@angri450/zvec-grep 0.2.2

package/docs/06-server.md

@@ -0,0 +1,279 @@
+# Server and execution modes
+
+[Documentation](./README.md) · [Agents](./01-agents.md) ·
+[CLI](./02-cli.md) · [MCP](./03-mcp.md) · [Pipeline](./04-pipeline.md) ·
+[Architecture](./05-architecture.md) · [Server](./06-server.md) ·
+[Embedding](./07-embedding.md) · [Roadmap](./08-roadmap.md)
+
+The zvec-grep Server is a local daemon shared by agents and terminal commands.
+It keeps the MCP endpoint available, coordinates active Workspace runtimes,
+supports background index refresh, and can reuse loaded Embedding models across
+requests.
+
+You do not need the Server for every use of zg. Exact managed ripgrep and
+indexed operations can also run directly in the current process.
+
+## Choose an execution mode
+
+Indexed CLI commands accept `--mode auto|server|direct`.
+
+| Mode | Use it when | Behavior |
+| --- | --- | --- |
+| `auto` | Almost all terminal use | Use a ready Server; otherwise run Direct before submitting the operation |
+| `server` | A script requires the daemon, shared state, or background refresh | Require the Server and fail if it is unavailable |
+| `direct` | One-off use, CI, foreground debugging, or no daemon is desired | Run entirely in the current process |
+
+`auto` is the default and the recommended mode for people. It does not start a
+missing Server; it simply chooses the Server when one is already ready.
+
+Use Server mode when:
+
+- an Agent connects through MCP;
+- indexed searches happen repeatedly across sessions;
+- background refresh should keep an active Workspace current;
+- multiple requests can benefit from a shared loaded model;
+- operational tooling needs an explicit ready/not-ready contract.
+
+Use Direct mode when:
+
+- a command is an isolated one-off operation;
+- a CI job should not leave a daemon running;
+- you want foreground failures and resource lifetime tied to one process.
+
+Managed `zg --rg` does not need an index or loaded Embedding model. It
+runs locally regardless of whether a Server is available.
+
+## Agent setup
+
+`zg --install` configures the selected Agent and starts the Server when possible:
+
+```bash
+zg --install
+```
+
+Most Agent users therefore never need to run `zg --server on` manually. Restart
+the Agent or open a new session after installation so it discovers the MCP
+endpoint.
+
+Set `ZVEC_GREP_INSTALL_SKIP_SERVER=1` only when another process manager will
+start the Server separately.
+
+See [Agent integrations](./01-agents.md) for managed configuration and
+[MCP](./03-mcp.md) for the exposed tools.
+
+## Server lifecycle
+
+Start the background daemon:
+
+```bash
+zg --server on
+```
+
+Inspect process readiness, endpoint, PID, and MCP toolset:
+
+```bash
+zg --server status
+zg --server status --check-ready
+```
+
+`--check-ready` preserves normal output and exits non-zero unless the Server is
+ready, making it suitable for scripts and health checks.
+
+Stop the daemon gracefully:
+
+```bash
+zg --server off
+```
+
+The shutdown endpoint accepts native clients without an `Origin` header. When
+`Origin` is present, its scheme, hostname, and port must match the request's
+loopback authority, and that authority must use the daemon's actual listening
+port. Pages on other local ports cannot request shutdown. Opaque (`null`), empty,
+duplicate, and malformed origins are rejected before shutdown runs. Optional
+Bearer authentication still applies to every shutdown request when configured;
+native CLI and agent clients need no additional configuration.
+
+MCP endpoints retain support for other HTTP loopback origins. Both MCP and
+shutdown reject malformed origins and invalid or duplicate Host authorities.
+
+Run it in the foreground for logs or process supervision:
+
+```bash
+zg --server run
+```
+
+Only one Server instance can own a given zvec-grep home. If a running Server
+uses the wrong MCP toolset, stop it before restarting with the new profile:
+
+```bash
+zg --server off
+zg --server on --mcp-toolset full
+```
+
+The Server releases a Workspace watcher and its lightweight runtime after four
+hours without a client request or a relevant file-system change. Periodic
+reconciliation does not extend this idle deadline. To select another timeout,
+set the number of seconds before starting or restarting the Server:
+
+```bash
+export ZVEC_GREP_WATCHER_IDLE_TIMEOUT_SECONDS=7200
+zg server off
+zg server on
+```
+
+Set the value to `0` to keep activated watchers until the Server stops.
+
+## Configure the mode
+
+Choose a mode for one command:
+
+```bash
+zg --mode direct "root-local index discovery"
+zg --status --mode server --check-ready
+```
+
+Set an environment default:
+
+```bash
+export ZVEC_GREP_MODE=auto
+```
+
+Or set `client.mode` in `~/.zvec-grep/config.json`:
+
+```json
+{
+  "version": 1,
+  "client": {
+    "mode": "auto"
+  }
+}
+```
+
+An explicit `--mode` wins over the environment and global configuration.
+
+## Refresh behavior
+
+The execution mode changes the default indexed-search refresh policy:
+
+| Policy | Server | Direct |
+| --- | --- | --- |
+| Default | `background` | `off` |
+| `--refresh background` | Return current results and schedule an update | Warn and behave as `off` |
+| `--refresh wait` | Wait for a fresh index | Update and wait in the current process |
+| `--refresh off` | Search without updating | Search without updating |
+
+Server searches return `freshness: possibly_stale` only with evidence of index
+drift. The first refresh after activation may use this conservative status; an
+hourly reconciliation remains `fresh` until its probe finds a mismatch. Use
+`--refresh wait` only when the latest file state is required.
+
+Watcher-reported path updates skip workspace-wide status scans. Because
+file-system watchers can silently miss events, the Server schedules an hourly
+full reconciliation probe; the next search uses it to scan the Workspace and
+repair index drift.
+
+Large bursts of exact watcher events are compacted into directory-scoped
+updates. A full reconciliation is reserved for watcher errors, missing event
+paths, resume drift, and other cases where events may have been lost.
+
+## Endpoint and toolset
+
+The default endpoint is:
+
+```text
+http://127.0.0.1:7999/mcp
+```
+
+The Server only accepts loopback listen addresses. Change the loopback address
+or port with:
+
+```bash
+zg --server on --listen 127.0.0.1:8999
+```
+
+Set `ZVEC_GREP_SERVER_URL` when a client should use a non-default configured
+endpoint.
+
+The default `agent` MCP toolset exposes only `zvec_grep_search`. Use
+`--mcp-toolset full` or `ZVEC_GREP_MCP_TOOLSET=full` to expose optional managed
+rg together with the index and status tools. See [MCP](./03-mcp.md) for the tool
+contract.
+
+## Bearer authentication
+
+Authentication is disabled by default because the Server is loopback-only. To
+require a token, provide at least 32 characters through the environment or a
+file:
+
+```bash
+export ZVEC_GREP_SERVER_TOKEN="replace-with-a-long-random-token"
+zg --server on
+```
+
+```bash
+zg --server on --token-file /secure/path/zvec-grep.token
+```
+
+Clients can use `ZVEC_GREP_SERVER_TOKEN` or `ZVEC_GREP_SERVER_TOKEN_FILE`.
+Supported Agent integrations can reference an environment-backed token:
+
+```bash
+zg --install \
+  --target codex \
+  --mcp-token-env ZVEC_GREP_SERVER_TOKEN \
+  --yes
+```
+
+The MCP Bearer token protects the local Server. It does not configure an
+Embedding provider or authorize remote data transfer.
+
+## Logs and state
+
+The default daemon directory is `~/.zvec-grep/daemon/`. Its structured JSON
+Lines log is written to:
+
+```text
+~/.zvec-grep/daemon/logs/server.log
+```
+
+Credential, authorization, token, API-key, and query fields are filtered from
+daemon log records. Repository identities are logged opaquely rather than as
+raw paths where identity is sufficient.
+
+Use `ZVEC_GREP_HOME` to relocate Server state. Check `zg --server status` before
+reading logs; routine searches do not need a status preflight.
+
+## Daemon logs
+
+The daemon writes JSON lines to `~/.zvec-grep/daemon/logs/server.log`.
+Size-based rotation is enabled by default: 10 MiB per file and five backups,
+numbered `server.log.1` (newest) through `server.log.5` (oldest). The oldest
+backup is removed on rotation. File writes and rotation use `rotating-file-stream`.
+Surplus numbered backups are removed when the logger opens after reducing `keep`.
+Existing logs are checked on the first write
+following a restart. Rotation happens after a complete record brings the file to
+or above the limit; the new active file may therefore be empty. Each JSON record
+is kept intact, so a rotated file may exceed the limit by one record.
+
+Configure logging in `~/.zvec-grep/config.json`, preserving your other settings:
+
+```json
+{
+  "version": 1,
+  "log": {
+    "maxBytes": 10485760,
+    "keep": 5,
+    "level": "info"
+  }
+}
+```
+
+`maxBytes` must be a positive safe integer; `keep` must be a non-negative safe
+integer and counts backups in addition to the active file. Setting `keep` to
+`0` removes the temporary backup after rotation completes.
+Restart the daemon after changing these settings.
+
+Successful `GET /healthz` completion events have level `debug` and are omitted
+at the default `info` level. Set `level` to `debug` to include them for diagnostics.
+Unsuccessful health checks and other requests continue to be logged at `info`.
+Each record includes a `level` field alongside the existing event fields.

package/docs/07-embedding.md

@@ -0,0 +1,227 @@
+# Embedding models
+
+[Documentation](./README.md) · [Agents](./01-agents.md) ·
+[CLI](./02-cli.md) · [MCP](./03-mcp.md) · [Pipeline](./04-pipeline.md) ·
+[Architecture](./05-architecture.md) · [Server](./06-server.md) ·
+[Embedding](./07-embedding.md) · [Roadmap](./08-roadmap.md)
+
+The Embedding model determines the vector representation used by indexed
+search. It affects language coverage, memory use, index size, input length, and
+indexing speed. A new index selects an explicit model, an environment default,
+or a configured default, and otherwise uses the built-in local default:
+
+```bash
+zg --index --embedding local/potion-code-16m-v2
+```
+
+Local models keep workspace content and query text on the machine. Their files
+are downloaded from Hugging Face on first use and cached under
+`~/.zvec-grep/models` by default. If a pinned Hugging Face artifact cannot be
+downloaded, zvec-grep automatically falls back to its pinned, integrity-checked
+ModelScope copy. A complete ModelScope snapshot is reused on later runs without
+retrying Hugging Face. ModelScope downloads use a separate `modelscope` cache
+subdirectory.
+
+Remote models avoid local inference but send disclosed query or workspace
+content to the configured provider after authorization.
+
+A first search without an index always creates one with a local model. It
+respects a configured local default but does not implicitly use a remote
+default. Select and authorize a remote model explicitly with `zg --index`.
+
+## Quick selection
+
+| Need | Start with | Why |
+| --- | --- | --- |
+| A fast first index for a code repository | `local/potion-code-16m-v2` | Small static Model2Vec model with a 1,024-token input limit |
+| Fast English document retrieval | `local/potion-retrieval-32m` | Retrieval-tuned static model with 512-dimensional vectors |
+| Fast multilingual document retrieval | `local/potion-multilingual-128m` | Static model trained for 101 languages with compact 256-dimensional vectors |
+| A Transformer model specialized for code | `local/jina-embeddings-v2-base-code` | Code-oriented, multilingual, and long-context |
+| General multilingual code and documents | `local/embeddinggemma-300m` | Broad language coverage in a local GGUF model |
+| A smaller multilingual model | `local/multilingual-e5-small` | Compact 384-dimensional Transformer model |
+| A lightweight English model | `local/all-minilm-l6-v2` | Small local model for short English text |
+| Long English documents | `local/gte-modernbert-base` or `local/nomic-embed-text-v1.5` | 8,192-token local context |
+| No local model runtime | `qwen/qwen3.7-text-embedding` | Managed text Embedding API |
+| Text and image retrieval | `qwen/qwen3-vl-embedding` | Managed multimodal Embedding API |
+
+The best model still depends on the repository and its real queries. Start with
+the smallest model that covers the required languages and input length, then
+compare representative results before committing to a larger model.
+
+## Supported models
+
+The input limit applies to each extracted entity or fragment, not the entire
+file.
+
+| Model | Runtime | Max input tokens | Dimensions |
+| --- | --- | ---: | ---: |
+| `local/potion-code-16m-v2` | Model2Vec FP16 | 1,024 | 256 |
+| `local/potion-retrieval-32m` | Model2Vec FP32 | 1,024 | 512 |
+| `local/potion-multilingual-128m` | Model2Vec FP32 | 1,024 | 256 |
+| `local/all-minilm-l6-v2` | ONNX Q4 | 256 | 384 |
+| `local/bge-small-en-v1.5` | ONNX Q4 | 512 | 384 |
+| `local/multilingual-e5-small` | ONNX Q8 | 512 | 384 |
+| `local/jina-embeddings-v2-base-code` | ONNX Q8 | 8,192 | 768 |
+| `local/gte-modernbert-base` | ONNX Q4 | 8,192 | 768 |
+| `local/nomic-embed-text-v1.5` | ONNX Q4 | 8,192 | 768 |
+| `local/embeddinggemma-300m` | GGUF Q8_0 | 2,048 | 768 |
+| `local/qwen3-embedding-0.6b` | GGUF Q8_0 | 8,192 | 1,024 |
+| `qwen/text-embedding-v4` | Remote text | 8,192 | 1,024 |
+| `qwen/qwen3.7-text-embedding` | Remote text | 128,000 | 1,024 |
+| `qwen/qwen3-vl-embedding` | Remote multimodal | 32,000 | 2,560 |
+
+All catalog entries currently use cosine similarity. Exact model revisions are
+pinned by zvec-grep so the same reference resolves consistently for a given
+release.
+
+## Configure a default model
+
+Set a default for new indexes:
+
+```bash
+zg --config model set local/potion-code-16m-v2 --default
+zg --index
+```
+
+`ZVEC_GREP_EMBEDDING` provides a process-level default that takes priority over
+the configured global default for new indexes:
+
+```bash
+export ZVEC_GREP_EMBEDDING=local/potion-code-16m-v2
+zg --index
+```
+
+An existing index always reuses its stored provider, model, dimensions, and
+metric unless `--embedding` and `--rebuild` explicitly change them. `zg --index`
+forwards the current CLI environment default in server and auto modes; direct
+MCP calls use the environment inherited by the daemon.
+
+## Local runtime and device
+
+Select a device for local Transformer and GGUF models:
+
+```bash
+zg --index \
+  --embedding local/jina-embeddings-v2-base-code \
+  --device auto
+```
+
+Supported values are `auto`, `cpu`, `metal`, `vulkan`, and `cuda`. Save a model
+preference globally with:
+
+```bash
+zg --config model set local/jina-embeddings-v2-base-code --device metal
+```
+
+The equivalent environment override is `ZVEC_GREP_DEVICE`. Model2Vec models
+such as Potion use static vector lookup, so selecting a GPU does not improve
+their runtime.
+
+For ONNX models using Transformers.js, `auto` uses the runtime's Node default
+(CPU). Select a GPU device explicitly when its hardware and runtime libraries
+are available. GGUF models retain their own automatic device selection.
+
+If Transformers.js cannot initialize a model, indexing stops instead of
+retrying the same load for every file. Correct the model or device configuration
+and restart the process or daemon before retrying; a failed first ONNX session
+can leave the runtime unusable for the rest of the process. For a GPU
+initialization error, restart and use `--device cpu` (or configure
+`ZVEC_GREP_DEVICE=cpu` in the daemon environment when no saved device overrides
+it). GPU inference failures after a successful initialization can still fall
+back to CPU. Model warnings are retained in the daemon log as `model.warning`
+events as well as sent through live progress.
+
+Override the download cache with `--model-cache` or `ZVEC_GREP_MODEL_CACHE`:
+
+```bash
+zg --index \
+  --embedding local/potion-code-16m-v2 \
+  --model-cache /path/to/model-cache
+```
+
+## Remote Embedding and authorization
+
+Configure the Qwen provider credential and, optionally, a model endpoint:
+
+```bash
+zg --config provider set qwen --api-key "$DASHSCOPE_API_KEY"
+zg --config model set qwen/text-embedding-v4 --default
+```
+
+One-off values can be passed directly or through `ZVEC_GREP_API_KEY` and
+`ZVEC_GREP_ENDPOINT`:
+
+```bash
+zg --index \
+  --embedding qwen/text-embedding-v4 \
+  --api-key "$DASHSCOPE_API_KEY" \
+  --allow-remote
+```
+
+Credentials configure access to a provider; they do not authorize data
+transfer. `--allow-remote` authorizes Remote Embedding only for the current
+command. To create a signed Workspace grant shared by the CLI and MCP server:
+
+```bash
+zg --auth grant \
+  --capability embedding \
+  --scope workspace \
+  --embedding qwen/text-embedding-v4
+
+zg --auth status
+zg --auth revoke
+```
+
+Before granting access, confirm that the workspace content is permitted to be
+sent to the selected provider and endpoint. MCP tool approval is separate from
+this data authorization.
+
+When Qoder CLI reports that it has no handler for `elicitation/create`, or
+declines or cancels without displaying the form, the `AGENTS.md` guidance
+installed by `zg --install --target qoder` uses the exact `AskUserQuestion` tool
+as a compatibility path. It offers workspace approval, local FTS only, or
+cancel.
+
+Qoder IDE uses the exact native tool name `ask_user_question`. Because Qoder IDE
+does not provide a supported global Rules location, the installer does not
+write equivalent IDE guidance. In stdio mode, the zvec-grep bridge instead
+turns a missing `elicitation/create` handler into an actionable error directing
+the top-level IDE agent to present the same choices with `ask_user_question`.
+This path still requires an end-to-end smoke test in a real Qoder IDE session.
+
+Either path runs the Workspace grant only after explicit approval and retries
+the original MCP search once. Choosing local FTS instead removes
+vector-producing query routes, sets `autoUpdate` to `false`, and keeps retrieval
+local without refreshing the remote-embedding index; a headless session never
+grants access automatically. Neither question tool should collect a token, API
+key, or password. `--allow-remote` is not a substitute for this path because it
+authorizes only the current CLI command, not a later MCP retry.
+
+## Input length and truncation
+
+zvec-grep uses the selected model's input limit when it creates fragments, but
+characters and tokens do not map one-to-one across languages. If an extracted
+input still exceeds the model limit, the provider truncates it and the index
+records the count.
+
+Inspect the index after changing models or file scope:
+
+```bash
+zg --status
+```
+
+Prefer a model with a larger input limit or narrow the indexed content when
+`truncated_fragments` is unexpectedly high.
+
+## Change models
+
+Vector spaces from different models are incompatible, even when their
+dimensions match. Rebuild explicitly when changing models:
+
+```bash
+zg --index --rebuild --embedding local/jina-embeddings-v2-base-code
+```
+
+Changing a remote endpoint also requires a rebuild because the endpoint is part
+of the stored index schema. API key and local device changes affect runtime and
+do not require rebuilding the vectors.

package/docs/08-roadmap.md

@@ -0,0 +1,85 @@
+# Roadmap
+
+[Documentation](./README.md) · [Agents](./01-agents.md) ·
+[CLI](./02-cli.md) · [MCP](./03-mcp.md) · [Pipeline](./04-pipeline.md) ·
+[Architecture](./05-architecture.md) · [Server](./06-server.md) ·
+[Embedding](./07-embedding.md) · [Roadmap](./08-roadmap.md)
+
+zvec-grep is under active development. The destination is one dependable,
+local-first search layer that people and agents can use without choosing among
+separate retrieval tools. This roadmap describes direction, not promised dates;
+priorities may change as we learn from real workloads.
+
+## Now — make the public preview dependable
+
+- Make installation, updates, and removal reliable across current platforms and
+  agent integrations.
+- Harden incremental indexing, freshness, Server recovery, and concurrent
+  workspace access.
+- Establish reproducible search-quality, performance, and agent-context
+  evaluation.
+- Stabilize the CLI, MCP contracts, configuration, diagnostics, and index
+  compatibility policy before a stable release.
+
+## Product direction
+
+The following four directions are part of the product vision, not optional
+experiments. Their sequencing may change, but each is required for zg to become
+the search layer we want it to be.
+
+### 1. Search more data formats, natively
+
+- Make PDF, PowerPoint (`.ppt` and `.pptx`), HTML, images, and other multimodal
+  documents first-class searchable inputs.
+- Add format-aware extraction that preserves useful structure, layout, metadata,
+  and relationships instead of flattening every file into plain text.
+- Combine text extraction, OCR, vision-language understanding, and multimodal
+  Embedding where each format benefits from them.
+- Use format- and content-aware retrieval so zg can choose a smarter strategy
+  for each query and source.
+
+### 2. Strengthen retrieval
+
+- Add knowledge-graph construction and graph retrieval to complement BM25,
+  vector search, and managed ripgrep.
+- Expand multi-route hybrid retrieval across lexical, vector, graph, structural,
+  and metadata signals.
+- Improve query planning so people and agents can express intent without
+  manually choosing retrieval routes.
+- Improve fusion, reranking, evaluation, and explainability while keeping the
+  returned context compact.
+
+### 3. Make zg more out of the box
+
+- Provide a local GUI for search, workspace management, indexing, model setup,
+  permissions, and diagnostics.
+- Support more installation paths beyond npm, including platform-native package
+  managers and installers where appropriate.
+- Make first-run setup, Agent discovery, model selection, updates, and recovery
+  increasingly automatic with useful defaults.
+- Preserve CLI and configuration control for advanced users while removing it
+  from the critical path for everyone else.
+
+### 4. Reach every platform, from PC to mobile
+
+- Deliver a consistent experience across macOS, Windows, and Linux desktops.
+- Extend the local search layer to iOS and Android instead of treating mobile as
+  a remote client to a required cloud service.
+- Adapt indexing, storage, and model execution to the memory, power, and lifecycle
+  constraints of mobile devices.
+- Keep the same local-first trust and permission model across desktop and mobile.
+
+## Guardrails
+
+- Local-first remains the default; remote content transfer always requires
+  explicit authorization.
+- zg should hide retrieval-tool choice from users without hiding useful control.
+- Better recall must not come at the cost of noisy, oversized agent context.
+- A hosted service must never be required for the core local workflow.
+
+## Help shape the roadmap
+
+Priorities should come from real use. Open a
+[GitHub issue](https://github.com/zvec-ai/zvec-grep/issues) to describe a
+workflow, limitation, or result that matters to you. Contributions are welcome;
+see the [Contributing Guide](../CONTRIBUTING.md).

package/docs/README.md

@@ -0,0 +1,53 @@
+# zvec-grep documentation
+
+**zg** is one local-first search layer for people and agents. These guides go
+beyond the short path in the project [README](../README.md) and explain how to
+connect an agent, shape an index, search it, and control the underlying
+interfaces.
+
+> [!IMPORTANT]
+> zvec-grep is a work in progress. Commands and configuration may change before
+> the first stable release.
+
+## Start here
+
+| I want to… | Read |
+| --- | --- |
+| Connect Codex, Claude Code, Qwen Code, Cursor, or OpenCode | [Agent integrations](./01-agents.md) |
+| Use zg directly from a terminal | [CLI guide](./02-cli.md) |
+| Understand the tools exposed to an agent | [MCP guide](./03-mcp.md) |
+| Understand indexing, updates, and search routes | [Retrieval pipeline](./04-pipeline.md) |
+| See how the components and trust boundaries fit together | [Architecture](./05-architecture.md) |
+| Choose between Auto, Server, and Direct execution | [Server and execution modes](./06-server.md) |
+| Choose and configure an Embedding model | [Embedding models](./07-embedding.md) |
+| See what is stable now and what comes next | [Roadmap](./08-roadmap.md) |
+
+## Recommended paths
+
+If you primarily use an agent, start with [Agent integrations](./01-agents.md),
+then read the [MCP guide](./03-mcp.md) and
+[Retrieval pipeline](./04-pipeline.md). The [Server guide](./06-server.md)
+explains the daemon that connects them.
+
+If you primarily use the terminal, start with the [CLI guide](./02-cli.md), then
+read the [Retrieval pipeline](./04-pipeline.md). Use the
+[Embedding guide](./07-embedding.md) when creating a new index. The default
+`auto` execution mode is explained in
+[Server and execution modes](./06-server.md).
+
+For the whole-system mental model and trust boundaries, read
+[Architecture](./05-architecture.md).
+
+The [Roadmap](./08-roadmap.md) tracks the path from work in progress to a stable
+release.
+
+The CLI remains the source of truth for flags in the installed version:
+
+```bash
+zg --help
+zg --help search
+zg --help index
+```
+
+For development setup and pull request conventions, see
+[CONTRIBUTING.md](../CONTRIBUTING.md).