@zvec/zvec-grep 0.1.5 → 0.1.7

package/docs/06-server.md

@@ -0,0 +1,215 @@
+# Server and execution modes
+
+[Documentation](./README.md) · [Agents](./01-agents.md) ·
+[CLI](./02-cli.md) · [MCP](./03-mcp.md) · [Pipeline](./04-pipeline.md) ·
+[Architecture](./05-architecture.md) · [Server](./06-server.md) ·
+[Embedding](./07-embedding.md) · [Roadmap](./08-roadmap.md)
+
+The zvec-grep Server is a local daemon shared by agents and terminal commands.
+It keeps the MCP endpoint available, coordinates active Workspace runtimes,
+supports background index refresh, and can reuse loaded Embedding models across
+requests.
+
+You do not need the Server for every use of zg. Exact managed ripgrep and
+indexed operations can also run directly in the current process.
+
+## Choose an execution mode
+
+Indexed CLI commands accept `--mode auto|server|direct`.
+
+| Mode | Use it when | Behavior |
+| --- | --- | --- |
+| `auto` | Almost all terminal use | Use a ready Server; otherwise run Direct before submitting the operation |
+| `server` | A script requires the daemon, shared state, or background refresh | Require the Server and fail if it is unavailable |
+| `direct` | One-off use, CI, foreground debugging, or no daemon is desired | Run entirely in the current process |
+
+`auto` is the default and the recommended mode for people. It does not start a
+missing Server; it simply chooses the Server when one is already ready.
+
+Use Server mode when:
+
+- an Agent connects through MCP;
+- indexed searches happen repeatedly across sessions;
+- background refresh should keep an active Workspace current;
+- multiple requests can benefit from a shared loaded model;
+- operational tooling needs an explicit ready/not-ready contract.
+
+Use Direct mode when:
+
+- a command is an isolated one-off operation;
+- a CI job should not leave a daemon running;
+- you want foreground failures and resource lifetime tied to one process.
+
+Managed `zg query --rg` does not need an index or loaded Embedding model. It
+runs locally regardless of whether a Server is available.
+
+## Agent setup
+
+`zg install` configures the selected Agent and starts the Server when possible:
+
+```bash
+zg install
+```
+
+Most Agent users therefore never need to run `zg server on` manually. Restart
+the Agent or open a new session after installation so it discovers the MCP
+endpoint.
+
+Set `ZVEC_GREP_INSTALL_SKIP_SERVER=1` only when another process manager will
+start the Server separately.
+
+See [Agent integrations](./01-agents.md) for managed configuration and
+[MCP](./03-mcp.md) for the exposed tools.
+
+## Server lifecycle
+
+Start the background daemon:
+
+```bash
+zg server on
+```
+
+Inspect process readiness, endpoint, PID, and MCP toolset:
+
+```bash
+zg server status
+zg server status --check-ready
+```
+
+`--check-ready` preserves normal output and exits non-zero unless the Server is
+ready, making it suitable for scripts and health checks.
+
+Stop the daemon gracefully:
+
+```bash
+zg server off
+```
+
+Run it in the foreground for logs or process supervision:
+
+```bash
+zg server run
+```
+
+Only one Server instance can own a given zvec-grep home. If a running Server
+uses the wrong MCP toolset, stop it before restarting with the new profile:
+
+```bash
+zg server off
+zg server on --mcp-toolset full
+```
+
+## Configure the mode
+
+Choose a mode for one command:
+
+```bash
+zg query --mode direct "root-local index discovery"
+zg status --mode server --check-ready
+```
+
+Set an environment default:
+
+```bash
+export ZVEC_GREP_MODE=auto
+```
+
+Or set `client.mode` in `~/.zvec-grep/config.json`:
+
+```json
+{
+  "version": 1,
+  "client": {
+    "mode": "auto"
+  }
+}
+```
+
+An explicit `--mode` wins over the environment and global configuration.
+
+## Refresh behavior
+
+The execution mode changes the default indexed-search refresh policy:
+
+| Policy | Server | Direct |
+| --- | --- | --- |
+| Default | `background` | `off` |
+| `--refresh background` | Return current results and schedule an update | Warn and behave as `off` |
+| `--refresh wait` | Wait for a fresh index | Update and wait in the current process |
+| `--refresh off` | Search without updating | Search without updating |
+
+Server searches may return useful results with `freshness: possibly_stale`
+while an update continues. Use the result immediately when it answers the
+question; use `--refresh wait` only when the latest file state is required.
+
+Watcher-reported path updates skip workspace-wide status scans. Because
+file-system watchers can silently miss events, the Server schedules an hourly
+full reconciliation probe; the next search uses it to scan the Workspace and
+repair index drift.
+
+## Endpoint and toolset
+
+The default endpoint is:
+
+```text
+http://127.0.0.1:7999/mcp
+```
+
+The Server only accepts loopback listen addresses. Change the loopback address
+or port with:
+
+```bash
+zg server on --listen 127.0.0.1:8999
+```
+
+Set `ZVEC_GREP_SERVER_URL` when a client should use a non-default configured
+endpoint.
+
+The default `agent` MCP toolset exposes only `zvec_grep_search`. Use
+`--mcp-toolset full` or `ZVEC_GREP_MCP_TOOLSET=full` to expose optional managed
+rg together with the index and status tools. See [MCP](./03-mcp.md) for the tool
+contract.
+
+## Bearer authentication
+
+Authentication is disabled by default because the Server is loopback-only. To
+require a token, provide at least 32 characters through the environment or a
+file:
+
+```bash
+export ZVEC_GREP_SERVER_TOKEN="replace-with-a-long-random-token"
+zg server on
+```
+
+```bash
+zg server on --token-file /secure/path/zvec-grep.token
+```
+
+Clients can use `ZVEC_GREP_SERVER_TOKEN` or `ZVEC_GREP_SERVER_TOKEN_FILE`.
+Supported Agent integrations can reference an environment-backed token:
+
+```bash
+zg install \
+  --target codex \
+  --mcp-token-env ZVEC_GREP_SERVER_TOKEN \
+  --yes
+```
+
+The MCP Bearer token protects the local Server. It does not configure an
+Embedding provider or authorize remote data transfer.
+
+## Logs and state
+
+The default daemon directory is `~/.zvec-grep/daemon/`. Its structured JSON
+Lines log is written to:
+
+```text
+~/.zvec-grep/daemon/logs/server.log
+```
+
+Credential, authorization, token, API-key, and query fields are filtered from
+daemon log records. Repository identities are logged opaquely rather than as
+raw paths where identity is sufficient.
+
+Use `ZVEC_GREP_HOME` to relocate Server state. Check `zg server status` before
+reading logs; routine searches do not need a status preflight.

package/docs/07-embedding.md

@@ -0,0 +1,182 @@
+# Embedding models
+
+[Documentation](./README.md) · [Agents](./01-agents.md) ·
+[CLI](./02-cli.md) · [MCP](./03-mcp.md) · [Pipeline](./04-pipeline.md) ·
+[Architecture](./05-architecture.md) · [Server](./06-server.md) ·
+[Embedding](./07-embedding.md) · [Roadmap](./08-roadmap.md)
+
+The Embedding model determines the vector representation used by indexed
+search. It affects language coverage, memory use, index size, input length, and
+indexing speed. A new index needs an explicit model, an environment default, or
+a configured default:
+
+```bash
+zg index --embedding local/potion-code-16m-v2
+```
+
+Local models keep workspace content and query text on the machine. Their files
+are downloaded on first use and cached under `~/.zvec-grep/models` by default.
+Remote models avoid local inference but send disclosed query or workspace
+content to the configured provider after authorization.
+
+## Quick selection
+
+| Need | Start with | Why |
+| --- | --- | --- |
+| A fast first index for a code repository | `local/potion-code-16m-v2` | Small static Model2Vec model with a 1,024-token input limit |
+| Fast English document retrieval | `local/potion-retrieval-32m` | Retrieval-tuned static model with 512-dimensional vectors |
+| Fast multilingual document retrieval | `local/potion-multilingual-128m` | Static model trained for 101 languages with compact 256-dimensional vectors |
+| A Transformer model specialized for code | `local/jina-embeddings-v2-base-code` | Code-oriented, multilingual, and long-context |
+| General multilingual code and documents | `local/embeddinggemma-300m` | Broad language coverage in a local GGUF model |
+| A smaller multilingual model | `local/multilingual-e5-small` | Compact 384-dimensional Transformer model |
+| A lightweight English model | `local/all-minilm-l6-v2` | Small local model for short English text |
+| Long English documents | `local/gte-modernbert-base` or `local/nomic-embed-text-v1.5` | 8,192-token local context |
+| No local model runtime | `qwen/qwen3.7-text-embedding` | Managed text Embedding API |
+| Text and image retrieval | `qwen/qwen3-vl-embedding` | Managed multimodal Embedding API |
+
+The best model still depends on the repository and its real queries. Start with
+the smallest model that covers the required languages and input length, then
+compare representative results before committing to a larger model.
+
+## Supported models
+
+The input limit applies to each extracted entity or fragment, not the entire
+file.
+
+| Model | Runtime | Max input tokens | Dimensions |
+| --- | --- | ---: | ---: |
+| `local/potion-code-16m-v2` | Model2Vec FP16 | 1,024 | 256 |
+| `local/potion-retrieval-32m` | Model2Vec FP32 | 1,024 | 512 |
+| `local/potion-multilingual-128m` | Model2Vec FP32 | 1,024 | 256 |
+| `local/all-minilm-l6-v2` | ONNX Q4 | 256 | 384 |
+| `local/bge-small-en-v1.5` | ONNX Q4 | 512 | 384 |
+| `local/multilingual-e5-small` | ONNX Q8 | 512 | 384 |
+| `local/jina-embeddings-v2-base-code` | ONNX Q8 | 8,192 | 768 |
+| `local/gte-modernbert-base` | ONNX Q4 | 8,192 | 768 |
+| `local/nomic-embed-text-v1.5` | ONNX Q4 | 8,192 | 768 |
+| `local/embeddinggemma-300m` | GGUF Q8_0 | 2,048 | 768 |
+| `local/qwen3-embedding-0.6b` | GGUF Q8_0 | 8,192 | 1,024 |
+| `qwen/text-embedding-v4` | Remote text | 8,192 | 1,024 |
+| `qwen/qwen3.7-text-embedding` | Remote text | 128,000 | 1,024 |
+| `qwen/qwen3-vl-embedding` | Remote multimodal | 32,000 | 2,560 |
+
+All catalog entries currently use cosine similarity. Exact model revisions are
+pinned by zvec-grep so the same reference resolves consistently for a given
+release.
+
+## Configure a default model
+
+Set a default for new indexes:
+
+```bash
+zg config model set local/potion-code-16m-v2 --default
+zg index
+```
+
+`ZVEC_GREP_EMBEDDING` provides a process-level default that takes priority over
+the configured global default for new indexes:
+
+```bash
+export ZVEC_GREP_EMBEDDING=local/potion-code-16m-v2
+zg index
+```
+
+An existing index always reuses its stored provider, model, dimensions, and
+metric unless `--embedding` and `--rebuild` explicitly change them. `zg index`
+forwards the current CLI environment default in server and auto modes; direct
+MCP calls use the environment inherited by the daemon.
+
+## Local runtime and device
+
+Select a device for local Transformer and GGUF models:
+
+```bash
+zg index \
+  --embedding local/jina-embeddings-v2-base-code \
+  --device auto
+```
+
+Supported values are `auto`, `cpu`, `metal`, `vulkan`, and `cuda`. Save a model
+preference globally with:
+
+```bash
+zg config model set local/jina-embeddings-v2-base-code --device metal
+```
+
+The equivalent environment override is `ZVEC_GREP_DEVICE`. Model2Vec models
+such as Potion use static vector lookup, so selecting a GPU does not improve
+their runtime.
+
+Override the download cache with `--model-cache` or `ZVEC_GREP_MODEL_CACHE`:
+
+```bash
+zg index \
+  --embedding local/potion-code-16m-v2 \
+  --model-cache /path/to/model-cache
+```
+
+## Remote Embedding and authorization
+
+Configure the Qwen provider credential and, optionally, a model endpoint:
+
+```bash
+zg config provider set qwen --api-key "$DASHSCOPE_API_KEY"
+zg config model set qwen/text-embedding-v4 --default
+```
+
+One-off values can be passed directly or through `ZVEC_GREP_API_KEY` and
+`ZVEC_GREP_ENDPOINT`:
+
+```bash
+zg index \
+  --embedding qwen/text-embedding-v4 \
+  --api-key "$DASHSCOPE_API_KEY" \
+  --allow-remote
+```
+
+Credentials configure access to a provider; they do not authorize data
+transfer. `--allow-remote` authorizes Remote Embedding only for the current
+command. To create a signed Workspace grant shared by the CLI and MCP server:
+
+```bash
+zg auth grant \
+  --capability embedding \
+  --scope workspace \
+  --embedding qwen/text-embedding-v4
+
+zg auth status
+zg auth revoke
+```
+
+Before granting access, confirm that the workspace content is permitted to be
+sent to the selected provider and endpoint. MCP tool approval is separate from
+this data authorization.
+
+## Input length and truncation
+
+zvec-grep uses the selected model's input limit when it creates fragments, but
+characters and tokens do not map one-to-one across languages. If an extracted
+input still exceeds the model limit, the provider truncates it and the index
+records the count.
+
+Inspect the index after changing models or file scope:
+
+```bash
+zg status
+```
+
+Prefer a model with a larger input limit or narrow the indexed content when
+`truncated_fragments` is unexpectedly high.
+
+## Change models
+
+Vector spaces from different models are incompatible, even when their
+dimensions match. Rebuild explicitly when changing models:
+
+```bash
+zg index --rebuild --embedding local/jina-embeddings-v2-base-code
+```
+
+Changing a remote endpoint also requires a rebuild because the endpoint is part
+of the stored index schema. API key and local device changes affect runtime and
+do not require rebuilding the vectors.

package/docs/08-roadmap.md

@@ -0,0 +1,85 @@
+# Roadmap
+
+[Documentation](./README.md) · [Agents](./01-agents.md) ·
+[CLI](./02-cli.md) · [MCP](./03-mcp.md) · [Pipeline](./04-pipeline.md) ·
+[Architecture](./05-architecture.md) · [Server](./06-server.md) ·
+[Embedding](./07-embedding.md) · [Roadmap](./08-roadmap.md)
+
+zvec-grep is under active development. The destination is one dependable,
+local-first search layer that people and agents can use without choosing among
+separate retrieval tools. This roadmap describes direction, not promised dates;
+priorities may change as we learn from real workloads.
+
+## Now — make the public preview dependable
+
+- Make installation, updates, and removal reliable across current platforms and
+  agent integrations.
+- Harden incremental indexing, freshness, Server recovery, and concurrent
+  workspace access.
+- Establish reproducible search-quality, performance, and agent-context
+  evaluation.
+- Stabilize the CLI, MCP contracts, configuration, diagnostics, and index
+  compatibility policy before a stable release.
+
+## Product direction
+
+The following four directions are part of the product vision, not optional
+experiments. Their sequencing may change, but each is required for zg to become
+the search layer we want it to be.
+
+### 1. Search more data formats, natively
+
+- Make PDF, PowerPoint (`.ppt` and `.pptx`), HTML, images, and other multimodal
+  documents first-class searchable inputs.
+- Add format-aware extraction that preserves useful structure, layout, metadata,
+  and relationships instead of flattening every file into plain text.
+- Combine text extraction, OCR, vision-language understanding, and multimodal
+  Embedding where each format benefits from them.
+- Use format- and content-aware retrieval so zg can choose a smarter strategy
+  for each query and source.
+
+### 2. Strengthen retrieval
+
+- Add knowledge-graph construction and graph retrieval to complement BM25,
+  vector search, and managed ripgrep.
+- Expand multi-route hybrid retrieval across lexical, vector, graph, structural,
+  and metadata signals.
+- Improve query planning so people and agents can express intent without
+  manually choosing retrieval routes.
+- Improve fusion, reranking, evaluation, and explainability while keeping the
+  returned context compact.
+
+### 3. Make zg more out of the box
+
+- Provide a local GUI for search, workspace management, indexing, model setup,
+  permissions, and diagnostics.
+- Support more installation paths beyond npm, including platform-native package
+  managers and installers where appropriate.
+- Make first-run setup, Agent discovery, model selection, updates, and recovery
+  increasingly automatic with useful defaults.
+- Preserve CLI and configuration control for advanced users while removing it
+  from the critical path for everyone else.
+
+### 4. Reach every platform, from PC to mobile
+
+- Deliver a consistent experience across macOS, Windows, and Linux desktops.
+- Extend the local search layer to iOS and Android instead of treating mobile as
+  a remote client to a required cloud service.
+- Adapt indexing, storage, and model execution to the memory, power, and lifecycle
+  constraints of mobile devices.
+- Keep the same local-first trust and permission model across desktop and mobile.
+
+## Guardrails
+
+- Local-first remains the default; remote content transfer always requires
+  explicit authorization.
+- zg should hide retrieval-tool choice from users without hiding useful control.
+- Better recall must not come at the cost of noisy, oversized agent context.
+- A hosted service must never be required for the core local workflow.
+
+## Help shape the roadmap
+
+Priorities should come from real use. Open a
+[GitHub issue](https://github.com/zvec-ai/zvec-grep/issues) to describe a
+workflow, limitation, or result that matters to you. Contributions are welcome;
+see the [Contributing Guide](../CONTRIBUTING.md).

package/docs/README.md

@@ -0,0 +1,53 @@
+# zvec-grep documentation
+
+**zg** is one local-first search layer for people and agents. These guides go
+beyond the short path in the project [README](../README.md) and explain how to
+connect an agent, shape an index, search it, and control the underlying
+interfaces.
+
+> [!IMPORTANT]
+> zvec-grep is a work in progress. Commands and configuration may change before
+> the first stable release.
+
+## Start here
+
+| I want to… | Read |
+| --- | --- |
+| Connect Codex, Claude Code, Cursor, or OpenCode | [Agent integrations](./01-agents.md) |
+| Use zg directly from a terminal | [CLI guide](./02-cli.md) |
+| Understand the tools exposed to an agent | [MCP guide](./03-mcp.md) |
+| Understand indexing, updates, and search routes | [Retrieval pipeline](./04-pipeline.md) |
+| See how the components and trust boundaries fit together | [Architecture](./05-architecture.md) |
+| Choose between Auto, Server, and Direct execution | [Server and execution modes](./06-server.md) |
+| Choose and configure an Embedding model | [Embedding models](./07-embedding.md) |
+| See what is stable now and what comes next | [Roadmap](./08-roadmap.md) |
+
+## Recommended paths
+
+If you primarily use an agent, start with [Agent integrations](./01-agents.md),
+then read the [MCP guide](./03-mcp.md) and
+[Retrieval pipeline](./04-pipeline.md). The [Server guide](./06-server.md)
+explains the daemon that connects them.
+
+If you primarily use the terminal, start with the [CLI guide](./02-cli.md), then
+read the [Retrieval pipeline](./04-pipeline.md). Use the
+[Embedding guide](./07-embedding.md) when creating a new index. The default
+`auto` execution mode is explained in
+[Server and execution modes](./06-server.md).
+
+For the whole-system mental model and trust boundaries, read
+[Architecture](./05-architecture.md).
+
+The [Roadmap](./08-roadmap.md) tracks the path from work in progress to a stable
+release.
+
+The CLI remains the source of truth for flags in the installed version:
+
+```bash
+zg help
+zg help query
+zg help index
+```
+
+For development setup and pull request conventions, see
+[CONTRIBUTING.md](../CONTRIBUTING.md).

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@zvec/zvec-grep",
-  "version": "0.1.5",
-  "description": "Agent-friendly hybrid code search with indexed context and managed ripgrep.",
+  "version": "0.1.7",
+  "description": "Agent-friendly hybrid workspace search across code and non-code content.",
   "type": "module",
   "license": "Apache-2.0",
   "repository": {
@@ -31,8 +31,8 @@
     "README.md",
     "README_CN.md",
     "LICENSE",
-    "dist",
-    "skills/zvec-grep"
+    "docs",
+    "dist"
   ],
   "scripts": {
     "build": "tsc -p tsconfig.json",
@@ -41,17 +41,21 @@
     "lint": "eslint .",
     "prepack": "npm run build",
     "test:coverage": "npm run build && c8 --all --include=dist/**/*.js --check-coverage --lines=80 --statements=80 --functions=80 --branches=80 npm run test:all",
-    "test:all": "node --test --test-concurrency=1 test/*.test.mjs test/unit/*.test.mjs test/integration/*.test.mjs test/e2e/*.test.mjs",
-    "test:e2e": "npm run build && node --test --test-concurrency=1 test/e2e/*.test.mjs",
-    "test:integration": "npm run build && node --test --test-concurrency=1 test/*.test.mjs test/integration/*.test.mjs",
+    "test:all": "npm run test:run:root && npm run test:run:unit && npm run test:run:integration && npm run test:run:e2e",
+    "test:e2e": "npm run build && npm run test:run:e2e",
+    "test:integration": "npm run build && npm run test:run:root && npm run test:run:integration",
     "test:package": "npm run build && node --test --test-concurrency=1 test/package/*.test.mjs",
-    "test:unit": "npm run build && node --test test/unit/*.test.mjs",
+    "test:run:e2e": "node --test --test-concurrency=1 test/e2e/*.test.mjs",
+    "test:run:integration": "node --test --test-concurrency=1 test/integration/*.test.mjs",
+    "test:run:root": "node --test --test-concurrency=1 test/*.test.mjs",
+    "test:run:unit": "node --test --test-concurrency=1 test/unit/*.test.mjs test/unit/extraction/*.test.mjs test/unit/models/*.test.mjs",
+    "test:unit": "npm run build && npm run test:run:unit",
     "typecheck": "tsc -p tsconfig.json --noEmit",
-    "workflow:check": "node scripts/check-workflows.mjs",
     "test": "npm run build && npm run test:all"
   },
   "devDependencies": {
     "@eslint/js": "^10.0.1",
+    "@modelcontextprotocol/conformance": "0.1.16",
     "@types/node": "^22.20.1",
     "c8": "^11.0.0",
     "eslint": "^10.7.0",
@@ -61,12 +65,17 @@
     "typescript-eslint": "^8.64.0"
   },
   "dependencies": {
-    "@modelcontextprotocol/sdk": "^1.29.0",
+    "@huggingface/tokenizers": "^0.1.3",
+    "@huggingface/transformers": "^3.8.1",
+    "@modelcontextprotocol/client": "2.0.0",
+    "@modelcontextprotocol/core": "2.0.0",
+    "@modelcontextprotocol/node": "2.0.0",
+    "@modelcontextprotocol/server": "2.0.0",
     "@vscode/ripgrep": "^1.18.0",
     "@zvec/zvec": "^0.5.0",
     "tree-sitter-wasms": "^0.1.13",
     "web-tree-sitter": "^0.20.8",
-    "zod": "^3.25.76"
+    "zod": "4.2.0"
   },
   "optionalDependencies": {
     "node-llama-cpp": "3.18.1"

package/dist/cli/mcp.d.ts

@@ -1,2 +0,0 @@
-import { type CreateZvecGrepOptions } from "../index.js";
-export declare function runMcpServer(options: CreateZvecGrepOptions): Promise<void>;