@zvec/zvec-grep 0.1.5 → 0.1.7

package/README.md

@@ -3,305 +3,223 @@
 </p>
 
 <div align="center">
+  <p>
+    <a href="./docs/08-roadmap.md"><img src="https://img.shields.io/badge/status-work%20in%20progress-F59E0B?style=for-the-badge" alt="Work in progress" /></a>
+  </p>
   <picture>
-    <source media="(prefers-color-scheme: dark)" srcset="https://zvec.oss-cn-hongkong.aliyuncs.com/logo/github_log_2.svg" />
-    <img src="https://zvec.oss-cn-hongkong.aliyuncs.com/logo/github_logo_1.svg" width="400" alt="zvec logo" />
+    <source media="(prefers-color-scheme: dark)" srcset="./.github/assets/zg-logo-dark.svg">
+    <img src="./.github/assets/zg-logo.svg" width="150" alt="zg logo" />
   </picture>
+  <p><strong>Know the words—or don’t. Just zg.</strong></p>
+  <p>The local-first search layer for humans and agents.</p>
+
+  <p>
+    <a href="https://www.npmjs.com/package/@zvec/zvec-grep"><img src="https://img.shields.io/npm/v/@zvec/zvec-grep.svg" alt="npm version" /></a>
+    <a href="https://github.com/zvec-ai/zvec-grep/actions/workflows/ci.yml"><img src="https://github.com/zvec-ai/zvec-grep/actions/workflows/ci.yml/badge.svg" alt="CI" /></a>
+    <a href="./LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="Apache 2.0 license" /></a>
+    <img src="https://img.shields.io/badge/node-%3E%3D22-blue.svg" alt="Node.js 22 or newer" />
+  </p>
+
+  <p>
+    <a href="#tour">🎬 <strong>Tour</strong></a> |
+    <a href="#features">💫 <strong>Features</strong></a> |
+    <a href="#try-it-yourself">🚀 <strong>Try it yourself</strong></a> |
+    <a href="./docs/README.md">📚 <strong>Docs</strong></a> |
+    <a href="#benchmarks">📊 <strong>Benchmarks</strong></a> |
+    <a href="#community">🤝 <strong>Community</strong></a>
+  </p>
 </div>
 
-<p align="center">
-  <a href="https://www.npmjs.com/package/@zvec/zvec-grep"><img src="https://img.shields.io/npm/v/@zvec/zvec-grep.svg" alt="npm Release"/></a>
-  <a href="https://github.com/zvec-ai/zvec-grep/actions/workflows/ci.yml"><img src="https://github.com/zvec-ai/zvec-grep/actions/workflows/ci.yml/badge.svg" alt="CI"/></a>
-  <a href="./LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License"/></a>
-  <img src="https://img.shields.io/badge/node-%3E%3D22-blue.svg" alt="Node.js >=22"/>
-  <img src="https://img.shields.io/badge/CLI-zg-2ea44f.svg" alt="zg CLI"/>
-</p>
-
-<p align="center">
-  <a href="#quickstart">🚀 <strong>Quickstart</strong></a> |
-  <a href="#features">💫 <strong>Features</strong></a> |
-  <a href="#installation">📦 <strong>Installation</strong></a> |
-  <a href="#models">🧠 <strong>Models</strong></a> |
-  <a href="#library-api">🛠️ <strong>Library API</strong></a>
-</p>
-
-**zvec-grep** is an agent-friendly hybrid code search tool built on Zvec. It gives repositories a root-local semantic index, combines vector search with full-text search, and keeps CLI output compact enough for AI agents while still offering rich terminal output for humans.
-
-The command is intentionally short:
-
-```bash
-zg query "where query auto update happens"
-```
-
-> [!IMPORTANT]
-> **Command interface**
->
-> - **Hybrid Code Search**: Query code with natural language, exact terms, or both in one command.
-> - **Explicit Index Lifecycle**: New repositories require `zg index --embedding <model>`; agents do not silently create indexes.
-> - **Automatic Refresh**: Existing anonymous indexes are checked and incrementally updated before normal queries.
-> - **Token-Efficient Output**: Agent output defaults to `--preview none`; `--human` defaults to full source previews.
-> - **No-Index Lexical Search**: `zg query --rg` provides managed ripgrep search without requiring an index.
-
-## <a id="features"></a>💫 Features
-
-- **Semantic + Lexical Retrieval**: Blend vector search and full-text search for code, docs, tests, scripts, and configuration.
-- **Root-Local Indexes**: Anonymous indexes live under `<repo>/.zvec-grep/`, so repository state stays with the repository.
-- **Agent-Ready Output**: Default output is grouped by file and keeps previews small to save tokens.
-- **Human Output Mode**: Add `--human` for a terminal-friendly summary with full previews by default.
-- **Managed ripgrep Route**: `zg query --rg` supports common `rg` flags and works even before a repository is indexed.
-- **Explicit Model Choice**: The first index build requires a model such as `local/embeddinggemma-300m`, `local/qwen3-embedding-0.6b`, or `qwen/text-embedding-v4`.
-- **Schema Reuse**: Re-running `zg index` on an existing index reuses the stored embedding schema unless you explicitly change it.
-- **MCP Server**: Run `zg serve --mcp` to expose indexed and no-index lexical search tools to MCP clients. Indexing and status remain CLI-only operations.
-- **Library API**: Use `createZvecGrep()` directly from Node.js tools, agents, or MCP servers.
-
-## <a id="installation"></a>📦 Installation
-
-Install the CLI from npm:
-
-```bash
-npm install -g @zvec/zvec-grep
-zg version
-```
-
-Or run it without a global install:
-
-```bash
-npx @zvec/zvec-grep help
-```
-
-Install the latest source checkout as a global `zg` command:
-
-```bash
-git clone https://github.com/zvec-ai/zvec-grep.git
-cd zvec-grep
-npm ci
-npm run build
-npm install -g .
-zg version
-```
-
-Run the stdio MCP server:
-
-```bash
-zg serve --mcp
-```
-
-Install the Codex MCP integration:
-
-```bash
-zg install --target codex --yes
-```
-
-Codex MCP tool calls default to a 600-second timeout. Override it during installation with `--mcp-tool-timeout <seconds>`.
+**zg** (**z**vec-**g**rep) unifies ripgrep, BM25, and vector search behind
+[one local-first interface](./docs/05-architecture.md). Use it directly from the
+terminal, or let your agent use it for you.
 
-### ✅ Requirements
+<a id="tour"></a>
 
-- Node.js 22 or newer
-- macOS, Linux, or Windows
-- A supported embedding model for indexed search
+## 🎬 See it in action
 
-`zg query --rg` works without any embedding model or index.
+<div align="center">
+  <img src="./.github/assets/zvec-grep-tour.gif" width="1000" alt="Install the agent integration, index a workspace, and let the agent search it with zvec-grep" />
+</div>
 
-## <a id="quickstart"></a>⚡ Quickstart
+Install the integration once, index the workspace, then search from the terminal
+or ask your agent naturally.
 
-Index a repository with an explicit embedding model:
+<a id="features"></a>
 
-```bash
-zg index \
-  --embedding local/embeddinggemma-300m \
-  -g "src/**" \
-  -g "docs/**" \
-  -g "test/**" \
-  -g "!dist/**" \
-  -g "!node_modules/**" \
-  -g "!coverage/**"
-```
+## 💫 Why zg?
 
-Check index state:
+- **Ready for humans and agents** — install once, index once, then use the same
+  workspace from the CLI or your agent on macOS, Linux, and Windows.
+- **Search beyond keywords** — discover by meaning, rank by relevance, then
+  verify with exact text or regex when needed.
+- **Multi-format search** — search source code, documents, and structured data
+  while preserving useful structure and source locations.
+- **Less searching, less context** — ranked, source-linked results surface the
+  right evidence with fewer tool calls, fewer tokens, and less noise.
+- **Local by default** — files, indexes, and local models stay on your machine;
+  remote embeddings receive data only with your permission.
 
-```bash
-zg status
-```
+<a id="try-it-yourself"></a>
 
-Search with natural language:
+## 🚀 Try it yourself
 
-```bash
-zg query "where query auto update happens"
-```
+Requires Node.js 22 or newer and a configured
+[OpenCode](https://opencode.ai/) installation. The agent example explicitly
+selects a free model, so it does not depend on your default provider.
 
-Combine semantic intent with exact anchors:
+### 1. Set up a sample bookshelf
 
 ```bash
-zg query "GPU fallback" --fts "usingCpuFallback" -g "src/**" -t ts --limit 5
-```
+npm install -g @zvec/zvec-grep
+zg install --target opencode --yes
 
-Use exhaustive lexical search without an index:
+mkdir zg-mystery && cd zg-mystery
+curl -L https://www.gutenberg.org/files/11/11-0.txt -o alice-in-wonderland.txt
+curl -L https://www.gutenberg.org/files/345/345-0.txt -o dracula.txt
+curl -L https://www.gutenberg.org/files/84/84-0.txt -o frankenstein.txt
+curl -L https://www.gutenberg.org/files/834/834-0.txt -o sherlock-holmes.txt
 
-```bash
-zg query --rg -F "ZVEC_GREP_HOME" src
+zg index --embedding local/potion-retrieval-32m
 ```
 
-Switch to human-readable output:
+This creates a small bookshelf of four public-domain novels and indexes it
+locally with Potion Retrieval.
 
-```bash
-zg query --human "root local index discovery" --limit 3
-```
+Installation starts the shared daemon immediately. The default MCP transport is
+stdio: the agent runs `zg server --stdio`, which safely starts or reuses it
+again after a reboot. Advanced users can keep direct HTTP configuration with
+`zg install --mcp-transport http` and `zg server on`.
 
-Use from an MCP client with `zvec_grep_search` and `zvec_grep_rg`. MCP inputs use JSON-friendly fields such as `include: ["src/**"]`; use the CLI for `zg status` and `zg index`. The Codex installer writes managed zvec-grep blocks to `${CODEX_HOME:-$HOME/.codex}/config.toml` and `${CODEX_HOME:-$HOME/.codex}/AGENTS.md`.
+Use `zg install --mcp-toolset full` when the installed MCP integration should
+also expose managed rg, index, and status tools. The default is `agent`.
 
-## <a id="models"></a>🧠 Models
+### 2. Choose how to search
 
-Local models run through `node-llama-cpp` and keep code search private to your machine:
+#### For agents: ask with OpenCode
 
 ```bash
-zg index --embedding local/embeddinggemma-300m
-zg index --embedding local/qwen3-embedding-0.6b
+opencode run --model opencode/deepseek-v4-flash-free \
+  "An unseen creature left a few marks. What did the detective infer? Cite local evidence."
 ```
 
-On Apple Silicon, local builds use quiet llama.cpp CMake defaults to avoid harmless OpenMP and ARM native-detection warnings. Override any llama.cpp CMake option with `NODE_LLAMA_CPP_CMAKE_OPTION_<name>`, for example `NODE_LLAMA_CPP_CMAKE_OPTION_GGML_NATIVE=ON` to opt back into native CPU tuning.
+OpenCode chooses zg on its own—the prompt does not name a tool. Query rewrites
+may vary by model, but a typical run looks like this:
 
-Remote Qwen embeddings are useful when you prefer a managed embedding service or want to avoid local model setup:
+```text
+⚙ zvec_grep_zvec_grep_search {"root":".../zg-mystery","query":"An unseen creature left a few marks. What did the detective infer?","fts":["marks","unseen creature","inferred","detective"],"fuse":true}
+→ Read sherlock-holmes.txt [offset=5440, limit=120]
 
-```bash
-zg index \
-  --embedding qwen/text-embedding-v4 \
-  --api-key "$DASHSCOPE_API_KEY"
-```
-
-After a successful index, explicitly passed global model and provider options are saved to `~/.zvec-grep/config.json` with user-only permissions. This lets an already-running `zg` MCP server load a newly configured remote API key on its next model request without restarting Codex. Values read only from environment variables are not persisted.
-
-```json
-{
-  "version": 1,
-  "defaults": {
-    "embedding": "qwen/text-embedding-v4"
-  },
-  "providers": {
-    "qwen": {
-      "apiKey": "...",
-      "endpoint": "https://dashscope.aliyuncs.com/compatible-mode/v1/embeddings"
-    }
-  }
-}
-```
+Answer:
+The detective is Sherlock Holmes in *The Adventure of the Crooked Man*.
+From the animal's foot-marks, he inferred:
 
-Resolution order is explicit CLI or library options, then environment variables, then global config, then built-in defaults. Repository roots and file-selection settings remain in each repository's `.zvec-grep` metadata rather than global config.
+- It was not a dog, cat, monkey, or other familiar creature: its print had
+  five foot-pads, long nails, and was nearly as large as a dessert-spoon
+  (sherlock-holmes.txt:5479-5486).
+- It had a long body and very short legs: nearly two feet long, but with a
+  stride of only about three inches (sherlock-holmes.txt:5497-5502).
+- It was carnivorous because it climbed the curtain toward a canary's cage
+  (sherlock-holmes.txt:5509-5510).
+- It was probably from the weasel or stoat tribe, though unusually large
+  (sherlock-holmes.txt:5514-5516).
 
-For existing indexes, `zg index` without `--embedding` reuses the stored schema. Use `--rebuild --embedding <model>` only when you intentionally want to rebuild with a different model:
-
-```bash
-zg index --rebuild --embedding local/qwen3-embedding-0.6b
+Holmes also inferred that a man entered through the window with it and later
+left with the key (sherlock-holmes.txt:5464-5470, 5527-5528).
 ```
 
-## 🔎 Query Patterns
+#### For humans: search directly
 
-Multiple quoted queries are treated as separate search groups:
+Search the same bookshelf directly, without an agent:
 
 ```bash
-zg query "request validation" "error handling" --limit 5
+zg query --human "An unseen creature left a few marks. What did the detective infer?" --limit 3
 ```
 
-Route options each consume one query and can be repeated. Add `--fuse` when
-you want all groups combined into one ranked result list:
-
-```bash
-zg query \
-  --hybrid "authentication flow" \
-  --fts "AuthService" \
-  --vector "where credentials are validated" \
-  --fuse \
-  --limit 10
-```
+zg returns the relevant passages from `sherlock-holmes.txt`, ranked ahead of the
+other novels.
 
-Use ripgrep-style path and file-type filters early to keep results focused.
-Positive globs include paths and globs beginning with `!` exclude them:
+<a id="benchmarks"></a>
 
-```bash
-zg query "cache invalidation" \
-  -g "src/**" \
-  -g "!test/**" \
-  -g "!tests/**" \
-  -g "!fixtures/**" \
-  -g "!dist/**" \
-  -t ts
-```
+## 📊 Benchmarks
 
-File-type filters are applied in addition to glob filters. For example,
-`-g "docs/**" -t ts` selects TypeScript files under `docs`, not every file in
-that directory.
+zg is a general-purpose retrieval tool. Current benchmarks cover code and text
+retrieval.
 
-Use `--preview` to control indexed source previews:
+### Repository code retrieval
 
-```bash
-zg query "plugin lifecycle" --preview none
-zg query "plugin lifecycle" --preview short --limit 5
-zg query "plugin lifecycle" --preview full --limit 2
-```
+We use [SWE-QA-Bench](https://github.com/peng-weihan/SWE-QA-Bench), which
+requires cross-file and multi-hop reasoning over real-world codebases.
 
-For exact text, symbols, flags, or error codes, use `--fts` or `--rg`:
+- **Coverage:** 20 retrieval-intensive tasks covering the paper's four
+  top-level categories—**What, Where, How, and Why**—across 8 intentions and
+  11 repositories.
+- **Agent:** Claude Code with **Claude Opus 5**.
+- **Embedding:** **Qwen3.7 Text Embedding** for the zg profile.
+- **Protocol:** 3 runs per task and profile.
 
-```bash
-zg query "authentication flow" --fts "AuthService" --fts "ForbiddenError"
-zg query --rg -i -C 2 -g "*.ts" "needle text" src
-```
+| Profile | Judge /100 ↑ | Input tokens ↓ | Tool calls ↓ | Time (s) ↓ | Cost ↓ |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| Native tools | 80.42 | 558,651 | 23.42 | 127.5 | $0.905 |
+| Native tools + zg | 81.92 | 294,262 | 9.70 | 79.7 | $0.558 |
+| Change | **+1.50 pp** | **−47.3%** | **−58.6%** | **−37.5%** | **−38.3%** |
 
-Managed rg mode accepts common ripgrep matching, context, type, glob, ignore,
-depth, size, symlink, encoding, and regex-engine options. Zvec-grep owns the
-result format, so output-changing modes such as `--json`, `--count`, `--files`,
-`-l`, `-o`, `--replace`, and `--vimgrep` are rejected. `.git/**` and
-`.zvec-grep/**` remain excluded.
+Across 20 tasks × 3 runs, zg increased the Judge score by 1.50 points while
+reducing input tokens by 47.3%, tool calls by 58.6%, execution time by 37.5%,
+and cost by 38.3%.
 
-## <a id="library-api"></a>🛠️ Library API
+### Large-scale text retrieval
 
-```ts
-import { createZvecGrep } from "@zvec/zvec-grep";
+We use [BrowseComp-Plus](https://github.com/texttron/BrowseComp-Plus), which
+requires multi-document evidence retrieval over a fixed web corpus. The
+evaluation includes 30 queries covering 10 mapped topics and three
+evidence-breadth levels, with 3 runs per profile.
 
-const zvecGrep = await createZvecGrep({
-  root: process.cwd(),
-});
+| Profile | Accuracy ↑ | Evidence recall ↑ | Search calls ↓ | Input tokens ↓ | Time (s) ↓ | Cost ↓ |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| Agent + BM25 | TBD | TBD | TBD | TBD | TBD | TBD |
+| Agent + zg | TBD | TBD | TBD | TBD | TBD | TBD |
+| Change | TBD | TBD | TBD | TBD | TBD | TBD |
 
-const result = await zvecGrep.context({
-  query: "ranking implementation",
-  limit: 5,
-});
+Across 30 queries × 3 runs, zg changed accuracy by TBD pp, evidence recall by
+TBD pp, search calls by TBD%, input tokens by TBD%, time by TBD%, and cost by
+TBD%.
 
-for (const item of result.items) {
-  const location =
-    item.range.kind === "text"
-      ? `${item.file.relativePath}:${item.range.startLine}`
-      : item.file.relativePath;
-  console.log(location);
-}
+To reproduce the evaluation, follow the
+[benchmark guide](./benchmarks/README.md) for environment setup, pinned
+revisions, task manifests, baseline configuration, exact commands, and result
+aggregation. All profiles use the same agent, model, prompt, tool budget, and
+three-run protocol.
 
-await zvecGrep.close();
-```
+## 📚 Documentation
 
-For explicit no-index lexical search:
+| Guide | What you can do |
+| :--- | :--- |
+| [Agent integrations](./docs/01-agents.md) | Connect zg to Codex, Claude Code, Cursor, or OpenCode and verify that it works. |
+| [CLI guide](./docs/02-cli.md) | Search, index, and manage your local workspaces from the terminal. |
+| [MCP guide](./docs/03-mcp.md) | Understand which zg tools your agent can use and how access is secured. |
+| [Retrieval pipeline](./docs/04-pipeline.md) | Choose what to index, keep it fresh, and get better search results. |
+| [Architecture](./docs/05-architecture.md) | See how zg handles your query and where your data stays. |
+| [Server and execution modes](./docs/06-server.md) | Choose between one-off commands and a long-running local server. |
+| [Embedding models](./docs/07-embedding.md) | Pick the right model for speed, search quality, privacy, and your hardware. |
+| [Roadmap](./docs/08-roadmap.md) | See what is coming next and help shape zg's priorities. |
 
-```ts
-const result = await zvecGrep.context({
-  query: "ZVEC_GREP_HOME",
-  rg: true,
-});
-```
+<a id="community"></a>
 
-## 🤝 Join the Zvec Community
+## 🤝 Join Our Community
 
 <div align="center">
 
-|                                                  💬 DingTalk                                                  |                                                   📱 WeChat                                                   |                                                                       🎮 Discord                                                                        |                                             X (Twitter)                                              |
-| :-----------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------: |
-| <img src="https://zvec.oss-cn-hongkong.aliyuncs.com/qrcode/dingding.png" width="150" alt="DingTalk QR Code"/> | <img src="https://zvec.oss-cn-hongkong.aliyuncs.com/qrcode/wechat.png?v=6" width="150" alt="WeChat QR Code"/> | [![Discord](https://img.shields.io/badge/Discord-Join%20Server-5865F2?style=for-the-badge&logo=discord&logoColor=white)](https://discord.gg/rKddFBBu9z) | [![X (formerly Twitter) Follow](https://img.shields.io/twitter/follow/ZvecAI)](https://x.com/ZvecAI) |
-|                                                 Scan to join                                                  |                                                 Scan to join                                                  |                                                                      Click to join                                                                      |                                           Click to follow                                            |
+| 💬 DingTalk | 📱 WeChat | 🎮 Discord | X (Twitter) |
+| :---: | :---: | :---: | :---: |
+| <img src="https://zvec.oss-cn-hongkong.aliyuncs.com/qrcode/dingding.png" width="150" alt="DingTalk QR Code"/> | <img src="https://zvec.oss-cn-hongkong.aliyuncs.com/qrcode/wechat.png?v1" width="150" alt="WeChat QR Code"/> | [![Discord](https://img.shields.io/badge/Discord-Join%20Server-5865F2?style=for-the-badge&logo=discord&logoColor=white)](https://discord.gg/rKddFBBu9z) | [![X (formerly Twitter) Follow](https://img.shields.io/twitter/follow/ZvecAI)](<https://x.com/ZvecAI>) |
+| Scan to join | Scan to join | Click to join | Click to follow |
 
 </div>
 
 ## ❤️ Contributing
 
-Issues and pull requests are welcome. Please keep changes focused, add tests for behavior changes, and run:
-
-```bash
-npm run check
-```
+Community contributions are always welcome—bug fixes, features, and
+documentation improvements all help make zvec-grep better.
 
-Pull request titles use Conventional Commits syntax.
+Check out our [Contributing Guide](./CONTRIBUTING.md) to get started!