@zvec/zvec-grep 0.1.6 → 0.1.8

package/docs/03-mcp.md

@@ -22,15 +22,33 @@ refresh, authentication, and logs.
 
 The default `agent` toolset intentionally exposes only search:
 
+Agents first decide whether the requested answer should be grounded in the
+current indexed workspace, then choose exact or semantic retrieval. The same
+rules apply to source code and non-code material such as documentation, books,
+research material, meeting notes, knowledge-base exports, manuals,
+configuration, and data.
+
+Workspace relevance requires a request to inspect, search, or ground the answer
+in local material, prior context that established local material as the intended
+source, or a question about whether relevant local material exists. Negative,
+incidental, or comparative workspace mentions do not establish relevance.
+
 | Tool | Use it when | Index required |
 | --- | --- | --- |
-| `zvec_grep_search` | The answer requires conceptual discovery, architecture, lifecycle, or cross-component flow | Yes |
+| `zvec_grep_search` | The answer is workspace-grounded and wording or location is unknown, or semantic, fuzzy, relationship, chronology, causality, comparison, or cross-file synthesis is required | Yes |
 
-Agents use native grep or rg when locating an exact word, symbol, filename, path,
-source fragment, or regex is sufficient. For mixed tasks, start with
-`zvec_grep_search`, then use native grep or rg for focused follow-up.
+Agents use native grep or rg when locating an exact word, quotation, name, date,
+key, filename, path, source fragment, or regex is sufficient. For mixed tasks,
+start with `zvec_grep_search`, then use native grep or rg for focused follow-up.
+When semantic discovery is selected because no sufficient exact anchor is
+available and the user asks whether conceptually related material exists
+locally, agents make at most one focused search probe and stop when its results
+are not relevant. The probe does not apply to exact quotations, configuration
+keys, filenames, regexes, or exhaustive occurrence requests. Unrelated
+open-world knowledge, current external facts, and web content that does not
+depend on local evidence use the appropriate external source instead.
 
-Every repository tool input uses an absolute `root` visible to the daemon.
+Every workspace tool input uses an absolute `root` visible to the daemon.
 
 ## `zvec_grep_search`
 
@@ -41,8 +59,8 @@ Minimal conceptual search:
 
 ```json
 {
-  "root": "/absolute/path/to/repository",
-  "query": "theme preference persistence on startup",
+  "root": "/absolute/path/to/workspace",
+  "query": "decision history behind the launch date",
   "limit": 5
 }
 ```
@@ -51,7 +69,7 @@ Explicit query routes and scope:
 
 ```json
 {
-  "root": "/absolute/path/to/repository",
+  "root": "/absolute/path/to/workspace",
   "query": "authentication flow",
   "fts": ["AuthService", "ForbiddenError"],
   "globs": ["src/**", "!src/generated/**"],
@@ -67,7 +85,7 @@ Important inputs:
 | --- | --- |
 | `query` | One hybrid natural-language or exact query |
 | `queries` | One or more hybrid query groups |
-| `fts` | Ranked lexical query groups |
+| `fts` | Ranked lexical constraints within indexed search, not exhaustive occurrence lookup |
 | `vector` | Semantic-only query groups |
 | `fuse` | Combine every group into one ranked plan |
 | `limit` | Maximum items per group, up to 50 |
@@ -110,7 +128,7 @@ arguments and is never executed by a shell:
 
 ```json
 {
-  "root": "/absolute/path/to/repository",
+  "root": "/absolute/path/to/workspace",
   "command": "rg -n -F 'loadTheme' -g '*.ts' src"
 }
 ```
@@ -120,7 +138,7 @@ requesting bounded output:
 
 ```json
 {
-  "root": "/absolute/path/to/repository",
+  "root": "/absolute/path/to/workspace",
   "command": "rg -n 'TODO|FIXME' src | head -50"
 }
 ```

package/docs/04-pipeline.md

@@ -67,6 +67,12 @@ File-type filters narrow the result after glob rules. For example,
 `-g "docs/**" -t ts` selects TypeScript files inside `docs`, not every file in
 that directory.
 
+Without an explicit `--max-filesize`, indexing uses type-aware safety limits:
+1 MiB for code, 256 MiB for text and Markdown, 16 MiB for structured data, and
+10 MiB for images. An explicit value replaces the type-aware defaults for every
+selected file. Files excluded by these limits remain silent during normal
+indexing; use `zg index --debug` to print skipped-file counts and samples.
+
 ### Supported formats and extraction
 
 The scanner assigns each admitted file to one extraction path. Structure-aware
@@ -200,8 +206,10 @@ between `auto`, `server`, `direct`, and `--refresh`.
 
 ## Output for agents and people
 
-Default CLI output is compact and grouped by file. Indexed source previews are
-omitted unless requested, reducing context passed to an agent:
+Default indexed CLI output is compact, grouped by query group, and ordered by
+that group's retrieval rank. A result recalled by several groups appears in
+each matching group. Indexed source previews are omitted unless requested,
+reducing context passed to an agent:
 
 ```bash
 zg query "plugin lifecycle" --preview short --limit 5

package/docs/06-server.md

@@ -142,6 +142,15 @@ Server searches may return useful results with `freshness: possibly_stale`
 while an update continues. Use the result immediately when it answers the
 question; use `--refresh wait` only when the latest file state is required.
 
+Watcher-reported path updates skip workspace-wide status scans. Because
+file-system watchers can silently miss events, the Server schedules an hourly
+full reconciliation probe; the next search uses it to scan the Workspace and
+repair index drift.
+
+Large bursts of exact watcher events are compacted into directory-scoped
+updates. A full reconciliation is reserved for watcher errors, missing event
+paths, resume drift, and other cases where events may have been lost.
+
 ## Endpoint and toolset
 
 The default endpoint is:

package/docs/07-embedding.md

@@ -23,7 +23,9 @@ content to the configured provider after authorization.
 
 | Need | Start with | Why |
 | --- | --- | --- |
-| A fast first index for a code repository | `local/potion-code-16m-v2` | Small static Model2Vec model with an 8,192-token input limit |
+| A fast first index for a code repository | `local/potion-code-16m-v2` | Small static Model2Vec model with a 1,024-token input limit |
+| Fast English document retrieval | `local/potion-retrieval-32m` | Retrieval-tuned static model with 512-dimensional vectors |
+| Fast multilingual document retrieval | `local/potion-multilingual-128m` | Static model trained for 101 languages with compact 256-dimensional vectors |
 | A Transformer model specialized for code | `local/jina-embeddings-v2-base-code` | Code-oriented, multilingual, and long-context |
 | General multilingual code and documents | `local/embeddinggemma-300m` | Broad language coverage in a local GGUF model |
 | A smaller multilingual model | `local/multilingual-e5-small` | Compact 384-dimensional Transformer model |
@@ -43,8 +45,9 @@ file.
 
 | Model | Runtime | Max input tokens | Dimensions |
 | --- | --- | ---: | ---: |
-| `local/potion-code-16m-v2` | Model2Vec FP16 | 8,192 | 256 |
-| `local/potion-base-8m` | Model2Vec FP32 | 512 | 256 |
+| `local/potion-code-16m-v2` | Model2Vec FP16 | 1,024 | 256 |
+| `local/potion-retrieval-32m` | Model2Vec FP32 | 1,024 | 512 |
+| `local/potion-multilingual-128m` | Model2Vec FP32 | 1,024 | 256 |
 | `local/all-minilm-l6-v2` | ONNX Q4 | 256 | 384 |
 | `local/bge-small-en-v1.5` | ONNX Q4 | 512 | 384 |
 | `local/multilingual-e5-small` | ONNX Q8 | 512 | 384 |

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@zvec/zvec-grep",
-  "version": "0.1.6",
-  "description": "Agent-friendly hybrid code search with indexed context and managed ripgrep.",
+  "version": "0.1.8",
+  "description": "Agent-friendly hybrid workspace search across code and non-code content.",
   "type": "module",
   "license": "Apache-2.0",
   "repository": {
@@ -32,8 +32,7 @@
     "README_CN.md",
     "LICENSE",
     "docs",
-    "dist",
-    "skills/zvec-grep"
+    "dist"
   ],
   "scripts": {
     "build": "tsc -p tsconfig.json",
@@ -46,6 +45,7 @@
     "test:e2e": "npm run build && npm run test:run:e2e",
     "test:integration": "npm run build && npm run test:run:root && npm run test:run:integration",
     "test:package": "npm run build && node --test --test-concurrency=1 test/package/*.test.mjs",
+    "test:smoke:local-model": "node --test --test-concurrency=1 test/smoke/local-model-package.test.mjs",
     "test:run:e2e": "node --test --test-concurrency=1 test/e2e/*.test.mjs",
     "test:run:integration": "node --test --test-concurrency=1 test/integration/*.test.mjs",
     "test:run:root": "node --test --test-concurrency=1 test/*.test.mjs",
@@ -66,6 +66,7 @@
     "typescript-eslint": "^8.64.0"
   },
   "dependencies": {
+    "@huggingface/tokenizers": "^0.1.3",
     "@huggingface/transformers": "^3.8.1",
     "@modelcontextprotocol/client": "2.0.0",
     "@modelcontextprotocol/core": "2.0.0",

package/skills/zvec-grep/SKILL.md

@@ -1,61 +0,0 @@
----
-name: zvec-grep
-description: Repository code search and indexing with zvec-grep. Use when exploring a codebase, locating symbols, references, or exact text, creating or inspecting a repository index, diagnosing the daemon, or whenever repository investigation would otherwise use grep, rg, or broad file reads.
----
-
-# zvec-grep
-
-## Select the transport
-
-Use zvec-grep indexed search for conceptual repository investigation. Use native
-`grep` or `rg` for exact lexical searches unless the optional managed-rg MCP tool
-is explicitly available.
-
-Use the public native HTTP MCP search tools as the primary interface when the matching `zvec_grep_*` tool is present. Call them directly; do not run `zg`, probe the daemon through shell, or choose CLI for convenience.
-
-Use CLI fallback only when one of these conditions is true:
-
-- The task requires index lifecycle or daemon diagnostics, which are intentionally
-  kept out of the default agent MCP toolset.
-- An available MCP search tool cannot perform the required operation.
-- MCP initialization, authentication, connection, or the required search call has
-  failed.
-- The repository has no index, the user has not authorized indexing, and an explicit no-index lexical search is needed.
-
-Do not retry a submitted indexing write through another transport after a connection interruption.
-
-## Use the MCP workflow
-
-Pass the repository's daemon-visible absolute path as `root` on every repository call.
-
-1. Use native `grep` or `rg` when an exact keyword, text, symbol, filename,
-   path, configuration key, error message, source fragment, literal, or regex
-   anchor is known and locating it is sufficient to answer the question.
-2. Call `zvec_grep_search` when the answer requires conceptual discovery,
-   architecture, lifecycle, data or control flow, comparison, design rationale,
-   or performance analysis. Search defaults to `freshness: "eventual"`; use
-   `freshness: "wait_for_fresh"` only when the result must include all pending
-   changes. Use hybrid `queries` for concepts plus known lexical constraints,
-   `fts` for indexed lexical-only intent, and `vector` for semantic-only intent.
-3. When exact symbols are present but the answer spans multiple components,
-   files, stages, or implementations, call `zvec_grep_search` first with the
-   concept and known symbols, then use native `grep` or `rg` for focused follow-up.
-4. Read the indexed search response's `freshness` and `indexing` fields. Use
-   `possibly_stale` results immediately when they are sufficient; do not call
-   status merely because a background update is active.
-5. Apply focused path and file-type filters early. Exclude dependencies,
-   generated output, caches, build artifacts, fixtures, and logs unless the
-   task concerns them.
-
-The default public MCP endpoint intentionally exposes only indexed search. If the
-index is missing, explain that indexed search requires an index and ask before
-creating one. After authorization, use the CLI lifecycle workflow; never silently
-create, rebuild, or drop an index. The optional full MCP toolset exposes
-`zvec_grep_rg`; the CLI `zg query --rg` route remains available for explicit
-managed-ripgrep fallback.
-
-Use multiple queries when comparing related concepts.
-
-## Use CLI fallback
-
-Read [references/cli-fallback.md](references/cli-fallback.md) only after a fallback condition above is satisfied. Leave CLI mode unset for status, fallback search, and authorized indexing commands so the default Auto mode can select Server or Direct; do not probe forced Server mode and then retry forced Direct mode. Keep the selected transport consistent for the investigation unless its availability changes.

package/skills/zvec-grep/agents/openai.yaml

@@ -1,4 +0,0 @@
-interface:
-  display_name: "zvec-grep"
-  short_description: "Repository search with explicit index lifecycle"
-  default_prompt: "Use $zvec-grep to investigate this repository before raw grep or rg. Prefer the public MCP search tools, and use the zg CLI only for authorized index lifecycle or daemon diagnostics."

package/skills/zvec-grep/references/cli-fallback.md

@@ -1,85 +0,0 @@
-# CLI fallback
-
-Use these commands only when the native MCP tools are unavailable or when an explicit no-index lexical search is required.
-
-## Choose the route
-
-Leave `--mode` unset for ordinary status and query commands. The default Auto mode uses the daemon when reachable and Direct mode otherwise:
-
-```bash
-zg status
-zg query "authentication flow" --fts "AuthService" -g "src/**" -g "!tests/**" -g "!dist/**"
-```
-
-Do not probe with `--mode server` and then retry with `--mode direct`. Use an explicit mode only when diagnosing routing or when the user requests one. Do not force Direct writes while a daemon lease is active.
-
-## Check index state
-
-Check status once per repository investigation and reuse it:
-
-```bash
-zg status
-```
-
-If the index is missing, do not build it automatically. Mention `zg index` and wait for the user to authorize persistent indexing.
-
-## Search an existing index
-
-Start with indexed hybrid search, then add exact or semantic routes as needed:
-
-```bash
-zg query "request validation" "error handling" -g "src/**" -g "!tests/**" -g "!dist/**" --limit 10 --preview short
-zg query "authentication flow" --fts "AuthService" --fts "ForbiddenError" -g "src/**"
-zg query --vector "where incoming requests are authorized" -g "src/**"
-zg query "changes that must be indexed before search" --refresh wait
-```
-
-When Auto selects Server, queries return the current index and refresh stale data in the background by default. Direct queries do not refresh by default. Add `--refresh wait` when the query must wait for pending index changes, or `--refresh off` to disable refresh explicitly. Use `--preview none` for broad candidate scans, `--preview short` for a small deterministic window, and `--preview full` only after narrowing.
-
-## Recover from a Direct model failure
-
-Direct mode already attempts CPU fallback when local GPU initialization fails. Do not repeat the query with an explicit CPU override.
-
-If the local model or embedding context remains unavailable and exact anchors are available, remove positional hybrid and vector queries and search those anchors through the existing indexed FTS route. Indexed FTS does not require an embedding model:
-
-```bash
-zg query --fts "AuthService" --fts "ForbiddenError" -g "src/**" --limit 20 --preview short
-```
-
-Do not switch to managed ripgrep merely because semantic search is unavailable. Use it only when the task requires exhaustive literal or regex matching, or when no index exists and the user has not authorized creating one.
-
-## Search exhaustively with managed ripgrep
-
-Use managed ripgrep only for exhaustive literal or regex matching; it does not require an index:
-
-```bash
-zg query --rg "SymbolName|LogMessage"
-zg query --rg -F "ExactSymbolOrText" src
-zg query --rg -i -C 2 -g "*.ts" -g "!dist/**" "needle text" src
-```
-
-Managed ripgrep always runs locally. It remains available while the daemon is running or an index writer is active; do not stop the daemon or switch client mode before using it.
-
-Use `-e` or `--regexp` when the pattern begins with `-`. Common ripgrep flags include `-n`, `-H`, `-F`, `-i`, `-w`, `-A`, `-B`, `-C`, `-g`, `--hidden`, `--type`, `--type-not`, `--no-ignore`, `--smart-case`, and `--pcre2`.
-
-## Index only with authorization
-
-When a server default model is known, let Auto submit the index without repeating the model:
-
-```bash
-zg index
-```
-
-Otherwise, use the embedding model selected by the user:
-
-```bash
-zg index --embedding local/embeddinggemma-300m
-```
-
-Use focused path filters for a new index:
-
-```bash
-zg index --embedding local/embeddinggemma-300m -g "src/**" -g "docs/**" -g "test/**" -g "!dist/**" -g "!node_modules/**" -g "!coverage/**"
-```
-
-Rebuild only when an incompatible embedding schema or index version requires it. Existing indexes reuse their stored embedding schema unless the user intentionally changes it.