pi-ast-sgrep 1.3.2 → 2.0.0

package/README.md

@@ -1,48 +1,242 @@
 # pi-ast-sgrep
 
-[![ast-sgrep for Pi](assets/preview.png)](https://github.com/AdityaVG13/ast-sgrep/blob/main/docs/pi-package.md)
+Native Code Mode, structural, graph, and semantic code search for [Pi](https://github.com/earendil-works/pi).
 
-Native intent, structural, definition, caller, chain, and semantic code search for Pi.
+[![pi-ast-sgrep: native code search inside Pi](https://cdn.jsdelivr.net/npm/pi-ast-sgrep/assets/preview.png)](https://pi.dev/packages/pi-ast-sgrep?name=pi-ast-sgrep)
+
+`pi-ast-sgrep` gives Pi a warm, project-aware search engine for understanding code. It finds behavior by intent, resolves definitions and callers, traces relationships, matches syntax-aware patterns, joins two indexed channels, and searches local semantic embeddings. The primary `asgrep` tool lets Pi compose several searches in one JavaScript program instead of spending one model round trip per lookup.
+
+**v2.0.0** · 13 languages · local-first semantic · critic + two-channel `AND` · **Code Mode** (on by default, no API key)
+
+**Upgrading to 2.0:** this is a breaking semver release because the cloud/Ollama embedding backends were removed. Update the Pi package normally; local hashed semantic search remains the default, optional neural embeddings remain in-process, and indexes that still store `embed_backend=cloud|ollama` fail closed until `/asgrep-reindex`. One-shot tools and Code Mode now put bounded hits in `content` so the model sees them, not only display-only `details`.
+
+### What's new for Pi in 2.0
+
+| Change | What you get |
+|--------|----------------|
+| Local-first embeddings | No `ASGREP_EMBED_API_KEY` / Ollama URL. Hashed semantic is default; optional ONNX stays in-process. |
+| Results on the model path | `asgrep_search` and Code Mode serialize hits into `content`. |
+| Two-channel queries | `asgrep.search({ query: 'callers:process_request AND pattern:fn $NAME($$$)' })` joins by span; `AND NOT` subtracts. Plain English `and` stays hybrid. |
+| Critic + follow-ups | Agent envelopes include `why` (`critic:` notes) and causal `follow_up_queries` from the actual top hit. |
+| Native work off the event loop | Index/search run as N-API worker tasks so Pi JS is not blocked on SQLite. |
+| Auto-registered tools | `asgrep` lands without requiring a skill file. |
+| Schema 12 | Older indexes rebuild through the normal compatibility path; `/asgrep-reindex` is the explicit full rebuild. |
+
+## Install
 
 ```bash
 pi install npm:pi-ast-sgrep
 ```
 
-Requires Node.js `>=22.19.0`, Pi `>=0.80.6 <1`, and a packaged host: macOS arm64/x64, glibc Linux arm64/x64, or Windows x64. The extension, `ast-sgrep` launcher, and selected native package manifests are exact-version matched at `1.3.2`; the embedded CLI compatibility identity is `1.3.2`. Alpine/musl, Windows arm64, and other hosts fail with an actionable unsupported-platform error; there is no source build or runtime download fallback.
+Restart Pi if the current session does not load newly installed package resources. For a project-local installation, add `-l`:
+
+```bash
+pi install -l npm:pi-ast-sgrep
+```
 
-## First use
+No Rust toolchain or separate MCP server is required. The npm package selects the native binary and in-process addon for the current supported platform.
 
-The package registers:
+## What this package adds
 
-- `asgrep_search`, `asgrep_index`, and `asgrep_status` tools;
-- `/asgrep-doctor`, `/asgrep-status`, `/asgrep-index`, and `/asgrep-reindex` commands;
-- the `ast-sgrep` skill.
+| Resource | Purpose |
+|---|---|
+| `asgrep` | Primary tool. Run a bounded JavaScript program that composes typed `asgrep.*` calls. Auto-registered with Pi (no skill file). |
+| `asgrep_search` | Run one natural, structural, symbol, graph, semantic, word, literal, or regex lookup. |
+| `asgrep_index` | Create, refresh, or explicitly rebuild the current project index. |
+| `asgrep_status` | Inspect the selected root, index, backend, counts, and capabilities. |
 
-Open Pi in a project and search. The first search lazily creates `.asgrep/`. Examples for `asgrep_search`:
+The package also registers `/asgrep-doctor`, `/asgrep-status`, `/asgrep-index`, and `/asgrep-reindex`.
+
+## Start with Code Mode
+
+Ask Pi:
+
+> Use ast-sgrep Code Mode to find where access tokens are refreshed, trace the top result's callers, and return only the relevant files, symbols, and lines.
+
+Pi can make one `asgrep` call like this:
 
 ```json
-{"query":"auth_refresh","mode":"defs"}
-{"query":"auth_refresh","mode":"callers"}
-{"query":"where are credentials renewed?","mode":"semantic"}
+{
+  "code": "async () => {\n  const seed = await asgrep.search({ query: 'where are access tokens refreshed?', limit: 5 });\n  const symbol = seed.hits?.[0]?.symbol;\n  if (!symbol) return { seed };\n  const [defs, callers] = await Promise.all([\n    asgrep.defs({ symbol, limit: 5 }),\n    asgrep.callers({ symbol, limit: 10 }),\n  ]);\n  return { symbol, defs: defs.hits, callers: callers.hits };\n}"
+}
 ```
 
-Run `/asgrep-doctor` to diagnose the runtime, binary, protocol, index, or configuration; run `/asgrep-status` to inspect the current project. The extension refreshes successful Pi write/edit changes before the next search and coalesces concurrent refreshes.
+This workflow narrows the first result, runs independent follow-up searches together, and returns a small shaped value to the model.
+
+### Code Mode API
+
+The Code Mode program receives these asynchronous methods on `asgrep`:
+
+| Method | Use |
+|---|---|
+| `asgrep.search({ query, limit?, excerptLines? })` | Search by intent, symbol, or a prefixed structural query. |
+| `asgrep.semantic({ query, limit?, excerptLines? })` | Search local semantic embeddings directly. |
+| `asgrep.defs({ symbol, limit? })` | Find definitions for one symbol. |
+| `asgrep.callers({ symbol, limit? })` | Find call sites for one symbol. |
+| `asgrep.imports({ module, limit? })` | Find imports of one module. |
+| `asgrep.chain({ query, limit? })` | Trace related symbols and graph edges. |
+| `asgrep.indexStatus()` | Read index and backend state. |
+| `asgrep.indexRepo({ force? })` | Create, refresh, or rebuild the index. |
+| `asgrep.catalogSearch({ query })` | Discover less common ast-sgrep operations. |
+| `asgrep.catalogDescribe({ name })` | Read the schema for a discovered operation. |
+
+Use `Promise.all` for independent calls. Filter, map, sort, and slice intermediate values in JavaScript. Return only the evidence needed for the next reasoning step.
+
+Code Mode runs in a disposable worker with a restricted `node:vm` context that exposes only a serialized `asgrep.*` bridge and console. String and WebAssembly code generation are disabled, ambient Node globals such as `process` and `require` are not exposed, and terminating the worker contains synchronous and microtask CPU loops. Node does not consider `vm` an adversarial-code security boundary, however, and the installed Pi package has full OS-user access; do not treat Code Mode as an OS jail. Prefer Code Mode **or** MCP for a client, never both.
+
+The bridge rejects oversized call arguments and serialized results, allows at
+most 256 host calls per program, and caps collected console output before it
+reaches the extension host. Raw-memory and WebAssembly globals are unavailable;
+worker heap/stack limits contain the remaining accidental memory growth. Native
+tool values are capped at 1 MiB each and complete batch responses at 4 MiB before
+Node-API converts them into extension-host objects. These bounds do not turn `node:vm` into an OS
+sandbox.
+
+## Direct one-shot search
+
+Use `asgrep_search` when one lookup is enough:
+
+```json
+{"query":"auth_refresh","mode":"defs","limit":8}
+{"query":"auth_refresh","mode":"callers","limit":8}
+{"query":"where are credentials renewed?","mode":"semantic","limit":8}
+{"query":"$CLIENT.post($URL)","mode":"pattern","limit":8}
+{"query":"callers:process_request AND pattern:fn $NAME($$$)", "mode":"natural","limit":8}
+{"query":"defs:handle AND NOT callers:test_","mode":"natural","limit":8}
+```
+
+Available modes:
+
+| Mode | Best for |
+|---|---|
+| `natural` | Intent or mixed code-language queries when exact spelling is unknown. Also the mode for two-channel `AND` / `AND NOT` query strings. |
+| `pattern` | Syntax-aware ast-sgrep patterns with metavariables. |
+| `defs`, `callers`, `imports` | Symbol and module navigation. |
+| `chain` | Multi-hop relationship tracing. |
+| `semantic` | Meaning-based local vector search. |
+| `word`, `literal`, `regex` | Explicit text-oriented matching. |
+
+`limit` accepts 1–100 and defaults to 8. Excerpts are disabled by default; set `excerptLines` only after narrowing the result set.
+
+## Why Code Mode is fast
+
+Official platform packages include `ast-sgrep-codemode.node`. The extension loads an in-process native `CodeModeSession` and keeps one warm Searcher per project root for Code Mode, direct tools, and freshness checks. Normal searches do not spawn a CLI process.
+
+Native index and search calls run as Promise-returning N-API worker tasks rather than on Node's event-loop thread. Calls for one warm session are serialized before entering libuv so concurrent Pi work does not occupy worker threads waiting on the same SQLite session.
+
+Independent calls created in the same JavaScript turn are coalesced into a batch. `Promise.all` can therefore fan out several lookups while the model makes one tool call. If the native addon is unavailable, the bundled CLI service is a degraded fallback; `/asgrep-doctor` reports the active backend.
+
+Code Mode and `ast-sgrep-mcp` are separate front ends over the same Rust search core. Pi uses Code Mode directly and does not use an MCP adapter.
+
+## Indexing and freshness
+
+Start Pi in the repository you want to search. The first search validates the index and lazily creates `<project-root>/.asgrep/` when needed. Run `/asgrep-index` if you want to build it before searching.
+
+After a successful Pi `write` or `edit`, the extension marks the affected path dirty and updates only known changed paths before the next search. It also watches the project for external filesystem changes: known file changes receive the same targeted update, while renames, directory changes, ignore-file edits, watcher errors, and ambiguous events trigger a correctness scan. `.asgrep` writes are excluded so indexing cannot dirty itself. If recursive watching is unavailable, an immediate scan plus the periodic full scan preserve correctness. Concurrent searches for the same root share one in-flight refresh.
+
+The periodic interval forces a full incremental reconciliation even when the watcher reports nothing, covering dropped or coalesced filesystem events. Run `/asgrep-index` when you need freshness immediately after a large external operation; use `/asgrep-reindex` only for an incompatible or corrupt index, or when you explicitly need a strict full rebuild.
+
+The package never edits `.gitignore`. Add this entry yourself if index data must stay untracked:
+
+```gitignore
+.asgrep/
+```
+
+## Commands
+
+| Command | Action |
+|---|---|
+| `/asgrep-doctor` | Check package versions, native runtime, protocol, index, and project settings. |
+| `/asgrep-status` | Show the current root and index state. |
+| `/asgrep-index` | Create or incrementally refresh the index. |
+| `/asgrep-reindex` | Strictly rebuild the index in one transaction while preserving the prior usable rows on failure. |
+
+These commands take no arguments.
+
+## Requirements
+
+- Node.js `>=22.19.0`.
+- Pi currently tested with `@earendil-works/pi-coding-agent >=0.80.6 <1`.
+- macOS arm64 or x64, glibc Linux arm64 or x64, or Windows x64.
+
+Alpine/musl Linux, Windows arm64, and other hosts are not packaged. The package does not compile Rust, search `PATH`, or download executables at runtime. On an unsupported host, run `/asgrep-doctor` for the exact platform error.
+
+The extension, `ast-sgrep` launcher, platform package, native addon, and embedded CLI are exact-version matched. Update or reinstall the complete package if doctor reports a version or protocol mismatch.
+
+## Typed Code Mode API
+
+Import the search-only programmatic surface from `pi-ast-sgrep/code-mode` and execute related lookups in one typed plan:
+
+```ts
+import { AstSgrepRuntime } from "pi-ast-sgrep/runtime";
+import { createSgrepCodeMode } from "pi-ast-sgrep/code-mode";
+
+const mode = createSgrepCodeMode(new AstSgrepRuntime(pi), { cwd: process.cwd() });
+const result = await mode.execute(async (sgrep) => {
+  const [text, ast, semantic] = await Promise.all([
+    sgrep.keywordSearch("refresh token"),
+    sgrep.astSearch("function_declaration"),
+    sgrep.semanticSearch("credential renewal"),
+  ]);
+  const bodies = await sgrep.codeRead(text.hits.slice(0, 3), { contextLines: 2 });
+  return { text, ast, semantic, bodies };
+});
+```
+
+- `keywordSearch` runs lexical retrieval only.
+- `astSearch` runs `pattern:` structural search only.
+- `semanticSearch` runs embedding retrieval only.
+- `codeRead` streams bounded `file#Lx-Ly` refs inside the project, including adjacent context, symlink containment, strict UTF-8 validation, cancellation, and an aggregate output budget.
+- `find`, `astFind`, `semantic`, and `read` remain typed aliases for the four methods above.
+
+The agent chooses the retrieval granularity; these methods never auto-fuse channels. One-shot CLI search retains fusion for human/direct engine use. The API exposes no rewrite or mutation operation. Structural rewrites remain delegated to ast-grep. Search responses retain signal, contributor, score, and margin provenance.
 
 ## Local by default
 
-Local semantic indexing/search works offline with no credential, telemetry, first-use model download, executable download, PATH lookup, or MCP adapter. Optional external embedding providers are opt-in and may receive the source text and queries needed to create embeddings.
+The default semantic backend works offline. It needs no credential, sends no telemetry, and downloads no model on first use. Search data stays under the project's `.asgrep/` directory.
+
+In-process neural embeddings are optional (`--features neural-embed`). They never send source text to a remote embedding API. Hashed local search remains the default.
+
+Pi packages are trusted code. Installation grants this JavaScript extension and its native code the permissions of the OS user running Pi. Project-root confinement is a package policy, not an operating-system security boundary.
+
+## Configuration
+
+Defaults are a 30-second operation timeout, 4 MiB output limit, and 30-second freshness interval. Supported environment settings are:
 
-Indexing writes database, embedding, metadata, and lock/rebuild files under the project's `.asgrep/`. The package never edits `.gitignore`; add `.asgrep/` yourself if you do not want it committed. Pi packages run with the OS user's full access and are not sandboxed.
+| Setting | Purpose |
+|---|---|
+| `ASGREP_ROOT` | Select the project root. |
+| `ASGREP_TIMEOUT_MS` | Set the native operation timeout. |
+| `ASGREP_MAX_OUTPUT_BYTES` | Bound native output. |
+| `ASGREP_REFRESH_INTERVAL_MS` | Set the idle freshness-check interval. |
+| `ASGREP_BIN` | Override the packaged binary for development. |
 
-## Update or remove
+Explicit project configuration can opt into `allowOutsideProject`; global settings and environment variables cannot relax the default project boundary. See the [complete package guide](https://github.com/AdityaVG13/ast-sgrep/blob/main/docs/pi-package.md) for schema and precedence details.
+
+## Update, rollback, or remove
 
 ```bash
 pi update npm:pi-ast-sgrep
 pi remove npm:pi-ast-sgrep
 ```
 
-Removal preserves every project's `.asgrep` data for reinstall or rollback. Delete that directory separately and explicitly only when you no longer need it. Compatible updates reuse validated data; incompatible formats rebuild atomically and preserve recoverable prior data on failure. Roll back by removing the package and installing `npm:pi-ast-sgrep@<previous-version>`, then run `/asgrep-doctor`.
+Removal preserves each project's `.asgrep/` data for reinstall or rollback. Delete that directory separately only when you no longer need the index.
+
+To roll back, install one prior version as a matched unit:
+
+```bash
+pi remove npm:pi-ast-sgrep
+pi install npm:pi-ast-sgrep@<previous-version>
+```
+
+Then run `/asgrep-doctor`. Compatible updates reuse validated data. Incompatible formats rebuild transactionally in place and preserve recoverable prior rows when a rebuild fails.
+
+## More documentation
 
-Read the [complete install, configuration, security, recovery, and uninstall guide](https://github.com/AdityaVG13/ast-sgrep/blob/main/docs/pi-package.md). Release provenance and package order are documented in [RELEASING.md](https://github.com/AdityaVG13/ast-sgrep/blob/main/docs/RELEASING.md).
+- [Complete Pi package guide](https://github.com/AdityaVG13/ast-sgrep/blob/main/docs/pi-package.md)
+- [Code Mode architecture and performance](https://github.com/AdityaVG13/ast-sgrep/blob/main/docs/codemode.md)
+- [Query grammar](https://github.com/AdityaVG13/ast-sgrep/blob/main/docs/QUERY_GRAMMAR.md) (prefixes and two-channel `AND`)
+- [Fusion ranking and critic](https://github.com/AdityaVG13/ast-sgrep/blob/main/docs/fusion-ranking.md)
+- [Release provenance](https://github.com/AdityaVG13/ast-sgrep/blob/main/docs/RELEASING.md)
 
 MIT

package/dist/code-mode.d.ts

@@ -0,0 +1,89 @@
+import { type AstSgrepRuntime, type MachineEnvelope, type RunOptions, type RuntimeContext } from "./runtime.js";
+export type SgrepKind = "asgrep" | "def" | "caller" | "graph" | "anchor" | "import" | "pattern" | "embed";
+export type SgrepSignal = "exact" | "structural" | "semantic";
+export type SgrepRef = `${string}#L${number}-L${number}`;
+/**
+ * Trusted search hit. Location is solely `ref` (parsed once at the CLI/JSON boundary).
+ * Wire may still dual-encode file/lines; those are not live fields on this type.
+ */
+export interface SgrepHit {
+    kind: SgrepKind;
+    signal: SgrepSignal;
+    contributors: SgrepKind[];
+    score: number;
+    margin: number;
+    ref: SgrepRef;
+    preview: string;
+    symbol?: string | null;
+    caller?: string | null;
+    callee?: string | null;
+    language?: string | null;
+    excerpt?: string;
+}
+export interface SgrepSearchResponse extends MachineEnvelope {
+    hits: SgrepHit[];
+    query?: string;
+    hit_count?: number;
+}
+export interface SgrepSearchOptions extends RunOptions {
+    limit?: number;
+    excerptLines?: number;
+}
+export interface SgrepReadOptions {
+    contextLines?: number;
+    /** Aggregate character budget across all refs. */
+    maxChars?: number;
+    signal?: AbortSignal;
+}
+/**
+ * Trusted read window. Location is solely `ref` (actual lines returned; may expand the
+ * request via contextLines). Derive file/lines with `parseSgrepRef` -- no live twins.
+ */
+export interface SgrepReadResult {
+    ref: SgrepRef;
+    content: string;
+    truncated: boolean;
+    /** Present when truncated: 1-indexed line to resume from (on the last shown line). */
+    resumeOffset?: number;
+    /** Named recovery hint for the model (empty/past-EOF/truncation). */
+    note?: string;
+}
+export interface SgrepApi {
+    keywordSearch(query: string, options?: SgrepSearchOptions): Promise<SgrepSearchResponse>;
+    astSearch(pattern: string, options?: SgrepSearchOptions): Promise<SgrepSearchResponse>;
+    semanticSearch(query: string, options?: SgrepSearchOptions): Promise<SgrepSearchResponse>;
+    codeRead(ids: SgrepRef | Pick<SgrepHit, "ref"> | readonly (SgrepRef | Pick<SgrepHit, "ref">)[], options?: SgrepReadOptions): Promise<SgrepReadResult[]>;
+    /** Alias for keywordSearch. */
+    find(query: string, options?: SgrepSearchOptions): Promise<SgrepSearchResponse>;
+    /** Alias for astSearch. */
+    astFind(pattern: string, options?: SgrepSearchOptions): Promise<SgrepSearchResponse>;
+    /** Alias for semanticSearch. */
+    semantic(query: string, options?: SgrepSearchOptions): Promise<SgrepSearchResponse>;
+    /** Alias for codeRead. */
+    read(ids: SgrepRef | Pick<SgrepHit, "ref"> | readonly (SgrepRef | Pick<SgrepHit, "ref">)[], options?: SgrepReadOptions): Promise<SgrepReadResult[]>;
+}
+export type SgrepPlan<T> = (sgrep: Readonly<SgrepApi>) => T | Promise<T>;
+type RuntimeLike = Pick<AstSgrepRuntime, "run" | "resolveRoot">;
+/** Derive file/lines from a branded ref (sole location encoding on SgrepHit). */
+export declare function parseSgrepRef(ref: SgrepRef): {
+    file: string;
+    start: number;
+    end: number;
+};
+export declare class SgrepCodeMode implements SgrepApi {
+    #private;
+    private readonly runtime;
+    private readonly context;
+    constructor(runtime: RuntimeLike, context: RuntimeContext);
+    execute<T>(plan: SgrepPlan<T>): Promise<T>;
+    keywordSearch(query: string, options?: SgrepSearchOptions): Promise<SgrepSearchResponse>;
+    astSearch(pattern: string, options?: SgrepSearchOptions): Promise<SgrepSearchResponse>;
+    semanticSearch(query: string, options?: SgrepSearchOptions): Promise<SgrepSearchResponse>;
+    find(query: string, options?: SgrepSearchOptions): Promise<SgrepSearchResponse>;
+    astFind(pattern: string, options?: SgrepSearchOptions): Promise<SgrepSearchResponse>;
+    semantic(query: string, options?: SgrepSearchOptions): Promise<SgrepSearchResponse>;
+    codeRead(ids: SgrepRef | Pick<SgrepHit, "ref"> | readonly (SgrepRef | Pick<SgrepHit, "ref">)[], options?: SgrepReadOptions): Promise<SgrepReadResult[]>;
+    read(ids: SgrepRef | Pick<SgrepHit, "ref"> | readonly (SgrepRef | Pick<SgrepHit, "ref">)[], options?: SgrepReadOptions): Promise<SgrepReadResult[]>;
+}
+export declare function createSgrepCodeMode(runtime: RuntimeLike, context: RuntimeContext): SgrepCodeMode;
+export {};