lmgrep 0.1.0

package/README.md

@@ -0,0 +1,181 @@
+# lmgrep
+
+Semantic code search powered by AI embeddings. Index your codebase with any embedding provider and search it using natural language.
+
+lmgrep uses [Tree-sitter](https://tree-sitter.github.io/tree-sitter/) to parse source code into meaningful chunks (functions, classes, interfaces, etc.), embeds them with the AI model of your choice, and stores the vectors in a local [LanceDB](https://lancedb.github.io/lancedb/) database. Queries are matched by semantic similarity, so you find code by *intent* rather than exact strings.
+
+## Features
+
+- **Any embedding provider** — works with Ollama, OpenAI, Google, or any provider supported by the [Vercel AI SDK](https://sdk.vercel.ai/)
+- **Tree-sitter chunking** — splits code at AST boundaries so search results are complete, meaningful units
+- **MCP server** — ships with an MCP server (`lmgrep-mcp`) for integration with Claude Code, Cursor, and other AI tools
+- **File watching** — `lmgrep serve` watches for changes and incrementally re-indexes
+- **Cross-project search** — search across multiple indexed projects
+- **Git-aware** — respects `.gitignore`, deduplicates across worktrees sharing the same remote
+- **Configurable** — global or per-project config, custom ignore patterns, extension filtering
+
+## Quick start
+
+### 1. Install
+
+```sh
+pnpm install -g lmgrep
+```
+
+### 2. Set up an embedding model
+
+The fastest way to get started is with [Ollama](https://ollama.com/):
+
+```sh
+# Install Ollama
+curl -fsSL https://ollama.com/install.sh | sh
+
+# Pull an embedding model
+ollama pull nomic-embed-text
+
+# Auto-detect and write config
+lmgrep init
+```
+
+This creates a config file at `~/.config/lmgrep/config.yml` (Linux) or `~/Library/Application Support/lmgrep/config.yml` (macOS).
+
+### 3. Index your project
+
+```sh
+cd /path/to/your/project
+lmgrep index
+```
+
+### 4. Search
+
+```sh
+lmgrep search "how are users authenticated"
+lmgrep search "database connection pooling" --limit 5
+lmgrep search "error handling" --file-prefix src/lib --language .ts
+```
+
+## CLI commands
+
+| Command | Description |
+|---|---|
+| `lmgrep index` | Index the current directory |
+| `lmgrep search <query>` | Search using natural language |
+| `lmgrep status` | Show index stats and embedding connectivity |
+| `lmgrep serve` | Watch for changes and re-index automatically |
+| `lmgrep init` | Detect embedding setup and create config |
+| `lmgrep config` | Open the global config in your editor |
+| `lmgrep repair` | Detect and fix index inconsistencies |
+| `lmgrep compact` | Compact the index to reclaim disk space |
+| `lmgrep prune` | Delete the index for the current directory |
+| `lmgrep import [db-path]` | Import chunks from another lmgrep database |
+| `lmgrep completions zsh` | Output or install zsh completions |
+
+### Search options
+
+```
+--limit <n>          Max results (default: 25)
+--file-prefix <path> Only search files under this path
+--language <exts>    Filter by file extension (e.g. .ts,.py)
+--type <types>       Filter by AST node type (e.g. function_declaration)
+--not <query>        Exclude results similar to this query
+--scores             Show relevance scores
+--compact            Show file paths only
+--json               Output as JSON
+--project <path>     Search a different project's index
+--across <paths>     Search multiple projects (comma-separated)
+```
+
+### Index options
+
+```
+--reset       Rebuild the entire index from scratch
+--since <dur> Only re-index files modified within duration (e.g. 10m, 2h, 1d)
+--force       Force re-embed even if file hash is unchanged
+--dry         Show what would be indexed without doing it
+--verbose     Show file-by-file progress
+```
+
+## MCP server
+
+lmgrep includes an MCP server for use with AI coding assistants. Add it to your tool's MCP configuration:
+
+```json
+{
+  "mcpServers": {
+    "lmgrep": {
+      "command": "lmgrep-mcp",
+      "args": []
+    }
+  }
+}
+```
+
+The MCP server exposes a `search` tool and a `list_other_indexed_projects` tool. It automatically watches for file changes and keeps the index up to date.
+
+## Configuration
+
+lmgrep looks for configuration in this order:
+
+1. `.lmgrep.yml` in the project root (per-project)
+2. `~/.config/lmgrep/config.yml` (global, Linux) or `~/Library/Application Support/lmgrep/config.yml` (macOS)
+3. `~/.lmgrep.yml` (legacy fallback)
+
+### Example config
+
+```yaml
+# Embedding model in "provider:model" format
+model: ollama:nomic-embed-text
+
+# Base URL for the embedding API
+baseURL: http://localhost:11434/v1
+
+# Batch size for embedding API calls
+batchSize: 100
+
+# Optional: embedding dimensions (if model supports it)
+# dimensions: 384
+
+# Optional: max tokens per chunk (estimated at 4 chars/token)
+# maxTokens: 8192
+
+# Optional: prefixes for asymmetric embedding models
+# queryPrefix: "search_query: "
+# documentPrefix: "search_document: "
+
+# Optional: additional ignore patterns (merged with .gitignore)
+# ignore:
+#   - "*.generated.ts"
+#   - "fixtures/"
+
+# Optional: file extension control
+# extensions:
+#   include: [".sql", ".graphql", ".proto"]
+#   exclude: [".json"]
+```
+
+### Using other providers
+
+Install the provider package globally and set the model accordingly:
+
+```sh
+# OpenAI
+npm install -g @ai-sdk/openai
+# then in config: model: openai:text-embedding-3-small
+
+# Google
+npm install -g @ai-sdk/google
+# then in config: model: google:text-embedding-004
+```
+
+## Development
+
+```sh
+pnpm install
+pnpm build        # compile TypeScript
+pnpm dev          # watch mode
+pnpm check        # format and lint (Biome)
+```
+
+## License
+
+Apache-2.0

package/completions/_lmgrep

@@ -0,0 +1,88 @@
+#compdef lmgrep
+
+_lmgrep() {
+  local -a commands
+  commands=(
+    'index:Index the current directory for semantic search'
+    'search:Search the codebase using natural language'
+    'status:Show index stats and check embedding connectivity'
+    'repair:Detect and fix index inconsistencies'
+    'serve:Watch the current directory and re-index on changes'
+    'init:Detect your embedding setup and create config'
+    'config:Open the global config file in your editor'
+    'compact:Compact the index to reclaim disk space'
+    'prune:Delete the index database for the current directory'
+    'import:Import chunks and file hashes from another lmgrep database'
+    'completions:Output shell completions'
+  )
+
+  _arguments -C \
+    '(-h --help)'{-h,--help}'[Show help]' \
+    '(-V --version)'{-V,--version}'[Show version]' \
+    '1:command:->command' \
+    '*::arg:->args'
+
+  case "$state" in
+    command)
+      _describe 'command' commands
+      ;;
+    args)
+      case "$words[1]" in
+        index)
+          _arguments \
+            '(-r --reset)'{-r,--reset}'[Reset and rebuild the entire index]' \
+            '(-v --verbose)'{-v,--verbose}'[Show file-by-file progress]' \
+            '(-s --since)'{-s,--since}'[Only consider files modified within duration]:duration' \
+            '(-f --force)'{-f,--force}'[Force re-embed even if file hash unchanged]' \
+            '(-d --dry)'{-d,--dry}'[Show what would be indexed without actually doing it]'
+          ;;
+        search)
+          _arguments \
+            '1:query' \
+            '(-m --limit)'{-m,--limit}'[Max results]:number' \
+            '--scores[Show relevance scores]' \
+            '--compact[Show file paths only]' \
+            '--json[Output results as JSON]' \
+            '--min-score[Minimum score threshold]:score' \
+            '--file-prefix[Only search files matching this path prefix]:prefix:_files -/' \
+            '--not[Exclude results similar to this query]:query' \
+            '--type[Only return chunks of these AST types]:types' \
+            '--language[Only return chunks from files with these extensions]:extensions' \
+            '--project[Search a different project index]:path:_files -/' \
+            '--across[Search multiple project indexes]:paths'
+          ;;
+        status)
+          _arguments \
+            '(-c --changes)'{-c,--changes}'[Scan for changed files since last index]' \
+            '--json[Output status as JSON]'
+          ;;
+        repair)
+          _arguments \
+            '(-d --dry)'{-d,--dry}'[Show what would be repaired without making changes]' \
+            '--json[Output repair results as JSON]'
+          ;;
+        init)
+          _arguments \
+            '--force[Overwrite existing config]' \
+            '--local[Write a project-local .lmgrep.yml instead of the global config]'
+          ;;
+        prune)
+          _arguments \
+            '--force[Skip confirmation]'
+          ;;
+        import)
+          _arguments \
+            '1:database path:_files' \
+            '--reset[Reset the current index before importing]'
+          ;;
+        completions)
+          _arguments \
+            '1:shell:(zsh)' \
+            '--install[Install to site-functions]'
+          ;;
+      esac
+      ;;
+  esac
+}
+
+_lmgrep "$@"

package/dist/chunker/context.d.ts

@@ -0,0 +1,25 @@
+import type Parser from "web-tree-sitter";
+import type { LanguageConfig } from "./languages.js";
+export type StructuralRole = "definition" | "orchestration" | "implementation";
+export interface ScopeEntry {
+    kind: string;
+    name: string;
+}
+export interface ChunkContext {
+    filePath: string;
+    scope: ScopeEntry[];
+    leadingComment: string | null;
+    role: StructuralRole;
+}
+/** Walk up from a node to collect typed parent scopes */
+export declare function extractScope(node: Parser.SyntaxNode, langConfig: LanguageConfig): ScopeEntry[];
+/** Extract import module names from the root of the tree */
+export declare function extractImports(tree: Parser.Tree, langConfig: LanguageConfig): string[];
+/** Extract sibling signatures from the same parent scope */
+export declare function extractSiblingSignatures(node: Parser.SyntaxNode, langConfig: LanguageConfig): string[];
+/** Extract leading comments and decorators immediately before a node */
+export declare function extractLeadingComment(node: Parser.SyntaxNode, source: string): string | null;
+/** Classify a chunk's structural role based on its AST node type */
+export declare function classifyRole(node: Parser.SyntaxNode): StructuralRole;
+/** Build the context prefix string for a chunk */
+export declare function buildContextString(ctx: ChunkContext): string;

package/dist/chunker/context.js

@@ -0,0 +1,204 @@
+/** Map AST node types to human-readable scope kinds */
+const SCOPE_KIND_MAP = {
+    // TypeScript / JavaScript
+    class_declaration: "class",
+    class_body: "class",
+    interface_declaration: "interface",
+    module: "module",
+    namespace_declaration: "namespace",
+    // Python
+    class_definition: "class",
+    // Rust
+    impl_item: "impl",
+    trait_item: "trait",
+    mod_item: "mod",
+    // Go
+    type_declaration: "type",
+    // Ruby
+    class: "class",
+    // C / C++
+    struct_specifier: "struct",
+    class_specifier: "class",
+    namespace_definition: "namespace",
+    // Swift
+    struct_declaration: "struct",
+    extension_declaration: "extension",
+    // Scala
+    object_definition: "object",
+    trait_definition: "trait",
+    // Generic
+    ContainerDecl: "container",
+};
+/** AST node types that represent type/structure definitions */
+const DEFINITION_TYPES = new Set([
+    "class_declaration",
+    "class_definition",
+    "interface_declaration",
+    "type_alias_declaration",
+    "enum_declaration",
+    "struct_item",
+    "enum_item",
+    "trait_item",
+    "type_item",
+    "type_declaration",
+    "struct_specifier",
+    "class_specifier",
+    "enum_specifier",
+    "type_definition",
+    "struct_declaration",
+    "protocol_declaration",
+    "class_definition",
+    "trait_definition",
+    "ContainerDecl",
+]);
+/** AST node types that typically orchestrate / glue logic */
+const ORCHESTRATION_TYPES = new Set([
+    "export_statement",
+    "decorated_definition",
+]);
+/** Walk up from a node to collect typed parent scopes */
+export function extractScope(node, langConfig) {
+    const scopes = [];
+    let current = node.parent;
+    while (current) {
+        if (langConfig.scopeTypes.includes(current.type)) {
+            const name = extractNodeName(current);
+            if (name) {
+                const kind = SCOPE_KIND_MAP[current.type] ?? current.type;
+                scopes.unshift({ kind, name });
+            }
+        }
+        current = current.parent;
+    }
+    return scopes;
+}
+/** Extract import module names from the root of the tree */
+export function extractImports(tree, langConfig) {
+    const modules = [];
+    for (const child of tree.rootNode.children) {
+        if (langConfig.importTypes.includes(child.type)) {
+            const mod = extractModuleName(child);
+            if (mod && !modules.includes(mod)) {
+                modules.push(mod);
+            }
+        }
+    }
+    return modules;
+}
+/** Pull out just the module/package name from an import node */
+function extractModuleName(node) {
+    // Look for string children (the module specifier)
+    const stringNode = findDescendant(node, (n) => n.type === "string" ||
+        n.type === "string_literal" ||
+        n.type === "interpreted_string_literal");
+    if (stringNode) {
+        // Strip quotes
+        return stringNode.text.replace(/^["']|["']$/g, "");
+    }
+    // Python: import foo.bar / from foo.bar import baz
+    const dottedName = findDescendant(node, (n) => n.type === "dotted_name" || n.type === "module_name");
+    if (dottedName)
+        return dottedName.text;
+    // Rust: use crate::foo::bar
+    const path = findDescendant(node, (n) => n.type === "scoped_identifier" || n.type === "use_list");
+    if (path)
+        return path.text;
+    // C/C++: #include <foo.h> or "foo.h"
+    const sysLib = findDescendant(node, (n) => n.type === "system_lib_string" || n.type === "string_literal");
+    if (sysLib)
+        return sysLib.text.replace(/^[<"]|[>"]$/g, "");
+    return null;
+}
+function findDescendant(node, predicate) {
+    for (const child of node.children) {
+        if (predicate(child))
+            return child;
+        const found = findDescendant(child, predicate);
+        if (found)
+            return found;
+    }
+    return null;
+}
+/** Extract sibling signatures from the same parent scope */
+export function extractSiblingSignatures(node, langConfig) {
+    const parent = node.parent;
+    if (!parent)
+        return [];
+    const siblings = [];
+    for (const child of parent.children) {
+        if (child.id === node.id)
+            continue;
+        if (!langConfig.chunkTypes.includes(child.type))
+            continue;
+        const name = extractNodeName(child);
+        if (name) {
+            const firstLine = child.text.split("\n")[0].trim();
+            if (firstLine.length < 200) {
+                siblings.push(firstLine);
+            }
+        }
+    }
+    return siblings.slice(0, 10);
+}
+/** Extract leading comments and decorators immediately before a node */
+export function extractLeadingComment(node, source) {
+    const lines = source.split("\n");
+    const nodeStartLine = node.startPosition.row;
+    const collected = [];
+    // Walk backwards from the line before the node
+    for (let i = nodeStartLine - 1; i >= 0 && i >= nodeStartLine - 10; i--) {
+        const line = lines[i].trim();
+        if (line.startsWith("//") ||
+            line.startsWith("#") ||
+            line.startsWith("*") ||
+            line.startsWith("/*") ||
+            line.startsWith("*/") ||
+            line.startsWith("///") ||
+            line.startsWith("--") ||
+            line.startsWith("@") ||
+            line.startsWith('"""') ||
+            line.startsWith("'''")) {
+            collected.unshift(lines[i]);
+        }
+        else if (line === "") {
+            // Allow one blank line gap
+            if (collected.length > 0)
+                break;
+        }
+        else {
+            break;
+        }
+    }
+    if (collected.length === 0)
+        return null;
+    return collected.join("\n").trim();
+}
+/** Classify a chunk's structural role based on its AST node type */
+export function classifyRole(node) {
+    if (DEFINITION_TYPES.has(node.type))
+        return "definition";
+    if (ORCHESTRATION_TYPES.has(node.type))
+        return "orchestration";
+    return "implementation";
+}
+/** Build the context prefix string for a chunk */
+export function buildContextString(ctx) {
+    const lines = [];
+    lines.push(`[file: ${ctx.filePath}]`);
+    lines.push(`[role: ${ctx.role}]`);
+    if (ctx.scope.length > 0) {
+        const scopeStr = ctx.scope.map((s) => `${s.kind} ${s.name}`).join(" > ");
+        lines.push(`[scope: ${scopeStr}]`);
+    }
+    if (ctx.leadingComment) {
+        lines.push(`[doc: ${ctx.leadingComment}]`);
+    }
+    return lines.join("\n");
+}
+/** Try to extract a name identifier from an AST node */
+function extractNodeName(node) {
+    const nameNode = node.childForFieldName("name") ??
+        node.children.find((c) => c.type === "identifier" || c.type === "type_identifier");
+    return nameNode?.text;
+}
+//# sourceMappingURL=context.js.map

package/dist/chunker/context.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"context.js","sourceRoot":"","sources":["../../src/chunker/context.ts"],"names":[],"mappings":"AAiBA,uDAAuD;AACvD,MAAM,cAAc,GAA2B;IAC9C,0BAA0B;IAC1B,iBAAiB,EAAE,OAAO;IAC1B,UAAU,EAAE,OAAO;IACnB,qBAAqB,EAAE,WAAW;IAClC,MAAM,EAAE,QAAQ;IAChB,qBAAqB,EAAE,WAAW;IAClC,SAAS;IACT,gBAAgB,EAAE,OAAO;IACzB,OAAO;IACP,SAAS,EAAE,MAAM;IACjB,UAAU,EAAE,OAAO;IACnB,QAAQ,EAAE,KAAK;IACf,KAAK;IACL,gBAAgB,EAAE,MAAM;IACxB,OAAO;IACP,KAAK,EAAE,OAAO;IACd,UAAU;IACV,gBAAgB,EAAE,QAAQ;IAC1B,eAAe,EAAE,OAAO;IACxB,oBAAoB,EAAE,WAAW;IACjC,QAAQ;IACR,kBAAkB,EAAE,QAAQ;IAC5B,qBAAqB,EAAE,WAAW;IAClC,QAAQ;IACR,iBAAiB,EAAE,QAAQ;IAC3B,gBAAgB,EAAE,OAAO;IACzB,UAAU;IACV,aAAa,EAAE,WAAW;CAC1B,CAAC;AAEF,+DAA+D;AAC/D,MAAM,gBAAgB,GAAG,IAAI,GAAG,CAAC;IAChC,mBAAmB;IACnB,kBAAkB;IAClB,uBAAuB;IACvB,wBAAwB;IACxB,kBAAkB;IAClB,aAAa;IACb,WAAW;IACX,YAAY;IACZ,WAAW;IACX,kBAAkB;IAClB,kBAAkB;IAClB,iBAAiB;IACjB,gBAAgB;IAChB,iBAAiB;IACjB,oBAAoB;IACpB,sBAAsB;IACtB,kBAAkB;IAClB,kBAAkB;IAClB,eAAe;CACf,CAAC,CAAC;AAEH,6DAA6D;AAC7D,MAAM,mBAAmB,GAAG,IAAI,GAAG,CAAC;IACnC,kBAAkB;IAClB,sBAAsB;CACtB,CAAC,CAAC;AAEH,yDAAyD;AACzD,MAAM,UAAU,YAAY,CAC3B,IAAuB,EACvB,UAA0B;IAE1B,MAAM,MAAM,GAAiB,EAAE,CAAC;IAChC,IAAI,OAAO,GAAG,IAAI,CAAC,MAAM,CAAC;IAC1B,OAAO,OAAO,EAAE,CAAC;QAChB,IAAI,UAAU,CAAC,UAAU,CAAC,QAAQ,CAAC,OAAO,CAAC,IAAI,CAAC,EAAE,CAAC;YAClD,MAAM,IAAI,GAAG,eAAe,CAAC,OAAO,CAAC,CAAC;YACtC,IAAI,IAAI,EAAE,CAAC;gBACV,MAAM,IAAI,GAAG,cAAc,CAAC,OAAO,CAAC,IAAI,CAAC,IAAI,OAAO,CAAC,IAAI,CAAC;gBAC1D,MAAM,CAAC,OAAO,CAAC,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC,CAAC;YAChC,CAAC;QACF,CAAC;QACD,OAAO,GAAG,OAAO,CAAC,MAAM,CAAC;IAC1B,CAAC;IACD,OAAO,MAAM,CAAC;AACf,CAAC;AAED,4DAA4D;AAC5D,MAAM,UAAU,cAAc,CAC7B,IAAiB,EACjB,UAA0B;IAE1B,MAAM,OAAO,GAAa,EAAE,CAAC;IAC7B,KAAK,MAAM,KAAK,IAAI,IAAI,CAAC,QAAQ,CAAC,QAAQ,EAAE,CAAC;QAC5C,IAAI,UAAU,CAAC,WAAW,CAAC,QAAQ,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,CAAC;YACjD,MAAM,GAAG,GAAG,iBAAiB,CAAC,KAAK,CAAC,CAAC;YACrC,IAAI,GAAG,IAAI,CAAC,OAAO,CAAC,QAAQ,CAAC,GAAG,CAAC,EAAE,CAAC;gBACnC,OAAO,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;YACnB,CAAC;QACF,CAAC;IACF,CAAC;IACD,OAAO,OAAO,CAAC;AAChB,CAAC;AAED,gEAAgE;AAChE,SAAS,iBAAiB,CAAC,IAAuB;IACjD,kDAAkD;IAClD,MAAM,UAAU,GAAG,cAAc,CAAC,IAAI,EAAE,CAAC,CAAC,EAAE,EAAE,CAC7C,CAAC,CAAC,IAAI,KAAK,QAAQ;QACnB,CAAC,CAAC,IAAI,KAAK,gBAAgB;QAC3B,CAAC,CAAC,IAAI,KAAK,4BAA4B,CACvC,CAAC;IACF,IAAI,UAAU,EAAE,CAAC;QAChB,eAAe;QACf,OAAO,UAAU,CAAC,IAAI,CAAC,OAAO,CAAC,cAAc,EAAE,EAAE,CAAC,CAAC;IACpD,CAAC;IAED,mDAAmD;IACnD,MAAM,UAAU,GAAG,cAAc,CAAC,IAAI,EAAE,CAAC,CAAC,EAAE,EAAE,CAC7C,CAAC,CAAC,IAAI,KAAK,aAAa,IAAI,CAAC,CAAC,IAAI,KAAK,aAAa,CACpD,CAAC;IACF,IAAI,UAAU;QAAE,OAAO,UAAU,CAAC,IAAI,CAAC;IAEvC,4BAA4B;IAC5B,MAAM,IAAI,GAAG,cAAc,CAAC,IAAI,EAAE,CAAC,CAAC,EAAE,EAAE,CACvC,CAAC,CAAC,IAAI,KAAK,mBAAmB,IAAI,CAAC,CAAC,IAAI,KAAK,UAAU,CACvD,CAAC;IACF,IAAI,IAAI;QAAE,OAAO,IAAI,CAAC,IAAI,CAAC;IAE3B,qCAAqC;IACrC,MAAM,MAAM,GAAG,cAAc,CAAC,IAAI,EAAE,CAAC,CAAC,EAAE,EAAE,CACzC,CAAC,CAAC,IAAI,KAAK,mBAAmB,IAAI,CAAC,CAAC,IAAI,KAAK,gBAAgB,CAC7D,CAAC;IACF,IAAI,MAAM;QAAE,OAAO,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC,cAAc,EAAE,EAAE,CAAC,CAAC;IAE3D,OAAO,IAAI,CAAC;AACb,CAAC;AAED,SAAS,cAAc,CACtB,IAAuB,EACvB,SAA4C;IAE5C,KAAK,MAAM,KAAK,IAAI,IAAI,CAAC,QAAQ,EAAE,CAAC;QACnC,IAAI,SAAS,CAAC,KAAK,CAAC;YAAE,OAAO,KAAK,CAAC;QACnC,MAAM,KAAK,GAAG,cAAc,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;QAC/C,IAAI,KAAK;YAAE,OAAO,KAAK,CAAC;IACzB,CAAC;IACD,OAAO,IAAI,CAAC;AACb,CAAC;AAED,4DAA4D;AAC5D,MAAM,UAAU,wBAAwB,CACvC,IAAuB,EACvB,UAA0B;IAE1B,MAAM,MAAM,GAAG,IAAI,CAAC,MAAM,CAAC;IAC3B,IAAI,CAAC,MAAM;QAAE,OAAO,EAAE,CAAC;IAEvB,MAAM,QAAQ,GAAa,EAAE,CAAC;IAC9B,KAAK,MAAM,KAAK,IAAI,MAAM,CAAC,QAAQ,EAAE,CAAC;QACrC,IAAI,KAAK,CAAC,EAAE,KAAK,IAAI,CAAC,EAAE;YAAE,SAAS;QACnC,IAAI,CAAC,UAAU,CAAC,UAAU,CAAC,QAAQ,CAAC,KAAK,CAAC,IAAI,CAAC;YAAE,SAAS;QAE1D,MAAM,IAAI,GAAG,eAAe,CAAC,KAAK,CAAC,CAAC;QACpC,IAAI,IAAI,EAAE,CAAC;YACV,MAAM,SAAS,GAAG,KAAK,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;YACnD,IAAI,SAAS,CAAC,MAAM,GAAG,GAAG,EAAE,CAAC;gBAC5B,QAAQ,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;YAC1B,CAAC;QACF,CAAC;IACF,CAAC;IACD,OAAO,QAAQ,CAAC,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;AAC9B,CAAC;AAED,wEAAwE;AACxE,MAAM,UAAU,qBAAqB,CACpC,IAAuB,EACvB,MAAc;IAEd,MAAM,KAAK,GAAG,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;IACjC,MAAM,aAAa,GAAG,IAAI,CAAC,aAAa,CAAC,GAAG,CAAC;IAC7C,MAAM,SAAS,GAAa,EAAE,CAAC;IAE/B,+CAA+C;IAC/C,KAAK,IAAI,CAAC,GAAG,aAAa,GAAG,CAAC,EAAE,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,aAAa,GAAG,EAAE,EAAE,CAAC,EAAE,EAAE,CAAC;QACxE,MAAM,IAAI,GAAG,KAAK,CAAC,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;QAC7B,IACC,IAAI,CAAC,UAAU,CAAC,IAAI,CAAC;YACrB,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC;YACpB,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC;YACpB,IAAI,CAAC,UAAU,CAAC,IAAI,CAAC;YACrB,IAAI,CAAC,UAAU,CAAC,IAAI,CAAC;YACrB,IAAI,CAAC,UAAU,CAAC,KAAK,CAAC;YACtB,IAAI,CAAC,UAAU,CAAC,IAAI,CAAC;YACrB,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC;YACpB,IAAI,CAAC,UAAU,CAAC,KAAK,CAAC;YACtB,IAAI,CAAC,UAAU,CAAC,KAAK,CAAC,EACrB,CAAC;YACF,SAAS,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC;QAC7B,CAAC;aAAM,IAAI,IAAI,KAAK,EAAE,EAAE,CAAC;YACxB,2BAA2B;YAC3B,IAAI,SAAS,CAAC,MAAM,GAAG,CAAC;gBAAE,MAAM;QACjC,CAAC;aAAM,CAAC;YACP,MAAM;QACP,CAAC;IACF,CAAC;IAED,IAAI,SAAS,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,IAAI,CAAC;IACxC,OAAO,SAAS,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,CAAC;AACpC,CAAC;AAED,oEAAoE;AACpE,MAAM,UAAU,YAAY,CAC3B,IAAuB;IAEvB,IAAI,gBAAgB,CAAC,GAAG,CAAC,IAAI,CAAC,IAAI,CAAC;QAAE,OAAO,YAAY,CAAC;IACzD,IAAI,mBAAmB,CAAC,GAAG,CAAC,IAAI,CAAC,IAAI,CAAC;QAAE,OAAO,eAAe,CAAC;IAC/D,OAAO,gBAAgB,CAAC;AACzB,CAAC;AAED,kDAAkD;AAClD,MAAM,UAAU,kBAAkB,CAAC,GAAiB;IACnD,MAAM,KAAK,GAAa,EAAE,CAAC;IAE3B,KAAK,CAAC,IAAI,CAAC,UAAU,GAAG,CAAC,QAAQ,GAAG,CAAC,CAAC;IACtC,KAAK,CAAC,IAAI,CAAC,UAAU,GAAG,CAAC,IAAI,GAAG,CAAC,CAAC;IAElC,IAAI,GAAG,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QAC1B,MAAM,QAAQ,GAAG,GAAG,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,GAAG,CAAC,CAAC,IAAI,IAAI,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;QACzE,KAAK,CAAC,IAAI,CAAC,WAAW,QAAQ,GAAG,CAAC,CAAC;IACpC,CAAC;IAED,IAAI,GAAG,CAAC,cAAc,EAAE,CAAC;QACxB,KAAK,CAAC,IAAI,CAAC,SAAS,GAAG,CAAC,cAAc,GAAG,CAAC,CAAC;IAC5C,CAAC;IAED,OAAO,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AACzB,CAAC;AAED,wDAAwD;AACxD,SAAS,eAAe,CAAC,IAAuB;IAC/C,MAAM,QAAQ,GACb,IAAI,CAAC,iBAAiB,CAAC,MAAM,CAAC;QAC9B,IAAI,CAAC,QAAQ,CAAC,IAAI,CACjB,CAAC,CAAoB,EAAE,EAAE,CACxB,CAAC,CAAC,IAAI,KAAK,YAAY,IAAI,CAAC,CAAC,IAAI,KAAK,iBAAiB,CACxD,CAAC;IACH,OAAO,QAAQ,EAAE,IAAI,CAAC;AACvB,CAAC"}

package/dist/chunker/index.d.ts

@@ -0,0 +1,3 @@
+import type { Chunk } from "../types.js";
+/** Chunk a single file into context-enriched chunks */
+export declare function chunkFile(filePath: string, cwd: string): Promise<Chunk[]>;

package/dist/chunker/index.js

@@ -0,0 +1,145 @@
+import { createHash } from "node:crypto";
+import { readFileSync } from "node:fs";
+import { join } from "node:path";
+import Parser from "web-tree-sitter";
+import { buildContextString, classifyRole, extractLeadingComment, extractScope, } from "./context.js";
+import { getLanguageForFile, getWasmPath, } from "./languages.js";
+const MAX_CHUNK_TOKENS = 8192;
+let parserInstance;
+const loadedLanguages = new Map();
+async function getParser() {
+    if (!parserInstance) {
+        await Parser.init();
+        parserInstance = new Parser();
+    }
+    return parserInstance;
+}
+async function getLanguage(langConfig) {
+    const cached = loadedLanguages.get(langConfig.id);
+    if (cached)
+        return cached;
+    const wasmPath = getWasmPath(langConfig);
+    if (!wasmPath)
+        return undefined;
+    const lang = await Parser.Language.load(wasmPath);
+    loadedLanguages.set(langConfig.id, lang);
+    return lang;
+}
+/** Chunk a single file into context-enriched chunks */
+export async function chunkFile(filePath, cwd) {
+    const langConfig = getLanguageForFile(filePath);
+    if (!langConfig)
+        return fallbackChunk(filePath, cwd);
+    const parser = await getParser();
+    const language = await getLanguage(langConfig);
+    if (!language)
+        return fallbackChunk(filePath, cwd);
+    parser.setLanguage(language);
+    const absolutePath = join(cwd, filePath);
+    const source = readFileSync(absolutePath, "utf-8");
+    const tree = parser.parse(source);
+    if (!tree)
+        return fallbackChunk(filePath, cwd);
+    const chunks = [];
+    collectChunks(tree.rootNode, langConfig, filePath, source, chunks);
+    // If no AST chunks found, fall back to the whole file
+    if (chunks.length === 0) {
+        return fallbackChunk(filePath, cwd);
+    }
+    return chunks;
+}
+function collectChunks(node, langConfig, filePath, source, chunks) {
+    if (langConfig.chunkTypes.includes(node.type)) {
+        const content = node.text;
+        // If too large, recurse into children instead of chunking the whole node
+        const estimatedTokens = Math.ceil(content.length / 4);
+        if (estimatedTokens > MAX_CHUNK_TOKENS && hasChunkableDescendants(node, langConfig)) {
+            for (const child of node.children) {
+                collectChunks(child, langConfig, filePath, source, chunks);
+            }
+            return;
+        }
+        const name = extractNodeName(node) ?? `anonymous_${node.startPosition.row}`;
+        // Skip very small chunks (less than 2 lines)
+        if (content.split("\n").length < 2 && content.length < 50) {
+            return;
+        }
+        const scope = extractScope(node, langConfig);
+        const leadingComment = extractLeadingComment(node, source);
+        const role = classifyRole(node);
+        const context = buildContextString({
+            filePath,
+            scope,
+            leadingComment,
+            role,
+        });
+        const hash = createHash("sha256")
+            .update(content)
+            .digest("hex")
+            .slice(0, 16);
+        chunks.push({
+            id: `${filePath}:${node.startPosition.row}:${hash}`,
+            filePath,
+            startLine: node.startPosition.row + 1,
+            endLine: node.endPosition.row + 1,
+            type: node.type,
+            name,
+            content,
+            context,
+            hash,
+        });
+        return;
+    }
+    // Recurse into children
+    for (const child of node.children) {
+        collectChunks(child, langConfig, filePath, source, chunks);
+    }
+}
+function hasChunkableDescendants(node, langConfig) {
+    for (const child of node.children) {
+        if (langConfig.chunkTypes.includes(child.type))
+            return true;
+        if (hasChunkableDescendants(child, langConfig))
+            return true;
+    }
+    return false;
+}
+function extractNodeName(node) {
+    const nameNode = node.childForFieldName("name") ??
+        node.children.find((c) => c.type === "identifier" || c.type === "type_identifier");
+    return nameNode?.text;
+}
+/** Fallback: chunk by sliding window for unsupported or empty-parse files */
+function fallbackChunk(filePath, cwd) {
+    const absolutePath = join(cwd, filePath);
+    const source = readFileSync(absolutePath, "utf-8");
+    const lines = source.split("\n");
+    if (lines.length === 0)
+        return [];
+    const WINDOW = 50;
+    const STRIDE = 25;
+    const chunks = [];
+    for (let i = 0; i < lines.length; i += STRIDE) {
+        const slice = lines.slice(i, i + WINDOW);
+        const content = slice.join("\n");
+        if (content.trim().length === 0)
+            continue;
+        const hash = createHash("sha256")
+            .update(content)
+            .digest("hex")
+            .slice(0, 16);
+        chunks.push({
+            id: `${filePath}:${i}:${hash}`,
+            filePath,
+            startLine: i + 1,
+            endLine: Math.min(i + WINDOW, lines.length),
+            type: "block",
+            name: `lines_${i + 1}_${Math.min(i + WINDOW, lines.length)}`,
+            content,
+            context: `[file: ${filePath}]`,
+            hash,
+        });
+    }
+    return chunks;
+}
+//# sourceMappingURL=index.js.map

package/dist/chunker/index.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/chunker/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AACzC,OAAO,EAAE,YAAY,EAAE,MAAM,SAAS,CAAC;AACvC,OAAO,EAAE,IAAI,EAAE,MAAM,WAAW,CAAC;AACjC,OAAO,MAAM,MAAM,iBAAiB,CAAC;AAErC,OAAO,EACN,kBAAkB,EAClB,YAAY,EACZ,qBAAqB,EACrB,YAAY,GACZ,MAAM,cAAc,CAAC;AACtB,OAAO,EAEN,kBAAkB,EAClB,WAAW,GACX,MAAM,gBAAgB,CAAC;AAExB,MAAM,gBAAgB,GAAG,IAAI,CAAC;AAE9B,IAAI,cAAkC,CAAC;AACvC,MAAM,eAAe,GAAG,IAAI,GAAG,EAA2B,CAAC;AAE3D,KAAK,UAAU,SAAS;IACvB,IAAI,CAAC,cAAc,EAAE,CAAC;QACrB,MAAM,MAAM,CAAC,IAAI,EAAE,CAAC;QACpB,cAAc,GAAG,IAAI,MAAM,EAAE,CAAC;IAC/B,CAAC;IACD,OAAO,cAAc,CAAC;AACvB,CAAC;AAED,KAAK,UAAU,WAAW,CACzB,UAA0B;IAE1B,MAAM,MAAM,GAAG,eAAe,CAAC,GAAG,CAAC,UAAU,CAAC,EAAE,CAAC,CAAC;IAClD,IAAI,MAAM;QAAE,OAAO,MAAM,CAAC;IAE1B,MAAM,QAAQ,GAAG,WAAW,CAAC,UAAU,CAAC,CAAC;IACzC,IAAI,CAAC,QAAQ;QAAE,OAAO,SAAS,CAAC;IAEhC,MAAM,IAAI,GAAG,MAAM,MAAM,CAAC,QAAQ,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC;IAClD,eAAe,CAAC,GAAG,CAAC,UAAU,CAAC,EAAE,EAAE,IAAI,CAAC,CAAC;IACzC,OAAO,IAAI,CAAC;AACb,CAAC;AAED,uDAAuD;AACvD,MAAM,CAAC,KAAK,UAAU,SAAS,CAC9B,QAAgB,EAChB,GAAW;IAEX,MAAM,UAAU,GAAG,kBAAkB,CAAC,QAAQ,CAAC,CAAC;IAChD,IAAI,CAAC,UAAU;QAAE,OAAO,aAAa,CAAC,QAAQ,EAAE,GAAG,CAAC,CAAC;IAErD,MAAM,MAAM,GAAG,MAAM,SAAS,EAAE,CAAC;IACjC,MAAM,QAAQ,GAAG,MAAM,WAAW,CAAC,UAAU,CAAC,CAAC;IAC/C,IAAI,CAAC,QAAQ;QAAE,OAAO,aAAa,CAAC,QAAQ,EAAE,GAAG,CAAC,CAAC;IAEnD,MAAM,CAAC,WAAW,CAAC,QAAQ,CAAC,CAAC;IAE7B,MAAM,YAAY,GAAG,IAAI,CAAC,GAAG,EAAE,QAAQ,CAAC,CAAC;IACzC,MAAM,MAAM,GAAG,YAAY,CAAC,YAAY,EAAE,OAAO,CAAC,CAAC;IACnD,MAAM,IAAI,GAAG,MAAM,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC;IAClC,IAAI,CAAC,IAAI;QAAE,OAAO,aAAa,CAAC,QAAQ,EAAE,GAAG,CAAC,CAAC;IAE/C,MAAM,MAAM,GAAY,EAAE,CAAC;IAE3B,aAAa,CAAC,IAAI,CAAC,QAAQ,EAAE,UAAU,EAAE,QAAQ,EAAE,MAAM,EAAE,MAAM,CAAC,CAAC;IAEnE,sDAAsD;IACtD,IAAI,MAAM,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACzB,OAAO,aAAa,CAAC,QAAQ,EAAE,GAAG,CAAC,CAAC;IACrC,CAAC;IAED,OAAO,MAAM,CAAC;AACf,CAAC;AAED,SAAS,aAAa,CACrB,IAAuB,EACvB,UAA0B,EAC1B,QAAgB,EAChB,MAAc,EACd,MAAe;IAEf,IAAI,UAAU,CAAC,UAAU,CAAC,QAAQ,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC;QAC/C,MAAM,OAAO,GAAG,IAAI,CAAC,IAAI,CAAC;QAE1B,yEAAyE;QACzE,MAAM,eAAe,GAAG,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC;QACtD,IAAI,eAAe,GAAG,gBAAgB,IAAI,uBAAuB,CAAC,IAAI,EAAE,UAAU,CAAC,EAAE,CAAC;YACrF,KAAK,MAAM,KAAK,IAAI,IAAI,CAAC,QAAQ,EAAE,CAAC;gBACnC,aAAa,CAAC,KAAK,EAAE,UAAU,EAAE,QAAQ,EAAE,MAAM,EAAE,MAAM,CAAC,CAAC;YAC5D,CAAC;YACD,OAAO;QACR,CAAC;QAED,MAAM,IAAI,GAAG,eAAe,CAAC,IAAI,CAAC,IAAI,aAAa,IAAI,CAAC,aAAa,CAAC,GAAG,EAAE,CAAC;QAE5E,6CAA6C;QAC7C,IAAI,OAAO,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,MAAM,GAAG,CAAC,IAAI,OAAO,CAAC,MAAM,GAAG,EAAE,EAAE,CAAC;YAC3D,OAAO;QACR,CAAC;QAED,MAAM,KAAK,GAAG,YAAY,CAAC,IAAI,EAAE,UAAU,CAAC,CAAC;QAC7C,MAAM,cAAc,GAAG,qBAAqB,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;QAC3D,MAAM,IAAI,GAAG,YAAY,CAAC,IAAI,CAAC,CAAC;QAEhC,MAAM,OAAO,GAAG,kBAAkB,CAAC;YAClC,QAAQ;YACR,KAAK;YACL,cAAc;YACd,IAAI;SACJ,CAAC,CAAC;QAEH,MAAM,IAAI,GAAG,UAAU,CAAC,QAAQ,CAAC;aAC/B,MAAM,CAAC,OAAO,CAAC;aACf,MAAM,CAAC,KAAK,CAAC;aACb,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;QAEf,MAAM,CAAC,IAAI,CAAC;YACX,EAAE,EAAE,GAAG,QAAQ,IAAI,IAAI,CAAC,aAAa,CAAC,GAAG,IAAI,IAAI,EAAE;YACnD,QAAQ;YACR,SAAS,EAAE,IAAI,CAAC,aAAa,CAAC,GAAG,GAAG,CAAC;YACrC,OAAO,EAAE,IAAI,CAAC,WAAW,CAAC,GAAG,GAAG,CAAC;YACjC,IAAI,EAAE,IAAI,CAAC,IAAI;YACf,IAAI;YACJ,OAAO;YACP,OAAO;YACP,IAAI;SACJ,CAAC,CAAC;QACH,OAAO;IACR,CAAC;IAED,wBAAwB;IACxB,KAAK,MAAM,KAAK,IAAI,IAAI,CAAC,QAAQ,EAAE,CAAC;QACnC,aAAa,CAAC,KAAK,EAAE,UAAU,EAAE,QAAQ,EAAE,MAAM,EAAE,MAAM,CAAC,CAAC;IAC5D,CAAC;AACF,CAAC;AAED,SAAS,uBAAuB,CAC/B,IAAuB,EACvB,UAA0B;IAE1B,KAAK,MAAM,KAAK,IAAI,IAAI,CAAC,QAAQ,EAAE,CAAC;QACnC,IAAI,UAAU,CAAC,UAAU,CAAC,QAAQ,CAAC,KAAK,CAAC,IAAI,CAAC;YAAE,OAAO,IAAI,CAAC;QAC5D,IAAI,uBAAuB,CAAC,KAAK,EAAE,UAAU,CAAC;YAAE,OAAO,IAAI,CAAC;IAC7D,CAAC;IACD,OAAO,KAAK,CAAC;AACd,CAAC;AAED,SAAS,eAAe,CAAC,IAAuB;IAC/C,MAAM,QAAQ,GACb,IAAI,CAAC,iBAAiB,CAAC,MAAM,CAAC;QAC9B,IAAI,CAAC,QAAQ,CAAC,IAAI,CACjB,CAAC,CAAoB,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,KAAK,YAAY,IAAI,CAAC,CAAC,IAAI,KAAK,iBAAiB,CACjF,CAAC;IACH,OAAO,QAAQ,EAAE,IAAI,CAAC;AACvB,CAAC;AAED,6EAA6E;AAC7E,SAAS,aAAa,CAAC,QAAgB,EAAE,GAAW;IACnD,MAAM,YAAY,GAAG,IAAI,CAAC,GAAG,EAAE,QAAQ,CAAC,CAAC;IACzC,MAAM,MAAM,GAAG,YAAY,CAAC,YAAY,EAAE,OAAO,CAAC,CAAC;IACnD,MAAM,KAAK,GAAG,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;IAEjC,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,EAAE,CAAC;IAElC,MAAM,MAAM,GAAG,EAAE,CAAC;IAClB,MAAM,MAAM,GAAG,EAAE,CAAC;IAClB,MAAM,MAAM,GAAY,EAAE,CAAC;IAE3B,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,KAAK,CAAC,MAAM,EAAE,CAAC,IAAI,MAAM,EAAE,CAAC;QAC/C,MAAM,KAAK,GAAG,KAAK,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,GAAG,MAAM,CAAC,CAAC;QACzC,MAAM,OAAO,GAAG,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;QACjC,IAAI,OAAO,CAAC,IAAI,EAAE,CAAC,MAAM,KAAK,CAAC;YAAE,SAAS;QAE1C,MAAM,IAAI,GAAG,UAAU,CAAC,QAAQ,CAAC;aAC/B,MAAM,CAAC,OAAO,CAAC;aACf,MAAM,CAAC,KAAK,CAAC;aACb,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;QAEf,MAAM,CAAC,IAAI,CAAC;YACX,EAAE,EAAE,GAAG,QAAQ,IAAI,CAAC,IAAI,IAAI,EAAE;YAC9B,QAAQ;YACR,SAAS,EAAE,CAAC,GAAG,CAAC;YAChB,OAAO,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,GAAG,MAAM,EAAE,KAAK,CAAC,MAAM,CAAC;YAC3C,IAAI,EAAE,OAAO;YACb,IAAI,EAAE,SAAS,CAAC,GAAG,CAAC,IAAI,IAAI,CAAC,GAAG,CAAC,CAAC,GAAG,MAAM,EAAE,KAAK,CAAC,MAAM,CAAC,EAAE;YAC5D,OAAO;YACP,OAAO,EAAE,UAAU,QAAQ,GAAG;YAC9B,IAAI;SACJ,CAAC,CAAC;IACJ,CAAC;IAED,OAAO,MAAM,CAAC;AACf,CAAC"}