brainbank 0.1.1 → 0.2.0

package/README.md

@@ -11,7 +11,7 @@ BrainBank gives LLMs a long-term memory that persists between sessions.
 - **Pluggable embeddings** — local WASM (free) or OpenAI (higher quality)
 - **Multi-repo** — index multiple repositories into one shared database
 - **Portable** — single `.brainbank/brainbank.db` file
-- **Optional packages** — [`@brainbank/memory`](#memory) (deterministic fact extraction), [`@brainbank/reranker`](#reranker) (Qwen3 cross-encoder), [`@brainbank/mcp`](#mcp-server) (MCP server)
+- **Optional packages** — [`@brainbank/memory`](#memory) (fact extraction + entity graph), [`@brainbank/reranker`](#reranker) (Qwen3 cross-encoder), [`@brainbank/mcp`](#mcp-server) (MCP server)
 
 ![BrainBank Architecture](assets/architecture.png)
 
@@ -27,19 +27,21 @@ Most AI memory solutions (mem0, Zep, LangMem) require cloud services, external d
 |---|:---:|:---:|:---:|:---:|
 | Infrastructure | **SQLite file** | Vector DB + cloud | Neo4j + cloud | LangGraph Platform |
 | LLM required to write | **No**¹ | Yes | Yes | Yes |
-| Code-aware | **30+ languages, git, co-edits** | ✗ | ✗ | ✗ |
+| Code-aware | **19 AST-parsed languages (tree-sitter), git, co-edits** | ✗ | ✗ | ✗ |
 | Custom indexers | **`.use()` plugin system** | ✗ | ✗ | ✗ |
-| Search | **Vector + BM25 + RRF** | Vector only | Vector + graph | Vector only |
+| Search | **Vector + BM25 + RRF** | Vector + graph² | Vector + BM25 + graph | Vector only |
 | Framework lock-in | **None** | Optional | Zep cloud | LangChain |
 | Portable | **Copy one file** | Tied to DB | Tied to cloud | Tied to platform |
 
 > ¹ mem0 and Zep use LLMs to auto-extract memories from raw text. BrainBank is explicit — you decide what gets stored. Less magic, more control.
+>
+> ² mem0's graph store (mem0g) is available in the paid platform version.
 
 **In short:**
 - **Code-first** — the only memory layer that understands code structure, git history, and file co-edit relationships
+- **Framework-agnostic** — plain TypeScript, works with any agent framework (LangChain, Vercel AI SDK, custom) or none at all. Unopinionated — doesn't force you into a specific pattern
 - **$0 memory bill** — no LLM calls to extract/consolidate. You store what you want, BrainBank embeds deterministically
 - **Truly portable** — `.brainbank/brainbank.db` is a normal file. Copy it, back it up, `git lfs` it
-- **No vendor lock-in** — plain TypeScript, works with any agent framework or none at all
 
 ### Table of Contents
 
@@ -68,6 +70,9 @@ Most AI memory solutions (mem0, Zep, LangMem) require cloud services, external d
   - [Re-embedding](#re-embedding)
 - [Architecture](#architecture)
   - [Search Pipeline](#search-pipeline)
+- [Benchmarks](#benchmarks)
+  - [Search Quality: AST vs Sliding Window](#search-quality-ast-vs-sliding-window)
+  - [Grammar Support](#grammar-support)
 
 ---
 
@@ -81,7 +86,7 @@ npm install brainbank
 
 | Package | When to install |
 |---------|----------------|
-| `@brainbank/memory` | Deterministic memory extraction for LLM conversations (mem0-style pipeline) |
+| `@brainbank/memory` | Deterministic memory extraction + entity graph for LLM conversations |
 | `@brainbank/reranker` | Cross-encoder reranker (Qwen3-0.6B, ~640MB model) |
 | `@brainbank/mcp` | MCP server for AI tool integration |
 
@@ -148,12 +153,15 @@ BrainBank can be used entirely from the command line — no config file needed.
 
 ### Indexing
 
-`index` processes **code files + git history** only. Document collections are indexed separately with `docs`.
+`index` processes **code files + git history** by default. Use `--only` to select specific modules, and `--docs` to include document collections.
 
 ```bash
 brainbank index [path]                      # Index code + git history
 brainbank index [path] --force              # Force re-index everything
 brainbank index [path] --depth 200          # Limit git commit depth
+brainbank index [path] --only code          # Index only code (skip git)
+brainbank index [path] --only git           # Index only git history
+brainbank index [path] --docs ~/docs        # Include a docs folder
 brainbank docs [--collection <name>]        # Index document collections
 ```
 
@@ -232,7 +240,7 @@ BrainBank uses pluggable indexers. Register only what you need with `.use()`:
 
 | Indexer | Import | Description |
 |---------|--------|-------------|
-| `code` | `brainbank/code` | Language-aware code chunking (30+ languages) |
+| `code` | `brainbank/code` | AST-aware code chunking via tree-sitter (19 languages) |
 | `git` | `brainbank/git` | Git commit history, diffs, co-edit relationships |
 | `docs` | `brainbank/docs` | Document collections (markdown, wikis) |
 
@@ -763,6 +771,8 @@ Without a reranker, BrainBank uses pure RRF fusion — which is already producti
 
 `@brainbank/memory` adds **deterministic memory extraction** to any LLM conversation. After every turn, it automatically extracts facts, deduplicates against existing memories, and decides `ADD` / `UPDATE` / `NONE` — no function calling needed.
 
+Optionally extracts **entities and relationships** (knowledge graph) from the same LLM call — no extra cost. Includes **LLM-powered entity resolution** to merge aliases (e.g. "TS" → "TypeScript").
+
 Inspired by [mem0](https://github.com/mem0ai/mem0)'s pipeline, but framework-agnostic and built on BrainBank collections.
 
 ```bash
@@ -771,22 +781,32 @@ npm install @brainbank/memory
 
 ```typescript
 import { BrainBank } from 'brainbank';
-import { Memory, OpenAIProvider } from '@brainbank/memory';
+import { Memory, EntityStore, OpenAIProvider } from '@brainbank/memory';
 
 const brain = new BrainBank({ dbPath: './memory.db' });
 await brain.initialize();
 
-const memory = new Memory(brain.collection('memories'), {
-  llm: new OpenAIProvider({ model: 'gpt-4.1-nano' }),
+const llm = new OpenAIProvider({ model: 'gpt-4.1-nano' });
+
+// Opt-in entity extraction (knowledge graph)
+const entityStore = new EntityStore(brain, {
+  onEntity: (op) => console.log(`${op.action}: ${op.name}`),
+});
+
+const memory = new Memory(brain, {
+  llm,              // auto-shared with EntityStore
+  entityStore,      // optional — omit for facts-only mode
+  onOperation: (op) => console.log(`${op.action}: ${op.fact}`),
 });
 
-// After every conversation turn (deterministic, automatic)
-await memory.process(userMessage, assistantResponse);
-// → extracts facts, deduplicates, executes ADD/UPDATE/NONE
+// After every conversation turn
+const result = await memory.process(userMessage, assistantResponse);
+// result.operations → [{ fact, action: "ADD", reason }]
+// result.entities   → { entitiesProcessed: 2, relationshipsProcessed: 1 }
 
-// For the system prompt
+// System prompt with memories + entities
 const context = memory.buildContext();
-// → "## Memories\n- User's name is Berna\n- Prefers TypeScript"
+// → "## Memories\n- User's name is Berna\n\n## Known Entities\n- Berna (person, 3x)\n..."
 ```
 
 The `LLMProvider` interface works with any framework:
@@ -899,6 +919,24 @@ Instances are cached in memory after first initialization, so subsequent queries
 
 ## Indexing
 
+### Code Chunking (tree-sitter)
+
+BrainBank uses **native tree-sitter** to parse source code into ASTs and extract semantic blocks — functions, classes, methods, interfaces — as individual chunks. This produces dramatically better embeddings than naive line-based splitting.
+
+**Supported languages (AST-parsed):**
+
+| Category | Languages |
+|----------|-----------|
+| Web | TypeScript, JavaScript, HTML, CSS |
+| Systems | Go, Rust, C, C++, Swift |
+| JVM | Java, Kotlin, Scala |
+| Scripting | Python, Ruby, PHP, Lua, Bash, Elixir |
+| .NET | C# |
+
+For large classes (>80 lines), the chunker descends into the class body and extracts each method as a separate chunk. For unsupported languages, it falls back to a sliding window with overlap.
+
+> Tree-sitter grammars are **optional dependencies**. If a grammar isn't installed, that language falls back to the generic sliding window. Install only the grammars you need: `npm install tree-sitter-ruby tree-sitter-go` etc.
+
 ### Incremental Indexing
 
 All indexing is **incremental by default** — only new or changed content is processed:
@@ -971,6 +1009,101 @@ brainbank reembed
 
 ---
 
+## Benchmarks
+
+BrainBank includes benchmark scripts to validate chunking quality and search relevance. Run them against your own codebase to see the impact.
+
+### Search Quality: AST vs Sliding Window
+
+We compared BrainBank's **tree-sitter AST chunker** against the traditional **sliding window** (80-line blocks) on a production NestJS backend (3,753 lines across 8 service files). Both strategies chunk the same files; all chunks are embedded and searched with the same 10 domain-specific queries.
+
+#### How It Works
+
+```
+Sliding Window                          Tree-Sitter AST
+┌────────────────────┐                  ┌────────────────────┐
+│ import { ... }     │                  │ ✓ constructor()    │  → named chunk
+│ @Injectable()      │  → L1-80 block   │ ✓ findAll()        │  → named chunk
+│ class JobsService {│                  │ ✓ createJob()      │  → named chunk
+│   constructor()    │                  │ ✓ cancelJob()      │  → named chunk
+│   findAll() { ... }│                  │ ✓ updateStatus()   │  → named chunk
+│   createJob()      │                  └────────────────────┘
+│   ...              │
+│ ────────────────── │  overlaps ↕
+│   cancelJob()      │  → L75-155 block
+│   updateStatus()   │
+│   ...              │
+└────────────────────┘
+```
+
+**Sliding window** mixes imports, constructors, and multiple methods into one embedding. Search for "cancel a job" and you get a generic block.
+**AST chunking** gives each method its own embedding. Search for "cancel a job" → direct hit on `cancelJob()`.
+
+#### Results (Production NestJS Backend — 3,753 lines)
+
+Tested with 10 domain-specific queries on 8 service files (`orders.service.ts`, `bookings.service.ts`, `notifications.service.ts`, etc.):
+
+| Metric | Sliding Window | Tree-Sitter AST |
+|--------|:-:|:-:|
+| **Query Wins** | 0/10 | **8/10** (2 ties) |
+| **Top-1 Relevant** | 3/10 | **8/10** |
+| **Avg Precision@3** | 1.1/3 | **1.7/3** |
+| **Avg Score Delta** | — | **+0.035** |
+
+#### Per-Query Breakdown
+
+| Query | SW Top Result | AST Top Result | Δ Score |
+|-------|:---:|:---:|:---:|
+| cancel an order | generic `L451-458` | **`updateOrderStatus`** | +0.005 |
+| create a booking | generic `L451-458` | **`createInstantBooking`** | +0.068 |
+| confirm booking | generic `L451-458` | **`confirm`** | +0.034 |
+| send notification | generic `L226-305` | **`publishNotificationEvent`** | +0.034 |
+| authenticate JWT | generic `L1-80` | **`AuthModule`** | +0.032 |
+| tenant DB connection | `L76-155` | **`onModuleDestroy`** | +0.037 |
+| list orders paginated | `L76-155` | **`findAllActive`** | +0.045 |
+| reject booking | generic `L451-458` | **`reject`** | +0.090 |
+
+> Notice how the sliding window returns the **same generic block `L451-458`** for 4 different queries. The AST chunker returns a different, correctly named method each time.
+
+#### Chunk Quality Comparison
+
+| | Sliding Window | Tree-Sitter AST |
+|---|:-:|:-:|
+| Total chunks | 53 | **83** |
+| Avg lines/chunk | 75 | **39** |
+| Named chunks | 0 | **83** (100%) |
+| Chunk types | `block` | `method`, `interface`, `class` |
+
+### Grammar Support
+
+All 9 core grammars verified, each parsing in **<0.05ms**:
+
+| Language | AST Nodes Extracted | Parse Time |
+|----------|:---:|:---:|
+| TypeScript | `export_statement`, `interface_declaration` | 0.04ms |
+| JavaScript | `function_declaration` × 3 | 0.04ms |
+| Python | `class_definition`, `function_definition` × 2 | 0.03ms |
+| Go | `function_declaration`, `method_declaration` × 3 | 0.04ms |
+| Rust | `struct_item`, `impl_item`, `function_item` | 0.03ms |
+| Ruby | `class`, `method` | 0.03ms |
+| Java | `class_declaration` | 0.02ms |
+| C | `function_definition` × 3 | 0.05ms |
+| PHP | `class_declaration` | 0.03ms |
+
+> Additional grammars available: C++, Swift, C#, Kotlin, Scala, Lua, Elixir, Bash, HTML, CSS
+
+### Running Benchmarks
+
+```bash
+# Grammar support (9 languages, parse speed)
+node test/benchmarks/grammar-support.mjs
+
+# Search quality A/B (uses BrainBank's own source files)
+node test/benchmarks/search-quality.mjs
+```
+
+---
+
 ## Architecture
 
 <details>
@@ -1035,7 +1168,7 @@ Final results (sorted by blended score)
 
 ### Data Flow
 
-1. **Index** — Indexers parse files into chunks
+1. **Index** — Indexers parse files into chunks (tree-sitter AST for code, heading-based for docs)
 2. **Embed** — Each chunk gets a vector (local WASM or OpenAI)
 3. **Store** — Chunks + vectors → SQLite, vectors → HNSW index
 4. **Search** — Query → HNSW k-NN + BM25 keyword → RRF fusion → optional reranker

package/dist/{types-Da_zLLOl.d.ts → base-9vfWRHCV.d.ts}

@@ -101,10 +101,14 @@ interface SearchHit {
 interface VectorIndex {
     /** Initialize the index. Must be called before add/search. */
     init(): Promise<this>;
-    /** Add a vector with an integer ID. */
+    /** Add a vector with an integer ID. Idempotent: duplicate IDs are skipped. */
     add(vector: Float32Array, id: number): void;
+    /** Mark a vector as deleted so it no longer appears in searches. */
+    remove(id: number): void;
     /** Search for k nearest neighbors. */
     search(query: Float32Array, k: number): SearchHit[];
+    /** Clear all vectors and reset to empty state. */
+    reinit(): void;
     /** Number of vectors in the index. */
     readonly size: number;
 }
@@ -140,7 +144,7 @@ interface GitCommitRecord {
     deletions: number;
     isMerge: boolean;
 }
-interface MemoryPattern {
+interface LearningPattern {
     id?: number;
     /** Category (e.g. 'api', 'refactor', 'debug') */
     taskType: string;
@@ -166,25 +170,88 @@ interface DistilledStrategy {
     updatedAt: number;
 }
 type SearchResultType = 'code' | 'commit' | 'pattern' | 'document' | 'collection';
-interface SearchResult {
-    type: SearchResultType;
+interface CodeResultMetadata {
+    chunkType: string;
+    name?: string;
+    startLine: number;
+    endLine: number;
+    language: string;
+    searchType?: string;
+}
+interface CommitResultMetadata {
+    hash: string;
+    shortHash: string;
+    author: string;
+    date: string;
+    files: string[];
+    additions?: number;
+    deletions?: number;
+    diff?: string;
+    searchType?: string;
+}
+interface PatternResultMetadata {
+    taskType: string;
+    task: string;
+    outcome?: string;
+    successRate: number;
+    critique?: string;
+    searchType?: string;
+}
+interface DocumentResultMetadata {
+    collection?: string;
+    title?: string;
+    seq?: number;
+    path?: string;
+    searchType?: string;
+}
+interface CodeResult {
+    type: 'code';
+    score: number;
+    filePath: string;
+    content: string;
+    context?: string;
+    metadata: CodeResultMetadata;
+}
+interface CommitResult {
+    type: 'commit';
+    score: number;
+    filePath?: string;
+    content: string;
+    context?: string;
+    metadata: CommitResultMetadata;
+}
+interface PatternResult {
+    type: 'pattern';
+    score: number;
+    filePath?: string;
+    content: string;
+    context?: string;
+    metadata: PatternResultMetadata;
+}
+interface DocumentResult {
+    type: 'document';
+    score: number;
+    filePath?: string;
+    content: string;
+    context?: string;
+    metadata: DocumentResultMetadata;
+}
+interface CollectionResult {
+    type: 'collection';
     score: number;
-    /** File path (for code results) or document path */
     filePath?: string;
-    /** Content / text */
     content: string;
-    /** Context description (for document results) */
     context?: string;
-    /** Extra metadata depending on type */
     metadata: Record<string, any>;
 }
+type SearchResult = CodeResult | CommitResult | PatternResult | DocumentResult | CollectionResult;
 interface ContextOptions {
     /** Max code chunks to include. Default: 6 */
     codeResults?: number;
     /** Max git commits to include. Default: 5 */
     gitResults?: number;
     /** Max memory patterns to include. Default: 4 */
-    memoryResults?: number;
+    patternResults?: number;
     /** Files the agent is about to modify (improves co-edit suggestions) */
     affectedFiles?: string[];
     /** Minimum similarity score threshold. Default: 0.25 */
@@ -252,7 +319,10 @@ interface IndexStats {
         long: number;
     };
 }
+/** File-level progress (used by indexers). */
 type ProgressCallback = (file: string, current: number, total: number) => void;
+/** Stage-level progress (used by BrainBank.index() orchestrator). */
+type StageProgressCallback = (stage: string, message: string) => void;
 interface IndexResult {
     indexed: number;
     skipped: number;
@@ -278,18 +348,34 @@ declare class HNSWIndex implements VectorIndex {
     private _efConstruction;
     private _efSearch;
     private _index;
-    private _count;
+    private _lib;
+    private _ids;
     constructor(_dims: number, _maxElements?: number, _M?: number, _efConstruction?: number, _efSearch?: number);
     /**
      * Initialize the HNSW index.
      * Must be called before add/search.
      */
     init(): Promise<this>;
+    /**
+     * Reinitialize the index in-place, clearing all vectors.
+     * Required after reembed or full re-index to avoid duplicate IDs.
+     * init() must have been called first.
+     */
+    reinit(): void;
+    private _createIndex;
+    /** Maximum capacity of this index. */
+    get maxElements(): number;
     /**
      * Add a vector with an integer ID.
      * The vector should be pre-normalized for cosine distance.
      */
     add(vector: Float32Array, id: number): void;
+    /**
+     * Mark a vector as deleted so it no longer appears in searches.
+     * Uses hnswlib-node markDelete under the hood.
+     * Safe to call with an ID that doesn't exist.
+     */
+    remove(id: number): void;
     /**
      * Search for the k nearest neighbors.
      * Returns results sorted by score (highest first).
@@ -434,41 +520,55 @@ interface Indexer {
     readonly name: string;
     /** Initialize the indexer (create HNSW, load vectors, etc.). */
     initialize(ctx: IndexerContext): Promise<void>;
-    /** Index content. Implemented by code and git indexers. */
+    /** Index content (code, git plugins). */
     index?(options?: any): Promise<any>;
-    /** Search indexed content. Implemented by docs indexer. */
+    /** Search indexed content (docs plugin). */
     search?(query: string, options?: any): Promise<any[]>;
-    /** Register a document collection. */
+    /** Register a document collection (docs plugin). */
     addCollection?(collection: any): void;
-    /** Remove a collection. */
+    /** Remove a collection (docs plugin). */
     removeCollection?(name: string): void;
-    /** List registered collections. */
+    /** List registered collections (docs plugin). */
     listCollections?(): any[];
-    /** Index all or specific collections. */
+    /** Index collections (docs plugin). */
     indexCollections?(options?: any): Promise<any>;
-    /** Add context description for a collection path. */
+    /** Add context for a collection path (docs plugin). */
     addContext?(collection: string, path: string, context: string): void;
-    /** Remove context for a collection path. */
+    /** Remove context (docs plugin). */
     removeContext?(collection: string, path: string): void;
-    /** List all context entries. */
+    /** List context entries (docs plugin). */
     listContexts?(): any[];
-    /**
-     * Called by watch mode when a file changes.
-     * Return true if this indexer handled the change.
-     * If not implemented, watch will fall back to brain.index().
-     */
+    /** Watch mode: handle file change (returns true if handled). */
     onFileChange?(filePath: string, event: 'create' | 'update' | 'delete'): Promise<boolean>;
-    /**
-     * Glob patterns this indexer watches.
-     * If not set, defaults to all supported code extensions.
-     */
+    /** Glob patterns for watch mode. */
     watchPatterns?(): string[];
     /** Return stats for this indexer. */
     stats?(): Record<string, any>;
     /** Clean up resources. */
     close?(): void;
 }
-type BrainBankModule = Indexer;
-type ModuleContext = IndexerContext;
+/** Indexers that can scan and index content. */
+interface IndexablePlugin extends Indexer {
+    index(options?: any): Promise<any>;
+}
+/** Indexers that can search indexed content. */
+interface SearchablePlugin extends Indexer {
+    search(query: string, options?: any): Promise<any[]>;
+}
+/** Indexers that support file watch mode. */
+interface WatchablePlugin extends Indexer {
+    onFileChange(filePath: string, event: 'create' | 'update' | 'delete'): Promise<boolean>;
+    watchPatterns(): string[];
+}
+/** Indexers that manage document collections. */
+interface CollectionPlugin extends Indexer {
+    addCollection(collection: any): void;
+    removeCollection(name: string): void;
+    listCollections(): any[];
+    indexCollections(options?: any): Promise<any>;
+    addContext?(collection: string, path: string, context: string): void;
+    removeContext?(collection: string, path: string): void;
+    listContexts?(): any[];
+}
 
-export { type BrainBankModule as B, Collection as C, type DocumentCollection as D, type EmbeddingProvider as E, type GitCommitRecord as G, HNSWIndex as H, type Indexer as I, type MemoryPattern as M, type ProgressCallback as P, type ResolvedConfig as R, type SearchResult as S, type VectorIndex as V, type BrainBankConfig as a, type IndexResult as b, type ContextOptions as c, type CoEditSuggestion as d, type IndexStats as e, type SearchHit as f, type CodeChunk as g, Database as h, type Reranker as i, type CollectionAddOptions as j, type CollectionItem as k, type CollectionSearchOptions as l, type DistilledStrategy as m, type DocChunk as n, type IndexerContext as o, type ModuleContext as p, type SearchResultType as q };
+export { type SearchResultType as A, type BrainBankConfig as B, Collection as C, type DocumentCollection as D, type EmbeddingProvider as E, type SearchablePlugin as F, type GitCommitRecord as G, HNSWIndex as H, type Indexer as I, type LearningPattern as L, type ProgressCallback as P, type ResolvedConfig as R, type StageProgressCallback as S, type VectorIndex as V, type WatchablePlugin as W, type IndexResult as a, type SearchResult as b, type ContextOptions as c, type CoEditSuggestion as d, type IndexStats as e, type SearchHit as f, type CodeChunk as g, Database as h, type Reranker as i, type CodeResult as j, type CodeResultMetadata as k, type CollectionAddOptions as l, type CollectionItem as m, type CollectionPlugin as n, type CollectionResult as o, type CollectionSearchOptions as p, type CommitResult as q, type CommitResultMetadata as r, type DistilledStrategy as s, type DocChunk as t, type DocumentResult as u, type DocumentResultMetadata as v, type IndexablePlugin as w, type IndexerContext as x, type PatternResult as y, type PatternResultMetadata as z };