claude_memory 0.5.1 → 0.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (68) hide show
  1. checksums.yaml +4 -4
  2. data/.claude/CLAUDE.md +1 -1
  3. data/.claude/memory.sqlite3 +0 -0
  4. data/.claude/memory.sqlite3-shm +0 -0
  5. data/.claude/memory.sqlite3-wal +0 -0
  6. data/.claude/rules/claude_memory.generated.md +1 -1
  7. data/.claude/settings.json +5 -0
  8. data/.claude/settings.local.json +19 -1
  9. data/.claude-plugin/marketplace.json +5 -2
  10. data/.claude-plugin/plugin.json +16 -3
  11. data/.gitattributes +1 -0
  12. data/CHANGELOG.md +91 -0
  13. data/CLAUDE.md +28 -14
  14. data/README.md +6 -2
  15. data/Rakefile +22 -0
  16. data/db/migrations/011_add_tool_call_summaries.rb +18 -0
  17. data/db/migrations/012_add_vec_indexing_support.rb +19 -0
  18. data/docs/improvements.md +225 -61
  19. data/docs/influence/claude-mem.md +253 -0
  20. data/docs/influence/claude-supermemory.md +158 -430
  21. data/docs/influence/episodic-memory.md +217 -0
  22. data/docs/influence/grepai.md +163 -839
  23. data/docs/influence/kbs.md +437 -0
  24. data/docs/influence/qmd.md +210 -481
  25. data/docs/quality_review.md +344 -56
  26. data/hooks/hooks.json +19 -15
  27. data/lefthook.yml +4 -0
  28. data/lib/claude_memory/commands/checks/database_check.rb +7 -0
  29. data/lib/claude_memory/commands/checks/vec_check.rb +73 -0
  30. data/lib/claude_memory/commands/compact_command.rb +104 -0
  31. data/lib/claude_memory/commands/doctor_command.rb +1 -0
  32. data/lib/claude_memory/commands/export_command.rb +116 -0
  33. data/lib/claude_memory/commands/git_lfs_command.rb +117 -0
  34. data/lib/claude_memory/commands/help_command.rb +2 -0
  35. data/lib/claude_memory/commands/hook_command.rb +110 -9
  36. data/lib/claude_memory/commands/index_command.rb +63 -8
  37. data/lib/claude_memory/commands/initializers/global_initializer.rb +26 -7
  38. data/lib/claude_memory/commands/initializers/project_initializer.rb +35 -12
  39. data/lib/claude_memory/commands/registry.rb +4 -1
  40. data/lib/claude_memory/commands/serve_mcp_command.rb +10 -1
  41. data/lib/claude_memory/commands/stats_command.rb +12 -1
  42. data/lib/claude_memory/configuration.rb +40 -1
  43. data/lib/claude_memory/core/snippet_extractor.rb +21 -19
  44. data/lib/claude_memory/hook/context_injector.rb +75 -0
  45. data/lib/claude_memory/hook/error_classifier.rb +67 -0
  46. data/lib/claude_memory/hook/handler.rb +21 -1
  47. data/lib/claude_memory/index/lexical_fts.rb +88 -16
  48. data/lib/claude_memory/index/vector_index.rb +171 -0
  49. data/lib/claude_memory/infrastructure/schema_validator.rb +5 -1
  50. data/lib/claude_memory/ingest/ingester.rb +26 -1
  51. data/lib/claude_memory/ingest/observation_compressor.rb +177 -0
  52. data/lib/claude_memory/mcp/instructions_builder.rb +76 -0
  53. data/lib/claude_memory/mcp/server.rb +3 -1
  54. data/lib/claude_memory/mcp/tool_definitions.rb +65 -27
  55. data/lib/claude_memory/mcp/tools.rb +137 -2
  56. data/lib/claude_memory/publish.rb +28 -27
  57. data/lib/claude_memory/recall/dual_query_template.rb +1 -12
  58. data/lib/claude_memory/recall.rb +71 -17
  59. data/lib/claude_memory/resolve/resolver.rb +22 -18
  60. data/lib/claude_memory/store/sqlite_store.rb +17 -1
  61. data/lib/claude_memory/store/store_manager.rb +19 -24
  62. data/lib/claude_memory/sweep/sweeper.rb +41 -2
  63. data/lib/claude_memory/version.rb +1 -1
  64. data/lib/claude_memory.rb +15 -0
  65. data/scripts/hook-runner.sh +14 -0
  66. data/scripts/serve-mcp.sh +14 -0
  67. data/skills/setup-memory/SKILL.md +6 -0
  68. metadata +36 -2
@@ -1,463 +1,172 @@
1
- # grepai Analysis
1
+ # grepai Analysis (Updated)
2
2
 
3
- *Analysis Date: 2026-01-29*
3
+ *Analysis Date: 2026-03-02*
4
+ *Previous Analysis: 2026-01-29*
4
5
  *Repository: https://github.com/yoanbernabeu/grepai*
5
- *Version/Commit: HEAD (main branch)*
6
+ *Version: 0.34.0 (commit 1c7aba9)*
6
7
 
7
8
  ---
8
9
 
9
10
  ## Executive Summary
10
11
 
11
12
  ### Project Purpose
12
- grepai is a privacy-first CLI tool for semantic code search using vector embeddings, enabling natural language queries that find relevant code based on intent rather than exact text matches.
13
13
 
14
- ### Key Innovation
15
- **Semantic search with zero cloud dependency** - Combines local vector embeddings (via Ollama) with file-watching for real-time index updates, drastically reducing AI agent token usage by ~80% through intelligent context retrieval instead of full codebase scans.
14
+ grepai is a privacy-first CLI for semantic code search using vector embeddings. It enables natural language queries that find relevant code by meaning, not just text — reducing AI agent input tokens by providing targeted context.
16
15
 
17
- ### Technology Stack
18
- | Component | Technology |
19
- |-----------|-----------|
20
- | **Language** | Go 1.22+ |
21
- | **Vector Store** | PostgreSQL with pgvector, Qdrant, or GOB (file-based) |
22
- | **Embedding** | Ollama (local), OpenAI, LM Studio |
23
- | **File Watching** | fsnotify |
24
- | **CLI Framework** | cobra |
25
- | **MCP Integration** | mark3labs/mcp-go |
26
- | **Call Graph** | tree-sitter for AST parsing |
27
- | **Testing** | Go stdlib testing with race detection |
28
- | **CI/CD** | GitHub Actions (multi-OS, cross-compile) |
29
-
30
- ### Production Readiness
31
- - **Maturity**: Production-ready (actively developed, popular on ProductHunt)
32
- - **Test Coverage**: Comprehensive with race detection enabled
33
- - **Documentation**: Excellent (dedicated docs site, blog, examples)
34
- - **Distribution**: Homebrew, shell installers, multi-platform binaries
35
- - **Community**: Active development, ~280K views on Reddit
36
- - **Performance**: Designed for efficiency (compact JSON, batching, debouncing)
37
-
38
- ---
39
-
40
- ## Architecture Overview
41
-
42
- ### Data Model
43
-
44
- **Two-Phase Indexing**:
45
-
46
- 1. **Vector Index** (semantic search)
47
- - **Chunk**: Code segment with vector embedding
48
- - Fields: ID, FilePath, StartLine, EndLine, Content, Vector, Hash, UpdatedAt
49
- - Chunk size: 512 tokens with 50-token overlap
50
- - Character-based chunking (handles minified files)
51
- - **Document**: File metadata tracking chunks
52
- - Fields: Path, Hash, ModTime, ChunkIDs
53
-
54
- 2. **Symbol Index** (call graph tracing)
55
- - **Symbol**: Function/method/class definitions
56
- - Fields: Name, Kind, File, Line, Signature, Receiver, Package, Exported
57
- - **Reference**: Symbol usage/call sites
58
- - Fields: SymbolName, File, Line, Context, CallerName, CallerFile
59
- - **CallEdge**: Caller → Callee relationships for graph traversal
60
-
61
- **Storage Backends** (pluggable via `VectorStore` interface):
62
- - GOB: File-based (`.grepai/index.gob`)
63
- - PostgreSQL: pgvector extension for similarity search
64
- - Qdrant: Dedicated vector database
16
+ ### Key Innovation (What's New Since Last Study)
65
17
 
66
- ### Design Patterns
18
+ 1. **RPG Semantic Graph Layer** (v0.31.0, `rpg/`): A full knowledge graph for code that maps functional areas, categories, symbols, and their relationships. Uses Jaccard similarity for feature matching, co-caller affinity edges, and hierarchical organization (Area → Category → Subcategory → File → Symbol → Chunk).
67
19
 
68
- 1. **Interface-Based Extensibility**
69
- - `Embedder` interface (embedder/embedder.go:6): Pluggable embedding providers
70
- - `VectorStore` interface (store/store.go:50): Pluggable storage backends
71
- - `SymbolExtractor` interface (trace/trace.go:113): Pluggable language parsers
20
+ 2. **Workspace Mode** (v0.31.0): Multi-project support with cross-project call graph analysis, workspace-aware file watching, and per-project symbol stores.
72
21
 
73
- 2. **Context-Aware Operations**
74
- - All I/O operations accept `context.Context` for cancellation
75
- - Example: `SaveChunks(ctx context.Context, chunks []Chunk) error`
22
+ 3. **Bubble Tea TUI** (v0.34.0): Interactive terminal UI for watch, status, trace, init, and workspace commands.
76
23
 
77
- 3. **Batch Processing with Progress Callbacks**
78
- - `BatchEmbedder` interface (embedder/embedder.go:29): Parallel batch embedding
79
- - Progress callback: `func(batchIndex, totalBatches, completedChunks, totalChunks, retrying, attempt, statusCode)`
24
+ 4. **MCP Discovery Commands** (v0.34.0): `grepai_list_workspaces` and `grepai_list_projects` tools for AI agents to discover available search scopes.
80
25
 
81
- 4. **Debounced File Watching**
82
- - Aggregates rapid file changes before triggering re-indexing
83
- - Prevents index thrashing during bulk operations (git checkout, etc.)
26
+ 5. **New Embedding Providers** (v0.32.0): Synthetic API and OpenRouter added alongside Ollama, LM Studio, and OpenAI. Factory pattern (`NewFromConfig`/`NewFromWorkspaceConfig`) centralizes provider initialization.
84
27
 
85
- 5. **Compact JSON Mode**
86
- - MCP tools support `--compact` flag to omit content (~80% token reduction)
87
- - Returns only file:line references, not full code chunks
28
+ 6. **Multi-Worktree Support** (v0.29.0-0.30.0): Git worktree detection, auto-init, and parallel watching via errgroup.
88
29
 
89
- ### Module Organization
90
-
91
- ```
92
- grepai/
93
- ├── cmd/
94
- │ └── grepai/ # CLI entry point
95
- ├── cli/ # Command implementations (init, watch, search, trace)
96
- ├── embedder/ # Embedding providers (Ollama, OpenAI, LMStudio)
97
- ├── indexer/ # Scanner, chunker, indexer orchestration
98
- ├── store/ # Vector storage backends (GOB, Postgres, Qdrant)
99
- ├── search/ # Semantic + hybrid search
100
- ├── trace/ # Symbol extraction, call graph (tree-sitter based)
101
- ├── watcher/ # File watching with debouncing
102
- ├── daemon/ # Background indexing daemon
103
- ├── config/ # YAML configuration management
104
- ├── updater/ # Self-update mechanism
105
- └── mcp/ # MCP server (5 tools)
106
- ```
107
-
108
- **Entry Point Flow**:
109
- ```
110
- cmd/grepai/main.go → cli/root.go → cli/{init,watch,search,trace}.go
111
- ```
30
+ ### Technology Stack
112
31
 
113
- **Data Flow**:
114
- ```
115
- FileSystem Scanner (respects .gitignore)
116
- Chunker (overlapping chunks)
117
- Embedder (batch API calls)
118
- VectorStore (persist with hash-based deduplication)
119
-
120
- Query Embedder (query vector)
121
- VectorStore.Search (cosine similarity)
122
- Results (sorted by score)
123
- ```
32
+ | Component | Technology |
33
+ |-----------|-----------|
34
+ | **Language** | Go 1.22+ |
35
+ | **Vector Store** | GOB (file-based), PostgreSQL/pgvector, Qdrant |
36
+ | **Embedding** | Ollama (default), LM Studio, OpenAI, Synthetic, OpenRouter |
37
+ | **File Watching** | fsnotify with debouncing |
38
+ | **CLI Framework** | cobra |
39
+ | **MCP** | mark3labs/mcp-go |
40
+ | **Call Graph** | tree-sitter (Go, TypeScript, C#, F#, more) |
41
+ | **Knowledge Graph** | RPG (custom, GOB-serialized) |
42
+ | **Testing** | Go stdlib + race detection |
43
+ | **TUI** | Bubble Tea |
124
44
 
125
- ### Comparison with ClaudeMemory
45
+ ### Production Readiness
126
46
 
127
- | Aspect | grepai | ClaudeMemory |
128
- |--------|--------|--------------|
129
- | **Language** | Go (compiled, fast) | Ruby (interpreted) |
130
- | **Primary Use** | Semantic code search | Long-term AI memory |
131
- | **Data Model** | Chunks + Symbols | Facts + Provenance |
132
- | **Storage** | Vector DB (pgvector/Qdrant/GOB) | SQLite (facts, no vectors) |
133
- | **Search** | Vector similarity + hybrid | Full-text search (FTS5) |
134
- | **Indexing** | Automatic (file watcher) | Manual (transcript ingest) |
135
- | **MCP Tools** | 5 (search, trace, status) | 8+ (recall, explain, promote) |
136
- | **Scope** | Project-local or workspace | Global + Project dual-DB |
137
- | **Truth Maintenance** | Hash-based deduplication | Supersession + conflict resolution |
138
- | **Performance** | Optimized for speed (Go, batching) | Optimized for accuracy (resolution) |
139
- | **Distribution** | Standalone CLI (Homebrew) | Ruby gem |
140
- | **Testing** | Go stdlib + race detection | RSpec with mocks |
141
- | **UI** | CLI with bubbletea (TUI) | Pure CLI |
142
- | **Call Graph** | Yes (tree-sitter AST) | No |
143
- | **File Watching** | Yes (fsnotify) | No (transcript-driven) |
144
-
145
- **Key Architectural Difference**: grepai is **proactive** (watches files, updates index automatically) while ClaudeMemory is **reactive** (responds to transcript events via hooks).
47
+ - **Maturity**: Production-ready (v0.34.0, active community)
48
+ - **Test Coverage**: Comprehensive (race detection, cross-platform)
49
+ - **Documentation**: Dedicated docs site, blog, benchmarks
50
+ - **Distribution**: Homebrew, multi-platform binaries
51
+ - **Community**: 280K+ views on Reddit, ProductHunt featured, active contributors
146
52
 
147
53
  ---
148
54
 
149
- ## Key Components Deep-Dive
150
-
151
- ### Component 1: Chunking Strategy
152
-
153
- **Purpose**: Split code into overlapping chunks optimized for embedding
154
-
155
- **Implementation** (indexer/chunker.go:47-100):
156
- ```go
157
- // Character-based chunking (not line-based) handles minified files
158
- maxChars := c.chunkSize * CharsPerToken // 512 tokens * 4 chars = 2048 chars
159
- overlapChars := c.overlap * CharsPerToken // 50 tokens * 4 chars = 200 chars
160
-
161
- for pos < len(content) {
162
- end := pos + maxChars
163
- // Try to break at newline for cleaner chunks
164
- if end < len(content) {
165
- lastNewline := strings.LastIndex(content[pos:end], "\n")
166
- if lastNewline > 0 {
167
- end = pos + lastNewline + 1
168
- }
169
- }
170
-
171
- // Calculate line numbers using pre-built index
172
- startLine := getLineNumber(lineStarts, pos)
173
- endLine := getLineNumber(lineStarts, end-1)
174
-
175
- chunks = append(chunks, ChunkInfo{
176
- ID: fmt.Sprintf("%s_%d", filePath, chunkIndex),
177
- FilePath: filePath,
178
- StartLine: startLine,
179
- EndLine: endLine,
180
- Content: chunkContent,
181
- Hash: sha256Hash, // For deduplication
182
- })
183
-
184
- pos = end - overlapChars // Overlap for context continuity
185
- }
186
- ```
187
-
188
- **Design Decisions**:
189
- - **Character-based**: Handles minified files with very long lines
190
- - **Overlap**: 50 tokens ensure context continuity across chunks
191
- - **Newline breaking**: Prefers natural boundaries when possible
192
- - **Line number mapping**: Pre-build index for O(log n) lookups
193
- - **Hash-based deduplication**: Skip re-embedding unchanged chunks
194
-
195
- **Performance**: ~4 chars per token is empirically validated for code.
196
-
197
- ---
55
+ ## Architecture Overview
198
56
 
199
- ### Component 2: File Watcher with Debouncing
57
+ ### Data Model (Updated)
200
58
 
201
- **Purpose**: Incrementally update index on file changes without thrashing
59
+ **Three-Layer Index**:
202
60
 
203
- **Implementation** (watcher/watcher.go:30-100):
204
- ```go
205
- type Watcher struct {
206
- pending map[string]FileEvent // Aggregates rapid changes
207
- pendingMu sync.Mutex // Thread-safe updates
208
- timer *time.Timer // Debounce timer
209
- }
61
+ 1. **Vector Index** (semantic search): Chunks with embeddings, pluggable backends
62
+ 2. **Symbol Index** (call graph): tree-sitter AST → Symbol/Reference/CallEdge
63
+ 3. **RPG Semantic Graph** (v0.31.0, NEW):
210
64
 
211
- func (w *Watcher) processEvents(ctx context.Context) {
212
- for {
213
- select {
214
- case event := <-w.watcher.Events:
215
- w.pendingMu.Lock()
216
-
217
- // Aggregate events (overwrites older events for same file)
218
- w.pending[relPath] = FileEvent{Type: eventType, Path: relPath}
219
-
220
- // Reset debounce timer
221
- if w.timer != nil {
222
- w.timer.Stop()
223
- }
224
- w.timer = time.AfterFunc(time.Duration(w.debounceMs)*time.Millisecond,
225
- w.flushPending)
226
-
227
- w.pendingMu.Unlock()
228
- }
229
- }
230
- }
65
+ ```
66
+ Area (functional area, top-level) [V_H]
67
+ └── Category [V_H]
68
+ └── Subcategory [V_H]
69
+ └── File (source file) [V_L]
70
+ └── Symbol (function/class) [V_L]
71
+ └── Chunk (vector ref) [V_L]
231
72
  ```
232
73
 
233
- **Design Decisions**:
234
- - **Debouncing**: Default 300ms prevents index thrashing during bulk operations
235
- - **Event aggregation**: Multiple edits to same file collapse to single update
236
- - **Recursive watching**: Watches all subdirectories automatically
237
- - **gitignore respect**: Uses `sabhiram/go-gitignore` for filtering
74
+ **Edge Types** (`rpg/model.go:27-38`):
75
+ - `feature_parent`: Hierarchy edges
76
+ - `contains`: File Symbol containment
77
+ - `invokes`: Call graph edges
78
+ - `imports`: Import relationships
79
+ - `maps_to_chunk`: Symbol → vector chunk
80
+ - `semantic_sim`: Feature similarity edges
238
81
 
239
- **Use Case**: During `git checkout`, hundreds of files change simultaneously. Debouncing waits for changes to settle before re-indexing.
82
+ ### Key Design Patterns (Updated)
240
83
 
241
- ---
84
+ 1. **RPG Encoder** (`rpg/indexer.go:29-75`): Orchestrates graph building from symbol store + vector store. Uses drift thresholds, feature extractors, and hierarchy builders.
242
85
 
243
- ### Component 3: MCP Server Design
86
+ 2. **Triple Query Engine** (`rpg/query.go:72-80`): Three operations — `SearchNode` (Jaccard similarity), `FetchNode` (detailed context with parents/children/edges), `Explore` (graph traversal with depth/direction/edge-type filters).
244
87
 
245
- **Purpose**: Expose grepai as native tool for AI agents (Claude Code, Cursor, Windsurf)
88
+ 3. **File Watcher with Debouncing** (`watcher/watcher.go:30-59`): fsnotify with configurable debounce, gitignore-aware, event deduplication via pending map.
246
89
 
247
- **Implementation** (mcp/server.go:100-168):
248
- ```go
249
- func (s *Server) registerTools() {
250
- // 1. grepai_search
251
- searchTool := mcp.NewTool("grepai_search",
252
- mcp.WithDescription("Semantic code search..."),
253
- mcp.WithString("query", mcp.Required(), mcp.Description("Natural language query")),
254
- mcp.WithNumber("limit", mcp.Description("Max results (default: 10)")),
255
- mcp.WithBoolean("compact", mcp.Description("Omit content (~80% token savings)")),
256
- mcp.WithString("workspace", mcp.Description("Cross-project search")),
257
- mcp.WithString("projects", mcp.Description("Filter by project names")),
258
- )
259
- s.mcpServer.AddTool(searchTool, s.handleSearch)
260
-
261
- // 2. grepai_trace_callers - Find who calls a function
262
- // 3. grepai_trace_callees - Find what a function calls
263
- // 4. grepai_trace_graph - Build full call graph
264
- // 5. grepai_index_status - Health check
265
- }
90
+ 4. **Embedder Factory** (`embedder/factory.go`): `NewFromConfig`/`NewFromWorkspaceConfig` centralizes provider initialization across CLI and MCP server. Supports Ollama, LM Studio, OpenAI, Synthetic, OpenRouter.
266
91
 
267
- func (s *Server) handleSearch(ctx context.Context, request mcp.CallToolRequest) (*mcp.CallToolResult, error) {
268
- query := request.RequireString("query")
269
- limit := request.GetInt("limit", 10)
270
- compact := request.GetBool("compact", false)
271
-
272
- // Initialize embedder + store
273
- emb := s.createEmbedder(cfg)
274
- st := s.createStore(ctx, cfg)
275
- defer emb.Close(); defer st.Close()
276
-
277
- // Search
278
- searcher := search.NewSearcher(st, emb, cfg.Search)
279
- results := searcher.Search(ctx, query, limit)
280
-
281
- // Return compact or full results
282
- if compact {
283
- return SearchResultCompact{FilePath, StartLine, EndLine, Score}
284
- } else {
285
- return SearchResult{FilePath, StartLine, EndLine, Score, Content}
286
- }
287
- }
288
- ```
92
+ 5. **GOB File Locking** (v0.29.0): Cross-process safety for shared index files. Prevents data corruption in multi-process scenarios.
289
93
 
290
- **Design Decisions**:
291
- - **5 focused tools**: Search, trace callers, trace callees, trace graph, status
292
- - **Compact mode**: Critical for token efficiency (~80% reduction)
293
- - **Workspace support**: Search across multiple projects with shared embedder/store
294
- - **Project filtering**: Comma-separated list filters workspace results
295
- - **Error handling**: Returns `ToolResultError` instead of throwing exceptions
296
- - **Resource cleanup**: Explicit `defer` for embedder/store cleanup
94
+ ### Comparison with ClaudeMemory
297
95
 
298
- **Integration**: Works out-of-box with Claude Code, Cursor, Windsurf via stdio transport.
96
+ | Aspect | grepai (0.34.0) | ClaudeMemory | Notes |
97
+ |--------|-----------------|--------------|-------|
98
+ | **Data Model** | Code chunks + RPG graph | Subject-predicate-object facts | Different domains |
99
+ | **Storage** | GOB/PostgreSQL/Qdrant | Dual SQLite | We're simpler |
100
+ | **Embeddings** | Ollama/OpenAI/etc (remote) | fastembed-rb (local ONNX) | We're self-contained |
101
+ | **File Watching** | fsnotify + debounce | None (hook-triggered) | They auto-index |
102
+ | **Knowledge Graph** | RPG semantic graph | Fact links (supersession/conflict) | Different purpose |
103
+ | **MCP** | mcp-go (Go) | Custom Ruby MCP server | Different languages |
104
+ | **CLI** | cobra (Go) | Custom Ruby CLI | Different stacks |
299
105
 
300
106
  ---
301
107
 
302
- ### Component 4: Call Graph via tree-sitter
108
+ ## Key Components Deep-Dive
303
109
 
304
- **Purpose**: Static analysis for "find callers" and "find callees" without executing code
110
+ ### Component 1: RPG Semantic Graph (NEW)
305
111
 
306
- **Implementation** (trace/extractor.go):
307
- - Uses tree-sitter parsers for Go, TypeScript, Python, JavaScript, C#, Java, Ruby, Rust
308
- - Extracts symbols (function definitions) and references (function calls)
309
- - Builds call graph: `Symbol → References → Callers/Callees`
112
+ **Purpose**: Map code structure into searchable knowledge graph with functional areas, categories, and semantic relationships.
310
113
 
311
- **Symbol Extraction** (trace/extractor.go:115):
312
- ```go
313
- type Symbol struct {
314
- Name string // Function/method name
315
- Kind SymbolKind // function, method, class, interface
316
- File string
317
- Line int
318
- Signature string // Full signature for disambiguation
319
- Receiver string // For methods (e.g., "MyStruct")
320
- Package string
321
- Exported bool // Public vs private
322
- Language string
323
- }
324
- ```
114
+ **Location**: `rpg/` (model.go, indexer.go, query.go, hierarchy.go, evolver.go)
325
115
 
326
- **Reference Extraction** (trace/trace.go:36):
116
+ **Implementation** (`rpg/model.go:42-80`):
327
117
  ```go
328
- type Reference struct {
329
- SymbolName string // What's being called
330
- File string // Where the call happens
331
- Line int
332
- Context string // Surrounding code for display
333
- CallerName string // Who's making the call
334
- CallerFile string // Where caller is defined
335
- CallerLine int
118
+ type Node struct {
119
+ ID string `json:"id"`
120
+ Kind NodeKind `json:"kind"` // area, category, file, symbol, chunk
121
+ Feature string `json:"feature"` // primary semantic label
122
+ Features []string `json:"features"` // atomic semantic features
123
+ Path string `json:"path,omitempty"`
124
+ SymbolName string `json:"symbol_name,omitempty"`
125
+ Summary string `json:"summary,omitempty"` // LLM-generated
336
126
  }
337
127
  ```
338
128
 
339
- **Call Graph Query** (trace/store.go):
340
- - `LookupCallers(symbolName)` → All references where symbol is called
341
- - `LookupCallees(symbolName, file)` → All references within symbol's body
342
- - `GetCallGraph(symbolName, depth)` → Multi-level traversal (BFS)
343
-
344
129
  **Design Decisions**:
345
- - **Fast mode**: tree-sitter AST parsing (no execution)
346
- - **Language-agnostic**: Interface-based extractor per language
347
- - **Context preservation**: Stores surrounding code for display
348
- - **Disambiguation**: Uses signature + file for overloaded functions
349
- - **Depth-limited**: Prevents exponential explosion in call graphs
130
+ - Hierarchical: Area Category Subcategory → File → Symbol → Chunk
131
+ - Feature-based similarity using Jaccard coefficient
132
+ - Co-caller affinity edges with occurrence threshold
133
+ - Drift threshold for incremental updates (`RPGEncoderConfig.DriftThreshold`)
134
+ - GOB serialization for persistence
350
135
 
351
- ---
136
+ ### Component 2: File Watcher
352
137
 
353
- ### Component 5: Hybrid Search
138
+ **Purpose**: Keep index fresh automatically as files change.
354
139
 
355
- **Purpose**: Combine vector similarity with keyword matching for better relevance
140
+ **Location**: `watcher/watcher.go:30-80`
356
141
 
357
- **Implementation** (search/search.go):
358
142
  ```go
359
- type HybridConfig struct {
360
- Enabled bool // Enable hybrid search
361
- K int // RRF constant (default: 60)
362
- }
363
-
364
- func (s *Searcher) Search(ctx context.Context, query string, limit int) ([]SearchResult, error) {
365
- if !s.cfg.Hybrid.Enabled {
366
- return s.vectorSearch(ctx, query, limit)
367
- }
368
-
369
- // 1. Vector search (semantic)
370
- vectorResults := s.vectorSearch(ctx, query, limit*2)
371
-
372
- // 2. Text search (keyword)
373
- textResults := s.textSearch(ctx, query, limit*2)
374
-
375
- // 3. Reciprocal Rank Fusion (RRF)
376
- fusedResults := s.fuseResults(vectorResults, textResults)
377
-
378
- return fusedResults[:limit]
379
- }
380
-
381
- func (s *Searcher) fuseResults(vec, text []Result) []Result {
382
- scores := make(map[string]float32)
383
- for rank, r := range vec {
384
- scores[r.Chunk.ID] += 1.0 / float32(rank + s.cfg.Hybrid.K)
385
- }
386
- for rank, r := range text {
387
- scores[r.Chunk.ID] += 1.0 / float32(rank + s.cfg.Hybrid.K)
388
- }
389
- // Sort by fused score
143
+ type Watcher struct {
144
+ root string
145
+ watcher *fsnotify.Watcher
146
+ ignore *indexer.IgnoreMatcher
147
+ debounceMs int
148
+ events chan FileEvent
149
+ pending map[string]FileEvent // debounce deduplication
390
150
  }
391
151
  ```
392
152
 
393
153
  **Design Decisions**:
394
- - **RRF (Reciprocal Rank Fusion)**: Proven effective for combining ranked lists
395
- - **K=60 default**: Standard RRF constant balancing vector vs text
396
- - **2x over-fetch**: Retrieve more results before fusion to avoid missing relevant items
397
- - **Configurable**: Can disable hybrid for pure semantic search
154
+ - Configurable debounce (default 300ms)
155
+ - Recursive directory watching
156
+ - Gitignore-aware filtering
157
+ - Event deduplication via pending map
158
+ - Multi-worktree support (v0.30.0)
398
159
 
399
- **Trade-off**: Slightly slower (2 searches + fusion) but significantly better relevance for queries with specific keywords.
160
+ ### Component 3: MCP Discovery Tools (NEW)
400
161
 
401
- ---
402
-
403
- ### Component 6: Pluggable Storage Backends
404
-
405
- **Purpose**: Support different deployment scenarios (local, team, cloud)
406
-
407
- **Interface** (store/store.go:50):
408
- ```go
409
- type VectorStore interface {
410
- SaveChunks(ctx context.Context, chunks []Chunk) error
411
- DeleteByFile(ctx context.Context, filePath string) error
412
- Search(ctx context.Context, queryVector []float32, limit int) ([]SearchResult, error)
413
- GetDocument(ctx context.Context, filePath string) (*Document, error)
414
- Load(ctx context.Context) error
415
- Persist(ctx context.Context) error
416
- Close() error
417
- GetStats(ctx context.Context) (*IndexStats, error)
418
- }
419
- ```
162
+ **Purpose**: Let AI agents discover available search scopes.
420
163
 
421
- **Implementations**:
422
-
423
- 1. **GOB Store** (store/gob.go): File-based, single-project
424
- - Storage: `.grepai/index.gob` (binary format)
425
- - Search: In-memory cosine similarity (brute force)
426
- - Use case: Individual developers, local projects
427
- - Pros: Zero dependencies, fast for small codebases (<10K files)
428
- - Cons: Memory usage grows with index size, no concurrent access
429
-
430
- 2. **PostgreSQL Store** (store/postgres.go): Database-backed, multi-project
431
- - Storage: Postgres with pgvector extension
432
- - Search: `<=>` operator (optimized with IVFFlat index)
433
- - Use case: Teams, shared index across projects
434
- - Pros: Concurrent access, persistence, scalable
435
- - Cons: Requires Postgres + pgvector setup
436
-
437
- 3. **Qdrant Store** (store/qdrant.go): Dedicated vector DB
438
- - Storage: Qdrant server (gRPC API)
439
- - Search: Native vector search with HNSW indexing
440
- - Use case: Large codebases (>50K files), production deployments
441
- - Pros: Fastest search, advanced filtering, scalable
442
- - Cons: Additional service to run
443
-
444
- **Configuration** (.grepai/config.yaml):
445
- ```yaml
446
- store:
447
- backend: "gob" # or "postgres" or "qdrant"
448
- postgres:
449
- dsn: "postgresql://user:pass@localhost/grepai"
450
- qdrant:
451
- endpoint: "localhost"
452
- port: 6334
453
- collection: "my-codebase"
454
- ```
164
+ **Location**: `mcp/server.go`
455
165
 
456
166
  **Design Decisions**:
457
- - **Same interface**: Swap backends without changing application code
458
- - **Project ID scoping**: Multi-project support in Postgres/Qdrant
459
- - **Hash-based deduplication**: All backends check chunk hash before embedding
460
- - **Graceful degradation**: GOB works offline, Postgres/Qdrant require connectivity
167
+ - `grepai_list_workspaces`: Returns workspace-level entries for scope selection
168
+ - `grepai_list_projects`: Returns relative paths within workspace
169
+ - Compact vs full output modes for token efficiency
461
170
 
462
171
  ---
463
172
 
@@ -465,110 +174,20 @@ store:
465
174
 
466
175
  ### What They Do Well
467
176
 
468
- #### 1. **Incremental Indexing with File Watching**
469
- - **Value**: Index stays fresh automatically without manual triggers
470
- - **Evidence**: watcher/watcher.go:61 - `fsnotify` integration with debouncing
471
- - **How**: Watches all directories recursively, respects gitignore, debounces rapid changes
472
- - **Result**: Zero maintenance overhead for users, always up-to-date search results
473
-
474
- #### 2. **Compact JSON Mode for Token Efficiency**
475
- - **Value**: ~80% reduction in AI agent token usage
476
- - **Evidence**: mcp/server.go:36 - `SearchResultCompact` omits content field
477
- - **How**: MCP tools support `--compact` flag returning only file:line:score, not full chunks
478
- - **Result**: Enables AI agents to search larger codebases without hitting context limits
479
-
480
- #### 3. **Tree-sitter for Call Graph Analysis**
481
- - **Value**: Language-agnostic static analysis without code execution
482
- - **Evidence**: trace/extractor.go - Parsers for 8+ languages
483
- - **How**: AST parsing extracts symbols + references, builds call graph
484
- - **Result**: "Find callers" feature shows impact before refactoring
485
-
486
- #### 4. **Interface-Based Extensibility**
487
- - **Value**: Easy to add new embedders or storage backends
488
- - **Evidence**: embedder/embedder.go:6, store/store.go:50
489
- - **How**: Clean interfaces with multiple implementations (Ollama, OpenAI, LMStudio | GOB, Postgres, Qdrant)
490
- - **Result**: Users choose deployment model (local, team, cloud) without code changes
491
-
492
- #### 5. **Character-Based Chunking**
493
- - **Value**: Handles minified files and long lines gracefully
494
- - **Evidence**: indexer/chunker.go:54 - `maxChars := c.chunkSize * CharsPerToken`
495
- - **How**: Splits by characters with newline preference, not by lines
496
- - **Result**: No failures on minified JS/CSS, consistent chunk sizes
497
-
498
- #### 6. **Hybrid Search (Vector + Text)**
499
- - **Value**: Better relevance for queries with specific keywords
500
- - **Evidence**: search/search.go - Reciprocal Rank Fusion (RRF)
501
- - **How**: Combines cosine similarity with full-text search using RRF
502
- - **Result**: Finds both semantically related and keyword-matched code
503
-
504
- #### 7. **Multi-Platform Distribution**
505
- - **Value**: Easy installation on Mac/Linux/Windows
506
- - **Evidence**: .goreleaser.yml, install.sh, install.ps1, Homebrew tap
507
- - **How**: GoReleaser builds cross-platform binaries, shell installers
508
- - **Result**: Friction-free adoption (`brew install grepai`)
509
-
510
- #### 8. **Workspace Mode**
511
- - **Value**: Search across multiple projects simultaneously
512
- - **Evidence**: mcp/server.go:252 - `handleWorkspaceSearch`
513
- - **How**: Shared embedder + store with project filtering
514
- - **Result**: Reuse patterns across related projects (microservices, monorepos)
515
-
516
- ---
177
+ 1. **RPG Knowledge Graph**: Rich semantic structure for code understanding
178
+ 2. **File Watching**: Automatic index updates without manual commands
179
+ 3. **Embedder Factory**: Clean abstraction for multiple embedding providers
180
+ 4. **Cross-Process Safety**: GOB file locking prevents data corruption
181
+ 5. **MCP Discovery**: Agent-friendly scope exploration tools
182
+ 6. **Multi-Worktree**: Git worktree detection and parallel watching
517
183
 
518
184
  ### What We Do Well
519
185
 
520
- #### 1. **Truth Maintenance with Conflict Resolution**
521
- - **Our Advantage**: Fact supersession and conflict detection
522
- - **Evidence**: resolve/resolver.rb - Determines equivalence, supersession, or conflicts
523
- - **Value**: Maintains consistent knowledge base even with contradictory information
524
- - **Why**: Memory requires long-term coherence; search can tolerate stale chunks
525
-
526
- #### 2. **Dual-Database Scoping (Global + Project)**
527
- - **Our Advantage**: User preferences (global) vs project-specific facts
528
- - **Evidence**: store/store_manager.rb - Manages two SQLite connections
529
- - **Value**: Facts apply at correct granularity (always vs this-project-only)
530
- - **Why**: Memory has semantic scope; search is purely location-based
531
-
532
- #### 3. **Provenance Tracking**
533
- - **Our Advantage**: Every fact links to source transcript content
534
- - **Evidence**: domain/provenance.rb - Links facts to content_items
535
- - **Value**: Users can verify where facts came from, assess confidence
536
- - **Why**: Memory requires trustworthiness; search assumes correctness
537
-
538
- #### 4. **Pluggable Distiller Interface**
539
- - **Our Advantage**: AI-powered fact extraction (future: Claude API)
540
- - **Evidence**: distill/distiller.rb - Extracts entities, facts, scope hints
541
- - **Value**: Understands context and intent, not just code structure
542
- - **Why**: Memory extracts meaning; search indexes literal content
543
-
544
- #### 5. **Hook-Based Integration**
545
- - **Our Advantage**: Seamless integration with Claude Code events
546
- - **Evidence**: hook/ - Reads stdin JSON from Claude Code hooks
547
- - **Value**: Zero-effort ingestion, automatic sweeping
548
- - **Why**: Memory is reactive to AI sessions; search is proactive (file watcher)
549
-
550
- ---
551
-
552
- ### Trade-offs
553
-
554
- | Approach | Pros | Cons |
555
- |----------|------|------|
556
- | **Their: Proactive file watching** | Always up-to-date, zero manual work | CPU/disk overhead, battery drain on laptops, irrelevant for non-code files |
557
- | **Ours: Reactive transcript ingest** | Only processes meaningful interactions | Requires hook setup, delayed until session ends |
558
- | **Their: Vector embeddings** | Semantic understanding, fuzzy matching | Requires embedding provider, slower than text search, cost (if OpenAI) |
559
- | **Ours: FTS5 full-text search** | Fast, zero dependencies, no cost | Exact/substring matching only, no semantic understanding |
560
- | **Their: Go (compiled)** | Fast startup, low memory, easy distribution | Harder to extend (compile step), less metaprogramming |
561
- | **Ours: Ruby (interpreted)** | Easy to extend, rich metaprogramming | Slower, requires Ruby runtime, harder to distribute |
562
- | **Their: Chunk-based storage** | Optimized for retrieval granularity | Redundancy across chunks, harder to track changes |
563
- | **Ours: Fact-based storage** | Deduplicated, structured, queryable | Requires distillation step, more complex schema |
564
- | **Their: Interface-based backends** | Pluggable storage (GOB/Postgres/Qdrant) | Complexity in maintaining multiple implementations |
565
- | **Ours: SQLite-only** | Simple, zero config, portable | Limited scalability, no built-in vector search |
566
- | **Their: Tree-sitter call graph** | Static analysis, language-agnostic | Setup cost (parsers), limited to supported languages |
567
- | **Ours: No call graph** | Simpler architecture | Can't answer "who calls this?" questions |
568
- | **Their: Compact JSON mode** | Token-efficient for AI agents | Requires Read tool for full content (two-step) |
569
- | **Ours: Full content in recall** | One-step retrieval | Higher token usage per query |
570
-
571
- **Key Trade-off**: grepai prioritizes **search speed and semantic understanding** at the cost of setup complexity. ClaudeMemory prioritizes **truth maintenance and provenance** at the cost of search sophistication.
186
+ 1. **Self-Contained Embeddings**: fastembed-rb needs no external service
187
+ 2. **Dual-Database Scope**: Project/global separation is cleaner
188
+ 3. **Truth Maintenance**: Temporal validity and conflict resolution
189
+ 4. **Rich MCP Tools**: 18 specialized tools vs their general search
190
+ 5. **Simpler Architecture**: No background processes or external databases
572
191
 
573
192
  ---
574
193
 
@@ -577,357 +196,62 @@ store:
577
196
  ### High Priority ⭐
578
197
 
579
198
  #### 1. Incremental Indexing with File Watching
580
- - **Value**: ClaudeMemory index could stay fresh automatically during coding sessions
581
- - **Evidence**: watcher/watcher.go:44 - `fsnotify` with debouncing, gitignore respect
582
- - **Implementation**:
583
- 1. Add `fsnotify` to Gemfile
584
- 2. Create `ClaudeMemory::Watcher` class wrapping `Listen` gem (Ruby equivalent of fsnotify)
585
- 3. Watch `.claude/projects/*/transcripts/*.jsonl` for new lines (tail-like behavior)
586
- 4. Debounce events (default 500ms to avoid thrashing during bulk writes)
587
- 5. Trigger `IngestCommand` automatically when new transcript data appears
588
- 6. Optional: Watch `.claude/rules/` for manual fact additions
589
- - **Effort**: 2-3 days (watcher class, integration with ingest, testing)
590
- - **Trade-off**: Adds background process (memory overhead ~10MB), may complicate testing
591
- - **Recommendation**: **ADOPT** - Eliminates manual `claude-memory ingest` calls, huge UX win
592
-
593
- #### 2. Compact Response Format for MCP Tools
594
- - **Value**: Reduce token usage by ~60% in MCP responses by omitting verbose content
595
- - **Evidence**: mcp/server.go:219 - `SearchResultCompact` omits content field, returns only metadata
596
- - **Implementation**:
597
- 1. Add `compact` boolean parameter to `memory.recall` and `memory.search_*` tools
598
- 2. Create `CompactFormatter` in `MCP::ResponseFormatter`:
599
- ```ruby
600
- def format_fact_compact(fact)
601
- {
602
- id: fact.id,
603
- subject: fact.subject,
604
- predicate: fact.predicate,
605
- object: fact.object,
606
- scope: fact.scope,
607
- confidence: fact.confidence
608
- }
609
- # Omit: provenance, supersession_chain, context excerpts
610
- end
611
- ```
612
- 3. Default to `compact: true` for all MCP tools (user can override with `compact: false`)
613
- 4. Update tool descriptions to explain compact mode
614
- - **Effort**: 4-6 hours (add parameter, update formatters, tests)
615
- - **Trade-off**: User needs follow-up `memory.explain <fact_id>` for full context (two-step interaction)
616
- - **Recommendation**: **ADOPT** - Critical for scaling to large fact databases (1000+ facts)
617
-
618
- #### 3. Hybrid Search (Vector + Text)
619
- - **Value**: Better relevance when users search for specific terms (e.g., "uses_database") while preserving semantic matching
620
- - **Evidence**: search/search.go - Reciprocal Rank Fusion (RRF) with K=60
621
- - **Implementation**:
622
- 1. Add `sqlite-vec` extension to Gemfile for vector similarity in SQLite
623
- 2. Add `embeddings` column to `facts` table (BLOB storing float32 array)
624
- 3. Create `ClaudeMemory::Embedder` interface:
625
- - Implementation: Call Anthropic API for embeddings (free with Claude usage)
626
- - Cache embeddings per fact (regenerate only when fact changes)
627
- 4. Implement RRF in `Recall#query`:
628
- ```ruby
629
- vector_results = vector_search(query, limit * 2) # Cosine similarity
630
- text_results = fts_search(query, limit * 2) # Existing FTS5
631
- fuse_with_rrf(vector_results, text_results, k: 60)
632
- ```
633
- 5. Make hybrid search optional via `.grepai/config.yaml`:
634
- ```yaml
635
- search:
636
- hybrid:
637
- enabled: true
638
- k: 60
639
- ```
640
- - **Effort**: 5-7 days (embedder setup, schema migration, RRF implementation, testing)
641
- - **Trade-off**: Requires API calls for embedding (cost ~$0.00001/fact), slower queries (2x search + fusion)
642
- - **Recommendation**: **CONSIDER** - High value but significant implementation effort. Start with FTS5, add vectors later if search quality issues arise.
643
-
644
- #### 4. Call Graph for Fact Dependencies
645
- - **Value**: Show which facts depend on others (supersession chains, conflict relationships) visually
646
- - **Evidence**: trace/trace.go:95 - `CallGraph` struct with nodes and edges
647
- - **Implementation**:
648
- 1. Create `memory.fact_graph <fact_id> --depth 2` MCP tool
649
- 2. Query `fact_links` table to build graph:
650
- - Nodes: Facts (subject/predicate/object)
651
- - Edges: Supersedes, Conflicts, Supports
652
- 3. Return JSON matching grepai's format:
653
- ```json
654
- {
655
- "root": "fact_123",
656
- "nodes": {"fact_123": {...}, "fact_456": {...}},
657
- "edges": [
658
- {"from": "fact_123", "to": "fact_456", "type": "supersedes"},
659
- {"from": "fact_123", "to": "fact_789", "type": "conflicts"}
660
- ],
661
- "depth": 2
662
- }
663
- ```
664
- 4. Depth-limited BFS traversal (avoid exponential explosion)
665
- - **Effort**: 2-3 days (graph builder, MCP tool, tests)
666
- - **Trade-off**: Adds complexity for a feature used mainly for debugging/exploration
667
- - **Recommendation**: **ADOPT** - Invaluable for understanding why facts were superseded or conflicted
668
-
669
- #### 5. Multi-Project Workspace Mode
670
- - **Value**: Search facts across multiple projects simultaneously (e.g., all Ruby projects)
671
- - **Evidence**: mcp/server.go:252 - `handleWorkspaceSearch` with project filtering
672
- - **Implementation**:
673
- 1. Extend `.claude/settings.json` with workspace config:
674
- ```json
675
- {
676
- "workspaces": {
677
- "ruby-projects": {
678
- "projects": [
679
- "/Users/me/project1",
680
- "/Users/me/project2"
681
- ],
682
- "scope": "project" // Only search project-scoped facts
683
- }
684
- }
685
- }
686
- ```
687
- 2. Add `workspace` parameter to `memory.recall`:
688
- ```ruby
689
- memory.recall(query: "authentication", workspace: "ruby-projects")
690
- ```
691
- 3. StoreManager opens all project databases, merges results
692
- 4. Filter by `project_path` matching workspace projects
693
- - **Effort**: 3-4 days (workspace config, multi-DB queries, result merging, tests)
694
- - **Trade-off**: Complexity in managing multiple DB connections, potential for confusion (which project is this fact from?)
695
- - **Recommendation**: **DEFER** - Nice-to-have but low ROI for current use case (most users work on one project at a time)
696
-
697
- ---
199
+ - **Value**: Eliminates manual `claude-memory ingest` calls
200
+ - **Evidence**: `watcher/watcher.go:30-59` fsnotify with debouncing, gitignore respect
201
+ - **Implementation**: Add `listen` gem (Ruby fsnotify equivalent), watch `.claude/projects/*/transcripts/*.jsonl`, debounce 500ms, trigger IngestCommand
202
+ - **Effort**: 2-3 days
203
+ - **Trade-off**: Background process ~10MB memory overhead
204
+ - **Recommendation**: **CONSIDER** Already in improvements.md (#3), reinforced by this study
698
205
 
699
206
  ### Medium Priority
700
207
 
701
- #### 6. Self-Update Mechanism
702
- - **Value**: Users get bug fixes and new features automatically without reinstalling gem
703
- - **Evidence**: updater/updater.go - Checks GitHub releases, downloads binary, replaces self
704
- - **Implementation**:
705
- 1. Add `claude-memory update` command
706
- 2. Check GitHub releases API for `anthropics/claude-memory` (or RubyGems API)
707
- 3. Compare current version (`ClaudeMemory::VERSION`) with latest
708
- 4. Download gem, extract, replace files in-place
709
- 5. Display changelog and prompt to confirm
710
- - **Effort**: 2-3 days (update logic, safe file replacement, testing)
711
- - **Trade-off**: Requires write permissions to gem directory (may fail in system-wide installs)
712
- - **Recommendation**: **CONSIDER** - Nice UX, but users can `gem update` manually
713
-
714
- #### 7. Configuration via YAML
715
- - **Value**: Easier configuration than editing JSON or ENV vars
716
- - **Evidence**: config/config.go - `.grepai/config.yaml` with typed structs
717
- - **Implementation**:
718
- 1. Add `.claude/memory.yaml` support:
719
- ```yaml
720
- ingest:
721
- auto_sweep: true
722
- sweep_budget_seconds: 5
723
- publish:
724
- mode: shared
725
- include_provenance: false
726
- recall:
727
- default_limit: 10
728
- scope: all
729
- ```
730
- 2. Load YAML in `Configuration` class, merge with ENV vars (ENV takes precedence)
731
- 3. Validate config on load (raise clear errors for invalid values)
732
- - **Effort**: 1-2 days (YAML parsing, validation, tests)
733
- - **Trade-off**: Another configuration method to document and support
734
- - **Recommendation**: **CONSIDER** - Better UX, but JSON in `.claude/settings.json` works fine for now
735
-
736
- #### 8. Batch MCP Tool Operations
737
- - **Value**: Reduce round-trips when agent needs to recall multiple facts
738
- - **Evidence**: embedder/embedder.go:10 - `EmbedBatch` for parallel processing
739
- - **Implementation**:
740
- 1. Add `memory.recall_batch` tool accepting array of queries:
741
- ```json
742
- {
743
- "queries": [
744
- {"query": "authentication", "limit": 5},
745
- {"query": "database schema", "limit": 3}
746
- ]
747
- }
748
- ```
749
- 2. Execute queries in parallel (use `Concurrent::Future` from concurrent-ruby gem)
750
- 3. Return merged results with query labels:
751
- ```json
752
- [
753
- {"query": "authentication", "results": [...]},
754
- {"query": "database schema", "results": [...]}
755
- ]
756
- ```
757
- - **Effort**: 1-2 days (batch tool, parallel execution, tests)
758
- - **Trade-off**: More complex error handling (what if one query fails?)
759
- - **Recommendation**: **CONSIDER** - Useful if agents frequently need multiple searches, but current single-query API is simpler
760
-
761
- ---
762
-
763
- ### Low Priority
764
-
765
- #### 9. TUI (Terminal UI) for Interactive Exploration
766
- - **Value**: Visual interface for browsing facts, conflicts, provenance
767
- - **Evidence**: cli/ uses `charmbracelet/bubbletea` for rich TUI
768
- - **Implementation**:
769
- 1. Add `claude-memory explore` command launching TUI
770
- 2. Use `tty-prompt` gem for interactive menus:
771
- - Browse facts by predicate
772
- - Explore supersession chains
773
- - View conflict details
774
- - Search facts interactively
775
- 3. Display provenance excerpts inline
776
- - **Effort**: 3-5 days (TUI design, navigation, rendering)
777
- - **Trade-off**: Adds dependency and complexity for limited use case (most interaction via MCP tools)
778
- - **Recommendation**: **DEFER** - Nice for power users, but MCP tools + `memory.explain` cover 90% of needs
779
-
780
- #### 10. Prometheus Metrics Endpoint
781
- - **Value**: Monitor memory system health (fact count, sweep duration, query latency)
782
- - **Evidence**: grepai exposes metrics via `grepai_index_status` tool
783
- - **Implementation**:
784
- 1. Add `claude-memory serve-metrics` command (HTTP server on `:9090/metrics`)
785
- 2. Expose Prometheus-format metrics:
786
- - `claude_memory_facts_total{scope="global|project"}`
787
- - `claude_memory_sweep_duration_seconds`
788
- - `claude_memory_recall_latency_seconds`
789
- 3. Optional: Export to StatsD, DataDog, etc.
790
- - **Effort**: 2-3 days (metrics collection, HTTP server, testing)
791
- - **Trade-off**: Requires running separate metrics server, overkill for most users
792
- - **Recommendation**: **DEFER** - Only needed for production deployments with SLAs
793
-
794
- ---
208
+ #### 2. MCP Discovery Tools
209
+ - **Value**: Let Claude discover available search scopes before querying
210
+ - **Evidence**: `mcp/server.go` `grepai_list_workspaces`, `grepai_list_projects`
211
+ - **Implementation**: Add `memory.list_projects` tool showing databases with fact counts
212
+ - **Effort**: 1 day
213
+ - **Trade-off**: Minimal useful metadata
214
+ - **Recommendation**: **CONSIDER**
215
+
216
+ #### 3. Embedder Factory Pattern
217
+ - **Value**: Clean abstraction for swapping embedding providers
218
+ - **Evidence**: `embedder/factory.go` `NewFromConfig` centralizes initialization
219
+ - **Implementation**: Our fastembed-rb is already self-contained, but a factory would help if we add sqlite-vec and need to support different models
220
+ - **Effort**: 1-2 days
221
+ - **Trade-off**: Premature unless we add multiple embedding providers
222
+ - **Recommendation**: **DEFER**
795
223
 
796
224
  ### Features to Avoid
797
225
 
798
- #### 1. Cloud-Based Embedding Service
799
- - **Why Avoid**: grepai's privacy-first approach (Ollama local embeddings) is a key selling point, but ClaudeMemory's use case (AI memory, not code search) doesn't require embeddings yet
800
- - **Our Alternative**: Stick with FTS5 full-text search until we need semantic matching. If we add embeddings, use Anthropic API (already authenticated) rather than separate embedding service
801
- - **Reasoning**: Adding embeddings adds cost, latency, and complexity. FTS5 is sufficient for fact recall (structured data) vs code search (unstructured data)
802
-
803
- #### 2. Multiple Storage Backends (Postgres, Qdrant)
804
- - **Why Avoid**: Increases maintenance burden (test matrix, docs, support) for unclear benefit
805
- - **Our Alternative**: SQLite is perfect for local storage, portable, and sufficient for fact databases (<100K facts). If we need remote storage, use libSQL (SQLite over HTTP) or Turso
806
- - **Reasoning**: grepai needs backends for team collaboration (shared index). ClaudeMemory is single-user by design (global + project scoping)
807
-
808
- #### 3. Daemon Mode with Background Indexing
809
- - **Why Avoid**: Adds complexity (process management, logging, crash recovery) and battery drain
810
- - **Our Alternative**: Hook-based reactive ingestion (current approach) is elegant and efficient. Only process transcripts when meaningful work happens (AI sessions)
811
- - **Reasoning**: grepai needs daemon for real-time file watching. ClaudeMemory doesn't need to watch files (transcripts are append-only, hooks trigger ingestion)
812
-
813
- ---
814
-
815
- ## Implementation Recommendations
816
-
817
- ### Phase 1: Quick Wins (1-2 weeks)
818
-
819
- **Goal**: Low-effort, high-value improvements
820
-
821
- - [ ] **Compact MCP responses** (4-6 hours)
822
- - Add `compact: true` parameter to all recall tools
823
- - Omit provenance and context excerpts by default
824
- - Test token reduction with realistic queries
825
- - Success criteria: 60% token reduction in MCP responses
826
-
827
- - [ ] **Fact dependency graph** (2-3 days)
828
- - Implement `memory.fact_graph <fact_id> --depth 2` tool
829
- - BFS traversal of fact_links table
830
- - Return JSON with nodes and edges
831
- - Success criteria: Visualize supersession chains and conflicts
832
-
833
- ### Phase 2: Incremental Indexing (2-3 weeks)
834
-
835
- **Goal**: Auto-update index during coding sessions
836
-
837
- - [ ] **File watcher for transcripts** (2-3 days)
838
- - Add `Listen` gem (Ruby equivalent of fsnotify)
839
- - Watch `.claude/projects/*/transcripts/*.jsonl` for changes
840
- - Debounce rapid changes (500ms)
841
- - Trigger `IngestCommand` automatically
842
- - Success criteria: Index updates within 1 second of transcript write
843
-
844
- - [ ] **Optional daemon mode** (2-3 days)
845
- - Add `claude-memory watch` command (background process)
846
- - Fork and daemonize, write PID file
847
- - Graceful shutdown on SIGTERM
848
- - Success criteria: Run in background, no manual ingest needed
849
-
850
- - [ ] **Integration with existing hooks** (1 day)
851
- - Keep existing hooks as fallback (if watcher not running)
852
- - Add `auto_watch: true` setting to enable watcher
853
- - Success criteria: Works with or without daemon
854
-
855
- ### Phase 3: Hybrid Search (4-6 weeks)
856
-
857
- **Goal**: Add semantic search for better relevance
858
-
859
- - [ ] **Embedder interface** (1 week)
860
- - Create `ClaudeMemory::Embedder` module
861
- - Implement Anthropic API embedder (call `/v1/embeddings`)
862
- - Cache embeddings in `fact_embeddings` table
863
- - Success criteria: Generate embeddings for facts
864
-
865
- - [ ] **Vector storage** (1 week)
866
- - Add `sqlite-vec` extension to dependencies
867
- - Migrate schema: add `embedding` BLOB column to `facts`
868
- - Implement cosine similarity search
869
- - Success criteria: Vector search returns similar facts
870
-
871
- - [ ] **RRF implementation** (1 week)
872
- - Implement Reciprocal Rank Fusion in `Recall#query`
873
- - Combine FTS5 results with vector results
874
- - Make hybrid search optional via config
875
- - Success criteria: Better relevance than FTS5 alone
876
-
877
- - [ ] **Performance tuning** (1 week)
878
- - Benchmark vector search vs FTS5
879
- - Add caching for frequently-queried embeddings
880
- - Optimize batch embedding (parallel API calls)
881
- - Success criteria: Hybrid search <500ms for typical queries
882
-
883
- ---
884
-
885
- ## Architecture Decisions
886
-
887
- ### What to Preserve
888
-
889
- - **SQLite-only storage**: Simple, portable, fast enough for fact databases
890
- - **Hook-based integration**: Elegant reactive model (no polling, no daemons unless opted-in)
891
- - **Fact-based data model**: Structured triples with provenance vs unstructured chunks
892
- - **Truth maintenance**: Supersession and conflict resolution (grepai has no equivalent)
893
- - **Dual-database scoping**: Global vs project facts (grepai has only project-local or workspace)
894
-
895
- ### What to Adopt
896
-
897
- - **File watcher for incremental indexing**: Huge UX win, eliminates manual `ingest` calls
898
- - **Compact JSON for MCP tools**: Critical for scaling to large fact databases
899
- - **Fact dependency graph visualization**: Invaluable for debugging supersession/conflicts
900
- - **Interface-based extensibility**: If we add embeddings, use pluggable `Embedder` interface
901
- - **Hybrid search (RRF)**: Better relevance, proven technique
902
-
903
- ### What to Reject
904
-
905
- - **Cloud-based embeddings**: Privacy concerns, cost, latency (use Anthropic API if needed)
906
- - **Multiple storage backends**: Adds complexity without clear benefit for single-user tool
907
- - **Daemon mode by default**: Optional is fine, but hooks are simpler and more efficient
908
- - **Tree-sitter call graphs**: Out of scope for memory system (we track fact dependencies, not code dependencies)
909
- - **Workspace mode**: Defer until multi-project use case is validated
226
+ - **RPG Knowledge Graph**: Wrong domain — code graph vs fact graph
227
+ - **Ollama/External Embeddings**: We use local ONNX embeddings (fastembed-rb)
228
+ - **PostgreSQL/Qdrant**: Over-engineering for our SQLite-based architecture
229
+ - **tree-sitter AST**: Not relevant to fact/memory retrieval
230
+ - **Bubble Tea TUI**: CLI output is sufficient
231
+ - **GOB Serialization**: We use SQLite which has built-in ACID
910
232
 
911
233
  ---
912
234
 
913
235
  ## Key Takeaways
914
236
 
915
- 1. **grepai excels at real-time semantic search** via file watching + vector embeddings. We should adopt file watching for transcript indexing but defer vector embeddings until FTS5 proves insufficient.
916
-
917
- 2. **Compact JSON mode is critical for token efficiency**. We should implement this immediately in all MCP tools (60% token reduction with minimal effort).
918
-
919
- 3. **Fact dependency graphs** (supersession chains, conflicts) are analogous to grepai's call graphs. Implementing `memory.fact_graph` would be highly valuable for understanding fact relationships.
920
-
921
- 4. **Interface-based extensibility** is a proven pattern. If we add embeddings, follow grepai's `Embedder` interface design for pluggability.
237
+ ### Changes Since Last Analysis (2026-01-29)
238
+ - RPG Semantic Graph Layer added (v0.31.0) — major new feature
239
+ - Workspace mode for multi-project support
240
+ - Bubble Tea TUI for interactive commands
241
+ - MCP discovery commands (list_workspaces, list_projects)
242
+ - Synthetic API + OpenRouter embedding providers
243
+ - Multi-worktree detection and parallel watching
244
+ - .grepaiignore support
245
+ - F# language support for trace
246
+
247
+ ### Main Learnings
248
+ 1. RPG graph is interesting but wrong domain for us
249
+ 2. File watching with debouncing remains the best auto-indexing approach
250
+ 3. MCP discovery tools are a nice UX pattern for multi-scope systems
251
+ 4. Factory patterns for embedding providers are smart if you support multiple backends
922
252
 
923
- 5. **Hybrid search (vector + text)** is more sophisticated than our FTS5-only approach, but adds significant complexity. Defer until search quality becomes a bottleneck.
924
-
925
- 6. **Go's performance and distribution advantages** are appealing, but Ruby is fine for our use case (not latency-critical, single-user tool). Don't rewrite unless clear performance issues arise.
926
-
927
- 7. **Recommended adoption order**:
928
- - **Immediate**: Compact JSON, fact dependency graph
929
- - **Short-term**: File watcher for transcripts
930
- - **Medium-term**: Hybrid search (if needed)
931
- - **Defer**: Workspace mode, cloud backends, TUI
253
+ ---
932
254
 
933
- **Expected impact**: Adopting file watching + compact JSON would eliminate manual `ingest` calls and reduce token usage by 60%, dramatically improving UX without major architectural changes.
255
+ *Analysis completed: 2026-03-02*
256
+ *Analyst: Claude Code*
257
+ *Review Status: Draft*