claude_memory 0.5.1 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.claude/CLAUDE.md +1 -1
- data/.claude/memory.sqlite3 +0 -0
- data/.claude/memory.sqlite3-shm +0 -0
- data/.claude/memory.sqlite3-wal +0 -0
- data/.claude/rules/claude_memory.generated.md +1 -1
- data/.claude/settings.json +5 -0
- data/.claude/settings.local.json +19 -1
- data/.claude-plugin/marketplace.json +5 -2
- data/.claude-plugin/plugin.json +16 -3
- data/.gitattributes +1 -0
- data/CHANGELOG.md +91 -0
- data/CLAUDE.md +28 -14
- data/README.md +6 -2
- data/Rakefile +22 -0
- data/db/migrations/011_add_tool_call_summaries.rb +18 -0
- data/db/migrations/012_add_vec_indexing_support.rb +19 -0
- data/docs/improvements.md +225 -61
- data/docs/influence/claude-mem.md +253 -0
- data/docs/influence/claude-supermemory.md +158 -430
- data/docs/influence/episodic-memory.md +217 -0
- data/docs/influence/grepai.md +163 -839
- data/docs/influence/kbs.md +437 -0
- data/docs/influence/qmd.md +210 -481
- data/docs/quality_review.md +344 -56
- data/hooks/hooks.json +19 -15
- data/lefthook.yml +4 -0
- data/lib/claude_memory/commands/checks/database_check.rb +7 -0
- data/lib/claude_memory/commands/checks/vec_check.rb +73 -0
- data/lib/claude_memory/commands/compact_command.rb +104 -0
- data/lib/claude_memory/commands/doctor_command.rb +1 -0
- data/lib/claude_memory/commands/export_command.rb +116 -0
- data/lib/claude_memory/commands/git_lfs_command.rb +117 -0
- data/lib/claude_memory/commands/help_command.rb +2 -0
- data/lib/claude_memory/commands/hook_command.rb +110 -9
- data/lib/claude_memory/commands/index_command.rb +63 -8
- data/lib/claude_memory/commands/initializers/global_initializer.rb +26 -7
- data/lib/claude_memory/commands/initializers/project_initializer.rb +35 -12
- data/lib/claude_memory/commands/registry.rb +4 -1
- data/lib/claude_memory/commands/serve_mcp_command.rb +10 -1
- data/lib/claude_memory/commands/stats_command.rb +12 -1
- data/lib/claude_memory/configuration.rb +40 -1
- data/lib/claude_memory/core/snippet_extractor.rb +21 -19
- data/lib/claude_memory/hook/context_injector.rb +75 -0
- data/lib/claude_memory/hook/error_classifier.rb +67 -0
- data/lib/claude_memory/hook/handler.rb +21 -1
- data/lib/claude_memory/index/lexical_fts.rb +88 -16
- data/lib/claude_memory/index/vector_index.rb +171 -0
- data/lib/claude_memory/infrastructure/schema_validator.rb +5 -1
- data/lib/claude_memory/ingest/ingester.rb +26 -1
- data/lib/claude_memory/ingest/observation_compressor.rb +177 -0
- data/lib/claude_memory/mcp/instructions_builder.rb +76 -0
- data/lib/claude_memory/mcp/server.rb +3 -1
- data/lib/claude_memory/mcp/tool_definitions.rb +65 -27
- data/lib/claude_memory/mcp/tools.rb +137 -2
- data/lib/claude_memory/publish.rb +28 -27
- data/lib/claude_memory/recall/dual_query_template.rb +1 -12
- data/lib/claude_memory/recall.rb +71 -17
- data/lib/claude_memory/resolve/resolver.rb +22 -18
- data/lib/claude_memory/store/sqlite_store.rb +17 -1
- data/lib/claude_memory/store/store_manager.rb +19 -24
- data/lib/claude_memory/sweep/sweeper.rb +41 -2
- data/lib/claude_memory/version.rb +1 -1
- data/lib/claude_memory.rb +15 -0
- data/scripts/hook-runner.sh +14 -0
- data/scripts/serve-mcp.sh +14 -0
- data/skills/setup-memory/SKILL.md +6 -0
- metadata +36 -2
data/docs/influence/grepai.md
CHANGED
|
@@ -1,463 +1,172 @@
|
|
|
1
|
-
# grepai Analysis
|
|
1
|
+
# grepai Analysis (Updated)
|
|
2
2
|
|
|
3
|
-
*Analysis Date: 2026-
|
|
3
|
+
*Analysis Date: 2026-03-02*
|
|
4
|
+
*Previous Analysis: 2026-01-29*
|
|
4
5
|
*Repository: https://github.com/yoanbernabeu/grepai*
|
|
5
|
-
*Version
|
|
6
|
+
*Version: 0.34.0 (commit 1c7aba9)*
|
|
6
7
|
|
|
7
8
|
---
|
|
8
9
|
|
|
9
10
|
## Executive Summary
|
|
10
11
|
|
|
11
12
|
### Project Purpose
|
|
12
|
-
grepai is a privacy-first CLI tool for semantic code search using vector embeddings, enabling natural language queries that find relevant code based on intent rather than exact text matches.
|
|
13
13
|
|
|
14
|
-
|
|
15
|
-
**Semantic search with zero cloud dependency** - Combines local vector embeddings (via Ollama) with file-watching for real-time index updates, drastically reducing AI agent token usage by ~80% through intelligent context retrieval instead of full codebase scans.
|
|
14
|
+
grepai is a privacy-first CLI for semantic code search using vector embeddings. It enables natural language queries that find relevant code by meaning, not just text — reducing AI agent input tokens by providing targeted context.
|
|
16
15
|
|
|
17
|
-
###
|
|
18
|
-
| Component | Technology |
|
|
19
|
-
|-----------|-----------|
|
|
20
|
-
| **Language** | Go 1.22+ |
|
|
21
|
-
| **Vector Store** | PostgreSQL with pgvector, Qdrant, or GOB (file-based) |
|
|
22
|
-
| **Embedding** | Ollama (local), OpenAI, LM Studio |
|
|
23
|
-
| **File Watching** | fsnotify |
|
|
24
|
-
| **CLI Framework** | cobra |
|
|
25
|
-
| **MCP Integration** | mark3labs/mcp-go |
|
|
26
|
-
| **Call Graph** | tree-sitter for AST parsing |
|
|
27
|
-
| **Testing** | Go stdlib testing with race detection |
|
|
28
|
-
| **CI/CD** | GitHub Actions (multi-OS, cross-compile) |
|
|
29
|
-
|
|
30
|
-
### Production Readiness
|
|
31
|
-
- **Maturity**: Production-ready (actively developed, popular on ProductHunt)
|
|
32
|
-
- **Test Coverage**: Comprehensive with race detection enabled
|
|
33
|
-
- **Documentation**: Excellent (dedicated docs site, blog, examples)
|
|
34
|
-
- **Distribution**: Homebrew, shell installers, multi-platform binaries
|
|
35
|
-
- **Community**: Active development, ~280K views on Reddit
|
|
36
|
-
- **Performance**: Designed for efficiency (compact JSON, batching, debouncing)
|
|
37
|
-
|
|
38
|
-
---
|
|
39
|
-
|
|
40
|
-
## Architecture Overview
|
|
41
|
-
|
|
42
|
-
### Data Model
|
|
43
|
-
|
|
44
|
-
**Two-Phase Indexing**:
|
|
45
|
-
|
|
46
|
-
1. **Vector Index** (semantic search)
|
|
47
|
-
- **Chunk**: Code segment with vector embedding
|
|
48
|
-
- Fields: ID, FilePath, StartLine, EndLine, Content, Vector, Hash, UpdatedAt
|
|
49
|
-
- Chunk size: 512 tokens with 50-token overlap
|
|
50
|
-
- Character-based chunking (handles minified files)
|
|
51
|
-
- **Document**: File metadata tracking chunks
|
|
52
|
-
- Fields: Path, Hash, ModTime, ChunkIDs
|
|
53
|
-
|
|
54
|
-
2. **Symbol Index** (call graph tracing)
|
|
55
|
-
- **Symbol**: Function/method/class definitions
|
|
56
|
-
- Fields: Name, Kind, File, Line, Signature, Receiver, Package, Exported
|
|
57
|
-
- **Reference**: Symbol usage/call sites
|
|
58
|
-
- Fields: SymbolName, File, Line, Context, CallerName, CallerFile
|
|
59
|
-
- **CallEdge**: Caller → Callee relationships for graph traversal
|
|
60
|
-
|
|
61
|
-
**Storage Backends** (pluggable via `VectorStore` interface):
|
|
62
|
-
- GOB: File-based (`.grepai/index.gob`)
|
|
63
|
-
- PostgreSQL: pgvector extension for similarity search
|
|
64
|
-
- Qdrant: Dedicated vector database
|
|
16
|
+
### Key Innovation (What's New Since Last Study)
|
|
65
17
|
|
|
66
|
-
|
|
18
|
+
1. **RPG Semantic Graph Layer** (v0.31.0, `rpg/`): A full knowledge graph for code that maps functional areas, categories, symbols, and their relationships. Uses Jaccard similarity for feature matching, co-caller affinity edges, and hierarchical organization (Area → Category → Subcategory → File → Symbol → Chunk).
|
|
67
19
|
|
|
68
|
-
|
|
69
|
-
- `Embedder` interface (embedder/embedder.go:6): Pluggable embedding providers
|
|
70
|
-
- `VectorStore` interface (store/store.go:50): Pluggable storage backends
|
|
71
|
-
- `SymbolExtractor` interface (trace/trace.go:113): Pluggable language parsers
|
|
20
|
+
2. **Workspace Mode** (v0.31.0): Multi-project support with cross-project call graph analysis, workspace-aware file watching, and per-project symbol stores.
|
|
72
21
|
|
|
73
|
-
|
|
74
|
-
- All I/O operations accept `context.Context` for cancellation
|
|
75
|
-
- Example: `SaveChunks(ctx context.Context, chunks []Chunk) error`
|
|
22
|
+
3. **Bubble Tea TUI** (v0.34.0): Interactive terminal UI for watch, status, trace, init, and workspace commands.
|
|
76
23
|
|
|
77
|
-
|
|
78
|
-
- `BatchEmbedder` interface (embedder/embedder.go:29): Parallel batch embedding
|
|
79
|
-
- Progress callback: `func(batchIndex, totalBatches, completedChunks, totalChunks, retrying, attempt, statusCode)`
|
|
24
|
+
4. **MCP Discovery Commands** (v0.34.0): `grepai_list_workspaces` and `grepai_list_projects` tools for AI agents to discover available search scopes.
|
|
80
25
|
|
|
81
|
-
|
|
82
|
-
- Aggregates rapid file changes before triggering re-indexing
|
|
83
|
-
- Prevents index thrashing during bulk operations (git checkout, etc.)
|
|
26
|
+
5. **New Embedding Providers** (v0.32.0): Synthetic API and OpenRouter added alongside Ollama, LM Studio, and OpenAI. Factory pattern (`NewFromConfig`/`NewFromWorkspaceConfig`) centralizes provider initialization.
|
|
84
27
|
|
|
85
|
-
|
|
86
|
-
- MCP tools support `--compact` flag to omit content (~80% token reduction)
|
|
87
|
-
- Returns only file:line references, not full code chunks
|
|
28
|
+
6. **Multi-Worktree Support** (v0.29.0-0.30.0): Git worktree detection, auto-init, and parallel watching via errgroup.
|
|
88
29
|
|
|
89
|
-
###
|
|
90
|
-
|
|
91
|
-
```
|
|
92
|
-
grepai/
|
|
93
|
-
├── cmd/
|
|
94
|
-
│ └── grepai/ # CLI entry point
|
|
95
|
-
├── cli/ # Command implementations (init, watch, search, trace)
|
|
96
|
-
├── embedder/ # Embedding providers (Ollama, OpenAI, LMStudio)
|
|
97
|
-
├── indexer/ # Scanner, chunker, indexer orchestration
|
|
98
|
-
├── store/ # Vector storage backends (GOB, Postgres, Qdrant)
|
|
99
|
-
├── search/ # Semantic + hybrid search
|
|
100
|
-
├── trace/ # Symbol extraction, call graph (tree-sitter based)
|
|
101
|
-
├── watcher/ # File watching with debouncing
|
|
102
|
-
├── daemon/ # Background indexing daemon
|
|
103
|
-
├── config/ # YAML configuration management
|
|
104
|
-
├── updater/ # Self-update mechanism
|
|
105
|
-
└── mcp/ # MCP server (5 tools)
|
|
106
|
-
```
|
|
107
|
-
|
|
108
|
-
**Entry Point Flow**:
|
|
109
|
-
```
|
|
110
|
-
cmd/grepai/main.go → cli/root.go → cli/{init,watch,search,trace}.go
|
|
111
|
-
```
|
|
30
|
+
### Technology Stack
|
|
112
31
|
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
32
|
+
| Component | Technology |
|
|
33
|
+
|-----------|-----------|
|
|
34
|
+
| **Language** | Go 1.22+ |
|
|
35
|
+
| **Vector Store** | GOB (file-based), PostgreSQL/pgvector, Qdrant |
|
|
36
|
+
| **Embedding** | Ollama (default), LM Studio, OpenAI, Synthetic, OpenRouter |
|
|
37
|
+
| **File Watching** | fsnotify with debouncing |
|
|
38
|
+
| **CLI Framework** | cobra |
|
|
39
|
+
| **MCP** | mark3labs/mcp-go |
|
|
40
|
+
| **Call Graph** | tree-sitter (Go, TypeScript, C#, F#, more) |
|
|
41
|
+
| **Knowledge Graph** | RPG (custom, GOB-serialized) |
|
|
42
|
+
| **Testing** | Go stdlib + race detection |
|
|
43
|
+
| **TUI** | Bubble Tea |
|
|
124
44
|
|
|
125
|
-
###
|
|
45
|
+
### Production Readiness
|
|
126
46
|
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
| **Storage** | Vector DB (pgvector/Qdrant/GOB) | SQLite (facts, no vectors) |
|
|
133
|
-
| **Search** | Vector similarity + hybrid | Full-text search (FTS5) |
|
|
134
|
-
| **Indexing** | Automatic (file watcher) | Manual (transcript ingest) |
|
|
135
|
-
| **MCP Tools** | 5 (search, trace, status) | 8+ (recall, explain, promote) |
|
|
136
|
-
| **Scope** | Project-local or workspace | Global + Project dual-DB |
|
|
137
|
-
| **Truth Maintenance** | Hash-based deduplication | Supersession + conflict resolution |
|
|
138
|
-
| **Performance** | Optimized for speed (Go, batching) | Optimized for accuracy (resolution) |
|
|
139
|
-
| **Distribution** | Standalone CLI (Homebrew) | Ruby gem |
|
|
140
|
-
| **Testing** | Go stdlib + race detection | RSpec with mocks |
|
|
141
|
-
| **UI** | CLI with bubbletea (TUI) | Pure CLI |
|
|
142
|
-
| **Call Graph** | Yes (tree-sitter AST) | No |
|
|
143
|
-
| **File Watching** | Yes (fsnotify) | No (transcript-driven) |
|
|
144
|
-
|
|
145
|
-
**Key Architectural Difference**: grepai is **proactive** (watches files, updates index automatically) while ClaudeMemory is **reactive** (responds to transcript events via hooks).
|
|
47
|
+
- **Maturity**: Production-ready (v0.34.0, active community)
|
|
48
|
+
- **Test Coverage**: Comprehensive (race detection, cross-platform)
|
|
49
|
+
- **Documentation**: Dedicated docs site, blog, benchmarks
|
|
50
|
+
- **Distribution**: Homebrew, multi-platform binaries
|
|
51
|
+
- **Community**: 280K+ views on Reddit, ProductHunt featured, active contributors
|
|
146
52
|
|
|
147
53
|
---
|
|
148
54
|
|
|
149
|
-
##
|
|
150
|
-
|
|
151
|
-
### Component 1: Chunking Strategy
|
|
152
|
-
|
|
153
|
-
**Purpose**: Split code into overlapping chunks optimized for embedding
|
|
154
|
-
|
|
155
|
-
**Implementation** (indexer/chunker.go:47-100):
|
|
156
|
-
```go
|
|
157
|
-
// Character-based chunking (not line-based) handles minified files
|
|
158
|
-
maxChars := c.chunkSize * CharsPerToken // 512 tokens * 4 chars = 2048 chars
|
|
159
|
-
overlapChars := c.overlap * CharsPerToken // 50 tokens * 4 chars = 200 chars
|
|
160
|
-
|
|
161
|
-
for pos < len(content) {
|
|
162
|
-
end := pos + maxChars
|
|
163
|
-
// Try to break at newline for cleaner chunks
|
|
164
|
-
if end < len(content) {
|
|
165
|
-
lastNewline := strings.LastIndex(content[pos:end], "\n")
|
|
166
|
-
if lastNewline > 0 {
|
|
167
|
-
end = pos + lastNewline + 1
|
|
168
|
-
}
|
|
169
|
-
}
|
|
170
|
-
|
|
171
|
-
// Calculate line numbers using pre-built index
|
|
172
|
-
startLine := getLineNumber(lineStarts, pos)
|
|
173
|
-
endLine := getLineNumber(lineStarts, end-1)
|
|
174
|
-
|
|
175
|
-
chunks = append(chunks, ChunkInfo{
|
|
176
|
-
ID: fmt.Sprintf("%s_%d", filePath, chunkIndex),
|
|
177
|
-
FilePath: filePath,
|
|
178
|
-
StartLine: startLine,
|
|
179
|
-
EndLine: endLine,
|
|
180
|
-
Content: chunkContent,
|
|
181
|
-
Hash: sha256Hash, // For deduplication
|
|
182
|
-
})
|
|
183
|
-
|
|
184
|
-
pos = end - overlapChars // Overlap for context continuity
|
|
185
|
-
}
|
|
186
|
-
```
|
|
187
|
-
|
|
188
|
-
**Design Decisions**:
|
|
189
|
-
- **Character-based**: Handles minified files with very long lines
|
|
190
|
-
- **Overlap**: 50 tokens ensure context continuity across chunks
|
|
191
|
-
- **Newline breaking**: Prefers natural boundaries when possible
|
|
192
|
-
- **Line number mapping**: Pre-build index for O(log n) lookups
|
|
193
|
-
- **Hash-based deduplication**: Skip re-embedding unchanged chunks
|
|
194
|
-
|
|
195
|
-
**Performance**: ~4 chars per token is empirically validated for code.
|
|
196
|
-
|
|
197
|
-
---
|
|
55
|
+
## Architecture Overview
|
|
198
56
|
|
|
199
|
-
###
|
|
57
|
+
### Data Model (Updated)
|
|
200
58
|
|
|
201
|
-
**
|
|
59
|
+
**Three-Layer Index**:
|
|
202
60
|
|
|
203
|
-
**
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
pending map[string]FileEvent // Aggregates rapid changes
|
|
207
|
-
pendingMu sync.Mutex // Thread-safe updates
|
|
208
|
-
timer *time.Timer // Debounce timer
|
|
209
|
-
}
|
|
61
|
+
1. **Vector Index** (semantic search): Chunks with embeddings, pluggable backends
|
|
62
|
+
2. **Symbol Index** (call graph): tree-sitter AST → Symbol/Reference/CallEdge
|
|
63
|
+
3. **RPG Semantic Graph** (v0.31.0, NEW):
|
|
210
64
|
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
w.pending[relPath] = FileEvent{Type: eventType, Path: relPath}
|
|
219
|
-
|
|
220
|
-
// Reset debounce timer
|
|
221
|
-
if w.timer != nil {
|
|
222
|
-
w.timer.Stop()
|
|
223
|
-
}
|
|
224
|
-
w.timer = time.AfterFunc(time.Duration(w.debounceMs)*time.Millisecond,
|
|
225
|
-
w.flushPending)
|
|
226
|
-
|
|
227
|
-
w.pendingMu.Unlock()
|
|
228
|
-
}
|
|
229
|
-
}
|
|
230
|
-
}
|
|
65
|
+
```
|
|
66
|
+
Area (functional area, top-level) [V_H]
|
|
67
|
+
└── Category [V_H]
|
|
68
|
+
└── Subcategory [V_H]
|
|
69
|
+
└── File (source file) [V_L]
|
|
70
|
+
└── Symbol (function/class) [V_L]
|
|
71
|
+
└── Chunk (vector ref) [V_L]
|
|
231
72
|
```
|
|
232
73
|
|
|
233
|
-
**
|
|
234
|
-
-
|
|
235
|
-
-
|
|
236
|
-
-
|
|
237
|
-
-
|
|
74
|
+
**Edge Types** (`rpg/model.go:27-38`):
|
|
75
|
+
- `feature_parent`: Hierarchy edges
|
|
76
|
+
- `contains`: File → Symbol containment
|
|
77
|
+
- `invokes`: Call graph edges
|
|
78
|
+
- `imports`: Import relationships
|
|
79
|
+
- `maps_to_chunk`: Symbol → vector chunk
|
|
80
|
+
- `semantic_sim`: Feature similarity edges
|
|
238
81
|
|
|
239
|
-
|
|
82
|
+
### Key Design Patterns (Updated)
|
|
240
83
|
|
|
241
|
-
|
|
84
|
+
1. **RPG Encoder** (`rpg/indexer.go:29-75`): Orchestrates graph building from symbol store + vector store. Uses drift thresholds, feature extractors, and hierarchy builders.
|
|
242
85
|
|
|
243
|
-
|
|
86
|
+
2. **Triple Query Engine** (`rpg/query.go:72-80`): Three operations — `SearchNode` (Jaccard similarity), `FetchNode` (detailed context with parents/children/edges), `Explore` (graph traversal with depth/direction/edge-type filters).
|
|
244
87
|
|
|
245
|
-
**
|
|
88
|
+
3. **File Watcher with Debouncing** (`watcher/watcher.go:30-59`): fsnotify with configurable debounce, gitignore-aware, event deduplication via pending map.
|
|
246
89
|
|
|
247
|
-
**
|
|
248
|
-
```go
|
|
249
|
-
func (s *Server) registerTools() {
|
|
250
|
-
// 1. grepai_search
|
|
251
|
-
searchTool := mcp.NewTool("grepai_search",
|
|
252
|
-
mcp.WithDescription("Semantic code search..."),
|
|
253
|
-
mcp.WithString("query", mcp.Required(), mcp.Description("Natural language query")),
|
|
254
|
-
mcp.WithNumber("limit", mcp.Description("Max results (default: 10)")),
|
|
255
|
-
mcp.WithBoolean("compact", mcp.Description("Omit content (~80% token savings)")),
|
|
256
|
-
mcp.WithString("workspace", mcp.Description("Cross-project search")),
|
|
257
|
-
mcp.WithString("projects", mcp.Description("Filter by project names")),
|
|
258
|
-
)
|
|
259
|
-
s.mcpServer.AddTool(searchTool, s.handleSearch)
|
|
260
|
-
|
|
261
|
-
// 2. grepai_trace_callers - Find who calls a function
|
|
262
|
-
// 3. grepai_trace_callees - Find what a function calls
|
|
263
|
-
// 4. grepai_trace_graph - Build full call graph
|
|
264
|
-
// 5. grepai_index_status - Health check
|
|
265
|
-
}
|
|
90
|
+
4. **Embedder Factory** (`embedder/factory.go`): `NewFromConfig`/`NewFromWorkspaceConfig` centralizes provider initialization across CLI and MCP server. Supports Ollama, LM Studio, OpenAI, Synthetic, OpenRouter.
|
|
266
91
|
|
|
267
|
-
|
|
268
|
-
query := request.RequireString("query")
|
|
269
|
-
limit := request.GetInt("limit", 10)
|
|
270
|
-
compact := request.GetBool("compact", false)
|
|
271
|
-
|
|
272
|
-
// Initialize embedder + store
|
|
273
|
-
emb := s.createEmbedder(cfg)
|
|
274
|
-
st := s.createStore(ctx, cfg)
|
|
275
|
-
defer emb.Close(); defer st.Close()
|
|
276
|
-
|
|
277
|
-
// Search
|
|
278
|
-
searcher := search.NewSearcher(st, emb, cfg.Search)
|
|
279
|
-
results := searcher.Search(ctx, query, limit)
|
|
280
|
-
|
|
281
|
-
// Return compact or full results
|
|
282
|
-
if compact {
|
|
283
|
-
return SearchResultCompact{FilePath, StartLine, EndLine, Score}
|
|
284
|
-
} else {
|
|
285
|
-
return SearchResult{FilePath, StartLine, EndLine, Score, Content}
|
|
286
|
-
}
|
|
287
|
-
}
|
|
288
|
-
```
|
|
92
|
+
5. **GOB File Locking** (v0.29.0): Cross-process safety for shared index files. Prevents data corruption in multi-process scenarios.
|
|
289
93
|
|
|
290
|
-
|
|
291
|
-
- **5 focused tools**: Search, trace callers, trace callees, trace graph, status
|
|
292
|
-
- **Compact mode**: Critical for token efficiency (~80% reduction)
|
|
293
|
-
- **Workspace support**: Search across multiple projects with shared embedder/store
|
|
294
|
-
- **Project filtering**: Comma-separated list filters workspace results
|
|
295
|
-
- **Error handling**: Returns `ToolResultError` instead of throwing exceptions
|
|
296
|
-
- **Resource cleanup**: Explicit `defer` for embedder/store cleanup
|
|
94
|
+
### Comparison with ClaudeMemory
|
|
297
95
|
|
|
298
|
-
|
|
96
|
+
| Aspect | grepai (0.34.0) | ClaudeMemory | Notes |
|
|
97
|
+
|--------|-----------------|--------------|-------|
|
|
98
|
+
| **Data Model** | Code chunks + RPG graph | Subject-predicate-object facts | Different domains |
|
|
99
|
+
| **Storage** | GOB/PostgreSQL/Qdrant | Dual SQLite | We're simpler |
|
|
100
|
+
| **Embeddings** | Ollama/OpenAI/etc (remote) | fastembed-rb (local ONNX) | We're self-contained |
|
|
101
|
+
| **File Watching** | fsnotify + debounce | None (hook-triggered) | They auto-index |
|
|
102
|
+
| **Knowledge Graph** | RPG semantic graph | Fact links (supersession/conflict) | Different purpose |
|
|
103
|
+
| **MCP** | mcp-go (Go) | Custom Ruby MCP server | Different languages |
|
|
104
|
+
| **CLI** | cobra (Go) | Custom Ruby CLI | Different stacks |
|
|
299
105
|
|
|
300
106
|
---
|
|
301
107
|
|
|
302
|
-
|
|
108
|
+
## Key Components Deep-Dive
|
|
303
109
|
|
|
304
|
-
|
|
110
|
+
### Component 1: RPG Semantic Graph (NEW)
|
|
305
111
|
|
|
306
|
-
**
|
|
307
|
-
- Uses tree-sitter parsers for Go, TypeScript, Python, JavaScript, C#, Java, Ruby, Rust
|
|
308
|
-
- Extracts symbols (function definitions) and references (function calls)
|
|
309
|
-
- Builds call graph: `Symbol → References → Callers/Callees`
|
|
112
|
+
**Purpose**: Map code structure into searchable knowledge graph with functional areas, categories, and semantic relationships.
|
|
310
113
|
|
|
311
|
-
**
|
|
312
|
-
```go
|
|
313
|
-
type Symbol struct {
|
|
314
|
-
Name string // Function/method name
|
|
315
|
-
Kind SymbolKind // function, method, class, interface
|
|
316
|
-
File string
|
|
317
|
-
Line int
|
|
318
|
-
Signature string // Full signature for disambiguation
|
|
319
|
-
Receiver string // For methods (e.g., "MyStruct")
|
|
320
|
-
Package string
|
|
321
|
-
Exported bool // Public vs private
|
|
322
|
-
Language string
|
|
323
|
-
}
|
|
324
|
-
```
|
|
114
|
+
**Location**: `rpg/` (model.go, indexer.go, query.go, hierarchy.go, evolver.go)
|
|
325
115
|
|
|
326
|
-
**
|
|
116
|
+
**Implementation** (`rpg/model.go:42-80`):
|
|
327
117
|
```go
|
|
328
|
-
type
|
|
329
|
-
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
|
|
335
|
-
|
|
118
|
+
type Node struct {
|
|
119
|
+
ID string `json:"id"`
|
|
120
|
+
Kind NodeKind `json:"kind"` // area, category, file, symbol, chunk
|
|
121
|
+
Feature string `json:"feature"` // primary semantic label
|
|
122
|
+
Features []string `json:"features"` // atomic semantic features
|
|
123
|
+
Path string `json:"path,omitempty"`
|
|
124
|
+
SymbolName string `json:"symbol_name,omitempty"`
|
|
125
|
+
Summary string `json:"summary,omitempty"` // LLM-generated
|
|
336
126
|
}
|
|
337
127
|
```
|
|
338
128
|
|
|
339
|
-
**Call Graph Query** (trace/store.go):
|
|
340
|
-
- `LookupCallers(symbolName)` → All references where symbol is called
|
|
341
|
-
- `LookupCallees(symbolName, file)` → All references within symbol's body
|
|
342
|
-
- `GetCallGraph(symbolName, depth)` → Multi-level traversal (BFS)
|
|
343
|
-
|
|
344
129
|
**Design Decisions**:
|
|
345
|
-
-
|
|
346
|
-
-
|
|
347
|
-
-
|
|
348
|
-
-
|
|
349
|
-
-
|
|
130
|
+
- Hierarchical: Area → Category → Subcategory → File → Symbol → Chunk
|
|
131
|
+
- Feature-based similarity using Jaccard coefficient
|
|
132
|
+
- Co-caller affinity edges with occurrence threshold
|
|
133
|
+
- Drift threshold for incremental updates (`RPGEncoderConfig.DriftThreshold`)
|
|
134
|
+
- GOB serialization for persistence
|
|
350
135
|
|
|
351
|
-
|
|
136
|
+
### Component 2: File Watcher
|
|
352
137
|
|
|
353
|
-
|
|
138
|
+
**Purpose**: Keep index fresh automatically as files change.
|
|
354
139
|
|
|
355
|
-
**
|
|
140
|
+
**Location**: `watcher/watcher.go:30-80`
|
|
356
141
|
|
|
357
|
-
**Implementation** (search/search.go):
|
|
358
142
|
```go
|
|
359
|
-
type
|
|
360
|
-
|
|
361
|
-
|
|
362
|
-
|
|
363
|
-
|
|
364
|
-
|
|
365
|
-
|
|
366
|
-
return s.vectorSearch(ctx, query, limit)
|
|
367
|
-
}
|
|
368
|
-
|
|
369
|
-
// 1. Vector search (semantic)
|
|
370
|
-
vectorResults := s.vectorSearch(ctx, query, limit*2)
|
|
371
|
-
|
|
372
|
-
// 2. Text search (keyword)
|
|
373
|
-
textResults := s.textSearch(ctx, query, limit*2)
|
|
374
|
-
|
|
375
|
-
// 3. Reciprocal Rank Fusion (RRF)
|
|
376
|
-
fusedResults := s.fuseResults(vectorResults, textResults)
|
|
377
|
-
|
|
378
|
-
return fusedResults[:limit]
|
|
379
|
-
}
|
|
380
|
-
|
|
381
|
-
func (s *Searcher) fuseResults(vec, text []Result) []Result {
|
|
382
|
-
scores := make(map[string]float32)
|
|
383
|
-
for rank, r := range vec {
|
|
384
|
-
scores[r.Chunk.ID] += 1.0 / float32(rank + s.cfg.Hybrid.K)
|
|
385
|
-
}
|
|
386
|
-
for rank, r := range text {
|
|
387
|
-
scores[r.Chunk.ID] += 1.0 / float32(rank + s.cfg.Hybrid.K)
|
|
388
|
-
}
|
|
389
|
-
// Sort by fused score
|
|
143
|
+
type Watcher struct {
|
|
144
|
+
root string
|
|
145
|
+
watcher *fsnotify.Watcher
|
|
146
|
+
ignore *indexer.IgnoreMatcher
|
|
147
|
+
debounceMs int
|
|
148
|
+
events chan FileEvent
|
|
149
|
+
pending map[string]FileEvent // debounce deduplication
|
|
390
150
|
}
|
|
391
151
|
```
|
|
392
152
|
|
|
393
153
|
**Design Decisions**:
|
|
394
|
-
-
|
|
395
|
-
-
|
|
396
|
-
-
|
|
397
|
-
-
|
|
154
|
+
- Configurable debounce (default 300ms)
|
|
155
|
+
- Recursive directory watching
|
|
156
|
+
- Gitignore-aware filtering
|
|
157
|
+
- Event deduplication via pending map
|
|
158
|
+
- Multi-worktree support (v0.30.0)
|
|
398
159
|
|
|
399
|
-
|
|
160
|
+
### Component 3: MCP Discovery Tools (NEW)
|
|
400
161
|
|
|
401
|
-
|
|
402
|
-
|
|
403
|
-
### Component 6: Pluggable Storage Backends
|
|
404
|
-
|
|
405
|
-
**Purpose**: Support different deployment scenarios (local, team, cloud)
|
|
406
|
-
|
|
407
|
-
**Interface** (store/store.go:50):
|
|
408
|
-
```go
|
|
409
|
-
type VectorStore interface {
|
|
410
|
-
SaveChunks(ctx context.Context, chunks []Chunk) error
|
|
411
|
-
DeleteByFile(ctx context.Context, filePath string) error
|
|
412
|
-
Search(ctx context.Context, queryVector []float32, limit int) ([]SearchResult, error)
|
|
413
|
-
GetDocument(ctx context.Context, filePath string) (*Document, error)
|
|
414
|
-
Load(ctx context.Context) error
|
|
415
|
-
Persist(ctx context.Context) error
|
|
416
|
-
Close() error
|
|
417
|
-
GetStats(ctx context.Context) (*IndexStats, error)
|
|
418
|
-
}
|
|
419
|
-
```
|
|
162
|
+
**Purpose**: Let AI agents discover available search scopes.
|
|
420
163
|
|
|
421
|
-
**
|
|
422
|
-
|
|
423
|
-
1. **GOB Store** (store/gob.go): File-based, single-project
|
|
424
|
-
- Storage: `.grepai/index.gob` (binary format)
|
|
425
|
-
- Search: In-memory cosine similarity (brute force)
|
|
426
|
-
- Use case: Individual developers, local projects
|
|
427
|
-
- Pros: Zero dependencies, fast for small codebases (<10K files)
|
|
428
|
-
- Cons: Memory usage grows with index size, no concurrent access
|
|
429
|
-
|
|
430
|
-
2. **PostgreSQL Store** (store/postgres.go): Database-backed, multi-project
|
|
431
|
-
- Storage: Postgres with pgvector extension
|
|
432
|
-
- Search: `<=>` operator (optimized with IVFFlat index)
|
|
433
|
-
- Use case: Teams, shared index across projects
|
|
434
|
-
- Pros: Concurrent access, persistence, scalable
|
|
435
|
-
- Cons: Requires Postgres + pgvector setup
|
|
436
|
-
|
|
437
|
-
3. **Qdrant Store** (store/qdrant.go): Dedicated vector DB
|
|
438
|
-
- Storage: Qdrant server (gRPC API)
|
|
439
|
-
- Search: Native vector search with HNSW indexing
|
|
440
|
-
- Use case: Large codebases (>50K files), production deployments
|
|
441
|
-
- Pros: Fastest search, advanced filtering, scalable
|
|
442
|
-
- Cons: Additional service to run
|
|
443
|
-
|
|
444
|
-
**Configuration** (.grepai/config.yaml):
|
|
445
|
-
```yaml
|
|
446
|
-
store:
|
|
447
|
-
backend: "gob" # or "postgres" or "qdrant"
|
|
448
|
-
postgres:
|
|
449
|
-
dsn: "postgresql://user:pass@localhost/grepai"
|
|
450
|
-
qdrant:
|
|
451
|
-
endpoint: "localhost"
|
|
452
|
-
port: 6334
|
|
453
|
-
collection: "my-codebase"
|
|
454
|
-
```
|
|
164
|
+
**Location**: `mcp/server.go`
|
|
455
165
|
|
|
456
166
|
**Design Decisions**:
|
|
457
|
-
-
|
|
458
|
-
-
|
|
459
|
-
-
|
|
460
|
-
- **Graceful degradation**: GOB works offline, Postgres/Qdrant require connectivity
|
|
167
|
+
- `grepai_list_workspaces`: Returns workspace-level entries for scope selection
|
|
168
|
+
- `grepai_list_projects`: Returns relative paths within workspace
|
|
169
|
+
- Compact vs full output modes for token efficiency
|
|
461
170
|
|
|
462
171
|
---
|
|
463
172
|
|
|
@@ -465,110 +174,20 @@ store:
|
|
|
465
174
|
|
|
466
175
|
### What They Do Well
|
|
467
176
|
|
|
468
|
-
|
|
469
|
-
|
|
470
|
-
|
|
471
|
-
-
|
|
472
|
-
|
|
473
|
-
|
|
474
|
-
#### 2. **Compact JSON Mode for Token Efficiency**
|
|
475
|
-
- **Value**: ~80% reduction in AI agent token usage
|
|
476
|
-
- **Evidence**: mcp/server.go:36 - `SearchResultCompact` omits content field
|
|
477
|
-
- **How**: MCP tools support `--compact` flag returning only file:line:score, not full chunks
|
|
478
|
-
- **Result**: Enables AI agents to search larger codebases without hitting context limits
|
|
479
|
-
|
|
480
|
-
#### 3. **Tree-sitter for Call Graph Analysis**
|
|
481
|
-
- **Value**: Language-agnostic static analysis without code execution
|
|
482
|
-
- **Evidence**: trace/extractor.go - Parsers for 8+ languages
|
|
483
|
-
- **How**: AST parsing extracts symbols + references, builds call graph
|
|
484
|
-
- **Result**: "Find callers" feature shows impact before refactoring
|
|
485
|
-
|
|
486
|
-
#### 4. **Interface-Based Extensibility**
|
|
487
|
-
- **Value**: Easy to add new embedders or storage backends
|
|
488
|
-
- **Evidence**: embedder/embedder.go:6, store/store.go:50
|
|
489
|
-
- **How**: Clean interfaces with multiple implementations (Ollama, OpenAI, LMStudio | GOB, Postgres, Qdrant)
|
|
490
|
-
- **Result**: Users choose deployment model (local, team, cloud) without code changes
|
|
491
|
-
|
|
492
|
-
#### 5. **Character-Based Chunking**
|
|
493
|
-
- **Value**: Handles minified files and long lines gracefully
|
|
494
|
-
- **Evidence**: indexer/chunker.go:54 - `maxChars := c.chunkSize * CharsPerToken`
|
|
495
|
-
- **How**: Splits by characters with newline preference, not by lines
|
|
496
|
-
- **Result**: No failures on minified JS/CSS, consistent chunk sizes
|
|
497
|
-
|
|
498
|
-
#### 6. **Hybrid Search (Vector + Text)**
|
|
499
|
-
- **Value**: Better relevance for queries with specific keywords
|
|
500
|
-
- **Evidence**: search/search.go - Reciprocal Rank Fusion (RRF)
|
|
501
|
-
- **How**: Combines cosine similarity with full-text search using RRF
|
|
502
|
-
- **Result**: Finds both semantically related and keyword-matched code
|
|
503
|
-
|
|
504
|
-
#### 7. **Multi-Platform Distribution**
|
|
505
|
-
- **Value**: Easy installation on Mac/Linux/Windows
|
|
506
|
-
- **Evidence**: .goreleaser.yml, install.sh, install.ps1, Homebrew tap
|
|
507
|
-
- **How**: GoReleaser builds cross-platform binaries, shell installers
|
|
508
|
-
- **Result**: Friction-free adoption (`brew install grepai`)
|
|
509
|
-
|
|
510
|
-
#### 8. **Workspace Mode**
|
|
511
|
-
- **Value**: Search across multiple projects simultaneously
|
|
512
|
-
- **Evidence**: mcp/server.go:252 - `handleWorkspaceSearch`
|
|
513
|
-
- **How**: Shared embedder + store with project filtering
|
|
514
|
-
- **Result**: Reuse patterns across related projects (microservices, monorepos)
|
|
515
|
-
|
|
516
|
-
---
|
|
177
|
+
1. **RPG Knowledge Graph**: Rich semantic structure for code understanding
|
|
178
|
+
2. **File Watching**: Automatic index updates without manual commands
|
|
179
|
+
3. **Embedder Factory**: Clean abstraction for multiple embedding providers
|
|
180
|
+
4. **Cross-Process Safety**: GOB file locking prevents data corruption
|
|
181
|
+
5. **MCP Discovery**: Agent-friendly scope exploration tools
|
|
182
|
+
6. **Multi-Worktree**: Git worktree detection and parallel watching
|
|
517
183
|
|
|
518
184
|
### What We Do Well
|
|
519
185
|
|
|
520
|
-
|
|
521
|
-
|
|
522
|
-
|
|
523
|
-
|
|
524
|
-
|
|
525
|
-
|
|
526
|
-
#### 2. **Dual-Database Scoping (Global + Project)**
|
|
527
|
-
- **Our Advantage**: User preferences (global) vs project-specific facts
|
|
528
|
-
- **Evidence**: store/store_manager.rb - Manages two SQLite connections
|
|
529
|
-
- **Value**: Facts apply at correct granularity (always vs this-project-only)
|
|
530
|
-
- **Why**: Memory has semantic scope; search is purely location-based
|
|
531
|
-
|
|
532
|
-
#### 3. **Provenance Tracking**
|
|
533
|
-
- **Our Advantage**: Every fact links to source transcript content
|
|
534
|
-
- **Evidence**: domain/provenance.rb - Links facts to content_items
|
|
535
|
-
- **Value**: Users can verify where facts came from, assess confidence
|
|
536
|
-
- **Why**: Memory requires trustworthiness; search assumes correctness
|
|
537
|
-
|
|
538
|
-
#### 4. **Pluggable Distiller Interface**
|
|
539
|
-
- **Our Advantage**: AI-powered fact extraction (future: Claude API)
|
|
540
|
-
- **Evidence**: distill/distiller.rb - Extracts entities, facts, scope hints
|
|
541
|
-
- **Value**: Understands context and intent, not just code structure
|
|
542
|
-
- **Why**: Memory extracts meaning; search indexes literal content
|
|
543
|
-
|
|
544
|
-
#### 5. **Hook-Based Integration**
|
|
545
|
-
- **Our Advantage**: Seamless integration with Claude Code events
|
|
546
|
-
- **Evidence**: hook/ - Reads stdin JSON from Claude Code hooks
|
|
547
|
-
- **Value**: Zero-effort ingestion, automatic sweeping
|
|
548
|
-
- **Why**: Memory is reactive to AI sessions; search is proactive (file watcher)
|
|
549
|
-
|
|
550
|
-
---
|
|
551
|
-
|
|
552
|
-
### Trade-offs
|
|
553
|
-
|
|
554
|
-
| Approach | Pros | Cons |
|
|
555
|
-
|----------|------|------|
|
|
556
|
-
| **Their: Proactive file watching** | Always up-to-date, zero manual work | CPU/disk overhead, battery drain on laptops, irrelevant for non-code files |
|
|
557
|
-
| **Ours: Reactive transcript ingest** | Only processes meaningful interactions | Requires hook setup, delayed until session ends |
|
|
558
|
-
| **Their: Vector embeddings** | Semantic understanding, fuzzy matching | Requires embedding provider, slower than text search, cost (if OpenAI) |
|
|
559
|
-
| **Ours: FTS5 full-text search** | Fast, zero dependencies, no cost | Exact/substring matching only, no semantic understanding |
|
|
560
|
-
| **Their: Go (compiled)** | Fast startup, low memory, easy distribution | Harder to extend (compile step), less metaprogramming |
|
|
561
|
-
| **Ours: Ruby (interpreted)** | Easy to extend, rich metaprogramming | Slower, requires Ruby runtime, harder to distribute |
|
|
562
|
-
| **Their: Chunk-based storage** | Optimized for retrieval granularity | Redundancy across chunks, harder to track changes |
|
|
563
|
-
| **Ours: Fact-based storage** | Deduplicated, structured, queryable | Requires distillation step, more complex schema |
|
|
564
|
-
| **Their: Interface-based backends** | Pluggable storage (GOB/Postgres/Qdrant) | Complexity in maintaining multiple implementations |
|
|
565
|
-
| **Ours: SQLite-only** | Simple, zero config, portable | Limited scalability, no built-in vector search |
|
|
566
|
-
| **Their: Tree-sitter call graph** | Static analysis, language-agnostic | Setup cost (parsers), limited to supported languages |
|
|
567
|
-
| **Ours: No call graph** | Simpler architecture | Can't answer "who calls this?" questions |
|
|
568
|
-
| **Their: Compact JSON mode** | Token-efficient for AI agents | Requires Read tool for full content (two-step) |
|
|
569
|
-
| **Ours: Full content in recall** | One-step retrieval | Higher token usage per query |
|
|
570
|
-
|
|
571
|
-
**Key Trade-off**: grepai prioritizes **search speed and semantic understanding** at the cost of setup complexity. ClaudeMemory prioritizes **truth maintenance and provenance** at the cost of search sophistication.
|
|
186
|
+
1. **Self-Contained Embeddings**: fastembed-rb needs no external service
|
|
187
|
+
2. **Dual-Database Scope**: Project/global separation is cleaner
|
|
188
|
+
3. **Truth Maintenance**: Temporal validity and conflict resolution
|
|
189
|
+
4. **Rich MCP Tools**: 18 specialized tools vs their general search
|
|
190
|
+
5. **Simpler Architecture**: No background processes or external databases
|
|
572
191
|
|
|
573
192
|
---
|
|
574
193
|
|
|
@@ -577,357 +196,62 @@ store:
|
|
|
577
196
|
### High Priority ⭐
|
|
578
197
|
|
|
579
198
|
#### 1. Incremental Indexing with File Watching
|
|
580
|
-
- **Value**:
|
|
581
|
-
- **Evidence**: watcher/watcher.go:
|
|
582
|
-
- **Implementation**:
|
|
583
|
-
|
|
584
|
-
|
|
585
|
-
|
|
586
|
-
4. Debounce events (default 500ms to avoid thrashing during bulk writes)
|
|
587
|
-
5. Trigger `IngestCommand` automatically when new transcript data appears
|
|
588
|
-
6. Optional: Watch `.claude/rules/` for manual fact additions
|
|
589
|
-
- **Effort**: 2-3 days (watcher class, integration with ingest, testing)
|
|
590
|
-
- **Trade-off**: Adds background process (memory overhead ~10MB), may complicate testing
|
|
591
|
-
- **Recommendation**: **ADOPT** - Eliminates manual `claude-memory ingest` calls, huge UX win
|
|
592
|
-
|
|
593
|
-
#### 2. Compact Response Format for MCP Tools
|
|
594
|
-
- **Value**: Reduce token usage by ~60% in MCP responses by omitting verbose content
|
|
595
|
-
- **Evidence**: mcp/server.go:219 - `SearchResultCompact` omits content field, returns only metadata
|
|
596
|
-
- **Implementation**:
|
|
597
|
-
1. Add `compact` boolean parameter to `memory.recall` and `memory.search_*` tools
|
|
598
|
-
2. Create `CompactFormatter` in `MCP::ResponseFormatter`:
|
|
599
|
-
```ruby
|
|
600
|
-
def format_fact_compact(fact)
|
|
601
|
-
{
|
|
602
|
-
id: fact.id,
|
|
603
|
-
subject: fact.subject,
|
|
604
|
-
predicate: fact.predicate,
|
|
605
|
-
object: fact.object,
|
|
606
|
-
scope: fact.scope,
|
|
607
|
-
confidence: fact.confidence
|
|
608
|
-
}
|
|
609
|
-
# Omit: provenance, supersession_chain, context excerpts
|
|
610
|
-
end
|
|
611
|
-
```
|
|
612
|
-
3. Default to `compact: true` for all MCP tools (user can override with `compact: false`)
|
|
613
|
-
4. Update tool descriptions to explain compact mode
|
|
614
|
-
- **Effort**: 4-6 hours (add parameter, update formatters, tests)
|
|
615
|
-
- **Trade-off**: User needs follow-up `memory.explain <fact_id>` for full context (two-step interaction)
|
|
616
|
-
- **Recommendation**: **ADOPT** - Critical for scaling to large fact databases (1000+ facts)
|
|
617
|
-
|
|
618
|
-
#### 3. Hybrid Search (Vector + Text)
|
|
619
|
-
- **Value**: Better relevance when users search for specific terms (e.g., "uses_database") while preserving semantic matching
|
|
620
|
-
- **Evidence**: search/search.go - Reciprocal Rank Fusion (RRF) with K=60
|
|
621
|
-
- **Implementation**:
|
|
622
|
-
1. Add `sqlite-vec` extension to Gemfile for vector similarity in SQLite
|
|
623
|
-
2. Add `embeddings` column to `facts` table (BLOB storing float32 array)
|
|
624
|
-
3. Create `ClaudeMemory::Embedder` interface:
|
|
625
|
-
- Implementation: Call Anthropic API for embeddings (free with Claude usage)
|
|
626
|
-
- Cache embeddings per fact (regenerate only when fact changes)
|
|
627
|
-
4. Implement RRF in `Recall#query`:
|
|
628
|
-
```ruby
|
|
629
|
-
vector_results = vector_search(query, limit * 2) # Cosine similarity
|
|
630
|
-
text_results = fts_search(query, limit * 2) # Existing FTS5
|
|
631
|
-
fuse_with_rrf(vector_results, text_results, k: 60)
|
|
632
|
-
```
|
|
633
|
-
5. Make hybrid search optional via `.grepai/config.yaml`:
|
|
634
|
-
```yaml
|
|
635
|
-
search:
|
|
636
|
-
hybrid:
|
|
637
|
-
enabled: true
|
|
638
|
-
k: 60
|
|
639
|
-
```
|
|
640
|
-
- **Effort**: 5-7 days (embedder setup, schema migration, RRF implementation, testing)
|
|
641
|
-
- **Trade-off**: Requires API calls for embedding (cost ~$0.00001/fact), slower queries (2x search + fusion)
|
|
642
|
-
- **Recommendation**: **CONSIDER** - High value but significant implementation effort. Start with FTS5, add vectors later if search quality issues arise.
|
|
643
|
-
|
|
644
|
-
#### 4. Call Graph for Fact Dependencies
|
|
645
|
-
- **Value**: Show which facts depend on others (supersession chains, conflict relationships) visually
|
|
646
|
-
- **Evidence**: trace/trace.go:95 - `CallGraph` struct with nodes and edges
|
|
647
|
-
- **Implementation**:
|
|
648
|
-
1. Create `memory.fact_graph <fact_id> --depth 2` MCP tool
|
|
649
|
-
2. Query `fact_links` table to build graph:
|
|
650
|
-
- Nodes: Facts (subject/predicate/object)
|
|
651
|
-
- Edges: Supersedes, Conflicts, Supports
|
|
652
|
-
3. Return JSON matching grepai's format:
|
|
653
|
-
```json
|
|
654
|
-
{
|
|
655
|
-
"root": "fact_123",
|
|
656
|
-
"nodes": {"fact_123": {...}, "fact_456": {...}},
|
|
657
|
-
"edges": [
|
|
658
|
-
{"from": "fact_123", "to": "fact_456", "type": "supersedes"},
|
|
659
|
-
{"from": "fact_123", "to": "fact_789", "type": "conflicts"}
|
|
660
|
-
],
|
|
661
|
-
"depth": 2
|
|
662
|
-
}
|
|
663
|
-
```
|
|
664
|
-
4. Depth-limited BFS traversal (avoid exponential explosion)
|
|
665
|
-
- **Effort**: 2-3 days (graph builder, MCP tool, tests)
|
|
666
|
-
- **Trade-off**: Adds complexity for a feature used mainly for debugging/exploration
|
|
667
|
-
- **Recommendation**: **ADOPT** - Invaluable for understanding why facts were superseded or conflicted
|
|
668
|
-
|
|
669
|
-
#### 5. Multi-Project Workspace Mode
|
|
670
|
-
- **Value**: Search facts across multiple projects simultaneously (e.g., all Ruby projects)
|
|
671
|
-
- **Evidence**: mcp/server.go:252 - `handleWorkspaceSearch` with project filtering
|
|
672
|
-
- **Implementation**:
|
|
673
|
-
1. Extend `.claude/settings.json` with workspace config:
|
|
674
|
-
```json
|
|
675
|
-
{
|
|
676
|
-
"workspaces": {
|
|
677
|
-
"ruby-projects": {
|
|
678
|
-
"projects": [
|
|
679
|
-
"/Users/me/project1",
|
|
680
|
-
"/Users/me/project2"
|
|
681
|
-
],
|
|
682
|
-
"scope": "project" // Only search project-scoped facts
|
|
683
|
-
}
|
|
684
|
-
}
|
|
685
|
-
}
|
|
686
|
-
```
|
|
687
|
-
2. Add `workspace` parameter to `memory.recall`:
|
|
688
|
-
```ruby
|
|
689
|
-
memory.recall(query: "authentication", workspace: "ruby-projects")
|
|
690
|
-
```
|
|
691
|
-
3. StoreManager opens all project databases, merges results
|
|
692
|
-
4. Filter by `project_path` matching workspace projects
|
|
693
|
-
- **Effort**: 3-4 days (workspace config, multi-DB queries, result merging, tests)
|
|
694
|
-
- **Trade-off**: Complexity in managing multiple DB connections, potential for confusion (which project is this fact from?)
|
|
695
|
-
- **Recommendation**: **DEFER** - Nice-to-have but low ROI for current use case (most users work on one project at a time)
|
|
696
|
-
|
|
697
|
-
---
|
|
199
|
+
- **Value**: Eliminates manual `claude-memory ingest` calls
|
|
200
|
+
- **Evidence**: `watcher/watcher.go:30-59` — fsnotify with debouncing, gitignore respect
|
|
201
|
+
- **Implementation**: Add `listen` gem (Ruby fsnotify equivalent), watch `.claude/projects/*/transcripts/*.jsonl`, debounce 500ms, trigger IngestCommand
|
|
202
|
+
- **Effort**: 2-3 days
|
|
203
|
+
- **Trade-off**: Background process ~10MB memory overhead
|
|
204
|
+
- **Recommendation**: **CONSIDER** — Already in improvements.md (#3), reinforced by this study
|
|
698
205
|
|
|
699
206
|
### Medium Priority
|
|
700
207
|
|
|
701
|
-
####
|
|
702
|
-
- **Value**:
|
|
703
|
-
- **Evidence**:
|
|
704
|
-
- **Implementation**:
|
|
705
|
-
|
|
706
|
-
|
|
707
|
-
|
|
708
|
-
|
|
709
|
-
|
|
710
|
-
- **
|
|
711
|
-
- **
|
|
712
|
-
- **
|
|
713
|
-
|
|
714
|
-
|
|
715
|
-
- **
|
|
716
|
-
- **Evidence**: config/config.go - `.grepai/config.yaml` with typed structs
|
|
717
|
-
- **Implementation**:
|
|
718
|
-
1. Add `.claude/memory.yaml` support:
|
|
719
|
-
```yaml
|
|
720
|
-
ingest:
|
|
721
|
-
auto_sweep: true
|
|
722
|
-
sweep_budget_seconds: 5
|
|
723
|
-
publish:
|
|
724
|
-
mode: shared
|
|
725
|
-
include_provenance: false
|
|
726
|
-
recall:
|
|
727
|
-
default_limit: 10
|
|
728
|
-
scope: all
|
|
729
|
-
```
|
|
730
|
-
2. Load YAML in `Configuration` class, merge with ENV vars (ENV takes precedence)
|
|
731
|
-
3. Validate config on load (raise clear errors for invalid values)
|
|
732
|
-
- **Effort**: 1-2 days (YAML parsing, validation, tests)
|
|
733
|
-
- **Trade-off**: Another configuration method to document and support
|
|
734
|
-
- **Recommendation**: **CONSIDER** - Better UX, but JSON in `.claude/settings.json` works fine for now
|
|
735
|
-
|
|
736
|
-
#### 8. Batch MCP Tool Operations
|
|
737
|
-
- **Value**: Reduce round-trips when agent needs to recall multiple facts
|
|
738
|
-
- **Evidence**: embedder/embedder.go:10 - `EmbedBatch` for parallel processing
|
|
739
|
-
- **Implementation**:
|
|
740
|
-
1. Add `memory.recall_batch` tool accepting array of queries:
|
|
741
|
-
```json
|
|
742
|
-
{
|
|
743
|
-
"queries": [
|
|
744
|
-
{"query": "authentication", "limit": 5},
|
|
745
|
-
{"query": "database schema", "limit": 3}
|
|
746
|
-
]
|
|
747
|
-
}
|
|
748
|
-
```
|
|
749
|
-
2. Execute queries in parallel (use `Concurrent::Future` from concurrent-ruby gem)
|
|
750
|
-
3. Return merged results with query labels:
|
|
751
|
-
```json
|
|
752
|
-
[
|
|
753
|
-
{"query": "authentication", "results": [...]},
|
|
754
|
-
{"query": "database schema", "results": [...]}
|
|
755
|
-
]
|
|
756
|
-
```
|
|
757
|
-
- **Effort**: 1-2 days (batch tool, parallel execution, tests)
|
|
758
|
-
- **Trade-off**: More complex error handling (what if one query fails?)
|
|
759
|
-
- **Recommendation**: **CONSIDER** - Useful if agents frequently need multiple searches, but current single-query API is simpler
|
|
760
|
-
|
|
761
|
-
---
|
|
762
|
-
|
|
763
|
-
### Low Priority
|
|
764
|
-
|
|
765
|
-
#### 9. TUI (Terminal UI) for Interactive Exploration
|
|
766
|
-
- **Value**: Visual interface for browsing facts, conflicts, provenance
|
|
767
|
-
- **Evidence**: cli/ uses `charmbracelet/bubbletea` for rich TUI
|
|
768
|
-
- **Implementation**:
|
|
769
|
-
1. Add `claude-memory explore` command launching TUI
|
|
770
|
-
2. Use `tty-prompt` gem for interactive menus:
|
|
771
|
-
- Browse facts by predicate
|
|
772
|
-
- Explore supersession chains
|
|
773
|
-
- View conflict details
|
|
774
|
-
- Search facts interactively
|
|
775
|
-
3. Display provenance excerpts inline
|
|
776
|
-
- **Effort**: 3-5 days (TUI design, navigation, rendering)
|
|
777
|
-
- **Trade-off**: Adds dependency and complexity for limited use case (most interaction via MCP tools)
|
|
778
|
-
- **Recommendation**: **DEFER** - Nice for power users, but MCP tools + `memory.explain` cover 90% of needs
|
|
779
|
-
|
|
780
|
-
#### 10. Prometheus Metrics Endpoint
|
|
781
|
-
- **Value**: Monitor memory system health (fact count, sweep duration, query latency)
|
|
782
|
-
- **Evidence**: grepai exposes metrics via `grepai_index_status` tool
|
|
783
|
-
- **Implementation**:
|
|
784
|
-
1. Add `claude-memory serve-metrics` command (HTTP server on `:9090/metrics`)
|
|
785
|
-
2. Expose Prometheus-format metrics:
|
|
786
|
-
- `claude_memory_facts_total{scope="global|project"}`
|
|
787
|
-
- `claude_memory_sweep_duration_seconds`
|
|
788
|
-
- `claude_memory_recall_latency_seconds`
|
|
789
|
-
3. Optional: Export to StatsD, DataDog, etc.
|
|
790
|
-
- **Effort**: 2-3 days (metrics collection, HTTP server, testing)
|
|
791
|
-
- **Trade-off**: Requires running separate metrics server, overkill for most users
|
|
792
|
-
- **Recommendation**: **DEFER** - Only needed for production deployments with SLAs
|
|
793
|
-
|
|
794
|
-
---
|
|
208
|
+
#### 2. MCP Discovery Tools
|
|
209
|
+
- **Value**: Let Claude discover available search scopes before querying
|
|
210
|
+
- **Evidence**: `mcp/server.go` — `grepai_list_workspaces`, `grepai_list_projects`
|
|
211
|
+
- **Implementation**: Add `memory.list_projects` tool showing databases with fact counts
|
|
212
|
+
- **Effort**: 1 day
|
|
213
|
+
- **Trade-off**: Minimal — useful metadata
|
|
214
|
+
- **Recommendation**: **CONSIDER**
|
|
215
|
+
|
|
216
|
+
#### 3. Embedder Factory Pattern
|
|
217
|
+
- **Value**: Clean abstraction for swapping embedding providers
|
|
218
|
+
- **Evidence**: `embedder/factory.go` — `NewFromConfig` centralizes initialization
|
|
219
|
+
- **Implementation**: Our fastembed-rb is already self-contained, but a factory would help if we add sqlite-vec and need to support different models
|
|
220
|
+
- **Effort**: 1-2 days
|
|
221
|
+
- **Trade-off**: Premature unless we add multiple embedding providers
|
|
222
|
+
- **Recommendation**: **DEFER**
|
|
795
223
|
|
|
796
224
|
### Features to Avoid
|
|
797
225
|
|
|
798
|
-
|
|
799
|
-
- **
|
|
800
|
-
- **
|
|
801
|
-
- **
|
|
802
|
-
|
|
803
|
-
|
|
804
|
-
- **Why Avoid**: Increases maintenance burden (test matrix, docs, support) for unclear benefit
|
|
805
|
-
- **Our Alternative**: SQLite is perfect for local storage, portable, and sufficient for fact databases (<100K facts). If we need remote storage, use libSQL (SQLite over HTTP) or Turso
|
|
806
|
-
- **Reasoning**: grepai needs backends for team collaboration (shared index). ClaudeMemory is single-user by design (global + project scoping)
|
|
807
|
-
|
|
808
|
-
#### 3. Daemon Mode with Background Indexing
|
|
809
|
-
- **Why Avoid**: Adds complexity (process management, logging, crash recovery) and battery drain
|
|
810
|
-
- **Our Alternative**: Hook-based reactive ingestion (current approach) is elegant and efficient. Only process transcripts when meaningful work happens (AI sessions)
|
|
811
|
-
- **Reasoning**: grepai needs daemon for real-time file watching. ClaudeMemory doesn't need to watch files (transcripts are append-only, hooks trigger ingestion)
|
|
812
|
-
|
|
813
|
-
---
|
|
814
|
-
|
|
815
|
-
## Implementation Recommendations
|
|
816
|
-
|
|
817
|
-
### Phase 1: Quick Wins (1-2 weeks)
|
|
818
|
-
|
|
819
|
-
**Goal**: Low-effort, high-value improvements
|
|
820
|
-
|
|
821
|
-
- [ ] **Compact MCP responses** (4-6 hours)
|
|
822
|
-
- Add `compact: true` parameter to all recall tools
|
|
823
|
-
- Omit provenance and context excerpts by default
|
|
824
|
-
- Test token reduction with realistic queries
|
|
825
|
-
- Success criteria: 60% token reduction in MCP responses
|
|
826
|
-
|
|
827
|
-
- [ ] **Fact dependency graph** (2-3 days)
|
|
828
|
-
- Implement `memory.fact_graph <fact_id> --depth 2` tool
|
|
829
|
-
- BFS traversal of fact_links table
|
|
830
|
-
- Return JSON with nodes and edges
|
|
831
|
-
- Success criteria: Visualize supersession chains and conflicts
|
|
832
|
-
|
|
833
|
-
### Phase 2: Incremental Indexing (2-3 weeks)
|
|
834
|
-
|
|
835
|
-
**Goal**: Auto-update index during coding sessions
|
|
836
|
-
|
|
837
|
-
- [ ] **File watcher for transcripts** (2-3 days)
|
|
838
|
-
- Add `Listen` gem (Ruby equivalent of fsnotify)
|
|
839
|
-
- Watch `.claude/projects/*/transcripts/*.jsonl` for changes
|
|
840
|
-
- Debounce rapid changes (500ms)
|
|
841
|
-
- Trigger `IngestCommand` automatically
|
|
842
|
-
- Success criteria: Index updates within 1 second of transcript write
|
|
843
|
-
|
|
844
|
-
- [ ] **Optional daemon mode** (2-3 days)
|
|
845
|
-
- Add `claude-memory watch` command (background process)
|
|
846
|
-
- Fork and daemonize, write PID file
|
|
847
|
-
- Graceful shutdown on SIGTERM
|
|
848
|
-
- Success criteria: Run in background, no manual ingest needed
|
|
849
|
-
|
|
850
|
-
- [ ] **Integration with existing hooks** (1 day)
|
|
851
|
-
- Keep existing hooks as fallback (if watcher not running)
|
|
852
|
-
- Add `auto_watch: true` setting to enable watcher
|
|
853
|
-
- Success criteria: Works with or without daemon
|
|
854
|
-
|
|
855
|
-
### Phase 3: Hybrid Search (4-6 weeks)
|
|
856
|
-
|
|
857
|
-
**Goal**: Add semantic search for better relevance
|
|
858
|
-
|
|
859
|
-
- [ ] **Embedder interface** (1 week)
|
|
860
|
-
- Create `ClaudeMemory::Embedder` module
|
|
861
|
-
- Implement Anthropic API embedder (call `/v1/embeddings`)
|
|
862
|
-
- Cache embeddings in `fact_embeddings` table
|
|
863
|
-
- Success criteria: Generate embeddings for facts
|
|
864
|
-
|
|
865
|
-
- [ ] **Vector storage** (1 week)
|
|
866
|
-
- Add `sqlite-vec` extension to dependencies
|
|
867
|
-
- Migrate schema: add `embedding` BLOB column to `facts`
|
|
868
|
-
- Implement cosine similarity search
|
|
869
|
-
- Success criteria: Vector search returns similar facts
|
|
870
|
-
|
|
871
|
-
- [ ] **RRF implementation** (1 week)
|
|
872
|
-
- Implement Reciprocal Rank Fusion in `Recall#query`
|
|
873
|
-
- Combine FTS5 results with vector results
|
|
874
|
-
- Make hybrid search optional via config
|
|
875
|
-
- Success criteria: Better relevance than FTS5 alone
|
|
876
|
-
|
|
877
|
-
- [ ] **Performance tuning** (1 week)
|
|
878
|
-
- Benchmark vector search vs FTS5
|
|
879
|
-
- Add caching for frequently-queried embeddings
|
|
880
|
-
- Optimize batch embedding (parallel API calls)
|
|
881
|
-
- Success criteria: Hybrid search <500ms for typical queries
|
|
882
|
-
|
|
883
|
-
---
|
|
884
|
-
|
|
885
|
-
## Architecture Decisions
|
|
886
|
-
|
|
887
|
-
### What to Preserve
|
|
888
|
-
|
|
889
|
-
- **SQLite-only storage**: Simple, portable, fast enough for fact databases
|
|
890
|
-
- **Hook-based integration**: Elegant reactive model (no polling, no daemons unless opted-in)
|
|
891
|
-
- **Fact-based data model**: Structured triples with provenance vs unstructured chunks
|
|
892
|
-
- **Truth maintenance**: Supersession and conflict resolution (grepai has no equivalent)
|
|
893
|
-
- **Dual-database scoping**: Global vs project facts (grepai has only project-local or workspace)
|
|
894
|
-
|
|
895
|
-
### What to Adopt
|
|
896
|
-
|
|
897
|
-
- **File watcher for incremental indexing**: Huge UX win, eliminates manual `ingest` calls
|
|
898
|
-
- **Compact JSON for MCP tools**: Critical for scaling to large fact databases
|
|
899
|
-
- **Fact dependency graph visualization**: Invaluable for debugging supersession/conflicts
|
|
900
|
-
- **Interface-based extensibility**: If we add embeddings, use pluggable `Embedder` interface
|
|
901
|
-
- **Hybrid search (RRF)**: Better relevance, proven technique
|
|
902
|
-
|
|
903
|
-
### What to Reject
|
|
904
|
-
|
|
905
|
-
- **Cloud-based embeddings**: Privacy concerns, cost, latency (use Anthropic API if needed)
|
|
906
|
-
- **Multiple storage backends**: Adds complexity without clear benefit for single-user tool
|
|
907
|
-
- **Daemon mode by default**: Optional is fine, but hooks are simpler and more efficient
|
|
908
|
-
- **Tree-sitter call graphs**: Out of scope for memory system (we track fact dependencies, not code dependencies)
|
|
909
|
-
- **Workspace mode**: Defer until multi-project use case is validated
|
|
226
|
+
- **RPG Knowledge Graph**: Wrong domain — code graph vs fact graph
|
|
227
|
+
- **Ollama/External Embeddings**: We use local ONNX embeddings (fastembed-rb)
|
|
228
|
+
- **PostgreSQL/Qdrant**: Over-engineering for our SQLite-based architecture
|
|
229
|
+
- **tree-sitter AST**: Not relevant to fact/memory retrieval
|
|
230
|
+
- **Bubble Tea TUI**: CLI output is sufficient
|
|
231
|
+
- **GOB Serialization**: We use SQLite which has built-in ACID
|
|
910
232
|
|
|
911
233
|
---
|
|
912
234
|
|
|
913
235
|
## Key Takeaways
|
|
914
236
|
|
|
915
|
-
|
|
916
|
-
|
|
917
|
-
|
|
918
|
-
|
|
919
|
-
|
|
920
|
-
|
|
921
|
-
|
|
237
|
+
### Changes Since Last Analysis (2026-01-29)
|
|
238
|
+
- RPG Semantic Graph Layer added (v0.31.0) — major new feature
|
|
239
|
+
- Workspace mode for multi-project support
|
|
240
|
+
- Bubble Tea TUI for interactive commands
|
|
241
|
+
- MCP discovery commands (list_workspaces, list_projects)
|
|
242
|
+
- Synthetic API + OpenRouter embedding providers
|
|
243
|
+
- Multi-worktree detection and parallel watching
|
|
244
|
+
- .grepaiignore support
|
|
245
|
+
- F# language support for trace
|
|
246
|
+
|
|
247
|
+
### Main Learnings
|
|
248
|
+
1. RPG graph is interesting but wrong domain for us
|
|
249
|
+
2. File watching with debouncing remains the best auto-indexing approach
|
|
250
|
+
3. MCP discovery tools are a nice UX pattern for multi-scope systems
|
|
251
|
+
4. Factory patterns for embedding providers are smart if you support multiple backends
|
|
922
252
|
|
|
923
|
-
|
|
924
|
-
|
|
925
|
-
6. **Go's performance and distribution advantages** are appealing, but Ruby is fine for our use case (not latency-critical, single-user tool). Don't rewrite unless clear performance issues arise.
|
|
926
|
-
|
|
927
|
-
7. **Recommended adoption order**:
|
|
928
|
-
- **Immediate**: Compact JSON, fact dependency graph
|
|
929
|
-
- **Short-term**: File watcher for transcripts
|
|
930
|
-
- **Medium-term**: Hybrid search (if needed)
|
|
931
|
-
- **Defer**: Workspace mode, cloud backends, TUI
|
|
253
|
+
---
|
|
932
254
|
|
|
933
|
-
|
|
255
|
+
*Analysis completed: 2026-03-02*
|
|
256
|
+
*Analyst: Claude Code*
|
|
257
|
+
*Review Status: Draft*
|