@lotargo/memory_plugin 1.3.2 → 1.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,324 +1,334 @@
1
- <div align="center">
2
-
3
- <img src="./assets/hero.jpg" alt="@lotargo/memory_plugin" width="800" style="max-width: 100%; border-radius: 12px; margin-bottom: 16px;">
4
-
5
- <br>
6
-
7
- <img src="./assets/title.svg" alt="@lotargo/memory_plugin" width="520" style="max-width: 100%; margin-bottom: 12px;">
8
-
9
- <br>
10
-
11
- [![npm version](https://img.shields.io/npm/v/@lotargo/memory_plugin)](https://www.npmjs.com/package/@lotargo/memory_plugin)
12
- [![npm downloads](https://img.shields.io/npm/dt/@lotargo/memory_plugin)](https://www.npmjs.com/package/@lotargo/memory_plugin)
13
- [![license](https://img.shields.io/badge/license-MIT-blue)](./LICENSE)
14
- [![node version](https://img.shields.io/badge/node-%3E%3D18.0.0-brightgreen)](https://nodejs.org)
15
- [![mcp](https://img.shields.io/badge/MCP-Supported-8A2BE2)](https://modelcontextprotocol.io)
16
- [![storage](https://img.shields.io/badge/Storage-100%25%20Local-success)](#storage--privacy)
17
-
18
- <br>
19
-
20
- **Zero-Docker Local Hybrid RAG Engine & Long-Term Memory for AI Coding Agents**
21
-
22
- Automatically remembers durable user facts, ingests complex document repositories, and performs high-precision hybrid retrieval across sessions and platforms.
23
-
24
- </div>
25
-
26
- ---
27
-
28
- ## Overview
29
-
30
- Standard AI coding assistants lose context as soon as a chat session closes or a conversation is reset. You end up repeatedly re-explaining your preferences, architectural decisions, coding style, or project conventions.
31
-
32
- `@lotargo/memory_plugin` gives your AI tools durable, **persistent**, 100% local long-term memory and document retrieval capabilities that persist across restarts and work seamlessly across all supported coding environments. Any LLM-based coding agent (OpenCode, Claude Code, Codex, Antigravity) can query its own memory and hybrid knowledge base via the **Model Context Protocol (MCP)**.
33
-
34
- > **Project Scope & Runtime Notes**:
35
- > `@lotargo/memory_plugin` is designed primarily as a practical plugin to expand capabilities and streamline daily interaction with AI coding tools. Benchmark scores in this repository represent internal synthetic evaluation runs and are not intended as generalized RAG benchmarks.
36
- >
37
- > **Hardware Acceleration**: GPU execution mode is an experimental feature and may vary in stability across different operating systems or models. For optimal stability and consistent runtime performance, using standard CPU mode with `multilingual-e5-small` or `multilingual-e5-base` is recommended.
38
-
39
- ### Practical Use Cases
40
-
41
- - **Architectural Decisions**: _"In this project, we use Fastify instead of Express and strict schema validation via Zod."_
42
- - **Coding Conventions**: _"Place all helper utilities inside `src/utils/` and cover new functions with Vitest tests."_
43
- - **Environment Constraints**: _"Our target deployment environment is Node.js 20 on AWS Lambda."_
44
- - **User Profile & Tone**: _"My name is Alex. I prefer concise, direct answers without conversational filler."_
45
-
46
- ---
47
-
48
- ## Quick Start
49
-
50
- ### Minimum System Requirements
51
-
52
- - **Node.js**: `18.0.0` or higher
53
- - **Package Manager**: `npm` / `npx` (included with Node.js)
54
- - **Supported Environment**: OpenCode, Antigravity / Gemini CLI, Claude Code, or Codex
55
-
56
- ### Installation
57
-
58
- Run the setup command to configure all detected AI environments automatically:
59
-
60
- ```bash
61
- # Recommended: Global installation & setup (works across local CLI, Docker, and CI)
62
- npm install -g @lotargo/memory_plugin && memory_plugin setup
63
-
64
- # Or via npx
65
- npx @lotargo/memory_plugin setup
66
- ```
67
-
68
- To target a specific environment:
69
-
70
- ```bash
71
- # Antigravity / Gemini CLI
72
- npm install -g @lotargo/memory_plugin && memory_plugin setup --antigravity
73
-
74
- # OpenCode
75
- npm install -g @lotargo/memory_plugin && memory_plugin setup --opencode
76
-
77
- # Claude Code
78
- npm install -g @lotargo/memory_plugin && memory_plugin setup --claude
79
-
80
- # Codex
81
- npm install -g @lotargo/memory_plugin && memory_plugin setup --codex
82
- ```
83
-
84
- `setup` also accepts `--gemini` (alias for Antigravity) and `--local` (also registers the MCP server in the project-local `.agents/` directory for Antigravity). Without a specific flag, all detected environments are configured.
85
-
86
- ---
87
-
88
- ## Dual-Layer Architecture
89
-
90
- 1. **Layer 1: Notebook Store (Durable Facts)**
91
- - **Tools**: `remember`, `recall`, `forget`, `update_fact`, `memory_info`
92
- - **Scope**: User preferences, identity, project conventions, system rules.
93
- - **Storage**: Human-readable Markdown format (`global` and per-project stores).
94
- - **Performance**: Guaranteed 100% precision instant lookup without vector degradation or threshold filtering.
95
- - **Project Scoping**: Project stores are bound to their canonical directory path (e.g. `f__projects_plugins_memory.md`), so identical project names in different folders never collide. Legacy basename stores are migrated automatically with a collision guard.
96
-
97
- 2. **Layer 2: RAG Knowledge Base (Technical Documents & Codebases)**
98
- - **Tools**: `ingest_document`, `query_knowledge_base`, `manage_knowledge_base`, `link_knowledge`
99
- - **Capabilities**: Ingests raw text files, Markdown, HTML, Web URLs, and full code repositories.
100
- - **Engine Components**: 3-tier hierarchy chunking (Big / Medium / Small), SQLite FTS5 BM25 search, ONNX dense vector embeddings (`multilingual-e5-small`), Reciprocal Rank Fusion (RRF / RSF), cross-encoder reranking (optional), and GraphRAG Lite code symbol extraction.
101
-
102
- ---
103
-
104
- ## Key Features
105
-
106
- - **Zero Heavy Infrastructure**: No Docker, no Python server, no C++ compilation (`node-gyp`). Uses Node.js native SQLite database.
107
- - **Bilingual & Multilingual Support**: State-of-the-art semantic precision across Russian, English, and technical code symbols.
108
- - **3-Tier Hierarchy Chunking**: Document (Big) -> Section (Medium) -> Micro-Chunk (Small).
109
- - **Hybrid RRF/RSF Fusion**: Combines SQLite FTS5 keyword precision with ONNX dense vector similarity; lexical-only fallback when embeddings are disabled.
110
- - **Semantic Search**: Cosine-similarity vector retrieval with multilingual ONNX embeddings (E5 / BGE model families).
111
- - **Path-Based Project Memory**: Per-project stores keyed by canonical directory path, with automatic migration of legacy stores.
112
- - **GraphRAG Lite**: Automatically links documents and extracted code symbols (classes, functions, types).
113
- - **Memory-to-Knowledge Linking**: Associate notebook facts with specific documents or line ranges in the RAG base.
114
- - **Content-Addressable Storage (CAS)**: Local S3-style compressed blob store for raw original documents.
115
- - **Dual-Source Model Failover**: Automatic HuggingFace CDN model downloading with GitHub Repository Mirror fallback.
116
- - **Interactive TUI**: Terminal GUI (CLI menu) for runtime engine tuning, snapshot export/import, model cache management, and diagnostics.
117
-
118
- ---
119
-
120
- ## Supported Platforms
121
-
122
- | Platform | Status | Configuration Mechanism |
123
- | :--------------------------- | :----------- | :-------------------------------------------------------------------------- |
124
- | **Antigravity / Gemini CLI** | Supported | MCP Server (`~/.gemini/config/mcp_config.json` & `.agents/mcp_config.json`) |
125
- | **OpenCode** | Native | Native plugin + MCP Server (`~/.config/opencode/opencode.json`) |
126
- | **Claude Code** | Supported | MCP Server (`~/.claude.json`) |
127
- | **Codex** | Supported | MCP Server (`~/.codex/config.toml`) |
128
- | **Google Jules** | Experimental | MCP Server via global install + setup (`npm install -g @lotargo/memory_plugin && memory_plugin setup`) |
129
-
130
- ### Google Jules Integration (Experimental)
131
-
132
- The plugin has been verified inside the **Google Jules** cloud workspace environment. This feature is **experimental**.
133
-
134
- - **Setup Method**: Global pre-installation with auto-setup:
135
- ```bash
136
- npm install -g @lotargo/memory_plugin && memory_plugin setup
137
- ```
138
- - **Verification**: All current tools and capabilities have been verified inside the Google Jules cloud workspace. Google Jules automatically discovers the registered MCP server upon workspace initialization and seamlessly interacts with the full set of memory & RAG tools — `remember`, `recall`, `forget`, `update_fact`, `memory_info`, `link_knowledge`, `ingest_document`, `query_knowledge_base`, and `manage_knowledge_base` — including project-scoped memory, knowledge linking, and snapshot export/import.
139
- - **Current Limitation**: All memory stores and vector indexes operate locally within the workspace environment. Cross-session cloud synchronization across different Jules runs is planned for upcoming releases.
140
-
141
- ---
142
-
143
- ## Available MCP Tools
144
-
145
- ### 1. Memory Tools (Key-Value Notebook)
146
-
147
- | Tool | Scope / Target | Description |
148
- | :-------------- | :-------------------------------------- | :---------------------------------------------------------------- |
149
- | `remember` | `global` or `project` | Save an important durable fact or preference |
150
- | `recall` | `project`, `global`, `all`, `list_projects` | Display saved facts; read another project's store via `project: '<path>'` |
151
- | `forget` | Index ID, range, or query | Remove a saved fact (e.g. `"3-30"` ranges; `force` for protected) |
152
- | `update_fact` | Index ID, metadata id, or text | Rewrite a fact while preserving its original date and links |
153
- | `memory_info` | - | Show storage paths, fact counts, RAG stats, and package version |
154
-
155
- ### 2. Hybrid RAG Knowledge Base Tools
156
-
157
- | Tool | Target | Description |
158
- | :---------------------- | :------------------------------ | :--------------------------------------------------------------------------- |
159
- | `ingest_document` | Local files, Web URLs, Raw text | Ingest into 3-tier index with ONNX vector embeddings & symbol extraction |
160
- | `query_knowledge_base` | Text / Code query | Perform hybrid RSF/RRF search (BM25 + Vector) to retrieve candidate sections |
161
- | `manage_knowledge_base` | Actions / Documents | Stats, list, read, delete documents, or export/import snapshots |
162
- | `link_knowledge` | Facts + Document ranges | Explicitly link a memory fact to a KB document or line range |
163
-
164
- ### 3. Native OpenCode Plugin
165
-
166
- When installed as an OpenCode plugin, all MCP tools above plus `list-mcp-tools` and `mcp-reminder` are exposed. A chat hook (`experimental.chat.messages.transform`) automatically injects your saved memory into every conversation as a `<MEMORY>` block, so your agent starts each session already knowing your preferences and project context.
167
-
168
- ---
169
-
170
- ## GraphRAG Lite
171
-
172
- The RAG engine includes a lightweight graph layer built on the same SQLite database. It combines code symbol extraction, hierarchy edges, and explicit memory-to-document links without requiring a separate graph store or an LLM at ingest time.
173
-
174
- **Code Symbol Extraction** — during `ingest_document`, code symbols are extracted from the chunk content using fast regex heuristics (no language model needed):
175
-
176
- - JavaScript / TypeScript: `function`, `class`, `interface`, `type`, `enum`, `const`, `let`, `var`
177
- - Python: `def`, `class`
178
- - Symbols shorter than 3 characters and reserved keywords (`const`, `let`, `var`, `function`, `class`, `import`, `export`, `from`, `return`, `if`, `for`, `while`, `def`, `self`) are filtered out.
179
-
180
- **Graph Edges** three built-in relation types are created automatically, and custom relation types are supported for explicit linking:
181
-
182
- | Relation Type | Direction / Example |
183
- | :------------------ | :----------------------------------------------------------- |
184
- | `CONTAINS` | Document -> Section -> Micro-Chunk (3-tier hierarchy) |
185
- | `DEFINES_SYMBOL` | Section -> `symbol:<name>` (extracted code symbol) |
186
- | `LINKS_TO` (default) | Memory fact -> Document or line range (via `link_knowledge`) |
187
-
188
- **Memory-to-Knowledge Linking** the `link_knowledge` tool connects a notebook fact to a specific document or line range (`RULES_FOR`, `IMPLEMENTS`, `EXPLAINS`, `REFERENCES`, ...):
189
-
190
- - `link`create the link and its graph edge
191
- - `list_links` — list all links, optionally filtered by fact key
192
- - `get_doc_links` list all links pointing to a given document
193
-
194
- Linked facts are surfaced automatically in `recall` results as `🔗 [Linked Docs: ...]`, and `remember` accepts an optional `docId` to link immediately.
195
-
196
- **Retrieval Integration** — `query_knowledge_base` augments each retrieved section with `defined_symbols`: the code symbols defined in that same section (a single-hop lookup along `DEFINES_SYMBOL` edges). Symbol extraction also improves BM25 scoring, since symbol names become searchable tokens.
197
-
198
- **Lifecycle** — edges are rebuilt transactionally on re-ingest of the same document, and `manage_knowledge_base` delete operations clean up all graph edges and knowledge links owned by the document (including `GLOB`-matched section/micro-chunk suffixes).
199
-
200
- ---
201
-
202
- ## Interactive TUI (CLI Menu)
203
-
204
- Launch the interactive terminal UI to manage engine settings, inspect databases, tune retrieval parameters, and run diagnostics:
205
-
206
- ```bash
207
- # From local repository folder:
208
- node mcp-server/index.js cli
209
- # or
210
- npx . cli
211
-
212
- # If installed / linked globally:
213
- memory_plugin cli
214
- # or
215
- memory-cli
216
- ```
217
-
218
- ### TUI Menu Overview
219
-
220
- The interactive menu exposes runtime parameters that `hybridQuery` honors, allowing search behavior modifications without restarting the MCP server. Use **Up / Down** arrows to navigate, **ENTER** to select, and **BACKSPACE** to go back.
221
-
222
- | Block | Menu Item | Functionality |
223
- | :-------------------------------------- | :------------------------------ | :----------------------------------------------------------------------------- |
224
- | **Engine & Hybrid Search Settings** | Fusion Algorithm | Switch between `rsf`, `rrf`, `semantic_only`, `lexical_only`. |
225
- | | RSF Alpha Balance | Weight of semantic over lexical in `rsf` fusion (`α ∈ [0,1]`). Default: `0.5`. |
226
- | | Embedding Model | Select ONNX model (e.g. `Xenova/multilingual-e5-small`, custom HF models). |
227
- | | Reranker Model | Enable Cross-Encoder reranking or disable for zero-latency fusion. |
228
- | | Vector Batch Size | Ingestion vector batch size `[1 - 256]` (default `12`). |
229
- | | GPU Attention Budget | GPU micro-batch attention budget `[1M - 16M]` (default `2.0M`, ~1.5 GB VRAM). |
230
- | | CPU WASM Threads | ONNX WASM threads: `0` auto-detect or `1-16`. |
231
- | | Execution Hardware | `cpu` or `webgpu` (experimental). |
232
- | **Knowledge Base & Storage Management** | Notebook (Layer 1 Facts) | Browse and manage `global` and per-project `.md` fact stores. |
233
- | | RAG Docs (Layer 2 Base) | List ingested documents, inspect chunk counts, and purge entries. |
234
- | | Snapshot Export / Import | Export or restore the full RAG base + blob store as a JSON snapshot. |
235
- | | Manage & Purge ML Model Cache | Inspect or purge downloaded ONNX model weights. |
236
- | | Hard Reset | Purge RAG base, blob storage, and graph edges. |
237
- | **Global Prompt & Integration** | Enable / Disable Global Prompt | Inject memory instructions into `~/.gemini/config/AGENTS.md`, `~/.codex/AGENTS.md`, `~/.claude/CLAUDE.md`. |
238
- | **Diagnostics & System Actions** | Search Quality Benchmark | Execute in-process search evaluation across the benchmark query set. |
239
- | | Verification Query | Run a test `hybridQuery` against the active index. |
240
- | | Clear Benchmark Corpus Cache | Clear cached benchmark corpus. |
241
- | | Reset Config to Factory Defaults| Restore default engine configuration. |
242
-
243
- Settings persist to `<memory-dir>/config.json` and are immediately loaded by the MCP server.
244
-
245
- ---
246
-
247
- ## Configuration
248
-
249
- The engine is configured through `<memory-dir>/config.json` (created with defaults on first run):
250
-
251
- | Key | Default | Description |
252
- | :-------------------- | :--------------------------------- | :-------------------------------------------------------------- |
253
- | `fusionAlgorithm` | `rsf` | `rsf`, `rrf`, `semantic_only`, or `lexical_only` |
254
- | `alpha` | `0.5` | Vector vs BM25 weight in RSF `[0.0 - 1.0]` |
255
- | `embeddingModel` | `Xenova/multilingual-e5-small` | ONNX dense embedding model (E5 / BGE families supported) |
256
- | `rerankerModel` | `none` | Cross-encoder reranker, or `Xenova/bge-reranker-base` |
257
- | `rerankerEnabled` | `false` | Enable cross-encoder re-ranking |
258
- | `batchSize` | `12` | Ingestion vector batch size `[1 - 256]` |
259
- | `gpuAttentionBudget` | `2000000` | GPU micro-batch attention budget `[1M - 16M]` |
260
- | `onnxThreads` | `0` | ONNX WASM threads: `0` auto-detect, or `1-16` |
261
- | `executionDevice` | `cpu` | `cpu` or `webgpu` (experimental) |
262
-
263
- ---
264
-
265
- ## Testing & Benchmarking
266
-
267
- To run the automated test suite and benchmarks locally:
268
-
269
- ```bash
270
- cd mcp-server
271
-
272
- # Run unit and integration tests
273
- npm test
274
-
275
- # Run search quality & ingestion benchmarks
276
- npm run benchmark
277
- ```
278
-
279
- ### Benchmark Methodology
280
-
281
- The benchmark suite (`mcp-server/benchmarks/`) evaluates retrieval quality across three phases:
282
-
283
- 1. **Dual-Layer Verification**: Asserts Notebook and RAG layers are isolated (zero crosstalk, 100% precision on `recall`).
284
- 2. **Ingestion Benchmark**: Ingests test documents with ONNX `multilingual-e5-small` embeddings, reporting throughput, DB size, CAS blob footprint, and heap delta.
285
- 3. **Search Quality Benchmark**: Evaluates cross-lingual and code-keyword queries against 4 retrieval strategies with bootstrap 95% CIs, paired t-tests, and hyperparameter sweeps over RSF $\alpha$ and RRF $k$.
286
-
287
- ### Search Quality Results (Smoke Test)
288
-
289
- _Note: The following metrics reflect a quick smoke-test evaluation run performed on a reduced subset of documents to verify retrieval logic precision._
290
-
291
- Evaluated across a reduced document subset using Mean Reciprocal Rank (MRR@5), Recall@5, and Normalized Discounted Cumulative Gain (NDCG@5):
292
-
293
- | Retrieval Strategy | MRR@5 | Recall@5 | NDCG@5 |
294
- | :---------------------------- | :--------: | :---------: | :--------: |
295
- | BM25 Lexical Search Only | 0.6706 | 76.19% | 0.6934 |
296
- | Dense ONNX Vector Only | 0.8135 | 100.00% | 0.8612 |
297
- | Hybrid RRF ($k=10$) | 0.8810 | 95.24% | 0.8997 |
298
- | **Hybrid RSF ($\alpha=0.5$)** | **0.9286** | **100.00%** | **0.9473** |
299
-
300
- For complete methodology details, see [`docs/BENCHMARKS.md`](./docs/BENCHMARKS.md).
301
-
302
- ---
303
-
304
- ## Documentation & Reports
305
-
306
- Detailed technical documentation and architectural specifications are available in the [`docs/`](./docs) directory:
307
-
308
- - [**Verification Report (`MEMORY_PLUGIN_REPORT.md`)**](./docs/MEMORY_PLUGIN_REPORT.md): Summary report covering MCP Tool Registry, JSON-RPC integration testing, layer isolation validation, and search precision.
309
- - [**Comprehensive Technical Report (`MEMORY_PLUGIN_COMPREHENSIVE_REPORT.md`)**](./docs/MEMORY_PLUGIN_COMPREHENSIVE_REPORT.md): Scientific analysis of system architecture, dual-layer model, hardware environment specifications, mathematical search formulations, and event-loop profiling.
310
- - [**Benchmark Methodology & Guide (`BENCHMARKS.md`)**](./docs/BENCHMARKS.md): Guide to automated benchmark execution, hyperparameter sweeps (RSF $\alpha$, RRF $k$), search quality metrics, and performance tracking across releases.
311
-
312
- ---
313
-
314
- ## Storage & Privacy
315
-
316
- - **100% Local Storage**: All SQLite indexes, ONNX models, CAS blobs, and Markdown notebooks are stored locally in the memory directory. The location resolves to, in order of priority: `$MEMORY_DIR`, `$OPENCODE_CONFIG_DIR/memory`, the legacy `~/.config/opencode/memory` (on Windows: `%LOCALAPPDATA%\opencode\memory`), or `$XDG_CONFIG_HOME/opencode/memory`.
317
- - **Dual-Source Failover Model Fetching**: Primary model weights are fetched from HuggingFace CDN with automatic failover to GitHub Repository Mirror.
318
- - **Zero External Telemetry**: No third-party network calls are required after initial model setup.
319
-
320
- ---
321
-
322
- ## License
323
-
324
- [MIT](./LICENSE)
1
+ <div align="center">
2
+
3
+ <img src="./assets/hero.jpg" alt="@lotargo/memory_plugin" width="800" style="max-width: 100%; border-radius: 12px; margin-bottom: 16px;">
4
+
5
+ <br>
6
+
7
+ <img src="./assets/title.svg" alt="@lotargo/memory_plugin" width="520" style="max-width: 100%; margin-bottom: 12px;">
8
+
9
+ <br>
10
+
11
+ [![npm version](https://img.shields.io/npm/v/@lotargo/memory_plugin)](https://www.npmjs.com/package/@lotargo/memory_plugin)
12
+ [![npm downloads](https://img.shields.io/npm/dt/@lotargo/memory_plugin)](https://www.npmjs.com/package/@lotargo/memory_plugin)
13
+ [![license](https://img.shields.io/badge/license-MIT-blue)](./LICENSE)
14
+ [![node version](https://img.shields.io/badge/node-%3E%3D18.0.0-brightgreen)](https://nodejs.org)
15
+ [![mcp](https://img.shields.io/badge/MCP-Supported-8A2BE2)](https://modelcontextprotocol.io)
16
+ [![storage](https://img.shields.io/badge/Storage-100%25%20Local-success)](#storage--privacy)
17
+
18
+ <br>
19
+
20
+ **Zero-Docker Local Hybrid RAG Engine & Long-Term Memory for AI Coding Agents**
21
+
22
+ Automatically remembers durable user facts, ingests complex document repositories, and performs high-precision hybrid retrieval across sessions and platforms.
23
+
24
+ </div>
25
+
26
+ ---
27
+
28
+ ## Overview
29
+
30
+ Standard AI coding assistants lose context as soon as a chat session closes or a conversation is reset. You end up repeatedly re-explaining your preferences, architectural decisions, coding style, or project conventions.
31
+
32
+ `@lotargo/memory_plugin` gives your AI tools durable, **persistent**, 100% local long-term memory and document retrieval capabilities that persist across restarts and work seamlessly across all supported coding environments. Any LLM-based coding agent (OpenCode, Claude Code, Codex, Antigravity) can query its own memory and hybrid knowledge base via the **Model Context Protocol (MCP)**.
33
+
34
+ > **Project Scope & Runtime Notes**:
35
+ > `@lotargo/memory_plugin` is designed primarily as a practical plugin to expand capabilities and streamline daily interaction with AI coding tools. Benchmark scores in this repository represent internal synthetic evaluation runs and are not intended as generalized RAG benchmarks.
36
+ >
37
+ > **Hardware Acceleration**: GPU execution mode is an experimental feature and may vary in stability across different operating systems or models. For optimal stability and consistent runtime performance, using standard CPU mode with `multilingual-e5-small` or `multilingual-e5-base` is recommended.
38
+
39
+ ### Practical Use Cases
40
+
41
+ - **Architectural Decisions**: _"In this project, we use Fastify instead of Express and strict schema validation via Zod."_
42
+ - **Coding Conventions**: _"Place all helper utilities inside `src/utils/` and cover new functions with Vitest tests."_
43
+ - **Environment Constraints**: _"Our target deployment environment is Node.js 20 on AWS Lambda."_
44
+ - **User Profile & Tone**: _"My name is Alex. I prefer concise, direct answers without conversational filler."_
45
+
46
+ ---
47
+
48
+ ## Quick Start
49
+
50
+ ### Minimum System Requirements
51
+
52
+ - **Node.js**: `18.0.0` or higher
53
+ - **Package Manager**: `npm` / `npx` (included with Node.js)
54
+ - **Supported Environment**: OpenCode, Antigravity / Gemini CLI, Claude Code, or Codex
55
+
56
+ ### Installation
57
+
58
+ Run the setup command to configure all detected AI environments automatically:
59
+
60
+ ```bash
61
+ # Recommended: Global installation & setup (works across local CLI, Docker, and CI)
62
+ npm install -g @lotargo/memory_plugin && memory_plugin setup
63
+
64
+ # Or via npx
65
+ npx @lotargo/memory_plugin setup
66
+ ```
67
+
68
+ To target a specific environment:
69
+
70
+ ```bash
71
+ # Antigravity / Gemini CLI
72
+ npm install -g @lotargo/memory_plugin && memory_plugin setup --antigravity
73
+
74
+ # OpenCode
75
+ npm install -g @lotargo/memory_plugin && memory_plugin setup --opencode
76
+
77
+ # Claude Code
78
+ npm install -g @lotargo/memory_plugin && memory_plugin setup --claude
79
+
80
+ # Codex
81
+ npm install -g @lotargo/memory_plugin && memory_plugin setup --codex
82
+ ```
83
+
84
+ `setup` also accepts `--gemini` (alias for Antigravity) and `--local` (also registers the MCP server in the project-local `.agents/` directory for Antigravity). Without a specific flag, all detected environments are configured.
85
+
86
+ ---
87
+
88
+ ## Dual-Layer Architecture
89
+
90
+ 1. **Layer 1: Notebook Store (Durable Facts)**
91
+ - **Tools**: `remember`, `recall`, `forget`, `update_fact`, `memory_info`
92
+ - **Scope**: User preferences, identity, project conventions, system rules.
93
+ - **Storage**: Human-readable Markdown format (`global` and per-project stores).
94
+ - **Performance**: Guaranteed 100% precision instant lookup without vector degradation or threshold filtering.
95
+ - **Project Scoping**: Project stores are bound to their canonical directory path (e.g. `f__projects_plugins_memory.md`), so identical project names in different folders never collide. Legacy basename stores are migrated automatically with a collision guard.
96
+
97
+ 2. **Layer 2: RAG Knowledge Base (Technical Documents & Codebases)**
98
+ - **Tools**: `ingest_document`, `query_knowledge_base`, `manage_knowledge_base`, `link_knowledge`
99
+ - **Capabilities**: Ingests raw text files, Markdown, HTML, Web URLs, and full code repositories.
100
+ - **Engine Components**: 3-tier hierarchy chunking (Big / Medium / Small), SQLite FTS5 BM25 search, ONNX dense vector embeddings (`multilingual-e5-small`), Reciprocal Rank Fusion (RRF / RSF), cross-encoder reranking (optional), and GraphRAG Lite code symbol extraction.
101
+
102
+ ---
103
+
104
+ ## Key Features
105
+
106
+ - **Zero Heavy Infrastructure**: No Docker, no Python server, no C++ compilation (`node-gyp`). Uses Node.js native SQLite database.
107
+ - **Bilingual & Multilingual Support**: State-of-the-art semantic precision across Russian, English, and technical code symbols.
108
+ - **Multilingual Code Symbol Parsing**: High-performance extraction of code entities across 10 programming languages (Python, Go, Rust, C++, Java, Kotlin, C#, PHP, Ruby, JS/TS).
109
+ - **Office Document Ingestion**: Native, pure-JS parsing of PDF, DOCX, XLSX, and CSV documents, removing the need for external CLI converters.
110
+ - **Hybrid Spreadsheet RAG Representation**: XLSX/CSV tables are converted to Markdown tables for raw document viewing, while row records are transformed into semantic key-value text lines to prevent vector database noise and boost search quality.
111
+ - **3-Tier Hierarchy Chunking**: Document (Big) -> Section (Medium) -> Micro-Chunk (Small).
112
+ - **Hybrid RRF/RSF Fusion**: Combines SQLite FTS5 keyword precision with ONNX dense vector similarity; lexical-only fallback when embeddings are disabled.
113
+ - **Semantic Search**: Cosine-similarity vector retrieval with multilingual ONNX embeddings (E5 / BGE model families).
114
+ - **Path-Based Project Memory**: Per-project stores keyed by canonical directory path, with automatic migration of legacy stores.
115
+ - **GraphRAG Lite**: Automatically links documents and extracted code symbols (classes, functions, types).
116
+ - **Memory-to-Knowledge Linking**: Associate notebook facts with specific documents or line ranges in the RAG base.
117
+ - **Content-Addressable Storage (CAS)**: Local S3-style compressed blob store for raw original documents.
118
+ - **Dual-Source Model Failover**: Automatic HuggingFace CDN model downloading with GitHub Repository Mirror fallback.
119
+ - **Interactive TUI**: Terminal GUI (CLI menu) for runtime engine tuning, snapshot export/import, model cache management, and diagnostics.
120
+
121
+ ---
122
+
123
+ ## Supported Platforms
124
+
125
+ | Platform | Status | Configuration Mechanism |
126
+ | :--------------------------- | :----------- | :-------------------------------------------------------------------------- |
127
+ | **Antigravity / Gemini CLI** | Supported | MCP Server (`~/.gemini/config/mcp_config.json` & `.agents/mcp_config.json`) |
128
+ | **OpenCode** | Native | Native plugin + MCP Server (`~/.config/opencode/opencode.json`) |
129
+ | **Claude Code** | Supported | MCP Server (`~/.claude.json`) |
130
+ | **Codex** | Supported | MCP Server (`~/.codex/config.toml`) |
131
+ | **Google Jules** | Experimental | MCP Server via global install + setup (`npm install -g @lotargo/memory_plugin && memory_plugin setup`) |
132
+
133
+ ### Google Jules Integration (Experimental)
134
+
135
+ The plugin has been verified inside the **Google Jules** cloud workspace environment. This feature is **experimental**.
136
+
137
+ - **Setup Method**: Global pre-installation with auto-setup:
138
+ ```bash
139
+ npm install -g @lotargo/memory_plugin && memory_plugin setup
140
+ ```
141
+ - **Verification**: All current tools and capabilities have been verified inside the Google Jules cloud workspace. Google Jules automatically discovers the registered MCP server upon workspace initialization and seamlessly interacts with the full set of memory & RAG tools — `remember`, `recall`, `forget`, `update_fact`, `memory_info`, `link_knowledge`, `ingest_document`, `query_knowledge_base`, and `manage_knowledge_base` — including project-scoped memory, knowledge linking, and snapshot export/import.
142
+ - **Current Limitation**: All memory stores and vector indexes operate locally within the workspace environment. Cross-session cloud synchronization across different Jules runs is planned for upcoming releases.
143
+
144
+ ---
145
+
146
+ ## Available MCP Tools
147
+
148
+ ### 1. Memory Tools (Key-Value Notebook)
149
+
150
+ | Tool | Scope / Target | Description |
151
+ | :-------------- | :-------------------------------------- | :---------------------------------------------------------------- |
152
+ | `remember` | `global` or `project` | Save an important durable fact or preference |
153
+ | `recall` | `project`, `global`, `all`, `list_projects` | Display saved facts; read another project's store via `project: '<path>'` |
154
+ | `forget` | Index ID, range, or query | Remove a saved fact (e.g. `"3-30"` ranges; `force` for protected) |
155
+ | `update_fact` | Index ID, metadata id, or text | Rewrite a fact while preserving its original date and links |
156
+ | `memory_info` | - | Show storage paths, fact counts, RAG stats, and package version |
157
+
158
+ ### 2. Hybrid RAG Knowledge Base Tools
159
+
160
+ | Tool | Target | Description |
161
+ | :---------------------- | :------------------------------ | :--------------------------------------------------------------------------- |
162
+ | `ingest_document` | Local files, Web URLs, Raw text | Ingest into 3-tier index with ONNX vector embeddings & symbol extraction |
163
+ | `query_knowledge_base` | Text / Code query | Perform hybrid RSF/RRF search (BM25 + Vector) to retrieve candidate sections |
164
+ | `manage_knowledge_base` | Actions / Documents | Stats, list, read, delete documents, or export/import snapshots |
165
+ | `link_knowledge` | Facts + Document ranges | Explicitly link a memory fact to a KB document or line range |
166
+
167
+ ### 3. Native OpenCode Plugin
168
+
169
+ When installed as an OpenCode plugin, all MCP tools above plus `list-mcp-tools` and `mcp-reminder` are exposed. A chat hook (`experimental.chat.messages.transform`) automatically injects your saved memory into every conversation as a `<MEMORY>` block, so your agent starts each session already knowing your preferences and project context.
170
+
171
+ ---
172
+
173
+ ## GraphRAG Lite
174
+
175
+ The RAG engine includes a lightweight graph layer built on the same SQLite database. It combines code symbol extraction, hierarchy edges, and explicit memory-to-document links without requiring a separate graph store or an LLM at ingest time.
176
+
177
+ **Code Symbol Extraction** — during `ingest_document`, code symbols are extracted from the chunk content using fast, highly-optimized regex heuristics (maintaining 100% portability and avoiding heavy binary parsers):
178
+
179
+ - **JavaScript / TypeScript**: `function`, `class`, `interface`, `type`, `enum`, `const`, `let`, `var`
180
+ - **Python**: `def`, `class`
181
+ - **Go**: `struct`, `interface`, `func` (including methods with receivers)
182
+ - **Rust**: `struct`, `enum`, `trait`, `fn` (including async/pub)
183
+ - **C++**: `class`, `struct`, `namespace`, functions and methods
184
+ - **Java & Kotlin**: `class`, `interface`, `record`, `enum`, `fun` and synchronized methods
185
+ - **C#**: `class`, `interface`, `struct`, `record`, methods and properties
186
+ - **PHP**: `class`, `interface`, `trait`, functions
187
+ - **Ruby**: `module`, `class`, methods
188
+ - Standard language keywords and symbols shorter than 3 characters are automatically filtered out using a comprehensive, cross-language ignored keyword list to prevent graph clutter.
189
+
190
+ **Graph Edges**three built-in relation types are created automatically, and custom relation types are supported for explicit linking:
191
+
192
+ | Relation Type | Direction / Example |
193
+ | :------------------ | :----------------------------------------------------------- |
194
+ | `CONTAINS` | Document -> Section -> Micro-Chunk (3-tier hierarchy) |
195
+ | `DEFINES_SYMBOL` | Section -> `symbol:<name>` (extracted code symbol) |
196
+ | `LINKS_TO` (default) | Memory fact -> Document or line range (via `link_knowledge`) |
197
+
198
+ **Memory-to-Knowledge Linking** — the `link_knowledge` tool connects a notebook fact to a specific document or line range (`RULES_FOR`, `IMPLEMENTS`, `EXPLAINS`, `REFERENCES`, ...):
199
+
200
+ - `link` — create the link and its graph edge
201
+ - `list_links` — list all links, optionally filtered by fact key
202
+ - `get_doc_links` list all links pointing to a given document
203
+
204
+ Linked facts are surfaced automatically in `recall` results as `🔗 [Linked Docs: ...]`, and `remember` accepts an optional `docId` to link immediately.
205
+
206
+ **Retrieval Integration** — `query_knowledge_base` augments each retrieved section with `defined_symbols`: the code symbols defined in that same section (a single-hop lookup along `DEFINES_SYMBOL` edges). Symbol extraction also improves BM25 scoring, since symbol names become searchable tokens.
207
+
208
+ **Lifecycle** — edges are rebuilt transactionally on re-ingest of the same document, and `manage_knowledge_base` delete operations clean up all graph edges and knowledge links owned by the document (including `GLOB`-matched section/micro-chunk suffixes).
209
+
210
+ ---
211
+
212
+ ## Interactive TUI (CLI Menu)
213
+
214
+ Launch the interactive terminal UI to manage engine settings, inspect databases, tune retrieval parameters, and run diagnostics:
215
+
216
+ ```bash
217
+ # From local repository folder:
218
+ node mcp-server/index.js cli
219
+ # or
220
+ npx . cli
221
+
222
+ # If installed / linked globally:
223
+ memory_plugin cli
224
+ # or
225
+ memory-cli
226
+ ```
227
+
228
+ ### TUI Menu Overview
229
+
230
+ The interactive menu exposes runtime parameters that `hybridQuery` honors, allowing search behavior modifications without restarting the MCP server. Use **Up / Down** arrows to navigate, **ENTER** to select, and **BACKSPACE** to go back.
231
+
232
+ | Block | Menu Item | Functionality |
233
+ | :-------------------------------------- | :------------------------------ | :----------------------------------------------------------------------------- |
234
+ | **Engine & Hybrid Search Settings** | Fusion Algorithm | Switch between `rsf`, `rrf`, `semantic_only`, `lexical_only`. |
235
+ | | RSF Alpha Balance | Weight of semantic over lexical in `rsf` fusion (`α ∈ [0,1]`). Default: `0.5`. |
236
+ | | Embedding Model | Select ONNX model (e.g. `Xenova/multilingual-e5-small`, custom HF models). |
237
+ | | Reranker Model | Enable Cross-Encoder reranking or disable for zero-latency fusion. |
238
+ | | Vector Batch Size | Ingestion vector batch size `[1 - 256]` (default `12`). |
239
+ | | GPU Attention Budget | GPU micro-batch attention budget `[1M - 16M]` (default `2.0M`, ~1.5 GB VRAM). |
240
+ | | CPU WASM Threads | ONNX WASM threads: `0` auto-detect or `1-16`. |
241
+ | | Execution Hardware | `cpu` or `webgpu` (experimental). |
242
+ | **Knowledge Base & Storage Management** | Notebook (Layer 1 Facts) | Browse and manage `global` and per-project `.md` fact stores. |
243
+ | | RAG Docs (Layer 2 Base) | List ingested documents, inspect chunk counts, and purge entries. |
244
+ | | Snapshot Export / Import | Export or restore the full RAG base + blob store as a JSON snapshot. |
245
+ | | Manage & Purge ML Model Cache | Inspect or purge downloaded ONNX model weights. |
246
+ | | Hard Reset | Purge RAG base, blob storage, and graph edges. |
247
+ | **Global Prompt & Integration** | Enable / Disable Global Prompt | Inject memory instructions into `~/.gemini/config/AGENTS.md`, `~/.codex/AGENTS.md`, `~/.claude/CLAUDE.md`. |
248
+ | **Diagnostics & System Actions** | Search Quality Benchmark | Execute in-process search evaluation across the benchmark query set. |
249
+ | | Verification Query | Run a test `hybridQuery` against the active index. |
250
+ | | Clear Benchmark Corpus Cache | Clear cached benchmark corpus. |
251
+ | | Reset Config to Factory Defaults| Restore default engine configuration. |
252
+
253
+ Settings persist to `<memory-dir>/config.json` and are immediately loaded by the MCP server.
254
+
255
+ ---
256
+
257
+ ## Configuration
258
+
259
+ The engine is configured through `<memory-dir>/config.json` (created with defaults on first run):
260
+
261
+ | Key | Default | Description |
262
+ | :-------------------- | :--------------------------------- | :-------------------------------------------------------------- |
263
+ | `fusionAlgorithm` | `rsf` | `rsf`, `rrf`, `semantic_only`, or `lexical_only` |
264
+ | `alpha` | `0.5` | Vector vs BM25 weight in RSF `[0.0 - 1.0]` |
265
+ | `embeddingModel` | `Xenova/multilingual-e5-small` | ONNX dense embedding model (E5 / BGE families supported) |
266
+ | `rerankerModel` | `none` | Cross-encoder reranker, or `Xenova/bge-reranker-base` |
267
+ | `rerankerEnabled` | `false` | Enable cross-encoder re-ranking |
268
+ | `batchSize` | `12` | Ingestion vector batch size `[1 - 256]` |
269
+ | `gpuAttentionBudget` | `2000000` | GPU micro-batch attention budget `[1M - 16M]` |
270
+ | `onnxThreads` | `0` | ONNX WASM threads: `0` auto-detect, or `1-16` |
271
+ | `executionDevice` | `cpu` | `cpu` or `webgpu` (experimental) |
272
+
273
+ ---
274
+
275
+ ## Testing & Benchmarking
276
+
277
+ To run the automated test suite and benchmarks locally:
278
+
279
+ ```bash
280
+ cd mcp-server
281
+
282
+ # Run unit and integration tests
283
+ npm test
284
+
285
+ # Run search quality & ingestion benchmarks
286
+ npm run benchmark
287
+ ```
288
+
289
+ ### Benchmark Methodology
290
+
291
+ The benchmark suite (`mcp-server/benchmarks/`) evaluates retrieval quality across three phases:
292
+
293
+ 1. **Dual-Layer Verification**: Asserts Notebook and RAG layers are isolated (zero crosstalk, 100% precision on `recall`).
294
+ 2. **Ingestion Benchmark**: Ingests test documents with ONNX `multilingual-e5-small` embeddings, reporting throughput, DB size, CAS blob footprint, and heap delta.
295
+ 3. **Search Quality Benchmark**: Evaluates cross-lingual and code-keyword queries against 4 retrieval strategies with bootstrap 95% CIs, paired t-tests, and hyperparameter sweeps over RSF $\alpha$ and RRF $k$.
296
+
297
+ ### Search Quality Results (Smoke Test)
298
+
299
+ _Note: The following metrics reflect a quick smoke-test evaluation run performed on a reduced subset of documents to verify retrieval logic precision._
300
+
301
+ Evaluated across a reduced document subset using Mean Reciprocal Rank (MRR@5), Recall@5, and Normalized Discounted Cumulative Gain (NDCG@5):
302
+
303
+ | Retrieval Strategy | MRR@5 | Recall@5 | NDCG@5 |
304
+ | :---------------------------- | :--------: | :---------: | :--------: |
305
+ | BM25 Lexical Search Only | 0.6706 | 76.19% | 0.6934 |
306
+ | Dense ONNX Vector Only | 0.8135 | 100.00% | 0.8612 |
307
+ | Hybrid RRF ($k=10$) | 0.8810 | 95.24% | 0.8997 |
308
+ | **Hybrid RSF ($\alpha=0.5$)** | **0.9286** | **100.00%** | **0.9473** |
309
+
310
+ For complete methodology details, see [`docs/BENCHMARKS.md`](./docs/BENCHMARKS.md).
311
+
312
+ ---
313
+
314
+ ## Documentation & Reports
315
+
316
+ Detailed technical documentation and architectural specifications are available in the [`docs/`](./docs) directory:
317
+
318
+ - [**Verification Report (`MEMORY_PLUGIN_REPORT.md`)**](./docs/MEMORY_PLUGIN_REPORT.md): Summary report covering MCP Tool Registry, JSON-RPC integration testing, layer isolation validation, and search precision.
319
+ - [**Comprehensive Technical Report (`MEMORY_PLUGIN_COMPREHENSIVE_REPORT.md`)**](./docs/MEMORY_PLUGIN_COMPREHENSIVE_REPORT.md): Scientific analysis of system architecture, dual-layer model, hardware environment specifications, mathematical search formulations, and event-loop profiling.
320
+ - [**Benchmark Methodology & Guide (`BENCHMARKS.md`)**](./docs/BENCHMARKS.md): Guide to automated benchmark execution, hyperparameter sweeps (RSF $\alpha$, RRF $k$), search quality metrics, and performance tracking across releases.
321
+
322
+ ---
323
+
324
+ ## Storage & Privacy
325
+
326
+ - **100% Local Storage**: All SQLite indexes, ONNX models, CAS blobs, and Markdown notebooks are stored locally in the memory directory. The location resolves to, in order of priority: `$MEMORY_DIR`, `$OPENCODE_CONFIG_DIR/memory`, the legacy `~/.config/opencode/memory` (on Windows: `%LOCALAPPDATA%\opencode\memory`), or `$XDG_CONFIG_HOME/opencode/memory`.
327
+ - **Dual-Source Failover Model Fetching**: Primary model weights are fetched from HuggingFace CDN with automatic failover to GitHub Repository Mirror.
328
+ - **Zero External Telemetry**: No third-party network calls are required after initial model setup.
329
+
330
+ ---
331
+
332
+ ## License
333
+
334
+ [MIT](./LICENSE)