@icex-labs/openclaw-memory-engine 3.5.1 → 3.5.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/README.md +256 -91
  2. package/package.json +1 -1
package/README.md CHANGED
@@ -15,14 +15,15 @@ OpenClaw agents wake up fresh every session. Without persistent memory, they for
15
15
 
16
16
  ## The Solution
17
17
 
18
- Two-tier memory inspired by [MemGPT/Letta](https://github.com/cpacker/MemGPT):
18
+ Five-layer memory architecture inspired by [MemGPT/Letta](https://github.com/cpacker/MemGPT):
19
19
 
20
- - **Core Memory** (~500 tokens) — user identity, relationship, preferences. Always loaded.
21
- - **Archival Memory** (unlimited) — facts, decisions, events. Retrieved on demand via hybrid semantic search.
20
+ 1. **Core Memory** (~500 tokens) — identity, relationship, preferences. Always loaded.
21
+ 2. **Archival Memory** (unlimited) — facts with importance scoring. Hybrid semantic search.
22
+ 3. **Knowledge Graph** — entity relations. "Who is my doctor?" → graph traversal.
23
+ 4. **Episodic Memory** — conversation summaries. "What did we discuss last time?"
24
+ 5. **Reflective Memory** — behavioral pattern analysis. "What topics dominate this week?"
22
25
 
23
- Plus: knowledge graph, episodic memory, behavioral reflection, importance scoring with forgetting curves, deduplication, SQLite backend, and a browsable HTML dashboard.
24
-
25
- The agent manages all of this autonomously.
26
+ The agent manages all five layers autonomously using 19 purpose-built tools.
26
27
 
27
28
  ---
28
29
 
@@ -34,50 +35,126 @@ bash ~/.openclaw/extensions/memory-engine/setup.sh
34
35
  openclaw gateway restart
35
36
  ```
36
37
 
37
- `setup.sh` handles everything:
38
- - Interactive core memory setup (prompts for your name, location, role, etc.)
39
- - Configures `openclaw.json`
40
- - Installs daily maintenance scheduler (macOS LaunchAgent / Linux systemd / Windows schtasks)
41
- - Patches agent instructions (AGENTS.md)
42
- - Registers 4 automated cron jobs (reflection, consolidation, dedup, dashboard)
43
- - `--non-interactive` flag available for scripted installs
38
+ ### What setup.sh does
39
+
40
+ 1. **Interactive core memory setup** — prompts for name, location, role, relationship dynamic
41
+ 2. **Legacy data migration** detects existing MEMORY.md / daily logs and imports them into archival (with dedup)
42
+ 3. **Platform scheduler** installs daily maintenance (macOS LaunchAgent / Linux systemd / Windows schtasks)
43
+ 4. **Config patching** enables plugin in `openclaw.json`
44
+ 5. **Agent instructions** patches AGENTS.md with memory tool guide
45
+ 6. **Cron registration** — 4 automated jobs (reflection, consolidation, dedup, dashboard)
46
+ 7. **Embedding backfill** — on next gateway restart, missing embeddings are auto-computed in background
47
+
48
+ `--non-interactive` flag available for scripted installs.
44
49
 
45
50
  ---
46
51
 
47
52
  ## Architecture
48
53
 
54
+ ### Memory Layers
55
+
49
56
  ```
50
- ┌──────────────────────────────────────────────────────────┐
51
- Agent Context Window
52
-
53
- core_memory_read() ──→ core.json (~500 tokens)
54
-
55
- archival_search("query") ──→ archival.jsonl (unlimited)
56
- keyword (2×) + embedding (5×) + recency + decay
57
-
58
- graph_query("entity") ──→ graph.jsonl (relations)
59
- "who is my doctor?" traverse knowledge graph
60
-
61
- episode_recall("topic") ──→ episodes.jsonl
62
- "what did we discuss last time?" conversation recall
63
-
64
- memory_reflect() ──→ behavioral pattern analysis
65
- memory_dashboard() ──→ browsable HTML report
66
- memory_export() ──→ full backup for migration
67
- └──────────────────────────────────────────────────────────┘
57
+ ┌──────────────────────────────────────────────────────────────────┐
58
+ Agent Context Window
59
+
60
+ ┌─ Layer 1: Core Memory ─────────────────────────────────────┐
61
+ core_memory_read() → core.json (~500 tokens) │ │
62
+ │ Identity, relationship, preferences, current_focus │
63
+ │ Agent reads on session start, updates atomically
64
+ └────────────────────────────────────────────────────────────┘
65
+
66
+ ┌─ Layer 2: Archival Memory ─────────────────────────────────┐
67
+ archival_insert(fact, entity, tags, importance) │ │
68
+ │ archival_search(query) hybrid 5-signal ranking
69
+ │ Unlimited JSONL. Each record: content + entity + tags + │
70
+ importance (1-10) + access tracking + embedding vector │ │
71
+ └────────────────────────────────────────────────────────────┘
72
+
73
+ ┌─ Layer 3: Knowledge Graph ─────────────────────────────────┐
74
+ │ │ graph_query(entity, relation?, depth?) │ │
75
+ │ │ Triple store: (subject, relation, object) │ │
76
+ │ │ Auto-extracted from archival_insert content │ │
77
+ │ │ "who is my doctor?" → User→has_doctor→Dr. Smith │ │
78
+ │ └────────────────────────────────────────────────────────────┘ │
79
+ │ │
80
+ │ ┌─ Layer 4: Episodic Memory ─────────────────────────────────┐ │
81
+ │ │ episode_save(summary, decisions, mood, topics) │ │
82
+ │ │ episode_recall(query) → hybrid search over conversations │ │
83
+ │ │ "what did we discuss about the car?" → full context │ │
84
+ │ └────────────────────────────────────────────────────────────┘ │
85
+ │ │
86
+ │ ┌─ Layer 5: Reflective Memory ───────────────────────────────┐ │
87
+ │ │ memory_reflect(window_days) → pattern analysis report │ │
88
+ │ │ Topic trends, time distribution, mood shifts, │ │
89
+ │ │ neglected entities, forgetting candidates │ │
90
+ │ └────────────────────────────────────────────────────────────┘ │
91
+ └──────────────────────────────────────────────────────────────────┘
68
92
  ```
69
93
 
70
- ### Multi-Agent Support
94
+ ### Data Flow
71
95
 
72
- Each agent automatically gets its own memory based on its configured workspace. Uses OpenClaw's session key to resolve the correct workspace at tool registration time — zero configuration needed.
96
+ ```
97
+ User says something important
98
+
99
+
100
+ archival_insert(content, entity, tags, importance)
101
+
102
+ ├──→ archival.jsonl (append fact)
103
+ ├──→ graph.jsonl (auto-extract relations)
104
+ └──→ embeddings cache (background, async)
105
+
106
+ User asks a question
107
+
108
+
109
+ archival_search(query)
110
+
111
+ ├──→ Keyword matching (2× weight per term)
112
+ ├──→ Embedding cosine similarity (5× weight)
113
+ ├──→ Recency boost (0-1, decays over 1 year)
114
+ ├──→ Access frequency boost (0-0.5)
115
+ └──→ Importance × forgetting curve (0.5× weight)
116
+
117
+ End of conversation
118
+
119
+
120
+ episode_save(summary, decisions, mood)
121
+
122
+ └──→ episodes.jsonl + embedding
123
+
124
+ Gateway restart
125
+
126
+
127
+ Auto-detect missing embeddings across ALL workspaces
128
+
129
+ └──→ Batch backfill (100/batch, 200ms rate limit, crash-safe)
130
+ ```
73
131
 
74
- Privacy flag: `"sharing": false` in plugin config for multi-user setups.
132
+ ### Multi-Agent Isolation
133
+
134
+ ```
135
+ openclaw.json
136
+ agents:
137
+ ├── main → workspace/ ← Agent A's memory (2700+ records)
138
+ ├── wife → workspace-wife/ ← Agent B's memory (300+ records)
139
+ └── discord → workspace/ ← Shares Agent A's memory
140
+
141
+ Plugin uses ToolFactory pattern:
142
+ ctx.sessionKey = "agent:wife:telegram:..."
143
+
144
+ extractAgentId() → "wife"
145
+
146
+ openclaw.json lookup → workspace-wife/
147
+
148
+ All 19 tools bound to correct workspace via closure
149
+ ```
150
+
151
+ Each agent's tools are bound to its own workspace at registration time. No cross-contamination. Privacy flag `"sharing": false` available for multi-user setups.
75
152
 
76
153
  ---
77
154
 
78
155
  ## Tools (19)
79
156
 
80
- ### Core Memory — Identity
157
+ ### Core Memory — Identity (3)
81
158
 
82
159
  | Tool | Description |
83
160
  |------|-------------|
@@ -85,46 +162,48 @@ Privacy flag: `"sharing": false` in plugin config for multi-user setups.
85
162
  | `core_memory_replace` | Update a field by dot-path (e.g., `user.location`). Auto-parses JSON strings. 3KB hard limit. |
86
163
  | `core_memory_append` | Append to an array field (e.g., `current_focus`). |
87
164
 
88
- ### Archival Memory — Facts
165
+ ### Archival Memory — Facts (5)
89
166
 
90
167
  | Tool | Description |
91
168
  |------|-------------|
92
- | `archival_insert` | Store a fact with entity, tags, and importance (1-10). Auto-extracts knowledge graph triples. |
93
- | `archival_search` | Hybrid search: keyword + semantic + recency + access decay + importance. |
94
- | `archival_update` | Correct an existing record by ID. |
95
- | `archival_delete` | Remove an outdated record. |
169
+ | `archival_insert` | Store a fact with entity, tags, and importance (1-10). Auto-extracts knowledge graph triples. Embedding computed in background. |
170
+ | `archival_search` | Hybrid 5-signal search: keyword + semantic + recency + access + importance. |
171
+ | `archival_update` | Correct an existing record by ID. Re-indexes embedding. |
172
+ | `archival_delete` | Remove an outdated record. Cleans embedding cache. |
96
173
  | `archival_stats` | Record count, entity/tag distribution, embedding coverage, storage size. |
97
174
 
98
- ### Knowledge Graph — Relations
175
+ ### Knowledge Graph — Relations (2)
99
176
 
100
177
  | Tool | Description |
101
178
  |------|-------------|
102
- | `graph_query` | Traverse from entity with depth control. |
103
- | `graph_add` | Manually add a relation triple. |
179
+ | `graph_query` | Traverse from entity with depth control. Answers relational questions. |
180
+ | `graph_add` | Manually add a relation triple `(subject, relation, object)`. |
181
+
182
+ Auto-extraction patterns on `archival_insert`: has_doctor, lives_in, has_condition, treated_by, owns, works_at, attends, price, has_lawyer, spouse, has_child.
104
183
 
105
- ### Episodic Memory — Conversations
184
+ ### Episodic Memory — Conversations (2)
106
185
 
107
186
  | Tool | Description |
108
187
  |------|-------------|
109
- | `episode_save` | Save conversation summary, decisions, mood, topics. |
110
- | `episode_recall` | Search past conversations by topic or get recent N. |
188
+ | `episode_save` | Save conversation summary with decisions, mood, topics, participants. |
189
+ | `episode_recall` | Search past conversations by topic/keyword, or get recent N. Hybrid search with embedding. |
111
190
 
112
- ### Intelligence
191
+ ### Intelligence (3)
113
192
 
114
193
  | Tool | Description |
115
194
  |------|-------------|
116
- | `memory_reflect` | Analyze behavioral patterns: topic trends, time distribution, mood shifts, forgetting candidates. |
117
- | `archival_deduplicate` | Find and remove near-duplicates via embedding cosine similarity. |
118
- | `memory_consolidate` | Extract structured facts from text. Sentence-level splitting (Chinese + English), entity inference, dedup. |
195
+ | `memory_reflect` | Statistical analysis: topic frequency, time-of-day distribution, mood trend, importance distribution, neglected entities, forgetting candidates. Configurable window (7/14/30 days). |
196
+ | `archival_deduplicate` | Find near-duplicates via embedding cosine similarity (≥0.92 threshold). Preview or auto-remove. |
197
+ | `memory_consolidate` | Extract structured facts from text blocks. Sentence-level splitting (Chinese + English), generic entity inference, keyword dedup against existing records. |
119
198
 
120
- ### Backup & Admin
199
+ ### Backup & Admin (4)
121
200
 
122
201
  | Tool | Description |
123
202
  |------|-------------|
124
- | `memory_export` | Full snapshot: core + archival + embeddings → JSON file. |
125
- | `memory_import` | Restore from snapshot. Merge or replace mode. |
126
- | `memory_migrate` | Migrate from JSONL to SQLite with FTS5 full-text search. |
127
- | `memory_dashboard` | Generate self-contained HTML dashboard. |
203
+ | `memory_export` | Full snapshot: core + archival + embeddings → single JSON file. Versioned format. |
204
+ | `memory_import` | Restore from snapshot. `merge` (add missing) or `replace` (overwrite) mode. |
205
+ | `memory_migrate` | Migrate from JSONL to SQLite with FTS5 full-text search. Preserves JSONL as backup. |
206
+ | `memory_dashboard` | Generate self-contained HTML dashboard: facts (searchable), graph, episodes, core memory, reflection report. Dark theme. |
128
207
 
129
208
  ---
130
209
 
@@ -135,12 +214,26 @@ Privacy flag: `"sharing": false` in plugin config for multi-user setups.
135
214
  | Signal | Weight | Description |
136
215
  |--------|--------|-------------|
137
216
  | Keyword | 2× per term | Term presence in content + entity + tags |
138
- | Semantic | 5× | Cosine similarity via OpenAI `text-embedding-3-small` (512d) |
139
- | Recency | 0–1 | Linear decay over 1 year |
140
- | Access | 0–0.5 | Boost for recently accessed records |
141
- | Importance | 0.5× | Weighted by forgetting curve: `importance × e^(-0.01 × days)` |
217
+ | Semantic | 5× | Cosine similarity via OpenAI `text-embedding-3-small` (512 dimensions) |
218
+ | Recency | 0–1 | Linear decay over 1 year from creation date |
219
+ | Access | 0–0.5 | Boost for recently accessed records (decays over 180 days) |
220
+ | Importance | 0.5× | Forgetting curve: `importance × e^(-0.01 × days_since_access)` |
142
221
 
143
- Falls back to keyword-only if no OpenAI key is configured. Cost with embeddings: ~$0.001/session.
222
+ Falls back to keyword-only if no OpenAI key is configured — no errors, just reduced quality. Cost with embeddings: ~$0.001/session for search, ~$0.02/1M tokens for batch indexing.
223
+
224
+ ---
225
+
226
+ ## Self-Healing
227
+
228
+ The plugin automatically detects and fixes issues on gateway restart:
229
+
230
+ | Issue | Auto-fix |
231
+ |-------|----------|
232
+ | Missing embeddings after migration | Batch backfill across all workspaces (100/batch, rate limited) |
233
+ | Core memory value serialized as JSON string | Auto-parse on `core_memory_replace` |
234
+ | Near-duplicate facts accumulating | Weekly `archival_deduplicate` cron |
235
+ | Daily logs piling up | Daily `memory-maintenance.sh` merges 7-day-old logs into weekly summaries |
236
+ | Stale `current_focus` items | Agent prompted to update via `memory_reflect` analysis |
144
237
 
145
238
  ---
146
239
 
@@ -165,23 +258,51 @@ Falls back to keyword-only if no OpenAI key is configured. Cost with embeddings:
165
258
 
166
259
  | Option | Default | Description |
167
260
  |--------|---------|-------------|
168
- | `workspace` | Auto-resolved | Override workspace directory |
261
+ | `workspace` | Auto-resolved per agent | Override workspace directory |
169
262
  | `coreSizeLimit` | `3072` (3KB) | Max core.json size |
170
263
  | `sharing` | `true` | Cross-agent memory sharing. Set `false` for multi-user privacy. |
171
264
 
172
- Semantic search requires `OPENAI_API_KEY` in environment (optional).
265
+ Semantic search requires `OPENAI_API_KEY` in environment (optional — graceful degradation).
173
266
 
174
267
  ---
175
268
 
176
269
  ## Automated Maintenance
177
270
 
178
- | Schedule | Job | What it does |
271
+ ### Cron Jobs (registered by setup.sh)
272
+
273
+ | Schedule | Job | Description |
179
274
  |----------|-----|-------------|
180
- | Daily 9:00am | Reflection | Analyze memory patterns, store observations |
181
- | Every 6h | Consolidation | Extract missed facts from daily logs |
182
- | Weekly Sunday | Deduplication | Clean near-duplicate records |
183
- | Daily 9:30am | Dashboard | Refresh browsable HTML report |
184
- | Daily 3:00am | File cleanup | Merge old logs into weekly summaries, archive old summaries |
275
+ | Daily 9:00am | `memory-reflect-daily` | Analyze patterns, store observations |
276
+ | Every 6h | `memory-consolidate-6h` | Extract missed facts from daily logs |
277
+ | Weekly Sunday 4am | `memory-dedup-weekly` | Clean near-duplicate records |
278
+ | Daily 9:30am | `memory-dashboard-daily` | Refresh browsable HTML dashboard |
279
+
280
+ ### File Maintenance (daily 3am)
281
+
282
+ | Action | Trigger | Description |
283
+ |--------|---------|-------------|
284
+ | Size check | MEMORY.md >4KB | Warn alert |
285
+ | Size check | MEMORY.md >5KB | Critical alert |
286
+ | Log merge | Daily logs >7 days old | Merge into weekly summaries |
287
+ | Archive | Weekly summaries >60 days | Move to archive/ |
288
+ | Topic check | Topic file >8KB | Warn alert |
289
+
290
+ Alerts written to `memory/maintenance-alerts.json`, checked by agent during heartbeats.
291
+
292
+ ---
293
+
294
+ ## Storage Files
295
+
296
+ | File | Purpose | Growth | Format |
297
+ |------|---------|--------|--------|
298
+ | `memory/core.json` | Identity block | Fixed ~1-3KB | JSON |
299
+ | `memory/archival.jsonl` | Fact storage | Grows with usage | JSONL |
300
+ | `memory/graph.jsonl` | Knowledge graph | Grows with relations | JSONL |
301
+ | `memory/episodes.jsonl` | Conversation summaries | Grows per conversation | JSONL |
302
+ | `memory/archival.embeddings.json` | Embedding vectors | ~2KB per record | JSON |
303
+ | `memory/memory.sqlite` | SQLite index (optional) | After `memory_migrate` | SQLite |
304
+ | `memory/dashboard.html` | Browsable report | Regenerated daily | HTML |
305
+ | `memory/maintenance-alerts.json` | Health alerts | Regenerated daily | JSON |
185
306
 
186
307
  ---
187
308
 
@@ -189,41 +310,85 @@ Semantic search requires `OPENAI_API_KEY` in environment (optional).
189
310
 
190
311
  ```
191
312
  memory-engine/
192
- ├── index.js # Plugin entry — 19 tools (factory pattern)
313
+ ├── index.js # Plugin entry — 19 tools via ToolFactory pattern
193
314
  ├── lib/
194
- │ ├── paths.js # Constants, workspace resolution
195
- │ ├── core.js # Core memory CRUD + auto-parse
196
- │ ├── archival.js # JSONL storage + in-memory cache
197
- │ ├── embedding.js # OpenAI embedding API + cache
198
- │ ├── search.js # Hybrid 5-signal search
199
- │ ├── graph.js # Knowledge graph: triples + traversal
200
- │ ├── episodes.js # Episodic memory: save + recall
201
- │ ├── reflection.js # Statistical pattern analysis
202
- │ ├── consolidate.js # Text → facts extraction
203
- │ ├── dedup.js # Embedding similarity dedup
204
- │ ├── backup.js # Export / import
205
- │ ├── store-sqlite.js # SQLite backend (FTS5)
206
- │ └── dashboard.js # HTML dashboard generator
315
+ │ ├── paths.js # Constants, multi-workspace resolution, agent mapping
316
+ │ ├── core.js # Core memory CRUD + dot-path + auto-parse
317
+ │ ├── archival.js # JSONL storage + in-memory cache
318
+ │ ├── embedding.js # OpenAI embedding API + cache + batch backfill
319
+ │ ├── search.js # Hybrid 5-signal search with forgetting curve
320
+ │ ├── graph.js # Knowledge graph: triple store + traversal + auto-extract
321
+ │ ├── episodes.js # Episodic memory: save + hybrid recall
322
+ │ ├── reflection.js # Statistical pattern analysis (8 dimensions)
323
+ │ ├── consolidate.js # Text → structured facts (sentence split + entity inference)
324
+ │ ├── dedup.js # Embedding cosine similarity deduplication
325
+ │ ├── backup.js # Export / import with format versioning
326
+ │ ├── store-sqlite.js # SQLite backend with FTS5 + WAL mode
327
+ │ └── dashboard.js # Self-contained HTML dashboard generator
207
328
  ├── extras/
208
- │ ├── memory-maintenance.sh
329
+ │ ├── memory-maintenance.sh # Daily file cleanup script
330
+ │ ├── migrate-legacy.mjs # Standalone legacy data migration tool
209
331
  │ └── auto-consolidation-crons.json
210
- ├── setup.sh # One-command install
211
- ├── .claude/CLAUDE.md # Dev guide
212
- ├── ROADMAP.md
213
- ├── openclaw.plugin.json
332
+ ├── setup.sh # One-command install (interactive, cross-platform)
333
+ ├── .claude/CLAUDE.md # Development guide for Claude Code
334
+ ├── ROADMAP.md # Planned features
335
+ ├── openclaw.plugin.json # Plugin manifest with config schema
214
336
  └── package.json
215
337
  ```
216
338
 
339
+ ---
340
+
341
+ ## Migrating from File-Based Memory
342
+
343
+ If you already have MEMORY.md and daily log files, `setup.sh` handles migration automatically:
344
+
345
+ ```
346
+ 📦 Found 15 legacy memory files.
347
+ Migrate into archival memory? [Y/n]: Y
348
+ MEMORY.md: +87 facts
349
+ 2026-03-28.md: +42 facts
350
+ 2026-03-30.md: +29 facts
351
+ ✅ Migration complete: 158 facts imported, 3 skipped
352
+ ```
353
+
354
+ Manual migration: `node ~/.openclaw/extensions/memory-engine/extras/migrate-legacy.mjs [workspace_path]`
355
+
356
+ After migration, embeddings are auto-computed on next gateway restart (no manual step needed).
357
+
358
+ ---
359
+
217
360
  ## Platforms
218
361
 
219
- | Platform | Scheduler | Status |
220
- |----------|----------|--------|
362
+ | Platform | Maintenance Scheduler | Status |
363
+ |----------|----------------------|--------|
221
364
  | macOS | LaunchAgent | Full support |
222
- | Linux | systemd timer | Full support |
365
+ | Linux | systemd user timer | Full support |
223
366
  | Windows | schtasks | Guided setup |
224
367
 
225
368
  ---
226
369
 
370
+ ## Roadmap
371
+
372
+ See [ROADMAP.md](ROADMAP.md) for details.
373
+
374
+ - [x] Core memory with size guard + auto-parse
375
+ - [x] Archival CRUD + hybrid 5-signal search
376
+ - [x] Knowledge graph with auto-extraction
377
+ - [x] Episodic memory with conversation recall
378
+ - [x] Importance scoring + forgetting curves
379
+ - [x] Behavioral reflection + auto-consolidation
380
+ - [x] SQLite backend (FTS5) + HTML dashboard
381
+ - [x] Multi-workspace isolation (ToolFactory pattern)
382
+ - [x] Legacy data migration + embedding auto-backfill
383
+ - [x] Cross-platform support (macOS / Linux / Windows)
384
+ - [ ] LanceDB vector-native backend
385
+ - [ ] Memory importance auto-rating via LLM
386
+ - [ ] Web dashboard served via gateway HTTP route
387
+
388
+ ---
389
+
227
390
  ## License
228
391
 
229
- MIT — Built for [OpenClaw](https://openclaw.ai). Inspired by [MemGPT/Letta](https://github.com/cpacker/MemGPT).
392
+ MIT
393
+
394
+ Built for [OpenClaw](https://openclaw.ai). Inspired by [MemGPT/Letta](https://github.com/cpacker/MemGPT).
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@icex-labs/openclaw-memory-engine",
3
- "version": "3.5.1",
3
+ "version": "3.5.2",
4
4
  "description": "MemGPT-style hierarchical memory plugin for OpenClaw — core memory block + archival storage with semantic search",
5
5
  "type": "module",
6
6
  "main": "index.js",