@icex-labs/openclaw-memory-engine 3.3.1

package/README.md

@@ -0,0 +1,254 @@
+# @icex-labs/openclaw-memory-engine
+
+> Give your AI agent a brain that survives restarts.
+
+[![npm](https://img.shields.io/npm/v/@icex-labs/openclaw-memory-engine)](https://www.npmjs.com/package/@icex-labs/openclaw-memory-engine)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
+
+A [MemGPT](https://github.com/cpacker/MemGPT)-inspired memory plugin for [OpenClaw](https://openclaw.ai). Your agent gets 12 tools to manage its own memory — what to remember, what to recall, what to forget.
+
+**The problem:** OpenClaw agents wake up fresh every session. Without persistent memory, they forget who you are.
+
+**The fix:** Two-tier memory architecture:
+- **Core Memory** (~500 tokens) — identity, relationship, preferences. Always loaded.
+- **Archival Memory** (unlimited) — facts, decisions, events. Retrieved on demand via hybrid semantic search.
+
+The agent manages both tiers autonomously using purpose-built tools.
+
+---
+
+## Install
+
+```bash
+git clone git@github.com:icex-labs/openclaw-memory-engine.git ~/.openclaw/extensions/memory-engine
+bash ~/.openclaw/extensions/memory-engine/setup.sh
+nano ~/.openclaw/workspace/memory/core.json   # fill in your info
+openclaw gateway restart
+```
+
+Or from npm:
+```bash
+npm install -g @icex-labs/openclaw-memory-engine
+```
+
+`setup.sh` handles everything: enables the plugin in `openclaw.json`, creates template files, installs the daily maintenance cron, and patches your agent's instructions.
+
+---
+
+## How It Works
+
+```
+┌──────────────────────────────────────────────────────┐
+│                 Agent Context Window                  │
+│                                                       │
+│   Session start → core_memory_read()                  │
+│                   └─→ core.json (~500 tokens)         │
+│                                                       │
+│   "Where does Alice's doctor work?"                  │
+│   → archival_search("doctor")                         │
+│     └─→ keyword match + embedding similarity          │
+│         + recency boost + access frequency             │
+│         → "Dr. Smith, City Medical..."    │
+│                                                       │
+│   Alice says something new                           │
+│   → archival_insert(fact, entity, tags)               │
+│     └─→ archival.jsonl + background embedding         │
+│                                                       │
+│   End of conversation                                 │
+│   → memory_consolidate(summary)                       │
+│     └─→ split sentences → infer entities → dedup      │
+│         → batch insert                                │
+└──────────────────────────────────────────────────────┘
+```
+
+---
+
+## Tools (12)
+
+### Core Memory — Your Identity
+
+| Tool | What it does |
+|------|-------------|
+| `core_memory_read` | Load the identity block. Call every session start. |
+| `core_memory_replace` | Update a field by dot-path (`user.location`, `current_focus`). Auto-parses JSON strings. 3KB hard limit. |
+| `core_memory_append` | Append to an array field (`current_focus`). Creates array if needed. |
+
+Core memory lives in `memory/core.json`:
+
+```json
+{
+  "user": { "name": "Alice", "location": "New York", "language": "bilingual" },
+  "relationship": { "dynamic": "intimate companion", "trust": "deep" },
+  "preferences": { "config_rule": "don't touch openclaw.json" },
+  "current_focus": ["quant trading", "immigration case"]
+}
+```
+
+### Archival Memory — Your Long-Term Storage
+
+| Tool | What it does |
+|------|-------------|
+| `archival_insert` | Store a fact. Tags it with `entity` + `tags`. Computes embedding in background. |
+| `archival_search` | Hybrid search: keyword (2×) + semantic similarity (5×) + recency (0-1) + access decay (0-0.5). |
+| `archival_update` | Correct an existing record by ID. Re-indexes embedding. |
+| `archival_delete` | Remove an outdated record. Cleans up embedding cache. |
+| `archival_stats` | Dashboard: record count, embedding coverage, entity/tag distribution, storage size. |
+
+Each record in `memory/archival.jsonl`:
+
+```json
+{"id":"arch-17120-abc","ts":"2026-04-01","content":"Alice's doctor is Dr. Smith","entity":"Alice","tags":["health"],"access_count":3}
+```
+
+### Maintenance — Keep It Clean
+
+| Tool | What it does |
+|------|-------------|
+| `archival_deduplicate` | Find near-duplicates via embedding cosine similarity (≥0.92). Preview or auto-remove. |
+| `memory_consolidate` | Extract facts from text blocks. Splits by sentence (中文/English), infers entity, deduplicates, batch inserts. |
+
+### Backup — Never Lose Your Memory
+
+| Tool | What it does |
+|------|-------------|
+| `memory_export` | Snapshot core + archival + embeddings → single JSON file. |
+| `memory_import` | Restore from snapshot. `merge` (add missing) or `replace` (overwrite all). |
+
+---
+
+## Search Quality
+
+`archival_search` uses four signals:
+
+| Signal | Weight | How |
+|--------|--------|-----|
+| Keyword | 2× per term | Term presence in content + entity + tags |
+| Semantic | 5× | Cosine similarity via OpenAI `text-embedding-3-small` (512d) |
+| Recency | 0–1 | Linear decay over 1 year |
+| Access | 0–0.5 | Boost for recently accessed records |
+
+Embeddings are computed on insert and cached in `archival.embeddings.json`. If no OpenAI key is available, search falls back to keyword-only — no errors, just lower quality.
+
+**Cost:** ~$0.02 per 1M tokens with `text-embedding-3-small`. A typical session with 10 inserts + 5 searches costs < $0.001.
+
+---
+
+## Configuration
+
+```json
+// openclaw.json
+{
+  "plugins": {
+    "allow": ["memory-engine"],
+    "entries": {
+      "memory-engine": {
+        "enabled": true,
+        "config": {
+          "workspace": "/path/to/workspace",
+          "coreSizeLimit": 3072
+        }
+      }
+    }
+  }
+}
+```
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `workspace` | Auto-resolved | Workspace directory path |
+| `coreSizeLimit` | `3072` (3KB) | Max bytes for core.json |
+
+**Requires:** `OPENAI_API_KEY` in environment for semantic search. Without it, keyword search still works.
+
+---
+
+## Agent Instructions
+
+Add to your `AGENTS.md` or system prompt (done automatically by `setup.sh`):
+
+```markdown
+## Every Session
+1. Call `core_memory_read` — load your identity
+2. When you learn something important → `archival_insert`
+3. When you need details → `archival_search` before guessing
+4. When facts change → `core_memory_replace`
+5. End of conversation → `memory_consolidate` with key points
+```
+
+---
+
+## Daily Maintenance
+
+`extras/memory-maintenance.sh` runs daily at 3am (installed as a LaunchAgent by `setup.sh`):
+
+- Checks core.json size (warns >4KB, critical >5KB)
+- Merges 7-day-old daily logs into weekly summaries
+- Archives 60-day-old weekly summaries
+- Alerts written to `memory/maintenance-alerts.json`
+
+---
+
+## Backup & Migration
+
+```bash
+# Export
+openclaw agent -m "memory_export"
+# → memory/export-2026-04-01.json
+
+# Import on new machine
+openclaw agent -m "memory_import input_path='path/to/export.json' mode='replace'"
+```
+
+---
+
+## Project Structure
+
+```
+memory-engine/
+├── index.js              # Plugin entry — tool registration only (250 lines)
+├── lib/
+│   ├── paths.js          # Constants + path resolution
+│   ├── core.js           # Core memory CRUD + dot-path + auto-parse
+│   ├── archival.js       # Archival JSONL CRUD + in-memory cache
+│   ├── embedding.js      # OpenAI embedding API + file cache
+│   ├── search.js         # Hybrid four-signal search
+│   ├── consolidate.js    # Text → structured facts extraction
+│   ├── dedup.js          # Embedding similarity dedup
+│   └── backup.js         # Export/import
+├── extras/
+│   └── memory-maintenance.sh
+├── setup.sh              # One-command install
+├── .claude/CLAUDE.md     # Dev guide for Claude Code
+├── package.json
+├── openclaw.plugin.json
+└── README.md
+```
+
+---
+
+## Roadmap
+
+- [x] Core memory with size guard and auto-parse
+- [x] Archival CRUD with in-memory index
+- [x] Hybrid search (keyword + embedding + recency + access decay)
+- [x] Auto-extract facts from text
+- [x] Embedding-based deduplication
+- [x] Full backup/restore
+- [x] Modular codebase (8 focused modules)
+- [ ] LanceDB / SQLite backend for 50K+ records
+- [ ] Cross-agent memory sharing
+- [ ] Scheduled auto-consolidation via OpenClaw cron
+- [ ] Memory importance scoring (agent rates memories 1-10)
+- [ ] Forgetting curve — auto-archive unaccessed memories after N days
+- [ ] ClawHub publishing
+- [ ] Web dashboard for memory browsing
+
+---
+
+## License
+
+MIT
+
+---
+
+Built for [OpenClaw](https://openclaw.ai). Inspired by [MemGPT/Letta](https://github.com/cpacker/MemGPT).

package/extras/auto-consolidation-crons.json

@@ -0,0 +1,37 @@
+{
+  "_comment": "OpenClaw cron jobs for memory auto-consolidation. Install via: openclaw cron create --from-file <this-file>",
+  "crons": [
+    {
+      "id": "memory-reflect-daily",
+      "schedule": "0 9 * * *",
+      "agent": "main",
+      "message": "Run memory_reflect with window_days=7. Review the report. If you notice any important patterns or observations about George's behavior/interests, store them via archival_insert with entity='reflection' tags=['reflection','pattern']. Do NOT output anything to the main chat — this is a background task.",
+      "model": "anthropic/claude-sonnet-4-6",
+      "description": "Daily reflection: analyze memory patterns and store observations"
+    },
+    {
+      "id": "memory-consolidate-6h",
+      "schedule": "0 */6 * * *",
+      "agent": "main",
+      "message": "Read today's memory/YYYY-MM-DD.md daily log file. If it has content that isn't already in archival memory, run memory_consolidate on it. Then run archival_stats to check health. Do NOT output anything to the main chat.",
+      "model": "anthropic/claude-sonnet-4-6",
+      "description": "Auto-consolidate: extract facts from daily logs every 6 hours"
+    },
+    {
+      "id": "memory-dedup-weekly",
+      "schedule": "0 4 * * 0",
+      "agent": "main",
+      "message": "Run archival_deduplicate with apply=true to clean up duplicate facts. Then run archival_stats. Do NOT output anything to the main chat.",
+      "model": "anthropic/claude-sonnet-4-6",
+      "description": "Weekly dedup: clean up near-duplicate archival records"
+    },
+    {
+      "id": "memory-dashboard-daily",
+      "schedule": "30 9 * * *",
+      "agent": "main",
+      "message": "Run memory_dashboard to regenerate the HTML dashboard. Do NOT output anything to the main chat.",
+      "model": "anthropic/claude-sonnet-4-6",
+      "description": "Daily dashboard refresh at 9:30am"
+    }
+  ]
+}

package/extras/memory-maintenance.sh

@@ -0,0 +1,176 @@
+#!/bin/bash
+# memory-maintenance.sh — Automated memory system maintenance
+# Replaces the old memory-cleanup.sh with a comprehensive maintenance system
+#
+# Runs daily at 3am via LaunchAgent
+# Actions:
+#   1. Health check: MEMORY.md size limits
+#   2. Daily logs: merge week-old dailies into weekly summaries
+#   3. Archive: move weekly summaries older than 60 days
+#   4. Topic files: warn if any topic file exceeds size limit
+#   5. Report: append to maintenance log
+
+set -euo pipefail
+
+WORKSPACE="${OPENCLAW_WORKSPACE:-$HOME/.openclaw/workspace}"
+MEMORY_DIR="$WORKSPACE/memory"
+TOPICS_DIR="$MEMORY_DIR/topics"
+WEEKLY_DIR="$MEMORY_DIR/weekly"
+ARCHIVE_DIR="$MEMORY_DIR/archive"
+LOG_FILE="$MEMORY_DIR/maintenance.log"
+ALERT_FILE="$MEMORY_DIR/maintenance-alerts.json"
+
+MEMORY_MD="$WORKSPACE/MEMORY.md"
+DATE=$(date +%Y-%m-%d)
+TIMESTAMP=$(date +%Y-%m-%dT%H:%M:%S)
+
+# Limits (bytes)
+MEMORY_MD_MAX=5120        # 5KB — hard limit for MEMORY.md
+MEMORY_MD_WARN=4096       # 4KB — warning threshold
+TOPIC_FILE_MAX=8192       # 8KB per topic file
+DAILY_RETAIN_DAYS=7       # Keep daily logs for 7 days
+WEEKLY_ARCHIVE_DAYS=60    # Archive weekly summaries after 60 days
+
+mkdir -p "$TOPICS_DIR" "$WEEKLY_DIR" "$ARCHIVE_DIR"
+
+# --- Helpers ---
+log() { echo "[$TIMESTAMP] $1" >> "$LOG_FILE"; }
+alert_json=""
+add_alert() {
+  local level="$1" msg="$2"
+  if [ -z "$alert_json" ]; then
+    alert_json="[{\"level\":\"$level\",\"msg\":\"$msg\",\"ts\":\"$TIMESTAMP\"}"
+  else
+    alert_json="$alert_json,{\"level\":\"$level\",\"msg\":\"$msg\",\"ts\":\"$TIMESTAMP\"}"
+  fi
+}
+
+# --- 1. MEMORY.md health check ---
+if [ -f "$MEMORY_MD" ]; then
+  mem_size=$(wc -c < "$MEMORY_MD" | tr -d ' ')
+  mem_lines=$(wc -l < "$MEMORY_MD" | tr -d ' ')
+  if [ "$mem_size" -gt "$MEMORY_MD_MAX" ]; then
+    add_alert "critical" "MEMORY.md is ${mem_size}B (>${MEMORY_MD_MAX}B limit). Needs trimming."
+    log "CRITICAL: MEMORY.md ${mem_size}B exceeds ${MEMORY_MD_MAX}B limit (${mem_lines} lines)"
+  elif [ "$mem_size" -gt "$MEMORY_MD_WARN" ]; then
+    add_alert "warn" "MEMORY.md is ${mem_size}B (>${MEMORY_MD_WARN}B warning). Consider trimming."
+    log "WARN: MEMORY.md ${mem_size}B approaching limit (${mem_lines} lines)"
+  else
+    log "OK: MEMORY.md ${mem_size}B / ${mem_lines} lines"
+  fi
+else
+  add_alert "critical" "MEMORY.md not found!"
+  log "CRITICAL: MEMORY.md missing"
+fi
+
+# --- 2. Topic file health check ---
+if [ -d "$TOPICS_DIR" ]; then
+  for f in "$TOPICS_DIR"/*.md; do
+    [ -f "$f" ] || continue
+    fsize=$(wc -c < "$f" | tr -d ' ')
+    fname=$(basename "$f")
+    if [ "$fsize" -gt "$TOPIC_FILE_MAX" ]; then
+      add_alert "warn" "Topic ${fname} is ${fsize}B (>${TOPIC_FILE_MAX}B). Needs pruning."
+      log "WARN: Topic ${fname} ${fsize}B exceeds limit"
+    fi
+  done
+fi
+
+# --- 3. Merge old daily logs into weekly summaries ---
+# Find daily logs (YYYY-MM-DD.md) older than DAILY_RETAIN_DAYS
+daily_count=0
+merged_count=0
+for f in "$MEMORY_DIR"/2???-??-??.md; do
+  [ -f "$f" ] || continue
+  fname=$(basename "$f" .md)
+  # Parse date from filename
+  file_date="$fname"
+
+  # Calculate age in days using date arithmetic
+  file_epoch=$(date -j -f "%Y-%m-%d" "$file_date" "+%s" 2>/dev/null || echo 0)
+  now_epoch=$(date "+%s")
+  if [ "$file_epoch" -eq 0 ]; then
+    continue
+  fi
+  age_days=$(( (now_epoch - file_epoch) / 86400 ))
+
+  if [ "$age_days" -gt "$DAILY_RETAIN_DAYS" ]; then
+    # Determine ISO week: YYYY-Www
+    week_label=$(date -j -f "%Y-%m-%d" "$file_date" "+%G-W%V" 2>/dev/null || continue)
+    weekly_file="$WEEKLY_DIR/${week_label}.md"
+
+    # Append daily content to weekly file with date header
+    if [ ! -f "$weekly_file" ]; then
+      echo "# Weekly Summary: ${week_label}" > "$weekly_file"
+      echo "" >> "$weekly_file"
+    fi
+    echo "## ${file_date}" >> "$weekly_file"
+    echo "" >> "$weekly_file"
+    # Extract just the section headers and key bullet points (compress)
+    grep -E '^##|^- |^\*' "$f" >> "$weekly_file" 2>/dev/null || true
+    echo "" >> "$weekly_file"
+
+    # Move original to archive
+    mv "$f" "$ARCHIVE_DIR/"
+    merged_count=$((merged_count + 1))
+  fi
+  daily_count=$((daily_count + 1))
+done
+log "Daily logs: ${daily_count} active, ${merged_count} merged into weekly"
+
+# --- 4. Archive old weekly summaries ---
+archive_count=0
+for f in "$WEEKLY_DIR"/*.md; do
+  [ -f "$f" ] || continue
+  fmod=$(stat -f %m "$f" 2>/dev/null || echo 0)
+  now_epoch=$(date "+%s")
+  age_days=$(( (now_epoch - fmod) / 86400 ))
+  if [ "$age_days" -gt "$WEEKLY_ARCHIVE_DAYS" ]; then
+    mv "$f" "$ARCHIVE_DIR/"
+    archive_count=$((archive_count + 1))
+  fi
+done
+if [ "$archive_count" -gt 0 ]; then
+  log "Archived ${archive_count} weekly summaries (>${WEEKLY_ARCHIVE_DAYS} days old)"
+fi
+
+# --- 5. Count total memory footprint ---
+total_active=$(du -sh "$MEMORY_DIR" 2>/dev/null | cut -f1)
+total_archive=$(du -sh "$ARCHIVE_DIR" 2>/dev/null | cut -f1)
+topic_count=$(ls "$TOPICS_DIR"/*.md 2>/dev/null | wc -l | tr -d ' ')
+daily_active=$(ls "$MEMORY_DIR"/2???-??-??.md 2>/dev/null | wc -l | tr -d ' ')
+weekly_active=$(ls "$WEEKLY_DIR"/*.md 2>/dev/null | wc -l | tr -d ' ')
+log "Totals: active=${total_active} archive=${total_archive} topics=${topic_count} dailies=${daily_active} weeklies=${weekly_active}"
+
+# --- 6. Handle special-name daily logs (e.g., 2026-03-28-maren-stuck.md) ---
+for f in "$MEMORY_DIR"/2???-??-??-*.md; do
+  [ -f "$f" ] || continue
+  fname=$(basename "$f" .md)
+  file_date=$(echo "$fname" | grep -oE '^[0-9]{4}-[0-9]{2}-[0-9]{2}')
+  [ -z "$file_date" ] && continue
+
+  file_epoch=$(date -j -f "%Y-%m-%d" "$file_date" "+%s" 2>/dev/null || echo 0)
+  now_epoch=$(date "+%s")
+  [ "$file_epoch" -eq 0 ] && continue
+  age_days=$(( (now_epoch - file_epoch) / 86400 ))
+
+  if [ "$age_days" -gt "$DAILY_RETAIN_DAYS" ]; then
+    mv "$f" "$ARCHIVE_DIR/"
+    log "Archived special log: $(basename "$f") (${age_days} days old)"
+  fi
+done
+
+# --- 7. Write alerts file ---
+if [ -n "$alert_json" ]; then
+  echo "${alert_json}]" > "$ALERT_FILE"
+  log "Alerts written: $(echo "${alert_json}]" | grep -o '"level"' | wc -l | tr -d ' ') items"
+else
+  echo "[]" > "$ALERT_FILE"
+fi
+
+# --- 8. Trim maintenance log (keep last 90 entries) ---
+if [ -f "$LOG_FILE" ]; then
+  tail -270 "$LOG_FILE" > "${LOG_FILE}.tmp" && mv "${LOG_FILE}.tmp" "$LOG_FILE"
+fi
+
+log "Maintenance complete"