memory-lancedb-pro 1.0.24 → 1.1.0-beta.1

package/CHANGELOG-v1.1.0.md

@@ -0,0 +1,227 @@
+# memory-lancedb-pro v1.1.0 — 智能记忆增强
+
+> **日期**: 2026-03-03  
+> **作者**: CJY  
+> **概述**: 基于对 AI Agent 记忆系统的深入理解，对记忆的写入质量、生命周期管理和去重能力进行了全面改进与完善
+
+---
+
+## 一、改进动机
+
+原有记忆系统在**检索侧**表现优异（Vector+BM25 混合检索、cross-encoder 重排序、多维评分），但在以下方面存在提升空间：
+
+- **记忆写入质量**：依赖正则表达式触发捕获，容易漏捕有价值信息或误捕噪声
+- **记忆结构层次**：扁平文本存储，缺乏分层索引能力
+- **记忆生命周期**：简单时间衰减，无法模拟人类记忆的遗忘与强化规律
+- **去重能力**：仅基于向量相似度的粗粒度去重，缺乏语义级判断
+
+本次改进针对这三个维度进行了系统性增强。
+
+---
+
+## 二、变更摘要
+
+| 改进维度     | 核心变更                                  | 效果                               |
+| ------------ | ----------------------------------------- | ---------------------------------- |
+| 智能提取     | LLM 驱动的 6 类别提取 + L0/L1/L2 分层存储 | 记忆写入更精准、结构更丰富         |
+| 生命周期管理 | Weibull 衰减模型 + 三层晋升/降级          | 重要记忆持久保留，过时记忆自然淡化 |
+| 智能去重     | 向量预过滤 + LLM 语义决策                 | 避免冗余记忆，支持信息演化合并     |
+
+---
+
+## 三、新增文件
+
+### 1. `src/memory-categories.ts` — 6 类别分类系统
+
+设计了语义明确的记忆分类体系，将记忆分为两大类六小类：
+
+- **用户记忆**：`profile`（身份属性）、`preferences`（偏好习惯）、`entities`（持续存在的实体）、`events`（发生的事件）
+- **Agent 记忆**：`cases`（问题-解决方案对）、`patterns`（可复用的处理流程）
+
+每个类别有不同的合并策略：
+
+- `profile` → 始终合并（用户身份信息持续累积）
+- `preferences` / `entities` / `patterns` → 支持智能合并
+- `events` / `cases` → 仅新增或跳过（独立记录，保留历史完整性）
+
+---
+
+### 2. `src/llm-client.ts` — LLM 客户端
+
+封装了 LLM 调用接口，专注于结构化 JSON 输出：
+
+- 复用现有 OpenAI SDK 依赖，零新增包
+- 内置 JSON 容错解析：支持 markdown 代码块包裹和平衡大括号提取
+- 低温度 (0.1) 保证输出一致性
+- 30 秒超时保护，失败时优雅降级
+
+---
+
+### 3. `src/extraction-prompts.ts` — 记忆提取提示模板
+
+精心设计了 3 个提示模板：
+
+| 函数                      | 用途                                                |
+| ------------------------- | --------------------------------------------------- |
+| `buildExtractionPrompt()` | 从对话中提取 6 类别 L0/L1/L2 记忆，含 few-shot 示例 |
+| `buildDedupPrompt()`      | CREATE / MERGE / SKIP 去重决策                      |
+| `buildMergePrompt()`      | 将新旧记忆合并为三层结构                            |
+
+提取提示包含完整的记忆价值判断标准、类别决策逻辑表、常见混淆澄清规则和 6 个 few-shot 示例。
+
+---
+
+### 4. `src/smart-extractor.ts` — 智能提取管线
+
+实现了完整的 LLM 驱动提取流水线：
+
+```
+对话文本 → LLM 提取 → 候选记忆 → 向量去重 → LLM 决策 → 持久化
+```
+
+核心设计：
+
+- **两阶段去重**：先用向量相似度（阈值 0.7）快速筛选候选，再用 LLM 进行语义级判断
+- **类别感知合并**：不同类别应用不同合并策略
+- **L0/L1/L2 三层存储**：L0 一句话索引用于检索注入，L1 结构化摘要用于精读，L2 完整叙述用于深度回顾
+- **向后兼容**：新增的 6 类别自动映射到已有的 5 类别存储，L0/L1/L2 存储在 metadata JSON 中
+- **按类别设定重要度**：profile (0.9) > patterns (0.85) > cases/preferences (0.8) > entities (0.7) > events (0.6)
+
+---
+
+### 5. `src/decay-engine.ts` — Weibull 衰减引擎
+
+基于认知心理学中的记忆遗忘曲线研究，实现了复合衰减模型：
+
+**复合分数 = 时效权重 × 时效 + 频率权重 × 频率 + 内在权重 × 内在价值**
+
+三个分量：
+
+| 分量                     | 机制                              | 含义                   |
+| ------------------------ | --------------------------------- | ---------------------- |
+| **时效 (recency)**       | Weibull 拉伸指数衰减 `exp(-λt^β)` | 越久远的记忆衰减越快   |
+| **频率 (frequency)**     | 对数饱和曲线 + 时间加权           | 越常被访问的记忆越活跃 |
+| **内在价值 (intrinsic)** | `importance × confidence`         | 高价值记忆天然抵抗遗忘 |
+
+层级特定的衰减形状 (β 参数)：
+
+- **Core** (β=0.8)：亚指数衰减 → 遗忘极慢，衰减地板 0.9
+- **Working** (β=1.0)：标准指数衰减，衰减地板 0.7
+- **Peripheral** (β=1.3)：超指数衰减 → 遗忘加速，衰减地板 0.5
+
+关键特性：
+
+- **重要性调制半衰期**：`effectiveHL = halfLife × exp(μ × importance)`，重要记忆持续更久
+- **搜索结果加权**：检索时自动应用衰减加权，让活跃记忆排名更高
+- **过期识别**：识别 composite < 0.3 的过期记忆
+
+---
+
+### 6. `src/tier-manager.ts` — 三层晋升/降级管理器
+
+模拟人类记忆的多级存储模型：
+
+```
+Peripheral（外围） ⟷ Working（工作） ⟷ Core（核心）
+```
+
+**晋升条件**：
+
+| 方向                 | 条件                                            |
+| -------------------- | ----------------------------------------------- |
+| Peripheral → Working | 访问次数 ≥ 3 且 衰减分数 ≥ 0.4                  |
+| Working → Core       | 访问次数 ≥ 10 且 衰减分数 ≥ 0.7 且 重要度 ≥ 0.8 |
+
+**降级条件**：
+
+| 方向                 | 条件                                             |
+| -------------------- | ------------------------------------------------ |
+| Working → Peripheral | 衰减分数 < 0.15 或（年龄 > 60 天且访问次数 < 3） |
+| Core → Working       | 衰减分数 < 0.15 且 访问次数 < 3（极少触发）      |
+
+---
+
+## 四、修改文件
+
+### `index.ts` — 插件入口
+
+#### 新增配置项
+
+```typescript
+smartExtraction?: boolean;    // 是否启用 LLM 智能提取（默认 true）
+llm?: {
+  apiKey?: string;            // LLM API Key（默认复用 embedding.apiKey）
+  model?: string;             // LLM 模型（默认 gpt-4o-mini）
+  baseURL?: string;           // LLM API 端点
+};
+extractMinMessages?: number;  // 最少消息数才触发提取（默认 4）
+extractMaxChars?: number;     // 送入 LLM 的最大字符数（默认 8000）
+```
+
+#### `agent_end` 钩子改进
+
+- 当 `smartExtraction` 启用时，优先使用 SmartExtractor 进行 LLM 6 类别提取
+- 当消息数不足或 SmartExtractor 未初始化时，降级回原有正则触发逻辑
+- 提取完成后输出统计日志：`smart-extracted N created, M merged, K skipped`
+
+#### `before_agent_start` 钩子改进
+
+- 注入的记忆上下文现在显示 L0 摘要而非原始文本
+- 新增 6 类别标签（如 `[preferences:global]`）
+- 新增层级标记（`[C]`ore / `[W]`orking / `[P]`eripheral）
+
+---
+
+## 五、配置指南
+
+### 最简配置（复用已有 API Key）
+
+```json
+{
+  "embedding": {
+    "apiKey": "${OPENAI_API_KEY}",
+    "model": "text-embedding-3-small"
+  },
+  "smartExtraction": true
+}
+```
+
+### 完整配置
+
+```json
+{
+  "embedding": {
+    "apiKey": "${OPENAI_API_KEY}",
+    "model": "text-embedding-3-small"
+  },
+  "smartExtraction": true,
+  "llm": {
+    "apiKey": "${OPENAI_API_KEY}",
+    "model": "gpt-4o-mini",
+    "baseURL": "https://api.openai.com/v1"
+  },
+  "extractMinMessages": 4,
+  "extractMaxChars": 8000
+}
+```
+
+### 禁用智能提取
+
+```json
+{
+  "smartExtraction": false
+}
+```
+
+---
+
+## 六、向后兼容性
+
+| 方面           | 兼容方式                                       |
+| -------------- | ---------------------------------------------- |
+| LanceDB Schema | 新字段存储在 `metadata` JSON 中，不修改表结构  |
+| 记忆类别       | 新 6 类别自动映射到原有 5 类别                 |
+| 混合检索       | Vector+BM25 检索管线完全保留                   |
+| 去重逻辑       | 仅在 `smartExtraction: true` 时生效            |
+| 已有数据       | 旧记忆正常读取，新记忆额外携带 L0/L1/L2 元数据 |
+| 配置           | 全部新增配置项均有默认值，零配置即可使用       |

package/CHANGELOG.md

@@ -1,5 +1,20 @@
 # Changelog
 
+## 1.1.0-beta.1 (Smart Memory Beta)
+
+This is a **beta** release published under the npm dist-tag **`beta`** (it will not affect the stable `latest` channel).
+
+Highlights:
+- **Smart Extraction (LLM-powered)**: 6-category extraction with L0/L1/L2 metadata (falls back to regex capture when disabled or init fails)
+- **Lifecycle scoring integrated into retrieval**: decay-based score adjustment + tier floors
+- **Tier transitions (best-effort)**: records access stats for top results and can promote/demote tiers via metadata
+- Rebases the smart-memory branch onto `main@1.0.25` (keeps multi-key rotation & other recent fixes)
+
+Notes:
+- Retrieval now performs small, bounded metadata write-backs for top results (access_count / last_accessed_at / tier).
+
+---
+
 
 ## 1.0.22
 

package/index.ts

@@ -19,6 +19,12 @@ import { registerAllMemoryTools } from "./src/tools.js";
 import { shouldSkipRetrieval } from "./src/adaptive-retrieval.js";
 import { createMemoryCLI } from "./cli.js";
 
+// Import smart extraction & lifecycle components
+import { SmartExtractor } from "./src/smart-extractor.js";
+import { createLlmClient } from "./src/llm-client.js";
+import { createDecayEngine, DEFAULT_DECAY_CONFIG } from "./src/decay-engine.js";
+import { createTierManager, DEFAULT_TIER_CONFIG } from "./src/tier-manager.js";
+
 // ============================================================================
 // Configuration & Types
 // ============================================================================
@@ -26,7 +32,7 @@ import { createMemoryCLI } from "./cli.js";
 interface PluginConfig {
   embedding: {
     provider: "openai-compatible";
-    apiKey: string;
+    apiKey: string | string[];
     model?: string;
     baseURL?: string;
     dimensions?: number;
@@ -57,6 +63,15 @@ interface PluginConfig {
     hardMinScore?: number;
     timeDecayHalfLifeDays?: number;
   };
+  // Smart extraction config (Phase 1: from epro-memory)
+  smartExtraction?: boolean;
+  llm?: {
+    apiKey?: string;
+    model?: string;
+    baseURL?: string;
+  };
+  extractMinMessages?: number;
+  extractMaxChars?: number;
   scopes?: {
     default?: string;
     definitions?: Record<string, { description: string }>;
@@ -344,17 +359,62 @@ const memoryLanceDBProPlugin = {
       taskPassage: config.embedding.taskPassage,
       normalized: config.embedding.normalized,
     });
-    const retriever = createRetriever(store, embedder, {
-      ...DEFAULT_RETRIEVAL_CONFIG,
-      ...config.retrieval,
-    });
+    // Initialize decay engine + tier manager (lifecycle scoring)
+    const decayEngine = createDecayEngine(DEFAULT_DECAY_CONFIG);
+    const tierManager = createTierManager(DEFAULT_TIER_CONFIG);
+
+    const retriever = createRetriever(
+      store,
+      embedder,
+      {
+        ...DEFAULT_RETRIEVAL_CONFIG,
+        ...config.retrieval,
+      },
+      { decayEngine, tierManager },
+    );
     const scopeManager = createScopeManager(config.scopes);
     const migrator = createMigrator(store);
 
+    // Initialize smart extraction (Phase 1: from epro-memory)
+    let smartExtractor: SmartExtractor | null = null;
+    if (config.smartExtraction !== false) {
+      try {
+        const embeddingKey = Array.isArray(config.embedding.apiKey)
+          ? config.embedding.apiKey[0]
+          : config.embedding.apiKey;
+        const llmApiKey = config.llm?.apiKey
+          ? resolveEnvVars(config.llm.apiKey)
+          : resolveEnvVars(embeddingKey);
+        const llmBaseURL = config.llm?.baseURL
+          ? resolveEnvVars(config.llm.baseURL)
+          : config.embedding.baseURL;
+        const llmModel = config.llm?.model || "gpt-4o-mini";
+
+        const llmClient = createLlmClient({
+          apiKey: llmApiKey,
+          model: llmModel,
+          baseURL: llmBaseURL,
+          timeoutMs: 30000,
+        });
+
+        smartExtractor = new SmartExtractor(store, embedder, llmClient, {
+          user: "User",
+          extractMinMessages: config.extractMinMessages ?? 4,
+          extractMaxChars: config.extractMaxChars ?? 8000,
+          defaultScope: config.scopes?.default ?? "global",
+          log: (msg: string) => api.logger.info(msg),
+        });
+
+        api.logger.info("memory-lancedb-pro: smart extraction enabled (LLM model: " + llmModel + ")");
+      } catch (err) {
+        api.logger.warn(`memory-lancedb-pro: smart extraction init failed, falling back to regex: ${String(err)}`);
+      }
+    }
+
     const pluginVersion = getPluginVersion();
 
     api.logger.info(
-      `memory-lancedb-pro@${pluginVersion}: plugin registered (db: ${resolvedDbPath}, model: ${config.embedding.model || "text-embedding-3-small"})`
+      `memory-lancedb-pro@${pluginVersion}: plugin registered (db: ${resolvedDbPath}, model: ${config.embedding.model || "text-embedding-3-small"}, smartExtraction: ${smartExtractor ? 'ON' : 'OFF'})`
     );
 
     // ========================================================================
@@ -417,8 +477,17 @@ const memoryLanceDBProPlugin = {
             return;
           }
 
+          // Format with L0 abstracts grouped by category when available
           const memoryContext = results
-            .map((r) => `- [${r.entry.category}:${r.entry.scope}] ${sanitizeForContext(r.entry.text)} (${(r.score * 100).toFixed(0)}%${r.sources?.bm25 ? ', vector+BM25' : ''}${r.sources?.reranked ? '+reranked' : ''})`)
+            .map((r) => {
+              let metaObj: Record<string, unknown> = {};
+              try { metaObj = JSON.parse(r.entry.metadata || "{}"); } catch {}
+              const displayCategory = (metaObj.memory_category as string) || r.entry.category;
+              const displayTier = (metaObj.tier as string) || "";
+              const tierPrefix = displayTier ? `[${displayTier.charAt(0).toUpperCase()}]` : "";
+              const abstract = (metaObj.l0_abstract as string) || r.entry.text;
+              return `- ${tierPrefix}[${displayCategory}:${r.entry.scope}] ${sanitizeForContext(abstract)} (${(r.score * 100).toFixed(0)}%${r.sources?.bm25 ? ', vector+BM25' : ''}${r.sources?.reranked ? '+reranked' : ''})`;
+            })
             .join("\n");
 
           api.logger.info?.(
@@ -488,7 +557,29 @@ const memoryLanceDBProPlugin = {
             }
           }
 
-          // Filter for capturable content
+          // ----------------------------------------------------------------
+          // Smart Extraction (Phase 1: LLM-powered 6-category extraction)
+          // ----------------------------------------------------------------
+          if (smartExtractor) {
+            const minMessages = config.extractMinMessages ?? 4;
+            if (texts.length >= minMessages) {
+              const conversationText = texts.join("\n");
+              const sessionKey = (event as any).sessionKey || "unknown";
+              const stats = await smartExtractor.extractAndPersist(
+                conversationText, sessionKey,
+              );
+              if (stats.created > 0 || stats.merged > 0) {
+                api.logger.info(
+                  `memory-lancedb-pro: smart-extracted ${stats.created} created, ${stats.merged} merged, ${stats.skipped} skipped for agent ${agentId}`
+                );
+              }
+              return; // Smart extraction handled everything
+            }
+          }
+
+          // ----------------------------------------------------------------
+          // Fallback: regex-triggered capture (original logic)
+          // ----------------------------------------------------------------
           const toCapture = texts.filter((text) => text && shouldCapture(text));
           if (toCapture.length === 0) {
             return;
@@ -743,11 +834,22 @@ function parsePluginConfig(value: unknown): PluginConfig {
   }
 
   // Accept single key (string) or array of keys for round-robin rotation
-  const apiKey: string | string[] = typeof embedding.apiKey === "string"
-    ? embedding.apiKey
-    : Array.isArray(embedding.apiKey) && embedding.apiKey.length > 0 && embedding.apiKey.every((k: unknown) => typeof k === "string")
-      ? (embedding.apiKey as string[])
-      : process.env.OPENAI_API_KEY || "";
+  let apiKey: string | string[];
+  if (typeof embedding.apiKey === "string") {
+    apiKey = embedding.apiKey;
+  } else if (Array.isArray(embedding.apiKey) && embedding.apiKey.length > 0) {
+    // Validate every element is a non-empty string
+    const invalid = embedding.apiKey.findIndex((k: unknown) => typeof k !== "string" || (k as string).trim().length === 0);
+    if (invalid !== -1) {
+      throw new Error(`embedding.apiKey[${invalid}] is invalid: expected non-empty string`);
+    }
+    apiKey = embedding.apiKey as string[];
+  } else if (embedding.apiKey !== undefined) {
+    // apiKey is present but wrong type — throw, don't silently fall back
+    throw new Error("embedding.apiKey must be a string or non-empty array of strings");
+  } else {
+    apiKey = process.env.OPENAI_API_KEY || "";
+  }
 
   if (!apiKey || (Array.isArray(apiKey) && apiKey.length === 0)) {
     throw new Error("embedding.apiKey is required (set directly or via OPENAI_API_KEY env var)");
@@ -773,6 +875,11 @@ function parsePluginConfig(value: unknown): PluginConfig {
     autoRecallMinLength: parsePositiveInt(cfg.autoRecallMinLength),
     captureAssistant: cfg.captureAssistant === true,
     retrieval: typeof cfg.retrieval === "object" && cfg.retrieval !== null ? cfg.retrieval as any : undefined,
+    // Smart extraction config (Phase 1)
+    smartExtraction: cfg.smartExtraction !== false, // Default ON
+    llm: typeof cfg.llm === "object" && cfg.llm !== null ? cfg.llm as any : undefined,
+    extractMinMessages: parsePositiveInt(cfg.extractMinMessages) ?? 4,
+    extractMaxChars: parsePositiveInt(cfg.extractMaxChars) ?? 8000,
     scopes: typeof cfg.scopes === "object" && cfg.scopes !== null ? cfg.scopes as any : undefined,
     enableManagementTools: cfg.enableManagementTools === true,
     sessionMemory: typeof cfg.sessionMemory === "object" && cfg.sessionMemory !== null

package/openclaw.plugin.json

@@ -2,7 +2,7 @@
   "id": "memory-lancedb-pro",
   "name": "Memory (LanceDB Pro)",
   "description": "Enhanced LanceDB-backed long-term memory with hybrid retrieval, multi-scope isolation, long-context chunking, and management CLI",
-  "version": "1.0.23",
+  "version": "1.1.0-beta.1",
   "kind": "memory",
   "configSchema": {
     "type": "object",
@@ -18,8 +18,18 @@
           },
           "apiKey": {
             "oneOf": [
-              { "type": "string" },
-              { "type": "array", "items": { "type": "string" }, "minItems": 1 }
+              {
+                "type": "string",
+                "minLength": 1
+              },
+              {
+                "type": "array",
+                "items": {
+                  "type": "string",
+                  "minLength": 1
+                },
+                "minItems": 1
+              }
             ],
             "description": "Single API key or array of keys for round-robin rotation"
           },
@@ -239,6 +249,44 @@
             }
           }
         }
+      },
+      "smartExtraction": {
+        "type": "boolean",
+        "default": true,
+        "description": "Enable LLM-powered smart memory extraction (6-category system). Falls back to regex capture when disabled or init fails."
+      },
+      "llm": {
+        "type": "object",
+        "additionalProperties": false,
+        "properties": {
+          "apiKey": {
+            "type": "string",
+            "description": "API key for LLM extraction (defaults to embedding apiKey)"
+          },
+          "model": {
+            "type": "string",
+            "default": "gpt-4o-mini",
+            "description": "LLM model for memory extraction and dedup"
+          },
+          "baseURL": {
+            "type": "string",
+            "description": "Base URL for LLM API (defaults to embedding baseURL)"
+          }
+        }
+      },
+      "extractMinMessages": {
+        "type": "integer",
+        "minimum": 1,
+        "maximum": 50,
+        "default": 4,
+        "description": "Minimum conversation messages before smart extraction triggers"
+      },
+      "extractMaxChars": {
+        "type": "integer",
+        "minimum": 500,
+        "maximum": 100000,
+        "default": 8000,
+        "description": "Maximum characters of conversation text to process for extraction"
       }
     },
     "required": [
@@ -297,6 +345,36 @@
       "help": "Directory path for the LanceDB database files",
       "advanced": true
     },
+    "smartExtraction": {
+      "label": "Smart Extraction",
+      "help": "Enable LLM-powered 6-category memory extraction. Falls back to regex capture when off."
+    },
+    "llm.apiKey": {
+      "label": "LLM API Key",
+      "sensitive": true,
+      "help": "API key for smart extraction LLM (defaults to embedding apiKey)",
+      "advanced": true
+    },
+    "llm.model": {
+      "label": "LLM Model",
+      "help": "Model for memory extraction and dedup (default: gpt-4o-mini)",
+      "advanced": true
+    },
+    "llm.baseURL": {
+      "label": "LLM Base URL",
+      "help": "Base URL for LLM API (defaults to embedding baseURL)",
+      "advanced": true
+    },
+    "extractMinMessages": {
+      "label": "Min Messages for Extraction",
+      "help": "Minimum conversation messages before smart extraction triggers",
+      "advanced": true
+    },
+    "extractMaxChars": {
+      "label": "Max Chars for Extraction",
+      "help": "Maximum conversation characters to process for extraction",
+      "advanced": true
+    },
     "autoCapture": {
       "label": "Auto-Capture",
       "help": "Automatically capture important information from conversations"

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "memory-lancedb-pro",
-  "version": "1.0.24",
+  "version": "1.1.0-beta.1",
   "description": "OpenClaw enhanced LanceDB memory plugin with hybrid retrieval (Vector + BM25), cross-encoder rerank, multi-scope isolation, long-context chunking, and management CLI",
   "type": "module",
   "main": "index.ts",