@pencil-agent/nano-pencil 2.0.1 → 2.0.3

package/docs/mem-core/346/212/200/346/234/257/346/226/207/346/241/243.md

@@ -0,0 +1,593 @@
+# mem-core 技术文档
+
+> nanoPencil 持久化记忆系统技术规格说明
+
+## 概述
+
+mem-core 是 nanoPencil 的持久化记忆模块，负责从对话中提取、存储、检索项目相关的知识与上下文。核心设计基于 Mem0 和 A-MEM 的理念，采用分层存储与渐进召回机制。
+
+**位置**: `packages/mem-core/`
+**依赖**: 无内部依赖（standalone）
+
+---
+
+## 核心架构
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                     NanoMemEngine                            │
+│                    (packages/mem-core/src/engine.ts)          │
+└──────────────────────┬──────────────────────────────────────┘
+                       │
+   ┌───────────────────┼───────────────────┐
+   │                   │                   │
+   ▼                   ▼                   ▼
+Extraction          Storage            Retrieval
+(extraction.ts)     (store.ts)         (scoring.ts)
+   │                   │                   │
+   ├─ LLM path         ├─ JSON files      ├─ scoreEntry()
+   └─ Heuristic        ├─ V2 layered      ├─ tierEntries()
+                       │   (store-v2.ts)  └─ pickTop()
+                       │
+   ▼                   ▼                   ▼
+Consolidation      Privacy            Linking
+(consolidation.ts) (privacy.ts)        (linking.ts)
+   │                   │                   │
+   ├─ LLM consolidation   ├─ PII filter    ├─ Graph neighborhood
+   └─ Heuristic fallback ├─ Scope filter  └─ Related IDs
+                       └─ TTL eviction
+
+Dream Lock (dream-lock.ts): File-based mutex for background consolidation
+Update (update.ts): Mem0-style add/update/delete/noop pipeline
+Dedup (dedup.ts): Tag overlap + content similarity deduplication
+```
+
+---
+
+## 存储结构
+
+### V1 存储（遗留兼容）
+
+路径: `~/.nanomem/memory/`
+
+| 文件 | 内容 | 默认 TTL |
+|------|------|----------|
+| `knowledge.json` | 事实、实体、决策 | 45 天 |
+| `lessons.json` | 经验教训（错误、解决方案） | 90 天 |
+| `events.json` | 关键事件 | 180 天 |
+| `preferences.json` | 用户偏好 | 120 天 |
+| `facets.json` | 行为模式（pattern + struggle） | 180 天 |
+| `work.json` | 工作/任务历史 | 21 天 |
+| `episodes/*.json` | 会话摘要 | 14 天 |
+| `meta.json` | 元数据（会话数、上次整合时间） | - |
+
+### V2 分层存储
+
+路径: `~/.nanomem/memory/v2/`
+
+| 文件 | 类型 | 说明 |
+|------|------|------|
+| `episodes.json` | EpisodeMemory | 会话情景记忆 |
+| `facets.json` | EpisodeFacet[] | 原子洞察（目标/决策/错误/洞察） |
+| `semantic.json` | SemanticMemory[] | 语义记忆（事实/教训/偏好） |
+| `procedural.json` | ProceduralMemory[] | 程序记忆（步骤性知识） |
+| `links.json` | MemoryLink[] | 记忆关系图 |
+
+---
+
+## 核心类型定义
+
+### MemoryEntry (types.ts)
+
+```typescript
+interface MemoryEntry {
+  id: string;
+  type: "fact" | "lesson" | "preference" | "decision"
+       | "entity" | "pattern" | "struggle" | "event";
+  name?: string;       // 短标题 (<=30 字符)
+  summary?: string;   // 单行提示 (<=150 字符)
+  detail?: string;    // 完整内容
+  tags: string[];
+  project: string;
+  importance: number;  // 1-10
+  strength?: number;   // 自适应记忆强度（半衰期天数）
+  created: string;     // 摄入时间 (ISO)
+  eventTime?: string;   // 事件实际发生时间
+  lastAccessed?: string;
+  accessCount: number;
+  relatedIds?: string[];
+  ttl?: number;        // 生存时间（天）
+  retention: "core" | "key-event" | "ambient";
+  stability: "stable" | "situational";
+  facetData?: { kind: "pattern" | "struggle"; ... };
+  eventData?: { kind: string; outcome?: string; emotionalWeight?: number };
+}
+```
+
+### V2 类型 (types-v2.ts)
+
+```typescript
+// 会话情景记忆
+interface EpisodeMemory {
+  id: string;
+  sessionId: string;
+  project: string;
+  date: string;
+  summary: string;
+  userGoal?: string;
+  filesModified: string[];
+  toolsUsed: Record<string, number>;
+  keyObservations: string[];
+  errors: string[];
+  tags: string[];
+  importance: number;
+  consolidated: boolean;
+}
+
+// 原子洞察
+interface EpisodeFacet {
+  id: string;
+  episodeId: string;
+  kind: "goal" | "decision" | "error" | "insight";
+  content: string;
+  tags: string[];
+  created: string;
+}
+
+// 语义记忆
+interface SemanticMemory {
+  id: string;
+  type: "fact" | "lesson" | "preference";
+  content: string;
+  tags: string[];
+  project: string;
+  importance: number;
+  strength: number;
+  created: string;
+  lastAccessed?: string;
+  accessCount: number;
+}
+
+// 程序记忆
+interface ProceduralMemory {
+  id: string;
+  title: string;
+  steps: string[];
+  tags: string[];
+  project: string;
+  importance: number;
+  created: string;
+}
+
+// 状态记忆
+interface StateMemory {
+  id: string;
+  kind: "mood" | "pressure" | "energy";
+  value: number;  // 0-1
+  context: string;
+  created: string;
+  expiresAt: string;
+}
+
+// 记忆链接
+interface MemoryLink {
+  id: string;
+  sourceId: string;
+  targetId: string;
+  relation: "relates" | "contradicts" | "refines" | "causes";
+  strength: number;
+}
+```
+
+---
+
+## 提取机制
+
+### 双路径提取 (extraction.ts)
+
+```
+┌─────────────────────────────────────────┐
+│        extractAndStore(conversation)     │
+└──────────────────┬──────────────────────┘
+                   │
+                   ▼
+         ┌─────────────────┐
+         │  extractWithLLM │
+         └────────┬────────┘
+                  │ 成功
+                  ▼
+         ┌─────────────────┐
+         │  解析为 JSON     │
+         │  ExtractedItem[]│
+         └────────┬────────┘
+                  │ 失败或超时
+                  ▼
+         ┌─────────────────┐
+         │ extractHeuristic│
+         └─────────────────┘
+```
+
+### LLM 提取
+
+调用配置的 LLM，使用结构化 prompt 生成 JSON 响应：
+
+```typescript
+interface ExtractedItem {
+  type: "lesson" | "preference" | "fact" | "decision"
+      | "event" | "pattern" | "struggle";
+  name: string;
+  summary: string;
+  detail?: string;
+  tags: string[];
+  importance: number;
+}
+```
+
+### 启发式回退
+
+当 LLM 不可用或失败时，使用正则匹配：
+
+| 类型 | 匹配模式 |
+|------|----------|
+| lesson | `error:`, `fixed:`, `bug:`, `warning:`, `learned:` |
+| preference | `i prefer`, `call me`, `always use`, `never do` |
+| fact | `project uses`, `api:`, `config:`, `version` |
+| decision | `decided to`, `chose to`, `because:` |
+| event | `launched`, `shipped`, `major incident` |
+| pattern | `i always`, `every time i`, `my habit is` |
+| struggle | `failed`, `couldn't get`, `struggled with` |
+
+---
+
+## 评分与检索
+
+### Stanford 三因素评分 (scoring.ts)
+
+```typescript
+scoreEntry(entry, project, contextTags, halfLife, weights):
+  // 1. 最近度 (Ebbinghaus 遗忘曲线)
+  recency = decay(daysSince(created), strength)
+          = e^(-t × ln2 / S)  // S = 半衰期
+
+  // 2. 重要性
+  importance = entry.importance / 10
+
+  // 3. 相关性
+  relevance = projectMatch
+            × (0.3 + 0.7 × tagOverlap(tags, contextTags))
+
+  return weights.recency  × recency
+       + weights.importance × importance
+       + weights.relevance × relevance
+```
+
+**默认权重**: `{ recency: 1, importance: 1, relevance: 1 }`
+
+### 半衰期配置 (config.ts)
+
+| 类型 | 半衰期（天） |
+|------|-------------|
+| lesson | 90 |
+| fact | 60 |
+| episode | 14 |
+| preference | 120 |
+| decision | 45 |
+| entity | 30 |
+| event | 180 |
+| work | 45 |
+| pattern | 180 |
+| struggle | 120 |
+
+### 渐进召回分层
+
+```typescript
+const PROGRESSIVE_RECALL = {
+  thresholdActive: 0.7,    // score >= 0.7 → Active 层
+  thresholdCue: 0.35,     // score >= 0.35 → Cue 层
+  budgetActive: 0.15,     // Active 占 15% token 预算
+  budgetCue: 0.70,        // Cue 占 70% token 预算
+  forceRecentHours: 24,   // 强制保留最近 24 小时
+  forceImportanceMin: 9,  // importance >= 9 强制保留
+};
+```
+
+**注入内容差异：**
+
+| 层级 | 内容 |
+|------|------|
+| Active | 完整细节（name + summary + detail） |
+| Cue | 名称 + 摘要 + ID（用于上下文提示） |
+| Dormant | 不注入 |
+
+---
+
+## 去重与合并
+
+### 去重逻辑 (dedup.ts)
+
+```typescript
+function isMemoryDuplicate(a: MemoryEntry, b: MemoryEntry): boolean {
+  tagMatch = tagOverlap(a.tags, b.tags) >= 0.75
+  contentMatch = contentSimilarity(a.text, b.text) >= 0.8
+  return tagMatch || contentMatch
+}
+```
+
+### 更新管道 (update.ts)
+
+```
+Mem0-style 管道: add | update | delete | noop
+
+┌─────────────────────────────────────────┐
+│            processMemoryUpdate()         │
+└──────────────────┬──────────────────────┘
+                   │
+         tagOverlap >= 0.7?
+         /                \
+        Yes               No
+         │                 │
+         ▼                 ▼
+    update            是否完全重复?
+    (合并详情)          /         \
+                      Yes        No
+                       │          │
+                       ▼          ▼
+                    noop        add
+```
+
+---
+
+## Dream 机制（后台整合）
+
+### Dream Lock (dream-lock.ts)
+
+防止多个进程同时执行整合：
+
+```typescript
+tryAcquireDreamLock(lockPath, staleMs = 30min):
+  1. 检查锁文件是否存在
+  2. 如果存在，检查持有进程是否存活
+  3. 如果进程死亡或锁陈旧，获取锁
+  4. 写入当前 PID 到锁文件
+  5. 验证锁归属
+  6. 定期 stampDreamLock() 更新心跳
+  7. 失败时 rollbackDreamLock() 回滚
+```
+
+### 整合触发条件
+
+- 未整合 episode 数量 >= `consolidationThreshold` (默认 10)
+- 或距上次整合超过 `consolidationInterval` (默认 24h)
+
+### 整合流程 (consolidation.ts)
+
+```
+┌─────────────────────────────────────────┐
+│         consolidateDetailed()            │
+└──────────────────┬──────────────────────┘
+                   │
+         ┌─────────┴─────────┐
+         ▼                   ▼
+    consolidateWithLLM  consolidateHeuristic
+         │                   │
+         ▼                   ▼
+    总结所有未整合 episodes   识别热文件 (修改 >= 3次)
+         │                   │
+         ▼                   提取重复错误
+    转换为 MemoryEntry       创建 event 条目
+         │
+         ▼
+    存储到 v2/semantic.json
+```
+
+---
+
+## 隐私保护
+
+### PII 模式 (privacy.ts)
+
+```typescript
+const PII_PATTERNS = [
+  /email/,                     // 邮箱
+  /phone/,                     // 电话号码
+  /1[3-9]\d{9}/,              // 中国手机号
+  /\d{3}-\d{2}-\d{4}/,        // SSN
+  /CN ID/,                     // 身份证
+  /credit card/,              // 信用卡
+  /password/i,                // 密码
+  /secret/i,                  // 密钥
+  /api[_-]?key/i,             // API Key
+];
+```
+
+### Scope 过滤
+
+```typescript
+interface MemoryScope {
+  userId?: string;
+  agentId?: string;
+  project?: string;
+}
+
+function matchesScope(entry, required):
+  if (!required) return true
+  if (required.userId && entry.scope.userId !== required.userId) return false
+  if (required.agentId && entry.scope.agentId !== required.agentId) return false
+  return true
+```
+
+---
+
+## 记忆注入 Prompt
+
+### getMemoryInjection() 输出格式
+
+```markdown
+## Memory
+
+### Recent (Active Tier)
+- [fact] API 路径: /api/v1 (importance: 8, accessed: 3)
+- [lesson] 创建用户时需要 email 字段 (importance: 7)
+
+### Context (Cue Tier)
+- entity: React Hooks - 组件状态管理方案
+- preference: 用户偏好函数式组件
+```
+
+### pickTop() Token 预算分配
+
+```typescript
+function pickTop(entries, contextTags, budget):
+  // 1. 按 score 排序
+  // 2. Active 层优先，占用 15% 预算
+  // 3. Cue 层填充到 85% 预算
+  // 4. 格式化输出
+```
+
+---
+
+## 关系图谱
+
+### Linking 机制 (linking.ts)
+
+```typescript
+interface NeighborhoodResult {
+  nodes: MemoryEntry[];   // 中心节点
+  edges: MemoryLink[];    // 关联边
+  depth: number;          // 遍历深度
+}
+```
+
+**关联发现**：
+1. 给定种子 memory ID
+2. 扩展相关 IDs（通过 links.json）
+3. 递归遍历直到深度阈值
+4. 返回子图
+
+---
+
+## 配置参数
+
+### EngineConfig
+
+```typescript
+interface EngineConfig {
+  storageDir?: string;           // 默认 ~/.nanomem/memory/
+  halfLife?: HalfLifeConfig;    // 各类型半衰期
+  progressiveRecall?: RecallConfig;
+  consolidationThreshold?: number; // 默认 10
+  consolidationInterval?: number;  // 默认 24h (ms)
+  embeddingFn?: EmbeddingFn;    // 向量化函数
+  llmFn?: LLMFn;                // LLM 调用函数
+}
+```
+
+### 关键常量
+
+```typescript
+// dedup.ts
+const TAG_OVERLAP_THRESHOLD = 0.75
+const CONTENT_SIMILARITY_THRESHOLD = 0.80
+
+// update.ts
+const UPDATE_TAG_OVERLAP = 0.7
+const NOOP_SIMILARITY = 0.85
+
+// scoring.ts
+const DEFAULT_WEIGHTS = { recency: 1, importance: 1, relevance: 1 }
+const HALF_LIFE_DEFAULT = 60  // 天
+```
+
+---
+
+## 使用示例
+
+### 初始化引擎
+
+```typescript
+import { NanoMemEngine } from '@pencil-agent/mem-core';
+
+const engine = new NanoMemEngine({
+  storageDir: '~/.nanomem/memory',
+  llmFn: yourLLMCaller,
+  embeddingFn: yourEmbeddingFn,
+});
+
+// 设置项目上下文
+engine.setProject('my-project');
+```
+
+### 提取与存储
+
+```typescript
+const conversation = [
+  { role: 'user', content: '这个项目用 React' },
+  { role: 'assistant', content: '好的，已记住' },
+];
+
+await engine.extractAndStore(conversation, 'my-project');
+```
+
+### 检索记忆
+
+```typescript
+const injection = await engine.getMemoryInjection(
+  'my-project',
+  ['react', 'frontend'],
+  'active'  // scope
+);
+
+console.log(injection);
+// 输出:
+// ## Memory
+// - [fact] 本项目使用 React 框架 (importance: 9)
+```
+
+### 触发整合
+
+```typescript
+await engine.consolidate();  // 使用 AbortSignal 可取消
+```
+
+---
+
+## 文件清单
+
+```
+packages/mem-core/
+├── src/
+│   ├── engine.ts          # 主引擎 NanoMemEngine
+│   ├── store.ts           # V1 JSON 存储
+│   ├── store-v2.ts        # V2 分层存储
+│   ├── extraction.ts      # LLM + 启发式提取
+│   ├── consolidation.ts   # Episode 整合
+│   ├── dedup.ts           # 去重逻辑
+│   ├── update.ts          # Mem0 风格更新管道
+│   ├── scoring.ts         # Stanford 评分算法
+│   ├── linking.ts         # 记忆关系图
+│   ├── privacy.ts         # PII 过滤
+│   ├── dream-lock.ts      # 后台整合锁
+│   ├── types.ts           # V1 类型定义
+│   ├── types-v2.ts        # V2 类型定义
+│   ├── config.ts          # 默认配置
+│   ├── eviction.ts        # TTL 淘汰算法
+│   ├── insights-html.ts   # HTML 洞察报告
+│   ├── insights.ts        # 洞察生成
+│   ├── full-insights.ts   # 完整洞察
+│   ├── human-insights.ts  # 人类可读洞察
+│   ├── i18n.ts            # 国际化
+│   ├── extension.ts       # nanoPencil 扩展适配器
+│   └── index.ts           # 导出
+├── test/
+│   └── *.test.ts          # 单元测试
+├── package.json
+└── CLAUDE.md              # 包文档
+```
+
+---
+
+## 相关文档
+
+- [记忆系统.md](./记忆系统.md) - 用户视角概述（旧版 1.3.0）
+- [soul-core 文档](./) - AI 性格系统
+- [AgentTeam 文档](./AgentTeam重构方案.md) - 多智能体协作

package/docs/models.md

@@ -1,27 +1,27 @@
----
-name: models
-description: Use when the user asks how to add, select, or configure models.
-surface: TODO  # user entry points: /command, --flag, config key, file location
-owner: core/model-registry.ts  # DIP P2 anchor — read its AGENT.md member list to find code
-status: draft
----
-
-# Models
-
-> TODO: one line — what this feature does for the user.
-
-## When to use
-TODO: the user intents that should pull this doc (mirrors the frontmatter `description`).
-
-## Usage
-TODO: commands / flags / config keys / file locations, with one minimal example.
-
-## Behavior & defaults
-TODO: default on/off, side effects, opt-in/opt-out.
-
-## Code map → DIP
-- Owner: `core/model-registry.ts` — read its DIP **P2 member list** (the nearest `AGENT.md`) to locate files.
-- Then follow **P3** file headers (WHO / FROM / TO / HERE) to navigate. Do **not** duplicate code paths here.
-
-## Related
-[[custom-provider]] [[providers]]
+---
+name: models
+description: Use when the user asks how to add, select, or configure models.
+surface: TODO  # user entry points: /command, --flag, config key, file location
+owner: core/model-registry.ts  # DIP P2 anchor — read its AGENT.md member list to find code
+status: draft
+---
+
+# Models
+
+> TODO: one line — what this feature does for the user.
+
+## When to use
+TODO: the user intents that should pull this doc (mirrors the frontmatter `description`).
+
+## Usage
+TODO: commands / flags / config keys / file locations, with one minimal example.
+
+## Behavior & defaults
+TODO: default on/off, side effects, opt-in/opt-out.
+
+## Code map → DIP
+- Owner: `core/model-registry.ts` — read its DIP **P2 member list** (the nearest `AGENT.md`) to locate files.
+- Then follow **P3** file headers (WHO / FROM / TO / HERE) to navigate. Do **not** duplicate code paths here.
+
+## Related
+[[custom-provider]] [[providers]]

package/docs/packages.md

@@ -1,27 +1,27 @@
----
-name: packages
-description: Use when the user asks about the published @pencil-agent/* packages.
-surface: TODO  # user entry points: /command, --flag, config key, file location
-owner: packages/  # DIP P2 anchor — read its AGENT.md member list to find code
-status: draft
----
-
-# Packages
-
-> TODO: one line — what this feature does for the user.
-
-## When to use
-TODO: the user intents that should pull this doc (mirrors the frontmatter `description`).
-
-## Usage
-TODO: commands / flags / config keys / file locations, with one minimal example.
-
-## Behavior & defaults
-TODO: default on/off, side effects, opt-in/opt-out.
-
-## Code map → DIP
-- Owner: `packages/` — read its DIP **P2 member list** (the nearest `AGENT.md`) to locate files.
-- Then follow **P3** file headers (WHO / FROM / TO / HERE) to navigate. Do **not** duplicate code paths here.
-
-## Related
-[[sdk]] [[extensions]]
+---
+name: packages
+description: Use when the user asks about the published @pencil-agent/* packages.
+surface: TODO  # user entry points: /command, --flag, config key, file location
+owner: packages/  # DIP P2 anchor — read its AGENT.md member list to find code
+status: draft
+---
+
+# Packages
+
+> TODO: one line — what this feature does for the user.
+
+## When to use
+TODO: the user intents that should pull this doc (mirrors the frontmatter `description`).
+
+## Usage
+TODO: commands / flags / config keys / file locations, with one minimal example.
+
+## Behavior & defaults
+TODO: default on/off, side effects, opt-in/opt-out.
+
+## Code map → DIP
+- Owner: `packages/` — read its DIP **P2 member list** (the nearest `AGENT.md`) to locate files.
+- Then follow **P3** file headers (WHO / FROM / TO / HERE) to navigate. Do **not** duplicate code paths here.
+
+## Related
+[[sdk]] [[extensions]]