@oyasmi/pipiclaw 0.5.2 → 0.5.4

package/docs/memory-optimization-round2.md

@@ -1,319 +0,0 @@
-# Memory 子系统第二轮优化计划
-
-> **Date**: 2026-04-02
-> **Base**: 第一轮优化后 (commit 845de82)
-> **Input**: memory-audit.md 审计报告 + 第一轮优化质量评估
-
----
-
-## 一、第一轮优化回顾
-
-### 已落地（保留不动）
-
-| 方案 | 实施质量 | 说明 |
-|------|---------|------|
-| 空闲触发 consolidation | A | 60s idle timer + snapshot + 自动重调度，设计精良 |
-| 中文分词改进 | A | 正向最大匹配 + bigram 兜底 + 130 词表 |
-| Rerank 参数提升 | A | timeout 8s, content clip 800 |
-| 降低 maintenance 阈值 | A | memory 5K/4块, history 8K/5块/保留3 |
-| Query Intent Detection | A | 6 种意图，中英双语，超出审计建议的创新 |
-| Consolidation prompt 强化 | B- | 仅加了"prefer"措辞，未加 few-shot |
-
-### 未落地或需改进
-
-| 方案 | 状态 | 本轮处理 |
-|------|------|---------|
-| Recall 默认配置调优 | 未实施 | 本轮实施 |
-| 评分算法仍有 priority 主导倾向 | 部分改善 | 本轮精炼 |
-| Intent seeding 可能注入无关候选 | 新发现 | 本轮修复 |
-| 运算符优先级不清晰 | 新发现 | 本轮修复 |
-| 中文词表覆盖不足 | 新发现 | 本轮扩充 |
-| 排序逻辑重复 | 新发现 | 本轮提取 |
-
----
-
-## 二、本轮优化内容
-
-### 2.1 [P0] 首轮强制注入 MEMORY.md（新策略）
-
-**动机**：当前召回依赖 token overlap + intent matching，但 session 首轮对话中用户可能只说了一句模糊的话（如"你好"、"帮我看看那个问题"），几乎不会命中任何 memory 候选。LLM 在冷启动时对频道上下文完全遗忘。
-
-**策略**：在每个 session 的第一轮对话中，无条件将 MEMORY.md 的前 3K 字符拼入 prompt，独立于召回机制。
-
-**实现要点**：
-
-1. 在 `ChannelRunner` 中新增 `isFirstUserTurn` 状态标记：
-
-```typescript
-// channel-runner.ts
-private isFirstUserTurn = true;
-
-// 在 run() 方法中，recall 之后：
-if (this.isFirstUserTurn) {
-  const memorySnapshot = await this.buildFirstTurnMemorySnapshot();
-  if (memorySnapshot) {
-    promptText = `${memorySnapshot}\n\n${promptText}`;
-  }
-  this.isFirstUserTurn = false;
-}
-```
-
-2. 构建注入文本：
-
-```typescript
-private async buildFirstTurnMemorySnapshot(): Promise<string> {
-  const memoryPath = getChannelMemoryPath(this.channelDir);
-  const raw = await readOptionalFile(memoryPath);
-  if (!raw.trim()) return "";
-
-  // 如果 recall 已返回 channel-memory 候选，跳过注入避免重复
-  // （由调用方判断，此处只负责构建文本）
-
-  const clipped = clipText(raw, 3000, { headRatio: 1.0 }); // 纯取头部
-  return [
-    "<channel_memory_snapshot>",
-    "Below is the first 3K of this channel's durable memory (MEMORY.md).",
-    "Use it as background context for this conversation.",
-    "",
-    clipped,
-    "</channel_memory_snapshot>",
-  ].join("\n");
-}
-```
-
-3. 与 recall 去重：如果 recall 结果已包含 `channel-memory` 来源的候选，则跳过强制注入。
-
-**优势**：
-- 解决冷启动遗忘问题，LLM 从第一句话就"记得"频道核心上下文
-- 实现极简（约 20 行），几乎零风险
-- MEMORY.md 头部经过 cleanup LLM 整理，天然是高价值信息（Identity/Preferences/Constraints）
-- 与 recall 互补而非冲突：recall 解决"针对具体问题找相关记忆"，强制注入解决"确保基础上下文始终在场"
-
-**风险与缓解**：
-- Token 预算占用约 1K-1.5K tokens — 钉钉短问答场景中上下文窗口充裕，影响可忽略
-- 与 recall 结果重复 — 通过检查 recall 是否已返回 channel-memory 候选来去重
-
----
-
-### 2.2 [P0] 实施 Recall 默认配置调优
-
-**动机**：第一轮审计明确建议但未实施。当前默认值过于保守。
-
-**改动位置**：`src/agent/channel-runner.ts` 中的 `recallRelevantMemory()` 调用参数，以及 `src/context.ts` 中的 `PipiclawMemoryRecallSettings` 默认值。
-
-| 参数 | 当前值 | 目标值 | 理由 |
-|------|-------|-------|------|
-| maxCandidates | 8 | 12 | 给 reranker 更多候选，提升选择质量 |
-| maxInjected | 3 | 5 | 注入更多上下文，覆盖更多相关信息 |
-| maxChars | 3500 | 5000 | 允许更长的回忆文本 |
-| rerankWithModel | false | true | 默认开启 LLM rerank，显著提升中文召回质量 |
-
----
-
-### 2.3 [P1] 评分算法精炼
-
-**问题**：当前 priority 范围 4-18，lexical 最高约 96。session（priority=18）和 history（priority=4）差距 14 分。当 query 与两个候选的词汇匹配度相同时，session 始终胜出。真正的"相关性优先"尚未实现。
-
-**方案**：改为乘法加权，structural 作为调节因子而非独立加分项。
-
-```typescript
-function scoreCandidate(...): ScoredCandidate | null {
-  // ... 计算 lexicalScore 和 structuralScore 同现有逻辑 ...
-
-  if (matchedTokens.size === 0 && exactBoost === 0 && intentBoost === 0) {
-    return null;
-  }
-
-  // 乘法加权：lexical 为主，structural 为调节
-  // structural 最高约 34（priority 18 + intent 10 + recency 6）
-  // 乘数范围约 1.0 ~ 1.34
-  const score = lexicalScore * (1 + structuralScore / 100);
-
-  return { candidate, score, lexicalMatchCount: matchedTokens.size, intentBoost };
-}
-```
-
-**效果**：
-- 当 lexical = 0 时，score = 0 — 无词汇匹配的候选不会仅靠 structural 上位
-- 当 lexical 相同时，structural 提供约 0-34% 的微调 — session 仍有优势但不再碾压
-- 当 history 候选 lexical 远高于 session 候选时，history 可以反超
-
-**注意**：`seedIntentCandidates()` 中 intentBoost > 0 但 lexicalMatchCount = 0 的候选，在乘法模式下 score 会变为 0。需要调整 seeding 逻辑（见 2.4）。
-
----
-
-### 2.4 [P1] 约束 Intent Seeding
-
-**问题**：`seedIntentCandidates()` 可注入 lexicalMatchCount=0 的候选。在 2.3 改为乘法加权后，这些候选 score 为 0，会被自然淘汰。但即使不改为乘法，纯 intent 注入也可能挤掉更相关的候选。
-
-**方案**：对 intent seeding 增加最低门槛。
-
-```typescript
-function seedIntentCandidates(
-  request: RecallRequest,
-  candidates: MemoryCandidate[],
-  existing: ScoredCandidate[],
-  intents: Set<QueryIntent>,
-  queryTokens: string[],  // 新增参数
-): ScoredCandidate[] {
-  // ...
-  for (const { candidate, intentBoost } of intentCandidates) {
-    // 至少需要 1 个 query token 匹配，才允许 intent 注入
-    const matched = collectMatchingQueryTokens(
-      queryTokens,
-      [candidate.title, candidate.searchText ?? candidate.content],
-    );
-    if (matched.size === 0) continue;
-
-    seeded.push({ ... });
-  }
-  // ...
-}
-```
-
-**效果**：意图匹配 + 至少 1 个词汇匹配 = 允许注入。纯意图匹配但内容完全无关 = 不注入。
-
----
-
-### 2.5 [P1] 扩充中文词表
-
-**问题**：当前 130 词中包含较多 pipiclaw 内部术语（"召回排序"、"排序失真"、"主导权重"），缺少大量通用技术词汇。
-
-**方案**：
-
-1. **移除**过于特定的内部术语（约 20 个），如"排序失真"、"主导权重"、"轻量词表"等
-2. **新增**通用技术词汇（约 200 个），覆盖以下领域：
-
-```
-基础设施: 服务器、数据库、缓存、队列、网络、集群、容器、网关、负载均衡
-编程概念: 函数、变量、接口、类型、模块、组件、依赖、注入、继承、多态
-开发流程: 需求、设计、实现、测试、部署、发布、回滚、监控、告警、日志
-Web/API:  请求、响应、认证、授权、会话、路由、中间件、过滤器、拦截器
-数据:     查询、索引、事务、迁移、备份、恢复、同步、异步、并发、锁
-工具:     编辑器、终端、浏览器、调试器、编译器、构建、打包、压缩
-```
-
-3. 目标词表规模约 300 个，保持文件 < 300 行
-
----
-
-### 2.6 [P1] 修复代码质量问题
-
-#### 2.6.1 运算符优先级明确化
-
-**位置**：`src/memory/lifecycle.ts:109-113`
-
-```typescript
-// 当前（行为正确但意图不清晰）
-if (
-  canTriggerThresholdRefresh &&
-  this.turnsSinceSessionUpdate >= settings.minTurnsBetweenUpdate ||
-  (canTriggerThresholdRefresh && this.toolCallsSinceSessionUpdate >= settings.minToolCallsBetweenUpdate)
-)
-
-// 修改为
-if (
-  canTriggerThresholdRefresh &&
-  (this.turnsSinceSessionUpdate >= settings.minTurnsBetweenUpdate ||
-   this.toolCallsSinceSessionUpdate >= settings.minToolCallsBetweenUpdate)
-)
-```
-
-#### 2.6.2 提取排序比较器
-
-**位置**：`src/memory/recall.ts:487-501`
-
-```typescript
-function compareScoredCandidates(a: ScoredCandidate, b: ScoredCandidate): number {
-  return (
-    b.score - a.score ||
-    b.lexicalMatchCount - a.lexicalMatchCount ||
-    b.candidate.priority - a.candidate.priority ||
-    a.candidate.title.localeCompare(b.candidate.title)
-  );
-}
-
-// 使用
-const scored = filteredCandidates.map(...).filter(...).sort(compareScoredCandidates);
-const shortlist = seedIntentCandidates(...).sort(compareScoredCandidates).slice(...);
-```
-
----
-
-### 2.7 [P2] Consolidation prompt 强化
-
-**问题**：当前 prompt 对 historyBlock 仅用"prefer"措辞，LLM 在短对话时仍可能返回空值。
-
-**方案**：添加 few-shot 示例到 system prompt。
-
-```
-Rules:
-...
-- historyBlock: concise Markdown summarizing the conversation chunk for later recovery.
-- For any conversation with at least one meaningful user question and assistant answer,
-  you MUST return a non-empty historyBlock with at least one bullet summarizing the exchange.
-- Prefer short bullets and short paragraphs.
-
-Example output for a short Q&A conversation:
-{
-  "memoryEntries": ["User prefers dark mode in the dashboard"],
-  "historyBlock": "- User asked how to toggle dashboard theme; confirmed dark mode preference."
-}
-```
-
----
-
-### 2.8 [P2] Sidecar Token 追踪
-
-**问题**：memory 系统通过 sidecar 发起的 LLM 调用（session memory update、inline consolidation、memory cleanup、history folding、LLM rerank）没有 token 使用量统计，无法评估实际 API 开销。
-
-**方案**：
-
-1. `runSidecarTask` 返回值中增加 `usage?: { inputTokens: number; outputTokens: number }`
-2. `MemoryLifecycle` 中累计追踪每类操作的 token 消耗
-3. 在 background maintenance 日志中输出累计 token 消耗
-
----
-
-## 三、实施顺序
-
-```
-Phase 1（核心功能）
-  2.1  首轮强制注入 MEMORY.md
-  2.2  Recall 默认配置调优
-  2.6  代码质量修复（运算符优先级 + 排序提取）
-
-Phase 2（评分优化）
-  2.3  评分乘法加权
-  2.4  Intent seeding 约束
-  2.5  中文词表扩充
-
-Phase 3（可靠性）
-  2.7  Consolidation prompt 强化
-  2.8  Sidecar token 追踪
-```
-
----
-
-## 四、验证标准
-
-### 功能验证
-
-- [ ] 首轮对话发送模糊消息（如"你好"），prompt 中应包含 `<channel_memory_snapshot>` 标签
-- [ ] 第二轮对话不再包含强制注入内容
-- [ ] 当 recall 已返回 channel-memory 候选时，跳过强制注入
-- [ ] rerankWithModel 默认启用，中文查询自动触发 rerank
-
-### 评分验证
-
-- [ ] history 候选在 lexical 高度匹配时可以排在不相关 session 候选之前
-- [ ] lexical = 0 的候选不会仅靠 structural 上位（乘法加权后）
-- [ ] intent seeding 不会注入与 query 零词汇匹配的候选
-
-### 回归验证
-
-- [ ] 全部现有测试通过
-- [ ] 新增首轮注入的单元测试
-- [ ] 新增 scoreCandidate 乘法加权的单元测试
-- [ ] 新增中文短查询（2-3 字）的召回集成测试
-

package/docs/specs/001-implement-memory/memory-rfc.md

@@ -1,297 +0,0 @@
-# Pipiclaw Memory Model RFC
-
-## Status
-
-Superseded in part by [`docs/improve-memory/spec.md`](/Users/oyasmi/projects/pipiclaw/docs/improve-memory/spec.md) and [`docs/improve-memory/design.md`](/Users/oyasmi/projects/pipiclaw/docs/improve-memory/design.md).
-
-This RFC remains useful for the original file-based memory rationale, but it no longer fully describes the current design because the runtime now introduces:
-
-1. `SESSION.md` as a distinct channel working-memory layer
-2. proactive relevant-memory injection
-3. a revised lifecycle between `SESSION.md`, `MEMORY.md`, and `HISTORY.md`
-
-## Goals
-
-1. Define a minimal memory model for `pipiclaw` aligned with the `nanobot` style.
-2. Make memory flow explicit: record, read, consolidate, and reuse.
-3. Keep raw transport/session files cold and separate from memory.
-4. Limit what is loaded into session context by default.
-5. Preserve simple file-based operations and avoid introducing a dedicated memory database or memory-specific tools.
-
-## Non-Goals
-
-1. No OpenClaw-style vector retrieval, embedding index, or memory plugins.
-2. No `memory_search` or `memory_get` tool.
-3. No automatic runtime mutation of workspace-level `MEMORY.md`, `SOUL.md`, or `AGENTS.md`.
-4. No automatic loading or scanning of `log.jsonl` or `context.jsonl` as part of memory behavior.
-
-## Design Summary
-
-Pipiclaw memory is split into two channel-level files plus one workspace-level admin file:
-
-- `workspace/MEMORY.md`
-  - Stable, admin-managed, shared background memory.
-  - Not automatically updated by runtime consolidation.
-- `<channel>/MEMORY.md`
-  - Durable channel memory.
-  - Updated automatically by consolidation.
-  - May also be updated manually by the agent.
-- `<channel>/HISTORY.md`
-  - Channel history summaries.
-  - Updated automatically by consolidation only.
-  - Not intended for direct manual maintenance by the agent.
-
-Raw storage remains separate:
-
-- `<channel>/log.jsonl`
-  - Raw message log only.
-  - Cold storage.
-  - Not proactively loaded or scanned by runtime.
-- `<channel>/context.jsonl`
-  - Raw session persistence only.
-  - Cold storage.
-  - Not proactively loaded or scanned by runtime for memory purposes.
-
-## File Model
-
-### Workspace-Level Files
-
-- `SOUL.md`
-  - Loaded into session context at session start.
-  - Read-only from the agent's perspective unless a human explicitly changes it.
-- `AGENTS.md`
-  - Loaded into session context at session start.
-  - Read-only from the agent's perspective unless a human explicitly changes it.
-- `MEMORY.md`
-  - Not loaded by default.
-  - Listed in the system prompt with its role and path.
-  - Intended to be read on demand by the agent.
-  - Stable and admin-managed.
-
-### Channel-Level Files
-
-- `MEMORY.md`
-  - Not loaded by default.
-  - Listed in the system prompt with its role and path.
-  - Intended to be read on demand by the agent.
-  - Primary durable memory for the channel.
-- `HISTORY.md`
-  - Not loaded by default.
-  - Listed in the system prompt with its role and path.
-  - Intended to be read on demand by the agent.
-  - Append-oriented summary history for older context.
-- `log.jsonl`
-  - Raw archive only.
-  - Not memory.
-  - Runtime must not proactively load or scan it.
-- `context.jsonl`
-  - Raw session archive only.
-  - Not memory.
-  - Runtime must not proactively load or scan it.
-
-## Default Context Loading
-
-At session start, runtime loads the following into session context:
-
-1. Workspace-level `SOUL.md`
-2. Workspace-level `AGENTS.md`
-3. Built-in tool descriptions
-4. Summaries of both workspace-level and channel-level skills
-
-At session start, runtime does not load the following by default:
-
-1. Workspace-level `MEMORY.md`
-2. Channel-level `MEMORY.md`
-3. Channel-level `HISTORY.md`
-4. `<channel>/log.jsonl`
-5. `<channel>/context.jsonl`
-
-The system prompt must explicitly tell the agent:
-
-1. Where `workspace/MEMORY.md`, `<channel>/MEMORY.md`, and `<channel>/HISTORY.md` live.
-2. What each file is for.
-3. That these files are not preloaded.
-4. That the agent is encouraged to read them on demand when memory or history is relevant.
-
-Changes to workspace-level `SOUL.md` and `AGENTS.md` take effect on new sessions. They are not reloaded on every turn.
-
-## Agent Read/Write Rules
-
-### Reads
-
-The agent should prefer:
-
-1. `<channel>/MEMORY.md` for durable channel facts, decisions, preferences, and ongoing state.
-2. `<channel>/HISTORY.md` for older summarized context.
-3. `workspace/MEMORY.md` for stable shared background.
-
-The agent should not treat `log.jsonl` or `context.jsonl` as normal memory sources. Those files are for raw recovery, debugging, or explicit transcript-level investigation only.
-
-### Writes
-
-The agent may:
-
-1. Read `workspace/MEMORY.md`
-2. Read `<channel>/MEMORY.md`
-3. Read `<channel>/HISTORY.md`
-4. Manually update `<channel>/MEMORY.md` when necessary
-
-The agent should not manually maintain `<channel>/HISTORY.md` during normal operation. `HISTORY.md` is runtime-managed.
-
-The runtime must never automatically update:
-
-1. `workspace/SOUL.md`
-2. `workspace/AGENTS.md`
-3. `workspace/MEMORY.md`
-
-## Consolidation
-
-Consolidation is the primary mechanism that keeps memory moving.
-
-Consolidation takes recent session material and produces two outputs:
-
-1. Durable facts and live channel state in `<channel>/MEMORY.md`
-2. Older summarized history in `<channel>/HISTORY.md`
-
-Consolidation must not write to workspace-level `MEMORY.md`.
-
-### Trigger Model
-
-Runtime should run consolidation automatically only when a session is about to compact or trim context.
-
-There is no separate turn-count trigger, token-watermark trigger, or per-turn memory-dirty trigger in this RFC.
-
-### Consolidation Input
-
-Consolidation operates on recent session material already in the active session state. It does not scan `log.jsonl` or `context.jsonl`.
-
-### Consolidation Output Rules
-
-`<channel>/MEMORY.md` should contain:
-
-1. Durable facts about the channel, user, project, preferences, constraints, and long-lived open threads.
-2. Current state that matters across turns.
-3. Cleaned and deduplicated entries.
-4. Structured sections rather than free-form prose.
-
-`<channel>/HISTORY.md` should contain:
-
-1. Append-oriented summaries of older conversation chunks.
-2. Resolved decisions and notable milestones.
-3. Enough context for later recovery without replaying raw transcripts.
-
-### Consolidation Semantics
-
-Consolidation should:
-
-1. Extract durable facts from recent session material.
-2. Append new channel-memory entries into `<channel>/MEMORY.md` during normal operation.
-3. Periodically clean up `<channel>/MEMORY.md` with a larger sweep that removes outdated entries, merges duplicates, and tightens wording.
-4. Append or roll forward summarized older material into `<channel>/HISTORY.md`.
-5. Periodically fold older `HISTORY.md` blocks into coarser summaries while keeping newer blocks more detailed.
-6. Remove redundancy between the two files.
-7. Prefer stable facts in `MEMORY.md` and narrative progression in `HISTORY.md`.
-
-This RFC adopts an append-first strategy for `MEMORY.md`, with periodic cleanup passes rather than full rewrite on every consolidation.
-
-Consolidation should not:
-
-1. Dump raw message transcripts into `HISTORY.md`
-2. Copy large blocks from `context.jsonl` or `log.jsonl`
-3. Preserve outdated facts just because they were once true
-
-## Suggested File Semantics
-
-`<channel>/MEMORY.md` should be organized as stable sections such as:
-
-1. `## Identity / Participants`
-2. `## Preferences`
-3. `## Ongoing Work`
-4. `## Constraints`
-5. `## Decisions`
-6. `## Open Loops`
-
-`<channel>/HISTORY.md` should be organized as chronological summary blocks such as:
-
-1. Dated headings
-2. Short summaries of work periods
-3. Key decisions and outcomes
-
-Exact formatting can evolve, but the split between durable memory and summarized history should remain stable.
-
-## Consolidation Execution Model
-
-Consolidation uses a two-phase execution model:
-
-1. Inline phase
-   - Runs synchronously when a compaction or trim is about to happen.
-   - Responsible for producing the minimum safe memory updates needed before context is reduced.
-2. Background phase
-   - Runs later as a per-channel queued maintenance pass.
-   - Responsible for larger cleanup work, including `MEMORY.md` cleanup sweeps and `HISTORY.md` folding.
-
-The background phase must not replace the inline phase for compaction safety. It is an additional maintenance pass, not the primary durability guarantee.
-
-## Compaction And Trimming Contract
-
-Consolidation is hard-gated before compaction or trim.
-
-This means:
-
-1. If a compaction or trim would discard context, runtime must first run inline consolidation.
-2. If inline consolidation fails, compaction or trim must not proceed.
-3. After successful inline consolidation, the runtime may compact or trim.
-
-This RFC does not allow trim-first or compact-first behavior when memory durability depends on consolidation.
-
-## Runtime Responsibilities
-
-Runtime is responsible for:
-
-1. Loading only the default context described in this RFC.
-2. Exposing file locations and roles clearly in the system prompt.
-3. Running automatic consolidation.
-4. Updating `<channel>/MEMORY.md` and `<channel>/HISTORY.md` during consolidation.
-5. Keeping `log.jsonl` and `context.jsonl` cold.
-
-Runtime is not responsible for:
-
-1. Auto-loading memory files into every turn
-2. Indexing memory into a vector database
-3. Providing special memory-only tools
-
-## Session Model
-
-Session context is warm.
-
-Memory files are warm-on-demand.
-
-`log.jsonl` and `context.jsonl` are cold.
-
-The intended flow is:
-
-1. Session starts with `SOUL.md`, `AGENTS.md`, built-in tool descriptions, and skill summaries.
-2. Agent reads channel or workspace memory files if the task needs them.
-3. Conversation progresses in session state.
-4. Runtime consolidates session material into `<channel>/MEMORY.md` and `<channel>/HISTORY.md`.
-5. Older raw storage remains available but is not part of ordinary memory behavior.
-
-## Migration Impact
-
-This RFC implies the following behavior changes from the current implementation:
-
-1. `workspace/MEMORY.md` and `<channel>/MEMORY.md` are no longer injected into the system prompt by default.
-2. `SOUL.md` and `AGENTS.md` are no longer reloaded on every turn.
-3. `log.jsonl` is no longer scanned to rebuild context as part of normal turn processing.
-4. `context.jsonl` is no longer treated as a memory source.
-5. Consolidation becomes the main persistence path for channel memory.
-
-## Fixed Decisions
-
-This RFC fixes the following design choices:
-
-1. Consolidation triggers only before compaction or trim.
-2. `<channel>/MEMORY.md` uses append-first updates with periodic cleanup sweeps.
-3. `<channel>/HISTORY.md` uses append-first updates with periodic folding of older blocks.
-4. Consolidation uses a two-phase execution model: inline for safety, background for maintenance.
-5. Consolidation is hard-gated before compaction or trim.