@oyasmi/pipiclaw 0.4.0 → 0.5.1

package/docs/improve-memory/spec.md

@@ -0,0 +1,357 @@
+# Pipiclaw Context Upgrade Spec
+
+## Status
+
+Draft
+
+## Purpose
+
+This spec defines the next-stage context architecture for `pipiclaw`.
+
+It extends the existing [memory RFC](/Users/oyasmi/projects/pipiclaw/docs/memory-rfc.md) rather than replacing it wholesale.
+
+The primary goal is not "more memory files". The goal is to materially improve:
+
+1. cross-turn continuity
+2. long-task stability
+3. compaction resilience
+4. sub-agent usefulness
+5. skill usefulness in a long-running DingTalk channel workspace
+
+The design remains file-based and runtime-local. It does not introduce embeddings, vector search, or a dedicated memory database.
+
+## Problems To Solve
+
+The current model is good enough to persist durable facts, but still leaves several gaps:
+
+1. `MEMORY.md` and `HISTORY.md` are available but not proactively surfaced, so the model often fails to use them at the right time.
+2. "current working state" is mixed into durable memory and is therefore either too weak, too stale, or too noisy.
+3. compaction happens with only indirect help from memory consolidation, which is weaker than having an explicit working-state artifact.
+4. sub-agents are context-isolated, but there is no standard way to give them the most relevant channel memory automatically.
+5. skills are loaded, but they do not yet participate in a richer lifecycle where memory and runtime hooks reinforce one another.
+
+## Non-Goals
+
+1. No vector retrieval, embedding index, semantic database, or memory plugin framework.
+2. No special standalone `memory_search` tool for the model.
+3. No automatic mutation of workspace-level `SOUL.md`, `AGENTS.md`, or workspace `MEMORY.md`.
+4. No proactive scanning of `log.jsonl` or `context.jsonl` as normal memory inputs.
+5. No attempt to clone the full `claude-code` agent OS, swarm runtime, or prompt-cache infrastructure in this phase.
+
+## Context Layers
+
+The upgraded context model has five layers.
+
+### 1. Identity Layer
+
+- `workspace/SOUL.md`
+- `workspace/AGENTS.md`
+
+Semantics:
+
+- loaded into session context at session start
+- human-managed
+- authoritative
+- not rewritten by runtime maintenance
+
+### 2. Shared Durable Memory
+
+- `workspace/MEMORY.md`
+
+Semantics:
+
+- stable shared background
+- read on demand
+- not auto-rewritten by runtime
+
+### 3. Channel Durable Memory
+
+- `<channel>/MEMORY.md`
+
+Semantics:
+
+- durable facts, decisions, preferences, constraints, medium-horizon open loops
+- append-first, then cleanup/fold
+- channel-scoped
+- runtime-managed
+
+Important change:
+
+`MEMORY.md` is no longer the primary owner of minute-by-minute "what am I doing right now" state. It may still contain open loops, but detailed active execution state belongs to `SESSION.md`.
+
+### 4. Channel Working Memory
+
+- `<channel>/SESSION.md`
+
+Semantics:
+
+- current task state
+- active files and commands
+- current hypotheses and next steps
+- recent important errors and corrections
+- channel-scoped handoff artifact across sessions and compactions
+- runtime-managed
+
+This is the major addition in this spec.
+
+### 5. Channel History
+
+- `<channel>/HISTORY.md`
+
+Semantics:
+
+- summarized chronological recovery material
+- older work periods, decisions, milestones
+- runtime-managed
+- not intended for normal manual maintenance
+
+## Cold Storage
+
+These files remain cold:
+
+- `<channel>/log.jsonl`
+- `<channel>/context.jsonl`
+
+They are not part of normal memory recall and are not proactively loaded into prompts.
+
+## Core Invariants
+
+The upgraded design must preserve these invariants:
+
+1. File-based operation remains the source of truth.
+2. The model should not need to remember to manually read the right memory file in common cases.
+3. `SESSION.md` is the working-state artifact.
+4. `MEMORY.md` is the durable-facts artifact.
+5. `HISTORY.md` is the chronological-summary artifact.
+6. Workspace memory remains human-managed.
+7. Raw transport/session archives remain cold.
+8. Failure of a background updater must not corrupt persisted memory files.
+9. The system must degrade gracefully when an updater fails.
+10. Memory improvements must help sub-agents and skills, not just the main agent.
+
+## File Semantics
+
+### `SESSION.md`
+
+`SESSION.md` is channel-scoped hot memory.
+
+It should answer:
+
+1. What is the user currently trying to achieve?
+2. What is the current state of work?
+3. Which files and commands matter right now?
+4. What constraints and recent failures should not be forgotten?
+5. What are the next likely steps?
+
+It should not become:
+
+1. a raw transcript
+2. a duplicate of all durable facts already in `MEMORY.md`
+3. an infinite worklog
+
+Read/write rule:
+
+1. runtime-managed by default
+2. eligible for automatic targeted recall
+3. not intended for normal manual maintenance by the main agent
+4. may be edited only when an explicit user/admin instruction makes that the task itself
+
+### `MEMORY.md`
+
+`MEMORY.md` remains the durable channel memory, but the threshold for what belongs here gets stricter.
+
+It should keep:
+
+1. stable preferences
+2. durable decisions
+3. medium-horizon constraints
+4. important open loops that must survive beyond the current execution burst
+
+It should avoid:
+
+1. step-by-step active worklog
+2. temporary local debugging observations unless they have lasting value
+3. detailed "current state" that will likely be obsolete after a few turns
+
+Read/write rule:
+
+1. readable on demand
+2. writable by runtime consolidation
+3. still manually writable when necessary
+4. cleanup is allowed to remove transient state that should now live in `SESSION.md`
+
+### `HISTORY.md`
+
+`HISTORY.md` remains the recovery narrative.
+
+It should keep:
+
+1. notable work periods
+2. milestones
+3. decision outcomes
+4. enough chronology for later recovery
+
+It should avoid:
+
+1. dense per-turn detail
+2. raw snippets copied from transcript
+3. facts better represented in `MEMORY.md`
+
+Read/write rule:
+
+1. readable on demand
+2. runtime-managed
+3. not intended for ordinary manual edits
+
+## Closed-Loop Lifecycle
+
+The upgraded model introduces an explicit closed loop:
+
+1. active session work happens in warm context
+2. recent work updates `SESSION.md`
+3. stable facts and medium-horizon open loops are promoted into `MEMORY.md`
+4. older narrative is summarized into `HISTORY.md`
+5. future prompts receive targeted recall from `SESSION.md`, `MEMORY.md`, and `HISTORY.md`
+
+This loop must work even when:
+
+1. the session compacts
+2. the user starts a new session in the same channel
+3. the process restarts
+4. the main agent delegates work to a sub-agent
+
+## Recall Model
+
+The system should stop relying purely on "the model may remember to read memory files".
+
+Instead, each turn may inject a small amount of relevant memory context chosen from:
+
+1. `SESSION.md`
+2. channel `MEMORY.md`
+3. workspace `MEMORY.md`
+4. channel `HISTORY.md`
+
+Selection rules:
+
+1. small budget
+2. high precision
+3. recency-aware
+4. section-aware
+5. prefer `SESSION.md` when current work state matters
+6. prefer `MEMORY.md` when durable constraints or preferences matter
+7. prefer `HISTORY.md` when recovery of older narrative matters
+
+## SESSION.md Lifecycle Contract
+
+`SESSION.md` must follow this lifecycle:
+
+1. created automatically with the channel memory files
+2. not loaded wholesale into every prompt
+3. eligible for targeted recall injection
+4. updated in the background during normal work
+5. synchronously refreshed before context-reduction boundaries when possible
+6. carried across `/new` session boundaries within the same channel
+7. cleaned and condensed periodically so it remains current
+8. recreated automatically if missing on an old channel directory
+
+Important semantic rule:
+
+`/new` starts a new model session, but does not imply "forget current channel work". `SESSION.md` is allowed to survive `/new` if the channel still has active work.
+
+## Relationship Between SESSION.md And Existing Memory Files
+
+The relationship is:
+
+1. `SESSION.md` owns high-churn working state
+2. `MEMORY.md` owns durable and semi-durable channel knowledge
+3. `HISTORY.md` owns older narrative recovery
+
+Promotion rules:
+
+1. stable facts discovered in `SESSION.md` may be promoted into `MEMORY.md`
+2. resolved work periods may be summarized into `HISTORY.md`
+3. stale transient content should be removed from `SESSION.md`
+4. cleanup may also remove overly transient content that had historically accumulated in `MEMORY.md`
+
+This lets the system gradually migrate away from using channel `MEMORY.md` as the sole carrier of both durable and active state.
+
+## Source Of Truth Precedence
+
+When the same topic appears in multiple files, runtime and prompts should treat them in this precedence order:
+
+1. `SESSION.md` for current active work state
+2. `MEMORY.md` for durable constraints and decisions
+3. `HISTORY.md` for older chronology
+
+Practical implication:
+
+1. a stale `MEMORY.md` note must not override a fresher `SESSION.md` current-state section
+2. a stale `HISTORY.md` block must not override a fresher durable decision in `MEMORY.md`
+
+## Compaction Contract
+
+Compaction should become `SESSION.md`-aware.
+
+The preferred order is:
+
+1. refresh `SESSION.md`
+2. run inline durable consolidation into `MEMORY.md` and `HISTORY.md`
+3. compact using the latest `SESSION.md` as part of the retained context strategy
+
+Graceful degradation rules:
+
+1. if `SESSION.md` refresh fails, use the last persisted `SESSION.md`
+2. if durable consolidation fails, compaction may still proceed if `SESSION.md` is available
+3. failures must be logged and retried in background maintenance
+
+This keeps compaction safe without turning memory maintenance into a hard availability dependency.
+
+## Migration Contract
+
+Existing channel directories already containing only `MEMORY.md` and `HISTORY.md` must migrate safely.
+
+Migration rules:
+
+1. missing `SESSION.md` is treated as normal and repaired by ensure-file bootstrap
+2. existing `MEMORY.md` and `HISTORY.md` content remains authoritative
+3. no one-time destructive rewrite of historical `MEMORY.md` content is required
+4. transient content historically living in `MEMORY.md` is cleaned gradually by normal maintenance
+
+This keeps rollout incremental and low-risk for live DingTalk channels.
+
+## Sub-Agent Contract
+
+Sub-agents remain isolated by default, but the runtime gains a standard way to provide them with the right memory context.
+
+Two modes should exist:
+
+1. isolated
+   - current behavior
+   - only runtime basics plus the explicitly provided task
+2. contextual
+   - runtime prepends relevant memory and working-state context automatically
+
+The contextual mode should pull from the same recall pipeline as the main agent, but with stricter budgets.
+
+## Skill Contract
+
+Skills should become lightweight participants in the upgraded context system.
+
+Expected improvements:
+
+1. richer frontmatter describing when a skill should be used
+2. optional scoped hooks for session lifecycle events
+3. optional declaration of memory relevance or path relevance
+
+The skill system is still intentionally lighter than the `claude-code` version. The goal is stronger reuse and better context shaping, not a full plugin platform.
+
+## Rollout Principle
+
+This spec is designed for staged implementation:
+
+1. first improve recall
+2. then add `SESSION.md`
+3. then bridge compaction
+4. then upgrade sub-agent and skill integration
+
+At each stage, existing `MEMORY.md` and `HISTORY.md` behavior must continue to work.

package/docs/memory-rfc.md

@@ -2,7 +2,13 @@
 
 ## Status
 
-Draft
+Superseded in part by [`docs/improve-memory/spec.md`](/Users/oyasmi/projects/pipiclaw/docs/improve-memory/spec.md) and [`docs/improve-memory/design.md`](/Users/oyasmi/projects/pipiclaw/docs/improve-memory/design.md).
+
+This RFC remains useful for the original file-based memory rationale, but it no longer fully describes the current design because the runtime now introduces:
+
+1. `SESSION.md` as a distinct channel working-memory layer
+2. proactive relevant-memory injection
+3. a revised lifecycle between `SESSION.md`, `MEMORY.md`, and `HISTORY.md`
 
 ## Goals
 

package/docs/proj-review.md

@@ -0,0 +1,188 @@
+Pipiclaw 项目全面审查报告
+
+  项目概述
+
+  Pipiclaw 是一个基于 @mariozechner/pi-coding-agent SDK 的 DingTalk AI 编码助手运行时，提供持久化记忆、子 Agent 委派、定时事件等企业级功能。当前 v0.4.0，约 6,844 行
+  TypeScript，22 个测试文件。
+
+  ---
+  一、当前设计/实现问题
+
+  1. 代码重复严重
+
+  extractJsonObject() 在 3 处重复实现：
+  - memory-consolidation.ts
+  - memory-recall.ts
+  - session-memory.ts
+
+  clipText() 在 3 处重复：
+  - session-memory.ts
+  - tools/subagent.ts
+  - memory-recall.ts
+
+  Markdown section 解析存在 3 个变体：
+  - splitMarkdownSections() (## 级别) — memory-files.ts
+  - splitLevelOneSections() — tools/subagent.ts
+  - splitLevelOneSections() — memory-candidates.ts
+
+  建议： 提取到 src/utils/ 共享模块，统一实现。
+
+  ---
+  2. 核心模块职责过重
+
+  ┌───────────────────┬──────┬────────────────────────────────────────────────────────────────────────────┐
+  │       文件        │ 行数 │                                    问题                                    │
+  ├───────────────────┼──────┼────────────────────────────────────────────────────────────────────────────┤
+  │ agent.ts          │ 907  │ Session 管理 + 事件订阅 + 消息格式化 + 记忆生命周期 + 工具配置全部混在一起 │
+  ├───────────────────┼──────┼────────────────────────────────────────────────────────────────────────────┤
+  │ dingtalk.ts       │ 881  │ 协议处理 + 消息队列 + AI Card 状态 + Token 缓存耦合                        │
+  ├───────────────────┼──────┼────────────────────────────────────────────────────────────────────────────┤
+  │ sub-agents.ts     │ 511  │ 发现 + 配置解析 + 验证 + 合并逻辑混杂                                      │
+  ├───────────────────┼──────┼────────────────────────────────────────────────────────────────────────────┤
+  │ tools/subagent.ts │ ~600 │ 配置解析 + 上下文构建 + 工具过滤 + Worker 创建 + 事件处理 + 预算跟踪       │
+  └───────────────────┴──────┴────────────────────────────────────────────────────────────────────────────┘
+
+  agent.ts 中的 ChannelRunner 承担了太多职责，既是会话编排器，又是事件分发器和记忆协调器。
+
+  ---
+  3. 类型安全缺陷
+
+  - 大量 as any 类型断言：agent.ts 的事件处理中频繁使用 event as any，缺少正确的联合类型定义
+  - noExplicitAny: Off：biome 配置关闭了 any 检查，降低了类型安全性
+  - JSON 提取使用正则：extractJsonObject() 基于正则匹配 {...} 非常脆弱，嵌套 JSON、字符串中的花括号等边界情况容易出错
+  - Partial 配置合并：使用展开运算符合并配置，缺少运行时校验
+
+  ---
+  4. 记忆系统设计问题
+
+  4.1 召回算法过于简单
+
+  当前的 token overlap 打分方式本质上是关键词匹配：
+  - 按空格分词，匹配命中数累加
+  - 不理解语义相似性（如 "登录" 和 "认证" 无法关联）
+  - 中文分词完全缺失（中文 token 化基于 \W+ 分割，对中文几乎无效）
+
+  4.2 合并管道缺少超时保护
+
+  runInlineConsolidation() 和 runBackgroundMaintenance() 依赖 LLM sidecar worker，但没有超时机制。如果 LLM 响应缓慢或挂起，整个管道会无限等待。
+
+  4.3 记忆候选者无缓存
+
+  buildMemoryCandidates() 每次调用都读取 4 个文件并解析。在一个 run 内可能被多次调用（recall + consolidation），但每次都从磁盘重新读取。
+
+  4.4 SESSION.md 的 LLM 更新缺乏一致性保证
+
+  session memory 更新依赖 LLM 生成结构化 JSON，但 LLM 输出格式不稳定，可能导致 SESSION.md 内容退化或丢失关键信息。
+
+  ---
+  5. 测试覆盖缺口
+
+  未测试的关键路径：
+
+  ┌───────────────────────────────────┬──────────────────────────────┐
+  │               模块                │             影响             │
+  ├───────────────────────────────────┼──────────────────────────────┤
+  │ agent.ts (ChannelRunner 核心编排) │ 最核心的业务逻辑完全没有单测 │
+  ├───────────────────────────────────┼──────────────────────────────┤
+  │ dingtalk.ts (连接/重连/消息路由)  │ 生产环境最易出问题的部分     │
+  ├───────────────────────────────────┼──────────────────────────────┤
+  │ main.ts (启动流程/配置校验)       │ 首次启动失败无法快速定位     │
+  ├───────────────────────────────────┼──────────────────────────────┤
+  │ delivery.ts (响应投递)            │ 消息丢失/重复风险            │
+  ├───────────────────────────────────┼──────────────────────────────┤
+  │ 完整消息流的集成测试              │ 无法验证端到端行为           │
+  └───────────────────────────────────┴──────────────────────────────┘
+
+  现有测试质量良好（工厂模式、临时目录清理、边界情况），但覆盖面集中在工具层和记忆文件 I/O，编排层几乎空白。
+
+  ---
+  6. 错误处理与韧性
+
+  - 用户消息大小无限制：恶意或意外的超大消息可以直接注入 prompt
+  - 并发控制不够健壮：每个 channel 的 channelStates 依赖简单的状态标记，竞态条件下可能出现重复 run
+  - 磁盘写入没有容量检查：log.jsonl 无上限增长，memory 文件合并失败时没有回退机制
+  - 子 Agent 异常传播不清晰：子 Agent 超时/失败的错误信息格式化后丢失了原始堆栈
+
+  ---
+  7. 架构耦合
+
+  - DingTalk 耦合深入核心：DingTalkContext 类型贯穿 agent.ts、delivery.ts、commands.ts，使得支持其他 IM 平台（飞书、企业微信）成本很高
+  - 记忆系统与文件系统强绑定：所有记忆操作直接读写 Markdown 文件，无抽象层，难以切换到数据库或向量存储
+  - 配置路径硬编码：paths.ts 中的 ~/.pi/pipiclaw/ 路径硬编码，不利于多实例部署
+
+  ---
+  8. 其他问题
+
+  - release.yml 使用 Node 20，但项目要求 >=22，CI 测试也只跑 22/24
+  - attach.ts 是空壳：抛出 not implemented 错误，但已导出为公开 API
+  - shell-escape.ts 仅 7 行：功能过于简单，不处理特殊字符（如 \0、unicode）
+  - 无日志轮转：log.jsonl 和 context.jsonl 会无限增长
+
+  ---
+  二、进一步迭代方向
+
+  高优先级
+
+  1. 拆分 ChannelRunner
+    - 将 agent.ts 拆为：session-orchestrator.ts（会话生命周期）、event-dispatcher.ts（事件流处理）、tool-configurator.ts（工具组装）
+    - 使核心编排可测试
+  2. 引入传输层抽象
+    - 定义 Transport 接口（send/receive/stream），DingTalk 作为一个实现
+    - 为未来支持飞书、Slack、CLI 等铺路
+  3. 修复中文记忆召回
+    - 当前分词对中文无效，需引入分词库（如 jieba-wasm）或转向 embedding 向量召回
+    - 至少作为 rerank 阶段的补充
+  4. 补全编排层测试
+    - 为 ChannelRunner 的 run() 流程编写集成测试
+    - Mock AgentSession，验证记忆召回 → prompt 构建 → 事件处理的完整链路
+  5. 提取共享工具函数
+    - extractJsonObject()、clipText()、splitMarkdownSections() 统一到 src/utils/
+
+  中优先级
+
+  6. 记忆系统改进
+    - 为 sidecar worker 添加超时（30s-60s）
+    - 记忆候选者添加 run 级别缓存
+    - SESSION.md 更新增加 schema 校验和回退机制
+    - 考虑引入 embedding 存储做语义检索
+  7. 添加防护措施
+    - 用户消息长度限制
+    - log.jsonl 轮转（按大小或时间）
+    - 磁盘写入前检查可用空间
+    - 并发 run 的互斥锁（替代状态标记）
+  8. 类型安全加固
+    - 定义 AgentEvent 联合类型，消除 as any
+    - 启用 noExplicitAny，逐步修复
+    - JSON 提取改用 proper parser（如先找到平衡的 {} 再 JSON.parse）
+  9. 可观测性增强
+    - 结构化日志（JSON 格式）替代当前的 chalk 彩色输出
+    - 关键操作添加 metrics（记忆召回耗时、合并频率、子 Agent 使用率）
+    - 健康检查端点
+
+  低优先级
+
+  10. 记忆存储抽象
+    - 定义 MemoryStore 接口（read/write/query），当前文件系统作为默认实现
+    - 为未来 SQLite / 向量数据库做准备
+  11. 子 Agent 改进
+    - 支持子 Agent 间通信
+    - 子 Agent 结果的结构化输出（而非纯文本）
+    - 子 Agent 池化（避免每次创建新实例）
+  12. release.yml 修复
+    - Node 版本改为 22，与 engines 字段一致
+  13. 文档补充
+    - 架构决策记录（ADR）
+    - 记忆系统的运维指南（如何手动清理/重建）
+    - 子 Agent 开发指南
+
+  ---
+  三、总结
+
+  Pipiclaw 在 v0.4.0 阶段已具备清晰的分层架构、完善的记忆管道、灵活的子 Agent 系统。主要短板集中在：
+
+  1. 核心编排层 (agent.ts) 过于臃肿且缺少测试 — 这是最大风险
+  2. 记忆召回对中文场景基本失效 — 作为面向钉钉的中文产品这是关键缺陷
+  3. DingTalk 耦合过深 — 限制了平台扩展能力
+  4. 代码重复和类型安全 — 影响长期维护效率
+
+  建议下一阶段优先处理 ChannelRunner 拆分 + 中文召回修复 + 编排层测试补全，这三项能同时降低风险和提升产品质量。