stable-harness 0.0.7 → 0.0.9

package/docs/memory/0.1.0-memory-design.zh.md

@@ -0,0 +1,719 @@
+# 0.1.0 Memory 设计说明
+
+## 目标
+
+`stable-harness` 的 memory 设计目标是让 agent workspace 拥有可控、可插拔、可长期测试的长期记忆能力，同时不重写 DeepAgents 的执行语义。
+
+当前设计同时支持两类 memory：
+
+1. `LangMem`：`stable-harness` 主路径。由 runtime 在 run 前召回、run 后抽取，交给 LangMem-style service 持久化到 SQLite 和 vector index。
+2. `deepagentsMem`：DeepAgents 原生 memory passthrough。通过 DeepAgents 的 `memory` 参数暴露 `/memories/<id>.md`，由 DeepAgents 自己读取或写入 memory 文件。
+
+这两类 memory 使用同一组 workspace Memory YAML 定义，但通过四个 boolean flag 独立控制读写。
+
+## 非目标
+
+- 不把 memory 做成领域路由规则。
+- 不在 core runtime 中 hardcode EasyNet、股票、Kubernetes、release 等 downstream 逻辑。
+- 不把 vector store 当成唯一事实源。
+- 不让 DeepAgents native memory 和 LangMem structured memory 互相依赖。
+- 不依赖 system prompt 作为唯一治理手段。
+- 不在 adapter 中重写 DeepAgents 的 planning、delegation、filesystem、memory 执行语义。
+
+## 核心概念
+
+| 概念 | 说明 |
+|---|---|
+| `Memory` YAML | 声明一个 memory profile，例如 `workspace`、`user`、`agent`、`session`。 |
+| `profile` | 决定 namespace 和抽取边界。常见值为 `workspace`、`user`。 |
+| `LangMem` | 后台抽取、去重、持久化、vector recall 的主 memory path。 |
+| `deepagentsMem` | DeepAgents 原生 file memory passthrough。 |
+| `memory_records` | SQLite 中的结构化 memory 事实表。 |
+| `memory_vectors` | SQLite 中的 embedding JSON 表，用于 vector search。 |
+| `metadata.memoryWrite=false` | 单次 request 只 recall，不写回 memory。 |
+
+## 四个 Boolean Flag
+
+runtime 配置面保持简单，只暴露四个 boolean：
+
+```yaml
+memory:
+  deepagentsMem:
+    read: true
+    write: true
+  LangMem:
+    read: true
+    write: true
+  maintenance:
+    enabled: true
+    schedule: periodic
+    prompt: durable preferences, workspace facts, reusable procedures, prior corrections, stale duplicates, and contradicted memories
+```
+
+| Flag | 默认建议 | 作用 |
+|---|---:|---|
+| `memory.deepagentsMem.read` | `true` | 是否把 enabled Memory YAML 映射成 DeepAgents 原生 `memory: ["/memories/<id>.md"]`。 |
+| `memory.deepagentsMem.write` | `true` | 是否允许 DeepAgents 写 `/memories/**`。为 `false` 时 adapter 增加 write deny permission。 |
+| `memory.LangMem.read` | `true` | 是否在 run 前从 LangMem provider 搜索并召回相关 memory records。 |
+| `memory.LangMem.write` | `true` | 是否在 run 后把 `User input + Agent output` 发给 LangMem provider 抽取并写入。 |
+
+单个 memory profile 的开关仍然在 Memory YAML 内：
+
+```yaml
+apiVersion: stable-harness.dev/v1
+kind: Memory
+metadata:
+  name: workspace
+spec:
+  provider: langmem
+  profile: workspace
+  enabled: true
+```
+
+## 推荐默认值
+
+生产默认推荐：
+
+```yaml
+memory:
+  deepagentsMem:
+    read: true
+    write: true
+  LangMem:
+    read: true
+    write: true
+  maintenance:
+    enabled: true
+    schedule: periodic
+    prompt: durable preferences, workspace facts, reusable procedures, prior corrections, stale duplicates, and contradicted memories
+```
+
+含义：
+
+- 默认双写：DeepAgents 原生 file memory 和 LangMem structured memory 都写入。
+- 默认双读：DeepAgents 可以读取 native memory files，LangMem 也会在 run 前做 top-k recall。
+- 两个 memory 的后台整理使用同一个 maintenance prompt 和 schedule，但 target 独立运行，互不依赖。
+- 需要关闭某一路时，只关对应 `read` 或 `write` boolean。
+
+## Memory YAML
+
+Memory YAML 是可扩展定义，类似 resources/tools/skills 的 runtime inventory。
+
+示例：
+
+```yaml
+apiVersion: stable-harness.dev/v1
+kind: Memory
+metadata:
+  name: workspace
+  description: Workspace memory for durable project facts and reusable operating evidence.
+spec:
+  provider: langmem
+  profile: workspace
+  enabled: true
+  mode: background
+  prompts:
+    semantic: |-
+      Remember durable workspace facts, repository structure, runtime defaults,
+      validation commands, model names, URLs, and sentinel tokens.
+    episodic: |-
+      Remember important workspace events, validation runs, incidents, migrations,
+      release checks, and fixes that are likely to matter later.
+    procedural: |-
+      Remember reusable operating procedures, recovery steps, release gates,
+      full-matrix test commands, and project-specific engineering constraints.
+```
+
+每个 Memory profile 都可以有自己的 prompt，但输入上下文来自同一次 run：
+
+```text
+User input
+Agent output
+runtime metadata
+profile prompts
+```
+
+这样可以让 `workspace` 和 `user` 使用同一段上下文，但分别抽取不同的 memory。
+
+## LangMem 主路径
+
+LangMem 主路径由 `stable-harness` 控制 lifecycle。
+
+### 写入来源
+
+`memory_records.content` 不是 client 直接写入，也不是 DeepAgents 自动写入。
+
+写入流程是：
+
+1. client 或 agent run 产生 `User input` 和 `Agent output`。
+2. `stable-harness` memory layer 把 run context、profile、prompts、namespace 发给 provider。
+3. LangMem service 调 LLM 抽取 durable memory。
+4. SQLite store 做过滤、合并、去重、embedding。
+5. 最终写入 `memory_records` 和 `memory_vectors`。
+
+### LangMem 写入 Sequence
+
+```mermaid
+sequenceDiagram
+  participant Client as Client
+  participant Runtime as stable-harness Runtime
+  participant Adapter as Backend Adapter
+  participant Agent as DeepAgents
+  participant MemLayer as Memory Plugin Layer
+  participant Service as LangMem Service
+  participant LLM as Memory Extraction LLM
+  participant Store as SQLite + Vector Store
+
+  Client->>Runtime: request(input, metadata)
+  Runtime->>Adapter: run(adapterContext)
+  Adapter->>Agent: invoke(user message)
+  Agent-->>Adapter: agent output
+  Adapter-->>Runtime: RuntimeOutput
+  Runtime->>MemLayer: write-after-run hook
+  MemLayer->>Service: propose(namespace, User input + Agent output, prompts)
+  Service->>LLM: extract semantic/episodic/procedural memories
+  LLM-->>Service: memory candidates
+  Service->>Store: upsert + dedupe + embed
+  Store-->>Service: MemoryRecord
+  Service-->>MemLayer: candidates accepted
+  MemLayer-->>Runtime: runtime.memory.plugin.completed event
+  Runtime-->>Client: response
+```
+
+### LangMem 召回 Sequence
+
+```mermaid
+sequenceDiagram
+  participant Client as Client
+  participant Runtime as stable-harness Runtime
+  participant MemLayer as Memory Plugin Layer
+  participant Service as LangMem Service
+  participant Store as SQLite + Vector Store
+  participant Adapter as DeepAgents Adapter
+  participant Agent as DeepAgents
+
+  Client->>Runtime: request(input)
+  Runtime->>MemLayer: read-before-run hook
+  MemLayer->>Service: search(namespace, input)
+  Service->>Store: embed query + vector search
+  Store-->>Service: relevant MemoryRecords
+  Service-->>MemLayer: records
+  MemLayer-->>Runtime: pluginMemories context
+  Runtime->>Adapter: run(context includes pluginMemories)
+  Adapter->>Agent: "Relevant long-term memory" + User request
+  Agent-->>Adapter: output using recalled facts
+  Adapter-->>Runtime: RuntimeOutput
+  Runtime-->>Client: response
+```
+
+## DeepAgents 原生 Memory
+
+DeepAgents 原生 memory 是上游框架自己的 memory primitive。
+
+当 `memory.deepagentsMem.read=true` 且 agent 没有显式配置 DeepAgents memory 时，adapter 会把 enabled Memory YAML 映射成：
+
+```text
+memory:
+  - /memories/workspace.md
+  - /memories/user.md
+```
+
+这表示 DeepAgents 可以读取这些 memory file。它和 LangMem SQLite/vector 不是同一个存储系统。
+
+### DeepAgents 原生 Read Sequence
+
+```mermaid
+sequenceDiagram
+  participant Runtime as stable-harness Runtime
+  participant Adapter as DeepAgents Adapter
+  participant DA as DeepAgents
+  participant FS as DeepAgents Memory Backend
+
+  Runtime->>Adapter: run(adapterContext)
+  Adapter->>Adapter: read memory.deepagentsMem.read
+  alt deepagentsMem.read is true
+    Adapter->>DA: createDeepAgent(memory=["/memories/workspace.md"])
+    DA->>FS: read /memories/workspace.md
+    FS-->>DA: markdown memory content
+  else deepagentsMem.read is false
+    Adapter->>DA: createDeepAgent(memory omitted)
+  end
+  DA-->>Adapter: output
+```
+
+### DeepAgents 原生 Write Sequence
+
+```mermaid
+sequenceDiagram
+  participant DA as DeepAgents
+  participant Adapter as DeepAgents Adapter
+  participant Policy as Permissions / Middleware
+  participant FS as DeepAgents Memory Backend
+
+  DA->>Adapter: edit_file(/memories/workspace.md)
+  Adapter->>Policy: check memory.deepagentsMem.write
+  alt deepagentsMem.write is true
+    Policy-->>Adapter: allow
+    Adapter->>FS: write memory file
+    FS-->>DA: write ok
+  else deepagentsMem.write is false
+    Policy-->>Adapter: deny write /memories/**
+    Adapter-->>DA: tool error
+  end
+```
+
+## Prompt 控制和 Governance 控制
+
+DeepAgents native memory 有两层控制：
+
+| 控制层 | 类型 | 能控制什么 | 局限 |
+|---|---|---|---|
+| system prompt | 软控制 | 告诉 agent 什么时候写、写什么格式、不要写什么 | 模型可能忘记、误解或被 prompt injection 影响 |
+| permission / middleware | 硬控制 | 是否允许写 `/memories/**`、是否需要 approval、是否允许某个 path | 只能控制能不能写，不负责判断内容是否高质量 |
+
+LangMem 主路径的控制点不同：
+
+| 控制层 | 类型 | 能控制什么 |
+|---|---|---|
+| Memory YAML prompts | 抽取边界 | semantic/episodic/procedural 各存什么 |
+| profile namespace | 隔离边界 | workspace/user/agent/session 分开 |
+| service filtering | 内容过滤 | 过滤低价值 run instruction，例如 “Reply exactly” |
+| store dedupe | 记录治理 | canonical token、相似度、跨 kind 合并 |
+| runtime flag | 生命周期 | 读写开关、单次 request read-only |
+
+## Side-by-side 对比
+
+| 维度 | `deepagentsMem` | `LangMem` |
+|---|---|---|
+| 归属 | DeepAgents 上游能力 | stable-harness runtime capability |
+| 存储形态 | memory files / markdown | SQLite `memory_records` + `memory_vectors` |
+| 写入者 | agent 通过 file edit 写 | LangMem service 从 run context 抽取 |
+| 读入方式 | DeepAgents 读取 `/memories/*.md` | runtime 先 vector search，再注入相关 records |
+| 是否动态学习 | 是，agent 自己决定是否写文件 | 是，runtime run 后后台抽取 |
+| 内容质量控制 | prompt + permissions | prompts + filter + dedupe + namespace |
+| 去重 | 需要 agent 自觉或额外 middleware | store 层合并 |
+| 审计 | 文件 diff | record metadata + sourceRef + runtime events |
+| 生产默认 | 开启读写 | 开启读写 |
+| 适合用途 | 轻量 notebook、上游能力测试 | 生产长期记忆主路径 |
+
+## 四个 Flag 的行为矩阵
+
+| `deepagentsMem.read` | `deepagentsMem.write` | `LangMem.read` | `LangMem.write` | 行为 |
+|---:|---:|---:|---:|---|
+| true | true | true | true | 推荐默认。DeepAgents native memory 和 LangMem 都读写；后台整理保持同 prompt、同 schedule、独立 target。 |
+| false | false | true | true | LangMem-only 模式。DeepAgents 不读写 native memory；LangMem 负责召回和写入。 |
+| true | false | true | true | 额外让 DeepAgents 读 memory files，但不允许写；LangMem 仍是主路径。 |
+| false | false | true | false | 只读 LangMem，用于回归或只想使用已有 memory。 |
+| false | false | false | true | 只写 LangMem，不召回；适合离线采集或影子写入测试。 |
+| false | false | false | false | memory 完全关闭。 |
+
+## 单次 Request 只读
+
+如果某次请求只是测试 recall，不希望把这次回答再次写回 memory，可传：
+
+```json
+{
+  "metadata": {
+    "memoryWrite": false
+  }
+}
+```
+
+Sequence：
+
+```mermaid
+sequenceDiagram
+  participant Runtime as Runtime
+  participant Service as LangMem Service
+  participant Agent as Agent
+
+  Runtime->>Service: search(namespace, input)
+  Service-->>Runtime: recalled records
+  Runtime->>Agent: prompt with memory
+  Agent-->>Runtime: answer
+  alt metadata.memoryWrite is false
+    Runtime-->>Service: no propose call
+  else default
+    Runtime->>Service: propose(run context)
+  end
+```
+
+## 后台整理 Daemon
+
+后台整理是 `stable-harness` runtime capability，不属于 DeepAgents 执行语义。它只负责调度和审计，具体整理逻辑由各 memory target 自己实现。
+
+原则：
+
+- `LangMem` 和 `deepagentsMem` 是两个独立 target。
+- 两个 target 使用同一个 `memory.maintenance.prompt`。
+- 两个 target 使用同一个 schedule。
+- 一个 target 失败不阻塞另一个 target。
+- LangMem target 调 provider `search -> consolidate`。
+- DeepAgents target 通过独立 maintainer 处理 native memory file，不依赖 LangMem records。
+- Skill Candidate Miner target 从长期稳定 memory pattern 中生成候选 Skill，但不直接启用。
+
+Sequence：
+
+```mermaid
+sequenceDiagram
+  participant Daemon as Memory Maintenance Daemon
+  participant LangMem as LangMem Target
+  participant DeepMem as DeepAgentsMem Target
+  participant Provider as LangMem Provider
+  participant Store as DeepAgents StoreBackend
+
+  Daemon->>Daemon: load shared schedule and prompt
+  par LangMem maintenance
+    Daemon->>LangMem: run(prompt)
+    LangMem->>Provider: search(namespace, prompt)
+    Provider-->>LangMem: records
+    LangMem->>Provider: consolidate(records)
+    Provider-->>LangMem: operations
+  and DeepAgents native maintenance
+    Daemon->>DeepMem: run(prompt)
+    DeepMem->>Store: read /memories/*.md
+    DeepMem->>Store: compact / dedupe / rewrite files
+    Store-->>DeepMem: saved
+  and Skill candidate mining
+    Daemon->>Daemon: scan stable repeated memory evidence
+    Daemon->>Daemon: upsert skill_candidates
+    Daemon->>Daemon: create skill_candidate approval
+  end
+  Daemon->>Daemon: emit maintenance events
+```
+
+## Skill Candidate
+
+`SkillCandidate` 是 control-plane 对象，不是普通 memory record。它表示系统从长期稳定、可靠、可验证的 memory evidence 中发现了一个可能值得沉淀成 Skill 的流程。
+
+默认规则：
+
+- 不写入 `memory_records`。
+- 不自动写入 `resources/skills/**/SKILL.md`。
+- 不自动启用。
+- 只生成 `review_required` candidate。
+- 同时创建 `skill_candidate` approval。
+
+推荐存储在 control-plane SQLite：
+
+```text
+.stable-harness/control-plane/stable.sqlite
+  skill_candidates
+  skill_candidate_evidence
+```
+
+真实 E2E 使用：
+
+```text
+npm run test:skill-mining:e2e
+```
+
+测试会启动真实 Python LangMem service，通过远端 `qwen3.5:4b` 在多个 session 中写入长期 memory evidence，再由 maintenance daemon 生成 `review_required` SkillCandidate。
+
+## Namespace 规则
+
+namespace 由 `workspaceId`、profile、request metadata 决定：
+
+| Profile | Namespace 形态 |
+|---|---|
+| `workspace` | `<workspaceId>:workspace:<memoryId>` |
+| `user` | `<workspaceId>:user:<userId>:<memoryId>` |
+| `agent` | `<workspaceId>:agent:<agentId>:<memoryId>` |
+| `session` | `<workspaceId>:session:<sessionId>:<memoryId>` |
+
+EasyNet 示例：
+
+```text
+easynet:workspace:workspace
+easynet:user:e2e-user:user
+```
+
+## 存储目录
+
+配置和生成数据分开。
+
+`stable-harness`：
+
+```text
+config/memory/                         # Memory YAML
+.stable-harness/memory/langmem/         # generated LangMem data
+.stable-harness/memory/langmem/e2e/     # disposable E2E SQLite
+```
+
+EasyNet：
+
+```text
+config/memory/                          # Memory YAML
+.botbotgo/memory/langmem/               # generated LangMem data
+.botbotgo/memory/langmem/e2e/           # disposable E2E SQLite
+.botbotgo/runtime/                      # runtime run records
+.botbotgo/knowledge/                    # existing native knowledge DBs
+```
+
+生成数据不提交到 git。
+
+## SQLite Schema
+
+LangMem SQLite service 当前维护两张表：
+
+```text
+memory_records
+├─ id
+├─ namespace
+├─ kind
+├─ scope
+├─ status
+├─ content
+├─ summary
+├─ confidence
+├─ tags_json
+├─ sensitivity
+├─ metadata_json
+└─ created_at
+
+memory_vectors
+├─ record_id
+├─ namespace
+├─ dimensions
+└─ embedding_json
+```
+
+`memory_records` 是事实源，`memory_vectors` 是召回索引。
+
+## 去重策略
+
+重复来源主要有两类：
+
+1. 读取 memory 的回答又被写回，形成派生 memory。
+2. LLM 对同一事实生成中英文、semantic/procedural/episodic 多种表述。
+
+当前处理：
+
+- 支持 `metadata.memoryWrite=false`，只召回不写回。
+- 过滤低价值内容，例如 `Reply exactly`、`memory stored`、`asked to remember`。
+- 同 namespace 内跨 kind 查找相似 memory。
+- 对 `npm run ...`、`EASYNET_*` 等 canonical token 做强匹配。
+- 相似内容合并到已有 record，而不是新增行。
+
+去重 Sequence：
+
+```mermaid
+sequenceDiagram
+  participant Service as LangMem Service
+  participant LLM as Extraction LLM
+  participant Store as SQLite Store
+  participant Vec as Vector Store
+
+  Service->>LLM: extract memories
+  LLM-->>Service: candidates
+  loop each candidate
+    Service->>Service: filter low-value run instructions
+    Service->>Store: find active records in namespace
+    Store-->>Service: existing records
+    Service->>Service: compare text similarity + canonical tokens
+    alt duplicate found
+      Service->>Store: merge into existing memory_record
+      Service->>Vec: replace embedding
+    else new fact
+      Service->>Store: insert memory_record
+      Service->>Vec: insert embedding
+    end
+  end
+```
+
+## 长期测试
+
+长期测试目标是验证多轮重复写入不会让 memory records 无限制增长，并确认后续 run 能使用已经生成的 memory。
+
+`stable-harness`：
+
+```text
+LANGMEM_E2E_ITERATIONS=3 npm run test:langmem:sqlite:e2e
+```
+
+期望：
+
+```text
+memoryRecords: 1
+secondOutput includes npm run check:rules
+```
+
+EasyNet：
+
+```text
+EASYNET_LANGMEM_E2E_ITERATIONS=3 npm run test:langmem:sqlite:e2e
+```
+
+期望：
+
+```text
+workspaceRecords: 1
+userRecords: 2
+secondOutput includes npm run test:botbotgo:full
+secondOutput includes EASYNET_USER_MEMORY_9137
+```
+
+长期测试 Sequence：
+
+```mermaid
+sequenceDiagram
+  participant Test as E2E Script
+  participant Runtime as Runtime
+  participant Service as LangMem Service
+  participant Store as SQLite Store
+  participant Agent as Agent
+
+  Test->>Service: start with clean E2E SQLite
+  loop N iterations
+    Test->>Runtime: write request
+    Runtime->>Service: propose(run context)
+    Service->>Store: upsert/dedupe
+    Test->>Runtime: read request with memoryWrite=false
+    Runtime->>Service: search(query)
+    Service-->>Runtime: stable records
+    Runtime->>Agent: recalled memory
+    Agent-->>Runtime: exact remembered answer
+  end
+  Test->>Store: count records
+  Store-->>Test: bounded record count
+```
+
+## 配置示例
+
+### 推荐生产默认
+
+```yaml
+memory:
+  enabled: true
+  deepagentsMem:
+    read: true
+    write: true
+  LangMem:
+    read: true
+    write: true
+  maintenance:
+    enabled: true
+    schedule: periodic
+    prompt: durable preferences, workspace facts, reusable procedures, prior corrections, stale duplicates, and contradicted memories
+  refs:
+    - workspace
+    - user
+```
+
+### LangMem-only 对比测试
+
+```yaml
+memory:
+  enabled: true
+  deepagentsMem:
+    read: false
+    write: false
+  LangMem:
+    read: true
+    write: true
+  refs:
+    - workspace
+    - user
+```
+
+### DeepAgents native read-only 对比测试
+
+```yaml
+memory:
+  enabled: true
+  deepagentsMem:
+    read: true
+    write: false
+  LangMem:
+    read: true
+    write: true
+  refs:
+    - workspace
+    - user
+```
+
+### 只读 LangMem
+
+```yaml
+memory:
+  enabled: true
+  deepagentsMem:
+    read: false
+    write: false
+  LangMem:
+    read: true
+    write: false
+  refs:
+    - workspace
+    - user
+```
+
+## 端到端总览
+
+```mermaid
+sequenceDiagram
+  participant User as User
+  participant Runtime as stable-harness Runtime
+  participant LangMem as LangMem Provider
+  participant SQLite as SQLite + Vector
+  participant Adapter as DeepAgents Adapter
+  participant DeepAgents as DeepAgents
+  participant Native as DeepAgents Native Memory
+
+  User->>Runtime: request(input, metadata)
+
+  alt LangMem.read is true
+    Runtime->>LangMem: search(namespace, input)
+    LangMem->>SQLite: vector search
+    SQLite-->>LangMem: MemoryRecords
+    LangMem-->>Runtime: recalled context
+  end
+
+  Runtime->>Adapter: run(pluginMemories)
+
+  alt deepagentsMem.read is true
+    Adapter->>DeepAgents: memory=["/memories/workspace.md", "/memories/user.md"]
+    DeepAgents->>Native: read memory files
+    Native-->>DeepAgents: markdown memory
+  else deepagentsMem.read is false
+    Adapter->>DeepAgents: no native memory paths
+  end
+
+  Adapter->>DeepAgents: user request + recalled LangMem context
+  DeepAgents-->>Adapter: output
+  Adapter-->>Runtime: RuntimeOutput
+
+  alt deepagentsMem.write is true
+    DeepAgents->>Native: may edit /memories/**
+  else deepagentsMem.write is false
+    Adapter-->>DeepAgents: deny writes to /memories/**
+  end
+
+  alt LangMem.write is true and memoryWrite is not false
+    Runtime->>LangMem: propose(User input + Agent output)
+    LangMem->>SQLite: upsert/dedupe/vectorize
+  else LangMem.write is false or memoryWrite=false
+    Runtime-->>LangMem: no write
+  end
+
+  Runtime-->>User: response
+```
+
+## 最终建议
+
+短期默认：
+
+```yaml
+deepagentsMem.read: true
+deepagentsMem.write: true
+LangMem.read: true
+LangMem.write: true
+maintenance.enabled: true
+```
+
+长期评估：
+
+- 默认双读双写，长期观察重复、污染、文件 diff、profile 边界。
+- 两个 memory 的整理使用同一个 prompt 和 schedule，但 target 独立。
+- 如果 DeepAgents native memory 污染明显，可只关闭 `deepagentsMem.write`。
+- 如果 LangMem recall 成本过高，可只关闭 `LangMem.read`，保留 `LangMem.write` 做后台积累。