metame-cli 1.6.2 → 1.6.3

package/scripts/docs/hermes-memory-upgrade-plan.md

@@ -0,0 +1,506 @@
+# MetaMe Hermes-Style Memory Upgrade Plan
+
+Status: draft for expert review
+Scope: MetaMe session memory, wiki recall, agent memory injection
+Non-goal: replacing the daemon runtime or introducing a new agent server
+
+## 1. Executive Summary
+
+MetaMe already has most primitives required for a Hermes-style long-term memory system:
+
+- `scripts/memory.js` owns `~/.metame/memory.db`, `memory_items`, FTS5, state, project/scope fields, and compatibility APIs.
+- `scripts/memory-wiki-schema.js` owns `wiki_pages`, `wiki_topics`, `content_chunks`, and `embedding_queue`.
+- `scripts/session-analytics.js` parses Claude Code JSONL transcripts into local skeleton/evidence.
+- `scripts/memory-extract.js` extracts atomic facts from unanalyzed sessions.
+- `scripts/wiki-reflect.js` builds topic wiki pages from accumulated facts.
+- `scripts/agent-layer.js` injects per-agent `memory-snapshot.md` into engine prompts.
+- `scripts/memory-search.js` already exposes cross-session hybrid recall.
+
+The gap is not storage. The gap is an explicit lifecycle:
+
+```
+raw session episode -> episode index -> session note/wiki page -> structured memory item
+-> hybrid recall -> bounded prompt injection -> audit/supersession
+```
+
+Hermes Agent validates the same architecture pattern: small curated memory is always injected; large session history is stored losslessly and searched only when needed. MetaMe should converge on that pattern without replacing its daemon, routing, or existing memory DB.
+
+## 2. Current Architecture Findings
+
+### 2.1 Existing Strengths
+
+1. Unified memory DB exists.
+   `memory.js` initializes `~/.metame/memory.db`, enables WAL and foreign keys, and creates `memory_items` plus `memory_items_fts`.
+
+2. Memory kinds already map well to Hermes layers.
+   Existing `memory_items.kind` covers `profile`, `convention`, `episode`, and `insight`.
+
+3. Wiki schema already supports topic pages and hybrid retrieval.
+   `memory-wiki-schema.js` creates `wiki_pages`, `wiki_topics`, `wiki_pages_fts`, `content_chunks`, and `embedding_queue`.
+
+4. Hybrid wiki search already exists.
+   `core/hybrid-search.js` combines FTS5 and vector chunk search with RRF fusion, falling back to FTS-only.
+
+5. Session extraction is already isolated.
+   `session-analytics.js` performs local skeleton extraction, while `memory-extract.js` calls the model only for high-value atomic facts.
+
+6. Agent snapshot injection already exists.
+   `agent-layer.js` reads per-agent `memory-snapshot.md` and injects it as `[Agent memory snapshot: ...]`.
+
+### 2.2 Current Gaps
+
+1. No first-class raw episode table.
+   Raw session text exists in external JSONL files and a daily markdown diary, but `memory.db` does not have a durable episode registry with transcript path, hash, engine, chat id, project key, status, and lineage.
+
+2. `episode` memory items are summaries, not provenance.
+   `saveSession()` stores an `episode` as one `memory_items` row, but it does not preserve the raw transcript pointer, turn counts, tool counts, or source hash needed for rebuild/audit.
+
+3. Session wiki pages are exported, not modeled as first-class episode artifacts.
+   `wiki-reflect.js` exports recent session summaries, but there is no explicit `session_notes` / `episode_notes` table connecting raw transcript -> generated session note -> extracted facts -> wiki pages.
+
+4. Prompt injection is mostly static.
+   `agent-layer.js` refreshes `memory-snapshot.md` from recent sessions/facts, but there is no query-conditioned recall block for ambiguous/new tasks.
+
+5. Candidate memory lifecycle is incomplete.
+   `memory-extract.js` saves facts as `candidate`; `searchFacts()` only searches active items. Promotion, deprecation, supersession, and source-level audit need to be made explicit.
+
+6. Subagent memory write boundaries are not codified in the memory layer.
+   Dispatch has safety guards, but memory writes should also carry `writer_type`, `writer_agent`, and a default quarantine rule for delegated work.
+
+7. Scope model is under-specified.
+   `project`, `scope`, `task_key`, `session_id`, and `agent_key` exist, but there is no single resolver that maps chat/thread/project/agent into a memory scope for both read and write.
+
+## 3. Target Architecture
+
+### 3.1 Memory Layers
+
+MetaMe should use four explicit layers:
+
+1. L0 Raw Episode
+   Lossless source of truth. Immutable transcript metadata plus pointer/hash to raw JSONL or Codex rollout DB. Never summarized in place.
+
+2. L1 Session Note
+   Derived markdown/wiki note for each significant session. Contains task, decisions, changed files, validation, risks, next steps, and evidence links.
+
+3. L2 Structured Memory
+   Atomic facts in `memory_items`: conventions, decisions, bug lessons, config facts, workflow rules, milestones, profile facts. Each item must reference L0/L1 provenance.
+
+4. L3 Curated Injection
+   Small per-agent `memory-snapshot.md` plus query-conditioned recall block. This is the only layer routinely injected into prompts.
+
+### 3.2 Data Flow
+
+```
+Claude/Codex transcript
+  -> session episode indexer
+  -> episode row + checksum + metadata
+  -> session note builder
+  -> memory fact extractor
+  -> topic/wiki staleness update
+  -> wiki rebuild/export
+  -> recall router
+  -> prompt snapshot / query recall block
+```
+
+### 3.3 Retrieval Policy
+
+MetaMe should follow Hermes' split:
+
+- Always inject: small curated memory snapshot for the active agent/project.
+- On demand: run recall when user asks "之前/记得/上次/不清楚/查一下历史/为什么这么定" or when intent router detects low-confidence context.
+- Never inject blindly: raw transcript, large wiki pages, or all recent sessions.
+
+## 4. Proposed Schema Extensions
+
+All changes belong in `scripts/memory-wiki-schema.js` or a new `scripts/core/memory-schema.js` if we split schema ownership later.
+
+### 4.1 `session_episodes`
+
+Purpose: first-class registry for L0 raw episodes.
+
+Columns:
+
+- `id TEXT PRIMARY KEY`
+- `engine TEXT NOT NULL CHECK (engine IN ('claude','codex','unknown'))`
+- `session_id TEXT NOT NULL`
+- `chat_id TEXT`
+- `project TEXT DEFAULT '*'`
+- `scope TEXT`
+- `agent_key TEXT`
+- `cwd TEXT`
+- `transcript_path TEXT`
+- `transcript_hash TEXT`
+- `parent_episode_id TEXT`
+- `status TEXT DEFAULT 'indexed' CHECK (status IN ('indexed','summarized','extracted','archived','error'))`
+- `message_count INTEGER DEFAULT 0`
+- `tool_call_count INTEGER DEFAULT 0`
+- `tool_error_count INTEGER DEFAULT 0`
+- `first_ts TEXT`
+- `last_ts TEXT`
+- `created_at TEXT DEFAULT (datetime('now'))`
+- `updated_at TEXT DEFAULT (datetime('now'))`
+
+Indexes:
+
+- `(session_id)`
+- `(project, scope, last_ts)`
+- `(agent_key, last_ts)`
+- `(transcript_hash)`
+
+### 4.2 `session_notes`
+
+Purpose: first-class L1 derived notes.
+
+Columns:
+
+- `id TEXT PRIMARY KEY`
+- `episode_id TEXT NOT NULL`
+- `slug TEXT UNIQUE NOT NULL`
+- `title TEXT NOT NULL`
+- `content TEXT NOT NULL`
+- `summary TEXT`
+- `tags TEXT DEFAULT '[]'`
+- `evidence_refs TEXT DEFAULT '[]'`
+- `note_hash TEXT`
+- `status TEXT DEFAULT 'active' CHECK (status IN ('active','stale','archived','error'))`
+- `created_at TEXT DEFAULT (datetime('now'))`
+- `updated_at TEXT DEFAULT (datetime('now'))`
+
+FTS:
+
+- `session_notes_fts(title, content, tags)`
+
+### 4.3 `memory_items` Additions
+
+Add idempotent `ALTER TABLE` migrations:
+
+- `valid_from TEXT`
+- `valid_to TEXT`
+- `source_episode_id TEXT`
+- `source_note_id TEXT`
+- `writer_type TEXT DEFAULT 'system'`
+- `writer_agent TEXT`
+- `review_state TEXT DEFAULT 'unreviewed' CHECK (review_state IN ('unreviewed','verified','rejected'))`
+
+Rationale:
+
+- `source_episode_id` and `source_note_id` make every memory traceable.
+- `valid_from` / `valid_to` make temporal override explicit.
+- `review_state` separates "active for recall" from "human-verified".
+
+## 5. Component Plan
+
+### Phase 0: Stabilize Existing Memory Contract
+
+No behavior change. Add tests around current invariants before refactor.
+
+Files:
+
+- `scripts/memory.js`
+- `scripts/memory-wiki-schema.js`
+- `scripts/core/memory-model.js`
+- `scripts/memory-wiki-integration.test.js`
+
+Acceptance:
+
+- Existing `memory_items` schema remains backward compatible.
+- `saveFacts()`, `searchFacts()`, `searchSessions()`, `hybridSearchWiki()` signatures remain stable.
+- No config or runtime copy under `~/.metame/` is edited directly.
+
+### Phase 1: Episode Registry
+
+Add `session_episodes` and an indexer module.
+
+New module:
+
+- `scripts/core/session-episode-db.js`
+
+Exports:
+
+- `upsertSessionEpisode(db, episode)`
+- `getSessionEpisode(db, episodeId)`
+- `findEpisodeBySessionId(db, sessionId)`
+- `listRecentEpisodes(db, opts)`
+- `markEpisodeStatus(db, episodeId, status, errorMessage?)`
+
+Integration points:
+
+- `memory.js` should apply the schema at DB init.
+- `memory-extract.js` should call the new episode DB before fact extraction.
+- `session-analytics.js` remains pure extraction and should not open memory DB directly.
+
+Acceptance:
+
+- Running extraction on a known transcript creates one `session_episodes` row.
+- Re-running extraction is idempotent by `session_id + transcript_hash`.
+- Raw transcript content is not copied into DB; only path/hash/metadata are stored.
+
+### Phase 2: Session Note Builder
+
+Add L1 session note generation separate from fact extraction.
+
+New module:
+
+- `scripts/session-note-build.js`
+
+Responsibilities:
+
+- Build a concise session note from skeleton + evidence.
+- Store note in `session_notes`.
+- Export note to `~/.metame/wiki/sessions/` through the existing wiki export path.
+- Link note to episode via `episode_id`.
+
+Suggested note structure:
+
+```
+# <title>
+
+## Task
+## Decisions
+## Changes
+## Validation
+## Risks
+## Next Steps
+## Evidence
+```
+
+Acceptance:
+
+- Every non-trivial episode can have exactly one active session note.
+- Session note generation failure does not block fact extraction.
+- Notes are rebuildable from L0 transcript.
+
+### Phase 3: Provenance-Aware Fact Extraction
+
+Update `memory-extract.js` and `memory.saveFacts()`.
+
+Changes:
+
+- Pass `source_episode_id` and `source_note_id` into `saveFacts()`.
+- Store extracted facts as `candidate` and `review_state='unreviewed'`.
+- Auto-promote only high-confidence low-risk facts after validation checks.
+- Keep manual `memory-write.js` writes as `source_type='manual'`, `review_state='verified'`.
+
+Acceptance:
+
+- Every extracted fact has either `source_episode_id` or a clear legacy fallback.
+- Manual facts are protected and searchable immediately.
+- Delegated/subagent facts default to unreviewed unless promoted by parent/main agent.
+
+### Phase 4: Recall Router
+
+Add a pure router that decides when and what to recall.
+
+New module:
+
+- `scripts/core/recall-router.js`
+
+Inputs:
+
+- current user text
+- current project/scope/agent
+- active session metadata
+- optional intent classification
+
+Outputs:
+
+- `shouldRecall`
+- `queries[]`
+- `modes[]`: `facts`, `episodes`, `wiki`, `session_notes`
+- `budget`
+- `reason`
+
+Rules:
+
+- Recall if the user asks about history: "之前", "上次", "记得", "为什么这么定", "有没有踩过坑".
+- Recall if command/router confidence is low and project is known.
+- Recall exact identifiers via FTS first: paths, function names, error codes, config keys.
+- Recall semantic questions via wiki/session notes first.
+- Do not recall on simple one-shot commands where history is unlikely to help.
+
+Acceptance:
+
+- Router is pure and unit-tested with Chinese and English triggers.
+- Router never reads files or DB directly.
+- Router returns no recall for obviously self-contained tasks.
+
+### Phase 5: Prompt Injection Upgrade
+
+Keep current per-agent snapshot, add query-conditioned recall block.
+
+Files:
+
+- `scripts/agent-layer.js`
+- `scripts/daemon-claude-engine.js`
+- `scripts/daemon-prompt-context.js`
+
+Design:
+
+- `memory-snapshot.md` stays small and stable.
+- A new `[Relevant memory recall: ...]` block is added only when `recall-router` says recall is needed.
+- Recall block must include source labels: fact id, session note slug, episode id, or wiki slug.
+- Recall block should be capped by character budget and grouped by type.
+
+Proposed API:
+
+```js
+memory.assembleRecallContext({
+  query,
+  scope: { project, scope, agent, session },
+  modes: ['facts', 'session_notes', 'wiki', 'episodes'],
+  budget: { totalChars: 4000 }
+})
+```
+
+Acceptance:
+
+- Prompt cache remains stable for normal messages without recall.
+- Recall block appears only when triggered.
+- The block contains provenance and does not include raw transcript dumps.
+
+### Phase 6: Wiki Integration
+
+Extend wiki pipeline so session notes are a first-class source, not just exported afterthought.
+
+Files:
+
+- `scripts/wiki-reflect.js`
+- `scripts/wiki-reflect-export.js`
+- `scripts/core/wiki-db.js`
+
+Changes:
+
+- Add `listRecentSessionNotes()` alongside `listRecentSessionSummaries()`.
+- Export `session_notes` with stable slugs.
+- Allow topic wiki pages to reference session notes as `raw_source_ids`.
+- Add backlinks from session note -> related wiki topics.
+
+Acceptance:
+
+- `~/.metame/wiki/sessions/_index.md` is generated from `session_notes`, not only `memory_items.kind='episode'`.
+- Each wiki page can trace claims back to facts and session notes.
+- Existing wiki export behavior stays backward compatible.
+
+### Phase 7: Review, Promotion, and GC
+
+Make memory lifecycle auditable.
+
+Files:
+
+- `scripts/memory-gc.js`
+- `scripts/memory-nightly-reflect.js`
+- `scripts/memory-search.js`
+- optional new `scripts/memory-review.js`
+
+Rules:
+
+- Candidate facts can become active by manual approval, repeated successful recall, or nightly reflection.
+- Facts with conflicts should set `valid_to` and `supersedes_id`, not be overwritten.
+- Low-confidence unused candidate facts expire.
+- Manual/protected facts are never archived automatically.
+
+Acceptance:
+
+- A rejected/candidate fact does not appear in normal prompt injection.
+- Search CLI can optionally include `--candidates` for review.
+- GC never deletes raw episode rows or session notes; it only archives derived memory.
+
+## 6. Safety and Privacy Rules
+
+1. Raw transcripts are sensitive.
+   Store paths and hashes in DB. Do not copy full transcript into `memory.db` unless explicitly needed later.
+
+2. Credentials are never extracted.
+   Extend extraction prompts and filters to reject tokens, secrets, bot tokens, app secrets, chat ids unless they are placeholder values.
+
+3. Agent scope is mandatory.
+   Reads and writes must include project/scope/agent when known. Global memory should be rare and explicit.
+
+4. Personal assistant boundary remains hard.
+   No automatic dispatch or memory sharing into `personal`. If future memory sharing exists, personal scope must be opt-in.
+
+5. Review state matters.
+   Automatically extracted memory is not equivalent to verified memory.
+
+## 7. Test Matrix
+
+Required unit tests:
+
+- `scripts/core/session-episode-db.test.js`
+- `scripts/session-note-build.test.js`
+- `scripts/core/recall-router.test.js`
+- Extend `scripts/memory-wiki-schema.test.js`
+- Extend `scripts/memory-wiki-integration.test.js`
+- Extend `scripts/memory-search.js` coverage if currently absent
+
+Required integration tests:
+
+- `node --test scripts/memory-wiki-integration.test.js`
+- `node --test scripts/memory-extract-step4.test.js`
+- `node --test scripts/daemon-session-store.test.js`
+- `node --test scripts/daemon-prompt-context.test.js`
+- `node --test scripts/daemon-claude-engine.test.js`
+
+Daemon edit rule:
+
+- If any `scripts/daemon*.js` file changes, run:
+
+```bash
+npx eslint scripts/daemon*.js
+node --test scripts/daemon-*.test.js
+```
+
+## 8. Rollout Plan
+
+1. Ship schema and episode registry first.
+   This is low-risk and backward compatible.
+
+2. Backfill historical sessions.
+   Add a dry-run mode before writing rows. Use small batch size and lock files.
+
+3. Enable session note generation for new sessions only.
+   Do not backfill all history with LLM immediately.
+
+4. Add recall router in observe-only mode.
+   Log `shouldRecall`, queries, and candidate hits without injecting into prompts.
+
+5. Enable recall injection behind config.
+   Suggested config key under `daemon`:
+
+```yaml
+daemon:
+  memory_recall:
+    enabled: false
+    max_chars: 4000
+    modes: [facts, session_notes, wiki]
+```
+
+Do not edit `daemon-default.yaml` for real user config. Runtime config remains `~/.metame/daemon.yaml`; template changes are only placeholders.
+
+6. Turn on by project/agent.
+   Global enablement should wait until recall precision is measured.
+
+## 9. Expert Review Checklist
+
+Ask reviewers to challenge these points first:
+
+1. Is `session_episodes` enough provenance, or must raw transcript chunks be indexed directly?
+2. Should `session_notes` live in `memory.db`, filesystem markdown, or both?
+3. What is the exact promotion policy from `candidate` to `active`?
+4. Should recall be router-triggered only, or also model-tool-triggered?
+5. Should `memory-snapshot.md` remain a file, or become generated at session start?
+6. How strict should project/scope isolation be for global facts?
+7. Is `valid_from` / `valid_to` sufficient for temporal facts, or do we need a graph later?
+8. What metrics define recall quality: hit rate, false-positive rate, saved tokens, user correction rate?
+
+## 10. Recommended First PR
+
+Keep the first PR deliberately boring:
+
+1. Add schema for `session_episodes` and `session_notes`.
+2. Add `core/session-episode-db.js`.
+3. Add unit tests for idempotent upsert, transcript hash dedup, and recent listing.
+4. Add no prompt injection and no LLM calls.
+
+This creates the foundation experts can inspect without debating model behavior.
+

package/scripts/feishu-adapter.js

@@ -61,10 +61,10 @@ function withTimeout(promise, ms = 10000) {
 
 // Wait for DNS to resolve a target host with exponential backoff.
 // Used after system wake / before reconnect: the OS may report clock/events
-// restored before WiFi+DNS are actually usable. Retries 1/2/4/8s, total cap 30s.
+// restored before WiFi+DNS are actually usable. Retries 1/2/4/8s, total cap 60s.
 async function waitForNetworkReady(hostname, opts = {}) {
   const log = opts.log || (() => {});
-  const totalBudget = Number.isFinite(opts.totalBudgetMs) ? opts.totalBudgetMs : 30000;
+  const totalBudget = Number.isFinite(opts.totalBudgetMs) ? opts.totalBudgetMs : 60000;
   const lookup = opts.lookup || dns.promises.lookup;
   const sleep = opts.sleep || ((ms) => new Promise((r) => setTimeout(r, ms)));
   const startedAt = Date.now();
@@ -429,6 +429,82 @@ function createBot(config) {
       }
     },
 
+    /**
+     * Create a new Feishu group chat. The bot is automatically a member of
+     * any chat it creates; pass `inviteOpenIds` to add humans at creation time.
+     * Requires the app to have `im:chat` (and `im:chat.member` for invitees)
+     * permission. Returns { ok, chatId, error }; never throws — callers can
+     * fall back to the manual /activate flow on failure.
+     *
+     * @param {object} opts
+     * @param {string} opts.name        Chat name shown in user's chat list.
+     * @param {string} [opts.description]  Optional description.
+     * @param {string[]} [opts.inviteOpenIds]  open_ids of humans to add now.
+     * @param {string} [opts.ownerOpenId]  open_id to mark as chat owner.
+     */
+    async createChat({ name, description = '', inviteOpenIds = [], ownerOpenId = null }) {
+      if (!name) return { ok: false, error: 'name is required' };
+      try {
+        const data = {
+          name: String(name).slice(0, 60),
+          description: String(description).slice(0, 256),
+          chat_mode: 'group',
+          chat_type: 'private',
+          // Owner is required by the API; default to the inviter if not given.
+          ...(ownerOpenId ? { owner_id: ownerOpenId } : {}),
+          ...(inviteOpenIds.length > 0 ? { user_id_list: inviteOpenIds.slice(0, 50) } : {}),
+        };
+        const res = await withTimeout(
+          client.im.chat.create({ params: { user_id_type: 'open_id' }, data }),
+          15000
+        );
+        const chatId = res?.data?.chat_id || null;
+        if (!chatId) {
+          return { ok: false, error: `chat.create returned no chat_id: ${JSON.stringify(res?.data || res)}` };
+        }
+        return { ok: true, chatId };
+      } catch (err) {
+        const errDetail = err?.response?.data || err;
+        const code = errDetail?.code;
+        const msg = errDetail?.msg || errDetail?.message || String(err);
+        // Permission denied is the common first-time failure — surface a hint.
+        if (code === 99991663 || /permission|forbidden|scope/i.test(msg)) {
+          return { ok: false, error: `飞书应用缺少 im:chat 权限（${msg}）`, code };
+        }
+        return { ok: false, error: msg, code };
+      }
+    },
+
+    /**
+     * Invite humans to an existing chat by open_id. Returns invalid_id_list
+     * so the caller can decide whether to surface the partial-failure case.
+     */
+    async inviteToChat(chatId, openIds = []) {
+      if (!chatId) return { ok: false, error: 'chatId is required' };
+      const list = (Array.isArray(openIds) ? openIds : [openIds]).filter(Boolean).slice(0, 50);
+      if (list.length === 0) return { ok: true, invalid: [] };
+      try {
+        const res = await withTimeout(
+          client.im.chat.members.create({
+            path: { chat_id: chatId },
+            params: { member_id_type: 'open_id' },
+            data: { id_list: list },
+          }),
+          15000
+        );
+        const invalid = res?.data?.invalid_id_list || [];
+        return { ok: true, invalid };
+      } catch (err) {
+        const errDetail = err?.response?.data || err;
+        const code = errDetail?.code;
+        const msg = errDetail?.msg || errDetail?.message || String(err);
+        if (code === 99991663 || /permission|forbidden|scope/i.test(msg)) {
+          return { ok: false, error: `飞书应用缺少 im:chat.member 权限（${msg}）`, code };
+        }
+        return { ok: false, error: msg, code };
+      }
+    },
+
     /**
      * Start WebSocket long connection to receive messages (with auto-reconnect)
      * @param {function} onMessage - callback(chatId, text, event)