metame-cli 1.6.2 → 1.6.4

package/scripts/docs/hermes-memory-upgrade-converged.md

@@ -0,0 +1,461 @@
+# MetaMe Hermes-Style Memory Upgrade: Converged Landing Plan
+
+Status: converged proposal for implementation review
+Principle: extend the existing memory pipeline; do not replace it
+
+## 0. Final Architecture Decision
+
+Do not build a parallel Hermes clone.
+
+MetaMe already has the right primitives:
+
+- `memory_items` in `scripts/memory.js` for facts, conventions, profiles, and episodes.
+- FTS5 over `memory_items`.
+- `wiki_pages`, `content_chunks`, and `embedding_queue` from `scripts/memory-wiki-schema.js`.
+- Hybrid wiki retrieval in `scripts/core/hybrid-search.js`.
+- Session skeleton/evidence extraction in `scripts/session-analytics.js`.
+- Atomic fact extraction in `scripts/memory-extract.js`.
+- Per-agent memory snapshot injection in `scripts/agent-layer.js`.
+
+The elegant landing is therefore:
+
+```
+raw transcript provenance table
+  + existing memory_items(kind='episode') as session note
+  + existing memory_items(kind='convention'|'insight') as extracted facts
+  + existing wiki_pages as compiled knowledge
+  + new recall-router for on-demand prompt injection
+```
+
+Only one new durable table is required in the first landing: `session_sources`.
+
+## 1. Why This Is the Converged Version
+
+The previous RFC suggested both `session_episodes` and `session_notes`. After reviewing current source, that is heavier than necessary:
+
+- `memory_items.kind='episode'` already represents session summaries and is consumed by `searchSessions()` and wiki export.
+- `wiki-reflect-export.js` already exports session summaries into `wiki/sessions/`.
+- `memory-extract.js` already calls `memory.saveSession()` after processing Claude and Codex sessions.
+- Adding a separate `session_notes` table would duplicate existing `episode` rows and force new export/search paths.
+
+So the minimum stable design is:
+
+- Add `session_sources` for L0 provenance.
+- Upgrade `saveSession()` content quality and metadata.
+- Add query-time recall routing.
+- Keep all existing public memory APIs backward compatible.
+
+## 2. Target Data Model
+
+### 2.1 New Table: `session_sources`
+
+Add in `scripts/memory-wiki-schema.js` or a new schema helper called by `memory.js`.
+
+Purpose: immutable-ish provenance index for raw session material.
+
+```sql
+CREATE TABLE IF NOT EXISTS session_sources (
+  id                TEXT PRIMARY KEY,
+  engine            TEXT NOT NULL DEFAULT 'unknown',
+  session_id        TEXT NOT NULL,
+  project           TEXT DEFAULT '*',
+  scope             TEXT,
+  agent_key         TEXT,
+  cwd               TEXT,
+  source_path       TEXT,
+  source_hash       TEXT,
+  source_size       INTEGER DEFAULT 0,
+  first_ts          TEXT,
+  last_ts           TEXT,
+  message_count     INTEGER DEFAULT 0,
+  tool_call_count   INTEGER DEFAULT 0,
+  tool_error_count  INTEGER DEFAULT 0,
+  status            TEXT DEFAULT 'indexed'
+                    CHECK (status IN ('indexed','summarized','extracted','error','archived')),
+  error_message     TEXT,
+  created_at        TEXT DEFAULT (datetime('now')),
+  updated_at        TEXT DEFAULT (datetime('now')),
+  UNIQUE(engine, session_id, source_hash)
+);
+```
+
+Indexes:
+
+```sql
+CREATE INDEX IF NOT EXISTS idx_session_sources_session ON session_sources(session_id);
+CREATE INDEX IF NOT EXISTS idx_session_sources_project ON session_sources(project, scope, last_ts);
+CREATE INDEX IF NOT EXISTS idx_session_sources_agent   ON session_sources(agent_key, last_ts);
+```
+
+### 2.2 Reuse Existing Table: `memory_items`
+
+Do not add `session_notes` table.
+
+Use:
+
+- `kind='episode'` for L1 session note.
+- `source_type='session' | 'codex'`.
+- `source_id=session_id`.
+- `session_id=session_id`.
+- `project`, `scope`, and `agent_key` as routing metadata.
+
+Add only these columns if reviewers agree they are worth it:
+
+```sql
+ALTER TABLE memory_items ADD COLUMN source_hash TEXT;
+ALTER TABLE memory_items ADD COLUMN review_state TEXT DEFAULT 'unreviewed';
+ALTER TABLE memory_items ADD COLUMN valid_from TEXT;
+ALTER TABLE memory_items ADD COLUMN valid_to TEXT;
+```
+
+Strict minimum: `source_hash` is enough for phase 1.
+
+## 3. Implementation Sequence
+
+### PR 1: Provenance Index Only
+
+Goal: make every processed Claude/Codex transcript traceable without changing behavior.
+
+Files:
+
+- Add `scripts/core/session-source-db.js`
+- Update `scripts/memory-wiki-schema.js`
+- Update `scripts/memory.js` DB init if schema is split
+- Add `scripts/core/session-source-db.test.js`
+
+`session-source-db.js` API:
+
+```js
+function upsertSessionSource(db, source) {}
+function getSessionSource(db, { engine, sessionId, sourceHash }) {}
+function findSessionSources(db, { project, scope, engine, limit }) {}
+function markSessionSourceStatus(db, id, status, errorMessage) {}
+```
+
+Rules:
+
+- The module is pure DB logic.
+- It does not read transcript files.
+- It does not call LLMs.
+- It does not know daemon state.
+
+Integration:
+
+- In `memory-extract.js`, after `extractSkeleton()` or `buildCodexInput()`, compute `source_hash` from the source file.
+- Upsert `session_sources`.
+- Continue existing fact/session extraction unchanged.
+
+Acceptance:
+
+- Re-running `memory-extract.js` on the same transcript does not duplicate rows.
+- Existing tests for `memory-extract` and wiki still pass.
+- `memory.searchSessions()` output is unchanged.
+
+### PR 2: Better Episode Notes, Same Storage
+
+Goal: make `memory_items.kind='episode'` useful as a session wiki source even when no atomic facts are extracted.
+
+Files:
+
+- Add `scripts/core/session-note.js`
+- Update `scripts/memory-extract.js`
+- Extend `scripts/memory.js saveSession()`
+- Add `scripts/core/session-note.test.js`
+
+`session-note.js` should be pure:
+
+```js
+function buildSessionNote({ skeleton, evidence, sessionName, facts }) {
+  return {
+    title,
+    summary,
+    keywords,
+    content,
+  };
+}
+```
+
+Recommended content format:
+
+```markdown
+## Task
+<from user snippets / inferred intent>
+
+## Outcome
+<facts or concise execution result>
+
+## Evidence
+- session_id: ...
+- project: ...
+- duration: ...
+- tools: ...
+- errors: ...
+```
+
+Use this content as the `summary` passed to `saveSession()`. Keep the API name for compatibility even if the content becomes richer than a one-line summary.
+
+Acceptance:
+
+- Sessions with zero extracted facts still produce a useful episode row.
+- `memory-search.js --sessions "<query>"` can find session notes by task words, file paths, and error terms.
+- `wiki-reflect-export.js` continues to export session summaries without a new path.
+
+### PR 3: Scope Resolver
+
+Goal: stop scattering project/scope/agent inference.
+
+Files:
+
+- Add `scripts/core/memory-scope.js`
+- Use it in `memory-extract.js`, `agent-layer.js`, later `daemon-claude-engine.js`
+- Add `scripts/core/memory-scope.test.js`
+
+API:
+
+```js
+function normalizeMemoryScope(input) {
+  return {
+    project,
+    scope,
+    agentKey,
+    sessionId,
+    engine,
+    cwd,
+  };
+}
+```
+
+Rules:
+
+- It is pure.
+- It accepts partial inputs.
+- It never reads `daemon.yaml`.
+- Callers that need daemon config resolve project before calling this helper.
+
+Acceptance:
+
+- Claude and Codex extraction produce consistent `project/scope/engine`.
+- Existing `project_id` behavior remains compatible.
+
+### PR 4: Recall Router in Observe-Only Mode
+
+Goal: decide when historical memory should be retrieved, but do not inject yet.
+
+Files:
+
+- Add `scripts/core/recall-router.js`
+- Add `scripts/core/recall-router.test.js`
+- Optionally log decisions from `daemon-claude-engine.js` without changing prompt
+
+API:
+
+```js
+function planRecall({ text, scope, hasActiveSession }) {
+  return {
+    shouldRecall,
+    reason,
+    queries,
+    modes,
+    maxChars,
+  };
+}
+```
+
+Initial trigger rules:
+
+- history words: `之前`, `上次`, `以前`, `记得`, `回忆`, `查历史`, `有没有踩过坑`
+- decision words: `为什么这么定`, `当时怎么决定`, `以前怎么处理`
+- bug recurrence: `又`, `再次`, `之前的 bug`, `同样的问题`
+- exact identifiers: file path, function name, config key, error code
+
+Initial non-triggers:
+
+- pure commands: `/status`, `/tasks`, `/agent`
+- obvious one-shot shell/math/date requests
+- messages shorter than 4 chars unless they are explicit history commands
+
+Acceptance:
+
+- Router is deterministic.
+- Unit tests cover Chinese triggers.
+- No prompt behavior changes in this PR.
+
+### PR 5: Recall Assembly API
+
+Goal: centralize query-conditioned recall.
+
+Files:
+
+- Extend `scripts/memory.js`
+- Possibly add `scripts/core/recall-format.js`
+- Add tests in `scripts/memory-wiki-integration.test.js` or new `scripts/memory-recall.test.js`
+
+API:
+
+```js
+async function assembleRecallContext({
+  query,
+  scope,
+  modes = ['facts', 'sessions', 'wiki'],
+  maxChars = 4000,
+  ftsOnly = false,
+}) {}
+```
+
+Output:
+
+```js
+{
+  text: "...bounded markdown block...",
+  sources: [
+    { type: 'fact', id: '...' },
+    { type: 'session', session_id: '...' },
+    { type: 'wiki', slug: '...' }
+  ],
+  truncated: false
+}
+```
+
+Rules:
+
+- Facts: active facts only by default.
+- Sessions: use `searchSessions()`.
+- Wiki: use existing `hybridSearchWiki()`.
+- Include provenance labels.
+- Never include raw transcripts.
+- Hard cap by characters.
+
+Acceptance:
+
+- Empty search returns empty text.
+- Relevance is stable under missing embeddings.
+- Works with FTS-only fallback.
+
+### PR 6: Prompt Injection Behind Feature Flag
+
+Goal: add Hermes-style on-demand memory without destabilizing normal chats.
+
+Files:
+
+- `scripts/daemon-claude-engine.js`
+- `scripts/daemon-prompt-context.js`
+- `scripts/daemon-prompt-context.test.js`
+- Maybe `scripts/daemon-claude-engine.test.js`
+
+Behavior:
+
+- Existing `memory-snapshot.md` injection remains unchanged.
+- Before composing the final prompt, call `planRecall()`.
+- If enabled and `shouldRecall`, call `memory.assembleRecallContext()`.
+- Inject as:
+
+```text
+[Relevant memory recall:
+...
+]
+```
+
+Feature flag:
+
+Do not require config in PR 1-5.
+
+For PR 6, read optional runtime config from `~/.metame/daemon.yaml`:
+
+```yaml
+daemon:
+  memory_recall_enabled: false
+  memory_recall_max_chars: 4000
+```
+
+Template changes in `scripts/daemon-default.yaml` are optional and must contain placeholders only.
+
+Acceptance:
+
+- Default behavior is unchanged because flag is false.
+- With flag true, recall block appears only for router-triggered messages.
+- No daemon prompt exceeds configured recall cap.
+- If DB/search fails, prompt proceeds without recall.
+
+## 4. What Not To Do
+
+Do not:
+
+- Create a second memory DB.
+- Create `session_notes` table in the first landing.
+- Copy full transcripts into `memory.db`.
+- Inject recent sessions into every prompt.
+- Let subagents write verified/global memory directly.
+- Change `plugin/scripts/` or deployed `~/.metame/` files directly.
+- Edit real user config in `~/.metame/daemon.yaml` as part of source changes.
+
+## 5. Review-Critical Invariants
+
+Experts should hold the implementation to these invariants:
+
+1. L0 provenance is immutable enough.
+   A raw transcript pointer + hash must survive all derived memory changes.
+
+2. Derived memory is rebuildable.
+   Episode notes and facts can be regenerated from source material.
+
+3. Recall is query-conditioned.
+   No broad "always inject history" behavior.
+
+4. Prompt cache is mostly preserved.
+   Normal messages should not receive dynamic recall blocks.
+
+5. Existing APIs stay compatible.
+   `saveFacts`, `saveSession`, `searchFacts`, `searchSessions`, `recentSessions`, and `hybridSearchWiki` keep working.
+
+6. Scope is explicit.
+   Project/agent/thread memory does not bleed into unrelated sessions.
+
+7. Failure is soft.
+   Memory extraction/search failures never block the user-facing task.
+
+## 6. Minimal Test Plan
+
+Run after PR 1-3:
+
+```bash
+node --test scripts/core/session-source-db.test.js
+node --test scripts/core/session-note.test.js
+node --test scripts/memory-wiki-integration.test.js
+node --test scripts/memory-extract-step4.test.js
+```
+
+Run after PR 4-5:
+
+```bash
+node --test scripts/core/recall-router.test.js
+node --test scripts/memory-wiki-integration.test.js
+node --test scripts/wiki-reflect-export.test.js
+```
+
+Run after PR 6:
+
+```bash
+npx eslint scripts/daemon*.js
+node --test scripts/daemon-prompt-context.test.js
+node --test scripts/daemon-claude-engine.test.js
+node --test scripts/daemon-*.test.js
+```
+
+## 7. First Implementation Patch Shape
+
+The first implementation patch should be this small:
+
+1. Add `session_sources` DDL.
+2. Add `scripts/core/session-source-db.js`.
+3. Add `scripts/core/session-source-db.test.js`.
+4. Call `upsertSessionSource()` from `memory-extract.js` for Claude and Codex paths.
+5. Do not add prompt recall.
+6. Do not add config.
+7. Do not alter wiki export behavior.
+
+Expected review outcome:
+
+- Easy to verify.
+- Easy to roll back.
+- Gives future PRs a provenance anchor.
+