@x-otto/memory 0.0.1-alpha.0

package/README.md

@@ -0,0 +1,161 @@
+# @x-otto/memory
+
+Agent working memory pipeline: LLM-driven compaction, LLM-free pruning, archive persistence, multi-source AGENTS.md memory, writable cross-session auto-memory, operational learning, and workspace file-state snapshots.
+
+## Install
+
+```bash
+pnpm add @x-otto/memory
+```
+
+## Quick Start
+
+```ts
+import { MemoryManager, OperationalLearningStore, FileStateManager } from '@x-otto/memory'
+
+// Pipeline: prune → compaction → archive
+const memory = new MemoryManager({
+  workspaceDir: process.cwd(),
+  model: 'claude-sonnet-4-5',
+  provider: myProvider,
+})
+const result = await memory.process(messages, sessionId)
+// Returns { messages, summary, archivePath, pruned, compacted }
+
+// Token estimation (CJK-aware, no LLM call)
+import { estimateMessagesTokens } from '@x-otto/memory'
+const tokens = estimateMessagesTokens(messages)
+
+// Operational learning
+const lessons = new OperationalLearningStore({ path: './.otto/lessons.json' })
+await lessons.save({ tags: ['testing'], trigger: 'flaky test', insight: 'reset state first' })
+
+// File state snapshot
+const fsm = new FileStateManager()
+const snapshot = await fsm.capture({ workspaceDir: '.' })
+// snapshot → { tree, recentFiles: { created, modified, deleted } }
+```
+
+## Pipeline: prune → compaction → archive
+
+### 1. Prune — no-LLM fast cleanup (`prune.ts`)
+
+- Triggered when estimated tokens exceed `prune.trigger` threshold
+- Protection window: last N messages (by token count) are always kept
+- `tool_result` content outside protection → `[output pruned — N tokens]`
+- `tool_call` arguments for write tools (write/edit/apply_patch) truncated beyond `truncateMaxLength`
+- Minimum gain gate: rolls back if net savings < `minimum` tokens
+
+### 2. Compaction — LLM-driven summary (`compaction.ts`)
+
+- `findCutPoint`: keep recent tokens, align to user/assistant boundary, detect split-turn
+- Separates messages-to-summarize / preserved / turn-prefix region
+- Extracts file operations (read/modified paths) from tool_results
+- Calls injected `summarize(system, messages, opts)` with structured prompt (Goal / Constraints / Progress / Key Decisions / Next Steps / File Operations / Critical Context)
+- Iterative: previous summary + new messages → `UPDATE_SUMMARIZATION_PROMPT`
+- Split turn: generates additional turn-prefix summary via `TURN_PREFIX_SUMMARIZATION_PROMPT`
+
+### 3. Archive — persist compaction output (`archive.ts`)
+
+- Formats compressed messages + summary as Markdown (single messages >2000 chars truncated)
+- Backends: `InMemoryArchiveStorage` (default), `PersistenceBackedArchiveStorage` (file/HTTP via `@x-otto/persistence`)
+- Factory: `createArchiveStorage(options)` — file default `$OTTO_HOME/agent/archives`
+
+## Persistent Memory (`persistent-memory.ts`)
+
+Multi-source AGENTS.md loading + writable:
+
+| Source | Path | Writable |
+|--------|------|----------|
+| Global user | `~/.otto/AGENTS.md` | ✓ |
+| Project | `./.otto/AGENTS.md` | ✓ (by default) |
+| Community | `./AGENTS.md` | ✗ (read-only) |
+
+### MemoryStore backends
+
+| Backend | Store | Notes |
+|---------|-------|-------|
+| File system | `FileSystemMemoryStore` | Direct fs — AGENTS.md must stay human-editable, bypasses persistence envelope |
+| In-memory | `InMemoryMemoryStore` | Testing |
+| HTTP | `HttpMemoryStore` | Remote via `@x-otto/persistence` HttpPersistence; 1 source = 1 doc |
+
+### AutoMemory (`auto-memory.ts`)
+
+Writable cross-session memory with progressive disclosure:
+- `<baseDir>/MEMORY.md` — index with one-line pointers (injected only, keeps tokens low)
+- `<baseDir>/memory/<slug>.md` — per-entry full content, fetched on demand
+- `record(name, content)`, `read(name)`, `forget(name)`, `getInjection()`, `asSource()`
+
+### Memory Injection
+
+`buildMemoryInjection(sources)` wraps content in `<agent_memory>` + `<memory_guidelines>` blocks for model injection.
+
+## Token Estimation (`token-estimator.ts`)
+
+CJK-aware classification heuristic (not `length/4`):
+
+- CJK: 1 token/char, ASCII/symbols: 0.25–0.5/char → conservatively high (triggers early rather than overflow)
+- `estimateMessageTokens`: content blocks (text/thinking/tool_call), images ~2000 tokens constant, + role overhead
+- `estimateContextTokens`: hybrid — last assistant's usage as baseline, estimate only trailing tail
+- Exports `messageToText` and re-exports `getTokensFromUsage` from `@x-otto/ai`
+
+## Operational Learning (`operational-learning.ts`)
+
+`OperationalLearningStore` — experience knowledge base:
+
+| Method | Description |
+|--------|-------------|
+| `save(input)` | Auto-assigns `lesson_<n>` id, writes snapshot |
+| `search(query, limit)` | Weighted scoring on trigger/insight/tags, increments `appliedCount` |
+| `list(filter)` | Filter by tags / query |
+| `get(id)`, `delete(id)` | Single entry CRUD |
+| `size` | Entry count |
+
+Persistence via `@x-otto/persistence` FilePersistence (atomic tmp+rename), single snapshot store. Auto-upgrades pre-M19 flat JSON format on write.
+
+## File State Snapshot (`file-state-snapshot.ts`)
+
+Read-only workspace snapshot for agent context:
+- `capture({workspaceDir, recentFiles, planStatus})` — directory tree walk (depth ≤4, ≤50 per dir, ignores node_modules/.git/dist) + recent file stat check (birthtime/mtime → created/modified/deleted)
+- `formatSnapshot(snapshot)` — text block for model injection
+- `getLastSnapshot()` — most recent snapshot
+
+## Key Files
+
+```
+src/
+  types.ts                  # Config types, MemoryStore, ArchiveStorage, ManagerConfig
+  memory-manager.ts         # Pipeline coordinator + persistent memory facade
+  prune.ts                  # No-LLM pruning (window, tool_result truncation, gain gate)
+  compaction.ts             # Cut point, LLM summary, split-turn, file ops extraction
+  archive.ts                # InMemoryArchiveStorage + formatArchive + createArchiveStorage
+  persistent-memory.ts      # AGENTS.md multi-source + FileSystemMemoryStore / InMemoryMemoryStore
+  http-memory-store.ts      # Remote HTTP memory store
+  auto-memory.ts            # Cross-session progressive-disclosure memory
+  auto-extract.ts           # Auto-memory candidate extraction
+  agents-discovery.ts       # AGENTS.md path discovery
+  memory-extractor.ts       # Derive persistable memory entries from session
+  token-estimator.ts        # CJK-aware token estimation + usage-hybrid context
+  defaults.ts               # computeMemoryDefaults / resolveContextSize
+  operational-learning.ts   # Lessons store (CRUD + weighted search + atomic persistence)
+  file-state-snapshot.ts    # Workspace read-only snapshot
+  prompts.ts                # Summary prompts + memory/archive injection builders
+  constants.ts              # Thresholds, paths
+  index.ts                  # Barrel exports
+```
+
+## Dependencies
+
+- Internal: `@x-otto/ai` (Message types, getTokensFromUsage), `@x-otto/persistence`, `@x-otto/shared`, `@x-otto/env` (thresholds, tool name constants)
+- External: `js-tiktoken`
+- LLM access: via injected `summarize` callback (not direct)
+
+## Testing
+
+```bash
+pnpm --filter @x-otto/memory typecheck
+pnpm --filter @x-otto/memory build
+pnpm vitest run packages/memory/tests/
+```
+
+11 test files covering: prune (window, gain rollback), compaction (cut-point, split-turn, iterative, fileOps), archive (memory + persistence), persistent-memory (multi-source, writable, injection), auto-memory (record/read/forget/slug dedup), token estimation (CJK/symbol/image/usage-hybrid), operational learning (CRUD/search/upgrade), file-state-snapshot, prompts, defaults.