anvil-dev-framework 0.1.6

package/docs/specifications/SPEC-ANV-anvil-memory.md

@@ -0,0 +1,573 @@
+---
+spec_id: SPEC-ANV-MEM
+title: Anvil Memory System - Persistent Cross-Session Context
+status: draft
+created: 2026-01-07
+updated: 2026-01-07
+linear_issue: TBD
+---
+
+# Anvil Memory System
+
+## Overview
+
+Build a proprietary, ground-up persistent memory system for the Anvil framework. Anvil Memory provides cross-session context preservation, deeply integrated with the Context Checkpoint System (CCS), Ralph Wiggum autonomous execution, and Anvil workflows.
+
+**This replaces Claude-Mem** with a purpose-built solution that:
+- Uses per-project databases (no cross-contamination)
+- Adds first-class CCS/Ralph observation types
+- Integrates with Linear, sharding, and handoffs
+- Uses Bun runtime (strategic alignment with Anthropic)
+
+## Background
+
+### Why Not Fork Claude-Mem?
+
+| Factor | Claude-Mem | Anvil Memory |
+|--------|------------|--------------|
+| License | AGPL-3.0 (copyleft) | Proprietary |
+| Database | Global (`~/.claude-mem/`) | Per-project (`.anvil/memory.db`) |
+| Observation Types | Generic (6 types) | Extended for CCS/Ralph (10+ types) |
+| Integration | Plugin-based | Native to Anvil |
+| Control | External dependency | Full ownership |
+
+### Why Bun?
+
+Anthropic acquired Bun (Oven) in December 2025:
+
+- **Claude Code IS Bun** - Ships as Bun executable, users already have it
+- **Strategic alignment** - Anthropic's chosen runtime for Claude tools
+- **Native SQLite** - `bun:sqlite` is fast and synchronous
+- **MIT licensed** - No licensing issues
+- **Future-proof** - Claude Agent SDK and future tools are Bun-first
+
+**Sources**:
+- [Bun is joining Anthropic](https://bun.com/blog/bun-joins-anthropic)
+- [Anthropic Acquires Bun](https://devops.com/anthropic-acquires-bun-to-accelerate-ai-coding-tools/)
+
+---
+
+## Design Decisions
+
+### Decision 1: Per-Project Databases
+
+**Choice**: Each project gets its own SQLite database at `.anvil/memory.db`
+
+**Rationale**:
+- Users load Anvil framework into different projects
+- No cross-contamination of memories between projects
+- Portable - memory travels with project
+- Simpler backup/restore per project
+- Aligns with Anvil's project-centric model
+
+**Trade-off**: Cannot search across projects (acceptable for isolation benefits)
+
+### Decision 2: Bun Runtime
+
+**Choice**: Build with Bun (TypeScript)
+
+**Rationale**:
+- Claude Code users already have Bun installed
+- Native SQLite support (`bun:sqlite`)
+- Strategic alignment with Anthropic's tooling direction
+- Fast startup and execution
+- Single binary distribution possible
+
+### Decision 3: CLI Tool (Not Worker Service)
+
+**Choice**: Direct CLI invocation, no background worker
+
+**Rationale**:
+- Simpler architecture (no port management, no daemon)
+- Claude Code hooks can invoke directly
+- `bun:sqlite` is synchronous - no worker needed
+- Easier debugging and maintenance
+- Lower resource footprint
+
+**Trade-off**: Slightly slower per-invocation (acceptable for simplicity)
+
+### Decision 4: Proprietary License
+
+**Choice**: Proprietary license (not open-source)
+
+**Rationale**:
+- User requirement
+- Avoids AGPL copyleft obligations
+- Full control over distribution and modifications
+
+---
+
+## Architecture
+
+### System Overview
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                       ANVIL MEMORY SYSTEM                            │
+├─────────────────────────────────────────────────────────────────────┤
+│                                                                     │
+│  CLI: anvil-memory (Bun executable)                                 │
+│  ┌────────────┐  ┌────────────┐  ┌────────────┐  ┌────────────┐    │
+│  │  search    │  │  timeline  │  │  observe   │  │  context   │    │
+│  └────────────┘  └────────────┘  └────────────┘  └────────────┘    │
+│         │               │               │               │           │
+│         └───────────────┴───────────────┴───────────────┘           │
+│                                   │                                 │
+│                         ┌─────────▼─────────┐                       │
+│                         │   AnvilMemory     │                       │
+│                         │   (Bun/TS lib)    │                       │
+│                         └─────────┬─────────┘                       │
+│                                   │                                 │
+│         ┌─────────────────────────┼─────────────────────────┐       │
+│         │                         │                         │       │
+│  ┌──────▼──────┐  ┌───────────────▼───────────────┐  ┌──────▼─────┐│
+│  │ Observation │  │        Checkpoint             │  │   Ralph    ││
+│  │   Store     │  │          Store                │  │  Iteration ││
+│  └─────────────┘  └───────────────────────────────┘  └────────────┘│
+│                                   │                                 │
+│                         ┌─────────▼─────────┐                       │
+│                         │     SQLite DB     │                       │
+│                         │ .anvil/memory.db  │                       │
+│                         └───────────────────┘                       │
+│                                                                     │
+│  Hooks (Bun scripts in global/hooks/):                              │
+│  ┌────────────┐  ┌────────────┐  ┌────────────┐  ┌────────────┐    │
+│  │SessionStart│  │PostToolUse │  │    Stop    │  │PreCompact  │    │
+│  │  (context) │  │  (observe) │  │  (summary) │  │(checkpoint)│    │
+│  └────────────┘  └────────────┘  └────────────┘  └────────────┘    │
+│                                                                     │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+### Directory Structure
+
+```
+anvil-dev-framework/
+├── global/
+│   ├── tools/
+│   │   └── anvil-memory/           # CLI tool
+│   │       ├── src/
+│   │       │   ├── index.ts        # CLI entry point
+│   │       │   ├── db.ts           # SQLite operations
+│   │       │   ├── types.ts        # Type definitions
+│   │       │   ├── observe.ts      # Observation creation
+│   │       │   ├── search.ts       # Search operations
+│   │       │   ├── timeline.ts     # Timeline queries
+│   │       │   └── context.ts      # Context generation
+│   │       ├── package.json
+│   │       └── tsconfig.json
+│   │
+│   └── hooks/
+│       ├── anvil_memory_session.ts # SessionStart hook
+│       ├── anvil_memory_observe.ts # PostToolUse hook
+│       ├── anvil_memory_stop.ts    # Stop hook
+│       └── anvil_memory_compact.ts # PreCompact hook
+│
+└── project/
+    └── .anvil/
+        └── memory.db               # Per-project database
+```
+
+---
+
+## Data Model
+
+### Database Schema
+
+```sql
+-- Observations: Core records of work done
+CREATE TABLE observations (
+    id INTEGER PRIMARY KEY AUTOINCREMENT,
+    type TEXT NOT NULL,              -- observation type
+    title TEXT NOT NULL,             -- short title
+    subtitle TEXT,                   -- brief summary
+    content TEXT,                    -- full narrative/details
+    file_path TEXT,                  -- primary file affected
+    files TEXT,                      -- JSON array of files
+    concepts TEXT,                   -- JSON array of concept tags
+    project TEXT NOT NULL,           -- project name
+    session_id INTEGER,              -- FK to sessions
+    linear_issue TEXT,               -- Linear issue key (e.g., "ANV-123")
+    tokens_read INTEGER DEFAULT 0,   -- tokens to read this observation
+    tokens_work INTEGER DEFAULT 0,   -- tokens spent creating this
+    created_at TEXT DEFAULT (datetime('now')),
+    created_at_epoch INTEGER DEFAULT (unixepoch() * 1000)
+);
+
+-- FTS5 for full-text search
+CREATE VIRTUAL TABLE observations_fts USING fts5(
+    title, subtitle, content, file_path, concepts,
+    content='observations',
+    content_rowid='id'
+);
+
+-- Sessions: Conversation boundaries
+CREATE TABLE sessions (
+    id INTEGER PRIMARY KEY AUTOINCREMENT,
+    claude_session_id TEXT,          -- Claude Code session ID
+    request TEXT,                    -- Initial user request
+    summary TEXT,                    -- AI-generated summary
+    project TEXT NOT NULL,
+    branch TEXT,                     -- Git branch
+    linear_issue TEXT,               -- Primary issue worked on
+    context_peak_percent INTEGER,    -- Highest context % reached
+    checkpoint_triggered BOOLEAN DEFAULT 0,
+    created_at TEXT DEFAULT (datetime('now')),
+    ended_at TEXT
+);
+
+CREATE VIRTUAL TABLE sessions_fts USING fts5(
+    request, summary,
+    content='sessions',
+    content_rowid='id'
+);
+
+-- Checkpoints: CCS checkpoint events
+CREATE TABLE checkpoints (
+    id INTEGER PRIMARY KEY AUTOINCREMENT,
+    session_id INTEGER NOT NULL,
+    level TEXT NOT NULL,             -- L1, L2, L3
+    context_percent INTEGER NOT NULL,
+    trigger TEXT NOT NULL,           -- "automatic", "manual", "ralph"
+    handoff_file TEXT,               -- Path to handoff document
+    resume_summary TEXT,             -- What to continue
+    files_in_progress TEXT,          -- JSON array of {path, lines}
+    current_task TEXT,               -- What was being worked on
+    task_progress TEXT,              -- Percentage or description
+    linear_issue TEXT,
+    created_at TEXT DEFAULT (datetime('now')),
+    FOREIGN KEY (session_id) REFERENCES sessions(id)
+);
+
+-- Ralph Iterations: Ralph Wiggum loop tracking
+CREATE TABLE ralph_iterations (
+    id INTEGER PRIMARY KEY AUTOINCREMENT,
+    ralph_session_id TEXT NOT NULL,  -- Ralph session identifier
+    iteration INTEGER NOT NULL,
+    context_peak_percent INTEGER,
+    checkpoint_triggered BOOLEAN DEFAULT 0,
+    todo_items_start INTEGER,        -- Items remaining at start
+    todo_items_end INTEGER,          -- Items remaining at end
+    items_completed TEXT,            -- JSON array of completed items
+    status TEXT,                     -- "completed", "checkpoint", "error"
+    created_at TEXT DEFAULT (datetime('now')),
+    ended_at TEXT
+);
+
+-- Prompts: User messages (for conversation reconstruction)
+CREATE TABLE prompts (
+    id INTEGER PRIMARY KEY AUTOINCREMENT,
+    session_id INTEGER,
+    prompt TEXT NOT NULL,
+    project TEXT NOT NULL,
+    created_at TEXT DEFAULT (datetime('now')),
+    FOREIGN KEY (session_id) REFERENCES sessions(id)
+);
+
+CREATE VIRTUAL TABLE prompts_fts USING fts5(
+    prompt,
+    content='prompts',
+    content_rowid='id'
+);
+
+-- Indexes for common queries
+CREATE INDEX idx_observations_type ON observations(type);
+CREATE INDEX idx_observations_project ON observations(project);
+CREATE INDEX idx_observations_session ON observations(session_id);
+CREATE INDEX idx_observations_linear ON observations(linear_issue);
+CREATE INDEX idx_observations_created ON observations(created_at_epoch DESC);
+CREATE INDEX idx_sessions_project ON sessions(project);
+CREATE INDEX idx_checkpoints_session ON checkpoints(session_id);
+CREATE INDEX idx_ralph_session ON ralph_iterations(ralph_session_id);
+```
+
+### Observation Types
+
+| Type | Source | Description |
+|------|--------|-------------|
+| `bugfix` | claude-mem | Bug fixes |
+| `feature` | claude-mem | New features |
+| `refactor` | claude-mem | Code restructuring |
+| `discovery` | claude-mem | Things found during work |
+| `decision` | claude-mem | Architectural/design choices |
+| `change` | claude-mem | General changes |
+| **`checkpoint`** | CCS | Context checkpoint event |
+| **`ralph_iteration`** | Ralph | Autonomous loop iteration |
+| **`handoff`** | Handoff | Session continuity document created |
+| **`shard`** | Shard | Task breakdown event |
+| **`linear_sync`** | Linear | Issue state change |
+
+### Concept Tags
+
+| Concept | Description |
+|---------|-------------|
+| `how-it-works` | Technical explanation |
+| `why-it-exists` | Rationale/motivation |
+| `what-changed` | Diff/delta description |
+| `problem-solution` | Problem and how it was solved |
+| `gotcha` | Pitfall or non-obvious behavior |
+| `pattern` | Reusable approach |
+| `trade-off` | Decision with pros/cons |
+| **`context-critical`** | Must preserve across checkpoints |
+| **`ralph-continuation`** | Important for Ralph resume |
+
+---
+
+## CLI Interface
+
+### Commands
+
+```bash
+# Initialize database for project
+anvil-memory init
+
+# Create observation
+anvil-memory observe \
+  --type feature \
+  --title "Implemented OAuth callback" \
+  --content "Added OAuth callback handler..." \
+  --file src/auth/oauth.ts \
+  --linear ANV-123
+
+# Search observations
+anvil-memory search "authentication" --type bugfix --limit 10
+
+# Get recent context (for session start)
+anvil-memory context --limit 5 --format inject
+
+# Timeline around observation
+anvil-memory timeline --anchor 1234 --before 10 --after 10
+
+# Record checkpoint
+anvil-memory checkpoint \
+  --level L2 \
+  --percent 87 \
+  --handoff .claude/handoffs/2026-01-07-1145.md \
+  --task "Implementing OAuth callback" \
+  --progress "70%"
+
+# Record Ralph iteration
+anvil-memory ralph-iteration \
+  --session ralph-abc123 \
+  --iteration 5 \
+  --status completed \
+  --items-completed '["Fix test 1", "Fix test 2"]'
+
+# Get session summary
+anvil-memory session --current
+
+# Export database
+anvil-memory export --format json --output backup.json
+```
+
+### Hook Integration
+
+**SessionStart Hook** (`anvil_memory_session.ts`):
+```typescript
+// Loads recent context and injects into session
+const context = await anvilMemory.getRecentContext({ limit: 5 });
+return { additionalContext: context };
+```
+
+**PostToolUse Hook** (`anvil_memory_observe.ts`):
+```typescript
+// Analyzes tool output and creates observation if significant
+if (shouldObserve(toolName, toolOutput)) {
+  await anvilMemory.observe({
+    type: classifyObservation(toolOutput),
+    title: extractTitle(toolOutput),
+    content: extractNarrative(toolOutput),
+    // ...
+  });
+}
+```
+
+**PreCompact Hook** (`anvil_memory_compact.ts`):
+```typescript
+// Creates checkpoint before compaction
+await anvilMemory.checkpoint({
+  level: 'L3',
+  trigger: 'auto-compact',
+  // ...
+});
+```
+
+---
+
+## Integration Points
+
+### CCS Integration
+
+Anvil Memory serves as the persistence layer for CCS:
+
+| CCS Event | Anvil Memory Action |
+|-----------|---------------------|
+| L1 Warning | Log observation (type: `checkpoint`, level: L1) |
+| L2 Handoff | Create checkpoint record + observation |
+| L3 Emergency | Create checkpoint + handoff observation |
+| Resume | Query `context` with checkpoint filter |
+
+### Ralph Integration
+
+Anvil Memory tracks Ralph autonomous execution:
+
+| Ralph Event | Anvil Memory Action |
+|-------------|---------------------|
+| Iteration start | Create ralph_iteration record |
+| Iteration end | Update record with status |
+| Checkpoint triggered | Link to checkpoints table |
+| Task completed | Update items_completed |
+
+### Linear Integration
+
+Observations link to Linear issues:
+
+```sql
+-- Find all observations for an issue
+SELECT * FROM observations WHERE linear_issue = 'ANV-123';
+
+-- Get issue history timeline
+SELECT * FROM observations
+WHERE linear_issue = 'ANV-123'
+ORDER BY created_at_epoch;
+```
+
+---
+
+## Requirements
+
+### Core Requirements
+
+| ID | Requirement | Priority |
+|----|-------------|----------|
+| MEM-001 | Per-project SQLite database at `.anvil/memory.db` | P0 |
+| MEM-002 | CLI tool `anvil-memory` built with Bun | P0 |
+| MEM-003 | Full-text search via FTS5 | P0 |
+| MEM-004 | SessionStart hook for context injection | P0 |
+| MEM-005 | PostToolUse hook for observation capture | P0 |
+| MEM-006 | Support all observation types (10+) | P0 |
+
+### CCS Integration Requirements
+
+| ID | Requirement | Priority |
+|----|-------------|----------|
+| MEM-010 | Checkpoint records with level, percent, handoff | P0 |
+| MEM-011 | Query checkpoints for resume context | P0 |
+| MEM-012 | Link observations to checkpoints | P1 |
+
+### Ralph Integration Requirements
+
+| ID | Requirement | Priority |
+|----|-------------|----------|
+| MEM-020 | Ralph iteration tracking table | P0 |
+| MEM-021 | Link iterations to checkpoints | P0 |
+| MEM-022 | Query iteration history for patterns | P1 |
+
+### Context Generation Requirements
+
+| ID | Requirement | Priority |
+|----|-------------|----------|
+| MEM-030 | Generate context summary for session start | P0 |
+| MEM-031 | Progressive disclosure (index → full) | P1 |
+| MEM-032 | Token budget awareness in context | P1 |
+| MEM-033 | Filter by type, date, file, concept | P1 |
+
+---
+
+## Implementation Phases
+
+### Phase 1: Core Database and CLI (Week 1)
+
+1. Set up Bun project structure
+2. Implement SQLite schema with migrations
+3. Create `anvil-memory` CLI with basic commands:
+   - `init` - Create database
+   - `observe` - Create observation
+   - `search` - Query observations
+4. Write unit tests
+
+### Phase 2: Hooks Integration (Week 2)
+
+1. Create SessionStart hook for context injection
+2. Create PostToolUse hook for observation capture
+3. Create Stop hook for session summary
+4. Integrate with existing Anvil hook system
+
+### Phase 3: CCS Integration (Week 2-3)
+
+1. Add checkpoints table and CLI commands
+2. Create PreCompact hook for checkpoint creation
+3. Implement checkpoint resume context generation
+4. Test CCS → Memory → Resume flow
+
+### Phase 4: Ralph Integration (Week 3)
+
+1. Add ralph_iterations table
+2. Integrate with `ralph_state.py` (or rewrite in TS)
+3. Link Ralph iterations to checkpoints
+4. Test full Ralph → Checkpoint → Resume cycle
+
+### Phase 5: Polish and Optimization (Week 4)
+
+1. Performance optimization (indexes, caching)
+2. Context budget management
+3. Export/import functionality
+4. Documentation and examples
+
+---
+
+## Migration from Claude-Mem
+
+### Coexistence Period
+
+During development, both systems can coexist:
+- Claude-Mem: Global `~/.claude-mem/`
+- Anvil Memory: Per-project `.anvil/memory.db`
+
+### Migration Path
+
+1. **Phase 1**: Anvil Memory runs alongside Claude-Mem
+2. **Phase 2**: Anvil hooks replace Claude-Mem hooks in Anvil projects
+3. **Phase 3**: Claude-Mem can be uninstalled for Anvil users
+
+### Data Migration (Optional)
+
+```bash
+# Export Claude-Mem data for project
+anvil-memory import --from claude-mem --project my-project
+```
+
+---
+
+## Success Metrics
+
+| Metric | Target |
+|--------|--------|
+| Session context load time | <500ms |
+| Observation capture latency | <100ms |
+| Search query time | <200ms |
+| Database size per 1000 observations | <10MB |
+| Context resume accuracy | >95% useful |
+
+---
+
+## Open Questions (Resolved)
+
+| Question | Decision |
+|----------|----------|
+| Open-source or proprietary? | **Proprietary** |
+| Coexist or replace Claude-Mem? | **Replace** |
+| Worker service or direct CLI? | **CLI** (direct) |
+| MCP server or CLI? | **CLI** |
+| Global or per-project DB? | **Per-project** |
+| Node.js or Bun? | **Bun** (Anthropic-owned) |
+
+---
+
+## References
+
+- [Bun is joining Anthropic](https://bun.com/blog/bun-joins-anthropic)
+- [Anthropic Acquires Bun](https://devops.com/anthropic-acquires-bun-to-accelerate-ai-coding-tools/)
+- Claude-Mem architecture analysis (internal)
+- CCS Specification: `docs/specifications/SPEC-ANV-context-checkpoints.md`
+- Ralph Wiggum: `global/lib/ralph_state.py`, `global/commands/ralph.md`