htm 0.0.1

data/docs/architecture/adrs/001-postgresql-timescaledb.md

@@ -0,0 +1,314 @@
+# ADR-001: PostgreSQL with TimescaleDB for Storage
+
+**Status**: Accepted
+
+**Date**: 2025-10-25
+
+**Decision Makers**: Dewayne VanHoozer, Claude (Anthropic)
+
+---
+
+## Quick Summary
+
+HTM uses **PostgreSQL with TimescaleDB** as its primary storage backend, providing time-series optimization, vector embeddings (pgvector), full-text search, and ACID compliance in a single, production-proven database system.
+
+**Why**: Consolidates time-series data, vector search, and full-text capabilities in one system rather than maintaining multiple specialized databases.
+
+**Impact**: Production-ready storage with excellent tooling, at the cost of some operational complexity compared to simpler alternatives.
+
+---
+
+## Context
+
+HTM requires a persistent storage solution that can handle:
+
+- Time-series data with efficient time-range queries
+- Vector embeddings for semantic search
+- Full-text search capabilities
+- ACID compliance for data integrity
+- Scalability for growing memory databases
+- Production-grade reliability
+
+### Alternative Options Considered
+
+1. **Pure PostgreSQL**: Solid relational database, pgvector support
+2. **TimescaleDB**: PostgreSQL extension optimized for time-series
+3. **Elasticsearch**: Strong full-text search, vector support added
+4. **Pinecone/Weaviate**: Specialized vector databases
+5. **SQLite + extensions**: Simple, embedded option
+
+---
+
+## Decision
+
+We will use **PostgreSQL with TimescaleDB** as the primary storage backend for HTM.
+
+---
+
+## Rationale
+
+### Why PostgreSQL?
+
+**Production-proven**:
+- Decades of reliability in demanding environments
+- ACID compliance guarantees data integrity for memory operations
+- Rich ecosystem with extensive tooling, monitoring, and support
+
+**Search capabilities**:
+- **pgvector extension**: Native vector similarity search with HNSW indexing
+- **Full-text search**: Built-in tsvector with GIN indexing
+- **pg_trgm extension**: Trigram-based fuzzy matching
+
+**Developer experience**:
+- Strong typing with schema enforcement prevents data corruption
+- Wide adoption means well-understood by developers
+- Standard SQL with PostgreSQL-specific enhancements
+
+### Why TimescaleDB?
+
+**Time-series optimization**:
+- **Hypertable partitioning**: Automatic chunk-based time partitioning
+- **Compression policies**: Automatic compression of old data (70-90% reduction)
+- **Time-range optimization**: Fast queries on temporal data via chunk exclusion
+
+**PostgreSQL compatibility**:
+- Drop-in extension, not a fork
+- All PostgreSQL features remain available
+- Standard PostgreSQL tools work seamlessly
+
+**Operational features**:
+- **Continuous aggregates**: Pre-computed summaries for analytics
+- **Retention policies**: Automatic data lifecycle management
+- **Cloud offering**: Managed service available (TimescaleDB Cloud)
+
+### Why Not Alternatives?
+
+!!! warning "Elasticsearch"
+    - High operational complexity (JVM tuning, cluster management)
+    - Higher resource usage
+    - Vector support more recent, less mature
+    - Superior full-text search not critical for our use case
+
+!!! info "Specialized Vector DBs (Pinecone, Weaviate, Qdrant)"
+    - Additional service dependency increases complexity
+    - Limited relational capabilities
+    - Vendor lock-in concerns
+    - Cost considerations for managed services
+    - Excellent vector search performance
+    - Purpose-built for embeddings
+
+!!! note "SQLite"
+    - Limited concurrency (write locks)
+    - No native vector search (extensions experimental)
+    - Not suitable for multi-robot scenarios
+    - Simple deployment
+    - Zero configuration
+
+---
+
+## Implementation Details
+
+### Schema Design
+
+```sql
+-- Nodes table as TimescaleDB hypertable
+CREATE TABLE nodes (
+  id SERIAL PRIMARY KEY,
+  key TEXT UNIQUE NOT NULL,
+  value TEXT NOT NULL,
+  embedding vector(1536),
+  robot_id TEXT NOT NULL,
+  created_at TIMESTAMP NOT NULL,
+  importance FLOAT DEFAULT 1.0,
+  type TEXT,
+  metadata JSONB
+);
+
+-- Convert to hypertable (TimescaleDB)
+SELECT create_hypertable('nodes', 'created_at');
+
+-- Vector indexing (HNSW for approximate nearest neighbor)
+CREATE INDEX nodes_embedding_idx ON nodes
+USING hnsw (embedding vector_cosine_ops);
+
+-- Full-text indexing
+CREATE INDEX nodes_fts_idx ON nodes
+USING GIN (to_tsvector('english', value));
+
+-- Additional indexes
+CREATE INDEX nodes_robot_id_idx ON nodes (robot_id);
+CREATE INDEX nodes_created_at_idx ON nodes (created_at DESC);
+CREATE INDEX nodes_type_idx ON nodes (type);
+```
+
+### Connection Configuration
+
+```ruby
+# Via environment variable (preferred)
+ENV['HTM_DBURL'] = "postgresql://user:pass@host:port/dbname?sslmode=require"
+
+# Parsed into connection hash
+{
+  host: 'host',
+  port: 5432,
+  dbname: 'tsdb',
+  user: 'tsdbadmin',
+  password: 'secret',
+  sslmode: 'require'
+}
+```
+
+### Key Features Enabled
+
+**Vector similarity search**:
+```sql
+-- Find semantically similar nodes
+SELECT *, 1 - (embedding <=> $1::vector) as similarity
+FROM nodes
+WHERE created_at > NOW() - INTERVAL '30 days'
+ORDER BY embedding <=> $1::vector
+LIMIT 10;
+```
+
+**Full-text search**:
+```sql
+-- Find nodes containing keywords
+SELECT *, ts_rank(to_tsvector('english', value),
+                  plainto_tsquery('english', $1)) as rank
+FROM nodes
+WHERE to_tsvector('english', value) @@ plainto_tsquery('english', $1)
+ORDER BY rank DESC
+LIMIT 10;
+```
+
+**Time-range queries** (optimized by chunk exclusion):
+```sql
+-- Fast time-range query (TimescaleDB prunes chunks)
+SELECT * FROM nodes
+WHERE created_at BETWEEN '2025-10-01' AND '2025-10-25'
+AND robot_id = 'robot-123'
+ORDER BY created_at DESC;
+```
+
+**Automatic compression**:
+```sql
+-- Compress chunks older than 30 days
+SELECT add_compression_policy('nodes', INTERVAL '30 days');
+
+-- Segment by robot_id and type for better compression
+ALTER TABLE nodes SET (
+  timescaledb.compress,
+  timescaledb.compress_segmentby = 'robot_id, type'
+);
+```
+
+---
+
+## Consequences
+
+### Positive
+
+- Production-ready with battle-tested reliability
+- Multi-modal search: vector, full-text, and hybrid strategies
+- Time-series optimization for efficient temporal queries
+- Cost-effective storage with compression reducing cloud costs
+- Familiar tooling: standard PostgreSQL tools and practices
+- Flexible querying: full SQL power for complex operations
+- ACID guarantees for critical memory operations
+
+### Negative
+
+- Operational complexity requires database management (mitigated by managed service)
+- Vertical scaling limits (mitigated by partitioning)
+- Connection overhead: PostgreSQL connections relatively heavy
+- Vector search performance slower than specialized vector DBs at massive scale
+
+### Neutral
+
+- Learning curve: developers need PostgreSQL + TimescaleDB knowledge
+- Cloud dependency: currently using TimescaleDB Cloud (could self-host)
+- Extension management requires extensions (timescaledb, pgvector, pg_trgm)
+
+---
+
+## Risks and Mitigations
+
+### Risk: Extension Availability
+
+!!! danger "Risk"
+    Extensions not available in all PostgreSQL environments
+
+**Likelihood**: Low (extensions widely available)
+**Impact**: High (breaks core functionality)
+**Mitigation**: Document requirements clearly, verify in setup process
+
+### Risk: Connection Exhaustion
+
+!!! warning "Risk"
+    PostgreSQL connections limited (default ~100)
+
+**Likelihood**: Medium (with many robots)
+**Impact**: Medium (service degradation)
+**Mitigation**: Implement connection pooling (ConnectionPool gem)
+
+### Risk: Storage Costs
+
+!!! info "Risk"
+    Vector data storage can be expensive at scale
+
+**Likelihood**: Medium (depends on usage)
+**Impact**: Medium (operational cost)
+**Mitigation**: Compression policies, retention policies, archival strategies
+
+### Risk: Query Performance at Scale
+
+!!! warning "Risk"
+    Complex hybrid searches may slow with millions of nodes
+
+**Likelihood**: Low (with proper indexing)
+**Impact**: Medium (user experience)
+**Mitigation**: Query optimization, read replicas, caching layer
+
+---
+
+## Alternatives Comparison
+
+| Solution | Pros | Cons | Decision |
+|----------|------|------|----------|
+| Pure PostgreSQL | Simple, reliable, pgvector | No time-series optimization | Rejected |
+| **PostgreSQL + TimescaleDB** | **Best of both worlds** | **Slight complexity** | **ACCEPTED** |
+| Elasticsearch | Excellent full-text search | High resource usage, complexity | Rejected |
+| Pinecone | Purpose-built vectors | Vendor lock-in, cost, limited relational | Rejected |
+| SQLite | Simple, embedded | Limited concurrency, no vectors | Rejected |
+
+---
+
+## Future Considerations
+
+- **Read replicas**: For query scaling when needed
+- **Partitioning strategies**: By robot_id for tenant isolation
+- **Caching layer**: Redis for hot nodes
+- **Archive tier**: S3/Glacier for very old memories
+- **Multi-region**: For global deployment
+
+---
+
+## References
+
+- [TimescaleDB Documentation](https://docs.timescale.com/)
+- [pgvector Documentation](https://github.com/pgvector/pgvector)
+- [PostgreSQL Full-Text Search](https://www.postgresql.org/docs/current/textsearch.html)
+- [HTM Database Schema Guide](../../development/schema.md)
+- [HTM Configuration Guide](../../installation.md)
+
+---
+
+## Review Notes
+
+**Systems Architect**: Solid choice for time-series + vector workload. Consider read replicas for scaling.
+
+**Database Architect**: Excellent indexing strategy. Monitor query performance as data grows.
+
+**Performance Specialist**: TimescaleDB compression will help with costs. Add connection pooling soon.
+
+**Maintainability Expert**: PostgreSQL tooling is mature and well-documented. Good choice for long-term maintenance.

data/docs/architecture/adrs/002-two-tier-memory.md

@@ -0,0 +1,411 @@
+# ADR-002: Two-Tier Memory Architecture
+
+**Status**: Accepted
+
+**Date**: 2025-10-25
+
+**Decision Makers**: Dewayne VanHoozer, Claude (Anthropic)
+
+---
+
+## Quick Summary
+
+HTM implements a **two-tier memory architecture** with token-limited working memory (hot tier) and unlimited long-term memory (cold tier), managing LLM context windows while preserving all historical data through RAG-based retrieval.
+
+**Why**: LLMs have limited context windows but need awareness across long conversations. Two tiers provide fast access to recent context while maintaining complete history.
+
+**Impact**: Efficient token budget management with never-forget guarantees, at the cost of coordination between two storage layers.
+
+---
+
+## Context
+
+LLM-based applications face a fundamental challenge: LLMs have limited context windows (typically 128K-200K tokens) but need to maintain awareness across long conversations and sessions spanning days, weeks, or months.
+
+### Requirements
+
+- Persist memories across sessions (durable storage)
+- Provide fast access to recent/relevant context
+- Manage token budgets efficiently
+- Never lose data accidentally
+- Support contextual recall from the past
+
+### Alternative Approaches
+
+1. **Database-only**: Store everything in PostgreSQL, load on demand
+2. **Memory-only**: Keep everything in RAM, serialize on shutdown
+3. **Two-tier**: Combine fast working memory with durable long-term storage
+4. **External service**: Use a managed memory service
+
+---
+
+## Decision
+
+We will implement a **two-tier memory architecture** with:
+
+- **Working Memory**: Token-limited, in-memory active context
+- **Long-term Memory**: Durable PostgreSQL storage
+
+---
+
+## Rationale
+
+### Working Memory (Hot Tier)
+
+**Characteristics**:
+- **Purpose**: Immediate context for LLM
+- **Storage**: In-memory Ruby data structures
+- **Capacity**: Token-limited (default 128K tokens)
+- **Eviction**: LRU-based eviction when full
+- **Access pattern**: Frequent reads, moderate writes
+- **Lifetime**: Process lifetime
+
+**Benefits**:
+- O(1) hash lookups for fast context access
+- Token budget control prevents context overflow
+- Explicit eviction policy with transparent behavior
+
+### Long-term Memory (Cold Tier)
+
+**Characteristics**:
+- **Purpose**: Permanent knowledge base
+- **Storage**: PostgreSQL with TimescaleDB
+- **Capacity**: Effectively unlimited
+- **Retention**: Permanent (explicit deletion only)
+- **Access pattern**: RAG-based retrieval
+- **Lifetime**: Forever
+
+**Benefits**:
+- Never lose data, survives restarts
+- Search historical context semantically
+- Time-series queries for temporal context
+
+### Data Flow
+
+```
+Add Memory:
+  User Input → Working Memory → Long-term Memory
+               (immediate)      (persisted)
+
+Recall Memory:
+  Query → Long-term Memory (RAG search) → Working Memory
+          (semantic + temporal)            (evict if needed)
+
+Eviction:
+  Working Memory (full) → Evict LRU → Long-term Memory (already there)
+                                       (mark as evicted, not deleted)
+```
+
+---
+
+## Implementation Details
+
+### Working Memory
+
+```ruby
+class WorkingMemory
+  attr_reader :max_tokens, :token_count
+
+  def initialize(max_tokens: 128_000)
+    @nodes = {}           # key => {value, token_count, importance, timestamp}
+    @max_tokens = max_tokens
+    @token_count = 0
+    @access_order = []    # Track access for LRU
+  end
+
+  def add(key, value, token_count:, importance: 1.0)
+    evict_to_make_space(token_count) if needs_eviction?(token_count)
+    @nodes[key] = {
+      value: value,
+      token_count: token_count,
+      importance: importance,
+      added_at: Time.now,
+      last_accessed: Time.now
+    }
+    @token_count += token_count
+    @access_order << key
+  end
+
+  def evict_to_make_space(needed_tokens)
+    # LRU eviction based on last access + importance
+    # See ADR-007 for detailed eviction strategy
+  end
+
+  def assemble_context(strategy: :balanced, max_tokens: nil)
+    # Sort by strategy and assemble within budget
+    # See ADR-006 for context assembly strategies
+  end
+end
+```
+
+### Long-term Memory
+
+```ruby
+class LongTermMemory
+  def add(key:, value:, embedding:, robot_id:, importance: 1.0, type: nil)
+    # Insert into PostgreSQL with vector embedding
+    @db.exec_params(<<~SQL, [key, value, embedding, robot_id, importance, type])
+      INSERT INTO nodes (key, value, embedding, robot_id, importance, type, created_at)
+      VALUES ($1, $2, $3, $4, $5, $6, CURRENT_TIMESTAMP)
+      RETURNING id
+    SQL
+  end
+
+  def search(timeframe:, query:, embedding_service:, limit:, strategy: :hybrid)
+    # RAG-based retrieval: temporal + semantic
+    # See ADR-005 for retrieval strategies
+  end
+
+  def mark_evicted(keys)
+    # Update in_working_memory flag (not deleted)
+    @db.exec_params(<<~SQL, [keys])
+      UPDATE nodes
+      SET in_working_memory = FALSE
+      WHERE key = ANY($1)
+    SQL
+  end
+end
+```
+
+### Coordination (HTM Class)
+
+```ruby
+class HTM
+  def initialize(robot_name:, robot_id: nil, max_tokens: 128_000, ...)
+    @working_memory = WorkingMemory.new(max_tokens: max_tokens)
+    @long_term_memory = LongTermMemory.new(db_config)
+    @embedding_service = EmbeddingService.new(...)
+    @robot_id = robot_id || SecureRandom.uuid
+    @robot_name = robot_name
+  end
+
+  def add_node(key, value, importance: 1.0, type: nil)
+    # 1. Generate embedding
+    embedding = @embedding_service.embed(value)
+
+    # 2. Store in long-term memory
+    @long_term_memory.add(
+      key: key,
+      value: value,
+      embedding: embedding,
+      robot_id: @robot_id,
+      importance: importance,
+      type: type
+    )
+
+    # 3. Add to working memory (evict if needed)
+    token_count = estimate_tokens(value)
+    @working_memory.add(key, value,
+                        token_count: token_count,
+                        importance: importance)
+  end
+
+  def recall(timeframe:, topic:, limit: 10, strategy: :hybrid)
+    # 1. Search long-term memory (RAG)
+    results = @long_term_memory.search(
+      timeframe: timeframe,
+      query: topic,
+      embedding_service: @embedding_service,
+      limit: limit,
+      strategy: strategy
+    )
+
+    # 2. Add results to working memory (evict if needed)
+    results.each do |node|
+      @working_memory.add(node[:key], node[:value],
+                          token_count: node[:token_count],
+                          importance: node[:importance])
+    end
+
+    # 3. Return nodes
+    results
+  end
+end
+```
+
+---
+
+## Consequences
+
+### Positive
+
+- Fast context access through O(1) working memory lookups
+- Durable storage ensures never lose data, survives restarts
+- Token budget control with automatic management
+- Explicit eviction policy provides transparent behavior
+- RAG-enabled search of historical context semantically
+- Never-delete philosophy: eviction moves data, never removes
+- Process-isolated: each robot instance has independent working memory
+
+### Negative
+
+- Complexity of coordinating two storage layers
+- Memory overhead from working memory consuming RAM
+- Synchronization challenges keeping both tiers consistent
+- Eviction overhead when moving data between tiers
+
+### Neutral
+
+- Token counting requires accurate estimation
+- Strategy tuning for eviction and assembly needs calibration
+- Per-process state means working memory not shared across processes
+
+---
+
+## Eviction Strategies
+
+### LRU-based (Implemented)
+
+```ruby
+def eviction_score(node)
+  recency = Time.now - node[:last_accessed]
+  importance = node[:importance]
+
+  # Lower score = evict first
+  importance / (recency + 1.0)
+end
+```
+
+See [ADR-007: Working Memory Eviction Strategy](007-eviction-strategy.md) for detailed eviction algorithm.
+
+### Future Strategies
+
+- **Importance-only**: Keep most important nodes
+- **Recency-only**: Pure LRU cache
+- **Frequency-based**: Track access counts
+- **Category-based**: Pin certain types (facts, preferences)
+- **Smart eviction**: ML-based prediction of future access
+
+---
+
+## Context Assembly Strategies
+
+### Recent (`:recent`)
+Sort by `created_at DESC`, newest first
+
+### Important (`:important`)
+Sort by `importance DESC`, most important first
+
+### Balanced (`:balanced`)
+```ruby
+score = importance * (1.0 / (1 + age_in_hours))
+```
+
+See [ADR-006: Context Assembly Strategies](006-context-assembly.md) for detailed assembly algorithms.
+
+---
+
+## Design Principles
+
+### Never Forget (Unless Told)
+
+- Eviction moves data, never deletes
+- Only `forget(confirm: :confirmed)` deletes
+- Long-term memory is append-only (updates rare)
+
+See [ADR-009: Never-Forget Philosophy](009-never-forget.md) for deletion policies.
+
+### Token Budget Management
+
+- Token counting happens at add time
+- Working memory enforces hard token limit
+- Context assembly respects token budget
+- Safety margin (10%) for token estimation errors
+
+### Transparent Behavior
+
+- Log all evictions
+- Track `in_working_memory` flag
+- Operations log for audit trail
+
+---
+
+## Risks and Mitigations
+
+### Risk: Token Count Inaccuracy
+
+!!! warning "Risk"
+    Tiktoken approximation differs from LLM's actual count
+
+**Likelihood**: Medium (different tokenizers)
+**Impact**: Medium (context overflow)
+**Mitigation**: Add safety margin (10%), use LLM-specific counters
+
+### Risk: Eviction Thrashing
+
+!!! info "Risk"
+    Constant eviction/recall cycles
+
+**Likelihood**: Low (with proper sizing)
+**Impact**: Medium (performance degradation)
+**Mitigation**: Larger working memory, smarter eviction, caching
+
+### Risk: Working Memory Growth
+
+!!! danger "Risk"
+    Memory leaks or unbounded growth
+
+**Likelihood**: Low (token budget enforced)
+**Impact**: High (OOM crashes)
+**Mitigation**: Hard limits, monitoring, alerts
+
+### Risk: Stale Working Memory
+
+!!! note "Risk"
+    Working memory doesn't reflect long-term updates
+
+**Likelihood**: Low (single-writer pattern)
+**Impact**: Low (eventual consistency OK)
+**Mitigation**: Refresh on recall, invalidation on update
+
+---
+
+## Performance Characteristics
+
+### Working Memory
+
+- **Add**: O(1) amortized (eviction is O(n) when needed)
+- **Retrieve**: O(1) hash lookup
+- **Eviction**: O(n log n) for sorting, O(k) for removing k nodes
+- **Context assembly**: O(n log n) for sorting, O(k) for selecting
+
+### Long-term Memory
+
+- **Add**: O(log n) PostgreSQL insert with indexes
+- **Vector search**: O(log n) with HNSW index (approximate)
+- **Full-text search**: O(log n) with GIN index
+- **Hybrid search**: O(log n) for both, then merge
+
+---
+
+## Future Enhancements
+
+1. **Shared working memory**: Redis-backed for multi-process
+2. **Lazy loading**: Load nodes on first access
+3. **Pre-fetching**: Anticipate needed context
+4. **Compression**: Compress old working memory nodes
+5. **Tiered eviction**: Multiple working memory levels
+6. **Smart assembly**: ML-driven context selection
+
+---
+
+## References
+
+- [Working Memory (Psychology)](https://en.wikipedia.org/wiki/Working_memory)
+- [Cache Eviction Policies](https://en.wikipedia.org/wiki/Cache_replacement_policies)
+- [LLM Context Window Management](https://www.anthropic.com/research/context-windows)
+- [ADR-001: PostgreSQL Storage](001-postgresql-timescaledb.md)
+- [ADR-006: Context Assembly](006-context-assembly.md)
+- [ADR-007: Eviction Strategy](007-eviction-strategy.md)
+
+---
+
+## Review Notes
+
+**Systems Architect**: Clean separation of concerns. Consider shared cache for horizontal scaling.
+
+**Performance Specialist**: Good balance of speed and durability. Monitor eviction frequency.
+
+**AI Engineer**: Token budget management is critical. Add safety margins for token count variance.
+
+**Ruby Expert**: Consider using Concurrent::Map for thread-safe working memory in future.