htm 0.0.1

data/.architecture/reviews/initial-system-analysis.md

@@ -0,0 +1,330 @@
+# HTM Initial System Analysis
+
+**Generated**: 2025-10-25
+**Version**: 0.1.0 (Initial Development)
+**Status**: Active Development
+
+## Executive Summary
+
+HTM (Hierarchical Temporary Memory) is a Ruby gem providing intelligent memory management for LLM-based applications ("robots"). The system implements a two-tier memory architecture combining durable PostgreSQL/TimescaleDB storage with token-limited working memory, enabling contextual recall through RAG (Retrieval-Augmented Generation) techniques.
+
+### Key Strengths
+- **Never-forget architecture**: Explicit deletion model prevents accidental data loss
+- **Multi-robot "hive mind"**: Shared memory enables cross-robot context awareness
+- **Time-series optimization**: TimescaleDB hypertables with automatic compression
+- **Flexible search**: Vector, full-text, and hybrid search strategies
+- **Production-grade storage**: PostgreSQL with pgvector for semantic search
+
+### Key Challenges
+- **Embedding service dependency**: Requires Ollama or external API for vector generation
+- **Early development stage**: Some features are stubs (OpenAI, Cohere providers)
+- **Schema evolution**: No migration framework currently in place
+- **Connection management**: Single connection model may not scale
+
+## System Architecture
+
+### Component Overview
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                         HTM API                              │
+│  (Main interface - lib/htm.rb)                              │
+│  • add_node, recall, retrieve, forget                       │
+│  • create_context, memory_stats                             │
+│  • which_robot_said, conversation_timeline                  │
+└────────────┬────────────────────────────┬───────────────────┘
+             │                            │
+             ▼                            ▼
+┌──────────────────────┐      ┌──────────────────────┐
+│   WorkingMemory      │      │  LongTermMemory      │
+│   (In-memory)        │◄────►│  (PostgreSQL)        │
+│                      │      │                      │
+│  • Token tracking    │      │  • Persistent nodes  │
+│  • LRU eviction      │      │  • Relationships     │
+│  • Context assembly  │      │  • Tags              │
+└──────────────────────┘      │  • Vector search     │
+                              │  • Full-text search  │
+                              └──────────┬───────────┘
+                                         │
+                                         ▼
+                              ┌──────────────────────┐
+                              │  EmbeddingService    │
+                              │  (via RubyLLM)       │
+                              │                      │
+                              │  • Ollama (default)  │
+                              │  • OpenAI (stub)     │
+                              │  • Token counting    │
+                              └──────────────────────┘
+```
+
+### Data Flow
+
+**Adding a Memory:**
+1. HTM.add_node() receives content and metadata
+2. EmbeddingService generates vector embedding (via Ollama)
+3. Token count calculated using Tiktoken
+4. LongTermMemory persists to PostgreSQL with embedding
+5. WorkingMemory adds to active context (with eviction if needed)
+6. Relationships and tags created
+7. Operation logged to audit trail
+
+**Recalling Memories:**
+1. HTM.recall() with timeframe and topic
+2. Natural language timeframe parsed ("last week" → date range)
+3. Search strategy selected (vector/fulltext/hybrid)
+4. EmbeddingService generates query embedding (for vector search)
+5. LongTermMemory executes search with time filter
+6. Results added to WorkingMemory (evicting if needed)
+7. Operation logged
+8. Nodes returned to caller
+
+**Context Assembly:**
+1. HTM.create_context() with strategy
+2. WorkingMemory sorts nodes by strategy (recent/important/balanced)
+3. Assembles text within token budget
+4. Returns context string for LLM
+
+## Technology Stack
+
+### Core Dependencies
+- **PostgreSQL 17+**: Primary data store
+- **TimescaleDB**: Time-series optimization, hypertables, compression
+- **pgvector**: Vector similarity search (cosine distance, HNSW indexing)
+- **pg_trgm**: Trigram-based fuzzy text matching
+- **Ruby 3.0+**: Implementation language
+- **Ollama**: Local embedding generation (default via RubyLLM)
+- **Tiktoken**: Token counting for context management
+
+### Development Tools
+- **Minitest**: Testing framework
+- **Rake**: Task automation
+- **debug_me**: Debugging utility (project standard)
+
+## Database Schema
+
+### Core Tables
+
+**nodes** (TimescaleDB hypertable on `created_at`):
+- Primary memory storage
+- Vector embeddings (1536 dimensions)
+- Token counts, importance scores
+- Robot ownership tracking
+- Compression after 30 days
+
+**relationships**:
+- Knowledge graph edges
+- From/to node references
+- Relationship types and strength
+
+**tags**:
+- Flexible categorization
+- Many-to-many with nodes
+
+**operations_log** (TimescaleDB hypertable):
+- Audit trail for all operations
+- Partitioned by timestamp
+
+**robots**:
+- Robot registry
+- Activity tracking
+
+### Indexing Strategy
+- **HNSW** on vector embeddings (cosine distance)
+- **GIN** on full-text search vectors
+- **GIN** with trigram ops for fuzzy matching
+- **B-tree** on temporal columns, robot_id, types
+
+## Current Implementation Status
+
+### Completed (Phase 1)
+- ✅ Core two-tier memory architecture
+- ✅ PostgreSQL/TimescaleDB schema
+- ✅ Ollama embedding integration
+- ✅ Token counting and budget management
+- ✅ Database connection and setup
+- ✅ Hypertable configuration
+- ✅ Basic testing framework
+
+### In Progress
+- 🔄 RAG retrieval implementation
+- 🔄 Working memory eviction strategies
+- 🔄 Relationship graph queries
+- 🔄 Tag-based filtering
+
+### Planned
+- 📋 Additional embedding providers (OpenAI, Cohere)
+- 📋 Connection pooling
+- 📋 Advanced context assembly
+- 📋 Memory consolidation
+- 📋 Observability and metrics
+- 📋 Migration framework
+- 📋 Production hardening
+
+## Design Patterns & Principles
+
+### Architecture Patterns
+- **Two-tier memory**: Separates hot (working) from cold (long-term) storage
+- **RAG (Retrieval-Augmented Generation)**: Semantic + temporal search
+- **Repository pattern**: Database abstraction in LongTermMemory
+- **Strategy pattern**: Multiple search and context assembly strategies
+- **Adapter pattern**: EmbeddingService abstracts provider differences
+
+### Design Principles Applied
+- **Explicit deletion**: Never delete without confirmation
+- **Fail-safe defaults**: Falls back to random embeddings if Ollama unavailable
+- **Separation of concerns**: Clear component boundaries
+- **Testability**: Components designed for isolation testing
+- **Documentation as code**: Inline documentation with examples
+
+## Key Architectural Decisions
+
+See ADRs for detailed decision records:
+
+1. **PostgreSQL + TimescaleDB for storage** (ADR-001)
+   - Time-series optimization
+   - Native vector search with pgvector
+   - Production-grade reliability
+
+2. **Two-tier memory architecture** (ADR-002)
+   - Token budget management
+   - LRU eviction to long-term storage
+   - Never-delete philosophy
+
+3. **Ollama as default embedding provider** (ADR-003)
+   - Local-first approach
+   - No API costs
+   - Privacy-preserving
+
+4. **Multi-robot shared memory (hive mind)** (ADR-004)
+   - Cross-robot context sharing
+   - Conversation attribution
+   - Timeline reconstruction
+
+5. **Hybrid search strategy** (ADR-005)
+   - Vector similarity for semantics
+   - Full-text for keywords
+   - Temporal filtering
+   - Weighted combination
+
+## Risk Assessment
+
+### Technical Risks
+
+**High Priority:**
+- **Ollama dependency**: Embedding generation fails if Ollama unavailable
+  - *Mitigation*: Fallback to stub embeddings, multi-provider support
+
+- **Schema evolution**: No migration framework
+  - *Mitigation*: Implement Rails-like migration system
+
+**Medium Priority:**
+- **Connection management**: Single connection per instance
+  - *Mitigation*: Implement connection pooling (ConnectionPool gem already included)
+
+- **Memory growth**: Working memory could grow unbounded
+  - *Mitigation*: Implement aggressive eviction strategies
+
+**Low Priority:**
+- **Embedding dimension mismatch**: Hardcoded 1536 dimensions
+  - *Mitigation*: Make configurable per provider
+
+### Operational Risks
+
+**Medium Priority:**
+- **Database costs**: TimescaleDB Cloud usage-based pricing
+  - *Mitigation*: Compression policies, retention policies
+
+- **Token counting accuracy**: Tiktoken approximation may differ from LLM
+  - *Mitigation*: Add safety margins, LLM-specific counters
+
+## Performance Considerations
+
+### Strengths
+- TimescaleDB chunk-based partitioning for time-range queries
+- HNSW indexing for fast vector similarity search
+- Compression for old data reduces storage costs
+- Token pre-calculation avoids runtime overhead
+
+### Optimization Opportunities
+- Connection pooling for concurrent access
+- Batch embedding generation
+- Caching frequently accessed nodes
+- Lazy loading of relationships
+- Prepared statements for common queries
+
+## Security Considerations
+
+### Current State
+- SSL required for TimescaleDB Cloud connection
+- Database credentials via environment variables
+- No encryption at rest (relies on database)
+- No access control beyond robot_id tracking
+
+### Recommendations
+- Implement row-level security for multi-tenant scenarios
+- Encrypt embeddings if sensitive
+- Add audit logging for forget() operations
+- Consider API key rotation for embedding providers
+- Validate and sanitize all user inputs
+
+## Scalability Analysis
+
+### Current Limitations
+- Single database connection per HTM instance
+- In-memory working memory (per-process)
+- No horizontal scaling strategy
+- Limited to single TimescaleDB instance
+
+### Growth Path
+- Add connection pooling
+- Consider Redis for shared working memory
+- Implement read replicas for query scaling
+- Partition by robot_id for tenant isolation
+- Add caching layer (Redis/Memcached)
+
+## Maintainability Assessment
+
+### Strengths
+- Clear component separation
+- Comprehensive inline documentation
+- Test framework in place
+- Debugging with debug_me standard
+- Frozen string literals enabled
+
+### Areas for Improvement
+- Increase test coverage (integration tests needed)
+- Add API documentation (YARD/RDoc)
+- Implement CI/CD pipeline
+- Add code quality metrics (RuboCop, SimpleCov)
+- Create migration framework
+
+## Next Steps
+
+### Immediate (Current Sprint)
+1. Complete RAG retrieval implementation
+2. Finalize working memory eviction
+3. Add comprehensive integration tests
+4. Document API with YARD
+
+### Short-term (Next 2-4 weeks)
+1. Implement connection pooling
+2. Add OpenAI embedding provider
+3. Create migration framework
+4. Add observability (logging, metrics)
+5. Performance profiling and optimization
+
+### Long-term (Next Quarter)
+1. Production hardening
+2. Horizontal scaling strategy
+3. Advanced RAG features (re-ranking, filtering)
+4. Memory consolidation algorithms
+5. Web UI for memory exploration
+6. Publish gem to RubyGems
+
+## Conclusion
+
+HTM demonstrates a solid architectural foundation with clear separation of concerns and production-grade technology choices. The two-tier memory model with RAG-based retrieval is well-suited for LLM applications requiring contextual awareness across conversations.
+
+Key strengths include the never-forget philosophy, multi-robot hive mind, and TimescaleDB time-series optimization. Primary areas for improvement are connection management, schema evolution, and comprehensive testing.
+
+The project is positioned well for growth from prototype to production-ready gem with focused attention on connection pooling, additional embedding providers, and operational tooling.

data/.envrc

@@ -0,0 +1,32 @@
+# htm/.envrc
+
+export RR=`pwd`
+
+# Database connection - Localhost PostgreSQL
+export HTM_DBHOST=localhost
+export HTM_DBPORT=5432
+export HTM_DBNAME=htm_development
+export HTM_DBUSER=${USER}
+export HTM_DBPASS=
+export HTM_DBURL="postgresql://${HTM_DBUSER}@${HTM_DBHOST}:${HTM_DBPORT}/${HTM_DBNAME}?sslmode=disable"
+
+# Uncomment if using TimescaleDB Cloud instead:
+# export HTM_SERVICE_NAME=$TIGER_SERVICE_NAME
+# export HTM_DBURL=$TIGER_DBURL
+# export HTM_DBNAME=$TIGER_DBNAME
+# export HTM_DBUSER=$TIGER_DBUSER
+# export HTM_DBPASS=$TIGER_DBPASS
+# export HTM_DBHOST=$TIGER_DBHOST
+# export HTM_DBPORT=$TIGER_DBPORT
+
+# Client-side embedding generation
+# HTM generates embeddings before inserting into database
+export HTM_EMBEDDINGS_PROVIDER=ollama
+export HTM_EMBEDDINGS_MODEL=embeddinggemma
+export HTM_EMBEDDINGS_BASE_URL=http://localhost:11434
+export HTM_EMBEDDINGS_DIMENSION=768
+
+# Topic extraction (client-side)
+export HTM_TOPIC_PROVIDER=ollama
+export HTM_TOPIC_MODEL=phi4
+export HTM_TOPIC_BASE_URL=http://localhost:11434

data/.irbrc

@@ -0,0 +1,145 @@
+# frozen_string_literal: true
+
+# HTM IRB Configuration
+# Load this with: irb -r ./.irbrc
+# Or just: irb (if in the htm directory)
+
+puts "Loading HTM library..."
+
+# Load the HTM library
+require_relative 'lib/htm'
+
+# Establish database connection
+HTM::ActiveRecordConfig.establish_connection! unless HTM::ActiveRecordConfig.connected?
+
+# Configure HTM with Ollama for embedding and tag generation
+HTM.configure do |c|
+  c.embedding_provider   = :ollama
+  c.embedding_model      = 'nomic-embed-text'
+  c.embedding_dimensions = 768
+  c.tag_provider = :ollama
+  c.tag_model    = 'gemma3'
+  c.reset_to_defaults
+end
+
+# Convenience aliases for models
+Node    = HTM::Models::Node
+Tag     = HTM::Models::Tag
+NodeTag = HTM::Models::NodeTag
+Robot   = HTM::Models::Robot
+
+# Helper methods
+def reload!
+  puts "Reloading HTM library..."
+  load 'lib/htm.rb'
+  puts "✓ Reloaded"
+end
+
+def db_stats
+  puts <<~STATS
+
+    === Database Statistics ===
+    Nodes:     #{Node.count}
+    Tags:      #{Tag.count}
+    NodeTags:  #{NodeTag.count}
+    Robots:    #{Robot.count}
+
+  STATS
+end
+
+def recent_nodes(limit = 5)
+  puts "\n=== Recent Nodes ==="
+  Node.order(created_at: :desc).limit(limit).each do |node|
+    tags = node.tags.pluck(:name).join(', ')
+    tags_str = tags.empty? ? "(no tags)" : tags
+    puts "Node #{node.id}: #{node.content[0..60]}..."
+    puts "  Tags: #{tags_str}"
+    puts "  Embedding: #{node.embedding ? '✓' : '✗'}"
+    puts ""
+  end
+end
+
+def recent_tags(limit = 10)
+  puts "\n=== Recent Tags ==="
+  Tag.order(created_at: :desc).limit(limit).each do |tag|
+    count = tag.nodes.count
+    puts "#{tag.name} (#{count} nodes)"
+  end
+  puts
+end
+
+def search_tags(pattern)
+  puts "\n=== Tags matching '#{pattern}' ==="
+  Tag.where("name LIKE ?", "%#{pattern}%").each do |tag|
+    count = tag.nodes.count
+    puts "#{tag.name} (#{count} nodes)"
+  end
+  puts
+end
+
+def node_with_tags(node_id)
+  node = Node.includes(:tags).find(node_id)
+  embedding_info = if node.embedding
+    "✓ (#{node.embedding.size} dimensions)"
+  else
+    '✗'
+  end
+
+  puts <<~NODE_INFO
+
+    === Node #{node_id} ===
+    Content: #{node.content}
+    Source: #{node.source}
+    Created: #{node.created_at}
+    Embedding: #{embedding_info}
+
+    Tags:
+  NODE_INFO
+
+  if node.tags.any?
+    node.tags.each { |tag| puts "  - #{tag.name}" }
+  else
+    puts "  (no tags)"
+  end
+  puts
+  node
+end
+
+def create_test_node(content, source: "irb")
+  htm = HTM.new(robot_name: "IRB User")
+  node_id = htm.remember(content, source: source)
+  puts "✓ Created node #{node_id}"
+  node_id
+end
+
+def htm_help
+  puts <<~WELCOME
+
+    ============================================================
+      HTM Interactive Console
+    ============================================================
+
+    Available models:
+      - Node      (HTM::Models::Node)
+      - Tag       (HTM::Models::Tag)
+      - NodeTag   (HTM::Models::NodeTag)
+      - Robot     (HTM::Models::Robot)
+
+    Helper methods:
+      htm_help              # Reprints this message
+      db_stats              # Show database statistics
+      recent_nodes(n)       # Show n recent nodes (default: 5)
+      recent_tags(n)        # Show n recent tags (default: 10)
+      search_tags(pattern)  # Search tags by pattern
+      node_with_tags(id)    # Show node details with tags
+      create_test_node(str) # Create a test node
+      reload!               # Reload HTM library
+
+    Database: #{HTM::Database.default_config[:dbname]}
+  WELCOME
+end
+
+htm_help
+db_stats
+
+print "HTM Ready!\n\n"

data/CHANGELOG.md

@@ -0,0 +1,150 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Added
+- Architecture documentation using ai-software-architect framework
+- Comprehensive ADRs (Architecture Decision Records):
+  - ADR-001: PostgreSQL with TimescaleDB for storage
+  - ADR-002: Two-tier memory architecture (working + long-term)
+  - ADR-003: Ollama as default embedding provider
+  - ADR-004: Multi-robot shared memory (hive mind)
+  - ADR-005: RAG-based retrieval with hybrid search
+  - ADR-006: Context assembly strategies (recent, important, balanced)
+  - ADR-007: Working memory eviction strategy (hybrid importance + recency)
+  - ADR-008: Robot identification system (UUID + name)
+  - ADR-009: Never-forget philosophy with explicit deletion
+- Architecture review team with 8 specialist perspectives
+- Had the robot convert my notss and system analysis documentation into Architectural Decision Records (ADR)
+
+## [0.1.0] - 2025-10-25
+
+### Added
+- Initial release of HTM (Hierarchical Temporary Memory)
+- Two-tier memory system:
+  - Working memory: Token-limited, in-memory active context
+  - Long-term memory: Durable PostgreSQL/TimescaleDB storage
+- Core memory operations:
+  - `add_node`: Store memories with metadata, embeddings, and relationships
+  - `recall`: RAG-based retrieval with temporal and semantic search
+  - `retrieve`: Direct memory lookup by key
+  - `forget`: Explicit deletion with confirmation requirement
+  - `create_context`: Assemble LLM context from working memory
+- Multi-robot "hive mind" architecture:
+  - Shared global memory database
+  - Robot attribution tracking
+  - Robot registry and activity monitoring
+  - Cross-robot knowledge sharing
+- Search strategies:
+  - Vector search: Semantic similarity using pgvector
+  - Full-text search: Keyword matching with PostgreSQL full-text search
+  - Hybrid search: Combined pre-filter + vector reranking
+  - Temporal filtering: Time-range queries with TimescaleDB
+- Context assembly strategies:
+  - Recent: Most recently accessed memories first
+  - Important: Highest importance score first
+  - Balanced: Hybrid with time-decay function (default)
+- Working memory management:
+  - Configurable token limits (default: 128,000)
+  - Hybrid eviction strategy (importance + recency)
+  - LRU access tracking
+  - Automatic eviction to long-term storage
+- Long-term memory features:
+  - PostgreSQL 17+ with TimescaleDB 2.22.1
+  - Vector embeddings with pgvector 0.8.1 (HNSW indexing)
+  - Full-text search with pg_trgm
+  - Relationship graphs between memories
+  - Tag system for categorization
+  - Operations logging and audit trail
+  - Memory statistics and analytics
+- Embedding service:
+  - Default: Ollama with gpt-oss model (local-first)
+  - Support for multiple providers (OpenAI, Cohere, local)
+  - Configurable models and endpoints
+  - Accurate token counting with tiktoken_ruby
+- Database schema:
+  - `nodes`: Core memory storage with embeddings
+  - `relationships`: Graph connections between memories
+  - `tags`: Flexible categorization system
+  - `robots`: Robot registry and activity tracking
+  - `operations_log`: Audit trail for all operations
+  - TimescaleDB hypertables for time-series optimization
+  - PostgreSQL views for statistics
+- Memory types:
+  - `:fact`: Factual information
+  - `:context`: Contextual information
+  - `:code`: Code snippets and technical content
+  - `:preference`: User preferences
+  - `:decision`: Architectural and strategic decisions
+  - `:question`: Questions and queries
+- Memory metadata:
+  - Importance scoring (0.0-10.0)
+  - Token counting
+  - Timestamps (created_at, last_accessed)
+  - Robot attribution
+  - Categories and tags
+  - Relationships to other memories
+- Robot identification:
+  - UUID-based robot_id (auto-generated)
+  - Optional human-readable robot_name
+  - Robot registry with activity tracking
+  - Memory attribution by robot
+- Never-forget philosophy:
+  - Memories never automatically deleted
+  - Eviction moves to long-term storage (no data loss)
+  - Explicit confirmation required for deletion (`:confirmed` symbol)
+  - All deletions logged for audit trail
+- Database utilities:
+  - Schema creation and migration scripts
+  - Extension installation (TimescaleDB, pgvector, pg_trgm)
+  - Hypertable configuration
+  - Compression policies
+  - Index creation for performance
+- Development tools:
+  - Comprehensive test suite (Minitest)
+  - Example scripts and usage patterns
+  - Rakefile with common tasks
+  - Environment configuration with direnv
+- Documentation:
+  - README with quick start guide
+  - SETUP.md with detailed installation instructions
+  - CLAUDE.md for AI assistant context
+  - Architecture documentation in `.architecture/`
+  - Inline code documentation
+
+### Dependencies
+- Ruby 3.0+
+- PostgreSQL 17+
+- TimescaleDB 2.22.1
+- pgvector 0.8.1
+- pg gem (~> 1.5)
+- pgvector gem (~> 0.8)
+- connection_pool gem (~> 2.4)
+- tiktoken_ruby gem (~> 0.0.9)
+- ruby-llm gem (~> 0.7.1)
+
+### Database Requirements
+- PostgreSQL 17+ with extensions:
+  - timescaledb (2.22.1+)
+  - vector (0.8.1+)
+  - pg_trgm (1.6+)
+- Recommended: TimescaleDB Cloud or local TimescaleDB installation
+
+### Environment Variables
+- `HTM_DBURL`: PostgreSQL connection string (required)
+- `OLLAMA_URL`: Ollama API endpoint (default: http://localhost:11434)
+
+### Notes
+- This is an initial release focused on core functionality
+- Database schema is stable but may evolve in future versions
+- Embedding models and providers are configurable
+- Working memory size is user-configurable
+- See ADRs for detailed architectural decisions and rationale
+
+[Unreleased]: https://github.com/madbomber/htm/compare/v0.1.0...HEAD
+[0.1.0]: https://github.com/madbomber/htm/releases/tag/v0.1.0