htm 0.0.11 → 0.0.15

data/notes/ARCHITECTURE_REVIEW.md

@@ -1,1167 +0,0 @@
-# HTM Architecture Review
-
-**Review Date**: 2025-10-25
-**HTM Version**: 0.1.0
-**Reviewers**: Multi-disciplinary architecture team
-
-## Executive Summary
-
-HTM (Hierarchical Temporary Memory) implements a sophisticated two-tier memory system for LLM-based applications. This review evaluates the architecture from six specialist perspectives, identifying strengths, weaknesses, and recommendations for improvement.
-
-**Overall Assessment**: ⭐⭐⭐⭐ (4/5)
-
-The architecture demonstrates solid design fundamentals with well-documented decisions via ADRs. Key strengths include the two-tier memory model, RAG-based retrieval, and hive mind capabilities. Primary areas for improvement include connection pooling, error handling, and thread safety.
-
----
-
-## 1. Systems Architect Perspective
-
-**Reviewer**: Systems Architecture Specialist
-
-### Strengths
-
-#### ✅ Two-Tier Memory Model (ADR-002)
-The separation of hot working memory and cold long-term storage is architecturally sound:
-- Clear responsibility separation (`lib/htm.rb:41-363`, `lib/htm/working_memory.rb`, `lib/htm/long_term_memory.rb`)
-- Appropriate technology choices for each tier
-- Well-defined eviction strategy
-
-#### ✅ Component Cohesion
-Each class has a single, clear responsibility:
-- `HTM` - Coordination layer (`lib/htm.rb:41`)
-- `WorkingMemory` - Token-limited active context (`lib/htm/working_memory.rb:9`)
-- `LongTermMemory` - Persistent storage (`lib/htm/long_term_memory.rb:17`)
-- `EmbeddingService` - Vector generation (`lib/htm/embedding_service.rb:15`)
-- `Database` - Schema management (`lib/htm/database.rb:9`)
-
-#### ✅ Hive Mind Architecture (ADR-004)
-Shared global memory with robot attribution enables cross-robot learning:
-- Robot registration (`lib/htm.rb:294-296`)
-- Activity tracking (`lib/htm.rb:298-300`)
-- Multi-robot queries (`lib/htm.rb:253-290`)
-
-### Concerns
-
-#### ⚠️ No Connection Pooling
-**Location**: `lib/htm/long_term_memory.rb:315-325`
-
-```ruby
-def with_connection
-  conn = PG.connect(@config)
-  result = yield(conn)
-  conn.close
-  result
-end
-```
-
-**Issue**: Creates a new database connection for every operation. Under load, this will:
-- Exhaust connection limits
-- Introduce latency (connection handshake overhead)
-- Waste resources (connection creation/teardown)
-
-**Recommendation**: Implement connection pooling
-
-```ruby
-require 'connection_pool'
-
-class LongTermMemory
-  def initialize(config)
-    @pool = ConnectionPool.new(size: 5, timeout: 5) do
-      PG.connect(config)
-    end
-  end
-
-  def with_connection
-    @pool.with { |conn| yield(conn) }
-  end
-end
-```
-
-#### ⚠️ Synchronous Embedding Generation
-**Location**: `lib/htm.rb:84-126`
-
-Every `add_node` call blocks on embedding generation:
-```ruby
-def add_node(key, value, ...)
-  embedding = @embedding_service.embed(value)  # Blocks here
-  # ...
-end
-```
-
-For large text or slow embedding services, this creates latency.
-
-**Recommendation**: Consider async embedding with job queue for non-critical paths
-
-#### ⚠️ No Circuit Breaker Pattern
-**Location**: `lib/htm/embedding_service.rb:70-108`
-
-Ollama failures fall back to random vectors with only a warning:
-```ruby
-rescue => e
-  warn "Error generating embedding with Ollama: #{e.message}"
-  Array.new(1536) { rand(-1.0..1.0) }  # Random fallback
-end
-```
-
-**Issue**: Silent degradation - the system continues with meaningless embeddings.
-
-**Recommendation**: Implement circuit breaker or explicit failure mode
-
-### Architecture Scalability
-
-| Aspect | Current State | Scalability Limit | Recommendation |
-|--------|---------------|-------------------|----------------|
-| **Working Memory** | Per-process in-memory | ~2GB RAM | ✅ Acceptable (process-local is intentional) |
-| **Database Reads** | No connection pool | ~100 concurrent connections | ⚠️ Add connection pooling |
-| **Database Writes** | No batching | ~1000 ops/sec | ⚠️ Consider batch inserts for bulk operations |
-| **Embeddings** | Synchronous, per-request | ~10 req/sec (Ollama bottleneck) | ⚠️ Add async queue for high-throughput scenarios |
-
-### Recommendations
-
-1. **High Priority**: Implement connection pooling
-2. **Medium Priority**: Add circuit breaker for embedding service
-3. **Low Priority**: Consider async embedding for high-throughput scenarios
-4. **Documentation**: Add deployment architecture diagram
-
----
-
-## 2. Database Architect Perspective
-
-**Reviewer**: PostgreSQL/TimescaleDB Specialist
-
-### Strengths
-
-#### ✅ TimescaleDB Integration (ADR-001)
-Excellent choice for time-series workloads:
-- Hypertable partitioning (`lib/htm/database.rb:143-153`)
-- Automatic compression (`lib/htm/database.rb:165-177`)
-- Time-range query optimization
-
-**Schema** (`sql/schema.sql:8-22`):
-```sql
-CREATE TABLE nodes (
-  id BIGSERIAL PRIMARY KEY,
-  created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-  embedding vector(1536),
-  ...
-);
-
--- Hypertable conversion
-SELECT create_hypertable('nodes', 'created_at', if_not_exists => TRUE);
-```
-
-#### ✅ Comprehensive Indexing
-Well-thought-out index strategy (`sql/schema.sql:64-95`):
-- **B-tree**: Standard queries (created_at, robot_id, type)
-- **HNSW**: Vector similarity (`idx_nodes_embedding`)
-- **GIN**: Full-text search (`idx_nodes_value_gin`)
-- **GIN Trigram**: Fuzzy matching (`idx_nodes_value_trgm`)
-
-The HNSW index configuration is appropriate:
-```sql
-CREATE INDEX idx_nodes_embedding ON nodes
-  USING hnsw (embedding vector_cosine_ops)
-  WITH (m = 16, ef_construction = 64);
-```
-
-#### ✅ Referential Integrity
-Proper foreign key constraints with cascade rules:
-```sql
-CREATE TABLE relationships (
-  from_node_id BIGINT REFERENCES nodes(id) ON DELETE CASCADE,
-  to_node_id BIGINT REFERENCES nodes(id) ON DELETE CASCADE,
-  ...
-);
-```
-
-### Concerns
-
-#### ⚠️ Missing Index on `in_working_memory`
-**Location**: `sql/schema.sql:70`
-
-```sql
-CREATE INDEX idx_nodes_in_working_memory ON nodes(in_working_memory);
-```
-
-**Issue**: This index exists but the column is never queried in the codebase. Either:
-1. The index is unused (wasted space/write overhead)
-2. There's a missing query that should use it
-
-**Recommendation**: Audit index usage or remove if unused
-
-#### ⚠️ No Partitioning Strategy for Large Datasets
-**Location**: Hypertable setup doesn't specify chunk interval
-
-```sql
-SELECT create_hypertable('nodes', 'created_at',
-  if_not_exists => TRUE,
-  migrate_data => TRUE
-);
-```
-
-**Missing**: `chunk_time_interval` parameter. Default is 7 days, which may not be optimal.
-
-**Recommendation**: Explicitly set chunk interval based on query patterns:
-
-```sql
-SELECT create_hypertable('nodes', 'created_at',
-  chunk_time_interval => INTERVAL '30 days',  -- Explicit
-  if_not_exists => TRUE
-);
-```
-
-#### ⚠️ Hybrid Search Performance
-**Location**: `lib/htm/long_term_memory.rb:159-182`
-
-The hybrid search uses a CTE (Common Table Expression):
-```sql
-WITH candidates AS (
-  SELECT * FROM nodes
-  WHERE to_tsvector(...) @@ plainto_tsquery(...)
-  LIMIT $5  -- prefilter_limit
-)
-SELECT ... FROM candidates
-ORDER BY embedding <=> $4::vector
-LIMIT $6
-```
-
-**Concern**: For large result sets, the `prefilter_limit` parameter (default 100) may:
-- Miss relevant results if full-text search returns many matches
-- Perform poorly if full-text returns few matches (unnecessary CTE overhead)
-
-**Recommendation**: Add EXPLAIN ANALYZE tests for different `prefilter_limit` values
-
-#### ⚠️ No Query Timeout
-**Location**: All database queries lack timeouts
-
-Long-running queries can block the system indefinitely.
-
-**Recommendation**: Add statement timeout:
-
-```ruby
-def with_connection
-  @pool.with do |conn|
-    conn.exec("SET statement_timeout = '30s'")  # Global timeout
-    yield(conn)
-  end
-end
-```
-
-### Schema Normalization Analysis
-
-| Table | Normal Form | Notes |
-|-------|-------------|-------|
-| `nodes` | 3NF | ✅ Well-normalized |
-| `relationships` | 3NF | ✅ Properly normalized |
-| `tags` | 3NF | ✅ Many-to-many via junction table |
-| `operations_log` | 3NF | ✅ JSONB used appropriately for flexible metadata |
-
-### Recommendations
-
-1. **High Priority**: Add query timeouts
-2. **High Priority**: Audit and optimize hybrid search `prefilter_limit`
-3. **Medium Priority**: Explicitly set TimescaleDB chunk intervals
-4. **Low Priority**: Review `in_working_memory` index usage
-5. **Monitoring**: Add slow query logging and EXPLAIN ANALYZE for critical paths
-
----
-
-## 3. AI Engineer Perspective
-
-**Reviewer**: Machine Learning Infrastructure Specialist
-
-### Strengths
-
-#### ✅ RAG Pattern Implementation (ADR-005)
-Solid implementation of retrieval-augmented generation:
-- Temporal filtering (time-range queries)
-- Semantic search (vector similarity)
-- Full-text search (keyword matching)
-- Hybrid approach (combines both)
-
-**Hybrid Search** (`lib/htm/long_term_memory.rb:159-182`):
-```ruby
-def search_hybrid(timeframe:, query:, limit:, embedding_service:)
-  # 1. Full-text prefilter (narrows search space)
-  # 2. Vector similarity on candidates
-  # = Best of both worlds
-end
-```
-
-#### ✅ Embedding Abstraction (ADR-003)
-Provider-agnostic design supports multiple embedding services:
-- Ollama (default, local)
-- OpenAI (cloud, production)
-- Cohere (cloud alternative)
-- Local transformers (edge deployment)
-
-**Interface** (`lib/htm/embedding_service.rb:41-54`):
-```ruby
-def embed(text)
-  case @provider
-  when :ollama then embed_ollama(text)
-  when :openai then embed_openai(text)
-  # ...
-  end
-end
-```
-
-#### ✅ Importance Scoring
-Explicit importance weighting enables better context assembly:
-```ruby
-def assemble_context(strategy:)
-  case strategy
-  when :balanced
-    # Hybrid: importance × time-decay
-    @nodes.sort_by { |k, v|
-      recency = Time.now - v[:added_at]
-      -(v[:importance] * (1.0 / (1 + recency / 3600.0)))
-    }
-  end
-end
-```
-
-### Concerns
-
-#### ⚠️ Fixed Embedding Dimensions
-**Location**: `sql/schema.sql:21`, `lib/htm/embedding_service.rb:107,126,132,138`
-
-All embeddings assumed to be 1536 dimensions (OpenAI text-embedding-3-small):
-```sql
-embedding vector(1536)  -- Hard-coded
-```
-
-**Issue**: Other models use different dimensions:
-- `nomic-embed-text`: 768 dimensions
-- `text-embedding-3-large`: 3072 dimensions
-- `gpt-oss`: **Dimension unknown** - defaults to 1536 but may be incorrect
-
-**Recommendation**: Make embedding dimensions configurable:
-
-```ruby
-class EmbeddingService
-  DIMENSIONS = {
-    'gpt-oss' => 768,           # Verify this!
-    'text-embedding-3-small' => 1536,
-    'text-embedding-3-large' => 3072,
-    'nomic-embed-text' => 768
-  }
-
-  def initialize(provider, model:)
-    @dimensions = DIMENSIONS.fetch(model) do
-      raise "Unknown embedding dimensions for model: #{model}"
-    end
-  end
-end
-```
-
-#### ⚠️ No Embedding Caching
-**Location**: `lib/htm.rb:84-126`
-
-Every `add_node` call generates a new embedding, even for duplicate text.
-
-**Recommendation**: Add embedding cache:
-
-```ruby
-class EmbeddingService
-  def initialize(...)
-    @cache = {}  # or Redis for distributed cache
-  end
-
-  def embed(text)
-    cache_key = Digest::SHA256.hexdigest(text)
-    @cache[cache_key] ||= embed_uncached(text)
-  end
-end
-```
-
-#### ⚠️ Similarity Score Not Returned
-**Location**: `lib/htm.rb:136-178`
-
-The `recall` method doesn't return similarity scores:
-```ruby
-def recall(timeframe:, topic:, limit:)
-  nodes = @long_term_memory.search(...)
-  # nodes contain similarity scores, but they're not exposed to caller
-  nodes
-end
-```
-
-**Issue**: Caller can't:
-- Filter low-relevance results
-- Implement confidence thresholds
-- Debug retrieval quality
-
-**Recommendation**: Return structured results with scores:
-
-```ruby
-def recall(...)
-  nodes.map do |node|
-    {
-      node: node,
-      similarity: node['similarity'],  # Expose score
-      retrieval_method: strategy
-    }
-  end
-end
-```
-
-#### ⚠️ No Re-ranking
-**Location**: Hybrid search uses simple approaches without re-ranking
-
-Modern RAG systems often use cross-encoder re-ranking for better precision.
-
-**Recommendation**: Consider adding optional re-ranking stage for production use
-
-### Embedding Quality Analysis
-
-| Aspect | Current State | Recommendation |
-|--------|---------------|----------------|
-| **Model Choice** | gpt-oss (Ollama) | ⚠️ Verify dimensions and quality vs alternatives |
-| **Dimension Handling** | Fixed 1536 | ⚠️ Make configurable per model |
-| **Caching** | None | ⚠️ Add for duplicate text |
-| **Normalization** | Handled by pgvector | ✅ Good |
-| **Distance Metric** | Cosine similarity | ✅ Appropriate for semantic search |
-
-### Recommendations
-
-1. **High Priority**: Fix embedding dimension handling
-2. **High Priority**: Return similarity scores from `recall`
-3. **Medium Priority**: Add embedding caching
-4. **Low Priority**: Benchmark gpt-oss vs other models
-5. **Low Priority**: Consider re-ranking for production
-
----
-
-## 4. Performance Specialist Perspective
-
-**Reviewer**: System Performance Engineer
-
-### Strengths
-
-#### ✅ Working Memory Performance
-O(1) hash-based lookups and efficient token counting:
-```ruby
-class WorkingMemory
-  def initialize(max_tokens:)
-    @nodes = {}           # O(1) access
-    @access_order = []    # LRU tracking
-  end
-end
-```
-
-#### ✅ HNSW Index for Vector Search
-Approximate nearest neighbor search with good recall/speed tradeoff:
-```sql
-USING hnsw (embedding vector_cosine_ops)
-WITH (m = 16, ef_construction = 64)
-```
-
-**Expected Performance**:
-- m=16: Good balance (16 neighbors per layer)
-- ef_construction=64: Moderate build time, good quality
-
-### Concerns
-
-#### ⚠️ Eviction Algorithm Complexity
-**Location**: `lib/htm/working_memory.rb:66-86`
-
-```ruby
-def evict_to_make_space(needed_tokens)
-  candidates = @nodes.sort_by do |key, node|
-    recency = Time.now - node[:added_at]
-    [node[:importance], -recency]
-  end
-  # ...
-end
-```
-
-**Complexity**: O(n log n) where n = number of nodes in working memory
-
-**Issue**: For large working memories (>1000 nodes), sorting on every eviction is expensive.
-
-**Recommendation**: Maintain a priority queue (min-heap) for O(log n) evictions:
-
-```ruby
-require 'priority_queue'
-
-class WorkingMemory
-  def initialize(max_tokens:)
-    @eviction_queue = PriorityQueue.new  # Sorted by eviction score
-  end
-
-  def evict_to_make_space(needed_tokens)
-    # Pop from heap: O(log n) per eviction
-    evicted = []
-    tokens_freed = 0
-    while tokens_freed < needed_tokens
-      key, score = @eviction_queue.pop
-      # ...
-    end
-  end
-end
-```
-
-#### ⚠️ Context Assembly Sorts Every Time
-**Location**: `lib/htm/working_memory.rb:94-122`
-
-```ruby
-def assemble_context(strategy:, max_tokens:)
-  nodes = case strategy
-  when :balanced
-    @nodes.sort_by { ... }  # Full sort every time
-  end
-end
-```
-
-**Issue**: If context is assembled frequently (every LLM call), this is O(n log n) overhead.
-
-**Recommendation**: Cache sorted results and invalidate on modification
-
-#### ⚠️ No Query Result Caching
-**Location**: `lib/htm/long_term_memory.rb:106-123`
-
-Identical queries hit the database every time:
-```ruby
-def search(timeframe:, query:, limit:, embedding_service:)
-  query_embedding = embedding_service.embed(query)  # Expensive
-  with_connection do |conn|
-    conn.exec_params(...)  # Database round-trip
-  end
-end
-```
-
-**Recommendation**: Add LRU cache for recent queries:
-
-```ruby
-require 'lru_redux'
-
-class LongTermMemory
-  def initialize(config)
-    @query_cache = LruRedux::ThreadSafeCache.new(1000)
-  end
-
-  def search(...)
-    cache_key = "#{timeframe}/#{query}/#{limit}"
-    @query_cache.getset(cache_key) do
-      search_uncached(...)
-    end
-  end
-end
-```
-
-### Performance Benchmark Estimates
-
-| Operation | Current Performance | Optimized Performance | Notes |
-|-----------|---------------------|----------------------|-------|
-| **add_node** | ~100ms | ~50ms | With connection pooling |
-| **recall (vector)** | ~200ms | ~100ms | With query caching |
-| **recall (hybrid)** | ~250ms | ~150ms | With query caching |
-| **evict** | ~10ms (100 nodes) | ~2ms | With priority queue |
-| **assemble_context** | ~5ms (100 nodes) | ~0.5ms | With caching |
-
-### Recommendations
-
-1. **High Priority**: Add connection pooling (5x improvement on DB ops)
-2. **High Priority**: Implement query result caching
-3. **Medium Priority**: Optimize eviction with priority queue
-4. **Medium Priority**: Cache sorted context assembly results
-5. **Monitoring**: Add performance instrumentation (StatsD/Prometheus)
-
----
-
-## 5. Ruby Expert Perspective
-
-**Reviewer**: Ruby Best Practices Specialist
-
-### Strengths
-
-#### ✅ Idiomatic Ruby Style
-Code follows Ruby community conventions:
-- Keyword arguments for clarity
-- Symbols for enums
-- Proper attr_reader usage
-- Frozen string literals
-
-**Example** (`lib/htm.rb:53-71`):
-```ruby
-def initialize(
-  working_memory_size: 128_000,        # Keyword arg with default
-  robot_id: nil,
-  robot_name: nil,
-  embedding_service: :ollama           # Symbol for enum
-)
-  @robot_id = robot_id || SecureRandom.uuid  # Idiomatic fallback
-  # ...
-end
-```
-
-#### ✅ Clear Method Signatures
-Methods have descriptive names and appropriate arity:
-```ruby
-def add_node(key, value, type: nil, category: nil, importance: 1.0, ...)
-def recall(timeframe:, topic:, limit: 20, strategy: :vector)
-def forget(key, confirm: false)  # Safety through explicit confirmation
-```
-
-#### ✅ Proper Encapsulation
-Private methods used appropriately:
-```ruby
-private
-
-def register_robot
-  # ...
-end
-
-def add_to_working_memory(node)
-  # ...
-end
-```
-
-### Concerns
-
-#### ⚠️ No Thread Safety
-**Location**: All classes lack synchronization
-
-```ruby
-class WorkingMemory
-  def add(key, value, ...)
-    @nodes[key] = { ... }  # Not thread-safe
-    update_access(key)     # Modifies @access_order
-  end
-end
-```
-
-**Issue**: Concurrent access from multiple threads will cause:
-- Race conditions in `@nodes` hash
-- Corrupted `@access_order` array
-- Incorrect token counts
-
-**Recommendation**: Add explicit documentation or thread-safety:
-
-```ruby
-# Option 1: Document thread-unsafety
-class WorkingMemory
-  # NOT THREAD-SAFE: Use one instance per thread or add external synchronization
-  def initialize(max_tokens:)
-    # ...
-  end
-end
-
-# Option 2: Add mutex for thread-safety
-class WorkingMemory
-  def initialize(max_tokens:)
-    @mutex = Mutex.new
-    @nodes = {}
-  end
-
-  def add(key, value, ...)
-    @mutex.synchronize do
-      @nodes[key] = { ... }
-    end
-  end
-end
-```
-
-#### ⚠️ Exception Handling Inconsistency
-**Location**: Different error handling strategies across classes
-
-**EmbeddingService** (`lib/htm/embedding_service.rb:103-108`):
-```ruby
-rescue => e
-  warn "Error: #{e.message}"
-  Array.new(1536) { rand }  # Silent fallback
-end
-```
-
-**LongTermMemory** (`lib/htm/long_term_memory.rb:322-325`):
-```ruby
-rescue => e
-  conn&.close
-  raise e  # Re-raises
-end
-```
-
-**Inconsistency**: Some methods fail silently, others raise exceptions.
-
-**Recommendation**: Define error handling policy:
-- Critical operations: Raise custom exceptions
-- Optional features: Return nil or Result monad
-- Never: Silent fallback with random data
-
-#### ⚠️ Mutable Default Arguments
-**Location**: No instances found, but worth noting for future development
-
-Ruby gotcha to avoid:
-```ruby
-# BAD
-def add_tags(node_id, tags = [])
-  tags << "default"  # Mutates shared default!
-end
-
-# GOOD
-def add_tags(node_id, tags: [])
-  tags = tags.dup  # Defensive copy
-  # ...
-end
-```
-
-Current code doesn't have this issue ✅
-
-#### ⚠️ Missing Return Value Documentation
-**Location**: Several methods unclear about return values
-
-```ruby
-def mark_evicted(keys)
-  return if keys.empty?
-  # ...
-  # Returns what? nil? true? count of rows updated?
-end
-```
-
-**Recommendation**: Use YARD comments consistently:
-
-```ruby
-# Mark nodes as evicted from working memory
-#
-# @param keys [Array<String>] Node keys
-# @return [void]
-def mark_evicted(keys)
-  # ...
-end
-```
-
-### Ruby Patterns Analysis
-
-| Pattern | Usage | Recommendation |
-|---------|-------|----------------|
-| **Dependency Injection** | ✅ Used (embedding_service, db_config) | Good |
-| **Factory Pattern** | ⚠️ Not used | Consider for embedding service creation |
-| **Observer Pattern** | ❌ Not used | Could be useful for memory events |
-| **Result Objects** | ❌ Not used | Would improve error handling |
-| **Value Objects** | ❌ Not used | Could wrap timeframes, embeddings |
-
-### Recommendations
-
-1. **High Priority**: Document thread-safety guarantees (or lack thereof)
-2. **High Priority**: Standardize error handling approach
-3. **Medium Priority**: Add comprehensive YARD documentation
-4. **Low Priority**: Consider introducing Result objects for operations that can fail
-5. **Code Quality**: Add RuboCop configuration for consistent style
-
----
-
-## 6. Security Specialist Perspective
-
-**Reviewer**: Application Security Engineer
-
-### Strengths
-
-#### ✅ SQL Injection Protection
-All queries use parameterized statements:
-```ruby
-conn.exec_params(
-  "INSERT INTO nodes (...) VALUES ($1, $2, $3, ...)",
-  [key, value, type, ...]  # ✅ Parameters, not string interpolation
-)
-```
-
-✅ **No instances of string interpolation in SQL queries found**
-
-#### ✅ Explicit Deletion Confirmation (ADR-009)
-The `forget` method requires explicit confirmation:
-```ruby
-def forget(key, confirm: false)
-  raise ArgumentError, "Must pass confirm: :confirmed" unless confirm == :confirmed
-  # ...
-end
-```
-
-Prevents accidental data loss ✅
-
-#### ✅ SSL/TLS Configuration
-Database connection supports SSL:
-```ruby
-def parse_connection_url(url)
-  {
-    sslmode: params['sslmode'] || 'prefer'  # SSL by default
-  }
-end
-```
-
-### Concerns
-
-#### ⚠️ Database Credentials in Environment Variables
-**Location**: `lib/htm/database.rb:79-87`
-
-```ruby
-def default_config
-  if ENV['HTM_DBURL']
-    parse_connection_url(ENV['HTM_DBURL'])
-  elsif ENV['HTM_DBNAME']
-    {
-      password: ENV['HTM_DBPASS'],  # Password in env var
-      # ...
-    }
-  end
-end
-```
-
-**Issue**: Environment variables are:
-- Visible in process listings (`ps aux`)
-- Logged in various places
-- Inherited by child processes
-
-**Recommendation**: Support external secret management:
-
-```ruby
-def default_config
-  # Option 1: Read from file
-  if File.exist?('/run/secrets/db_password')
-    password = File.read('/run/secrets/db_password').strip
-  end
-
-  # Option 2: Use secret management service
-  if ENV['USE_AWS_SECRETS_MANAGER']
-    password = AwsSecretsManager.get('htm/db_password')
-  end
-
-  # Option 3: Fall back to env var for development
-  password ||= ENV['HTM_DBPASS']
-end
-```
-
-#### ⚠️ No Input Validation
-**Location**: Public methods lack input sanitization
-
-```ruby
-def add_node(key, value, type: nil, ...)
-  # No validation:
-  # - key length? (could exceed column size)
-  # - value length? (could be gigabytes)
-  # - type enum? (any string accepted)
-  # - importance range? (could be negative or huge)
-
-  @long_term_memory.add(key: key, value: value, ...)
-end
-```
-
-**Risks**:
-- DoS via large inputs
-- Database errors from invalid types
-- Undefined behavior from out-of-range values
-
-**Recommendation**: Add input validation:
-
-```ruby
-class HTM
-  MAX_KEY_LENGTH = 255
-  MAX_VALUE_LENGTH = 1_000_000  # 1MB
-  VALID_TYPES = [:fact, :context, :code, :preference, :decision, :question]
-
-  def add_node(key, value, type: nil, importance: 1.0, ...)
-    validate_key!(key)
-    validate_value!(value)
-    validate_type!(type) if type
-    validate_importance!(importance)
-
-    # ...
-  end
-
-  private
-
-  def validate_key!(key)
-    raise ArgumentError, "Key cannot be nil" if key.nil?
-    raise ArgumentError, "Key too long (max #{MAX_KEY_LENGTH})" if key.length > MAX_KEY_LENGTH
-    raise ArgumentError, "Key cannot be empty" if key.empty?
-  end
-
-  def validate_importance!(importance)
-    raise ArgumentError, "Importance must be 0.0-10.0" unless (0.0..10.0).cover?(importance)
-  end
-end
-```
-
-#### ⚠️ Embedding Service Failures Silently Degrade
-**Location**: `lib/htm/embedding_service.rb:103-108`
-
-```ruby
-rescue => e
-  warn "Error generating embedding with Ollama: #{e.message}"
-  warn "Falling back to stub embeddings (random vectors)"
-  Array.new(1536) { rand(-1.0..1.0) }  # ⚠️ Security/reliability issue
-end
-```
-
-**Security Issue**: Random embeddings break semantic search, allowing:
-- Incorrect memory retrieval
-- Potential information leakage (wrong context returned)
-- Silent data corruption
-
-**Recommendation**: Fail loudly or implement proper fallback:
-
-```ruby
-rescue => e
-  if @allow_fallback
-    warn "CRITICAL: Embedding service failure, using fallback"
-    Array.new(1536) { 0.0 }  # Zero vector (at least deterministic)
-  else
-    raise EmbeddingServiceError, "Failed to generate embedding: #{e.message}"
-  end
-end
-```
-
-#### ⚠️ No Rate Limiting
-**Location**: All public methods lack rate limiting
-
-In a multi-robot environment, a compromised or misconfigured robot could:
-- Flood the system with `add_node` calls
-- Execute expensive searches continuously
-- Exhaust database resources
-
-**Recommendation**: Add rate limiting:
-
-```ruby
-require 'redis'
-require 'redis-throttle'
-
-class HTM
-  def initialize(...)
-    @throttle = Redis::Throttle.new(
-      key: "htm:#{@robot_id}",
-      limit: 100,       # 100 operations
-      period: 60        # per minute
-    )
-  end
-
-  def add_node(key, value, ...)
-    @throttle.exceeded? and raise RateLimitExceeded
-
-    # ...
-  end
-end
-```
-
-#### ⚠️ Operations Log Contains Potentially Sensitive Data
-**Location**: `lib/htm.rb:116-122`
-
-```ruby
-@long_term_memory.log_operation(
-  operation: 'add',
-  node_id: node_id,
-  robot_id: @robot_id,
-  details: { key: key, type: type }  # What if key contains sensitive data?
-)
-```
-
-**Issue**: The `operations_log` table stores details in JSONB, which might include:
-- Sensitive keys
-- Query terms revealing confidential information
-- User identifiers
-
-**Recommendation**: Add PII scrubbing for audit logs
-
-### Security Checklist
-
-| Concern | Status | Priority |
-|---------|--------|----------|
-| SQL Injection | ✅ Protected | - |
-| XSS | N/A (no web interface) | - |
-| CSRF | N/A (no web interface) | - |
-| Authentication | ❌ Not implemented | Low (out of scope) |
-| Authorization | ❌ Not implemented | Medium (robot access control) |
-| Input Validation | ⚠️ Missing | High |
-| Output Encoding | ✅ Handled by pg gem | - |
-| Secret Management | ⚠️ Env vars only | Medium |
-| Rate Limiting | ❌ Not implemented | Medium |
-| Audit Logging | ✅ Implemented | - |
-| PII Handling | ⚠️ Not scrubbed | Medium |
-
-### Recommendations
-
-1. **High Priority**: Add input validation for all public methods
-2. **High Priority**: Fail loudly on embedding service failures (no random fallbacks)
-3. **Medium Priority**: Implement robot-level authorization
-4. **Medium Priority**: Add rate limiting
-5. **Medium Priority**: Support external secret management (Docker secrets, Vault, etc.)
-6. **Low Priority**: Add PII scrubbing for audit logs
-7. **Documentation**: Add security considerations section to docs
-
----
-
-## Cross-Cutting Concerns
-
-### Observability
-
-**Current State**: Minimal observability
-
-**Issues**:
-- No structured logging
-- No metrics collection (request rates, latencies, errors)
-- No distributed tracing
-- No health checks
-
-**Recommendations**:
-
-```ruby
-require 'logger'
-require 'statsd'
-
-class HTM
-  def initialize(...)
-    @logger = Logger.new(STDOUT)
-    @logger.formatter = JsonFormatter.new  # Structured logging
-
-    @metrics = Statsd.new('localhost', 8125)
-  end
-
-  def add_node(key, value, ...)
-    start_time = Time.now
-
-    @logger.info("Adding node", {
-      robot_id: @robot_id,
-      key: key,
-      type: type
-    })
-
-    begin
-      # ... operation ...
-
-      @metrics.increment('htm.add_node.success')
-      @metrics.histogram('htm.add_node.duration', Time.now - start_time)
-    rescue => e
-      @logger.error("Failed to add node", {
-        error: e.class.name,
-        message: e.message,
-        backtrace: e.backtrace[0..5]
-      })
-      @metrics.increment('htm.add_node.error')
-      raise
-    end
-  end
-end
-```
-
-### Testing
-
-**Current State**: Basic test coverage (from conversation summary: "2 errors, 3 failures")
-
-**Recommendations**:
-1. Add integration tests for database operations
-2. Add property-based tests for eviction algorithm
-3. Add benchmark suite for performance regression detection
-4. Mock embedding service to avoid network dependency in tests
-5. Add load tests for concurrent access patterns
-
-### Documentation
-
-**Current State**: Excellent ADR documentation, comprehensive guides
-
-**Strengths**:
-- 9 ADRs covering major decisions
-- 37 documentation files
-- API documentation with examples
-
-**Gaps**:
-- No performance tuning guide
-- No security hardening guide
-- No troubleshooting guide
-- No operational runbook
-
----
-
-## Priority Matrix
-
-### High Priority (Address in next release)
-
-1. **Connection Pooling** (Performance + Scalability)
-   - Implement in `LongTermMemory`
-   - Add configuration for pool size
-   - Estimated effort: 4 hours
-
-2. **Input Validation** (Security)
-   - Add validators for all public methods
-   - Define clear constraints
-   - Estimated effort: 8 hours
-
-3. **Embedding Dimension Configuration** (AI/Correctness)
-   - Make dimensions configurable
-   - Verify gpt-oss actual dimensions
-   - Estimated effort: 4 hours
-
-4. **Query Timeouts** (Database Reliability)
-   - Add statement timeout
-   - Make configurable
-   - Estimated effort: 2 hours
-
-5. **Error Handling Standardization** (Ruby Best Practices)
-   - Define error handling policy
-   - Remove random embedding fallback
-   - Estimated effort: 6 hours
-
-**Total High Priority Effort**: ~24 hours (3 days)
-
-### Medium Priority (Next milestone)
-
-1. **Query Result Caching** (Performance)
-2. **Circuit Breaker for Embeddings** (Systems)
-3. **Robot Authorization** (Security)
-4. **Observability Infrastructure** (Operations)
-5. **Eviction Algorithm Optimization** (Performance)
-
-### Low Priority (Future consideration)
-
-1. **Async Embedding Generation** (Scalability)
-2. **Re-ranking for RAG** (AI Quality)
-3. **Thread-Safety** (If multi-threaded use case emerges)
-4. **Factory Pattern for Services** (Ruby Design Patterns)
-
----
-
-## Conclusion
-
-HTM demonstrates a solid architectural foundation with well-documented decisions and appropriate technology choices. The two-tier memory model, RAG-based retrieval, and hive mind capabilities are particularly strong.
-
-**Key Strengths**:
-- Excellent architecture documentation (ADRs)
-- Sound database design with TimescaleDB
-- Clean separation of concerns
-- Idiomatic Ruby code
-
-**Primary Gaps**:
-- Connection pooling for database operations
-- Input validation and error handling
-- Embedding dimension configuration
-- Observability and monitoring
-
-**Recommendation**: Address high-priority items in the next release to improve production-readiness, then systematically work through medium and low priority improvements.
-
-**Overall Architecture Grade**: **A- (Strong foundation with clear improvement path)**
-
----
-
-## Appendix: Code Quality Metrics
-
-### Complexity Analysis
-
-| Class | Lines | Methods | Avg Method Length | Complexity |
-|-------|-------|---------|-------------------|------------|
-| `HTM` | 363 | 15 | 24 | Medium |
-| `WorkingMemory` | 159 | 10 | 16 | Low |
-| `LongTermMemory` | 327 | 20 | 16 | Medium |
-| `EmbeddingService` | 141 | 7 | 20 | Low |
-| `Database` | 184 | 7 | 26 | Medium |
-
-### Test Coverage
-- Current: Unknown (tests exist but coverage not measured)
-- Recommendation: Add SimpleCov, target 80%+ coverage
-
-### Dependencies
-- Core: `pg`, `pgvector`, `connection_pool`, `tiktoken_ruby`, `ruby_llm`
-- All dependencies are actively maintained ✅
-- No known security vulnerabilities ✅
-
----
-
-**Review Complete**: 2025-10-25
-**Next Review**: After high-priority improvements implemented