htm 0.0.1 → 0.0.10

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0a25e07ce28cc74ddb8b2fca0a9b316108b97362c2281f2e77b7bec5124da0f7
-  data.tar.gz: 9978ce3bd0c1b3c589436a5c8637bf8043bac3b0dfb542657851fa7cb96e1b49
+  metadata.gz: b552d9b1e6a197c35d1ff2c64d4c8eb94a36a0e67d05efe1c10d305cf8f23d78
+  data.tar.gz: 58fa199e1a3af9fb9c3a9e8fc0fe9ee779209a44baecbccdfdf6f570f5e7c692
 SHA512:
-  metadata.gz: bbd241d5bda941b6d3df5b35bb13e8dbf630af1be02debf814b4396fba5e76e67b6a22756690d3f510d4a0ad441710d61b3d8d20018db8ba1347220818ecefcf
-  data.tar.gz: 57e49d9629c934dd875e3d65a9c4b476bd96bc04dbc168455d3652c8a8b0aededaa2cd0163e377feadf28507550f6adbd0d13b5a9870b9c54b95437e7fd4e8bb
+  metadata.gz: 9326f318f1677ea5cad5e5f6cdda4125421a13bde13d1c7bef0a2b2e8096873a7762f11d008c8d0e64365f90272aad58699b39a3af40f0c188f7e961eef6d2df
+  data.tar.gz: 6f25fa0089349b0872db11427747e525ab46a9528d1ada22be183be260bad40d48ef692f78d922ba0803b817f2f661fa143721bf180882f34c575f4650e65448

data/.aigcm_msg

@@ -0,0 +1 @@
+Error generating commit message: status=404 Not Found body=404 page not found

data/.architecture/reviews/comprehensive-codebase-review.md

@@ -0,0 +1,577 @@
+# Architecture Review: HTM Comprehensive Codebase Review
+
+**Date**: 2025-11-28 (Updated: 2025-11-29)
+**Review Type**: Comprehensive Codebase Review
+**Version**: 0.0.7+
+**Reviewers**: Systems Architect, Domain Expert, Security Specialist, Maintainability Expert, Performance Specialist, AI Engineer, Ruby Expert, Database Architect
+
+## Executive Summary
+
+HTM (Hierarchical Temporary Memory) is a well-architected Ruby gem providing intelligent memory management for LLM-based applications. The codebase demonstrates strong architectural decisions, comprehensive documentation through ADRs, and thoughtful implementation of RAG-based retrieval with PostgreSQL/pgvector.
+
+**Overall Assessment**: **Strong** - The codebase shows mature architectural thinking with good separation of concerns, proper use of patterns, and solid security foundations. Recent improvements addressed critical thread safety and resilience concerns.
+
+**Key Findings**:
+- Excellent architectural documentation (16 ADRs documenting key decisions)
+- Solid two-tier memory architecture with proper separation of concerns
+- Good security practices with input validation, parameterized queries, and soft delete
+- Multi-provider LLM support via RubyLLM abstraction
+- Well-designed async job system with pluggable backends
+- **NEW**: Circuit breaker pattern for LLM service resilience
+- **NEW**: Thread-safe WorkingMemory implementation
+
+**Resolved Since Initial Review**:
+- Circuit breaker pattern implemented for EmbeddingService and TagService
+- WorkingMemory now fully thread-safe with Mutex protection
+- Comprehensive test suites added for EmbeddingService and TagService
+
+**Remaining Critical Actions**:
+- Add rate limiting for LLM API calls to prevent cost overruns
+- Add observability/metrics collection for production monitoring
+- Consider connection pooling for high-concurrency scenarios
+
+---
+
+## Resolved Items Summary
+
+The following high-priority items from the initial review have been addressed:
+
+| Item | Status | Implementation |
+|------|--------|----------------|
+| Circuit Breaker Pattern | **DONE** | `HTM::CircuitBreaker` class with closed/open/half-open states |
+| Thread-Safe WorkingMemory | **DONE** | All public methods protected by Mutex |
+| EmbeddingService Tests | **DONE** | 19 tests covering validation, generation, circuit breaker |
+| TagService Tests | **DONE** | 4 new circuit breaker tests (33 total) |
+| CircuitBreaker Tests | **NEW** | 13 tests for state transitions and thread safety |
+
+**Files Added/Modified**:
+- `lib/htm/circuit_breaker.rb` - New circuit breaker implementation
+- `lib/htm/working_memory.rb` - Added Mutex synchronization
+- `lib/htm/embedding_service.rb` - Integrated circuit breaker
+- `lib/htm/tag_service.rb` - Integrated circuit breaker
+- `lib/htm/errors.rb` - Updated CircuitBreakerOpenError hierarchy
+- `test/circuit_breaker_test.rb` - New test suite
+- `test/embedding_service_test.rb` - New test suite
+- `test/tag_service_test.rb` - Updated with circuit breaker tests
+
+---
+
+## System Overview
+
+HTM implements a two-tier memory system for LLM applications:
+
+1. **Working Memory**: Token-limited in-memory cache for immediate LLM context
+2. **Long-term Memory**: PostgreSQL with pgvector for persistent semantic storage
+
+Core capabilities:
+- RAG-based retrieval (vector, fulltext, hybrid search)
+- Hierarchical tagging with LLM-driven extraction
+- Multi-robot "hive mind" shared memory
+- Async background processing for embeddings and tags
+- File loading with chunking support
+
+---
+
+## Individual Member Reviews
+
+### Systems Architect
+
+**Perspective**: Distributed systems architecture, scalability patterns, and system decomposition
+
+#### Key Observations
+- Clean separation between Working Memory (in-memory) and Long-term Memory (PostgreSQL)
+- Good use of dependency injection for LLM providers
+- Pluggable job backend architecture supports diverse deployment scenarios
+- Content deduplication via SHA-256 hashing prevents storage bloat
+
+#### Strengths
+1. **Two-tier Memory Architecture**: Well-reasoned separation aligns with cognitive memory models and provides clear performance boundaries
+2. **Hive Mind Design**: Global memory sharing between robots enables knowledge reuse without complex synchronization
+3. **Async Processing Pattern**: Background job architecture prevents blocking on LLM calls
+
+#### Concerns
+1. **Single Database Dependency** (Impact: Medium)
+   - Issue: All robots share one PostgreSQL instance with no sharding strategy
+   - Recommendation: Document scaling guidance; consider read replicas for high-read workloads
+
+2. ~~**No Circuit Breaker for External LLM Services** (Impact: High)~~ **RESOLVED**
+   - ~~Issue: `CircuitBreakerOpenError` exists but no implementation found~~
+   - **Resolution**: Implemented `HTM::CircuitBreaker` class with closed/open/half-open states, integrated into EmbeddingService and TagService. Configurable failure threshold (5) and reset timeout (60s).
+
+3. **Working Memory State Not Persisted** (Impact: Low)
+   - Issue: `working_memories` table exists but WorkingMemory class is purely in-memory
+   - Recommendation: Either implement persistence or remove unused table
+
+#### Recommendations
+1. **Add horizontal scaling documentation** (Priority: Medium, Effort: Small)
+2. ~~**Implement circuit breaker for LLM calls**~~ **DONE**
+3. **Add connection pool monitoring** (Priority: Medium, Effort: Small)
+
+---
+
+### Domain Expert
+
+**Perspective**: Domain-driven design, business logic accuracy, and semantic modeling
+
+#### Key Observations
+- Domain model accurately represents memory concepts (nodes, working memory, recall)
+- Hierarchical tagging system maps well to knowledge organization principles
+- "Hive mind" metaphor effectively communicates shared memory architecture
+
+#### Strengths
+1. **Semantic Memory Types**: Six memory types (fact, context, code, preference, decision, question) provide meaningful categorization
+2. **Hierarchical Tags**: Colon-separated ontology (e.g., `database:postgresql:extensions`) enables flexible knowledge organization
+3. **Temporal Reasoning**: Natural language timeframe parsing ("last week", "few days ago") improves usability
+
+#### Concerns
+1. **Memory Types Not Enforced** (Impact: Medium)
+   - Issue: Memory types documented but not used in schema or code
+   - Recommendation: Either implement type field or remove from documentation
+
+2. **Tag Ontology Validation Limited** (Impact: Low)
+   - Issue: LLM extraction may produce inconsistent hierarchies
+   - Recommendation: Consider ontology schema enforcement or human-in-loop validation
+
+#### Recommendations
+1. **Implement memory_type column** (Priority: Low, Effort: Small)
+2. **Add ontology consistency checks** (Priority: Low, Effort: Medium)
+
+---
+
+### Security Specialist
+
+**Perspective**: Threat modeling, vulnerability assessment, and security controls
+
+#### Key Observations
+- SQL injection prevention via parameterized queries throughout
+- Input validation with size limits (1MB content, 1000 array items)
+- Soft delete prevents accidental permanent data loss
+- Sensitive embedding vectors not exposed in standard queries
+
+#### Strengths
+1. **Parameterized Queries**: All SQL uses ActiveRecord sanitization or parameterized queries
+2. **Input Validation**: Comprehensive validation in `htm.rb:116-132` for content and tags
+3. **Embedding Sanitization**: `sanitize_embedding_for_sql` validates numeric values before SQL use
+4. **Soft Delete with Confirmation**: Permanent deletion requires explicit `confirm: :confirmed`
+
+#### Concerns
+1. **No Rate Limiting on LLM Calls** (Impact: High)
+   - Issue: Malicious input could trigger unlimited LLM API calls, causing cost overruns
+   - Recommendation: Implement rate limiting per robot/time window
+
+2. **API Keys in Environment Variables** (Impact: Medium)
+   - Issue: Standard practice but no guidance on secrets management
+   - Recommendation: Document secure secrets management (Vault, AWS Secrets Manager)
+
+3. **Content Size Limit May Be Circumvented** (Impact: Low)
+   - Issue: MAX_VALUE_LENGTH checks `bytesize` but file loading has no aggregate limit
+   - Recommendation: Add per-session/per-robot storage quotas
+
+4. **No Audit Trail for Deletions** (Impact: Medium)
+   - Issue: Soft delete tracks `deleted_at` but not who/why
+   - Recommendation: Consider adding `deleted_by` and `deletion_reason` columns
+
+#### Recommendations
+1. **Add rate limiting for LLM APIs** (Priority: High, Effort: Medium)
+2. **Document secrets management best practices** (Priority: Medium, Effort: Small)
+3. **Add deletion audit trail** (Priority: Low, Effort: Small)
+
+---
+
+### Maintainability Expert
+
+**Perspective**: Code quality, technical debt, and long-term evolution
+
+#### Key Observations
+- Frozen string literals enforced across all files
+- Consistent code style with clear method documentation
+- 16 ADRs provide excellent architectural decision trail
+- Good separation of concerns across modules
+
+#### Strengths
+1. **Comprehensive ADR Documentation**: Every major decision documented with rationale
+2. **Clear Module Structure**: `lib/htm/` organized by responsibility
+3. **Ruby Idioms**: Follows Ruby conventions (method naming, block usage)
+4. **Error Hierarchy**: Well-structured error classes in `errors.rb`
+
+#### Concerns
+1. ~~**Test Coverage Gaps** (Impact: Medium)~~ **PARTIALLY RESOLVED**
+   - ~~Issue: 14 test files exist but several core classes lack dedicated tests~~
+   - **Resolution**: Added comprehensive test suites:
+     - `test/embedding_service_test.rb` - 19 tests for validation, generation, circuit breaker
+     - `test/circuit_breaker_test.rb` - 13 tests for state transitions and thread safety
+     - Updated `test/tag_service_test.rb` - Added 4 circuit breaker tests (total 33)
+   - Remaining: Node model methods, integration tests for end-to-end workflows
+
+2. **Large File Smell** (Impact: Low)
+   - Issue: `long_term_memory.rb` is 1157 lines; could benefit from extraction
+   - Recommendation: Extract search strategies into separate modules
+
+3. **Duplicate Validation Logic** (Impact: Low)
+   - Issue: Tag validation regex appears in multiple places (TagService, Tag model, HTM)
+   - Recommendation: Extract to shared constant or validator module
+
+4. **Commented Code / Dead Code** (Impact: Low)
+   - Issue: `mark_evicted` is a no-op but retained "for API compatibility"
+   - Recommendation: Either implement or remove with deprecation
+
+#### Recommendations
+1. ~~**Increase test coverage for services**~~ **DONE** (EmbeddingService, TagService, CircuitBreaker)
+2. **Extract search strategies from LongTermMemory** (Priority: Low, Effort: Medium)
+3. **Consolidate validation constants** (Priority: Low, Effort: Small)
+
+---
+
+### Performance Specialist
+
+**Perspective**: System optimization, resource utilization, and bottleneck identification
+
+#### Key Observations
+- HNSW index on embeddings enables fast approximate nearest neighbor search
+- LRU cache with TTL for query results reduces database load
+- Batch tag loading avoids N+1 queries
+- Query timeout protection (30s default)
+
+#### Strengths
+1. **Efficient Indexing Strategy**:
+   - GIN index for full-text search
+   - GIN index with pg_trgm for fuzzy matching
+   - HNSW index for vector similarity
+   - Partial index on non-deleted nodes
+2. **Query Caching**: ThreadSafe LRU cache with configurable size/TTL
+3. **Batch Operations**: `batch_load_node_tags` and `track_access` use bulk updates
+
+#### Concerns
+1. **Embedding Padding Overhead** (Impact: Medium)
+   - Issue: All embeddings padded to 2000 dimensions regardless of actual size
+   - Recommendation: Consider variable-dimension storage or dynamic padding
+
+2. **Cache Invalidation on Every Write** (Impact: Low)
+   - Issue: `invalidate_cache!` clears entire cache on any data change
+   - Recommendation: Consider more granular cache invalidation
+
+3. **No Connection Pool Metrics** (Impact: Medium)
+   - Issue: Pool exhaustion could cause silent failures
+   - Recommendation: Add pool utilization monitoring
+
+4. **Large Content Tokenization** (Impact: Low)
+   - Issue: Token counting loads entire Tiktoken encoder per call
+   - Recommendation: Cache encoder instance
+
+#### Recommendations
+1. **Add database connection pool monitoring** (Priority: High, Effort: Small)
+2. **Implement encoder caching in token counter** (Priority: Medium, Effort: Small)
+3. **Profile embedding dimension impact** (Priority: Low, Effort: Medium)
+
+---
+
+### AI Engineer
+
+**Perspective**: LLM integration, RAG systems, and AI-powered features
+
+#### Key Observations
+- Multi-provider LLM support via RubyLLM abstraction
+- Hybrid search combines vector + fulltext + tag relevance
+- Hierarchical tag extraction with ontology context
+- Async job processing prevents blocking on LLM calls
+
+#### Strengths
+1. **Provider Abstraction**: Single interface for 9+ LLM providers
+2. **Hybrid Search**: Three-signal ranking (semantic, keyword, categorical) improves recall
+3. **Ontology-Aware Extraction**: Existing tags provided to LLM for consistency
+4. **Relevance Scoring**: Multi-factor composite score (semantic, tag, recency, access)
+
+#### Concerns
+1. **No Embedding Versioning** (Impact: Medium)
+   - Issue: Changing embedding model invalidates all existing vectors
+   - Recommendation: Store embedding model identifier; support migration
+
+2. **Tag Extraction Prompt Engineering** (Impact: Low)
+   - Issue: Complex prompt in Configuration could drift from TagService constraints
+   - Recommendation: Centralize prompt with validation rules
+
+3. **No Similarity Threshold** (Impact: Low)
+   - Issue: Vector search returns K results regardless of quality
+   - Recommendation: Add minimum similarity threshold option
+
+4. **Missing Reranking Stage** (Impact: Medium)
+   - Issue: Hybrid search uses fixed 70/30 vector/tag weighting
+   - Recommendation: Consider learned reranking or configurable weights
+
+#### Recommendations
+1. **Add embedding model versioning** (Priority: Medium, Effort: Medium)
+2. **Implement configurable similarity threshold** (Priority: Low, Effort: Small)
+3. **Add configurable hybrid search weights** (Priority: Low, Effort: Small)
+
+---
+
+### Ruby Expert
+
+**Perspective**: Ruby best practices, gem development, and ecosystem integration
+
+#### Key Observations
+- Proper gem structure with version file, gemspec, and Rakefile
+- Frozen string literals enforced
+- Minitest for testing (appropriate for gem)
+- Good use of Ruby idioms (blocks, modules, inheritance)
+
+#### Strengths
+1. **Clean Gem Structure**: Follows standard layout with clear entry point
+2. **Dependency Management**: Reasonable runtime deps; dev deps properly separated
+3. **Configuration DSL**: Block-based configuration pattern is idiomatic Ruby
+4. **Pluggable Components**: Callable (lambda/proc) for custom implementations
+
+#### Concerns
+1. **Mixed Method Visibility** (Impact: Low)
+   - Issue: Some classes have `private` declared multiple times
+   - Recommendation: Single `private` declaration followed by private methods
+
+2. ~~**Thread Safety Incomplete** (Impact: Medium)~~ **RESOLVED**
+   - ~~Issue: WorkingMemory uses Hash without synchronization~~
+   - **Resolution**: All public WorkingMemory methods now protected by Mutex:
+     - `add`, `remove`, `has_space?`, `evict_to_make_space` synchronized
+     - `assemble_context`, `token_count`, `utilization_percentage`, `node_count` synchronized
+     - Internal helpers renamed to `*_unlocked` variants for safe internal use
+
+3. **No Keyword Argument Defaults Documentation** (Impact: Low)
+   - Issue: Some methods have many optional kwargs without YARD docs
+   - Recommendation: Add YARD documentation for all public methods
+
+4. **Gem Not Signed** (Impact: Low)
+   - Issue: No gem signing configured
+   - Recommendation: Consider signing for production use
+
+#### Recommendations
+1. ~~**Add thread safety to WorkingMemory**~~ **DONE**
+2. **Add comprehensive YARD documentation** (Priority: Medium, Effort: Medium)
+3. **Consider gem signing** (Priority: Low, Effort: Small)
+
+---
+
+### Database Architect
+
+**Perspective**: PostgreSQL, ActiveRecord, pgvector optimization, schema design
+
+#### Key Observations
+- Well-designed schema with proper normalization
+- Foreign key constraints ensure referential integrity
+- Comprehensive indexing strategy
+- Good use of PostgreSQL-specific features (JSONB, text search, vector)
+
+#### Strengths
+1. **Schema Design**: Clean many-to-many relationships via join tables
+2. **Index Coverage**:
+   - Unique indexes on content_hash, file_path, tag name
+   - Partial indexes for common query patterns
+   - HNSW for vector similarity
+3. **Database Comments**: Excellent documentation in schema
+4. **Soft Delete Implementation**: Partial index on non-deleted for performance
+
+#### Concerns
+1. **No Table Partitioning Strategy** (Impact: Medium)
+   - Issue: nodes table could grow very large
+   - Recommendation: Consider range partitioning by created_at for large deployments
+
+2. **HNSW Index Parameters** (Impact: Low)
+   - Issue: Using default m=16, ef_construction=64; may not be optimal
+   - Recommendation: Document tuning guidance based on dataset size
+
+3. **No Read Replica Support** (Impact: Medium)
+   - Issue: Single database connection for all operations
+   - Recommendation: Add configuration for replica for read-heavy workloads
+
+4. **Missing Database Constraints** (Impact: Low)
+   - Issue: No CHECK constraint on tag name format at database level
+   - Recommendation: Add CHECK constraint matching application validation
+
+#### Recommendations
+1. **Document HNSW tuning guidance** (Priority: Medium, Effort: Small)
+2. **Add table partitioning strategy for scale** (Priority: Medium, Effort: Large)
+3. **Add read replica configuration option** (Priority: Low, Effort: Medium)
+
+---
+
+## Collaborative Discussion
+
+### Common Concerns Identified
+
+1. **Rate Limiting Gap**: Security and AI Engineer both highlighted missing rate limiting for LLM calls
+2. ~~**Thread Safety**: Ruby Expert and Systems Architect note WorkingMemory lacks synchronization~~ **RESOLVED**
+3. **Observability**: Performance and Systems Architect recommend metrics collection
+4. ~~**Test Coverage**: Maintainability Expert and Ruby Expert recommend expanded testing~~ **PARTIALLY RESOLVED**
+
+### Priority Consensus (Updated)
+
+The team agrees on the following priority ranking:
+
+1. **Critical**: Rate limiting for LLM APIs (cost protection, security)
+2. ~~**High**: Thread safety in WorkingMemory~~ **DONE**
+3. ~~**High**: Circuit breaker for LLM services~~ **DONE**
+4. ~~**High**: Test coverage for EmbeddingService, TagService~~ **DONE**
+5. **Medium**: Connection pool monitoring
+6. **Medium**: Embedding model versioning
+7. **Medium**: Integration test expansion (remaining)
+8. **Low**: Various code quality improvements
+
+### Trade-off Discussions
+
+1. **Embedding Padding**: Performance suggests variable dimensions, but AI Engineer notes fixed dimensions simplify pgvector indexing. Consensus: Document trade-off, keep current approach for simplicity.
+
+2. **Cache Granularity**: Full cache invalidation is simple but wasteful. Consensus: Keep simple approach unless profiling shows it's a bottleneck.
+
+3. **WorkingMemory Persistence**: Table exists but unused. Consensus: Either implement for session recovery or remove table to avoid confusion.
+
+---
+
+## Consolidated Findings
+
+### Strengths
+1. **Excellent Architectural Documentation**: 16 ADRs provide decision rationale and context
+2. **Clean Separation of Concerns**: Two-tier memory, pluggable LLM providers, flexible job backends
+3. **Security Foundations**: Input validation, parameterized queries, soft delete
+4. **Multi-provider LLM Support**: Single abstraction for 9+ providers
+5. **Efficient Search**: Hybrid vector/fulltext/tag search with relevance scoring
+6. **NEW: LLM Service Resilience**: Circuit breaker pattern prevents cascading failures
+7. **NEW: Thread-Safe WorkingMemory**: Mutex protection for concurrent access
+
+### Areas for Improvement
+1. **Rate Limiting**: No → Implement per-robot/time-window limits (High Priority)
+2. ~~**Thread Safety**: Incomplete → Add synchronization to WorkingMemory~~ **DONE**
+3. ~~**Test Coverage**: Moderate → Expand service and integration tests~~ **PARTIALLY DONE**
+4. **Observability**: None → Add metrics collection (Medium Priority)
+5. **Documentation**: Good → Add YARD docs for public API (Medium Priority)
+
+### Technical Debt
+
+**High Priority**:
+- ~~`CircuitBreakerOpenError` defined but not implemented~~ **RESOLVED**
+  - ~~Impact: LLM service failures not properly handled~~
+  - **Resolution**: Implemented `HTM::CircuitBreaker` with full state machine
+
+**Medium Priority**:
+- `working_memories` table unused
+  - Impact: Schema confusion, wasted storage
+  - Resolution: Implement or remove
+  - Effort: Small
+
+- Duplicate validation logic for tags
+  - Impact: Maintenance burden, potential drift
+  - Resolution: Extract to shared module
+  - Effort: Small
+
+**Low Priority**:
+- `mark_evicted` no-op method
+  - Impact: Dead code
+  - Resolution: Remove with deprecation notice
+  - Effort: Small
+
+### Risks
+
+**Technical Risks**:
+- **LLM Cost Overrun**: No rate limiting could lead to unexpected costs
+  - Likelihood: Medium
+  - Impact: High
+  - Mitigation: Implement rate limiting, add spending alerts
+
+- ~~**Thread Safety Issues**: WorkingMemory corruption under concurrent access~~ **MITIGATED**
+  - ~~Likelihood: Low (mostly single-threaded use)~~
+  - ~~Impact: High (data corruption)~~
+  - **Resolution**: All WorkingMemory methods now protected by Mutex
+
+- **Embedding Model Migration**: Changing models invalidates all vectors
+  - Likelihood: High (models improve frequently)
+  - Impact: Medium (requires rebuild)
+  - Mitigation: Add model versioning, document migration process
+
+**Operational Risks**:
+- **Database Single Point of Failure**: No documented HA strategy
+  - Likelihood: Low (standard postgres)
+  - Impact: High (complete outage)
+  - Mitigation: Document HA setup, consider read replicas
+
+---
+
+## Recommendations
+
+### Immediate (0-2 weeks)
+
+1. **Add Rate Limiting for LLM APIs**
+   - Why: Prevent cost overruns and potential abuse
+   - How: Add per-robot rate limiter with configurable limits
+   - Owner: Core maintainer
+   - Success Criteria: Rate limit errors returned when exceeded
+
+2. ~~**Thread Safety for WorkingMemory**~~ **DONE**
+   - ~~Why: Prevent data corruption in concurrent scenarios~~
+   - ~~How: Add Mutex around @nodes and @access_order operations~~
+   - **Completed**: All public methods now synchronized
+
+3. ~~**Implement Circuit Breaker**~~ **DONE**
+   - ~~Why: Graceful degradation when LLM services fail~~
+   - ~~How: Use existing error class with threshold/timeout logic~~
+   - **Completed**: `HTM::CircuitBreaker` with closed/open/half-open states
+
+### Short-term (2-8 weeks)
+
+1. ~~**Expand Test Coverage**~~ **PARTIALLY DONE**
+   - ~~Add unit tests for EmbeddingService, TagService~~
+   - **Completed**: 19 tests for EmbeddingService, 33 tests for TagService, 13 tests for CircuitBreaker
+   - **Remaining**: Add integration tests for end-to-end workflows
+   - Target 80% coverage for core modules
+
+2. **Add Observability**
+   - Implement metrics collection (embedding generation time, cache hit rate)
+   - Add structured logging
+   - Document monitoring setup
+
+3. **Document Production Deployment**
+   - HA configuration
+   - Secrets management
+   - Performance tuning
+
+4. **Embedding Model Versioning**
+   - Add `embedding_model` column to nodes
+   - Support concurrent model versions
+   - Document migration procedure
+
+### Long-term (2-6 months)
+
+1. **Working Memory Persistence**
+   - Decide: implement or remove
+   - If implementing: session recovery, distributed caching
+   - If removing: migration to drop table
+
+2. **Scaling Documentation**
+   - Table partitioning strategy
+   - Read replica configuration
+   - Sharding considerations
+
+3. **YARD Documentation**
+   - Complete public API documentation
+   - Usage examples for all methods
+   - Generate documentation site
+
+---
+
+## Success Metrics
+
+1. **Test Coverage**: ~~Current (estimated 60%) -> Target 80% (8 weeks)~~ Improved to ~70% with new test suites
+2. **Rate Limit Compliance**: 0% -> 100% of LLM calls rate-limited (2 weeks)
+3. ~~**Thread Safety Issues**: Unknown -> 0 race conditions in test suite (2 weeks)~~ **DONE**
+4. **Documentation Completeness**: Good -> YARD coverage 100% public methods (8 weeks)
+
+---
+
+## Follow-up
+
+**Next Review**: After v1.0.0 release or 3 months
+**Tracking**: Create GitHub issues for High and Medium priority items
+
+---
+
+## Related Documentation
+
+- [ADR-001: Use PostgreSQL/TimescaleDB Storage](../decisions/adrs/001-use-postgresql-timescaledb-storage.md)
+- [ADR-002: Two-tier Memory Architecture](../decisions/adrs/002-two-tier-memory-architecture.md)
+- [ADR-013: ActiveRecord ORM and Many-to-Many Tagging](../decisions/adrs/013-activerecord-orm-and-many-to-many-tagging.md)
+- [ADR-016: Async Embedding and Tag Generation](../decisions/adrs/016-async-embedding-and-tag-generation.md)

data/.claude/settings.local.json

@@ -0,0 +1,92 @@
+{
+  "permissions": {
+    "allow": [
+      "WebFetch(domain:raw.githubusercontent.com)",
+      "Bash(mkdir:*)",
+      "Bash(mkdocs build:*)",
+      "Bash(ffmpeg:*)",
+      "Bash(gh issue create:*)",
+      "Bash(gh issue list:*)",
+      "Bash(rake test:*)",
+      "Bash(ruby -Ilib -e:*)",
+      "Bash(gh issue view:*)",
+      "Bash(gh issue close:*)",
+      "Bash(bundle install:*)",
+      "Bash(echo:*)",
+      "Bash(ruby test_connection.rb:*)",
+      "Bash(psql:*)",
+      "Bash(ruby -r ./lib/htm -e:*)",
+      "Bash(curl:*)",
+      "Bash(ollama list:*)",
+      "Bash(ruby:*)",
+      "Bash(rake:*)",
+      "Bash(export HTM_DBURL=$TIGER_DBURL)",
+      "Bash(python3:*)",
+      "Bash(chmod:*)",
+      "Bash(git clone:*)",
+      "Bash(cat:*)",
+      "Bash(source:*)",
+      "Bash(bundle exec rake:*)",
+      "Bash(./examples/example_app/app.rb)",
+      "Bash(gem list:*)",
+      "Read(//Users/dewayne/.rbenv/versions/3.4.7/lib/ruby/gems/3.4.0/gems/pgvector-0.3.2/lib/pgvector/**)",
+      "Bash(bundle check:*)",
+      "Bash(find:*)",
+      "Bash(test/caching_test.rb)",
+      "Bash(xargs:*)",
+      "Bash(git log:*)",
+      "Bash(done)",
+      "Bash(brew services:*)",
+      "Bash(lsof:*)",
+      "Bash(env)",
+      "Bash(createdb:*)",
+      "Bash(export HTM_DBURL:*)",
+      "Bash(dropdb:*)",
+      "Bash(bundle exec ruby:*)",
+      "WebFetch(domain:rubyllm.com)",
+      "Bash(HTM_DBURL=\"postgresql://dewayne@localhost:5432/htm_development\" bundle exec rake:*)",
+      "mcp__playwright-mcp__playwright_navigate",
+      "mcp__playwright-mcp__playwright_screenshot",
+      "mcp__playwright-mcp__playwright_fill",
+      "mcp__playwright-mcp__playwright_click",
+      "mcp__playwright-mcp__playwright_select",
+      "mcp__playwright-mcp__playwright_get",
+      "mcp__playwright-mcp__playwright_post",
+      "mcp__playwright-mcp__playwright_close",
+      "Bash(HTM_DBURL=\"postgresql://dewayne@localhost:5432/htm_development\" bundle exec ruby:*)",
+      "Bash(HTM_DBURL=\"postgresql://dewayne@localhost:5432/htm_development\" bundle exec rake test:*)",
+      "Bash(git mv:*)",
+      "Bash(kill:*)",
+      "WebFetch(domain:github.com)",
+      "WebFetch(domain:api.github.com)",
+      "Bash(git add:*)",
+      "Bash(git commit:*)",
+      "Bash(HTM_DBURL=\"postgresql://dewayne@localhost:5432/htm_development\" ruby:*)",
+      "Bash(HTM_DBURL=\"postgresql://dewayne@localhost:5432/htm_development\" ruby test/loaders/markdown_loader_test.rb:*)",
+      "Bash(HTM_DBURL=\"postgresql://dewayne@localhost:5432/htm_development\" bundle exec rake htm:db:stats:*)",
+      "Skill(architecture-review)",
+      "Bash(HTM_DBURL=\"postgresql://dewayne@localhost:5432/htm_development\" psql:*)",
+      "Bash(git reset:*)",
+      "Bash(/Users/dewayne/sandbox/git_repos/madbomber/htm/db/migrate/20250101000002_create_robots.rb )",
+      "Bash(/Users/dewayne/sandbox/git_repos/madbomber/htm/db/migrate/20250101000003_create_nodes.rb )",
+      "Bash(/Users/dewayne/sandbox/git_repos/madbomber/htm/db/migrate/20250101000005_create_tags.rb )",
+      "Bash(/Users/dewayne/sandbox/git_repos/madbomber/htm/db/migrate/20250101000007_add_node_vector_indexes.rb )",
+      "Bash(/Users/dewayne/sandbox/git_repos/madbomber/htm/db/migrate/20250125000001_add_content_hash_to_nodes.rb )",
+      "Bash(/Users/dewayne/sandbox/git_repos/madbomber/htm/db/migrate/20250125000002_create_robot_nodes.rb )",
+      "Bash(/Users/dewayne/sandbox/git_repos/madbomber/htm/db/migrate/20250125000003_remove_source_and_robot_id_from_nodes.rb )",
+      "Bash(/Users/dewayne/sandbox/git_repos/madbomber/htm/db/migrate/20250126000001_create_working_memories.rb )",
+      "Bash(/Users/dewayne/sandbox/git_repos/madbomber/htm/db/migrate/20250126000002_remove_unused_columns.rb )",
+      "Bash(/Users/dewayne/sandbox/git_repos/madbomber/htm/db/migrate/20251128000001_add_deleted_at_to_nodes.rb )",
+      "Bash(/Users/dewayne/sandbox/git_repos/madbomber/htm/db/migrate/20251128000002_create_file_sources.rb )",
+      "Bash(/Users/dewayne/sandbox/git_repos/madbomber/htm/db/migrate/20251128000003_add_source_to_nodes.rb )",
+      "Bash(/Users/dewayne/sandbox/git_repos/madbomber/htm/db/migrate/20251129000002_remove_unused_embedding_columns.rb)",
+      "WebSearch",
+      "WebFetch(domain:www.barndominiumlife.com)",
+      "Bash(git rm:*)",
+      "Bash(HTM_DBURL=\"postgresql://dewayne@localhost:5432/htm_development\" psql -c \"DROP TABLE IF EXISTS working_memories CASCADE;\" htm_development)",
+      "Bash(git checkout:*)"
+    ],
+    "deny": [],
+    "ask": []
+  }
+}