htm 0.0.2 → 0.0.11

data/docs/api/yard/HTM/LongTermMemory.md

@@ -0,0 +1,342 @@
+# Class: HTM::LongTermMemory
+**Inherits:** Object
+    
+
+Long-term Memory - PostgreSQL/TimescaleDB-backed permanent storage
+
+LongTermMemory provides durable storage for all memory nodes with:
+*   Vector similarity search (RAG)
+*   Full-text search
+*   Time-range queries
+*   Relationship graphs
+*   Tag system
+*   ActiveRecord ORM for data access
+*   Query result caching for efficiency
+
+
+# Attributes
+## query_timeout[RW] {: #attribute-i-query_timeout }
+Returns the value of attribute query_timeout.
+
+
+# Instance Methods
+## add(content:, token_count:0, robot_id:, embedding:nil, metadata:{}) {: #method-i-add }
+Add a node to long-term memory (with deduplication)
+
+If content already exists (by content_hash), links the robot to the existing
+node and updates timestamps. Otherwise creates a new node.
+
+**`@param`** [String] Conversation message/utterance
+
+**`@param`** [Integer] Token count
+
+**`@param`** [Integer] Robot identifier
+
+**`@param`** [Array<Float>, nil] Pre-generated embedding vector
+
+**`@param`** [Hash] Flexible metadata for the node (default: {})
+
+**`@return`** [Hash] { node_id:, is_new:, robot_node: }
+
+## add_tag(node_id:, tag:) {: #method-i-add_tag }
+Add a tag to a node
+
+**`@param`** [Integer] Node database ID
+
+**`@param`** [String] Tag name
+
+**`@return`** [void] 
+
+## batch_load_node_tags(node_ids) {: #method-i-batch_load_node_tags }
+Batch load tags for multiple nodes (avoids N+1 queries)
+
+**`@param`** [Array<Integer>] Node database IDs
+
+**`@return`** [Hash<Integer, Array<String>>] Map of node_id to array of tag names
+
+## calculate_relevance(node:, query_tags:[], vector_similarity:nil, node_tags:nil) {: #method-i-calculate_relevance }
+Calculate dynamic relevance score for a node given query context
+
+Combines multiple signals:
+*   Vector similarity (semantic match)
+*   Tag overlap (categorical match)
+*   Recency (freshness)
+*   Access frequency (popularity/utility)
+
+**`@param`** [Hash] Node data with similarity, tags, created_at, access_count
+
+**`@param`** [Array<String>] Tags associated with the query
+
+**`@param`** [Float, nil] Pre-computed vector similarity (0-1)
+
+**`@param`** [Array<String>, nil] Pre-loaded tags for this node (avoids N+1 query)
+
+**`@return`** [Float] Composite relevance score (0-10)
+
+## clear_cache!() {: #method-i-clear_cache! }
+Clear the query cache
+
+Call this after any operation that modifies data (soft delete, restore, etc.)
+to ensure subsequent queries see fresh results.
+
+**`@return`** [void] 
+
+## delete(node_id) {: #method-i-delete }
+Delete a node
+
+**`@param`** [Integer] Node database ID
+
+**`@return`** [void] 
+
+## exists?(node_id) {: #method-i-exists? }
+Check if a node exists
+
+**`@param`** [Integer] Node database ID
+
+**`@return`** [Boolean] True if node exists
+
+## find_query_matching_tags(query, include_extracted:false) {: #method-i-find_query_matching_tags }
+Find tags that match terms in the query
+
+Searches the tags table for tags where any hierarchy level matches query
+words. For example, query "PostgreSQL database" would match tags like
+"database:postgresql", "database:sql", etc. Find tags matching a query using
+semantic extraction
+
+**`@param`** [String] Search query
+
+**`@param`** [Boolean] If true, returns hash with :extracted and :matched keys
+
+**`@return`** [Array<String>] Matching tag names (default)
+
+**`@return`** [Hash] If include_extracted: { extracted: [...], matched: [...] }
+
+## get_node_tags(node_id) {: #method-i-get_node_tags }
+Get tags for a specific node
+
+**`@param`** [Integer] Node database ID
+
+**`@return`** [Array<String>] Tag names
+
+## initialize(config, pool_size:nil, query_timeout:DEFAULT_QUERY_TIMEOUT, cache_size:DEFAULT_CACHE_SIZE, cache_ttl:DEFAULT_CACHE_TTL) {: #method-i-initialize }
+Initialize long-term memory storage
+
+**`@param`** [Hash] Database configuration (host, port, dbname, user, password)
+
+**`@param`** [Integer, nil] Connection pool size (uses ActiveRecord default if nil)
+
+**`@param`** [Integer] Query timeout in milliseconds (default: 30000)
+
+**`@param`** [Integer] Number of query results to cache (default: 1000, use 0 to disable)
+
+**`@param`** [Integer] Cache time-to-live in seconds (default: 300)
+
+**`@return`** [LongTermMemory] a new instance of LongTermMemory
+
+
+**`@example`**
+```ruby
+ltm = LongTermMemory.new(HTM::Database.default_config)
+```
+**`@example`**
+```ruby
+ltm = LongTermMemory.new(config, cache_size: 500, cache_ttl: 600)
+```
+**`@example`**
+```ruby
+ltm = LongTermMemory.new(config, cache_size: 0)
+```
+## link_robot_to_node(robot_id:, node:, working_memory:false) {: #method-i-link_robot_to_node }
+Link a robot to a node (create or update robot_node record)
+
+**`@param`** [Integer] Robot ID
+
+**`@param`** [HTM::Models::Node] Node to link
+
+**`@param`** [Boolean] Whether node is in working memory (default: false)
+
+**`@return`** [HTM::Models::RobotNode] The robot_node link record
+
+## mark_evicted(robot_id:, node_ids:) {: #method-i-mark_evicted }
+Mark nodes as evicted from working memory
+
+Sets working_memory = false on the robot_nodes join table for the specified
+robot and node IDs.
+
+**`@param`** [Integer] Robot ID whose working memory is being evicted
+
+**`@param`** [Array<Integer>] Node IDs to mark as evicted
+
+**`@return`** [void] 
+
+## node_topics(node_id) {: #method-i-node_topics }
+Get topics for a specific node
+
+**`@param`** [Integer] Node database ID
+
+**`@return`** [Array<String>] Topic paths
+
+## nodes_by_topic(topic_path, exact:false, limit:50) {: #method-i-nodes_by_topic }
+Retrieve nodes by ontological topic
+
+**`@param`** [String] Topic hierarchy path
+
+**`@param`** [Boolean] Exact match or prefix match
+
+**`@param`** [Integer] Maximum results
+
+**`@return`** [Array<Hash>] Matching nodes
+
+## ontology_structure() {: #method-i-ontology_structure }
+Get ontology structure view
+
+**`@return`** [Array<Hash>] Ontology structure
+
+## pool_size() {: #method-i-pool_size }
+For backwards compatibility with tests/code that expect pool_size
+
+## popular_tags(limit:20, timeframe:nil) {: #method-i-popular_tags }
+Get most popular tags
+
+**`@param`** [Integer] Number of tags to return
+
+**`@param`** [Range, nil] Optional time range filter
+
+**`@return`** [Array<Hash>] Tags with usage counts
+
+## register_robot(robot_name) {: #method-i-register_robot }
+Register a robot
+
+**`@param`** [String] Robot identifier
+
+**`@param`** [String] Robot name
+
+**`@return`** [void] 
+
+## retrieve(node_id) {: #method-i-retrieve }
+Retrieve a node by ID
+
+Automatically tracks access by incrementing access_count and updating
+last_accessed
+
+**`@param`** [Integer] Node database ID
+
+**`@return`** [Hash, nil] Node data or nil
+
+## search(timeframe:, query:, limit:, embedding_service:, metadata:{}) {: #method-i-search }
+Vector similarity search
+
+**`@param`** [nil, Range, Array<Range>] Time range(s) to search (nil = no filter)
+
+**`@param`** [String] Search query
+
+**`@param`** [Integer] Maximum results
+
+**`@param`** [Object] Service to generate embeddings
+
+**`@param`** [Hash] Filter by metadata fields (default: {})
+
+**`@return`** [Array<Hash>] Matching nodes
+
+## search_by_tags(tags:, match_all:false, timeframe:nil, limit:20) {: #method-i-search_by_tags }
+Search nodes by tags
+
+**`@param`** [Array<String>] Tags to search for
+
+**`@param`** [Boolean] If true, match ALL tags; if false, match ANY tag
+
+**`@param`** [Range, nil] Optional time range filter
+
+**`@param`** [Integer] Maximum results
+
+**`@return`** [Array<Hash>] Matching nodes with relevance scores
+
+## search_fulltext(timeframe:, query:, limit:, metadata:{}) {: #method-i-search_fulltext }
+Full-text search
+
+**`@param`** [Range] Time range to search
+
+**`@param`** [String] Search query
+
+**`@param`** [Integer] Maximum results
+
+**`@param`** [Hash] Filter by metadata fields (default: {})
+
+**`@return`** [Array<Hash>] Matching nodes
+
+## search_hybrid(timeframe:, query:, limit:, embedding_service:, prefilter_limit:100, metadata:{}) {: #method-i-search_hybrid }
+Hybrid search (full-text + vector)
+
+**`@param`** [Range] Time range to search
+
+**`@param`** [String] Search query
+
+**`@param`** [Integer] Maximum results
+
+**`@param`** [Object] Service to generate embeddings
+
+**`@param`** [Integer] Candidates to consider (default: 100)
+
+**`@param`** [Hash] Filter by metadata fields (default: {})
+
+**`@return`** [Array<Hash>] Matching nodes
+
+## search_with_relevance(timeframe:, query:nil, query_tags:[], limit:20, embedding_service:nil, metadata:{}) {: #method-i-search_with_relevance }
+Search with dynamic relevance scoring
+
+Returns nodes with calculated relevance scores based on query context
+
+**`@param`** [nil, Range, Array<Range>] Time range(s) to search (nil = no filter)
+
+**`@param`** [String, nil] Search query
+
+**`@param`** [Array<String>] Tags to match
+
+**`@param`** [Integer] Maximum results
+
+**`@param`** [Object, nil] Service to generate embeddings
+
+**`@param`** [Hash] Filter by metadata fields (default: {})
+
+**`@return`** [Array<Hash>] Nodes with relevance scores
+
+## shutdown() {: #method-i-shutdown }
+Shutdown - no-op with ActiveRecord (connection pool managed by ActiveRecord)
+
+## stats() {: #method-i-stats }
+Get memory statistics
+
+**`@return`** [Hash] Statistics
+
+## topic_relationships(min_shared_nodes:2, limit:50) {: #method-i-topic_relationships }
+Get topic relationships (co-occurrence)
+
+**`@param`** [Integer] Minimum shared nodes
+
+**`@param`** [Integer] Maximum relationships
+
+**`@return`** [Array<Hash>] Topic relationships
+
+## track_access(node_ids) {: #method-i-track_access }
+Track access for multiple nodes (bulk operation)
+
+Updates access_count and last_accessed for all nodes in the array
+
+**`@param`** [Array<Integer>] Node IDs that were accessed
+
+**`@return`** [void] 
+
+## update_last_accessed(node_id) {: #method-i-update_last_accessed }
+Update last_accessed timestamp
+
+**`@param`** [Integer] Node database ID
+
+**`@return`** [void] 
+
+## update_robot_activity(robot_id) {: #method-i-update_robot_activity }
+Update robot activity timestamp
+
+**`@param`** [String] Robot identifier
+
+**`@return`** [void] 
+

data/docs/api/yard/HTM/NotFoundError.md

@@ -0,0 +1,17 @@
+# Exception: HTM::NotFoundError
+**Inherits:** HTM::Error
+    
+
+Raised when a requested resource cannot be found
+
+Common causes:
+*   Node ID does not exist
+*   Robot not registered
+*   File source not found
+
+
+**`@example`**
+```ruby
+htm.forget(999999)  # => raises NotFoundError if node doesn't exist
+```
+

data/docs/api/yard/HTM/Observability.md

@@ -0,0 +1,107 @@
+# Module: HTM::Observability
+    
+
+Observability module for monitoring and metrics collection
+
+Provides comprehensive monitoring of HTM components including:
+*   Connection pool health monitoring with alerts
+*   Query timing and performance metrics
+*   Cache efficiency tracking
+*   Service health checks
+*   Memory usage statistics
+
+
+**`@example`**
+```ruby
+stats = HTM::Observability.collect_all
+puts stats[:connection_pool][:status]  # => :healthy
+```
+**`@example`**
+```ruby
+pool_stats = HTM::Observability.connection_pool_stats
+if pool_stats[:status] == :exhausted
+  logger.error "Connection pool exhausted!"
+end
+```
+**`@example`**
+```ruby
+if HTM::Observability.healthy?
+  puts "All systems operational"
+else
+  puts "Health check failed: #{HTM::Observability.health_check[:issues]}"
+end
+```
+# Class Methods
+## cache_stats() {: #method-c-cache_stats }
+Get query cache statistics
+**`@return`** [Hash, nil] Cache stats or nil if unavailable
+
+## circuit_breaker_stats() {: #method-c-circuit_breaker_stats }
+Get circuit breaker states for all services
+**`@return`** [Hash] Circuit breaker states:
+- :embedding_service - State and failure count
+- :tag_service - State and failure count
+
+## collect_all() {: #method-c-collect_all }
+Collect all observability metrics
+**`@return`** [Hash] Comprehensive metrics including:
+- :connection_pool - Pool stats with health status
+- :cache - Query cache hit rates and size
+- :circuit_breakers - Service circuit breaker states
+- :query_timings - Recent query performance
+- :service_timings - Embedding/tag generation times
+- :memory_usage - System memory stats
+
+## connection_pool_stats() {: #method-c-connection_pool_stats }
+Get connection pool statistics with health status
+**`@return`** [Hash] Pool statistics including:
+- :size - Maximum pool size
+- :connections - Current total connections
+- :in_use - Connections currently checked out
+- :available - Connections available for checkout
+- :utilization - Usage percentage (0.0-1.0)
+- :status - Health status (:healthy, :warning, :critical, :exhausted)
+- :wait_timeout - Connection wait timeout (ms)
+
+## health_check() {: #method-c-health_check }
+Perform comprehensive health check
+**`@return`** [Hash] Health check results:
+- :healthy - Boolean overall health status
+- :checks - Individual check results
+- :issues - Array of identified issues
+
+## healthy?() {: #method-c-healthy? }
+Quick health check - returns boolean
+**`@return`** [Boolean] true if system is healthy
+
+## memory_stats() {: #method-c-memory_stats }
+Get memory usage statistics
+**`@return`** [Hash] Memory stats
+
+## query_timing_stats() {: #method-c-query_timing_stats }
+Get query timing statistics
+**`@return`** [Hash] Timing statistics including avg, min, max, p95
+
+## record_embedding_timing(duration_ms ) {: #method-c-record_embedding_timing }
+Record embedding generation timing
+**`@param`** [Float] Generation duration in milliseconds
+
+## record_query_timing(duration_ms , query_type: :unknown) {: #method-c-record_query_timing }
+Record query timing for metrics
+**`@param`** [Float] Query duration in milliseconds
+
+**`@param`** [Symbol] Type of query (:vector, :fulltext, :hybrid)
+
+## record_tag_timing(duration_ms ) {: #method-c-record_tag_timing }
+Record tag extraction timing
+**`@param`** [Float] Extraction duration in milliseconds
+
+## reset_metrics!() {: #method-c-reset_metrics! }
+Clear all collected timing metrics
+**`@return`** [void] 
+
+## service_timing_stats() {: #method-c-service_timing_stats }
+Get service timing statistics (embedding and tag extraction)
+**`@return`** [Hash] Timing stats for embedding and tag services
+
+

data/docs/api/yard/HTM/QueryTimeoutError.md

@@ -0,0 +1,19 @@
+# Exception: HTM::QueryTimeoutError
+**Inherits:** HTM::DatabaseError
+    
+
+Raised when a database query exceeds the configured timeout
+
+Default timeout is 30 seconds. Configure via db_query_timeout parameter when
+initializing HTM.
+
+
+**`@example`**
+```ruby
+begin
+  htm.recall("complex query", strategy: :hybrid)
+rescue HTM::QueryTimeoutError
+  # Retry with simpler query or smaller limit
+end
+```
+

data/docs/api/yard/HTM/Railtie.md

@@ -0,0 +1,27 @@
+# Class: HTM::Railtie
+**Inherits:** Rails::Railtie
+    
+
+Rails Railtie for automatic HTM configuration in Rails applications
+
+This railtie automatically configures HTM when Rails boots:
+*   Sets logger to Rails.logger
+*   Sets job backend to :active_job
+*   Loads Rake tasks
+*   Configures test environment for synchronous jobs
+
+
+**`@example`**
+```ruby
+# HTM is automatically configured on Rails boot
+# No additional setup required
+```
+**`@example`**
+```ruby
+# config/initializers/htm.rb
+HTM.configure do |config|
+  config.embedding_model = 'custom-model'
+  config.tag_model = 'custom-tag-model'
+end
+```
+

data/docs/api/yard/HTM/ResourceExhaustedError.md

@@ -0,0 +1,13 @@
+# Exception: HTM::ResourceExhaustedError
+**Inherits:** HTM::Error
+    
+
+Raised when system resources are exhausted
+
+Common causes:
+*   Working memory token limit exceeded
+*   Database connection pool exhausted
+*   Memory allocation failures
+
+
+

data/docs/api/yard/HTM/TagError.md

@@ -0,0 +1,18 @@
+# Exception: HTM::TagError
+**Inherits:** HTM::Error
+    
+
+Raised when tag extraction fails
+
+Common causes:
+*   LLM provider API errors
+*   Invalid tag response format
+*   Network connectivity issues
+*   Model not available
+
+Note: This error is distinct from CircuitBreakerOpenError. TagError indicates
+a single failure, while CircuitBreakerOpenError indicates repeated failures
+have triggered protective circuit breaking.
+
+
+

data/docs/api/yard/HTM/TagService.md

@@ -0,0 +1,67 @@
+# Class: HTM::TagService
+**Inherits:** Object
+    
+
+Tag Service - Processes and validates hierarchical tags
+
+This service wraps the configured tag extractor and provides:
+*   Response parsing (string or array)
+*   Format validation (lowercase, alphanumeric, hyphens, colons)
+*   Depth validation (max 5 levels)
+*   Ontology consistency
+*   Circuit breaker protection for external LLM failures
+
+The actual LLM call is delegated to HTM.configuration.tag_extractor
+
+
+# Class Methods
+## circuit_breaker() {: #method-c-circuit_breaker }
+Get or create the circuit breaker for tag service
+**`@return`** [HTM::CircuitBreaker] The circuit breaker instance
+
+## extract(content , existing_ontology: []) {: #method-c-extract }
+Extract tags with validation and processing
+**`@param`** [String] Text to analyze
+
+**`@param`** [Array<String>] Sample of existing tags for context
+
+**`@raise`** [CircuitBreakerOpenError] If circuit breaker is open
+
+**`@return`** [Array<String>] Validated tag names
+
+## parse_hierarchy(tag ) {: #method-c-parse_hierarchy }
+Parse hierarchical structure of a tag
+**`@param`** [String] Hierarchical tag (e.g., "ai:llm:embedding")
+
+**`@return`** [Hash] Hierarchy structure
+{
+  full: "ai:llm:embedding",
+  root: "ai",
+  parent: "ai:llm",
+  levels: ["ai", "llm", "embedding"],
+  depth: 3
+}
+
+## parse_tags(raw_tags ) {: #method-c-parse_tags }
+Parse tag response (handles string or array input)
+**`@param`** [String, Array] Raw response from extractor
+
+**`@return`** [Array<String>] Parsed tag strings
+
+## reset_circuit_breaker!() {: #method-c-reset_circuit_breaker! }
+Reset the circuit breaker (useful for testing)
+**`@return`** [void] 
+
+## valid_tag?(tag ) {: #method-c-valid_tag? }
+Validate single tag format
+**`@param`** [String] Tag to validate
+
+**`@return`** [Boolean] True if valid
+
+## validate_and_filter_tags(tags ) {: #method-c-validate_and_filter_tags }
+Validate and filter tags
+**`@param`** [Array<String>] Parsed tags
+
+**`@return`** [Array<String>] Valid tags only
+
+

data/docs/api/yard/HTM/Timeframe/Result.md

@@ -0,0 +1,24 @@
+# Class: HTM::Timeframe::Result
+**Inherits:** Struct
+    
+
+Result structure for :auto mode
+
+
+# Attributes
+## extracted[RW] {: #attribute-i-extracted }
+Returns the value of attribute extracted
+
+**`@return`** [Object] the current value of extracted
+
+## query[RW] {: #attribute-i-query }
+Returns the value of attribute query
+
+**`@return`** [Object] the current value of query
+
+## timeframe[RW] {: #attribute-i-timeframe }
+Returns the value of attribute timeframe
+
+**`@return`** [Object] the current value of timeframe
+
+

data/docs/api/yard/HTM/Timeframe.md

@@ -0,0 +1,40 @@
+# Class: HTM::Timeframe
+**Inherits:** Object
+    
+
+Timeframe - Normalizes various timeframe inputs for database queries
+
+Handles multiple input types and normalizes them to either:
+*   nil (no timeframe filter)
+*   Range (single time window)
+*   Array<Range> (multiple time windows, OR'd together)
+
+
+**`@example`**
+```ruby
+Timeframe.normalize(nil)                    # => nil (no filter)
+Timeframe.normalize(Date.today)             # => Range for entire day
+Timeframe.normalize(Time.now)               # => Range for entire day
+Timeframe.normalize("last week")            # => Range from chronic/extractor
+Timeframe.normalize(:auto, query: "...")    # => Extract from query text
+Timeframe.normalize(range1..range2)         # => Pass through
+Timeframe.normalize([range1, range2])       # => Array of ranges
+```
+# Class Methods
+## normalize(input , query: nil) {: #method-c-normalize }
+Normalize a timeframe input to nil, Range, or Array<Range>
+**`@param`** [nil, Range, Array, Date, DateTime, Time, String, Symbol] Timeframe specification
+
+**`@param`** [String, nil] Query text (required when input is :auto)
+
+**`@return`** [nil, Range, Array<Range>] Normalized timeframe
+
+**`@return`** [Result] When input is :auto, returns Result with :timeframe, :query, :extracted
+
+## valid?(input ) {: #method-c-valid? }
+Check if a value is a valid timeframe input
+**`@param`** [Object] Value to check
+
+**`@return`** [Boolean] 
+
+

data/docs/api/yard/HTM/TimeframeExtractor/Result.md

@@ -0,0 +1,24 @@
+# Class: HTM::TimeframeExtractor::Result
+**Inherits:** Struct
+    
+
+Result structure for extracted timeframe
+
+
+# Attributes
+## original_expression[RW] {: #attribute-i-original_expression }
+Returns the value of attribute original_expression
+
+**`@return`** [Object] the current value of original_expression
+
+## query[RW] {: #attribute-i-query }
+Returns the value of attribute query
+
+**`@return`** [Object] the current value of query
+
+## timeframe[RW] {: #attribute-i-timeframe }
+Returns the value of attribute timeframe
+
+**`@return`** [Object] the current value of timeframe
+
+