htm 0.0.2 → 0.0.10

data/docs/api/htm.md

@@ -141,12 +141,12 @@ htm.long_term_memory.stats  # => {...}
 
 ## Public Methods
 
-### `remember(content, tags:)` {: #remember }
+### `remember(content, tags:, metadata:)` {: #remember }
 
 Remember new information by storing it in long-term memory.
 
 ```ruby
-remember(content, tags: [])
+remember(content, tags: [], metadata: {})
 ```
 
 #### Parameters
@@ -155,6 +155,7 @@ remember(content, tags: [])
 |-----------|------|---------|-------------|
 | `content` | String | *required* | The information to remember |
 | `tags` | Array\<String\> | `[]` | Manual tags to assign (in addition to auto-extracted tags) |
+| `metadata` | Hash | `{}` | Arbitrary key-value metadata stored as JSONB. Keys must be strings or symbols. |
 
 #### Returns
 
@@ -190,6 +191,19 @@ node_id = htm.remember(
   tags: ["database:timescaledb", "performance"]
 )
 
+# With metadata
+node_id = htm.remember(
+  "User prefers dark mode for all interfaces",
+  metadata: { category: "preference", priority: "high", source_app: "settings" }
+)
+
+# With both tags and metadata
+node_id = htm.remember(
+  "API rate limit is 1000 requests per minute",
+  tags: ["api:rate-limiting", "infrastructure"],
+  metadata: { environment: "production", version: 2 }
+)
+
 # Multiple robots remembering the same content
 robot1 = HTM.new(robot_name: "assistant_1")
 robot2 = HTM.new(robot_name: "assistant_2")
@@ -205,6 +219,7 @@ robot2.remember("Ruby 3.3 was released in December 2023")
 - Embeddings and hierarchical tags are generated asynchronously via background jobs
 - Empty content returns the ID of the most recent node without creating a duplicate
 - Token count is calculated automatically using the configured token counter
+- Metadata is stored in a JSONB column with a GIN index for efficient queries
 
 ---
 
@@ -220,6 +235,7 @@ recall(
   strategy: :vector,
   with_relevance: false,
   query_tags: [],
+  metadata: {},
   raw: false
 )
 ```
@@ -234,6 +250,7 @@ recall(
 | `strategy` | Symbol | `:vector` | Search strategy (`:vector`, `:fulltext`, `:hybrid`) |
 | `with_relevance` | Boolean | `false` | Include dynamic relevance scores |
 | `query_tags` | Array\<String\> | `[]` | Tags to boost relevance |
+| `metadata` | Hash | `{}` | Filter results by metadata (uses JSONB `@>` containment) |
 | `raw` | Boolean | `false` | Return full node hashes instead of content strings |
 
 #### Timeframe Formats
@@ -284,6 +301,7 @@ When `raw: true`, each hash contains:
   "access_count" => 5,            # Times accessed
   "created_at" => "2025-01-15...", # Creation timestamp
   "token_count" => 125,           # Token count
+  "metadata" => { "category" => "preference", "priority" => "high" },  # JSONB metadata
   "similarity" => 0.87,           # Similarity score (hybrid/vector)
   "tag_boost" => 0.3,             # Tag boost score (hybrid only)
   "combined_score" => 0.79        # Combined score (hybrid only)
@@ -344,6 +362,23 @@ memories = htm.recall(
   timeframe: start_time..end_time,
   limit: 50
 )
+
+# Filter by metadata
+memories = htm.recall(
+  "user preferences",
+  metadata: { category: "preference" }
+)
+# => Returns only nodes with metadata containing { category: "preference" }
+
+# Combine metadata with other filters
+memories = htm.recall(
+  "API configuration",
+  timeframe: "last month",
+  strategy: :hybrid,
+  metadata: { environment: "production", version: 2 },
+  raw: true
+)
+# => Returns production configs with version 2, sorted by relevance
 ```
 
 #### Performance Notes
@@ -410,6 +445,166 @@ end
 
 ---
 
+### `load_file(path, force: false)` {: #load_file }
+
+Load a markdown file into long-term memory with automatic chunking and source tracking.
+
+```ruby
+load_file(path, force: false)
+```
+
+#### Parameters
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `path` | String | *required* | Path to the markdown file to load |
+| `force` | Boolean | `false` | Force re-sync even if file hasn't changed |
+
+#### Returns
+
+- `Hash` with keys:
+  - `file_source_id` - ID of the FileSource record
+  - `chunks_created` - Number of new nodes created
+  - `chunks_updated` - Number of existing nodes updated
+  - `chunks_deleted` - Number of nodes soft-deleted
+
+#### Side Effects
+
+- Creates or updates a FileSource record for tracking
+- Parses YAML frontmatter and stores as metadata
+- Chunks content by paragraph, preserving code blocks
+- Creates nodes for each chunk with `source_id` linking to file
+- Triggers async embedding and tag extraction for new nodes
+
+#### Examples
+
+```ruby
+# Load a file
+result = htm.load_file("docs/guide.md")
+# => { file_source_id: 1, chunks_created: 5, chunks_updated: 0, chunks_deleted: 0 }
+
+# Force reload even if unchanged
+result = htm.load_file("docs/guide.md", force: true)
+
+# File with frontmatter
+# ---
+# title: User Guide
+# tags: [documentation, tutorial]
+# ---
+# Content here...
+result = htm.load_file("docs/guide.md")
+# Frontmatter stored in FileSource.frontmatter
+```
+
+---
+
+### `load_directory(path, pattern: '**/*.md', force: false)` {: #load_directory }
+
+Load all matching files in a directory into long-term memory.
+
+```ruby
+load_directory(path, pattern: '**/*.md', force: false)
+```
+
+#### Parameters
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `path` | String | *required* | Directory path to scan |
+| `pattern` | String | `'**/*.md'` | Glob pattern for matching files |
+| `force` | Boolean | `false` | Force re-sync all files |
+
+#### Returns
+
+- `Array<Hash>` - Results for each file loaded, each containing:
+  - `file_path` - Path of the loaded file
+  - `file_source_id` - ID of the FileSource record
+  - `chunks_created` - Number of new nodes created
+  - `chunks_updated` - Number of existing nodes updated
+  - `chunks_deleted` - Number of nodes soft-deleted
+
+#### Examples
+
+```ruby
+# Load all markdown files
+results = htm.load_directory("docs/")
+
+# Load with custom pattern
+results = htm.load_directory("content/", pattern: "**/*.md")
+
+# Force reload all
+results = htm.load_directory("docs/", force: true)
+```
+
+---
+
+### `nodes_from_file(file_path)` {: #nodes_from_file }
+
+Get all nodes loaded from a specific file.
+
+```ruby
+nodes_from_file(file_path)
+```
+
+#### Parameters
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `file_path` | String | Path of the source file |
+
+#### Returns
+
+- `Array<HTM::Models::Node>` - Nodes from the file, ordered by chunk position
+
+#### Examples
+
+```ruby
+nodes = htm.nodes_from_file("docs/guide.md")
+nodes.each do |node|
+  puts "Chunk #{node.chunk_position}: #{node.content[0..50]}..."
+end
+```
+
+---
+
+### `unload_file(file_path)` {: #unload_file }
+
+Remove a file from memory by soft-deleting all its chunks and the file source.
+
+```ruby
+unload_file(file_path)
+```
+
+#### Parameters
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `file_path` | String | Path of the source file to unload |
+
+#### Returns
+
+- `true` if file was found and unloaded
+- `false` if file was not found
+
+#### Side Effects
+
+- Soft-deletes all nodes from the file (sets `deleted_at`)
+- Destroys the FileSource record
+
+#### Examples
+
+```ruby
+# Unload a file
+htm.unload_file("docs/guide.md")
+
+# Check if file is loaded
+if htm.nodes_from_file("docs/guide.md").empty?
+  puts "File not loaded"
+end
+```
+
+---
+
 ## Error Handling
 
 ### ArgumentError

data/docs/api/yard/HTM/ActiveRecordConfig.md

@@ -0,0 +1,23 @@
+# Class: HTM::ActiveRecordConfig
+**Inherits:** Object
+    
+
+ActiveRecord database configuration and model loading
+
+
+# Class Methods
+## connected?() {: #method-c-connected? }
+Check if connection is established and active
+**`@return`** [Boolean] 
+
+## connection_stats() {: #method-c-connection_stats }
+Get connection pool statistics
+## disconnect!() {: #method-c-disconnect! }
+Close all database connections
+## establish_connection!() {: #method-c-establish_connection! }
+Establish database connection from config/database.yml
+## load_database_config() {: #method-c-load_database_config }
+Load and parse database configuration from YAML with ERB
+## verify_extensions!() {: #method-c-verify_extensions! }
+Verify required extensions are available
+

data/docs/api/yard/HTM/AuthorizationError.md

@@ -0,0 +1,11 @@
+# Exception: HTM::AuthorizationError
+**Inherits:** HTM::Error
+    
+
+Raised when an operation is not authorized
+
+Reserved for future multi-tenant scenarios where access control may restrict
+certain operations.
+
+
+

data/docs/api/yard/HTM/CircuitBreaker.md

@@ -0,0 +1,92 @@
+# Class: HTM::CircuitBreaker
+**Inherits:** Object
+    
+
+Circuit Breaker - Prevents cascading failures from external LLM services
+
+Implements the circuit breaker pattern to protect against repeated failures
+when calling external LLM APIs for embeddings or tag extraction.
+
+States:
+*   :closed - Normal operation, requests flow through
+*   :open - Circuit tripped, requests fail fast with CircuitBreakerOpenError
+*   :half_open - Testing if service recovered, allows limited requests
+
+
+**`@example`**
+```ruby
+breaker = HTM::CircuitBreaker.new(name: 'embedding')
+result = breaker.call { external_api_call }
+```
+**`@example`**
+```ruby
+breaker = HTM::CircuitBreaker.new(
+  name: 'tag_extraction',
+  failure_threshold: 3,
+  reset_timeout: 30
+)
+```
+# Attributes
+## failure_count[RW] {: #attribute-i-failure_count }
+Returns the value of attribute failure_count.
+
+## last_failure_time[RW] {: #attribute-i-last_failure_time }
+Returns the value of attribute last_failure_time.
+
+## name[RW] {: #attribute-i-name }
+Returns the value of attribute name.
+
+## state[RW] {: #attribute-i-state }
+Returns the value of attribute state.
+
+
+# Instance Methods
+## call() {: #method-i-call }
+Execute a block with circuit breaker protection
+
+**`@raise`** [CircuitBreakerOpenError] If circuit is open
+
+**`@raise`** [StandardError] If the block raises an error (after recording failure)
+
+**`@return`** [Object] Result of the block if successful
+
+**`@yield`** [] Block containing the protected operation
+
+## closed?() {: #method-i-closed? }
+Check if circuit is currently closed (normal operation)
+
+**`@return`** [Boolean] true if circuit is closed
+
+## half_open?() {: #method-i-half_open? }
+Check if circuit is in half-open state (testing recovery)
+
+**`@return`** [Boolean] true if circuit is half-open
+
+## initialize(name:, failure_threshold:DEFAULT_FAILURE_THRESHOLD, reset_timeout:DEFAULT_RESET_TIMEOUT, half_open_max_calls:DEFAULT_HALF_OPEN_MAX_CALLS) {: #method-i-initialize }
+Initialize a new circuit breaker
+
+**`@param`** [String] Identifier for this circuit breaker (for logging)
+
+**`@param`** [Integer] Number of failures before opening circuit
+
+**`@param`** [Integer] Seconds to wait before attempting recovery
+
+**`@param`** [Integer] Successful calls needed to close circuit
+
+**`@return`** [CircuitBreaker] a new instance of CircuitBreaker
+
+## open?() {: #method-i-open? }
+Check if circuit is currently open
+
+**`@return`** [Boolean] true if circuit is open
+
+## reset!() {: #method-i-reset! }
+Manually reset the circuit breaker to closed state
+
+**`@return`** [void] 
+
+## stats() {: #method-i-stats }
+Get current circuit breaker statistics
+
+**`@return`** [Hash] Statistics including state, failure count, etc.
+

data/docs/api/yard/HTM/CircuitBreakerOpenError.md

@@ -0,0 +1,34 @@
+# Exception: HTM::CircuitBreakerOpenError
+**Inherits:** HTM::Error
+    
+
+Raised when circuit breaker is open due to repeated failures
+
+The circuit breaker pattern protects against cascading failures when external
+LLM services are unavailable. When too many consecutive failures occur, the
+circuit "opens" and subsequent calls fail fast without attempting the
+operation.
+
+Circuit states:
+*   :closed - Normal operation, requests flow through
+*   :open - Too many failures, requests fail immediately
+*   :half_open - Testing if service recovered
+
+After a reset timeout (default: 60 seconds), the circuit transitions to
+half-open and tests if the service has recovered.
+
+**`@see`** [] 
+
+**`@see`** [] 
+
+
+**`@example`**
+```ruby
+begin
+  htm.remember("new content")
+rescue HTM::CircuitBreakerOpenError
+  # LLM service unavailable, but node is still saved
+  # Embeddings/tags will be generated later when service recovers
+end
+```
+

data/docs/api/yard/HTM/Configuration.md

@@ -0,0 +1,175 @@
+# Class: HTM::Configuration
+**Inherits:** Object
+    
+
+HTM Configuration
+
+HTM uses RubyLLM for multi-provider LLM support. Supported providers:
+*   :openai (OpenAI API)
+*   :anthropic (Anthropic Claude)
+*   :gemini (Google Gemini)
+*   :azure (Azure OpenAI)
+*   :ollama (Local Ollama - default)
+*   :huggingface (HuggingFace Inference API)
+*   :openrouter (OpenRouter)
+*   :bedrock (AWS Bedrock)
+*   :deepseek (DeepSeek)
+
+
+**`@example`**
+```ruby
+HTM.configure do |config|
+  config.embedding_provider = :openai
+  config.embedding_model = 'text-embedding-3-small'
+  config.tag_provider = :openai
+  config.tag_model = 'gpt-4o-mini'
+  config.openai_api_key = ENV['OPENAI_API_KEY']
+end
+```
+**`@example`**
+```ruby
+HTM.configure do |config|
+  config.embedding_provider = :ollama
+  config.embedding_model = 'nomic-embed-text'
+  config.tag_provider = :ollama
+  config.tag_model = 'llama3'
+  config.ollama_url = 'http://localhost:11434'
+end
+```
+**`@example`**
+```ruby
+HTM.configure do |config|
+  config.embedding_provider = :openai
+  config.embedding_model = 'text-embedding-3-small'
+  config.openai_api_key = ENV['OPENAI_API_KEY']
+  config.tag_provider = :anthropic
+  config.tag_model = 'claude-3-haiku-20240307'
+  config.anthropic_api_key = ENV['ANTHROPIC_API_KEY']
+end
+```
+**`@example`**
+```ruby
+HTM.configure do |config|
+  config.embedding_generator = ->(text) {
+    MyApp::LLMService.embed(text)  # Returns Array<Float>
+  }
+  config.tag_extractor = ->(text, ontology) {
+    MyApp::LLMService.extract_tags(text, ontology)  # Returns Array<String>
+  }
+  config.logger = Rails.logger
+end
+```
+# Attributes
+## anthropic_api_key[RW] {: #attribute-i-anthropic_api_key }
+Returns the value of attribute anthropic_api_key.
+
+## azure_api_key[RW] {: #attribute-i-azure_api_key }
+Returns the value of attribute azure_api_key.
+
+## azure_api_version[RW] {: #attribute-i-azure_api_version }
+Returns the value of attribute azure_api_version.
+
+## azure_endpoint[RW] {: #attribute-i-azure_endpoint }
+Returns the value of attribute azure_endpoint.
+
+## bedrock_access_key[RW] {: #attribute-i-bedrock_access_key }
+Returns the value of attribute bedrock_access_key.
+
+## bedrock_region[RW] {: #attribute-i-bedrock_region }
+Returns the value of attribute bedrock_region.
+
+## bedrock_secret_key[RW] {: #attribute-i-bedrock_secret_key }
+Returns the value of attribute bedrock_secret_key.
+
+## connection_timeout[RW] {: #attribute-i-connection_timeout }
+Returns the value of attribute connection_timeout.
+
+## deepseek_api_key[RW] {: #attribute-i-deepseek_api_key }
+Returns the value of attribute deepseek_api_key.
+
+## embedding_dimensions[RW] {: #attribute-i-embedding_dimensions }
+Returns the value of attribute embedding_dimensions.
+
+## embedding_generator[RW] {: #attribute-i-embedding_generator }
+Returns the value of attribute embedding_generator.
+
+## embedding_model[RW] {: #attribute-i-embedding_model }
+Returns the value of attribute embedding_model.
+
+## embedding_provider[RW] {: #attribute-i-embedding_provider }
+Returns the value of attribute embedding_provider.
+
+## embedding_timeout[RW] {: #attribute-i-embedding_timeout }
+Returns the value of attribute embedding_timeout.
+
+## gemini_api_key[RW] {: #attribute-i-gemini_api_key }
+Returns the value of attribute gemini_api_key.
+
+## huggingface_api_key[RW] {: #attribute-i-huggingface_api_key }
+Returns the value of attribute huggingface_api_key.
+
+## job_backend[RW] {: #attribute-i-job_backend }
+Returns the value of attribute job_backend.
+
+## logger[RW] {: #attribute-i-logger }
+Returns the value of attribute logger.
+
+## ollama_url[RW] {: #attribute-i-ollama_url }
+Returns the value of attribute ollama_url.
+
+## openai_api_key[RW] {: #attribute-i-openai_api_key }
+Provider-specific API keys and endpoints
+
+## openai_organization[RW] {: #attribute-i-openai_organization }
+Provider-specific API keys and endpoints
+
+## openai_project[RW] {: #attribute-i-openai_project }
+Provider-specific API keys and endpoints
+
+## openrouter_api_key[RW] {: #attribute-i-openrouter_api_key }
+Returns the value of attribute openrouter_api_key.
+
+## tag_extractor[RW] {: #attribute-i-tag_extractor }
+Returns the value of attribute tag_extractor.
+
+## tag_model[RW] {: #attribute-i-tag_model }
+Returns the value of attribute tag_model.
+
+## tag_provider[RW] {: #attribute-i-tag_provider }
+Returns the value of attribute tag_provider.
+
+## tag_timeout[RW] {: #attribute-i-tag_timeout }
+Returns the value of attribute tag_timeout.
+
+## token_counter[RW] {: #attribute-i-token_counter }
+Returns the value of attribute token_counter.
+
+## week_start[RW] {: #attribute-i-week_start }
+Returns the value of attribute week_start.
+
+
+# Instance Methods
+## configure_ruby_llm(providernil) {: #method-i-configure_ruby_llm }
+Configure RubyLLM with the appropriate provider credentials
+
+**`@param`** [Symbol] The provider to configure (:openai, :anthropic, etc.)
+
+## initialize() {: #method-i-initialize }
+**`@return`** [Configuration] a new instance of Configuration
+
+## normalize_ollama_model(model_name) {: #method-i-normalize_ollama_model }
+Normalize Ollama model name to include tag if missing
+
+Ollama models require a tag (e.g., :latest, :7b, :13b). If the user specifies
+a model without a tag, we append :latest by default.
+
+**`@param`** [String] Original model name
+
+**`@return`** [String] Normalized model name with tag
+
+## reset_to_defaults() {: #method-i-reset_to_defaults }
+Reset to default RubyLLM-based implementations
+
+## validate!() {: #method-i-validate! }
+Validate configuration
+

data/docs/api/yard/HTM/Database.md

@@ -0,0 +1,99 @@
+# Class: HTM::Database
+**Inherits:** Object
+    
+
+Database setup and configuration for HTM Handles schema creation and database
+initialization
+
+
+# Class Methods
+## default_config() {: #method-c-default_config }
+Get default database configuration
+**`@return`** [Hash, nil] Connection configuration hash
+
+## drop(db_url nil) {: #method-c-drop }
+Drop all HTM tables
+**`@param`** [String] Database connection URL (uses ENV['HTM_DBURL'] if not provided)
+
+**`@return`** [void] 
+
+## dump_schema(db_url nil) {: #method-c-dump_schema }
+Dump current database schema to db/schema.sql
+
+Uses pg_dump to create a clean SQL schema file without data
+**`@param`** [String] Database connection URL (uses ENV['HTM_DBURL'] if not provided)
+
+**`@return`** [void] 
+
+## generate_docs(db_url nil) {: #method-c-generate_docs }
+Generate database documentation using tbls
+
+Uses .tbls.yml configuration file for output directory and settings. Creates
+comprehensive database documentation including:
+*   Entity-relationship diagrams
+*   Table schemas with comments
+*   Index information
+*   Relationship diagrams
+**`@param`** [String] Database connection URL (uses ENV['HTM_DBURL'] if not provided)
+
+**`@return`** [void] 
+
+## info(db_url nil) {: #method-c-info }
+Show database info
+**`@param`** [String] Database connection URL (uses ENV['HTM_DBURL'] if not provided)
+
+**`@return`** [void] 
+
+## load_schema(db_url nil) {: #method-c-load_schema }
+Load schema from db/schema.sql
+
+Uses psql to load the schema file
+**`@param`** [String] Database connection URL (uses ENV['HTM_DBURL'] if not provided)
+
+**`@return`** [void] 
+
+## migrate(db_url nil) {: #method-c-migrate }
+Run pending database migrations
+**`@param`** [String] Database connection URL (uses ENV['HTM_DBURL'] if not provided)
+
+**`@return`** [void] 
+
+## migration_status(db_url nil) {: #method-c-migration_status }
+Show migration status
+**`@param`** [String] Database connection URL (uses ENV['HTM_DBURL'] if not provided)
+
+**`@return`** [void] 
+
+## parse_connection_params() {: #method-c-parse_connection_params }
+Build config from individual environment variables
+**`@return`** [Hash, nil] Connection configuration hash
+
+## parse_connection_url(url ) {: #method-c-parse_connection_url }
+Parse database connection URL
+**`@param`** [String] Connection URL (e.g., postgresql://user:pass@host:port/dbname)
+
+**`@raise`** [ArgumentError] If URL format is invalid
+
+**`@return`** [Hash, nil] Connection configuration hash
+
+## seed(db_url nil) {: #method-c-seed }
+Seed database with sample data
+
+Loads and executes db/seeds.rb file following Rails conventions. All seeding
+logic is contained in db/seeds.rb and reads data from markdown files in
+db/seed_data/ directory.
+**`@param`** [String] Database connection URL (uses ENV['HTM_DBURL'] if not provided)
+
+**`@return`** [void] 
+
+## setup(db_url nil, run_migrations: true, dump_schema: false) {: #method-c-setup }
+Set up the HTM database schema
+**`@param`** [String] Database connection URL (uses ENV['HTM_DBURL'] if not provided)
+
+**`@param`** [Boolean] Whether to run migrations (default: true)
+
+**`@param`** [Boolean] Whether to dump schema to db/schema.sql after setup (default: false)
+
+**`@return`** [void] 
+
+