htm 0.0.18 → 0.0.30

data/docs/api/embedding-service.md

@@ -4,15 +4,21 @@ Client-side embedding generation service for HTM.
 
 ## Overview
 
-`HTM::EmbeddingService` generates vector embeddings for text content before database insertion. It supports multiple embedding providers:
+`HTM::EmbeddingService` generates vector embeddings for text content before database insertion. It uses RubyLLM to support multiple embedding providers:
 
-- **Ollama** - Local embedding server (default, via `nomic-embed-text` model)
-- **OpenAI** - OpenAI's `text-embedding-3-small` model
+- **Ollama** - Local embedding server (default for development)
+- **OpenAI** - OpenAI's embedding models
+- **Anthropic** - For tag extraction (via chat models)
+- **Gemini** - Google's embedding models
+- **Azure** - Azure OpenAI deployments
+- **Bedrock** - AWS Bedrock models
+- **DeepSeek** - DeepSeek embeddings
 
 The service also provides token counting for working memory management.
 
 **Architecture:**
-- Ruby application generates embeddings via HTTP call to Ollama/OpenAI
+- Ruby application generates embeddings via RubyLLM
+- RubyLLM handles provider-specific API calls
 - Embeddings are passed to PostgreSQL during INSERT
 - Simple, reliable, cross-platform operation
 
@@ -34,7 +40,6 @@ Create a new embedding service instance.
 HTM::EmbeddingService.new(
   provider = :ollama,
   model: 'nomic-embed-text',
-  ollama_url: nil,
   dimensions: nil
 )
 ```
@@ -43,11 +48,12 @@ HTM::EmbeddingService.new(
 
 | Parameter | Type | Default | Description |
 |-----------|------|---------|-------------|
-| `provider` | Symbol | `:ollama` | Embedding provider (`:ollama`, `:openai`) |
-| `model` | String | `'nomic-embed-text'` | Model name for the provider |
-| `ollama_url` | String, nil | `ENV['OLLAMA_URL']` or `'http://localhost:11434'` | Ollama server URL |
+| `provider` | Symbol | `:ollama` | Embedding provider (`:ollama`, `:openai`, `:gemini`, `:azure`, `:bedrock`, `:deepseek`) |
+| `model` | String | Provider-dependent | Model name for the provider |
 | `dimensions` | Integer, nil | Auto-detected | Expected embedding dimensions |
 
+**Provider-specific configuration** is handled via environment variables (see RubyLLM documentation).
+
 #### Returns
 
 `HTM::EmbeddingService` - Configured embedding service instance
@@ -58,20 +64,19 @@ HTM::EmbeddingService.new(
 
 #### Examples
 
-**Default Ollama configuration:**
+**Default configuration (uses Ollama):**
 
 ```ruby
 service = HTM::EmbeddingService.new
-# Uses Ollama at http://localhost:11434 with nomic-embed-text (768 dimensions)
+# Uses Ollama with nomic-embed-text (768 dimensions)
 ```
 
-**Custom Ollama model:**
+**Ollama with custom model:**
 
 ```ruby
 service = HTM::EmbeddingService.new(
   :ollama,
   model: 'mxbai-embed-large',
-  ollama_url: 'http://localhost:11434',
   dimensions: 1024
 )
 ```
@@ -87,15 +92,27 @@ service = HTM::EmbeddingService.new(
 )
 ```
 
-**HTM automatically initializes EmbeddingService:**
+**Gemini configuration:**
 
 ```ruby
-htm = HTM.new(
-  robot_name: "Assistant",
-  embedding_provider: :ollama,
-  embedding_model: 'nomic-embed-text'
+# Requires GEMINI_API_KEY environment variable
+service = HTM::EmbeddingService.new(
+  :gemini,
+  model: 'text-embedding-004',
+  dimensions: 768
 )
-# EmbeddingService configured automatically
+```
+
+**HTM global configuration (recommended):**
+
+```ruby
+HTM.configure do |config|
+  config.embedding.provider = :openai  # or :ollama, :gemini, etc.
+  config.embedding.model = 'text-embedding-3-small'
+end
+
+htm = HTM.new(robot_name: "Assistant")
+# EmbeddingService configured automatically from global config
 ```
 
 ---
@@ -144,21 +161,24 @@ begin
   embedding = service.embed("some text")
 rescue HTM::EmbeddingError => e
   puts "Embedding failed: #{e.message}"
-  # Check Ollama is running: curl http://localhost:11434/api/tags
+  # For Ollama: Check if running with `curl http://localhost:11434/api/tags`
+  # For cloud providers: Check API key is set correctly
 end
 ```
 
 #### Implementation Details
 
-**Ollama provider:**
-- Makes HTTP POST to `/api/embeddings`
-- Returns dense vector representation
-- Requires Ollama server running locally
+All providers are handled through RubyLLM, which provides a consistent interface across providers.
+
+**Ollama:** Local HTTP calls, requires Ollama server running
 
-**OpenAI provider:**
-- Makes HTTP POST to OpenAI API
-- Requires `OPENAI_API_KEY` environment variable
-- API costs: $0.0001 per 1K tokens
+**OpenAI:** Cloud API calls, requires `OPENAI_API_KEY`
+
+**Gemini:** Cloud API calls, requires `GEMINI_API_KEY`
+
+**Azure:** Cloud API calls, requires Azure credentials
+
+**Bedrock:** AWS API calls, requires AWS credentials
 
 ---
 
@@ -207,11 +227,13 @@ htm.add_message(
 
 ## Embedding Providers
 
-### Ollama (Default)
+HTM uses RubyLLM which supports multiple providers. Choose based on your requirements for privacy, cost, and quality.
+
+### Ollama (Default for Development)
 
 **Status**: ✅ Fully implemented
 
-Local embedding server with various models, accessed via HTTP.
+Local embedding server with various models.
 
 **Installation:**
 
@@ -234,25 +256,10 @@ ollama pull nomic-embed-text
 **Configuration:**
 
 ```ruby
-service = HTM::EmbeddingService.new(
-  :ollama,
-  model: 'nomic-embed-text',
-  ollama_url: 'http://localhost:11434'
-)
-
-embedding = service.embed("test text")
-```
-
-**Troubleshooting:**
-
-If Ollama is unavailable, embedding generation will fail:
-
-```ruby
-# Check Ollama is running
-system("curl http://localhost:11434/api/tags")
-
-# Start Ollama if needed
-system("ollama serve")
+HTM.configure do |config|
+  config.embedding.provider = :ollama
+  config.embedding.model = 'nomic-embed-text'
+end
 ```
 
 **Advantages:**
@@ -264,15 +271,14 @@ system("ollama serve")
 **Disadvantages:**
 - ❌ Requires local installation
 - ❌ Uses local compute resources
-- ❌ Slightly lower quality than OpenAI
 
 ---
 
-### OpenAI
+### OpenAI (Recommended for Production)
 
 **Status**: ✅ Fully implemented
 
-Uses OpenAI's embedding API, accessed via HTTP.
+Uses OpenAI's embedding API.
 
 **Configuration:**
 
@@ -281,13 +287,10 @@ export OPENAI_API_KEY="sk-..."
 ```
 
 ```ruby
-service = HTM::EmbeddingService.new(
-  :openai,
-  model: 'text-embedding-3-small'
-)
-
-# Add message - embedding generated via OpenAI API
-embedding = service.embed("test text")
+HTM.configure do |config|
+  config.embedding.provider = :openai
+  config.embedding.model = 'text-embedding-3-small'
+end
 ```
 
 **Models:**
@@ -295,20 +298,7 @@ embedding = service.embed("test text")
 | Model | Dimensions | Speed | Cost |
 |-------|------------|-------|------|
 | `text-embedding-3-small` | 1536 | Fast | $0.0001/1K tokens |
-| `text-embedding-ada-002` | 1536 | Fast | $0.0001/1K tokens |
-
-**Error Handling:**
-
-```ruby
-begin
-  service = HTM::EmbeddingService.new(:openai)
-  embedding = service.embed("test")
-rescue HTM::EmbeddingError => e
-  if e.message.include?("API key")
-    puts "Set OPENAI_API_KEY environment variable"
-  end
-end
-```
+| `text-embedding-3-large` | 3072 | Fast | $0.00013/1K tokens |
 
 **Advantages:**
 - ✅ High quality embeddings
@@ -316,10 +306,44 @@ end
 - ✅ Managed service
 
 **Disadvantages:**
-- ❌ API costs ($0.0001 per 1K tokens)
+- ❌ API costs
 - ❌ Requires internet connection
-- ❌ Data sent to OpenAI servers
-- ❌ Requires API key management
+- ❌ Data sent to cloud
+
+---
+
+### Other Providers
+
+**Gemini:**
+```bash
+export GEMINI_API_KEY="..."
+```
+```ruby
+HTM.configure do |config|
+  config.embedding.provider = :gemini
+  config.embedding.model = 'text-embedding-004'
+end
+```
+
+**Azure OpenAI:**
+```bash
+export AZURE_OPENAI_API_KEY="..."
+export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com"
+```
+
+**AWS Bedrock:**
+```bash
+export AWS_ACCESS_KEY_ID="..."
+export AWS_SECRET_ACCESS_KEY="..."
+export AWS_REGION="us-east-1"
+```
+
+**DeepSeek:**
+```bash
+export DEEPSEEK_API_KEY="..."
+```
+
+See the [RubyLLM documentation](https://rubyllm.com/) for complete provider configuration.
 
 ---
 
@@ -335,15 +359,17 @@ end
 system("ollama serve")
 ```
 
-**OpenAI API key missing:**
+**API key missing (cloud providers):**
 
 ```ruby
-# Error: OPENAI_API_KEY not set
+# Error: API key not set
 # Solution: Set environment variable
-ENV['OPENAI_API_KEY'] = 'sk-...'
+ENV['OPENAI_API_KEY'] = 'sk-...'     # For OpenAI
+ENV['ANTHROPIC_API_KEY'] = 'sk-...'  # For Anthropic
+ENV['GEMINI_API_KEY'] = '...'        # For Gemini
 ```
 
-**Invalid model:**
+**Invalid model (Ollama):**
 
 ```ruby
 # Error: Model not found
@@ -368,9 +394,10 @@ Based on typical production workloads:
 | Ollama | nomic-embed-text | 20ms | 40ms | Free |
 | Ollama | mxbai-embed-large | 30ms | 60ms | Free |
 | OpenAI | text-embedding-3-small | 40ms | 80ms | $0.10 |
+| Gemini | text-embedding-004 | 50ms | 90ms | Varies |
 
 **Factors affecting latency:**
-- Network latency (Ollama local vs OpenAI remote)
+- Network latency (local providers vs cloud)
 - Text length (longer text = more tokens = slower)
 - Model size (larger models = slower)
 - System load (CPU/GPU utilization)
@@ -380,14 +407,17 @@ Based on typical production workloads:
 **Use appropriate model size:**
 
 ```ruby
-# Fast but lower quality
-service = HTM::EmbeddingService.new(:ollama, model: 'all-minilm')
+# Fast but lower quality (Ollama)
+HTM.configure { |c| c.embedding.model = 'all-minilm' }
 
-# Balanced (recommended)
-service = HTM::EmbeddingService.new(:ollama, model: 'nomic-embed-text')
+# Balanced - Ollama (recommended for development)
+HTM.configure { |c| c.embedding.model = 'nomic-embed-text' }
 
-# Slower but higher quality
-service = HTM::EmbeddingService.new(:ollama, model: 'mxbai-embed-large')
+# High quality - OpenAI (recommended for production)
+HTM.configure do |c|
+  c.embedding.provider = :openai
+  c.embedding.model = 'text-embedding-3-small'
+end
 ```
 
 **Batch operations:**
@@ -410,12 +440,12 @@ end
 HTM initializes `EmbeddingService` automatically:
 
 ```ruby
-htm = HTM.new(
-  robot_name: "Assistant",
-  embedding_provider: :ollama,        # Optional, default
-  embedding_model: 'nomic-embed-text' # Optional, default
-)
+HTM.configure do |config|
+  config.embedding.provider = :ollama  # or :openai, :gemini, etc.
+  config.embedding.model = 'nomic-embed-text'
+end
 
+htm = HTM.new(robot_name: "Assistant")
 # EmbeddingService is ready to use internally
 ```
 
@@ -426,13 +456,13 @@ sequenceDiagram
     participant App as Application
     participant HTM as HTM
     participant ES as EmbeddingService
-    participant Ollama as Ollama/OpenAI
+    participant LLM as LLM Provider (via RubyLLM)
     participant DB as PostgreSQL
 
     App->>HTM: add_message(content)
     HTM->>ES: embed(content)
-    ES->>Ollama: HTTP POST /api/embeddings
-    Ollama->>ES: embedding vector
+    ES->>LLM: Generate embedding
+    LLM->>ES: embedding vector
     ES->>HTM: Array<Float>
     HTM->>DB: INSERT with embedding
     DB->>HTM: node_id
@@ -484,21 +514,20 @@ puts "Token count: #{tokens}"
 ### Multiple Providers
 
 ```ruby
-# Ollama for development
-dev_service = HTM::EmbeddingService.new(
-  :ollama,
-  model: 'nomic-embed-text'
-)
+# Configure for development (Ollama)
+HTM.configure do |config|
+  config.embedding.provider = :ollama
+  config.embedding.model = 'nomic-embed-text'
+end
 
-# OpenAI for production
-prod_service = HTM::EmbeddingService.new(
-  :openai,
-  model: 'text-embedding-3-small'
-)
+# Configure for production (OpenAI)
+HTM.configure do |config|
+  config.embedding.provider = :openai
+  config.embedding.model = 'text-embedding-3-small'
+end
 
-# Same interface
-dev_embedding = dev_service.embed("test")
-prod_embedding = prod_service.embed("test")
+# Same interface regardless of provider
+embedding = HTM::EmbeddingService.new.embed("test")
 ```
 
 ### Custom Model Dimensions
@@ -521,6 +550,7 @@ embedding = service.embed("text")
 
 - [HTM API](htm.md) - Main HTM class
 - [LongTermMemory API](long-term-memory.md) - Storage layer
-- [ADR-003: Ollama Embeddings](../architecture/adrs/003-ollama-embeddings.md) - Architecture decision
-- [Ollama Documentation](https://ollama.ai/docs) - Ollama setup guide
-- [OpenAI Embeddings](https://platform.openai.com/docs/guides/embeddings) - OpenAI API docs
+- [ADR-003: Default Embedding Provider](../architecture/adrs/003-ollama-embeddings.md) - Architecture decision for defaults
+- [RubyLLM Documentation](https://rubyllm.com/) - Multi-provider LLM interface
+- [Ollama Documentation](https://ollama.ai/docs) - Local LLM provider
+- [OpenAI Embeddings](https://platform.openai.com/docs/guides/embeddings) - Cloud embeddings

data/docs/api/yard/HTM/ActiveRecordConfig.md

@@ -4,6 +4,10 @@
 
 ActiveRecord database configuration and model loading
 
+Uses HTM::Config for database settings. Configuration can come from:
+*   Environment variables (HTM_DATABASE__URL, HTM_DATABASE__HOST, etc.)
+*   Programmatic configuration via HTM.configure
+
 
 # Class Methods
 ## connected?() {: #method-c-connected? }
@@ -18,6 +22,8 @@ Close all database connections
 Establish database connection from HTM::Config
 ## load_database_config() {: #method-c-load_database_config }
 Load database configuration from HTM::Config
+**`@return`** [Hash] ActiveRecord-compatible configuration hash
+
 ## verify_extensions!() {: #method-c-verify_extensions! }
 Verify required extensions are available
 

data/docs/api/yard/HTM/Config.md

@@ -0,0 +1,173 @@
+# Class: HTM::Config
+**Inherits:** Anyway::Config
+    
+
+HTM Configuration using Anyway Config
+
+Schema is defined in lib/htm/config/defaults.yml (single source of truth)
+Configuration uses nested sections for better organization:
+    - HTM.config.database.host
+    - HTM.config.embedding.provider
+    - HTM.config.providers.openai.api_key
+
+Configuration sources (lowest to highest priority):
+1.  Bundled defaults: lib/htm/config/defaults.yml (ships with gem)
+2.  XDG user config:
+    *   ~/Library/Application Support/htm/htm.yml (macOS only)
+    *   ~/.config/htm/htm.yml (XDG default)
+    *   $XDG_CONFIG_HOME/htm/htm.yml (if XDG_CONFIG_HOME is set)
+3.  Project config: ./config/htm.yml (environment-specific)
+4.  Local overrides: ./config/htm.local.yml (gitignored)
+5.  Environment variables (HTM_*)
+6.  Explicit values passed to configure block
+
+
+**`@example`**
+```ruby
+export HTM_EMBEDDING__PROVIDER=openai
+export HTM_EMBEDDING__MODEL=text-embedding-3-small
+export HTM_PROVIDERS__OPENAI__API_KEY=sk-xxx
+```
+**`@example`**
+```ruby
+embedding:
+  provider: ollama
+  model: nomic-embed-text:latest
+providers:
+  ollama:
+    url: http://localhost:11434
+```
+**`@example`**
+```ruby
+HTM.configure do |config|
+  config.embedding.provider = :openai
+  config.embedding.model = 'text-embedding-3-small'
+end
+```
+# Class Methods
+## active_xdg_config_file() {: #method-c-active_xdg_config_file }
+## config_section_with_defaults(section_key ) {: #method-c-config_section_with_defaults }
+Create a coercion that merges incoming value with SCHEMA defaults for a
+section. This ensures env vars like HTM_DATABASE__URL don't lose other
+defaults.
+## deep_merge_hashes(base , overlay ) {: #method-c-deep_merge_hashes }
+Deep merge helper for coercion
+## env() {: #method-c-env }
+## xdg_config_file() {: #method-c-xdg_config_file }
+## xdg_config_paths() {: #method-c-xdg_config_paths }
+XDG Config Path Helpers
+
+# Attributes
+## embedding_generator[RW] {: #attribute-i-embedding_generator }
+Callable Accessors (not loaded from config sources)
+
+
+## logger[RW] {: #attribute-i-logger }
+Returns the value of attribute logger.
+
+## proposition_extractor[RW] {: #attribute-i-proposition_extractor }
+Callable Accessors (not loaded from config sources)
+
+
+## tag_extractor[RW] {: #attribute-i-tag_extractor }
+Callable Accessors (not loaded from config sources)
+
+
+## token_counter[RW] {: #attribute-i-token_counter }
+Returns the value of attribute token_counter.
+
+
+# Instance Methods
+## anthropic_api_key() {: #method-i-anthropic_api_key }
+## azure_api_key() {: #method-i-azure_api_key }
+## azure_api_version() {: #method-i-azure_api_version }
+## azure_endpoint() {: #method-i-azure_endpoint }
+## bedrock_access_key() {: #method-i-bedrock_access_key }
+## bedrock_region() {: #method-i-bedrock_region }
+## bedrock_secret_key() {: #method-i-bedrock_secret_key }
+## chunk_overlap() {: #method-i-chunk_overlap }
+## chunk_size() {: #method-i-chunk_size }
+Chunking convenience accessors
+
+## circuit_breaker_failure_threshold() {: #method-i-circuit_breaker_failure_threshold }
+Circuit breaker convenience accessors
+
+## circuit_breaker_half_open_max_calls() {: #method-i-circuit_breaker_half_open_max_calls }
+## circuit_breaker_reset_timeout() {: #method-i-circuit_breaker_reset_timeout }
+## configure_ruby_llm(providernil) {: #method-i-configure_ruby_llm }
+## database_config() {: #method-i-database_config }
+## database_configured?() {: #method-i-database_configured? }
+**`@return`** [Boolean] 
+
+## database_url() {: #method-i-database_url }
+Database convenience methods
+
+## deepseek_api_key() {: #method-i-deepseek_api_key }
+## development?() {: #method-i-development? }
+**`@return`** [Boolean] 
+
+## embedding_dimensions() {: #method-i-embedding_dimensions }
+## embedding_model() {: #method-i-embedding_model }
+## embedding_provider() {: #method-i-embedding_provider }
+Embedding convenience accessors
+
+## embedding_timeout() {: #method-i-embedding_timeout }
+## environment() {: #method-i-environment }
+## extract_propositions() {: #method-i-extract_propositions }
+## gemini_api_key() {: #method-i-gemini_api_key }
+## huggingface_api_key() {: #method-i-huggingface_api_key }
+## initialize() {: #method-i-initialize }
+Instance Methods
+
+
+**`@return`** [Config] a new instance of Config
+
+## job_backend() {: #method-i-job_backend }
+Job backend convenience accessor
+
+## max_embedding_dimension() {: #method-i-max_embedding_dimension }
+## max_tag_depth() {: #method-i-max_tag_depth }
+## normalize_ollama_model(model_name) {: #method-i-normalize_ollama_model }
+Ollama Helpers
+
+
+## ollama_url() {: #method-i-ollama_url }
+## openai_api_key() {: #method-i-openai_api_key }
+Provider credential convenience accessors
+
+## openai_organization() {: #method-i-openai_organization }
+## openai_project() {: #method-i-openai_project }
+## openrouter_api_key() {: #method-i-openrouter_api_key }
+## production?() {: #method-i-production? }
+**`@return`** [Boolean] 
+
+## proposition_model() {: #method-i-proposition_model }
+## proposition_provider() {: #method-i-proposition_provider }
+Proposition convenience accessors
+
+## proposition_timeout() {: #method-i-proposition_timeout }
+## refresh_ollama_models!() {: #method-i-refresh_ollama_models! }
+## relevance_access_weight() {: #method-i-relevance_access_weight }
+## relevance_recency_half_life_hours() {: #method-i-relevance_recency_half_life_hours }
+## relevance_recency_weight() {: #method-i-relevance_recency_weight }
+## relevance_semantic_weight() {: #method-i-relevance_semantic_weight }
+Relevance scoring convenience accessors
+
+## relevance_tag_weight() {: #method-i-relevance_tag_weight }
+## reset_to_defaults() {: #method-i-reset_to_defaults }
+## service_name() {: #method-i-service_name }
+Service name convenience accessor
+
+## tag_model() {: #method-i-tag_model }
+## tag_provider() {: #method-i-tag_provider }
+Tag convenience accessors
+
+## tag_timeout() {: #method-i-tag_timeout }
+## test?() {: #method-i-test? }
+Environment Helpers
+
+
+**`@return`** [Boolean] 
+
+## validate!() {: #method-i-validate! }
+## validate_settings!() {: #method-i-validate_settings! }

data/docs/api/yard/HTM/ConfigSection.md

@@ -0,0 +1,28 @@
+# Class: HTM::ConfigSection
+**Inherits:** Object
+    
+
+ConfigSection provides method access to nested configuration hashes
+
+
+**`@example`**
+```ruby
+section = ConfigSection.new(host: 'localhost', port: 5432)
+section.host  # => 'localhost'
+section.port  # => 5432
+```
+
+# Instance Methods
+## `[](key)` {: #method-i-[] }
+## `[]=(key, value)` {: #method-i-[]= }
+## `each(&block)` {: #method-i-each }
+## `initialize(hash = {})` {: #method-i-initialize }
+**`@return`** [ConfigSection] a new instance of ConfigSection
+
+## `keys()` {: #method-i-keys }
+## `merge(other)` {: #method-i-merge }
+## `method_missing(method, *args, &block)` {: #method-i-method_missing }
+## `respond_to_missing?(method, include_private = false)` {: #method-i-respond_to_missing? }
+**`@return`** [Boolean] 
+
+## `to_h()` {: #method-i-to_h }

data/docs/api/yard/HTM/Database.md

@@ -8,7 +8,7 @@ initialization
 
 # Class Methods
 ## default_config() {: #method-c-default_config }
-Get default database configuration (respects HTM_ENV)
+Get default database configuration
 
 Uses HTM::Config for database settings.
 **`@return`** [Hash, nil] Connection configuration hash with PG-style keys

data/docs/api/yard/HTM/Railtie.md

@@ -20,8 +20,8 @@ This railtie automatically configures HTM when Rails boots:
 ```ruby
 # config/initializers/htm.rb
 HTM.configure do |config|
-  config.embedding.model = 'custom-model'
-  config.tag.model = 'custom-tag-model'
+  config.embedding_model = 'custom-model'
+  config.tag_model = 'custom-tag-model'
 end
 ```