htm 0.0.18 → 0.0.20

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7379c23800bf3f00cc328be9725b14bcea06a9d2158eaf3b3bca80b11a6efb74
-  data.tar.gz: ec96095a72ab89fed1dd7df0ceaeb3888938f5ddbdea6fd3f3c06aa15b80bf67
+  metadata.gz: 73eb1c7c8727363608ac47f7c1afd26ebb47021a173c105e5bba364598bf143f
+  data.tar.gz: e9c6954e8bb6da65e26ea55878da3e817e59f28e3383d74f9b48a0501ebd882b
 SHA512:
-  metadata.gz: 428b9c8f1db0f3a7a4be6e0b8bc4cc27f0bf058e7847c409cfca423c1bf9722a8efad92dbdd276d69844ab771670ea7cdc62e292018bec4705ed3f9f66255c16
-  data.tar.gz: af62a40339de40eac22f90b919a626b0f2d61a14440cdd5ca9e0aaf87ca088cc086cd35bc1eb4394d87a0646512c0a30d2fb47fdee7ec9ea2c4f11e469f2a44b
+  metadata.gz: cb76637105f6033ba58d35f5fe3d73b41b990bd327069c7c526c196c787c62d2287e0fdac58261a2ee7b6d739af9f11bd58843c77b7ad6b486da95e02468665e
+  data.tar.gz: 9e4da2a308bfb5efe31a050406caee29e7a653e401a9665a271b3c094d540d50115663966f11f26b7b58a51f7a9354047ee6117ce69250e7dc5ffabb30469a48

data/CHANGELOG.md

@@ -6,6 +6,57 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
+
+## [0.0.20] - 2025-12-22
+### Added
+- **Fiber-based job backend** - New `:fiber` backend for I/O-bound background jobs
+  - Uses `async` gem for cooperative concurrency
+  - Non-blocking execution ideal for LLM API calls
+  - `JobAdapter.enqueue_parallel(jobs)` for concurrent job execution
+  - Fiber backend runs jobs with `Async::Barrier` for parallel coordination
+- **RememberWorkflow using simple_flow** - Parallel processing pipeline for node enrichment
+  - Orchestrates save_node → (embedding, tags, propositions in parallel) → finalize
+  - Uses `SimpleFlow::Pipeline` with dependency-based step execution
+  - Configurable concurrency model (`:auto`, `:threads`, `:async`)
+  - Visualization support: `to_mermaid` and `execution_plan` methods
+
+### Changed
+- **async gem is now a required dependency** - Previously optional, now always available
+  - Enables fiber-based concurrency for all HTM installations
+  - `:fiber` is now the default job backend (was `:thread`)
+- **Default job backend changed from `:thread` to `:fiber`** - Better performance for I/O-bound LLM operations
+
+### Removed
+- **`JobAdapter.async_available?` method** - No longer needed since async is always available
+
+### Fixed
+- **`connection_timeout` default inconsistency** - Unified default to 60 seconds across all files
+  - `test/configuration_test.rb`, `docs/guides/configuration.md`, and `db/seeds.rb` now match `defaults.yml`
+
+### Dependencies
+- Added `async` (~> 2.0) as required runtime dependency
+- Added `simple_flow` for workflow orchestration
+
+## [0.0.19] - 2025-12-21
+
+### Changed
+- **Reorganized "Robots!" documentation** - Consolidated 5 robot-related docs into `docs/robots/` subdirectory
+  - Created `docs/robots/index.md` as section overview with navigation table and architecture diagram
+  - Moved `docs/robots.md` → `docs/robots/why-robots.md`
+  - Moved `docs/guides/multi-robot.md` → `docs/robots/multi-robot.md`
+  - Moved `docs/guides/robot-groups.md` → `docs/robots/robot-groups.md`
+  - Moved `docs/architecture/two-tier-memory.md` → `docs/robots/two-tier-memory.md`
+  - Moved `docs/architecture/hive-mind.md` → `docs/robots/hive-mind.md`
+  - Updated `mkdocs.yml` navigation with logical information flow
+  - Fixed 30+ broken relative links across documentation files
+- **Extracted inline SVG graphics to standalone files** - Improved documentation maintainability
+  - From `hive-mind.md`: `hive-mind-shared-memory.svg`, `memory-topology.svg`
+  - From `two-tier-memory.md`: `two-tier-memory-architecture.svg`, `eviction-priority.svg`, `balanced-strategy-decay.svg`
+  - All SVGs now in `docs/assets/images/` and referenced via markdown image syntax
+
+### Fixed
+- **ConfigSection.md unrecognized link warning** - Wrapped method signatures in backticks to prevent MkDocs interpreting bracket parameters as links
+
 ## [0.0.18] - 2025-12-20
 ### Added
 - **Anyway::Config-based configuration system** - Replaced custom Configuration class with robust multi-source config management
@@ -689,7 +740,14 @@ HTM.config.embedding.model
 - Working memory size is user-configurable
 - See ADRs for detailed architectural decisions and rationale
 
-[Unreleased]: https://github.com/madbomber/htm/compare/v0.0.12...HEAD
+[Unreleased]: https://github.com/madbomber/htm/compare/v0.0.19...HEAD
+[0.0.19]: https://github.com/madbomber/htm/compare/v0.0.18...v0.0.19
+[0.0.18]: https://github.com/madbomber/htm/compare/v0.0.17...v0.0.18
+[0.0.17]: https://github.com/madbomber/htm/compare/v0.0.15...v0.0.17
+[0.0.15]: https://github.com/madbomber/htm/compare/v0.0.14...v0.0.15
+[0.0.14]: https://github.com/madbomber/htm/compare/v0.0.13...v0.0.14
+[0.0.13]: https://github.com/madbomber/htm/compare/v0.0.11...v0.0.13
+[0.0.11]: https://github.com/madbomber/htm/compare/v0.0.10...v0.0.11
 [0.0.12]: https://github.com/madbomber/htm/compare/v0.0.10...v0.0.12
 [0.0.10]: https://github.com/madbomber/htm/compare/v0.0.9...v0.0.10
 [0.0.9]: https://github.com/madbomber/htm/compare/v0.0.8...v0.0.9

data/README.md

@@ -1,4 +1,16 @@
+
 <div align="center">
+  <div style="background: linear-gradient(135deg, #90EE90 0%, #32CD32 100%); border: 4px solid #228B22; border-radius: 12px; padding: 20px; margin: 20px auto; max-width: 800px; box-shadow: 0 8px 16px rgba(34, 139, 34, 0.3);">
+    <p style="color: #000000; font-size: 42px; font-weight: bold; margin: 0;">
+        🚀 v0.0.20 🚀
+    </p>
+    <p style="color: #000; font-size: 27px; font-weight: bold; margin: 10px 0 0 0; line-height: 1.6;">
+        <strong>Fiber-based concurrency</strong> is now the default.<br/>
+        Parallel embedding, tagging, and proposition extraction via simple_flow.
+    </p>
+  </div>
+
+
   <h1>HTM</h1>
   <img src="docs/assets/images/htm_demo.gif" alt="Tree of Knowledge is Growing" width="400">
 

data/db/seeds.rb

@@ -37,7 +37,7 @@ tag_provider = (ENV['HTM_TAG_PROVIDER'] || 'ollama').to_sym
 tag_model = ENV['HTM_TAG_MODEL'] || 'gemma3'
 embedding_timeout = (ENV['HTM_EMBEDDING_TIMEOUT'] || '120').to_i
 tag_timeout = (ENV['HTM_TAG_TIMEOUT'] || '180').to_i
-connection_timeout = (ENV['HTM_CONNECTION_TIMEOUT'] || '30').to_i
+connection_timeout = (ENV['HTM_CONNECTION_TIMEOUT'] || '60').to_i
 
 puts "Configuration:"
 puts "  Embedding Provider: #{embedding_provider}"

data/docs/api/embedding-service.md

@@ -4,15 +4,21 @@ Client-side embedding generation service for HTM.
 
 ## Overview
 
-`HTM::EmbeddingService` generates vector embeddings for text content before database insertion. It supports multiple embedding providers:
+`HTM::EmbeddingService` generates vector embeddings for text content before database insertion. It uses RubyLLM to support multiple embedding providers:
 
-- **Ollama** - Local embedding server (default, via `nomic-embed-text` model)
-- **OpenAI** - OpenAI's `text-embedding-3-small` model
+- **Ollama** - Local embedding server (default for development)
+- **OpenAI** - OpenAI's embedding models
+- **Anthropic** - For tag extraction (via chat models)
+- **Gemini** - Google's embedding models
+- **Azure** - Azure OpenAI deployments
+- **Bedrock** - AWS Bedrock models
+- **DeepSeek** - DeepSeek embeddings
 
 The service also provides token counting for working memory management.
 
 **Architecture:**
-- Ruby application generates embeddings via HTTP call to Ollama/OpenAI
+- Ruby application generates embeddings via RubyLLM
+- RubyLLM handles provider-specific API calls
 - Embeddings are passed to PostgreSQL during INSERT
 - Simple, reliable, cross-platform operation
 
@@ -34,7 +40,6 @@ Create a new embedding service instance.
 HTM::EmbeddingService.new(
   provider = :ollama,
   model: 'nomic-embed-text',
-  ollama_url: nil,
   dimensions: nil
 )
 ```
@@ -43,11 +48,12 @@ HTM::EmbeddingService.new(
 
 | Parameter | Type | Default | Description |
 |-----------|------|---------|-------------|
-| `provider` | Symbol | `:ollama` | Embedding provider (`:ollama`, `:openai`) |
-| `model` | String | `'nomic-embed-text'` | Model name for the provider |
-| `ollama_url` | String, nil | `ENV['OLLAMA_URL']` or `'http://localhost:11434'` | Ollama server URL |
+| `provider` | Symbol | `:ollama` | Embedding provider (`:ollama`, `:openai`, `:gemini`, `:azure`, `:bedrock`, `:deepseek`) |
+| `model` | String | Provider-dependent | Model name for the provider |
 | `dimensions` | Integer, nil | Auto-detected | Expected embedding dimensions |
 
+**Provider-specific configuration** is handled via environment variables (see RubyLLM documentation).
+
 #### Returns
 
 `HTM::EmbeddingService` - Configured embedding service instance
@@ -58,20 +64,19 @@ HTM::EmbeddingService.new(
 
 #### Examples
 
-**Default Ollama configuration:**
+**Default configuration (uses Ollama):**
 
 ```ruby
 service = HTM::EmbeddingService.new
-# Uses Ollama at http://localhost:11434 with nomic-embed-text (768 dimensions)
+# Uses Ollama with nomic-embed-text (768 dimensions)
 ```
 
-**Custom Ollama model:**
+**Ollama with custom model:**
 
 ```ruby
 service = HTM::EmbeddingService.new(
   :ollama,
   model: 'mxbai-embed-large',
-  ollama_url: 'http://localhost:11434',
   dimensions: 1024
 )
 ```
@@ -87,15 +92,27 @@ service = HTM::EmbeddingService.new(
 )
 ```
 
-**HTM automatically initializes EmbeddingService:**
+**Gemini configuration:**
 
 ```ruby
-htm = HTM.new(
-  robot_name: "Assistant",
-  embedding_provider: :ollama,
-  embedding_model: 'nomic-embed-text'
+# Requires GEMINI_API_KEY environment variable
+service = HTM::EmbeddingService.new(
+  :gemini,
+  model: 'text-embedding-004',
+  dimensions: 768
 )
-# EmbeddingService configured automatically
+```
+
+**HTM global configuration (recommended):**
+
+```ruby
+HTM.configure do |config|
+  config.embedding.provider = :openai  # or :ollama, :gemini, etc.
+  config.embedding.model = 'text-embedding-3-small'
+end
+
+htm = HTM.new(robot_name: "Assistant")
+# EmbeddingService configured automatically from global config
 ```
 
 ---
@@ -144,21 +161,24 @@ begin
   embedding = service.embed("some text")
 rescue HTM::EmbeddingError => e
   puts "Embedding failed: #{e.message}"
-  # Check Ollama is running: curl http://localhost:11434/api/tags
+  # For Ollama: Check if running with `curl http://localhost:11434/api/tags`
+  # For cloud providers: Check API key is set correctly
 end
 ```
 
 #### Implementation Details
 
-**Ollama provider:**
-- Makes HTTP POST to `/api/embeddings`
-- Returns dense vector representation
-- Requires Ollama server running locally
+All providers are handled through RubyLLM, which provides a consistent interface across providers.
+
+**Ollama:** Local HTTP calls, requires Ollama server running
 
-**OpenAI provider:**
-- Makes HTTP POST to OpenAI API
-- Requires `OPENAI_API_KEY` environment variable
-- API costs: $0.0001 per 1K tokens
+**OpenAI:** Cloud API calls, requires `OPENAI_API_KEY`
+
+**Gemini:** Cloud API calls, requires `GEMINI_API_KEY`
+
+**Azure:** Cloud API calls, requires Azure credentials
+
+**Bedrock:** AWS API calls, requires AWS credentials
 
 ---
 
@@ -207,11 +227,13 @@ htm.add_message(
 
 ## Embedding Providers
 
-### Ollama (Default)
+HTM uses RubyLLM which supports multiple providers. Choose based on your requirements for privacy, cost, and quality.
+
+### Ollama (Default for Development)
 
 **Status**: ✅ Fully implemented
 
-Local embedding server with various models, accessed via HTTP.
+Local embedding server with various models.
 
 **Installation:**
 
@@ -234,25 +256,10 @@ ollama pull nomic-embed-text
 **Configuration:**
 
 ```ruby
-service = HTM::EmbeddingService.new(
-  :ollama,
-  model: 'nomic-embed-text',
-  ollama_url: 'http://localhost:11434'
-)
-
-embedding = service.embed("test text")
-```
-
-**Troubleshooting:**
-
-If Ollama is unavailable, embedding generation will fail:
-
-```ruby
-# Check Ollama is running
-system("curl http://localhost:11434/api/tags")
-
-# Start Ollama if needed
-system("ollama serve")
+HTM.configure do |config|
+  config.embedding.provider = :ollama
+  config.embedding.model = 'nomic-embed-text'
+end
 ```
 
 **Advantages:**
@@ -264,15 +271,14 @@ system("ollama serve")
 **Disadvantages:**
 - ❌ Requires local installation
 - ❌ Uses local compute resources
-- ❌ Slightly lower quality than OpenAI
 
 ---
 
-### OpenAI
+### OpenAI (Recommended for Production)
 
 **Status**: ✅ Fully implemented
 
-Uses OpenAI's embedding API, accessed via HTTP.
+Uses OpenAI's embedding API.
 
 **Configuration:**
 
@@ -281,13 +287,10 @@ export OPENAI_API_KEY="sk-..."
 ```
 
 ```ruby
-service = HTM::EmbeddingService.new(
-  :openai,
-  model: 'text-embedding-3-small'
-)
-
-# Add message - embedding generated via OpenAI API
-embedding = service.embed("test text")
+HTM.configure do |config|
+  config.embedding.provider = :openai
+  config.embedding.model = 'text-embedding-3-small'
+end
 ```
 
 **Models:**
@@ -295,20 +298,7 @@ embedding = service.embed("test text")
 | Model | Dimensions | Speed | Cost |
 |-------|------------|-------|------|
 | `text-embedding-3-small` | 1536 | Fast | $0.0001/1K tokens |
-| `text-embedding-ada-002` | 1536 | Fast | $0.0001/1K tokens |
-
-**Error Handling:**
-
-```ruby
-begin
-  service = HTM::EmbeddingService.new(:openai)
-  embedding = service.embed("test")
-rescue HTM::EmbeddingError => e
-  if e.message.include?("API key")
-    puts "Set OPENAI_API_KEY environment variable"
-  end
-end
-```
+| `text-embedding-3-large` | 3072 | Fast | $0.00013/1K tokens |
 
 **Advantages:**
 - ✅ High quality embeddings
@@ -316,10 +306,44 @@ end
 - ✅ Managed service
 
 **Disadvantages:**
-- ❌ API costs ($0.0001 per 1K tokens)
+- ❌ API costs
 - ❌ Requires internet connection
-- ❌ Data sent to OpenAI servers
-- ❌ Requires API key management
+- ❌ Data sent to cloud
+
+---
+
+### Other Providers
+
+**Gemini:**
+```bash
+export GEMINI_API_KEY="..."
+```
+```ruby
+HTM.configure do |config|
+  config.embedding.provider = :gemini
+  config.embedding.model = 'text-embedding-004'
+end
+```
+
+**Azure OpenAI:**
+```bash
+export AZURE_OPENAI_API_KEY="..."
+export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com"
+```
+
+**AWS Bedrock:**
+```bash
+export AWS_ACCESS_KEY_ID="..."
+export AWS_SECRET_ACCESS_KEY="..."
+export AWS_REGION="us-east-1"
+```
+
+**DeepSeek:**
+```bash
+export DEEPSEEK_API_KEY="..."
+```
+
+See the [RubyLLM documentation](https://rubyllm.com/) for complete provider configuration.
 
 ---
 
@@ -335,15 +359,17 @@ end
 system("ollama serve")
 ```
 
-**OpenAI API key missing:**
+**API key missing (cloud providers):**
 
 ```ruby
-# Error: OPENAI_API_KEY not set
+# Error: API key not set
 # Solution: Set environment variable
-ENV['OPENAI_API_KEY'] = 'sk-...'
+ENV['OPENAI_API_KEY'] = 'sk-...'     # For OpenAI
+ENV['ANTHROPIC_API_KEY'] = 'sk-...'  # For Anthropic
+ENV['GEMINI_API_KEY'] = '...'        # For Gemini
 ```
 
-**Invalid model:**
+**Invalid model (Ollama):**
 
 ```ruby
 # Error: Model not found
@@ -368,9 +394,10 @@ Based on typical production workloads:
 | Ollama | nomic-embed-text | 20ms | 40ms | Free |
 | Ollama | mxbai-embed-large | 30ms | 60ms | Free |
 | OpenAI | text-embedding-3-small | 40ms | 80ms | $0.10 |
+| Gemini | text-embedding-004 | 50ms | 90ms | Varies |
 
 **Factors affecting latency:**
-- Network latency (Ollama local vs OpenAI remote)
+- Network latency (local providers vs cloud)
 - Text length (longer text = more tokens = slower)
 - Model size (larger models = slower)
 - System load (CPU/GPU utilization)
@@ -380,14 +407,17 @@ Based on typical production workloads:
 **Use appropriate model size:**
 
 ```ruby
-# Fast but lower quality
-service = HTM::EmbeddingService.new(:ollama, model: 'all-minilm')
+# Fast but lower quality (Ollama)
+HTM.configure { |c| c.embedding.model = 'all-minilm' }
 
-# Balanced (recommended)
-service = HTM::EmbeddingService.new(:ollama, model: 'nomic-embed-text')
+# Balanced - Ollama (recommended for development)
+HTM.configure { |c| c.embedding.model = 'nomic-embed-text' }
 
-# Slower but higher quality
-service = HTM::EmbeddingService.new(:ollama, model: 'mxbai-embed-large')
+# High quality - OpenAI (recommended for production)
+HTM.configure do |c|
+  c.embedding.provider = :openai
+  c.embedding.model = 'text-embedding-3-small'
+end
 ```
 
 **Batch operations:**
@@ -410,12 +440,12 @@ end
 HTM initializes `EmbeddingService` automatically:
 
 ```ruby
-htm = HTM.new(
-  robot_name: "Assistant",
-  embedding_provider: :ollama,        # Optional, default
-  embedding_model: 'nomic-embed-text' # Optional, default
-)
+HTM.configure do |config|
+  config.embedding.provider = :ollama  # or :openai, :gemini, etc.
+  config.embedding.model = 'nomic-embed-text'
+end
 
+htm = HTM.new(robot_name: "Assistant")
 # EmbeddingService is ready to use internally
 ```
 
@@ -426,13 +456,13 @@ sequenceDiagram
     participant App as Application
     participant HTM as HTM
     participant ES as EmbeddingService
-    participant Ollama as Ollama/OpenAI
+    participant LLM as LLM Provider (via RubyLLM)
     participant DB as PostgreSQL
 
     App->>HTM: add_message(content)
     HTM->>ES: embed(content)
-    ES->>Ollama: HTTP POST /api/embeddings
-    Ollama->>ES: embedding vector
+    ES->>LLM: Generate embedding
+    LLM->>ES: embedding vector
     ES->>HTM: Array<Float>
     HTM->>DB: INSERT with embedding
     DB->>HTM: node_id
@@ -484,21 +514,20 @@ puts "Token count: #{tokens}"
 ### Multiple Providers
 
 ```ruby
-# Ollama for development
-dev_service = HTM::EmbeddingService.new(
-  :ollama,
-  model: 'nomic-embed-text'
-)
+# Configure for development (Ollama)
+HTM.configure do |config|
+  config.embedding.provider = :ollama
+  config.embedding.model = 'nomic-embed-text'
+end
 
-# OpenAI for production
-prod_service = HTM::EmbeddingService.new(
-  :openai,
-  model: 'text-embedding-3-small'
-)
+# Configure for production (OpenAI)
+HTM.configure do |config|
+  config.embedding.provider = :openai
+  config.embedding.model = 'text-embedding-3-small'
+end
 
-# Same interface
-dev_embedding = dev_service.embed("test")
-prod_embedding = prod_service.embed("test")
+# Same interface regardless of provider
+embedding = HTM::EmbeddingService.new.embed("test")
 ```
 
 ### Custom Model Dimensions
@@ -521,6 +550,7 @@ embedding = service.embed("text")
 
 - [HTM API](htm.md) - Main HTM class
 - [LongTermMemory API](long-term-memory.md) - Storage layer
-- [ADR-003: Ollama Embeddings](../architecture/adrs/003-ollama-embeddings.md) - Architecture decision
-- [Ollama Documentation](https://ollama.ai/docs) - Ollama setup guide
-- [OpenAI Embeddings](https://platform.openai.com/docs/guides/embeddings) - OpenAI API docs
+- [ADR-003: Default Embedding Provider](../architecture/adrs/003-ollama-embeddings.md) - Architecture decision for defaults
+- [RubyLLM Documentation](https://rubyllm.com/) - Multi-provider LLM interface
+- [Ollama Documentation](https://ollama.ai/docs) - Local LLM provider
+- [OpenAI Embeddings](https://platform.openai.com/docs/guides/embeddings) - Cloud embeddings

data/docs/api/yard/HTM/ActiveRecordConfig.md

@@ -4,6 +4,10 @@
 
 ActiveRecord database configuration and model loading
 
+Uses HTM::Config for database settings. Configuration can come from:
+*   Environment variables (HTM_DATABASE__URL, HTM_DATABASE__HOST, etc.)
+*   Programmatic configuration via HTM.configure
+
 
 # Class Methods
 ## connected?() {: #method-c-connected? }
@@ -18,6 +22,8 @@ Close all database connections
 Establish database connection from HTM::Config
 ## load_database_config() {: #method-c-load_database_config }
 Load database configuration from HTM::Config
+**`@return`** [Hash] ActiveRecord-compatible configuration hash
+
 ## verify_extensions!() {: #method-c-verify_extensions! }
 Verify required extensions are available