htm 0.0.18 → 0.0.30

data/docs/development/rake-tasks.md

@@ -1,6 +1,6 @@
 # HTM Rake Tasks Reference
 
-Complete reference for all HTM rake tasks. HTM provides 44 rake tasks organized into five namespaces for managing the database, documentation, file loading, background jobs, and tag management.
+Complete reference for all HTM rake tasks. HTM provides 46+ rake tasks organized into five namespaces for managing the database, documentation, file loading, background jobs, and tag management.
 
 ## Quick Reference
 
@@ -337,6 +337,64 @@ $ rake htm:db:reset
 
 ---
 
+#### `rake htm:db:purge_all`
+
+Permanently removes all soft-deleted records from all tables, including orphaned join table entries, orphaned propositions, and orphaned robots.
+
+!!! danger "Warning"
+    This task permanently deletes data and cannot be undone!
+
+```bash
+$ rake htm:db:purge_all
+
+HTM Purge All Soft-Deleted Records
+============================================================
+
+Records to permanently delete:
+--------------------------------------------------------------
+Soft-deleted nodes:              23
+Soft-deleted node_tags:          45
+Soft-deleted robot_nodes:        12
+Orphaned node_tags:              3
+Orphaned robot_nodes:            1
+Orphaned propositions:           5
+Orphaned robots (no nodes):      2
+--------------------------------------------------------------
+Total records to delete:         91
+
+Proceed with permanent deletion? (yes/no): yes
+
+Deleting records...
+  Deleted 45 soft-deleted node_tags entries
+  Deleted 3 orphaned node_tags entries
+  Deleted 12 soft-deleted robot_nodes entries
+  Deleted 1 orphaned robot_nodes entries
+  Deleted 5 orphaned propositions
+  Deleted 23 soft-deleted nodes
+  Deleted 2 orphaned robots
+
+✓ Purge complete!
+```
+
+**What it removes:**
+
+- **Soft-deleted records**: Nodes, node_tags, and robot_nodes with `deleted_at` set
+- **Orphaned join entries**: node_tags and robot_nodes pointing to non-existent nodes
+- **Orphaned propositions**: Proposition nodes whose `source_node_id` no longer exists
+- **Orphaned robots**: Robots with no associated memory nodes
+
+**Deletion order** (for referential integrity):
+
+1. Soft-deleted node_tags
+2. Orphaned node_tags
+3. Soft-deleted robot_nodes
+4. Orphaned robot_nodes
+5. Orphaned propositions
+6. Soft-deleted nodes
+7. Orphaned robots
+
+---
+
 ## Documentation Tasks (`htm:doc:*`)
 
 Tasks for generating API documentation, database diagrams, and building the documentation site.
@@ -645,75 +703,62 @@ Sync Status:
 
 ## Job Tasks (`htm:jobs:*`)
 
-Tasks for managing HTM background jobs (embedding generation, tag extraction, proposition extraction).
+Tasks for managing HTM background jobs (embedding generation, tag extraction, proposition extraction). All processing tasks include progress bars with ETA display for visual feedback during long-running operations.
 
 ### Processing Jobs
 
-#### `rake htm:jobs:process`
+#### `rake htm:jobs:process_all`
 
-Processes all pending background jobs (embeddings, tags, propositions).
+Processes all pending background jobs in sequence: embeddings, then tags, then propositions.
 
 ```bash
-$ rake htm:jobs:process
-Processing pending jobs...
-  Embedding jobs: 5 pending
-  Tag jobs: 8 pending
-  Proposition jobs: 0 pending
-  Processing...
-✓ Processed 13 jobs (13 successful, 0 failed)
+$ rake htm:jobs:process_all
+# Runs process_embeddings, process_tags, process_propositions in sequence
 ```
 
 ---
 
 #### `rake htm:jobs:process_embeddings`
 
-Processes only pending embedding generation jobs.
+Processes nodes without embeddings, generating vector embeddings via the configured LLM provider. Shows a progress bar with ETA for visual feedback.
 
 ```bash
 $ rake htm:jobs:process_embeddings
-Processing embedding jobs...
-  5 pending jobs
-  Processing...
-    ✓ Node 1542 - embedding generated (768 dimensions)
-    ✓ Node 1543 - embedding generated (768 dimensions)
-    ...
-✓ Processed 5 embedding jobs
+Processing 50 nodes without embeddings...
+Embeddings: |████████████████████████████████| 50/50 (100%) Time: 00:01:45
+✓ Generated embeddings for 50 nodes
 ```
 
+**Progress bar format:** `Title: |████████████| count/total (percent%) ETA: mm:ss`
+
 ---
 
 #### `rake htm:jobs:process_tags`
 
-Processes only pending tag extraction jobs.
+Processes nodes without tags, extracting hierarchical tags via the configured LLM provider. Shows a progress bar with ETA for visual feedback.
 
 ```bash
 $ rake htm:jobs:process_tags
-Processing tag jobs...
-  8 pending jobs
-  Processing...
-    ✓ Node 1542 - extracted 3 tags
-    ✓ Node 1543 - extracted 2 tags
-    ...
-✓ Processed 8 tag jobs
+Processing 50 nodes without tags...
+Tags: |████████████████████████████████| 50/50 (100%) Time: 00:02:30
+✓ Extracted tags for 50 nodes
 ```
 
 ---
 
 #### `rake htm:jobs:process_propositions`
 
-Processes only pending proposition extraction jobs.
+Extracts atomic propositions from nodes that haven't been processed yet. Tracks processed nodes via `source_node_id` metadata on proposition nodes. Shows a progress bar with ETA.
 
 ```bash
 $ rake htm:jobs:process_propositions
-Processing proposition jobs...
-  3 pending jobs
-  Processing...
-    ✓ Node 1542 - extracted 5 propositions
-    ✓ Node 1543 - extracted 3 propositions
-    ...
-✓ Processed 3 proposition jobs, created 12 proposition nodes
+Processing 25 nodes for proposition extraction...
+Propositions: |████████████████████████████████| 25/25 (100%) Time: 00:03:15
+✓ Extracted propositions from 25 nodes, created 89 proposition nodes
 ```
 
+**Note:** Only nodes that don't already have propositions extracted will be processed. The task tracks this by checking for proposition nodes with matching `source_node_id` in metadata.
+
 ---
 
 ### Job Status

data/docs/development/setup.md

@@ -10,7 +10,7 @@ Setting up HTM for development involves:
 2. Installing Ruby and system dependencies
 3. Installing Ruby gem dependencies
 4. Setting up TimescaleDB database
-5. Configuring Ollama for embeddings
+5. Configuring an LLM provider (Ollama, OpenAI, etc.)
 6. Verifying your setup
 7. Running tests and examples
 
@@ -274,77 +274,94 @@ ruby enable_extensions.rb
 
 This ensures that TimescaleDB, pgvector, and pg_trgm extensions are enabled.
 
-## Step 5: Set Up Ollama for Embeddings
+## Step 5: Configure LLM Provider
 
-HTM uses Ollama (via RubyLLM) to generate vector embeddings for semantic search.
+HTM uses RubyLLM to generate vector embeddings and extract tags. RubyLLM supports multiple providers, so you can choose what works best for your development environment.
 
-### Install Ollama
+### Supported Providers
 
-#### macOS
+| Provider | Best For | Setup Required |
+|----------|----------|----------------|
+| **Ollama** (default) | Local development, privacy | Install Ollama + models |
+| **OpenAI** | Production, high-quality | API key only |
+| **Anthropic** | Claude models for tags | API key only |
+| **Gemini** | Google Cloud users | API key only |
 
+### Option A: Ollama (Recommended for Development)
+
+Ollama runs locally with no API costs.
+
+#### Install Ollama
+
+**macOS:**
 ```bash
-# Download and install from official site
+# Direct download
 curl https://ollama.ai/install.sh | sh
 
 # Or using Homebrew
 brew install ollama
 ```
 
-#### Linux
-
+**Linux:**
 ```bash
 curl https://ollama.ai/install.sh | sh
 ```
 
-### Start Ollama Service
+#### Start Ollama Service
 
-Ollama typically starts automatically after installation. Verify it's running:
+Ollama typically starts automatically. Verify it's running:
 
 ```bash
-# Check if Ollama is running
 curl http://localhost:11434/api/version
-
-# Expected output:
-# {"version":"0.1.x"}
 ```
 
-If not running, start it manually:
+If not running:
 
 ```bash
-# macOS - Ollama runs as a background service
-# Check Activity Monitor or start from Applications
-
+# macOS - Check Activity Monitor or start from Applications
 # Linux
 ollama serve &
 ```
 
-### Pull the gpt-oss Model
-
-HTM uses the `gpt-oss` model by default:
+#### Pull Required Models
 
 ```bash
-# Pull the model (downloads ~4GB)
-ollama pull gpt-oss
+# Pull embedding model
+ollama pull nomic-embed-text
+
+# Pull tag extraction model
+ollama pull gemma3:latest
 
-# Verify the model is available
+# Verify models
 ollama list
-# Should show gpt-oss in the list
 ```
 
-### Test Embedding Generation
+#### Configure Custom URL (Optional)
 
 ```bash
-# Test that embeddings work
-ollama run gpt-oss "Hello, HTM!"
+export OLLAMA_URL="http://custom-host:11434"
 ```
 
-### Optional: Configure Custom Ollama URL
+### Option B: OpenAI (Recommended for Production)
 
-If Ollama is running on a different host or port:
+```bash
+export OPENAI_API_KEY="sk-..."
+```
 
+Configure in your code:
+```ruby
+HTM.configure do |config|
+  config.embedding.provider = :openai
+  config.embedding.model = 'text-embedding-3-small'
+end
+```
+
+### Option C: Other Providers
+
+Set the appropriate API key:
 ```bash
-# Add to ~/.bashrc__tiger
-export OLLAMA_URL="http://custom-host:11434"
+export ANTHROPIC_API_KEY="sk-ant-..."
+export GEMINI_API_KEY="..."
 ```
 
 ## Step 6: Initialize Database Schema
@@ -420,11 +437,11 @@ HTM Basic Usage Example
 ============================================================
 
 1. Initializing HTM for 'Code Helper' robot...
-   Using RubyLLM with Ollama provider and gpt-oss model for embeddings
+   Using RubyLLM with configured provider for embeddings
 ✓ HTM initialized
   Robot ID: robot-abc123
   Robot Name: Code Helper
-  Embedding Service: Ollama (gpt-oss via RubyLLM)
+  Embedding Service: Configured provider via RubyLLM
 
 2. Adding memory nodes...
 ✓ Added decision about database choice
@@ -485,11 +502,14 @@ HTM uses environment variables for configuration. Here's a complete reference:
 | `HTM_DATABASE__PORT` | Database port | `37807` |
 | `HTM_SERVICE_NAME` | Service identifier (informational) | `db-67977` |
 
-### Ollama Variables
+### LLM Provider Variables
 
 | Variable | Description | Example |
 |----------|-------------|---------|
-| `OLLAMA_URL` | Ollama API URL (optional) | `http://localhost:11434` |
+| `OLLAMA_URL` | Ollama API URL (if using Ollama) | `http://localhost:11434` |
+| `OPENAI_API_KEY` | OpenAI API key (if using OpenAI) | `sk-...` |
+| `ANTHROPIC_API_KEY` | Anthropic API key (if using Anthropic) | `sk-ant-...` |
+| `GEMINI_API_KEY` | Gemini API key (if using Gemini) | `...` |
 
 ### Managing Environment Files
 
@@ -497,7 +517,7 @@ You can organize your environment variables using multiple files:
 
 ```bash
 # ~/.bashrc__tiger - Database configuration
-# ~/.bashrc__ollama - Ollama configuration (if needed)
+# LLM provider environment variables (set based on your chosen provider)
 
 # Load all configuration files in ~/.bashrc
 source ~/.bashrc__tiger
@@ -527,11 +547,11 @@ psql $HTM_DATABASE__URL
 docker ps | grep timescale
 ```
 
-#### "Ollama connection refused"
+#### "LLM provider connection failed"
 
 **Symptoms**: Embedding generation fails
 
-**Solutions**:
+**Solutions for Ollama**:
 
 ```bash
 # Verify Ollama is running
@@ -541,11 +561,23 @@ curl http://localhost:11434/api/version
 # macOS: Start from Applications or Activity Monitor
 # Linux: ollama serve &
 
-# Check if model is downloaded
-ollama list | grep gpt-oss
+# Check if models are downloaded
+ollama list | grep nomic-embed-text
+
+# Pull models if missing
+ollama pull nomic-embed-text
+ollama pull gemma3:latest
+```
+
+**Solutions for Cloud Providers**:
+
+```bash
+# Verify API key is set
+echo $OPENAI_API_KEY
+echo $ANTHROPIC_API_KEY
 
-# Pull model if missing
-ollama pull gpt-oss
+# Test API connectivity
+curl https://api.openai.com/v1/models -H "Authorization: Bearer $OPENAI_API_KEY"
 ```
 
 #### "Extension not available"
@@ -602,7 +634,7 @@ rake db_setup
 source ~/.bashrc__tiger
 env | grep TIGER
 
-# Check Ollama is running
+# Check LLM provider (if using Ollama)
 curl http://localhost:11434/api/version
 
 # Run tests with verbose output

data/docs/examples/basic-usage.md

@@ -0,0 +1,133 @@
+# Basic Usage Example
+
+This example demonstrates the core HTM operations: configuring the system, remembering information, and recalling memories.
+
+**Source:** [`examples/basic_usage.rb`](https://github.com/madbomber/htm/blob/main/examples/basic_usage.rb)
+
+## Overview
+
+The basic usage example shows:
+
+- Configuring HTM with a provider (Ollama by default)
+- Initializing an HTM instance for a robot
+- Storing memories with `remember()`
+- Retrieving memories with `recall()`
+- Using different search strategies
+
+## Running the Example
+
+```bash
+export HTM_DATABASE__URL="postgresql://user@localhost:5432/htm_development"
+ruby examples/basic_usage.rb
+```
+
+## Code Walkthrough
+
+### Configuration
+
+```ruby
+HTM.configure do |config|
+  # Provider configuration (Ollama is default)
+  config.embedding.provider = :ollama
+  config.embedding.model = 'nomic-embed-text:latest'
+  config.embedding.dimensions = 768
+
+  config.tag.provider = :ollama
+  config.tag.model = 'gemma3:latest'
+
+  # Use inline job backend for synchronous execution
+  # In production, use :thread or :sidekiq for async
+  config.job.backend = :inline
+end
+```
+
+### Initialize HTM
+
+```ruby
+htm = HTM.new(
+  robot_name: "Code Helper",
+  working_memory_size: 128_000  # Token limit
+)
+```
+
+### Storing Memories
+
+```ruby
+# Remember automatically generates embeddings and tags
+node_id = htm.remember(
+  "PostgreSQL supports native vector search with pgvector."
+)
+
+# Tags are extracted via LLM
+# Embeddings enable semantic search
+```
+
+### Recalling Memories
+
+```ruby
+# Full-text search (keyword matching)
+memories = htm.recall(
+  "PostgreSQL",
+  timeframe: (Time.now - 3600)..Time.now,
+  limit: 5,
+  strategy: :fulltext
+)
+
+# Vector search (semantic similarity)
+memories = htm.recall("database features", strategy: :vector)
+
+# Hybrid search (combines both)
+memories = htm.recall("database features", strategy: :hybrid)
+```
+
+## Expected Output
+
+```
+HTM Basic Usage Example
+============================================================
+
+1. Configuring HTM with Ollama provider...
+✓ HTM configured with Ollama provider (inline job backend)
+
+2. Initializing HTM for 'Code Helper' robot...
+✓ HTM initialized
+  Robot ID: 1
+  Robot Name: Code Helper
+  Embedding Service: ollama (nomic-embed-text:latest)
+
+3. Remembering information...
+✓ Remembered decision about database choice (node 1)
+✓ Remembered decision about RAG approach (node 2)
+✓ Remembered fact about user preferences (node 3)
+
+4. Recalling memories about 'PostgreSQL'...
+✓ Found 1 memories
+  - Node 1: We decided to use PostgreSQL for HTM storage because...
+
+============================================================
+✓ Example completed successfully!
+```
+
+## Key Concepts
+
+### Search Strategies
+
+| Strategy | Description | Best For |
+|----------|-------------|----------|
+| `:fulltext` | Keyword matching with PostgreSQL tsvector | Exact term matches |
+| `:vector` | Semantic similarity with pgvector | Conceptual queries |
+| `:hybrid` | Combines both with RRF scoring | General queries |
+
+### Job Backends
+
+| Backend | Behavior | Use Case |
+|---------|----------|----------|
+| `:inline` | Synchronous execution | Examples, testing |
+| `:thread` | Background threads | Development |
+| `:sidekiq` | Sidekiq workers | Production |
+
+## See Also
+
+- [LLM Configuration Example](llm-configuration.md)
+- [Adding Memories Guide](../guides/adding-memories.md)
+- [Search Strategies Guide](../guides/search-strategies.md)

data/docs/examples/config-files.md

@@ -0,0 +1,170 @@
+# Configuration Files Example
+
+This example demonstrates HTM's flexible, layered configuration management using the `anyway_config` gem.
+
+**Source:** [`examples/config_file_example/`](https://github.com/madbomber/htm/tree/main/examples/config_file_example)
+
+## Overview
+
+The configuration example shows:
+
+- Configuration source hierarchy and priority
+- Using `htm_mcp config` to generate config files
+- Environment-specific configuration sections
+- Environment variable overrides with `HTM_*`
+- Custom config file paths with `HTM_CONF`
+- Programmatic configuration with `HTM.configure`
+
+## Configuration Priority
+
+HTM loads configuration from multiple sources, merged in priority order:
+
+| Priority | Source | Location |
+|----------|--------|----------|
+| 1 (lowest) | Bundled Defaults | `lib/htm/config/defaults.yml` |
+| 2 | XDG User Config | `~/.config/htm/htm.yml` |
+| 3 | Project Config | `./config/htm.yml` |
+| 4 | Local Overrides | `./config/htm.local.yml` |
+| 5 | HTM_CONF File | Path in `HTM_CONF` env var |
+| 6 | Environment Variables | `HTM_*` |
+| 7 (highest) | Code Block | `HTM.configure { }` |
+
+## Running the Example
+
+```bash
+cd examples/config_file_example
+
+# Basic usage - loads ./config/htm.local.yml automatically
+ruby show_config.rb
+
+# Different environment
+HTM_ENV=production ruby show_config.rb
+
+# Override with environment variable
+HTM_EMBEDDING__MODEL=mxbai-embed-large ruby show_config.rb
+
+# Use custom config file
+HTM_CONF=./custom_config.yml ruby show_config.rb
+
+# Combine multiple overrides
+HTM_CONF=./custom_config.yml HTM_ENV=test HTM_LOG_LEVEL=warn ruby show_config.rb
+```
+
+## Generating Config Files
+
+Use the `htm_mcp config` command to output the complete default configuration:
+
+```bash
+# Create a project config file
+htm_mcp config > ./config/htm.yml
+
+# Create a local overrides file (add to .gitignore)
+htm_mcp config > ./config/htm.local.yml
+
+# Create a user-wide XDG config
+mkdir -p ~/.config/htm
+htm_mcp config > ~/.config/htm/htm.yml
+```
+
+## Example Config Files
+
+### Project Config with Environment Sections
+
+```yaml
+# ./config/htm.yml
+defaults:
+  database:
+    host: localhost
+    port: 5432
+
+development:
+  database:
+    name: myapp_development
+  log_level: debug
+
+production:
+  database:
+    name: myapp_production
+    pool_size: 25
+  telemetry_enabled: true
+```
+
+### Local Overrides (Gitignored)
+
+```yaml
+# ./config/htm.local.yml
+database:
+  name: my_local_db
+
+providers:
+  openai:
+    api_key: sk-your-actual-key
+
+log_level: debug
+```
+
+## Environment Variables
+
+Use double underscores (`__`) for nested values:
+
+```bash
+# Database configuration
+export HTM_DATABASE__URL="postgresql://user:pass@host:5432/dbname"
+export HTM_DATABASE__POOL_SIZE=25
+
+# Embedding configuration
+export HTM_EMBEDDING__PROVIDER=openai
+export HTM_EMBEDDING__MODEL=text-embedding-3-small
+
+# Provider credentials
+export HTM_PROVIDERS__OPENAI__API_KEY=sk-your-key
+```
+
+## Example Output
+
+The `show_config.rb` script displays loaded configuration with source tracing:
+
+```
+HTM Configuration Example
+============================================================
+
+Environment: development
+
+Config sources checked:
+  - Bundled defaults: lib/htm/config/defaults.yml
+  - XDG config: ~/.config/htm/htm.yml
+  - Project config: ./config/htm.yml
+  - Local overrides: ./config/htm.local.yml
+  - HTM_CONF override: (not set)
+  - Environment variables: HTM_*
+
+------------------------------------------------------------
+Configuration with Sources:
+------------------------------------------------------------
+
+database:
+  host: "localhost"  # from: htm.local.yml
+  port: 5432  # from: htm.local.yml
+  name: "htm_example"  # from: htm.local.yml
+  pool_size: 10  # from: bundled_defaults
+embedding:
+  provider: "ollama"  # from: htm.local.yml
+  model: "nomic-embed-text:latest"  # from: htm.local.yml
+...
+```
+
+## Files in This Example
+
+```
+config_file_example/
+├── README.md              # Detailed documentation
+├── show_config.rb         # Demo script with source tracing
+├── custom_config.yml      # Example for HTM_CONF usage
+└── config/
+    └── htm.local.yml      # Auto-loaded local overrides
+```
+
+## See Also
+
+- [Configuration Guide](../guides/configuration.md)
+- [anyway_config Documentation](https://github.com/palkan/anyway_config)