htm 0.0.1 → 0.0.10

data/examples/README.md

@@ -0,0 +1,280 @@
+# HTM Examples
+
+This directory contains example applications demonstrating various ways to use the HTM (Hierarchical Temporary Memory) gem.
+
+## Prerequisites
+
+All examples require:
+
+1. **PostgreSQL Database** with pgvector extension:
+   ```bash
+   export HTM_DBURL="postgresql://user@localhost:5432/htm_development"
+   ```
+
+2. **Ollama** (recommended for local LLM):
+   ```bash
+   ollama pull nomic-embed-text  # For embeddings
+   ollama pull gemma3            # For tag extraction
+   ```
+
+3. **Ruby Dependencies**:
+   ```bash
+   bundle install
+   ```
+
+---
+
+## Standalone Scripts
+
+### basic_usage.rb
+
+**Getting started with HTM fundamentals.**
+
+Demonstrates the core API: configuring HTM, registering a robot, and using the three primary methods (`remember`, `recall`, `forget`).
+
+```bash
+ruby examples/basic_usage.rb
+```
+
+**Features:**
+- HTM configuration with Ollama provider
+- Robot initialization
+- Storing memories with `remember()`
+- Retrieving memories with `recall()` using timeframes
+- Understanding async embedding/tag generation
+
+---
+
+### custom_llm_configuration.rb
+
+**Flexible LLM integration patterns.**
+
+Shows how to configure HTM with custom embedding and tag generation methods, supporting multiple LLM providers or custom infrastructure.
+
+```bash
+ruby examples/custom_llm_configuration.rb
+```
+
+**Features:**
+- Default configuration (RubyLLM + Ollama)
+- Custom lambdas for embedding generation
+- Custom lambdas for tag extraction
+- Service object integration pattern
+- Mixed configuration (custom embedding + default tags)
+- Provider settings (OpenAI, Anthropic, Gemini, etc.)
+
+---
+
+### file_loader_usage.rb
+
+**Loading documents into long-term memory.**
+
+Demonstrates loading markdown files with automatic paragraph chunking, YAML frontmatter extraction, and source tracking for re-sync.
+
+```bash
+ruby examples/file_loader_usage.rb
+```
+
+**Features:**
+- Single file loading with `load_file()`
+- Directory loading with glob patterns via `load_directory()`
+- YAML frontmatter extraction (title, author, tags)
+- Querying nodes from a specific file
+- Re-sync behavior (skip unchanged files)
+- Force reload option
+- Unloading files with `unload_file()`
+
+---
+
+### timeframe_demo.rb
+
+**Flexible time-based filtering for recall.**
+
+Comprehensive demonstration of all timeframe options supported by `recall()`, including natural language parsing.
+
+```bash
+ruby examples/timeframe_demo.rb
+```
+
+**Features:**
+- No filter (`nil`)
+- Date/DateTime/Time objects (entire day)
+- Range for precise time windows
+- Natural language strings ("yesterday", "last week", "few days ago")
+- Weekend expressions ("last weekend", "2 weekends ago")
+- Automatic extraction (`:auto`) from query text
+- Multiple time windows (array of ranges)
+
+---
+
+## Application Examples
+
+### example_app/
+
+**Full-featured HTM demonstration with RubyLLM integration.**
+
+A standalone application showing complete HTM workflow with database connection, Ollama integration, memory operations, and multiple search strategies.
+
+```bash
+ruby examples/example_app/app.rb
+```
+
+**Features:**
+- Database connection verification
+- RubyLLM configuration for embeddings and tags
+- Async embedding/tag generation with wait
+- Comparison of search strategies (:fulltext, :vector, :hybrid)
+- Detailed output of generated tags and embeddings
+
+---
+
+### sinatra_app/
+
+**Web application with Sidekiq background processing.**
+
+A Sinatra-based web application demonstrating HTM in a multi-user web context with async job processing.
+
+```bash
+cd examples/sinatra_app
+bundle install
+bundle exec ruby app.rb
+```
+
+**Features:**
+- Sidekiq integration for background jobs
+- Session-based robot identification
+- RESTful API endpoints:
+  - `POST /api/remember` - Store information
+  - `GET /api/recall` - Search memories with timeframe filtering
+  - `GET /api/stats` - Memory statistics
+  - `GET /api/tags` - Tag tree structure
+  - `GET /api/health` - Health check
+- Interactive HTML UI with hybrid search scoring display
+- Tag tree visualization
+
+**Environment Variables:**
+- `HTM_DBURL` - PostgreSQL connection (required)
+- `REDIS_URL` - Redis for Sidekiq (default: redis://localhost:6379/0)
+- `SESSION_SECRET` - Session encryption key
+
+---
+
+### cli_app/
+
+**Interactive command-line application.**
+
+A REPL-style CLI demonstrating synchronous job execution with the `:inline` backend, ideal for CLI tools and scripts.
+
+```bash
+ruby examples/cli_app/htm_cli.rb
+```
+
+**Commands:**
+- `remember <text>` - Store information (waits for embedding/tags)
+- `recall <topic>` - Hybrid search with LLM-powered response generation
+- `tags [prefix]` - List all tags with linked node content
+- `stats` - Memory statistics
+- `help` - Show help
+- `exit` - Quit
+
+**Features:**
+- Synchronous job execution (`:inline` backend)
+- Real-time progress feedback
+- Tag extraction visibility during search
+- Hybrid search with scoring (similarity, tag_boost, combined)
+- RubyLLM chat integration for context-aware responses
+- Response storage in long-term memory
+
+See [cli_app/README.md](cli_app/README.md) for detailed documentation.
+
+---
+
+### robot_groups/
+
+**Multi-robot coordination with shared working memory.**
+
+Demonstrates high-availability patterns with shared working memory, failover, and real-time synchronization via PostgreSQL LISTEN/NOTIFY.
+
+#### same_process.rb
+
+Single-process demonstration of robot groups:
+
+```bash
+ruby examples/robot_groups/same_process.rb
+```
+
+**Scenarios demonstrated:**
+1. Creating a group with primary + standby robots
+2. Adding shared memories
+3. Verifying synchronization
+4. Simulating failover (primary dies, standby takes over)
+5. Verifying standby has full context
+6. Dynamic scaling (adding new robots)
+7. Collaborative memory (multiple robots adding)
+8. Real-time sync via PostgreSQL LISTEN/NOTIFY
+
+#### multi_process.rb
+
+Cross-process demonstration with separate Ruby processes:
+
+```bash
+ruby examples/robot_groups/multi_process.rb
+```
+
+**Scenarios demonstrated:**
+1. Spawning robot worker processes
+2. Cross-process memory sharing
+3. Collaborative memory updates
+4. Failover when a process dies
+5. Dynamic scaling (adding new processes)
+
+**Key concepts:**
+- **Shared Working Memory**: Multiple robots share context via `robot_nodes` table
+- **Active/Passive Roles**: Active robots participate; passive robots maintain warm standby
+- **Failover**: Instant takeover with full context already loaded
+- **Real-time Sync**: PostgreSQL LISTEN/NOTIFY for in-memory cache coordination
+
+---
+
+## Directory Structure
+
+```
+examples/
+├── README.md                      # This file
+├── basic_usage.rb                 # Core API demonstration
+├── custom_llm_configuration.rb    # LLM integration patterns
+├── file_loader_usage.rb           # Document loading
+├── timeframe_demo.rb              # Time-based filtering
+├── example_app/
+│   ├── app.rb                     # Full-featured demo app
+│   └── Rakefile
+├── sinatra_app/
+│   ├── app.rb                     # Sinatra web application
+│   ├── Gemfile
+│   └── Gemfile.lock
+├── cli_app/
+│   ├── htm_cli.rb                 # Interactive CLI
+│   └── README.md                  # Detailed CLI documentation
+└── robot_groups/
+    ├── same_process.rb            # Single-process robot groups
+    ├── multi_process.rb           # Multi-process coordination
+    ├── robot_worker.rb            # Worker process for multi_process.rb
+    └── lib/
+        ├── robot_group.rb         # RobotGroup coordination class
+        └── working_memory_channel.rb  # PostgreSQL pub/sub
+```
+
+---
+
+## Choosing the Right Example
+
+| Use Case | Example |
+|----------|---------|
+| Learning HTM basics | `basic_usage.rb` |
+| Custom LLM integration | `custom_llm_configuration.rb` |
+| Loading documents/files | `file_loader_usage.rb` |
+| Time-based queries | `timeframe_demo.rb` |
+| Web application | `sinatra_app/` |
+| CLI tool | `cli_app/` |
+| Multi-robot coordination | `robot_groups/` |
+| High availability | `robot_groups/` |

data/examples/basic_usage.rb

@@ -4,8 +4,8 @@
 # Basic usage example for HTM
 #
 # Prerequisites:
-# 1. Source environment variables: source ~/.bashrc__tiger
-# 2. Initialize database schema: ruby -r ./lib/htm -e "HTM::Database.setup"
+# 1. Set HTM_DBURL environment variable (see SETUP.md)
+# 2. Initialize database schema: rake db_setup
 # 3. Install dependencies: bundle install
 
 require_relative '../lib/htm'
@@ -15,19 +15,21 @@ puts "=" * 60
 
 # Check environment
 unless ENV['HTM_DBURL']
-  puts "ERROR: HTM_DBURL not set. Please run: source ~/.bashrc__tiger"
+  puts "ERROR: HTM_DBURL not set. Please set it:"
+  puts "  export HTM_DBURL=\"postgresql://postgres@localhost:5432/htm_development\""
+  puts "See SETUP.md for details."
   exit 1
 end
 
 begin
-  # Configure HTM globally (uses RubyLLM with Ollama by default)
+  # Configure HTM globally (uses Ollama by default)
   puts "\n1. Configuring HTM with Ollama provider..."
   HTM.configure do |config|
     config.embedding_provider = :ollama
-    config.embedding_model = 'nomic-embed-text'
+    config.embedding_model = 'nomic-embed-text:latest'  # Ollama models need :tag suffix
     config.embedding_dimensions = 768
     config.tag_provider = :ollama
-    config.tag_model = 'llama3'
+    config.tag_model = 'gemma3:latest'  # Ollama models need :tag suffix
     config.reset_to_defaults  # Apply settings
   end
   puts "✓ HTM configured with Ollama provider"
@@ -47,20 +49,17 @@ begin
   puts "\n3. Remembering information..."
 
   node_id_1 = htm.remember(
-    "We decided to use PostgreSQL for HTM storage because it provides excellent time-series optimization and native vector search with pgvector.",
-    source: "architect"
+    "We decided to use PostgreSQL for HTM storage because it provides excellent time-series optimization and native vector search with pgvector."
   )
   puts "✓ Remembered decision about database choice (node #{node_id_1})"
 
   node_id_2 = htm.remember(
-    "We chose RAG (Retrieval-Augmented Generation) for memory recall, combining temporal filtering with semantic vector search.",
-    source: "architect"
+    "We chose RAG (Retrieval-Augmented Generation) for memory recall, combining temporal filtering with semantic vector search."
   )
   puts "✓ Remembered decision about RAG approach (node #{node_id_2})"
 
   node_id_3 = htm.remember(
-    "The user's name is Dewayne and they prefer using debug_me for debugging instead of puts.",
-    source: "system"
+    "The user's name is Dewayne and they prefer using debug_me for debugging instead of puts."
   )
   puts "✓ Remembered fact about user preferences (node #{node_id_3})"
 
@@ -72,18 +71,21 @@ begin
   memories = htm.recall(
     "database",
     timeframe: (Time.now - 3600)..Time.now,  # Last hour
-    limit: 5
+    limit: 5,
+    raw: true  # Return full node data (id, content, etc.)
   )
   puts "✓ Found #{memories.length} memories"
   memories.each do |memory|
-    puts "  - Node #{memory['id']}: #{memory['content'][0..60]}..."
+    content = memory['content'] || memory[:content]
+    node_id = memory['id'] || memory[:id]
+    puts "  - Node #{node_id}: #{content[0..60]}..."
   end
 
   puts "\n" + "=" * 60
   puts "✓ Example completed successfully!"
   puts "\nThe HTM API provides 3 core methods:"
-  puts "  - htm.remember(content, source:) - Store information"
-  puts "  - htm.recall(timeframe:, topic:, ...) - Retrieve memories"
+  puts "  - htm.remember(content, tags: []) - Store information"
+  puts "  - htm.recall(topic, timeframe:, ...) - Retrieve memories"
   puts "  - htm.forget(node_id, confirm:) - Delete a memory"
 
 rescue => e

data/examples/cli_app/htm_cli.rb

@@ -19,6 +19,20 @@
 
 require_relative '../../lib/htm'
 require 'io/console'
+require 'ruby_llm'
+
+PROVIDER = :ollama
+MODEL    = 'gpt-oss:latest'
+
+# Configure RubyLLM for Ollama provider (same pattern as HTM uses)
+RubyLLM.configure do |config|
+  ollama_url = ENV.fetch('OLLAMA_URL', 'http://localhost:11434')
+  ollama_api_base = ollama_url.end_with?('/v1') ? ollama_url : "#{ollama_url}/v1"
+  config.ollama_api_base = ollama_api_base
+end
+
+# Create chat with Ollama model - use assume_model_exists to bypass registry check
+# AI = RubyLLM.chat(model: MODEL, provider: PROVIDER, assume_model_exists: true)
 
 class HTMCli
   def initialize
@@ -43,6 +57,9 @@ class HTMCli
       end
     end
 
+    # Initialize RubyLLM chat for context-aware responses
+    @chat = RubyLLM.chat(model: MODEL, provider: PROVIDER)
+
     # Initialize HTM instance
     @htm = HTM.new(robot_name: "cli_assistant")
   end
@@ -59,6 +76,7 @@ class HTMCli
     puts "Commands:"
     puts "  remember <text>  - Store information"
     puts "  recall <topic>   - Search memories"
+    puts "  tags             - List all tags with linked nodes"
     puts "  stats            - Show memory statistics"
     puts "  help             - Show this help"
     puts "  exit             - Quit"
@@ -91,6 +109,8 @@ class HTMCli
       handle_remember(args)
     when 'recall'
       handle_recall(args)
+    when 'tags'
+      handle_tags(args)
     when 'stats'
       handle_stats
     when 'help'
@@ -113,7 +133,7 @@ class HTMCli
     puts "Processing..."
 
     start_time = Time.now
-    node_id = @htm.remember(text, source: 'cli_user')
+    node_id = @htm.remember(text)
     duration = ((Time.now - start_time) * 1000).round(2)
 
     puts "[✓] Stored as node #{node_id} (#{duration}ms)"
@@ -141,7 +161,30 @@ class HTMCli
     end
 
     puts "\nSearching for: \"#{topic}\""
-    puts "Strategy: hybrid (vector + fulltext)"
+    puts "Strategy: hybrid (vector + fulltext + tags)"
+
+    # Show tags extracted from query and which ones matched
+    tag_result = @htm.long_term_memory.find_query_matching_tags(topic, include_extracted: true)
+
+    if tag_result[:extracted].any?
+      puts "Extracted tags: #{tag_result[:extracted].join(', ')}"
+
+      # Show what was actually searched (exact + prefixes)
+      searched = tag_result[:extracted].dup
+      tag_result[:extracted].each do |tag|
+        levels = tag.split(':')
+        (1...levels.size).each { |i| searched << levels[0, i].join(':') }
+      end
+      puts "Searched for:   #{searched.uniq.join(', ')}"
+
+      if tag_result[:matched].any?
+        puts "Matched in DB:  #{tag_result[:matched].join(', ')}"
+      else
+        puts "Matched in DB:  (none)"
+      end
+    else
+      puts "Extracted tags: (none)"
+    end
 
     start_time = Time.now
     memories = @htm.recall(
@@ -161,16 +204,103 @@ class HTMCli
 
     memories.each_with_index do |memory, index|
       puts
-      puts "#{index + 1}. Node #{memory['id']} (#{memory['source']})"
+      puts "#{index + 1}. Node #{memory['id']}"
       puts "   Created: #{memory['created_at']}"
       puts "   Content: #{memory['content']}"
 
+      # Show scores from hybrid search
+      if memory['combined_score']
+        similarity = (memory['similarity'].to_f * 100).round(1)
+        tag_boost = (memory['tag_boost'].to_f * 100).round(1)
+        combined = (memory['combined_score'].to_f * 100).round(1)
+        puts "   Scores: similarity=#{similarity}%, tag_boost=#{tag_boost}%, combined=#{combined}%"
+      end
+
       # Show tags if any
       node = HTM::Models::Node.includes(:tags).find(memory['id'])
       if node.tags.any?
         puts "   Tags: #{node.tags.map(&:name).join(', ')}"
+      else
+        puts "   Tags: (none)"
       end
     end
+
+    # Build LLM prompt with context from retrieved memories
+    context_content = memories.map { |m| m['content'] }.join("\n\n")
+
+    llm_prompt = <<~PROMPT
+      #{topic}
+      Your response should highlight information also found in the
+      following context:
+      <CONTEXT>
+      #{context_content}
+      </CONTEXT>
+    PROMPT
+
+    puts "\n" + "=" * 60
+    puts "Generating response for this prompt..."
+    puts llm_prompt
+    puts "=" * 60
+
+    begin
+      response = @chat.ask(llm_prompt)
+      puts "\n#{response.content}"
+
+      # Remember the LLM response in long-term memory
+      node_id = @htm.remember(response.content)
+      puts "\n[✓] Response stored as node #{node_id}"
+    rescue StandardError => e
+      puts "[✗] LLM Error: #{e.message}"
+    end
+  end
+
+  def handle_tags(filter = nil)
+    puts "\nTags Overview:"
+    puts
+
+    # Get all tags with their nodes, optionally filtered by prefix
+    tags_query = HTM::Models::Tag.includes(:nodes).order(:name)
+    tags_query = tags_query.where("name LIKE ?", "#{filter}%") if filter && !filter.empty?
+
+    tags = tags_query.to_a
+
+    if tags.empty?
+      if filter
+        puts "[•] No tags found matching '#{filter}'"
+      else
+        puts "[•] No tags in database"
+      end
+      return
+    end
+
+    puts "[✓] Found #{tags.length} tags#{filter ? " matching '#{filter}'" : ""}"
+    puts
+
+    tags.each do |tag|
+      nodes = tag.nodes.to_a
+      puts "━" * 60
+      puts "Tag: #{tag.name}"
+      puts "     Nodes: #{nodes.length}"
+      puts
+
+      if nodes.any?
+        nodes.each do |node|
+          content_preview = node.content.to_s[0..70]
+          content_preview += "..." if node.content.to_s.length > 70
+          puts "     [#{node.id}] #{content_preview}"
+        end
+      else
+        puts "     (no nodes linked)"
+      end
+      puts
+    end
+
+    # Show tag hierarchy summary
+    root_tags = tags.map { |t| t.name.split(':').first }.uniq.sort
+    if root_tags.length > 1
+      puts "━" * 60
+      puts "Root categories: #{root_tags.join(', ')}"
+    end
   end
 
   def handle_stats
@@ -201,7 +331,7 @@ class HTMCli
     puts
 
     puts "Current Robot (#{@htm.robot_name}):"
-    robot_nodes = HTM::Models::Node.where(robot_id: @htm.robot_id).count
+    robot_nodes = HTM::Models::RobotNode.where(robot_id: @htm.robot_id).count
     puts "  Nodes: #{robot_nodes}"
   end
 
@@ -216,8 +346,17 @@ class HTMCli
     puts
     puts "  recall <topic>"
     puts "    Search for relevant memories using hybrid search"
-    puts "    Combines vector similarity and full-text search"
+    puts "    Combines vector similarity, full-text search, and tag matching"
+    puts "    Shows matching tags, scores (similarity, tag_boost, combined)"
     puts "    Example: recall PostgreSQL"
+    puts "    Example: recall database optimization"
+    puts
+    puts "  tags [prefix]"
+    puts "    List all tags with their linked node content"
+    puts "    Optionally filter by tag prefix"
+    puts "    Example: tags"
+    puts "    Example: tags database"
+    puts "    Example: tags ai:machine"
     puts
     puts "  stats"
     puts "    Show memory statistics and current state"
@@ -242,10 +381,9 @@ unless ENV['HTM_DBURL']
   puts "[✗] Error: HTM_DBURL environment variable not set"
   puts
   puts "Please set your database connection URL:"
-  puts "  export HTM_DBURL='postgresql://user:pass@host:port/dbname'"
+  puts "  export HTM_DBURL='postgresql://postgres@localhost:5432/htm_development'"
   puts
-  puts "Or source your environment configuration:"
-  puts "  source ~/.bashrc__tiger"
+  puts "See SETUP.md for database setup instructions."
   puts
   exit 1
 end