htm 0.0.20 → 0.0.30

data/docs/development/rake-tasks.md

@@ -1,6 +1,6 @@
 # HTM Rake Tasks Reference
 
-Complete reference for all HTM rake tasks. HTM provides 44 rake tasks organized into five namespaces for managing the database, documentation, file loading, background jobs, and tag management.
+Complete reference for all HTM rake tasks. HTM provides 46+ rake tasks organized into five namespaces for managing the database, documentation, file loading, background jobs, and tag management.
 
 ## Quick Reference
 
@@ -337,6 +337,64 @@ $ rake htm:db:reset
 
 ---
 
+#### `rake htm:db:purge_all`
+
+Permanently removes all soft-deleted records from all tables, including orphaned join table entries, orphaned propositions, and orphaned robots.
+
+!!! danger "Warning"
+    This task permanently deletes data and cannot be undone!
+
+```bash
+$ rake htm:db:purge_all
+
+HTM Purge All Soft-Deleted Records
+============================================================
+
+Records to permanently delete:
+--------------------------------------------------------------
+Soft-deleted nodes:              23
+Soft-deleted node_tags:          45
+Soft-deleted robot_nodes:        12
+Orphaned node_tags:              3
+Orphaned robot_nodes:            1
+Orphaned propositions:           5
+Orphaned robots (no nodes):      2
+--------------------------------------------------------------
+Total records to delete:         91
+
+Proceed with permanent deletion? (yes/no): yes
+
+Deleting records...
+  Deleted 45 soft-deleted node_tags entries
+  Deleted 3 orphaned node_tags entries
+  Deleted 12 soft-deleted robot_nodes entries
+  Deleted 1 orphaned robot_nodes entries
+  Deleted 5 orphaned propositions
+  Deleted 23 soft-deleted nodes
+  Deleted 2 orphaned robots
+
+✓ Purge complete!
+```
+
+**What it removes:**
+
+- **Soft-deleted records**: Nodes, node_tags, and robot_nodes with `deleted_at` set
+- **Orphaned join entries**: node_tags and robot_nodes pointing to non-existent nodes
+- **Orphaned propositions**: Proposition nodes whose `source_node_id` no longer exists
+- **Orphaned robots**: Robots with no associated memory nodes
+
+**Deletion order** (for referential integrity):
+
+1. Soft-deleted node_tags
+2. Orphaned node_tags
+3. Soft-deleted robot_nodes
+4. Orphaned robot_nodes
+5. Orphaned propositions
+6. Soft-deleted nodes
+7. Orphaned robots
+
+---
+
 ## Documentation Tasks (`htm:doc:*`)
 
 Tasks for generating API documentation, database diagrams, and building the documentation site.
@@ -645,75 +703,62 @@ Sync Status:
 
 ## Job Tasks (`htm:jobs:*`)
 
-Tasks for managing HTM background jobs (embedding generation, tag extraction, proposition extraction).
+Tasks for managing HTM background jobs (embedding generation, tag extraction, proposition extraction). All processing tasks include progress bars with ETA display for visual feedback during long-running operations.
 
 ### Processing Jobs
 
-#### `rake htm:jobs:process`
+#### `rake htm:jobs:process_all`
 
-Processes all pending background jobs (embeddings, tags, propositions).
+Processes all pending background jobs in sequence: embeddings, then tags, then propositions.
 
 ```bash
-$ rake htm:jobs:process
-Processing pending jobs...
-  Embedding jobs: 5 pending
-  Tag jobs: 8 pending
-  Proposition jobs: 0 pending
-  Processing...
-✓ Processed 13 jobs (13 successful, 0 failed)
+$ rake htm:jobs:process_all
+# Runs process_embeddings, process_tags, process_propositions in sequence
 ```
 
 ---
 
 #### `rake htm:jobs:process_embeddings`
 
-Processes only pending embedding generation jobs.
+Processes nodes without embeddings, generating vector embeddings via the configured LLM provider. Shows a progress bar with ETA for visual feedback.
 
 ```bash
 $ rake htm:jobs:process_embeddings
-Processing embedding jobs...
-  5 pending jobs
-  Processing...
-    ✓ Node 1542 - embedding generated (768 dimensions)
-    ✓ Node 1543 - embedding generated (768 dimensions)
-    ...
-✓ Processed 5 embedding jobs
+Processing 50 nodes without embeddings...
+Embeddings: |████████████████████████████████| 50/50 (100%) Time: 00:01:45
+✓ Generated embeddings for 50 nodes
 ```
 
+**Progress bar format:** `Title: |████████████| count/total (percent%) ETA: mm:ss`
+
 ---
 
 #### `rake htm:jobs:process_tags`
 
-Processes only pending tag extraction jobs.
+Processes nodes without tags, extracting hierarchical tags via the configured LLM provider. Shows a progress bar with ETA for visual feedback.
 
 ```bash
 $ rake htm:jobs:process_tags
-Processing tag jobs...
-  8 pending jobs
-  Processing...
-    ✓ Node 1542 - extracted 3 tags
-    ✓ Node 1543 - extracted 2 tags
-    ...
-✓ Processed 8 tag jobs
+Processing 50 nodes without tags...
+Tags: |████████████████████████████████| 50/50 (100%) Time: 00:02:30
+✓ Extracted tags for 50 nodes
 ```
 
 ---
 
 #### `rake htm:jobs:process_propositions`
 
-Processes only pending proposition extraction jobs.
+Extracts atomic propositions from nodes that haven't been processed yet. Tracks processed nodes via `source_node_id` metadata on proposition nodes. Shows a progress bar with ETA.
 
 ```bash
 $ rake htm:jobs:process_propositions
-Processing proposition jobs...
-  3 pending jobs
-  Processing...
-    ✓ Node 1542 - extracted 5 propositions
-    ✓ Node 1543 - extracted 3 propositions
-    ...
-✓ Processed 3 proposition jobs, created 12 proposition nodes
+Processing 25 nodes for proposition extraction...
+Propositions: |████████████████████████████████| 25/25 (100%) Time: 00:03:15
+✓ Extracted propositions from 25 nodes, created 89 proposition nodes
 ```
 
+**Note:** Only nodes that don't already have propositions extracted will be processed. The task tracks this by checking for proposition nodes with matching `source_node_id` in metadata.
+
 ---
 
 ### Job Status

data/docs/guides/mcp-server.md

@@ -57,7 +57,7 @@ The server logs to STDERR to avoid corrupting the JSON-RPC protocol on STDOUT.
 
 ## CLI Commands
 
-The `htm_mcp` executable includes management commands for database setup and diagnostics:
+The `htm_mcp` executable includes management commands for database setup, diagnostics, and rake task execution:
 
 | Command | Description |
 |---------|-------------|
@@ -69,6 +69,9 @@ The `htm_mcp` executable includes management commands for database setup and dia
 | `htm_mcp stats` | Show memory statistics (nodes, tags, robots, database size) |
 | `htm_mcp version` | Show HTM version |
 | `htm_mcp help` | Show help with all environment variables |
+| `htm_mcp rake <task>` | Run any HTM rake task |
+| `htm_mcp rake -T` | List all available HTM rake tasks |
+| `htm_mcp rake -T <pattern>` | List HTM rake tasks matching pattern |
 
 ### First-Time Setup
 
@@ -101,6 +104,72 @@ Migration Status
   3 applied, 1 pending
 ```
 
+### Rake Task Passthrough
+
+The `htm_mcp rake` command allows you to run any HTM rake task directly through the MCP CLI. This is useful when working with HTM without a full Rails/Rake environment.
+
+**List all available tasks:**
+
+```bash
+$ htm_mcp rake -T
+# or
+$ htm_mcp rake --tasks
+
+HTM Rake Tasks
+================================================================================
+htm:db:console              # Open psql console to database
+htm:db:create               # Create the database if it doesn't exist
+htm:db:drop                 # Drop all HTM tables (WARNING: destructive!)
+htm:db:info                 # Show database information
+htm:db:migrate              # Run pending database migrations
+htm:db:purge_all            # Permanently delete all soft-deleted records
+...
+```
+
+**Filter tasks by pattern** (like standard `rake -T`):
+
+```bash
+$ htm_mcp rake -T htm:jobs
+
+HTM Rake Tasks
+================================================================================
+htm:jobs:process_all        # Process all pending jobs (embeddings, tags, propositions)
+htm:jobs:process_embeddings # Process pending embedding jobs
+htm:jobs:process_propositions # Process pending proposition extraction jobs
+htm:jobs:process_tags       # Process pending tag extraction jobs
+htm:jobs:stats              # Show job processing statistics
+
+$ htm_mcp rake -T db:rebuild
+
+HTM Rake Tasks
+================================================================================
+htm:db:rebuild:embeddings   # Clear and regenerate all embeddings
+htm:db:rebuild:propositions # Extract propositions from all non-proposition nodes
+```
+
+**Run specific tasks:**
+
+```bash
+# Database tasks
+$ htm_mcp rake htm:db:stats
+$ htm_mcp rake htm:db:verify
+$ htm_mcp rake htm:db:purge_all
+
+# Job processing tasks
+$ htm_mcp rake htm:jobs:process_all
+$ htm_mcp rake htm:jobs:process_embeddings
+
+# Tag tasks
+$ htm_mcp rake htm:tags:tree
+$ htm_mcp rake 'htm:tags:tree[database]'  # With argument
+
+# File tasks
+$ htm_mcp rake htm:files:list
+$ htm_mcp rake htm:files:sync
+```
+
+**Note:** Tasks requiring arguments use the standard rake syntax with brackets quoted for shell safety: `htm_mcp rake 'htm:files:load[path/to/file.md]'`
+
 ## Tools Reference
 
 ### SetRobotTool

data/examples/.envrc

@@ -0,0 +1,6 @@
+# htm/examples/.envrc
+
+source_up
+
+export HTM_ENV=examples
+export HTM_DATABASE__URL="postgresql://dewayne@localhost:5432/htm_$HTM_ENV"

data/examples/.gitignore

@@ -0,0 +1,2 @@
+*.log
+*.log.0

data/examples/00_create_examples_db.rb

@@ -0,0 +1,94 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+# Creates and sets up the htm_examples database
+#
+# This script should be run before any other examples to ensure
+# the examples database exists and has the correct schema.
+#
+# Run via:
+#   ruby examples/00_create_examples_db.rb
+#   # or
+#   ./examples/00_create_examples_db.rb
+
+$LOAD_PATH.unshift File.expand_path("../lib", __dir__)
+
+# Set examples environment BEFORE loading HTM
+ENV['HTM_ENV'] = 'examples'
+
+service_name = ENV['HTM_SERVICE__NAME'] || 'htm'
+db_name = "#{service_name}_examples"
+db_url = "postgresql://#{ENV['USER']}@localhost:5432/#{db_name}"
+
+ENV['HTM_DATABASE__URL'] = db_url
+
+puts <<~HEADER
+
+  ==========================================
+  == HTM Examples Database Setup ==
+  ==========================================
+
+  Database: #{db_name}
+
+HEADER
+
+# Step 1: Create database if it doesn't exist
+puts "1. Checking if database exists..."
+
+check_cmd = "psql -lqt | cut -d \\| -f 1 | grep -qw #{db_name}"
+db_exists = system(check_cmd)
+
+if db_exists
+  puts "   Database '#{db_name}' already exists."
+else
+  puts "   Creating database '#{db_name}'..."
+  unless system("createdb #{db_name}")
+    abort "   ERROR: Failed to create database '#{db_name}'"
+  end
+  puts "   Database created."
+end
+
+# Step 2: Enable extensions
+puts "\n2. Enabling PostgreSQL extensions..."
+
+extensions_sql = <<~SQL
+  CREATE EXTENSION IF NOT EXISTS vector;
+  CREATE EXTENSION IF NOT EXISTS pg_trgm;
+SQL
+
+unless system("psql -d #{db_name} -c \"#{extensions_sql}\" > /dev/null 2>&1")
+  abort "   ERROR: Failed to enable extensions"
+end
+puts "   Extensions enabled (pgvector, pg_trgm)."
+
+# Step 3: Run migrations
+puts "\n3. Running schema migrations..."
+
+require "htm"
+
+HTM.configure do |config|
+  config.job.backend = :inline
+  config.log_level = :info
+  config.telemetry_enabled = false
+end
+
+begin
+  HTM::SequelConfig.establish_connection!
+  HTM::Database.setup
+  puts "   Schema setup complete."
+rescue => e
+  abort "   ERROR: #{e.message}"
+end
+
+puts <<~FOOTER
+
+  ==========================================
+  Examples database is ready!
+
+  You can now run examples:
+    ruby examples/basic_usage.rb
+    rake examples:basic
+    rake examples:all
+  ==========================================
+
+FOOTER

data/examples/{basic_usage.rb → 01_basic_usage.rb}

@@ -4,25 +4,21 @@
 # Basic usage example for HTM
 #
 # Prerequisites:
-# 1. Configure database via environment or config file:
-#    - HTM_DATABASE__URL="postgresql://user@localhost:5432/htm_development"
-#    - Or individual vars: HTM_DATABASE__HOST, HTM_DATABASE__NAME, etc.
-# 2. Initialize database schema: rake htm:db:setup
-# 3. Install dependencies: bundle install
+# 1. Set up examples database: rake examples:setup
+# 2. Install dependencies: bundle install
+#
+# Run via:
+#   rake examples:basic
+#   # or
+#   ruby examples/basic_usage.rb
 
-require_relative '../lib/htm'
+require_relative 'examples_helper'
 
-puts "HTM Basic Usage Example"
-puts "=" * 60
+ExamplesHelper.section "HTM Basic Usage Example"
+ExamplesHelper.print_environment
 
-# Check database configuration using the config system
-unless HTM.config.database_configured?
-  puts "ERROR: Database not configured. Set one of:"
-  puts "  export HTM_DATABASE__URL=\"postgresql://user@localhost:5432/htm_development\""
-  puts "  Or configure in ~/.config/htm/htm.yml"
-  puts "Run 'bin/htm_mcp help' for all configuration options."
-  exit 1
-end
+# Verify database is available
+ExamplesHelper.require_database!
 
 begin
   # Configure HTM globally (uses Ollama by default from defaults.yml)

data/examples/{custom_llm_configuration.rb → 03_custom_llm_configuration.rb}

@@ -5,11 +5,21 @@
 #
 # This example demonstrates how to configure HTM with custom LLM methods
 # for embedding generation and tag extraction, as well as using defaults.
+#
+# Prerequisites:
+# 1. Set up examples database: rake examples:setup
+# 2. Install dependencies: bundle install
+#
+# Run via:
+#   ruby examples/custom_llm_configuration.rb
 
-require_relative '../lib/htm'
+require_relative 'examples_helper'
 
-puts "HTM Custom LLM Configuration Example"
-puts "=" * 50
+ExamplesHelper.section "HTM Custom LLM Configuration Example"
+ExamplesHelper.print_environment
+
+# Verify database is available
+ExamplesHelper.require_database!
 
 # Example 1: Use Default Configuration (RubyLLM with Ollama)
 puts "\n1. Using Default Configuration (RubyLLM + Ollama)"

data/examples/{file_loader_usage.rb → 04_file_loader_usage.rb}

@@ -10,24 +10,21 @@
 # - Unloading files
 #
 # Prerequisites:
-# 1. Set HTM_DATABASE__URL environment variable (see SETUP.md)
-# 2. Initialize database schema: rake db_setup
-# 3. Install dependencies: bundle install
+# 1. Set up examples database: rake examples:setup
+# 2. Install dependencies: bundle install
+#
+# Run via:
+#   ruby examples/file_loader_usage.rb
 
-require_relative '../lib/htm'
+require_relative 'examples_helper'
 require 'tempfile'
 require 'fileutils'
 
-puts "HTM File Loader Example"
-puts "=" * 60
+ExamplesHelper.section "HTM File Loader Example"
+ExamplesHelper.print_environment
 
-# Check environment
-unless ENV['HTM_DATABASE__URL']
-  puts "ERROR: HTM_DATABASE__URL not set. Please set it:"
-  puts "  export HTM_DATABASE__URL=\"postgresql://postgres@localhost:5432/htm_development\""
-  puts "See SETUP.md for details."
-  exit 1
-end
+# Verify database is available
+ExamplesHelper.require_database!
 
 begin
   # Configure HTM globally (uses Ollama by default)
@@ -102,7 +99,7 @@ begin
   puts "   Skipped: #{result[:skipped]}"
 
   # Access the file source to show frontmatter
-  source = HTM::Models::FileSource.find(result[:file_source_id])
+  source = HTM::Models::FileSource[result[:file_source_id]]
   puts "   Frontmatter title: #{source.title}"
   puts "   Frontmatter author: #{source.author}"
   puts "   Frontmatter tags: #{source.frontmatter_tags.join(', ')}"

data/examples/{timeframe_demo.rb → 05_timeframe_demo.rb}

@@ -3,10 +3,17 @@
 
 # Timeframe Demo - Demonstrates the various ways to use timeframes with recall
 #
-# Run with:
-#   HTM_DATABASE__URL="postgresql://localhost/htm_development" ruby examples/timeframe_demo.rb
+# Prerequisites:
+# 1. Set up examples database: rake examples:setup
+# 2. Install dependencies: bundle install
+#
+# Run via:
+#   rake examples:basic  # runs basic_usage.rb
+#   ruby examples/timeframe_demo.rb
+
+require_relative 'examples_helper'
 
-require_relative "../lib/htm"
+ExamplesHelper.require_database!
 
 puts <<~HEADER
   ╔══════════════════════════════════════════════════════════════════╗

data/examples/{example_app → 06_example_app}/app.rb

@@ -2,27 +2,27 @@
 # Example HTM Application
 #
 # This demonstrates a simple application using the HTM gem with full RubyLLM integration.
+#
+# Prerequisites:
+# 1. Set up examples database: rake examples:setup
+# 2. Install dependencies: bundle install
+#
+# Run via:
+#   ruby examples/example_app/app.rb
 
-require_relative '../../lib/htm'
+require_relative '../examples_helper'
 require 'ruby_llm'
 require 'logger'
 
 class ExampleApp
   def self.run
     puts "\n=== HTM Full-Featured Example Application ==="
-    puts "\nChecking database connection..."
-
-    # Check if database is configured
-    config = HTM::Database.default_config
-    unless config
-      puts "\n⚠ Database not configured!"
-      puts "Set HTM_DATABASE__URL environment variable:"
-      puts "  export HTM_DATABASE__URL='postgresql://user:pass@host:port/dbname'"
-      puts "\nOr run: rake app:bootstrap"
-      return
-    end
+    ExamplesHelper.print_environment
+
+    # Verify database is available
+    ExamplesHelper.require_database!
 
-    puts "✓ Database configured: #{config[:dbname]} @ #{config[:host]}"
+    puts "Database configured: #{HTM.config.actual_database_name}"
 
     # Configure HTM with RubyLLM for embeddings and tag generation
     puts "\nConfiguring HTM with RubyLLM..."
@@ -95,7 +95,7 @@ class ExampleApp
     # Check what was generated
     puts "\n--- Generated Tags ---"
     [node_1, node_2, node_3].each do |node_id|
-      node = HTM::Models::Node.includes(:tags).find(node_id)
+      node = HTM::Models::Node[node_id]
       if node.tags.any?
         puts "Node #{node_id}:"
         node.tags.each { |tag| puts "  - #{tag.name}" }
@@ -107,7 +107,7 @@ class ExampleApp
     # Check embeddings
     puts "\n--- Embedding Status ---"
     [node_1, node_2, node_3].each do |node_id|
-      node = HTM::Models::Node.find(node_id)
+      node = HTM::Models::Node[node_id]
       if node.embedding
         dimensions = node.embedding.is_a?(Array) ? node.embedding.size : node.embedding_dimension
         status = "✓ Generated (#{dimensions} dimensions)"

data/examples/{cli_app → 07_cli_app}/htm_cli.rb

@@ -9,15 +9,18 @@
 # - Progress feedback
 # - Database persistence
 #
+# Prerequisites:
+# 1. Set up examples database: rake examples:setup
+# 2. Install dependencies: bundle install
+#
 # Usage:
-#   ruby htm_cli.rb
+#   ruby examples/cli_app/htm_cli.rb
 #
 # Environment:
-#   HTM_DATABASE__URL - PostgreSQL connection URL (required)
 #   OLLAMA_URL - Ollama server URL (default: http://localhost:11434)
 #
 
-require_relative '../../lib/htm'
+require_relative '../examples_helper'
 require 'io/console'
 require 'ruby_llm'
 
@@ -86,7 +89,7 @@ class HTMCli
 
     loop do
       print "\nhtm> "
-      input = gets&.chomp
+      input = $stdin.gets&.chomp
       break if input.nil? || input == 'exit'
 
       handle_command(input)
@@ -139,7 +142,7 @@ class HTMCli
     puts "[✓] Stored as node #{node_id} (#{duration}ms)"
 
     # Show what was generated (inline mode, so already complete)
-    node = HTM::Models::Node.includes(:tags).find(node_id)
+    node = HTM::Models::Node[node_id]
 
     if node.embedding
       puts "    Embedding: #{node.embedding_dimension} dimensions"
@@ -217,7 +220,7 @@ class HTMCli
       end
 
       # Show tags if any
-      node = HTM::Models::Node.includes(:tags).find(memory['id'])
+      node = HTM::Models::Node[memory['id']]
       if node.tags.any?
         puts "   Tags: #{node.tags.map(&:name).join(', ')}"
       else
@@ -259,10 +262,10 @@ class HTMCli
     puts
 
     # Get all tags with their nodes, optionally filtered by prefix
-    tags_query = HTM::Models::Tag.includes(:nodes).order(:name)
-    tags_query = tags_query.where("name LIKE ?", "#{filter}%") if filter && !filter.empty?
+    tags_query = HTM::Models::Tag.order(:name)
+    tags_query = tags_query.where(Sequel.like(:name, "#{filter}%")) if filter && !filter.empty?
 
-    tags = tags_query.to_a
+    tags = tags_query.eager(:nodes).all
 
     if tags.empty?
       if filter
@@ -308,8 +311,8 @@ class HTMCli
     puts
 
     total_nodes = HTM::Models::Node.count
-    nodes_with_embeddings = HTM::Models::Node.where.not(embedding: nil).count
-    nodes_with_tags = HTM::Models::Node.joins(:tags).distinct.count
+    nodes_with_embeddings = HTM::Models::Node.exclude(embedding: nil).count
+    nodes_with_tags = HTM::Models::NodeTag.distinct.select(:node_id).count
     total_tags = HTM::Models::Tag.count
     total_robots = HTM::Models::Robot.count
 
@@ -376,17 +379,7 @@ class HTMCli
 end
 
 # Check database configuration
-unless ENV['HTM_DATABASE__URL']
-  puts
-  puts "[✗] Error: HTM_DATABASE__URL environment variable not set"
-  puts
-  puts "Please set your database connection URL:"
-  puts "  export HTM_DATABASE__URL='postgresql://postgres@localhost:5432/htm_development'"
-  puts
-  puts "See SETUP.md for database setup instructions."
-  puts
-  exit 1
-end
+ExamplesHelper.require_database!
 
 # Check Ollama connection (optional but recommended)
 begin