htm 0.0.18 → 0.0.30

data/docs/guides/file-loading.md

@@ -0,0 +1,322 @@
+# File Loading
+
+HTM can load text-based files (currently markdown) into long-term memory with automatic chunking, source tracking, and re-sync support. This is ideal for building knowledge bases from documentation, notes, or any text content.
+
+## Overview
+
+The file loading system provides:
+
+- **Automatic chunking**: Large files are split into semantically-aware chunks
+- **YAML frontmatter extraction**: Metadata from file headers is preserved
+- **Source tracking**: Files are tracked for re-sync when content changes
+- **Duplicate detection**: Content hashing prevents duplicate chunks
+- **Soft delete**: Unloading files uses soft delete for recovery
+
+## Quick Start
+
+```ruby
+require 'htm'
+
+htm = HTM.new(robot_name: "Document Loader")
+
+# Load a single markdown file
+result = htm.load_file("docs/guide.md")
+# => { file_source_id: 1, chunks_created: 5, chunks_updated: 0, skipped: false }
+
+# Load all markdown files from a directory
+results = htm.load_directory("docs/", pattern: "**/*.md")
+# => [{ file_path: "docs/guide.md", ... }, { file_path: "docs/api.md", ... }]
+
+# Query nodes from a specific file
+nodes = htm.nodes_from_file("docs/guide.md")
+
+# Unload a file (soft deletes chunks)
+htm.unload_file("docs/guide.md")
+```
+
+## API Reference
+
+### load_file(path, force: false)
+
+Loads a single file into long-term memory.
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `path` | String | required | Path to the file |
+| `force` | Boolean | `false` | Force reload even if file unchanged |
+
+**Returns:** Hash with keys:
+- `file_source_id`: ID of the FileSource record
+- `chunks_created`: Number of new chunks created
+- `chunks_updated`: Number of existing chunks updated
+- `chunks_deleted`: Number of chunks removed
+- `skipped`: Whether file was skipped (unchanged)
+
+```ruby
+# Normal load - skips unchanged files
+result = htm.load_file("docs/guide.md")
+
+# Force reload even if file hasn't changed
+result = htm.load_file("docs/guide.md", force: true)
+```
+
+### load_directory(path, pattern: "**/*.md", force: false)
+
+Loads all matching files from a directory.
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `path` | String | required | Directory path |
+| `pattern` | String | `"**/*.md"` | Glob pattern for files |
+| `force` | Boolean | `false` | Force reload all files |
+
+**Returns:** Array of result hashes (one per file)
+
+```ruby
+# Load all markdown files
+results = htm.load_directory("docs/")
+
+# Load only top-level markdown files
+results = htm.load_directory("docs/", pattern: "*.md")
+
+# Load specific subdirectory
+results = htm.load_directory("docs/guides/", pattern: "**/*.md")
+```
+
+### nodes_from_file(path)
+
+Returns all nodes loaded from a specific file.
+
+```ruby
+nodes = htm.nodes_from_file("docs/guide.md")
+nodes.each do |node|
+  puts "#{node.id}: #{node.content[0..50]}..."
+end
+```
+
+### unload_file(path)
+
+Soft deletes all nodes from a file and removes the file source.
+
+```ruby
+count = htm.unload_file("docs/guide.md")
+puts "Removed #{count} chunks"
+```
+
+## YAML Frontmatter
+
+Files with YAML frontmatter have their metadata extracted and stored:
+
+```markdown
+---
+title: PostgreSQL Guide
+author: HTM Team
+tags:
+  - database
+  - postgresql
+version: 1.2
+---
+
+# PostgreSQL Guide
+
+Content starts here...
+```
+
+Access frontmatter via the FileSource model:
+
+```ruby
+source = HTM::Models::FileSource.find_by(file_path: "docs/guide.md")
+source.title            # => "PostgreSQL Guide"
+source.author           # => "HTM Team"
+source.frontmatter_tags # => ["database", "postgresql"]
+source.frontmatter      # => { "title" => "...", "author" => "...", ... }
+```
+
+## Chunking Strategy
+
+HTM uses the [Baran gem](https://github.com/baran) with `MarkdownSplitter` for intelligent chunking that respects markdown structure:
+
+- **Headers**: Chunks break at header boundaries
+- **Code blocks**: Code blocks are kept intact
+- **Horizontal rules**: Natural section breaks
+- **Configurable size**: Control chunk size and overlap
+
+### Configuration
+
+```ruby
+# Global configuration
+HTM.configure do |config|
+  config.chunk_size = 1024    # Characters per chunk (default: 1024)
+  config.chunk_overlap = 64   # Overlap between chunks (default: 64)
+end
+
+# Or via environment variables
+# HTM_CHUNK_SIZE=512
+# HTM_CHUNK_OVERLAP=50
+```
+
+### Per-Loader Configuration
+
+```ruby
+loader = HTM::Loaders::MarkdownLoader.new(
+  htm,
+  chunk_size: 512,
+  chunk_overlap: 50
+)
+loader.load("docs/guide.md")
+```
+
+## Re-Sync Behavior
+
+The file loading system tracks file modification times for efficient re-syncing:
+
+1. **First load**: Creates FileSource record and chunks
+2. **Subsequent loads**: Compares mtime, skips unchanged files
+3. **Changed files**: Re-chunks and updates nodes
+4. **Force reload**: Bypasses mtime check
+
+```ruby
+# First load - creates chunks
+htm.load_file("docs/guide.md")
+# => { skipped: false, chunks_created: 5 }
+
+# Second load - skipped (unchanged)
+htm.load_file("docs/guide.md")
+# => { skipped: true }
+
+# After editing file - re-syncs
+htm.load_file("docs/guide.md")
+# => { skipped: false, chunks_updated: 2, chunks_created: 1 }
+
+# Force reload
+htm.load_file("docs/guide.md", force: true)
+# => { skipped: false, chunks_updated: 5 }
+```
+
+## FileSource Model
+
+The `HTM::Models::FileSource` tracks loaded files:
+
+```ruby
+source = HTM::Models::FileSource.find_by(file_path: "docs/guide.md")
+
+source.file_path       # Full path to file
+source.mtime           # Last modification time
+source.needs_sync?     # Check if file changed since load
+source.chunks          # Associated nodes (ordered by position)
+source.frontmatter     # Parsed YAML frontmatter
+source.title           # Frontmatter title (convenience)
+source.author          # Frontmatter author (convenience)
+source.frontmatter_tags # Tags from frontmatter
+```
+
+## Rake Tasks
+
+HTM provides rake tasks for file management:
+
+```bash
+# Load a single file
+rake 'htm:files:load[docs/guide.md]'
+
+# Load directory
+rake 'htm:files:load_dir[docs/]'
+rake 'htm:files:load_dir[docs/,**/*.md]'
+
+# List loaded files
+rake htm:files:list
+
+# Show file details
+rake 'htm:files:info[docs/guide.md]'
+
+# Unload a file
+rake 'htm:files:unload[docs/guide.md]'
+
+# Sync all files (reload changed)
+rake htm:files:sync
+
+# Show statistics
+rake htm:files:stats
+
+# Force reload with FORCE=true
+FORCE=true rake 'htm:files:load[docs/guide.md]'
+```
+
+## Best Practices
+
+### Organize Files Logically
+
+```ruby
+# Load by category
+htm.load_directory("docs/guides/", pattern: "**/*.md")
+htm.load_directory("docs/api/", pattern: "**/*.md")
+htm.load_directory("docs/tutorials/", pattern: "**/*.md")
+```
+
+### Use Frontmatter for Metadata
+
+```markdown
+---
+title: API Authentication
+category: api
+tags:
+  - security
+  - authentication
+priority: high
+---
+```
+
+### Tune Chunk Size for Your Content
+
+```ruby
+# Smaller chunks for dense technical content
+HTM.configure { |c| c.chunk_size = 512 }
+
+# Larger chunks for narrative content
+HTM.configure { |c| c.chunk_size = 2048 }
+```
+
+### Regular Sync for Updated Content
+
+```ruby
+# Sync all loaded files periodically
+htm.sync_files  # Re-checks all FileSource records
+```
+
+## Example: Building a Knowledge Base
+
+```ruby
+require 'htm'
+
+# Initialize
+htm = HTM.new(robot_name: "Knowledge Base")
+
+# Configure chunking for technical docs
+HTM.configure do |config|
+  config.chunk_size = 768
+  config.chunk_overlap = 100
+end
+
+# Load documentation
+htm.load_directory("docs/", pattern: "**/*.md")
+htm.load_directory("README.md")
+htm.load_directory("CHANGELOG.md")
+
+# Query the knowledge base
+results = htm.recall(
+  "How do I configure authentication?",
+  strategy: :hybrid,
+  limit: 5
+)
+
+results.each do |result|
+  puts result['content']
+  puts "---"
+end
+```
+
+## Related Documentation
+
+- [Adding Memories](adding-memories.md) - Core memory operations
+- [Search Strategies](search-strategies.md) - Querying loaded content
+- [API Reference: HTM](../api/htm.md) - Complete API documentation
+- [Example: File Loading](../examples/file-loading.md) - Working example

data/docs/guides/getting-started.md

@@ -8,26 +8,47 @@ Before starting, ensure you have:
 
 1. **Ruby 3.0+** installed
 2. **PostgreSQL with TimescaleDB** (or access to a TimescaleDB cloud instance)
-3. **Ollama** installed and running (for embeddings)
+3. **LLM Provider** configured - Ollama (default for local development), OpenAI, Anthropic, Gemini, or others via RubyLLM
 4. Basic understanding of Ruby and LLMs
 
-### Installing Ollama
+### Configuring an LLM Provider
 
-HTM uses Ollama for generating vector embeddings by default:
+HTM uses RubyLLM which supports multiple providers for generating embeddings and extracting tags.
+
+**Option A: Ollama (Recommended for Local Development)**
 
 ```bash
 # Install Ollama
 curl https://ollama.ai/install.sh | sh
 
-# Pull the gpt-oss model (default for HTM)
-ollama pull gpt-oss
+# Pull required models
+ollama pull nomic-embed-text
+ollama pull gemma3:latest
 
 # Verify Ollama is running
 curl http://localhost:11434/api/version
 ```
 
+**Option B: OpenAI (Recommended for Production)**
+
+```bash
+export OPENAI_API_KEY="sk-..."
+```
+
+Configure HTM:
+```ruby
+HTM.configure do |config|
+  config.embedding.provider = :openai
+  config.embedding.model = 'text-embedding-3-small'
+end
+```
+
+**Option C: Other Providers** (Anthropic, Gemini, Azure, Bedrock, DeepSeek)
+
+Set the appropriate API key and configure HTM with your preferred provider.
+
 !!! tip
-    The gpt-oss model provides high-quality embeddings optimized for semantic search. HTM uses these embeddings to understand the meaning of your memories, not just keyword matches.
+    HTM uses vector embeddings to understand the semantic meaning of your memories, not just keyword matches. Any provider will work—choose based on your privacy, cost, and quality requirements.
 
 ## Installation
 
@@ -463,9 +484,9 @@ htm.forget(node_id, soft: false, confirm: :confirmed)
 
 ## Troubleshooting
 
-### Ollama Connection Issues
+### LLM Provider Connection Issues
 
-If you see embedding errors:
+**If using Ollama:**
 
 ```bash
 # Check Ollama is running
@@ -474,10 +495,20 @@ curl http://localhost:11434/api/version
 # If not running, start it
 ollama serve
 
-# Verify the model is available
+# Verify the models are available
 ollama list
 ```
 
+**If using cloud providers:**
+
+```bash
+# Verify API key is set
+echo $OPENAI_API_KEY      # or ANTHROPIC_API_KEY, GEMINI_API_KEY, etc.
+
+# Test connectivity
+curl https://api.openai.com/v1/models -H "Authorization: Bearer $OPENAI_API_KEY"
+```
+
 ### Database Connection Issues
 
 ```ruby

data/docs/guides/index.md

@@ -32,7 +32,7 @@ Learn how to work with HTM's memory system effectively.
 
 Dive deeper into HTM's powerful capabilities.
 
-- [**Multi-Robot Usage**](multi-robot.md) - Building hive mind systems with multiple robots
+- [**Multi-Robot Usage**](../robots/multi-robot.md) - Building hive mind systems with multiple robots
 - [**Search Strategies**](search-strategies.md) - Vector, full-text, and hybrid search
 - [**Context Assembly**](context-assembly.md) - Creating optimized context for LLMs
 
@@ -62,7 +62,7 @@ We recommend the following progression:
    - [Search Strategies](search-strategies.md) - Optimize retrieval
 
 4. **Advanced Topics**: Multi-Robot Systems
-   - [Multi-Robot Usage](multi-robot.md) - Build collaborative systems
+   - [Multi-Robot Usage](../robots/multi-robot.md) - Build collaborative systems
 
 ## Quick Reference
 
@@ -73,7 +73,7 @@ We recommend the following progression:
 - **Search for memories**: See [Recalling Memories](recalling-memories.md#basic-recall)
 - **Create LLM context**: See [Context Assembly](context-assembly.md#basic-usage)
 - **Monitor memory usage**: See [Working Memory](working-memory.md#monitoring-utilization)
-- **Multi-robot setup**: See [Multi-Robot Usage](multi-robot.md#setting-up-multiple-robots)
+- **Multi-robot setup**: See [Multi-Robot Usage](../robots/multi-robot.md#setting-up-multiple-robots)
 - **Use with Claude/AIA**: See [MCP Server](mcp-server.md#client-configuration)
 
 ### Memory Types

data/docs/guides/mcp-server.md

@@ -30,11 +30,19 @@ Before using the MCP server, ensure you have:
    htm_mcp setup
    ```
 
-3. **Ollama running** (for embeddings and tag extraction)
+3. **LLM provider configured** (for embeddings and tag extraction)
+
+   **Option A: Ollama (default for local development)**
    ```bash
    ollama serve
    ollama pull nomic-embed-text
-   ollama pull llama3
+   ollama pull gemma3:latest
+   ```
+
+   **Option B: Cloud providers** (OpenAI, Anthropic, etc.)
+   ```bash
+   export OPENAI_API_KEY="sk-..."
+   # Configure via HTM.configure or environment variables
    ```
 
 ## Starting the Server
@@ -49,7 +57,7 @@ The server logs to STDERR to avoid corrupting the JSON-RPC protocol on STDOUT.
 
 ## CLI Commands
 
-The `htm_mcp` executable includes management commands for database setup and diagnostics:
+The `htm_mcp` executable includes management commands for database setup, diagnostics, and rake task execution:
 
 | Command | Description |
 |---------|-------------|
@@ -61,6 +69,9 @@ The `htm_mcp` executable includes management commands for database setup and dia
 | `htm_mcp stats` | Show memory statistics (nodes, tags, robots, database size) |
 | `htm_mcp version` | Show HTM version |
 | `htm_mcp help` | Show help with all environment variables |
+| `htm_mcp rake <task>` | Run any HTM rake task |
+| `htm_mcp rake -T` | List all available HTM rake tasks |
+| `htm_mcp rake -T <pattern>` | List HTM rake tasks matching pattern |
 
 ### First-Time Setup
 
@@ -93,6 +104,72 @@ Migration Status
   3 applied, 1 pending
 ```
 
+### Rake Task Passthrough
+
+The `htm_mcp rake` command allows you to run any HTM rake task directly through the MCP CLI. This is useful when working with HTM without a full Rails/Rake environment.
+
+**List all available tasks:**
+
+```bash
+$ htm_mcp rake -T
+# or
+$ htm_mcp rake --tasks
+
+HTM Rake Tasks
+================================================================================
+htm:db:console              # Open psql console to database
+htm:db:create               # Create the database if it doesn't exist
+htm:db:drop                 # Drop all HTM tables (WARNING: destructive!)
+htm:db:info                 # Show database information
+htm:db:migrate              # Run pending database migrations
+htm:db:purge_all            # Permanently delete all soft-deleted records
+...
+```
+
+**Filter tasks by pattern** (like standard `rake -T`):
+
+```bash
+$ htm_mcp rake -T htm:jobs
+
+HTM Rake Tasks
+================================================================================
+htm:jobs:process_all        # Process all pending jobs (embeddings, tags, propositions)
+htm:jobs:process_embeddings # Process pending embedding jobs
+htm:jobs:process_propositions # Process pending proposition extraction jobs
+htm:jobs:process_tags       # Process pending tag extraction jobs
+htm:jobs:stats              # Show job processing statistics
+
+$ htm_mcp rake -T db:rebuild
+
+HTM Rake Tasks
+================================================================================
+htm:db:rebuild:embeddings   # Clear and regenerate all embeddings
+htm:db:rebuild:propositions # Extract propositions from all non-proposition nodes
+```
+
+**Run specific tasks:**
+
+```bash
+# Database tasks
+$ htm_mcp rake htm:db:stats
+$ htm_mcp rake htm:db:verify
+$ htm_mcp rake htm:db:purge_all
+
+# Job processing tasks
+$ htm_mcp rake htm:jobs:process_all
+$ htm_mcp rake htm:jobs:process_embeddings
+
+# Tag tasks
+$ htm_mcp rake htm:tags:tree
+$ htm_mcp rake 'htm:tags:tree[database]'  # With argument
+
+# File tasks
+$ htm_mcp rake htm:files:list
+$ htm_mcp rake htm:files:sync
+```
+
+**Note:** Tasks requiring arguments use the standard rake syntax with brackets quoted for shell safety: `htm_mcp rake 'htm:files:load[path/to/file.md]'`
+
 ## Tools Reference
 
 ### SetRobotTool
@@ -681,8 +758,8 @@ Memory statistics as JSON.
   "current_robot": "my-assistant",
   "robot_id": 5,
   "robot_initialized": true,
-  "embedding_provider": "ollama",
-  "embedding_model": "nomic-embed-text"
+  "embedding_provider": "ollama",  // or "openai", "gemini", etc.
+  "embedding_model": "nomic-embed-text"  // provider-specific model
 }
 ```
 
@@ -986,12 +1063,18 @@ psql htm_development -c "CREATE EXTENSION IF NOT EXISTS pg_trgm;"
 
 ### Embedding/Tag Errors
 
-**Error: `Connection refused` (Ollama)**
+**Error: `Connection refused` (when using Ollama)**
 1. Start Ollama: `ollama serve`
 2. Pull required models:
    ```bash
    ollama pull nomic-embed-text
-   ollama pull llama3
+   ollama pull gemma3:latest
+   ```
+
+**Error: `API key invalid` (when using cloud providers)**
+1. Verify the API key is set:
+   ```bash
+   echo $OPENAI_API_KEY  # or ANTHROPIC_API_KEY, GEMINI_API_KEY
    ```
 
 ### Debugging
@@ -1026,19 +1109,23 @@ Run `htm_mcp help` for a complete list. Key variables:
 
 ### LLM Providers
 
+HTM uses RubyLLM which supports multiple providers. Defaults to Ollama for local development.
+
 | Variable | Description | Default |
 |----------|-------------|---------|
-| `HTM_EMBEDDING_PROVIDER` | Embedding provider | `ollama` |
-| `HTM_EMBEDDING_MODEL` | Embedding model | `nomic-embed-text:latest` |
+| `HTM_EMBEDDING_PROVIDER` | Embedding provider (`ollama`, `openai`, `gemini`, etc.) | `ollama` |
+| `HTM_EMBEDDING_MODEL` | Embedding model (provider-specific) | `nomic-embed-text` |
 | `HTM_TAG_PROVIDER` | Tag extraction provider | `ollama` |
 | `HTM_TAG_MODEL` | Tag model | `gemma3:latest` |
-| `HTM_OLLAMA_URL` | Ollama server URL | `http://localhost:11434` |
+| `HTM_OLLAMA_URL` | Ollama server URL (if using Ollama) | `http://localhost:11434` |
 
-### Other Providers (set API keys as needed)
+### Cloud Provider API Keys
 
 | Variable | Description |
 |----------|-------------|
-| `HTM_OPENAI_API_KEY` | OpenAI API key |
+| `OPENAI_API_KEY` | OpenAI API key |
+| `ANTHROPIC_API_KEY` | Anthropic API key |
+| `GEMINI_API_KEY` | Google Gemini API key |
 | `HTM_ANTHROPIC_API_KEY` | Anthropic API key |
 | `HTM_GEMINI_API_KEY` | Google Gemini API key |
 | `HTM_AZURE_API_KEY` | Azure OpenAI API key |
@@ -1049,4 +1136,4 @@ Run `htm_mcp help` for a complete list. Key variables:
 - [Getting Started](getting-started.md) - HTM basics
 - [Adding Memories](adding-memories.md) - Learn about tags and metadata
 - [Recalling Memories](recalling-memories.md) - Search strategies
-- [Multi-Robot Systems](multi-robot.md) - Working with multiple robots
+- [Multi-Robot Systems](../robots/multi-robot.md) - Working with multiple robots