htm 0.0.2 → 0.0.10

data/docs/development/schema.md

@@ -37,13 +37,13 @@ For detailed table definitions, columns, indexes, and constraints, see the auto-
 | [robots](../database/public.robots.md) | Registry of all LLM robots using the HTM system | Stores robot metadata and activity tracking |
 | [nodes](../database/public.nodes.md) | Core memory storage for conversation messages and context | Vector embeddings, full-text search, deduplication |
 | [tags](../database/public.tags.md) | Unique hierarchical tag names for categorization | Colon-separated namespaces (e.g., `ai:llm:embeddings`) |
-| [working_memories](../database/public.working_memories.md) | Per-robot working memory state | Optional persistence for token-limited context |
+| file_sources | Source file metadata for loaded documents | Path, mtime, frontmatter, sync tracking |
 
 ### Join Tables
 
 | Table | Description | Details |
 |-------|-------------|---------|
-| [robot_nodes](../database/public.robot_nodes.md) | Links robots to nodes (many-to-many) | Enables "hive mind" shared memory architecture |
+| [robot_nodes](../database/public.robot_nodes.md) | Links robots to nodes (many-to-many) | Enables "hive mind" shared memory; includes `working_memory` boolean for per-robot working memory state |
 | [node_tags](../database/public.node_tags.md) | Links nodes to tags (many-to-many) | Flexible multi-tag categorization |
 
 ### System Tables
@@ -65,6 +65,40 @@ Content deduplication is enforced via SHA-256 hashing in the `nodes` table:
 3. A new `robot_nodes` association is created (or updated if it already exists)
 4. This ensures identical memories are stored once but can be "remembered" by multiple robots
 
+### JSONB Metadata
+
+The `nodes` table includes a `metadata` JSONB column for flexible key-value storage:
+
+| Column | Type | Default | Description |
+|--------|------|---------|-------------|
+| `metadata` | jsonb | `{}` | Arbitrary key-value data |
+
+**Features:**
+- Stores any valid JSON data (strings, numbers, booleans, arrays, objects)
+- GIN index (`idx_nodes_metadata`) for efficient containment queries
+- Queried using PostgreSQL's `@>` containment operator
+
+**Query examples:**
+```sql
+-- Find nodes with specific metadata
+SELECT * FROM nodes WHERE metadata @> '{"priority": "high"}'::jsonb;
+
+-- Find nodes with nested metadata
+SELECT * FROM nodes WHERE metadata @> '{"user": {"role": "admin"}}'::jsonb;
+
+-- Find nodes with multiple conditions
+SELECT * FROM nodes WHERE metadata @> '{"environment": "production", "version": 2}'::jsonb;
+```
+
+**Ruby usage:**
+```ruby
+# Store with metadata
+htm.remember("API config", metadata: { environment: "production", version: 2 })
+
+# Recall filtering by metadata
+htm.recall("config", metadata: { environment: "production" })
+```
+
 ### Hierarchical Tags
 
 Tags use colon-separated hierarchies for organization:
@@ -78,6 +112,35 @@ SELECT * FROM tags WHERE name LIKE 'database:%';  -- All database-related tags
 SELECT * FROM tags WHERE name LIKE 'ai:llm:%';    -- All LLM-related tags
 ```
 
+### File Source Tracking
+
+The `file_sources` table tracks loaded documents for re-sync support:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | bigint | Primary key |
+| `file_path` | text | Absolute path to the source file |
+| `file_hash` | varchar(64) | SHA-256 hash of file contents |
+| `mtime` | timestamptz | File modification time for change detection |
+| `file_size` | integer | File size in bytes |
+| `frontmatter` | jsonb | Parsed YAML frontmatter metadata |
+| `last_synced_at` | timestamptz | When file was last synced |
+| `created_at` | timestamptz | When source was first loaded |
+| `updated_at` | timestamptz | When source was last updated |
+
+Nodes loaded from files have:
+- `source_id` - Foreign key to file_sources (nullable, ON DELETE SET NULL)
+- `chunk_position` - Integer position within the file (0-indexed)
+
+Query nodes from a file:
+```sql
+SELECT n.*
+FROM nodes n
+JOIN file_sources fs ON n.source_id = fs.id
+WHERE fs.file_path = '/path/to/file.md'
+ORDER BY n.chunk_position;
+```
+
 ### Remember Tracking
 
 The `robot_nodes` table tracks per-robot remember metadata:
@@ -278,6 +341,8 @@ The schema is managed through ActiveRecord migrations located in `db/migrate/`:
 1. `20250101000001_create_robots.rb` - Creates robots table
 2. `20250101000002_create_nodes.rb` - Creates nodes table with all indexes
 3. `20250101000005_create_tags.rb` - Creates tags and nodes_tags tables
+4. `20251128000002_create_file_sources.rb` - Creates file_sources table for document tracking
+5. `20251128000003_add_source_to_nodes.rb` - Adds source_id and chunk_position to nodes
 
 To apply migrations:
 ```bash

data/docs/guides/adding-memories.md

@@ -7,7 +7,7 @@ This guide covers everything you need to know about storing information in HTM e
 The primary method for adding memories is `remember`:
 
 ```ruby
-node_id = htm.remember(content, tags: [])
+node_id = htm.remember(content, tags: [], metadata: {})
 ```
 
 **Parameters:**
@@ -16,6 +16,7 @@ node_id = htm.remember(content, tags: [])
 |-----------|------|---------|-------------|
 | `content` | String | *required* | The information to remember |
 | `tags` | Array\<String\> | `[]` | Manual tags to assign (in addition to auto-extracted tags) |
+| `metadata` | Hash | `{}` | Arbitrary key-value metadata stored as JSONB |
 
 The method returns the database ID of the created node.
 
@@ -161,6 +162,86 @@ htm.remember("We're using Redis for session caching with a 24-hour TTL")
 # Background job might extract: ["database:redis", "caching:session", "performance"]
 ```
 
+## Using Metadata
+
+Metadata provides flexible key-value storage for arbitrary data that doesn't fit into tags. Unlike tags (which are for hierarchical categorization), metadata is for structured data like version numbers, priorities, source systems, or any custom attributes.
+
+### Basic Metadata Usage
+
+```ruby
+# Store with metadata
+htm.remember(
+  "User prefers dark mode",
+  metadata: { category: "preference", priority: "high" }
+)
+
+# Multiple metadata fields
+htm.remember(
+  "API endpoint changed from /v1 to /v2",
+  metadata: {
+    category: "migration",
+    version: 2,
+    breaking_change: true,
+    affected_services: ["web", "mobile"]
+  }
+)
+```
+
+### Metadata vs Tags
+
+| Feature | Tags | Metadata |
+|---------|------|----------|
+| Structure | Hierarchical (colon-separated) | Flat key-value pairs |
+| Type | String only | Any JSON type (string, number, boolean, array, object) |
+| Search | Prefix matching (`LIKE 'ai:%'`) | JSONB containment (`@>`) |
+| Purpose | Categorization & navigation | Arbitrary attributes & filtering |
+| Auto-extraction | Yes (via LLM) | No (always explicit) |
+
+### Common Metadata Patterns
+
+```ruby
+# Version tracking
+htm.remember("API uses OAuth 2.0", metadata: { version: 3, deprecated: false })
+
+# Source tracking
+htm.remember("Error rate is 0.1%", metadata: { source: "monitoring", dashboard: "errors" })
+
+# Priority/importance
+htm.remember("Deploy to prod on Fridays is forbidden", metadata: { priority: "critical" })
+
+# Environment-specific
+htm.remember("Database connection limit is 100", metadata: { environment: "production" })
+
+# Combining with tags
+htm.remember(
+  "Use connection pooling for better performance",
+  tags: ["database:postgresql", "performance"],
+  metadata: { priority: "high", reviewed: true, author: "dba-team" }
+)
+```
+
+### Querying by Metadata
+
+Use the `metadata` parameter in `recall()` to filter by metadata:
+
+```ruby
+# Find all high-priority items
+htm.recall("settings", metadata: { priority: "high" })
+
+# Find production-specific configurations
+htm.recall("database", metadata: { environment: "production" })
+
+# Combine with other filters
+htm.recall(
+  "API changes",
+  timeframe: "last month",
+  metadata: { breaking_change: true },
+  strategy: :hybrid
+)
+```
+
+Metadata filtering uses PostgreSQL's JSONB containment operator (`@>`), which means the node's metadata must contain all the key-value pairs you specify.
+
 ## Content Deduplication
 
 HTM automatically deduplicates content across all robots using SHA-256 hashing.
@@ -355,13 +436,14 @@ htm.remember(
   "Alice Thompson is a senior software engineer specializing in distributed systems"
 )
 
-# Add a preference
+# Add a preference with metadata
 htm.remember(
-  "Alice prefers Vim for editing and tmux for terminal management"
+  "Alice prefers Vim for editing and tmux for terminal management",
+  metadata: { category: "preference", source: "user-interview" }
 )
 
-# Add a decision with context
-htm.remember(<<~DECISION, tags: ["architecture", "messaging"])
+# Add a decision with context, tags, and metadata
+htm.remember(<<~DECISION, tags: ["architecture", "messaging"], metadata: { priority: "high", approved: true, version: 1 })
   Decision: Use RabbitMQ for async job processing
 
   Rationale:
@@ -374,8 +456,8 @@ htm.remember(<<~DECISION, tags: ["architecture", "messaging"])
   - Kafka (overkill for our scale)
 DECISION
 
-# Add implementation code
-htm.remember(<<~RUBY, tags: ["code:ruby", "messaging:rabbitmq"])
+# Add implementation code with metadata
+htm.remember(<<~RUBY, tags: ["code:ruby", "messaging:rabbitmq"], metadata: { language: "ruby", tested: true })
   require 'bunny'
 
   connection = Bunny.new(ENV['RABBITMQ_URL'])
@@ -387,4 +469,8 @@ RUBY
 
 puts "Added memories with relationships and rich metadata"
 puts "Stats: #{HTM::Models::Node.count} total nodes"
+
+# Query by metadata
+high_priority = htm.recall("decisions", metadata: { priority: "high" })
+puts "High priority decisions: #{high_priority.count}"
 ```

data/docs/guides/recalling-memories.md

@@ -390,9 +390,43 @@ results = htm.recall(topic: "JWT authentication", strategy: :fulltext)
 results = htm.recall(topic: "user validation methods", strategy: :vector)
 ```
 
+## Filtering by Metadata
+
+HTM supports metadata filtering directly in the `recall()` method. This is more efficient than post-filtering because the database does the work.
+
+```ruby
+# Filter by single metadata field
+memories = htm.recall(
+  topic: "user settings",
+  metadata: { category: "preference" }
+)
+# => Returns only nodes with metadata containing { category: "preference" }
+
+# Filter by multiple metadata fields
+memories = htm.recall(
+  topic: "API configuration",
+  metadata: { environment: "production", version: 2 }
+)
+# => Returns nodes with BOTH environment: "production" AND version: 2
+
+# Combine with other filters
+memories = htm.recall(
+  topic: "database changes",
+  timeframe: "last month",
+  strategy: :hybrid,
+  metadata: { breaking_change: true },
+  limit: 10
+)
+```
+
+Metadata filtering uses PostgreSQL's JSONB containment operator (`@>`), which means:
+- The node's metadata must contain ALL the key-value pairs you specify
+- The node's metadata can have additional fields (they're ignored)
+- Nested objects work: `metadata: { user: { role: "admin" } }` matches `{ user: { role: "admin", name: "..." } }`
+
 ## Combining Search with Filters
 
-While `recall` handles timeframes and topics, you can filter results further:
+While `recall` handles timeframes, topics, and metadata, you can filter results further:
 
 ```ruby
 # Recall memories
@@ -622,6 +656,7 @@ memory = {
   'created_at' => "2024-01-15 10:30:00", # Timestamp
   'robot_id' => "uuid...",               # Which robot added it
   'token_count' => 150,                  # Token count
+  'metadata' => { 'priority' => 'high', 'version' => 2 },  # JSONB metadata
   'similarity' => 0.85                   # Similarity score (vector/hybrid)
   # or 'rank' for fulltext
 }

data/examples/README.md

@@ -0,0 +1,280 @@
+# HTM Examples
+
+This directory contains example applications demonstrating various ways to use the HTM (Hierarchical Temporary Memory) gem.
+
+## Prerequisites
+
+All examples require:
+
+1. **PostgreSQL Database** with pgvector extension:
+   ```bash
+   export HTM_DBURL="postgresql://user@localhost:5432/htm_development"
+   ```
+
+2. **Ollama** (recommended for local LLM):
+   ```bash
+   ollama pull nomic-embed-text  # For embeddings
+   ollama pull gemma3            # For tag extraction
+   ```
+
+3. **Ruby Dependencies**:
+   ```bash
+   bundle install
+   ```
+
+---
+
+## Standalone Scripts
+
+### basic_usage.rb
+
+**Getting started with HTM fundamentals.**
+
+Demonstrates the core API: configuring HTM, registering a robot, and using the three primary methods (`remember`, `recall`, `forget`).
+
+```bash
+ruby examples/basic_usage.rb
+```
+
+**Features:**
+- HTM configuration with Ollama provider
+- Robot initialization
+- Storing memories with `remember()`
+- Retrieving memories with `recall()` using timeframes
+- Understanding async embedding/tag generation
+
+---
+
+### custom_llm_configuration.rb
+
+**Flexible LLM integration patterns.**
+
+Shows how to configure HTM with custom embedding and tag generation methods, supporting multiple LLM providers or custom infrastructure.
+
+```bash
+ruby examples/custom_llm_configuration.rb
+```
+
+**Features:**
+- Default configuration (RubyLLM + Ollama)
+- Custom lambdas for embedding generation
+- Custom lambdas for tag extraction
+- Service object integration pattern
+- Mixed configuration (custom embedding + default tags)
+- Provider settings (OpenAI, Anthropic, Gemini, etc.)
+
+---
+
+### file_loader_usage.rb
+
+**Loading documents into long-term memory.**
+
+Demonstrates loading markdown files with automatic paragraph chunking, YAML frontmatter extraction, and source tracking for re-sync.
+
+```bash
+ruby examples/file_loader_usage.rb
+```
+
+**Features:**
+- Single file loading with `load_file()`
+- Directory loading with glob patterns via `load_directory()`
+- YAML frontmatter extraction (title, author, tags)
+- Querying nodes from a specific file
+- Re-sync behavior (skip unchanged files)
+- Force reload option
+- Unloading files with `unload_file()`
+
+---
+
+### timeframe_demo.rb
+
+**Flexible time-based filtering for recall.**
+
+Comprehensive demonstration of all timeframe options supported by `recall()`, including natural language parsing.
+
+```bash
+ruby examples/timeframe_demo.rb
+```
+
+**Features:**
+- No filter (`nil`)
+- Date/DateTime/Time objects (entire day)
+- Range for precise time windows
+- Natural language strings ("yesterday", "last week", "few days ago")
+- Weekend expressions ("last weekend", "2 weekends ago")
+- Automatic extraction (`:auto`) from query text
+- Multiple time windows (array of ranges)
+
+---
+
+## Application Examples
+
+### example_app/
+
+**Full-featured HTM demonstration with RubyLLM integration.**
+
+A standalone application showing complete HTM workflow with database connection, Ollama integration, memory operations, and multiple search strategies.
+
+```bash
+ruby examples/example_app/app.rb
+```
+
+**Features:**
+- Database connection verification
+- RubyLLM configuration for embeddings and tags
+- Async embedding/tag generation with wait
+- Comparison of search strategies (:fulltext, :vector, :hybrid)
+- Detailed output of generated tags and embeddings
+
+---
+
+### sinatra_app/
+
+**Web application with Sidekiq background processing.**
+
+A Sinatra-based web application demonstrating HTM in a multi-user web context with async job processing.
+
+```bash
+cd examples/sinatra_app
+bundle install
+bundle exec ruby app.rb
+```
+
+**Features:**
+- Sidekiq integration for background jobs
+- Session-based robot identification
+- RESTful API endpoints:
+  - `POST /api/remember` - Store information
+  - `GET /api/recall` - Search memories with timeframe filtering
+  - `GET /api/stats` - Memory statistics
+  - `GET /api/tags` - Tag tree structure
+  - `GET /api/health` - Health check
+- Interactive HTML UI with hybrid search scoring display
+- Tag tree visualization
+
+**Environment Variables:**
+- `HTM_DBURL` - PostgreSQL connection (required)
+- `REDIS_URL` - Redis for Sidekiq (default: redis://localhost:6379/0)
+- `SESSION_SECRET` - Session encryption key
+
+---
+
+### cli_app/
+
+**Interactive command-line application.**
+
+A REPL-style CLI demonstrating synchronous job execution with the `:inline` backend, ideal for CLI tools and scripts.
+
+```bash
+ruby examples/cli_app/htm_cli.rb
+```
+
+**Commands:**
+- `remember <text>` - Store information (waits for embedding/tags)
+- `recall <topic>` - Hybrid search with LLM-powered response generation
+- `tags [prefix]` - List all tags with linked node content
+- `stats` - Memory statistics
+- `help` - Show help
+- `exit` - Quit
+
+**Features:**
+- Synchronous job execution (`:inline` backend)
+- Real-time progress feedback
+- Tag extraction visibility during search
+- Hybrid search with scoring (similarity, tag_boost, combined)
+- RubyLLM chat integration for context-aware responses
+- Response storage in long-term memory
+
+See [cli_app/README.md](cli_app/README.md) for detailed documentation.
+
+---
+
+### robot_groups/
+
+**Multi-robot coordination with shared working memory.**
+
+Demonstrates high-availability patterns with shared working memory, failover, and real-time synchronization via PostgreSQL LISTEN/NOTIFY.
+
+#### same_process.rb
+
+Single-process demonstration of robot groups:
+
+```bash
+ruby examples/robot_groups/same_process.rb
+```
+
+**Scenarios demonstrated:**
+1. Creating a group with primary + standby robots
+2. Adding shared memories
+3. Verifying synchronization
+4. Simulating failover (primary dies, standby takes over)
+5. Verifying standby has full context
+6. Dynamic scaling (adding new robots)
+7. Collaborative memory (multiple robots adding)
+8. Real-time sync via PostgreSQL LISTEN/NOTIFY
+
+#### multi_process.rb
+
+Cross-process demonstration with separate Ruby processes:
+
+```bash
+ruby examples/robot_groups/multi_process.rb
+```
+
+**Scenarios demonstrated:**
+1. Spawning robot worker processes
+2. Cross-process memory sharing
+3. Collaborative memory updates
+4. Failover when a process dies
+5. Dynamic scaling (adding new processes)
+
+**Key concepts:**
+- **Shared Working Memory**: Multiple robots share context via `robot_nodes` table
+- **Active/Passive Roles**: Active robots participate; passive robots maintain warm standby
+- **Failover**: Instant takeover with full context already loaded
+- **Real-time Sync**: PostgreSQL LISTEN/NOTIFY for in-memory cache coordination
+
+---
+
+## Directory Structure
+
+```
+examples/
+├── README.md                      # This file
+├── basic_usage.rb                 # Core API demonstration
+├── custom_llm_configuration.rb    # LLM integration patterns
+├── file_loader_usage.rb           # Document loading
+├── timeframe_demo.rb              # Time-based filtering
+├── example_app/
+│   ├── app.rb                     # Full-featured demo app
+│   └── Rakefile
+├── sinatra_app/
+│   ├── app.rb                     # Sinatra web application
+│   ├── Gemfile
+│   └── Gemfile.lock
+├── cli_app/
+│   ├── htm_cli.rb                 # Interactive CLI
+│   └── README.md                  # Detailed CLI documentation
+└── robot_groups/
+    ├── same_process.rb            # Single-process robot groups
+    ├── multi_process.rb           # Multi-process coordination
+    ├── robot_worker.rb            # Worker process for multi_process.rb
+    └── lib/
+        ├── robot_group.rb         # RobotGroup coordination class
+        └── working_memory_channel.rb  # PostgreSQL pub/sub
+```
+
+---
+
+## Choosing the Right Example
+
+| Use Case | Example |
+|----------|---------|
+| Learning HTM basics | `basic_usage.rb` |
+| Custom LLM integration | `custom_llm_configuration.rb` |
+| Loading documents/files | `file_loader_usage.rb` |
+| Time-based queries | `timeframe_demo.rb` |
+| Web application | `sinatra_app/` |
+| CLI tool | `cli_app/` |
+| Multi-robot coordination | `robot_groups/` |
+| High availability | `robot_groups/` |

data/examples/cli_app/htm_cli.rb

@@ -19,6 +19,20 @@
 
 require_relative '../../lib/htm'
 require 'io/console'
+require 'ruby_llm'
+
+PROVIDER = :ollama
+MODEL    = 'gpt-oss:latest'
+
+# Configure RubyLLM for Ollama provider (same pattern as HTM uses)
+RubyLLM.configure do |config|
+  ollama_url = ENV.fetch('OLLAMA_URL', 'http://localhost:11434')
+  ollama_api_base = ollama_url.end_with?('/v1') ? ollama_url : "#{ollama_url}/v1"
+  config.ollama_api_base = ollama_api_base
+end
+
+# Create chat with Ollama model - use assume_model_exists to bypass registry check
+# AI = RubyLLM.chat(model: MODEL, provider: PROVIDER, assume_model_exists: true)
 
 class HTMCli
   def initialize
@@ -43,6 +57,9 @@ class HTMCli
       end
     end
 
+    # Initialize RubyLLM chat for context-aware responses
+    @chat = RubyLLM.chat(model: MODEL, provider: PROVIDER)
+
     # Initialize HTM instance
     @htm = HTM.new(robot_name: "cli_assistant")
   end
@@ -146,12 +163,27 @@ class HTMCli
     puts "\nSearching for: \"#{topic}\""
     puts "Strategy: hybrid (vector + fulltext + tags)"
 
-    # Show which tags match the query
-    matching_tags = @htm.long_term_memory.find_query_matching_tags(topic)
-    if matching_tags.any?
-      puts "Matching tags: #{matching_tags.join(', ')}"
+    # Show tags extracted from query and which ones matched
+    tag_result = @htm.long_term_memory.find_query_matching_tags(topic, include_extracted: true)
+
+    if tag_result[:extracted].any?
+      puts "Extracted tags: #{tag_result[:extracted].join(', ')}"
+
+      # Show what was actually searched (exact + prefixes)
+      searched = tag_result[:extracted].dup
+      tag_result[:extracted].each do |tag|
+        levels = tag.split(':')
+        (1...levels.size).each { |i| searched << levels[0, i].join(':') }
+      end
+      puts "Searched for:   #{searched.uniq.join(', ')}"
+
+      if tag_result[:matched].any?
+        puts "Matched in DB:  #{tag_result[:matched].join(', ')}"
+      else
+        puts "Matched in DB:  (none)"
+      end
     else
-      puts "Matching tags: (none found)"
+      puts "Extracted tags: (none)"
     end
 
     start_time = Time.now
@@ -192,6 +224,34 @@ class HTMCli
         puts "   Tags: (none)"
       end
     end
+
+    # Build LLM prompt with context from retrieved memories
+    context_content = memories.map { |m| m['content'] }.join("\n\n")
+
+    llm_prompt = <<~PROMPT
+      #{topic}
+      Your response should highlight information also found in the
+      following context:
+      <CONTEXT>
+      #{context_content}
+      </CONTEXT>
+    PROMPT
+
+    puts "\n" + "=" * 60
+    puts "Generating response for this prompt..."
+    puts llm_prompt
+    puts "=" * 60
+
+    begin
+      response = @chat.ask(llm_prompt)
+      puts "\n#{response.content}"
+
+      # Remember the LLM response in long-term memory
+      node_id = @htm.remember(response.content)
+      puts "\n[✓] Response stored as node #{node_id}"
+    rescue StandardError => e
+      puts "[✗] LLM Error: #{e.message}"
+    end
   end
 
   def handle_tags(filter = nil)