htm 0.0.18 → 0.0.20

data/docs/guides/propositions.md

@@ -0,0 +1,264 @@
+# Propositions: Atomic Fact Extraction
+
+Proposition extraction breaks complex text into atomic, self-contained factual statements. This improves RAG retrieval accuracy by storing granular facts that can be matched more precisely.
+
+## Overview
+
+When proposition extraction is enabled, HTM:
+
+1. Stores the original content as a node
+2. Extracts atomic propositions from the content
+3. Creates independent nodes for each proposition
+4. Each proposition gets its own embedding and tags
+
+## What is a Proposition?
+
+A proposition is an atomic factual statement that:
+
+- Expresses a **single fact** or claim
+- Is **understandable without context**
+- Uses **full names**, not pronouns
+- Includes relevant **dates, times, and qualifiers**
+- Contains **one subject-predicate relationship**
+
+### Example
+
+**Original text:**
+> "In 1969, Neil Armstrong became the first person to walk on the Moon during Apollo 11."
+
+**Extracted propositions:**
+- "Neil Armstrong was an astronaut."
+- "Neil Armstrong walked on the Moon in 1969."
+- "Neil Armstrong was the first person to walk on the Moon."
+- "Neil Armstrong walked on the Moon during the Apollo 11 mission."
+- "The Apollo 11 mission occurred in 1969."
+
+## Configuration
+
+### Enable Proposition Extraction
+
+```ruby
+# Via configuration block
+HTM.configure do |config|
+  config.extract_propositions = true
+  config.proposition_provider = :ollama  # or :openai, :anthropic, etc.
+  config.proposition_model = 'gemma3:latest'
+end
+
+# Or via environment variable
+# HTM_EXTRACT_PROPOSITIONS=true
+```
+
+### Provider Options
+
+Proposition extraction uses LLM chat completion. Configure your preferred provider:
+
+| Provider | Model Examples |
+|----------|----------------|
+| `:ollama` (default) | `gemma3:latest`, `llama3`, `mistral` |
+| `:openai` | `gpt-4o-mini`, `gpt-4o` |
+| `:anthropic` | `claude-3-haiku-20240307` |
+| `:gemini` | `gemini-1.5-flash` |
+
+```ruby
+HTM.configure do |config|
+  config.extract_propositions = true
+
+  # Use OpenAI for higher quality extraction
+  config.proposition_provider = :openai
+  config.proposition_model = 'gpt-4o-mini'
+end
+```
+
+## How It Works
+
+### Workflow
+
+```mermaid
+sequenceDiagram
+    participant User
+    participant HTM
+    participant PropositionService
+    participant LLM
+    participant Database
+
+    User->>HTM: remember("Complex text...")
+    HTM->>Database: Save original node
+    HTM->>PropositionService: Extract propositions
+    PropositionService->>LLM: Parse into atomic facts
+    LLM-->>PropositionService: ["Prop 1", "Prop 2", ...]
+    loop For each proposition
+        PropositionService->>Database: Create proposition node
+    end
+    HTM-->>User: node_id
+```
+
+### Proposition Nodes
+
+Proposition nodes are stored with special metadata:
+
+```ruby
+{
+  "is_proposition" => true,
+  "source_node_id" => 123  # ID of the original node
+}
+```
+
+This metadata allows you to:
+- Identify proposition nodes
+- Trace propositions back to source
+- Filter propositions in queries
+
+## Usage
+
+### Basic Usage
+
+```ruby
+htm = HTM.new(robot_name: "Proposition Demo")
+
+# With extraction enabled, this creates multiple nodes
+node_id = htm.remember(
+  "PostgreSQL 16 was released in September 2023 with improved query performance and new JSON features."
+)
+
+# The original node plus propositions are created:
+# - "PostgreSQL 16 was released in September 2023."
+# - "PostgreSQL 16 includes improved query performance."
+# - "PostgreSQL 16 includes new JSON features."
+```
+
+### Direct Extraction
+
+You can extract propositions without storing them:
+
+```ruby
+propositions = HTM.extract_propositions(
+  "Ruby 3.3 introduced YJIT improvements and the Prism parser."
+)
+# => [
+#   "Ruby 3.3 introduced YJIT improvements.",
+#   "Ruby 3.3 introduced the Prism parser.",
+#   "YJIT is a just-in-time compiler for Ruby.",
+#   "Prism is a parser for Ruby."
+# ]
+
+# Manually store if needed
+propositions.each { |p| htm.remember(p) }
+```
+
+### Querying Propositions
+
+```ruby
+# Find all proposition nodes
+propositions = HTM::Models::Node.where("metadata->>'is_proposition' = ?", 'true')
+
+# Find propositions from a specific source
+source_node_id = 123
+related = HTM::Models::Node.where(
+  "metadata->>'source_node_id' = ?",
+  source_node_id.to_s
+)
+
+# Include propositions in recall (default behavior)
+results = htm.recall("PostgreSQL features", strategy: :hybrid)
+```
+
+## Recursion Prevention
+
+Proposition nodes do **not** trigger further proposition extraction. This prevents infinite recursion:
+
+```ruby
+# Original node → triggers proposition extraction
+htm.remember("Complex statement about many things.")
+
+# Proposition nodes → do NOT trigger extraction
+# (metadata.is_proposition = true prevents this)
+```
+
+## Performance Considerations
+
+### Processing Time
+
+Proposition extraction adds latency:
+
+| Provider | Typical Latency |
+|----------|-----------------|
+| Ollama (local) | 1-3 seconds |
+| OpenAI | 0.5-1 second |
+| Anthropic | 0.5-1 second |
+
+### Async Processing
+
+With async job backend, extraction happens in background:
+
+```ruby
+HTM.configure do |config|
+  config.extract_propositions = true
+  config.job.backend = :thread  # or :sidekiq
+end
+
+# Returns immediately, propositions created async
+node_id = htm.remember("Complex content...")
+```
+
+### Storage Impact
+
+Proposition extraction increases storage:
+
+- Original node: 1 record
+- Propositions: 3-10 additional records (typical)
+- Each proposition gets its own embedding
+
+## Best Practices
+
+### When to Use Propositions
+
+**Good use cases:**
+- Dense factual content (Wikipedia, documentation)
+- Complex statements with multiple facts
+- Content that will be queried for specific facts
+
+**Less suitable:**
+- Simple, atomic statements
+- Conversational content
+- Content where context is critical
+
+### Quality Tuning
+
+Use a capable model for better extraction:
+
+```ruby
+# Higher quality (slower, costs more)
+config.proposition_provider = :openai
+config.proposition_model = 'gpt-4o'
+
+# Balanced (faster, local)
+config.proposition_provider = :ollama
+config.proposition_model = 'gemma3:latest'
+```
+
+### Selective Extraction
+
+Enable/disable per operation if needed:
+
+```ruby
+# Temporarily disable for specific content
+original_setting = HTM.configuration.extract_propositions
+HTM.configuration.extract_propositions = false
+htm.remember("Simple fact that doesn't need decomposition.")
+HTM.configuration.extract_propositions = original_setting
+```
+
+## Rake Tasks
+
+```bash
+# Rebuild all propositions (clears and regenerates)
+rake htm:db:rebuild:propositions
+```
+
+## Related Documentation
+
+- [Adding Memories](adding-memories.md) - Core memory operations
+- [Search Strategies](search-strategies.md) - Querying memories
+- [Tags](tags.md) - Hierarchical tagging (propositions get tags too)
+- [API Reference: PropositionService](../api/yard/HTM/PropositionService.md)

data/docs/guides/recalling-memories.md

@@ -39,7 +39,7 @@ end
   <!-- Step 2: Generate Embedding -->
   <rect x="290" y="70" width="200" height="80" fill="rgba(33, 150, 243, 0.2)" stroke="#2196F3" stroke-width="2" rx="5"/>
   <text x="390" y="95" text-anchor="middle" fill="#2196F3" font-size="14" font-weight="bold">2. Generate Embedding</text>
-  <text x="390" y="120" text-anchor="middle" fill="#B0B0B0" font-size="10">Ollama/OpenAI</text>
+  <text x="390" y="120" text-anchor="middle" fill="#B0B0B0" font-size="10">LLM Provider (RubyLLM)</text>
   <text x="390" y="135" text-anchor="middle" fill="#B0B0B0" font-size="10">[0.23, -0.57, ...]</text>
 
   <!-- Arrow 2 to 3 -->
@@ -221,7 +221,7 @@ memories = htm.recall(
 
 **How it works**:
 
-1. Converts your topic to a vector embedding via Ollama
+1. Converts your topic to a vector embedding via your configured provider (Ollama, OpenAI, etc.)
 2. Finds memories with similar embeddings using cosine similarity
 3. Returns results ordered by semantic similarity
 
@@ -865,9 +865,9 @@ htm.recall(timeframe: "...", topic: "...", limit: 100)
   .first(10)
 ```
 
-### Ollama Connection Issues
+### LLM Provider Connection Issues
 
-If vector search fails:
+If vector search fails (Ollama not running, API key invalid, etc.):
 
 ```ruby
 begin

data/docs/guides/search-strategies.md

@@ -108,9 +108,9 @@ Vector search finds memories based on semantic similarity using embeddings.
 ```
 User Query: "database optimization techniques"
       ↓
-   Ollama Embedding (gpt-oss)
+   Embedding via RubyLLM (Ollama, OpenAI, etc.)
       ↓
-  [0.234, -0.567, 0.123, ...]  ← 1536-dimensional vector
+  [0.234, -0.567, 0.123, ...]  ← Vector representation
       ↓
    PostgreSQL + pgvector
       ↓
@@ -927,7 +927,7 @@ end
 
 ```ruby
 # If vector search returns nothing:
-# 1. Check Ollama is running
+# 1. Check your LLM provider is accessible (Ollama running, API key set, etc.)
 # 2. Try broader query
 # 3. Widen timeframe
 # 4. Fall back to full-text

data/docs/guides/tags.md

@@ -0,0 +1,318 @@
+# Hierarchical Tags
+
+HTM uses a hierarchical tagging system to organize memories semantically. Tags use colon-separated namespaces (like `database:postgresql:extensions`) enabling both specific and broad queries.
+
+## Overview
+
+The tagging system provides:
+
+- **Hierarchical organization**: `category:subcategory:topic`
+- **LLM-powered extraction**: Tags auto-generated from content
+- **Ontology awareness**: New tags consider existing taxonomy
+- **Prefix queries**: Find all `database:*` tags easily
+- **Visualization**: Export as text tree, Mermaid, or SVG
+
+## Quick Start
+
+```ruby
+htm = HTM.new(robot_name: "Tag Demo")
+
+# Tags are auto-extracted from content
+htm.remember("PostgreSQL supports JSON and vector search via pgvector.")
+# Auto-tags: ["database:postgresql", "database:postgresql:json",
+#             "database:postgresql:pgvector", "search:vector"]
+
+# Or specify tags manually
+htm.remember(
+  "Redis is an in-memory data store.",
+  tags: ["database:redis", "database:nosql", "caching"]
+)
+
+# Query by tag
+results = htm.recall("database features", tags: ["database:postgresql"])
+```
+
+## Tag Format
+
+### Hierarchical Structure
+
+Tags use colon (`:`) as the hierarchy separator:
+
+```
+category:subcategory:topic
+    │         │        │
+    └─────────┴────────┴── More specific →
+```
+
+**Examples:**
+- `database:postgresql`
+- `database:postgresql:extensions`
+- `database:postgresql:extensions:pgvector`
+- `programming:ruby:gems`
+- `api:rest:authentication`
+
+### Naming Conventions
+
+- **Lowercase**: Use lowercase for consistency
+- **Singular nouns**: `database` not `databases`
+- **Hierarchical**: Most general → most specific
+- **Descriptive**: Clear, semantic meaning
+
+## Automatic Tag Extraction
+
+HTM uses LLM to automatically extract relevant tags from content:
+
+```ruby
+HTM.configure do |config|
+  config.tag.provider = :ollama  # or :openai, :anthropic, etc.
+  config.tag.model = 'gemma3:latest'
+end
+
+# Tags extracted automatically
+htm.remember("Ruby on Rails uses ActiveRecord for database access.")
+# Extracted: ["programming:ruby:rails", "database:orm:activerecord",
+#             "web:framework:rails"]
+```
+
+### Ontology Awareness
+
+The tag extractor receives existing tags to maintain consistency:
+
+```ruby
+# First memory creates initial tags
+htm.remember("PostgreSQL is a relational database.")
+# Tags: ["database:postgresql", "database:relational"]
+
+# Later memories align with existing ontology
+htm.remember("MySQL is also a relational database.")
+# Tags: ["database:mysql", "database:relational"]  # Reuses existing structure
+```
+
+### Custom Tag Extractor
+
+Provide your own tag extraction logic:
+
+```ruby
+HTM.configure do |config|
+  config.tag_extractor = lambda do |text, existing_ontology|
+    # Your custom logic here
+    # Must return Array<String>
+    ["custom:tag:one", "custom:tag:two"]
+  end
+end
+```
+
+## Manual Tag Operations
+
+### Adding Tags
+
+```ruby
+# Via remember
+htm.remember("Content here", tags: ["topic:subtopic"])
+
+# Via long-term memory directly
+htm.long_term_memory.add_tag(node_id: node.id, tag: "new:tag")
+```
+
+### Querying Tags
+
+```ruby
+# Get tags for a node
+tags = htm.long_term_memory.node_topics(node.id)
+# => ["database:postgresql", "search:vector"]
+
+# Find nodes by tag
+nodes = HTM::Models::Node.joins(:tags).where(tags: { name: "database:postgresql" })
+
+# Find by tag prefix
+nodes = HTM::Models::Node.joins(:tags).where("tags.name LIKE ?", "database:%")
+```
+
+### Tag Relationships
+
+Find tags that co-occur frequently:
+
+```ruby
+relationships = htm.long_term_memory.topic_relationships(min_shared_nodes: 2)
+# => [
+#   { tag1: "database:postgresql", tag2: "search:vector", shared_count: 15 },
+#   { tag1: "programming:ruby", tag2: "web:rails", shared_count: 12 }
+# ]
+```
+
+## Tag Visualization
+
+### Text Tree
+
+```ruby
+# All tags as directory-style tree
+puts HTM::Models::Tag.all.tree_string
+```
+
+Output:
+```
+database
+├── postgresql
+│   ├── extensions
+│   │   └── pgvector
+│   └── json
+├── mysql
+└── redis
+programming
+├── ruby
+│   ├── rails
+│   └── gems
+└── python
+```
+
+### Mermaid Flowchart
+
+```ruby
+# Generate Mermaid diagram
+mermaid = HTM::Models::Tag.all.tree_mermaid
+File.write("tags.md", "```mermaid\n#{mermaid}\n```")
+
+# Left-to-right orientation
+mermaid = HTM::Models::Tag.all.tree_mermaid(direction: 'LR')
+```
+
+### SVG Diagram
+
+```ruby
+# Generate SVG (dark theme, transparent background)
+svg = HTM::Models::Tag.all.tree_svg
+File.write("tags.svg", svg)
+
+# With custom title
+svg = HTM::Models::Tag.all.tree_svg(title: "Knowledge Taxonomy")
+```
+
+## Rake Tasks
+
+```bash
+# Display text tree (all tags)
+rake htm:tags:tree
+
+# Display tags with prefix
+rake 'htm:tags:tree[database]'
+
+# Export to Mermaid format
+rake htm:tags:mermaid
+rake 'htm:tags:mermaid[api]'
+
+# Export to SVG
+rake htm:tags:svg
+rake 'htm:tags:svg[web]'
+
+# Export all formats
+rake htm:tags:export
+rake 'htm:tags:export[database]'
+
+# Rebuild all tags (regenerate via LLM)
+rake htm:tags:rebuild
+```
+
+## Filtering by Tags
+
+### In Recall
+
+```ruby
+# Filter by specific tag
+results = htm.recall("query", tags: ["database:postgresql"])
+
+# Filter by multiple tags (AND)
+results = htm.recall("query", tags: ["database:postgresql", "search:vector"])
+
+# Combine with other filters
+results = htm.recall(
+  "performance optimization",
+  tags: ["database:postgresql"],
+  timeframe: "last week",
+  strategy: :hybrid,
+  limit: 10
+)
+```
+
+### Direct Queries
+
+```ruby
+# Find all nodes with a tag
+HTM::Models::Node.with_tag("database:postgresql")
+
+# Find nodes with any of several tags
+HTM::Models::Node.with_any_tags(["database:postgresql", "database:mysql"])
+
+# Find nodes with all specified tags
+HTM::Models::Node.with_all_tags(["database:postgresql", "search:vector"])
+```
+
+## Database Schema
+
+### Tags Table
+
+```sql
+CREATE TABLE tags (
+  id SERIAL PRIMARY KEY,
+  name VARCHAR(255) NOT NULL UNIQUE,
+  created_at TIMESTAMP DEFAULT NOW()
+);
+
+CREATE INDEX idx_tags_name ON tags(name);
+CREATE INDEX idx_tags_name_prefix ON tags USING btree (name text_pattern_ops);
+```
+
+### Node-Tag Association
+
+```sql
+CREATE TABLE node_tags (
+  node_id INTEGER REFERENCES nodes(id),
+  tag_id INTEGER REFERENCES tags(id),
+  created_at TIMESTAMP DEFAULT NOW(),
+  PRIMARY KEY (node_id, tag_id)
+);
+```
+
+## Best Practices
+
+### Design a Consistent Hierarchy
+
+Plan your top-level categories:
+
+```
+database:      # Database-related
+programming:   # Programming languages and frameworks
+api:           # API design and integration
+infrastructure: # DevOps, cloud, servers
+concept:       # Abstract concepts and patterns
+```
+
+### Use Appropriate Depth
+
+- **2-3 levels**: Typical for most use cases
+- **4+ levels**: Only for highly specialized domains
+
+```ruby
+# Good
+"database:postgresql:extensions"
+
+# Too deep (usually)
+"database:sql:relational:postgresql:extensions:pgvector:hnsw"
+```
+
+### Combine Auto and Manual Tags
+
+```ruby
+# Let LLM extract, but add specific tags you need
+htm.remember(
+  "PostgreSQL 16 introduces new parallel query features.",
+  tags: ["version:postgresql:16", "release:2023"]  # Manual additions
+)
+# LLM will also add: ["database:postgresql", "performance:parallel"]
+```
+
+## Related Documentation
+
+- [Adding Memories](adding-memories.md) - Core memory operations
+- [Search Strategies](search-strategies.md) - Using tags in queries
+- [Propositions](propositions.md) - Proposition nodes get tags too
+- [API Reference: TagService](../api/yard/HTM/TagService.md)