htm 0.0.1 → 0.0.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0a25e07ce28cc74ddb8b2fca0a9b316108b97362c2281f2e77b7bec5124da0f7
-  data.tar.gz: 9978ce3bd0c1b3c589436a5c8637bf8043bac3b0dfb542657851fa7cb96e1b49
+  metadata.gz: d381f41347974376534b3f6da3e1cbff46320e8ada7e078d432b27abed546165
+  data.tar.gz: 7d387e9daddbafef1f9bdc6d3538fe80c3c03ad0a78220b4b6722e56214e2682
 SHA512:
-  metadata.gz: bbd241d5bda941b6d3df5b35bb13e8dbf630af1be02debf814b4396fba5e76e67b6a22756690d3f510d4a0ad441710d61b3d8d20018db8ba1347220818ecefcf
-  data.tar.gz: 57e49d9629c934dd875e3d65a9c4b476bd96bc04dbc168455d3652c8a8b0aededaa2cd0163e377feadf28507550f6adbd0d13b5a9870b9c54b95437e7fd4e8bb
+  metadata.gz: 660b77256ddb31aeb4ae7cf2eff34dae9099b41303527291fe3640d991071df3691ed76b17ad5c2404ce3e1fafcde483ffea8359b0629dddf9cd86dafc0f3e45
+  data.tar.gz: fee193c2e2138abfd2dfdd9bb20806de8e51751a61dcea44e8b840ed12d6fb41098e1980d1755acf81f70e69d4286772c6770bbc2520cc2a44dd6fc12adcdab3

data/.envrc

@@ -9,6 +9,7 @@ export HTM_DBNAME=htm_development
 export HTM_DBUSER=${USER}
 export HTM_DBPASS=
 export HTM_DBURL="postgresql://${HTM_DBUSER}@${HTM_DBHOST}:${HTM_DBPORT}/${HTM_DBNAME}?sslmode=disable"
+# export HTM_DBURL="postgresql://dewayne@localhost:5432/htm_development"
 
 # Uncomment if using TimescaleDB Cloud instead:
 # export HTM_SERVICE_NAME=$TIGER_SERVICE_NAME

data/.tbls.yml

@@ -0,0 +1,30 @@
+# tbls configuration for HTM database documentation
+# https://github.com/k1LoW/tbls
+
+# Database connection - uses HTM_DBURL environment variable
+dsn: env://HTM_DBURL
+
+# Output documentation to docs/database directory
+docPath: docs/database
+
+# Documentation format options
+format:
+  adjust_table_width: true
+  sort: true
+
+# ER diagram settings
+er:
+  format: svg
+  comment: true
+  distance: 2
+
+# Include schema metadata
+include:
+  - public.*
+
+# Lint rules for schema quality
+lint:
+  requireColumnComment:
+    enabled: false
+  requireTableComment:
+    enabled: false

data/CHANGELOG.md

@@ -8,6 +8,29 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Added
+- **Multi-provider LLM support via RubyLLM** - HTM now supports 9 LLM providers:
+  - OpenAI (`text-embedding-3-small`, `gpt-4o-mini`)
+  - Anthropic (`claude-3-haiku-20240307`)
+  - Google Gemini (`text-embedding-004`)
+  - Azure OpenAI
+  - Ollama (default, local-first)
+  - HuggingFace Inference API
+  - OpenRouter
+  - AWS Bedrock
+  - DeepSeek
+- Provider-specific configuration attributes:
+  - `openai_api_key`, `openai_organization`, `openai_project`
+  - `anthropic_api_key`
+  - `gemini_api_key`
+  - `azure_api_key`, `azure_endpoint`, `azure_api_version`
+  - `ollama_url`
+  - `huggingface_api_key`
+  - `openrouter_api_key`
+  - `bedrock_access_key`, `bedrock_secret_key`, `bedrock_region`
+  - `deepseek_api_key`
+- `HTM::Configuration#configure_ruby_llm` method for provider credential setup
+- `SUPPORTED_PROVIDERS` constant listing all available providers
+- `DEFAULT_DIMENSIONS` hash with typical embedding dimensions per provider
 - Architecture documentation using ai-software-architect framework
 - Comprehensive ADRs (Architecture Decision Records):
   - ADR-001: PostgreSQL with TimescaleDB for storage
@@ -22,6 +45,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Architecture review team with 8 specialist perspectives
 - Had the robot convert my notss and system analysis documentation into Architectural Decision Records (ADR)
 
+### Changed
+- **Embedding generator now uses `RubyLLM.embed()`** instead of raw HTTP calls to Ollama
+- **Tag extractor now uses `RubyLLM.chat()`** instead of raw HTTP calls to Ollama
+- Configuration validation now checks provider is in `SUPPORTED_PROVIDERS`
+- Updated CLAUDE.md with multi-provider documentation and examples
+- Environment variables section expanded with all provider API keys
+
 ## [0.1.0] - 2025-10-25
 
 ### Added

data/SETUP.md

@@ -3,122 +3,165 @@
 ## Prerequisites
 
 1. **Ruby** (version 3.0 or higher)
-2. **TimescaleDB Cloud Account** (already set up)
-3. **Database Environment Variables** (already configured)
-4. **Ollama** (for embeddings via RubyLLM)
+2. **PostgreSQL** (14+ with pgvector and pg_trgm extensions)
+3. **Ollama** (for embeddings via RubyLLM)
 
-## Ollama Setup
+## PostgreSQL Setup
 
-HTM uses RubyLLM with the Ollama provider for generating embeddings. You need to install and run Ollama locally.
+### 1. Install PostgreSQL
 
-### 1. Install Ollama
+**macOS (via Homebrew):**
+```bash
+brew install postgresql@17
+brew services start postgresql@17
+```
 
-**macOS:**
+**Ubuntu/Debian:**
 ```bash
-curl https://ollama.ai/install.sh | sh
+sudo apt install postgresql postgresql-contrib
+sudo systemctl start postgresql
 ```
 
-**Or download from:** https://ollama.ai/download
+### 2. Install pgvector Extension
 
-### 2. Start Ollama Service
+**macOS:**
+```bash
+brew install pgvector
+```
 
+**Ubuntu/Debian:**
 ```bash
-# Ollama typically starts automatically after installation
-# Verify it's running:
-curl http://localhost:11434/api/version
+sudo apt install postgresql-17-pgvector
 ```
 
-### 3. Pull the gpt-oss Model
+**From source:**
+```bash
+git clone https://github.com/pgvector/pgvector.git
+cd pgvector
+make
+sudo make install
+```
+
+### 3. Create Database and Enable Extensions
 
 ```bash
-# Pull the default model used by HTM
-ollama pull gpt-oss
+# Create the development database
+createdb htm_development
 
-# Verify the model is available
-ollama list
+# Enable required extensions
+psql htm_development -c "CREATE EXTENSION IF NOT EXISTS vector;"
+psql htm_development -c "CREATE EXTENSION IF NOT EXISTS pg_trgm;"
+
+# Verify extensions
+psql htm_development -c "SELECT extname, extversion FROM pg_extension;"
 ```
 
-### 4. Test Embedding Generation
+### 4. Set Environment Variable
 
 ```bash
-# Test that embeddings work
-ollama run gpt-oss "Hello, world!"
-```
+# Add to your ~/.bashrc or ~/.zshrc
+export HTM_DBURL="postgresql://postgres@localhost:5432/htm_development"
 
-### Optional: Custom Ollama URL
+# Or for a specific user with password
+export HTM_DBURL="postgresql://username:password@localhost:5432/htm_development"
+```
 
-If Ollama is running on a different host/port, set the environment variable:
+### 5. Verify Connection
 
 ```bash
-export OLLAMA_URL="http://custom-host:11434"
+cd /path/to/HTM
+ruby test_connection.rb
+```
+
+You should see:
+```
+✓ Connected successfully!
+✓ pgvector Extension: Version 0.8.x
+✓ pg_trgm Extension: Version 1.6
 ```
 
-## Database Setup
+## Ollama Setup
 
-### 1. Load Database Credentials
+HTM uses RubyLLM with the Ollama provider for generating embeddings. You need to install and run Ollama locally.
 
-The HTM project uses environment variables to manage database credentials. These are defined in `~/.bashrc__tiger`.
+### 1. Install Ollama
 
+**macOS:**
 ```bash
-# Load the Tiger database environment variables
-source ~/.bashrc__tiger
+curl https://ollama.ai/install.sh | sh
 ```
 
-To make these variables available automatically in new shell sessions, ensure `~/.bashrc__tiger` is sourced in your `~/.bashrc` or `~/.bash_profile`.
-
-### 2. Verify Connection
+**Or download from:** https://ollama.ai/download
 
-Test the database connection:
+### 2. Start Ollama Service
 
 ```bash
-cd /path/to/HTM
-ruby test_connection.rb
+# Ollama typically starts automatically after installation
+# Verify it's running:
+curl http://localhost:11434/api/version
 ```
 
-You should see:
+### 3. Pull Required Models
+
+```bash
+# Pull the embedding model
+ollama pull nomic-embed-text
+
+# Pull the chat model (for tag extraction)
+ollama pull llama3
+
+# Verify models are available
+ollama list
 ```
-✓ Connected successfully!
-✓ TimescaleDB Extension: Version 2.22.1
-✓ pgvector Extension: Version 0.8.1
-✓ pg_trgm Extension: Version 1.6
+
+### 4. Test Embedding Generation
+
+```bash
+# Test that embeddings work
+curl http://localhost:11434/api/embeddings -d '{
+  "model": "nomic-embed-text",
+  "prompt": "Hello, world!"
+}'
 ```
 
-### 3. Enable Extensions (One-time)
+### Optional: Custom Ollama URL
 
-Enable required PostgreSQL extensions (already done, but can be re-run safely):
+If Ollama is running on a different host/port, set the environment variable:
 
 ```bash
-ruby enable_extensions.rb
+export OLLAMA_URL="http://custom-host:11434"
 ```
 
 ## Environment Variables Reference
 
-After sourcing `~/.bashrc__tiger`, these variables are available:
-
 | Variable | Description | Example Value |
 |----------|-------------|---------------|
-| `HTM_SERVICE_NAME` | Service identifier | `db-67977` |
-| `HTM_DBNAME` | Database name | `tsdb` |
-| `HTM_DBUSER` | Database user | `tsdbadmin` |
-| `HTM_DBPASS` | Database password | `***` |
-| `HTM_DBURL` | Full connection URL (preferred) | `postgres://...` |
-| `HTM_DBPORT` | Database port | `37807` |
+| `HTM_DBURL` | Full PostgreSQL connection URL (preferred) | `postgresql://postgres@localhost:5432/htm_development` |
+| `HTM_DBNAME` | Database name (fallback) | `htm_development` |
+| `HTM_DBUSER` | Database user (fallback) | `postgres` |
+| `HTM_DBPASS` | Database password (fallback) | `` |
+| `HTM_DBHOST` | Database host (fallback) | `localhost` |
+| `HTM_DBPORT` | Database port (fallback) | `5432` |
+| `OLLAMA_URL` | Ollama server URL | `http://localhost:11434` |
 
 ## Development Workflow
 
 ### Quick Start
 
 ```bash
-# 1. Source environment variables (if not in .bashrc)
-source ~/.bashrc__tiger
+# 1. Set database URL (if not in shell config)
+export HTM_DBURL="postgresql://postgres@localhost:5432/htm_development"
 
-# 2. Install dependencies (when gem is created)
+# 2. Install dependencies
 bundle install
 
-# 3. Initialize database schema (when ready)
-ruby -r ./lib/htm -e "HTMDatabase.setup"
+# 3. Initialize database schema
+rake db_setup
+
+# 4. Run tests
+rake test
 
-# 4. Test HTM functionality (when implemented)
+# 5. Try the basic example
 ruby examples/basic_usage.rb
 ```
 
@@ -130,10 +173,8 @@ HTM uses Minitest for testing:
 # Run all tests
 rake test
 
-# Or run directly with Ruby
-ruby test/htm_test.rb
-
 # Run specific test file
+ruby test/htm_test.rb
 ruby test/embedding_service_test.rb
 
 # Run integration tests (requires database)
@@ -150,14 +191,18 @@ HTM/
 │   │   ├── database.rb           # Database setup and schema
 │   │   ├── long_term_memory.rb   # PostgreSQL-backed storage
 │   │   ├── working_memory.rb     # In-memory active context
-│   │   ├── embedding_service.rb  # RubyLLM embedding generation (Ollama/gpt-oss)
+│   │   ├── embedding_service.rb  # RubyLLM embedding generation
+│   │   ├── tag_service.rb        # Hierarchical tag extraction
+│   │   ├── configuration.rb      # Multi-provider LLM config
 │   │   └── version.rb            # Version constant
-├── sql/
+├── config/
+│   └── database.yml              # Database configuration
+├── db/
 │   └── schema.sql                # Database schema
 ├── test/
 │   ├── test_helper.rb            # Minitest configuration
 │   ├── htm_test.rb               # Basic HTM tests
-│   ├── embedding_service_test.rb # Embedding tests (RubyLLM/Ollama)
+│   ├── embedding_service_test.rb # Embedding tests
 │   └── integration_test.rb       # Full integration tests
 ├── examples/
 │   └── basic_usage.rb            # Basic usage example
@@ -165,24 +210,12 @@ HTM/
 ├── enable_extensions.rb          # Enable PostgreSQL extensions
 ├── SETUP.md                      # This file
 ├── README.md                     # Project overview
-├── htm_teamwork.md               # Planning and design doc
+├── CLAUDE.md                     # AI assistant instructions
 ├── Gemfile
 ├── htm.gemspec
 └── Rakefile                      # Rake tasks
 ```
 
-## Next Steps
-
-1. **Phase 1**: Create basic gem structure
-2. **Phase 2**: Implement database schema
-3. **Phase 3**: Implement LongTermMemory class
-4. **Phase 4**: Implement WorkingMemory class
-5. **Phase 5**: Implement HTM main class
-6. **Phase 6**: Add tests
-7. **Phase 7**: Create examples
-
-See `htm_teamwork.md` for detailed roadmap.
-
 ## Troubleshooting
 
 ### Ollama Issues
@@ -193,24 +226,20 @@ If you encounter embedding errors:
 # Verify Ollama is running
 curl http://localhost:11434/api/version
 
-# Check if gpt-oss model is available
-ollama list | grep gpt-oss
+# Check if models are available
+ollama list
 
 # Test embedding generation
-ollama run gpt-oss "Test embedding"
+curl http://localhost:11434/api/embeddings -d '{"model": "nomic-embed-text", "prompt": "Test"}'
 
-# View Ollama logs
-ollama logs
-
-# Restart Ollama service
-# On macOS, Ollama runs as a background service
-# Check Activity Monitor or restart from the menu bar
+# View Ollama logs (macOS)
+# Check Console.app or Activity Monitor
 ```
 
 **Common Ollama Errors:**
 
 - **"connection refused"**: Ollama service is not running. Start Ollama from Applications or via CLI.
-- **"model not found"**: Run `ollama pull gpt-oss` to download the model.
+- **"model not found"**: Run `ollama pull nomic-embed-text` to download the model.
 - **Custom URL not working**: Ensure `OLLAMA_URL` environment variable is set correctly.
 
 ### Database Connection Issues
@@ -218,14 +247,15 @@ ollama logs
 If you get connection errors:
 
 ```bash
-# Verify environment variables are set
+# Verify environment variable is set
 echo $HTM_DBURL
 
 # Test connection manually
-psql $HTM_DBURL
+psql $HTM_DBURL -c "SELECT 1"
 
-# Check if ~/.bashrc__tiger is sourced
-grep "bashrc__tiger" ~/.bashrc
+# Check PostgreSQL is running
+brew services list | grep postgresql  # macOS
+systemctl status postgresql           # Linux
 ```
 
 ### Extension Issues
@@ -233,31 +263,32 @@ grep "bashrc__tiger" ~/.bashrc
 If extensions aren't available:
 
 ```bash
+# Check if pgvector is installed
+psql htm_development -c "SELECT * FROM pg_available_extensions WHERE name = 'vector';"
+
 # Re-run extension setup
 ruby enable_extensions.rb
 
-# Check extension status manually
-psql $HTM_DBURL -c "SELECT extname, extversion FROM pg_extension ORDER BY extname"
+# Check extension status
+psql htm_development -c "SELECT extname, extversion FROM pg_extension ORDER BY extname"
 ```
 
-### SSL Issues
+### Test Database
 
-The TimescaleDB Cloud instance requires SSL. If you see SSL errors:
+For running tests, create a separate test database:
 
 ```bash
-# Ensure sslmode is set in connection URL
-echo $HTM_DBURL | grep sslmode
-# Should show: sslmode=require
+createdb htm_development_test
+psql htm_development_test -c "CREATE EXTENSION IF NOT EXISTS vector; CREATE EXTENSION IF NOT EXISTS pg_trgm;"
 ```
 
 ## Resources
 
 - **Ollama**: https://ollama.ai/
-- **RubyLLM**: https://github.com/madbomber/ruby_llm
-- **TimescaleDB Docs**: https://docs.timescale.com/
+- **RubyLLM**: https://github.com/crmne/ruby_llm
 - **pgvector Docs**: https://github.com/pgvector/pgvector
-- **Planning Document**: `htm_teamwork.md`
 - **PostgreSQL Docs**: https://www.postgresql.org/docs/
+- **Planning Document**: `htm_teamwork.md`
 
 ## Support
 

data/db/migrate/20250125000001_add_content_hash_to_nodes.rb

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+class AddContentHashToNodes < ActiveRecord::Migration[7.1]
+  def change
+    unless column_exists?(:nodes, :content_hash)
+      # Add content_hash column for SHA-256 hash of content (64 hex characters)
+      add_column :nodes, :content_hash, :string, limit: 64,
+                 comment: 'SHA-256 hash of content for deduplication'
+
+      # Unique index to prevent duplicate content
+      add_index :nodes, :content_hash, unique: true, name: 'idx_nodes_content_hash_unique'
+    end
+  end
+end

data/db/migrate/20250125000002_create_robot_nodes.rb

@@ -0,0 +1,35 @@
+# frozen_string_literal: true
+
+class CreateRobotNodes < ActiveRecord::Migration[7.1]
+  def change
+    unless table_exists?(:robot_nodes)
+      create_table :robot_nodes, comment: 'Join table connecting robots to nodes (many-to-many)' do |t|
+        t.bigint :robot_id, null: false, comment: 'ID of the robot that remembered this node'
+        t.bigint :node_id, null: false, comment: 'ID of the node being remembered'
+        t.timestamptz :first_remembered_at, default: -> { 'CURRENT_TIMESTAMP' },
+                      comment: 'When this robot first remembered this content'
+        t.timestamptz :last_remembered_at, default: -> { 'CURRENT_TIMESTAMP' },
+                      comment: 'When this robot last tried to remember this content'
+        t.integer :remember_count, default: 1, null: false,
+                  comment: 'Number of times this robot has tried to remember this content'
+        t.timestamptz :created_at, default: -> { 'CURRENT_TIMESTAMP' }
+        t.timestamptz :updated_at, default: -> { 'CURRENT_TIMESTAMP' }
+      end
+
+      # Unique constraint: each robot can only link to a node once
+      add_index :robot_nodes, [:robot_id, :node_id], unique: true, name: 'idx_robot_nodes_unique'
+      add_index :robot_nodes, :robot_id, name: 'idx_robot_nodes_robot_id'
+      add_index :robot_nodes, :node_id, name: 'idx_robot_nodes_node_id'
+      add_index :robot_nodes, :last_remembered_at, name: 'idx_robot_nodes_last_remembered_at'
+    end
+
+    # Add foreign keys
+    unless foreign_key_exists?(:robot_nodes, :robots, column: :robot_id)
+      add_foreign_key :robot_nodes, :robots, column: :robot_id, primary_key: :id, on_delete: :cascade
+    end
+
+    unless foreign_key_exists?(:robot_nodes, :nodes, column: :node_id)
+      add_foreign_key :robot_nodes, :nodes, column: :node_id, primary_key: :id, on_delete: :cascade
+    end
+  end
+end

data/db/migrate/20250125000003_remove_source_and_robot_id_from_nodes.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+class RemoveSourceAndRobotIdFromNodes < ActiveRecord::Migration[7.1]
+  def change
+    # Remove foreign key constraint first
+    if foreign_key_exists?(:nodes, :robots, column: :robot_id)
+      remove_foreign_key :nodes, :robots, column: :robot_id
+    end
+
+    # Remove indexes
+    if index_exists?(:nodes, :robot_id, name: 'idx_nodes_robot_id')
+      remove_index :nodes, name: 'idx_nodes_robot_id'
+    end
+
+    if index_exists?(:nodes, :source, name: 'idx_nodes_source')
+      remove_index :nodes, name: 'idx_nodes_source'
+    end
+
+    # Remove columns
+    if column_exists?(:nodes, :robot_id)
+      remove_column :nodes, :robot_id
+    end
+
+    if column_exists?(:nodes, :source)
+      remove_column :nodes, :source
+    end
+  end
+end

data/db/migrate/20250126000001_create_working_memories.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+class CreateWorkingMemories < ActiveRecord::Migration[7.1]
+  def change
+    create_table :working_memories, comment: 'Per-robot working memory state (optional persistence)' do |t|
+      t.bigint :robot_id, null: false, comment: 'Robot whose working memory this belongs to'
+      t.bigint :node_id, null: false, comment: 'Node currently in working memory'
+      t.timestamptz :added_at, default: -> { 'CURRENT_TIMESTAMP' }, comment: 'When node was added to working memory'
+      t.integer :token_count, comment: 'Cached token count for budget tracking'
+    end
+
+    add_index :working_memories, :robot_id, name: 'idx_working_memories_robot_id'
+    add_index :working_memories, :node_id, name: 'idx_working_memories_node_id'
+    add_index :working_memories, [:robot_id, :node_id], unique: true, name: 'idx_working_memories_unique'
+
+    add_foreign_key :working_memories, :robots, on_delete: :cascade
+    add_foreign_key :working_memories, :nodes, on_delete: :cascade
+  end
+end

data/db/migrate/20250126000002_remove_unused_columns.rb

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+class RemoveUnusedColumns < ActiveRecord::Migration[7.1]
+  def change
+    # Remove in_working_memory from nodes - now tracked per-robot in working_memories table
+    remove_index :nodes, name: 'idx_nodes_in_working_memory', if_exists: true
+    remove_column :nodes, :in_working_memory, :boolean, default: false
+
+    # Remove unused metadata column from robots
+    remove_column :robots, :metadata, :jsonb
+  end
+end