htm 0.0.18 → 0.0.30

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7379c23800bf3f00cc328be9725b14bcea06a9d2158eaf3b3bca80b11a6efb74
-  data.tar.gz: ec96095a72ab89fed1dd7df0ceaeb3888938f5ddbdea6fd3f3c06aa15b80bf67
+  metadata.gz: 5765fbc2b92d89be3f2bae2f53b4353a76343efd3d092456138bea73ac7803d9
+  data.tar.gz: e594cc16f789745267ce527dcc182fbd533b7484c36e09887f702007388803a3
 SHA512:
-  metadata.gz: 428b9c8f1db0f3a7a4be6e0b8bc4cc27f0bf058e7847c409cfca423c1bf9722a8efad92dbdd276d69844ab771670ea7cdc62e292018bec4705ed3f9f66255c16
-  data.tar.gz: af62a40339de40eac22f90b919a626b0f2d61a14440cdd5ca9e0aaf87ca088cc086cd35bc1eb4394d87a0646512c0a30d2fb47fdee7ec9ea2c4f11e469f2a44b
+  metadata.gz: 78b4b7e226b9911b429e7e0d8735dd0d50d1e56abf7e9e5f494ab50321e33e574605d6a3991d4f0ce18a95ae6dd8ee7928004e3d455730aa388ee2515ffc53c3
+  data.tar.gz: c19a0c15d79342d08e406724724cf64afc68c75ea26f1bf332b4a9b661f3ebe9d50e3f9dbf41db780c0542d4cd7a77ac0bb0da9bf45b5ea7d171def21b4ffc4a

data/CHANGELOG.md

@@ -6,6 +6,117 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
+
+### Added
+- **`sslmode` database configuration support** - SSL mode now extracted from URL and included when building URL
+  - `parse_database_url` extracts `sslmode` from URL query string (e.g., `?sslmode=require`)
+  - `build_database_url` includes `sslmode` as query parameter when set
+  - `reconcile_from_url` syncs `sslmode` along with other database components
+  - Default `sslmode` is `prefer` (from `defaults.yml`)
+- **`htm:db:purge_all` rake task** - Permanently remove all soft-deleted records from database
+  - Displays record counts by table before deletion
+  - Detects and removes orphaned join table entries (`node_tags`, `robot_nodes`)
+  - Detects and removes orphaned propositions (where `source_node_id` no longer exists)
+  - Detects and removes orphaned robots (with no associated memory nodes)
+  - Requires confirmation before proceeding
+  - Deletes in correct order for referential integrity
+- **`htm:jobs:process_propositions` rake task** - Incremental proposition extraction for unprocessed nodes
+  - Uses `ruby-progressbar` with ETA display
+  - Only processes nodes not yet extracted (tracks via `source_node_id` metadata)
+  - Added to `htm:jobs:process_all` task chain
+- **Rake task passthrough in MCP CLI** - Run rake tasks via `htm_mcp rake <task>`
+  - `htm_mcp rake htm:db:stats` - Run any HTM rake task
+  - `htm_mcp rake -T` / `htm_mcp rake --tasks` - List available tasks
+  - **Pattern filtering** - `htm_mcp rake -T htm:jobs` filters to matching namespace (like standard rake)
+- **Meta-response filtering in PropositionService** - Filters LLM responses that ask for input
+  - `META_RESPONSE_PATTERNS` constant with common patterns ("please provide", "I need the text", etc.)
+  - `meta_response?` method for detecting invalid responses
+  - Prevents storing "Please provide the text" as propositions
+- **Progress bars for job processing tasks** - Visual progress with ETA for long-running operations
+  - `htm:jobs:process_embeddings` - Shows progress when generating embeddings
+  - `htm:jobs:process_tags` - Shows progress when extracting tags
+  - Format: `Processing: |████████████████| 50/100 (50%) ETA: 00:01:30`
+
+### Changed
+- **Centralized HTM loading in rake tasks** - Single `require_relative` in `lib/htm/tasks.rb`
+  - HTM is now loaded once before any rake task files are loaded
+  - Removed ~25 redundant `require 'htm'` statements from individual rake tasks
+  - Ensures local development codebase is always used via `require_relative`
+  - Removed `$LOAD_PATH` manipulation that was previously needed for `require 'htm'`
+- **Removed `:validate` rake task dependency** - Config validation now automatic
+  - `HTM::Config.new` handles all validation at require time (environment, URL/component reconciliation, naming convention)
+  - Rake tasks no longer need explicit validation step
+- **Cleaned up Rakefile** - Removed unnecessary code
+  - Removed deprecated `db_setup` and `db_test` backward-compatibility tasks
+  - Removed redundant `task :test` block (`:set_test_env` prerequisite handles environment setup)
+- **PropositionService validation now fully configurable** - Moved hardcoded constants to `defaults.yml`
+  - `proposition.min_length` (default: 10) - Minimum characters for valid proposition
+  - `proposition.max_length` (default: 1000) - Maximum characters for valid proposition
+  - `proposition.min_words` (default: 5) - Minimum words for valid proposition
+  - Added `min_length`, `max_length`, `min_words` class methods that read from config
+- **Improved proposition extraction prompt** - Better quality propositions
+  - Added explicit BAD/GOOD examples for pronoun replacement
+  - Added context enrichment examples (e.g., "wiring" → "solar panel wiring for Oklahoma barndominium")
+  - System prompt now explicitly prevents meta-responses
+  - Increased specificity requirements for self-contained facts
+
+### Fixed
+- **Test database isolation** - Two-layer protection prevents tests from polluting development/production
+  - `Rakefile`: `set_test_env` task now ALWAYS overrides `HTM_DATABASE__URL` to test database
+  - Uses `#{service_name}_test` pattern (e.g., `htm_test`) based on `HTM_SERVICE__NAME` env var
+  - `test_helper.rb`: Safety check aborts with helpful message if database URL doesn't contain `_test`
+  - Prevents accidental test execution against non-test databases
+
+## [0.0.20] - 2025-12-22
+### Added
+- **Fiber-based job backend** - New `:fiber` backend for I/O-bound background jobs
+  - Uses `async` gem for cooperative concurrency
+  - Non-blocking execution ideal for LLM API calls
+  - `JobAdapter.enqueue_parallel(jobs)` for concurrent job execution
+  - Fiber backend runs jobs with `Async::Barrier` for parallel coordination
+- **RememberWorkflow using simple_flow** - Parallel processing pipeline for node enrichment
+  - Orchestrates save_node → (embedding, tags, propositions in parallel) → finalize
+  - Uses `SimpleFlow::Pipeline` with dependency-based step execution
+  - Configurable concurrency model (`:auto`, `:threads`, `:async`)
+  - Visualization support: `to_mermaid` and `execution_plan` methods
+
+### Changed
+- **async gem is now a required dependency** - Previously optional, now always available
+  - Enables fiber-based concurrency for all HTM installations
+  - `:fiber` is now the default job backend (was `:thread`)
+- **Default job backend changed from `:thread` to `:fiber`** - Better performance for I/O-bound LLM operations
+
+### Removed
+- **`JobAdapter.async_available?` method** - No longer needed since async is always available
+
+### Fixed
+- **`connection_timeout` default inconsistency** - Unified default to 60 seconds across all files
+  - `test/configuration_test.rb`, `docs/guides/configuration.md`, and `db/seeds.rb` now match `defaults.yml`
+
+### Dependencies
+- Added `async` (~> 2.0) as required runtime dependency
+- Added `simple_flow` for workflow orchestration
+
+## [0.0.19] - 2025-12-21
+
+### Changed
+- **Reorganized "Robots!" documentation** - Consolidated 5 robot-related docs into `docs/robots/` subdirectory
+  - Created `docs/robots/index.md` as section overview with navigation table and architecture diagram
+  - Moved `docs/robots.md` → `docs/robots/why-robots.md`
+  - Moved `docs/guides/multi-robot.md` → `docs/robots/multi-robot.md`
+  - Moved `docs/guides/robot-groups.md` → `docs/robots/robot-groups.md`
+  - Moved `docs/architecture/two-tier-memory.md` → `docs/robots/two-tier-memory.md`
+  - Moved `docs/architecture/hive-mind.md` → `docs/robots/hive-mind.md`
+  - Updated `mkdocs.yml` navigation with logical information flow
+  - Fixed 30+ broken relative links across documentation files
+- **Extracted inline SVG graphics to standalone files** - Improved documentation maintainability
+  - From `hive-mind.md`: `hive-mind-shared-memory.svg`, `memory-topology.svg`
+  - From `two-tier-memory.md`: `two-tier-memory-architecture.svg`, `eviction-priority.svg`, `balanced-strategy-decay.svg`
+  - All SVGs now in `docs/assets/images/` and referenced via markdown image syntax
+
+### Fixed
+- **ConfigSection.md unrecognized link warning** - Wrapped method signatures in backticks to prevent MkDocs interpreting bracket parameters as links
+
 ## [0.0.18] - 2025-12-20
 ### Added
 - **Anyway::Config-based configuration system** - Replaced custom Configuration class with robust multi-source config management
@@ -689,7 +800,14 @@ HTM.config.embedding.model
 - Working memory size is user-configurable
 - See ADRs for detailed architectural decisions and rationale
 
-[Unreleased]: https://github.com/madbomber/htm/compare/v0.0.12...HEAD
+[Unreleased]: https://github.com/madbomber/htm/compare/v0.0.19...HEAD
+[0.0.19]: https://github.com/madbomber/htm/compare/v0.0.18...v0.0.19
+[0.0.18]: https://github.com/madbomber/htm/compare/v0.0.17...v0.0.18
+[0.0.17]: https://github.com/madbomber/htm/compare/v0.0.15...v0.0.17
+[0.0.15]: https://github.com/madbomber/htm/compare/v0.0.14...v0.0.15
+[0.0.14]: https://github.com/madbomber/htm/compare/v0.0.13...v0.0.14
+[0.0.13]: https://github.com/madbomber/htm/compare/v0.0.11...v0.0.13
+[0.0.11]: https://github.com/madbomber/htm/compare/v0.0.10...v0.0.11
 [0.0.12]: https://github.com/madbomber/htm/compare/v0.0.10...v0.0.12
 [0.0.10]: https://github.com/madbomber/htm/compare/v0.0.9...v0.0.10
 [0.0.9]: https://github.com/madbomber/htm/compare/v0.0.8...v0.0.9

data/README.md

@@ -1,4 +1,16 @@
+
 <div align="center">
+  <div style="background: linear-gradient(135deg, #90EE90 0%, #32CD32 100%); border: 4px solid #228B22; border-radius: 12px; padding: 20px; margin: 20px auto; max-width: 800px; box-shadow: 0 8px 16px rgba(34, 139, 34, 0.3);">
+    <p style="color: #000000; font-size: 42px; font-weight: bold; margin: 0;">
+        🚀 v0.0.20 🚀
+    </p>
+    <p style="color: #000; font-size: 27px; font-weight: bold; margin: 10px 0 0 0; line-height: 1.6;">
+        <strong>Fiber-based concurrency</strong> is now the default.<br/>
+        Parallel embedding, tagging, and proposition extraction via simple_flow.
+    </p>
+  </div>
+
+
   <h1>HTM</h1>
   <img src="docs/assets/images/htm_demo.gif" alt="Tree of Knowledge is Growing" width="400">
 

data/Rakefile

@@ -10,21 +10,27 @@ Rake::TestTask.new(:test) do |t|
   t.verbose = true
 end
 
-# Ensure test task runs with HTM_ENV=test (takes priority over RAILS_ENV/RACK_ENV)
-task :test do
-  ENV['HTM_ENV'] = 'test'
-end
-
 # Prepend environment setup before test runs
 Rake::Task[:test].enhance [:set_test_env]
 
 task :set_test_env do
   ENV['HTM_ENV'] = 'test'
-  # Set database URL if not already configured
-  # Uses current system user for local PostgreSQL connection
-  unless ENV['HTM_DATABASE__URL']
-    ENV['HTM_DATABASE__URL'] = "postgresql://#{ENV['USER']}@localhost:5432/htm_test"
+
+  # Build test database name from service name + environment
+  # Uses HTM_SERVICE__NAME env var if set, otherwise defaults to 'htm'
+  service_name = ENV['HTM_SERVICE__NAME'] || 'htm'
+  test_db_name = "#{service_name}_test"
+
+  # ALWAYS use the test database - never allow tests to run against other databases
+  # This prevents accidental pollution of development/production data
+  test_db_url = "postgresql://#{ENV['USER']}@localhost:5432/#{test_db_name}"
+
+  if ENV['HTM_DATABASE__URL'] && !ENV['HTM_DATABASE__URL'].include?('_test')
+    warn "WARNING: HTM_DATABASE__URL was set to '#{ENV['HTM_DATABASE__URL']}'"
+    warn "         Overriding to use test database: #{test_db_url}"
   end
+
+  ENV['HTM_DATABASE__URL'] = test_db_url
 end
 
 task default: :test
@@ -33,21 +39,101 @@ task default: :test
 # This uses the same loader that external applications use
 require_relative "lib/htm/tasks"
 
-# Legacy tasks for backwards compatibility
-desc "Run database setup (deprecated: use htm:db:setup)"
-task :db_setup => "htm:db:setup"
+# =============================================================================
+# Examples Tasks
+# =============================================================================
+
+# Prepend environment setup before running any example
+task :set_examples_env do
+  ENV['HTM_ENV'] = 'examples'
+
+  # Build examples database name from service name + environment
+  service_name = ENV['HTM_SERVICE__NAME'] || 'htm'
+  examples_db_name = "#{service_name}_examples"
 
-desc "Verify database connection (deprecated: use htm:db:verify)"
-task :db_test => "htm:db:verify"
+  # ALWAYS use the examples database
+  examples_db_url = "postgresql://#{ENV['USER']}@localhost:5432/#{examples_db_name}"
 
-desc "Run example"
-task :example do
-  ruby "examples/basic_usage.rb"
+  if ENV['HTM_DATABASE__URL'] && !ENV['HTM_DATABASE__URL'].include?('_examples')
+    warn "WARNING: HTM_DATABASE__URL was set to '#{ENV['HTM_DATABASE__URL']}'"
+    warn "         Overriding to use examples database: #{examples_db_url}"
+  end
+
+  ENV['HTM_DATABASE__URL'] = examples_db_url
 end
 
+namespace :examples do
+  desc "Set up examples database (create + setup schema)"
+  task setup: :set_examples_env do
+    Rake::Task['htm:db:create'].invoke rescue nil
+    Rake::Task['htm:db:setup'].invoke
+  end
+
+  desc "Reset examples database (drop + create + setup)"
+  task reset: :set_examples_env do
+    Rake::Task['htm:db:reset'].invoke
+  end
+
+  desc "Run basic_usage example"
+  task basic: :set_examples_env do
+    ruby "examples/01_basic_usage.rb"
+  end
+
+  desc "Run all standalone examples"
+  task :all => :set_examples_env do
+    examples = %w[
+      examples/01_basic_usage.rb
+      examples/03_custom_llm_configuration.rb
+      examples/04_file_loader_usage.rb
+      examples/05_timeframe_demo.rb
+    ]
+    examples.each do |example|
+      if File.exist?(example)
+        puts "\n#{'=' * 60}"
+        puts "Running: #{example}"
+        puts "#{'=' * 60}"
+        ruby example
+      end
+    end
+  end
+
+  desc "Show examples database status"
+  task status: :set_examples_env do
+    require_relative 'lib/htm'
+    puts "Examples Environment Status"
+    puts "=" * 40
+    puts "HTM_ENV: #{ENV['HTM_ENV']}"
+    puts "Database URL: #{ENV['HTM_DATABASE__URL']}"
+    puts "Expected database: #{HTM.config.expected_database_name}"
+    if HTM.config.database_configured?
+      puts "Database configured: Yes"
+      begin
+        HTM::ActiveRecordConfig.establish_connection!
+        if HTM::ActiveRecordConfig.connected?
+          puts "Database connected: Yes"
+          puts "\nTable counts:"
+          %w[nodes robots tags file_sources].each do |table|
+            count = ActiveRecord::Base.connection.execute("SELECT COUNT(*) FROM #{table}").first['count']
+            puts "  #{table}: #{count}"
+          end
+        else
+          puts "Database connected: No"
+        end
+      rescue => e
+        puts "Database connected: No (#{e.message})"
+      end
+    else
+      puts "Database configured: No"
+    end
+  end
+end
+
+desc "Run example (alias for examples:basic)"
+task :example => 'examples:basic'
+
 desc "Run timeframe demo"
 task :timeframe_demo do
-  ruby "examples/timeframe_demo.rb"
+  ruby "examples/05_timeframe_demo.rb"
 end
 
 desc "Show gem stats"

data/db/migrate/00001_enable_extensions.rb

@@ -1,13 +1,17 @@
 # frozen_string_literal: true
 
-class EnableExtensions < ActiveRecord::Migration[7.1]
+require_relative '../../lib/htm/migration'
+
+class EnableExtensions < HTM::Migration
   def up
-    enable_extension 'vector'
-    enable_extension 'pg_trgm'
+    run "CREATE EXTENSION IF NOT EXISTS vector"
+    run "CREATE EXTENSION IF NOT EXISTS pg_trgm"
+    run "CREATE EXTENSION IF NOT EXISTS pg_search"
   end
 
   def down
-    disable_extension 'pg_trgm'
-    disable_extension 'vector'
+    run "DROP EXTENSION IF EXISTS pg_search"
+    run "DROP EXTENSION IF EXISTS pg_trgm"
+    run "DROP EXTENSION IF EXISTS vector"
   end
 end

data/db/migrate/00002_create_robots.rb

@@ -1,11 +1,23 @@
 # frozen_string_literal: true
 
-class CreateRobots < ActiveRecord::Migration[7.1]
-  def change
-    create_table :robots, comment: 'Registry of all LLM robots using the HTM system' do |t|
-      t.text :name, comment: 'Human-readable name for the robot'
-      t.timestamptz :created_at, default: -> { 'CURRENT_TIMESTAMP' }, comment: 'When the robot was first registered'
-      t.timestamptz :last_active, default: -> { 'CURRENT_TIMESTAMP' }, comment: 'Last time the robot accessed the system'
+require_relative '../../lib/htm/migration'
+
+class CreateRobots < HTM::Migration
+  def up
+    create_table(:robots) do
+      primary_key :id
+      String :name, text: true
+      DateTime :created_at, default: Sequel::CURRENT_TIMESTAMP
+      DateTime :last_active, default: Sequel::CURRENT_TIMESTAMP
     end
+
+    run "COMMENT ON TABLE robots IS 'Registry of all LLM robots using the HTM system'"
+    run "COMMENT ON COLUMN robots.name IS 'Human-readable name for the robot'"
+    run "COMMENT ON COLUMN robots.created_at IS 'When the robot was first registered'"
+    run "COMMENT ON COLUMN robots.last_active IS 'Last time the robot accessed the system'"
+  end
+
+  def down
+    drop_table(:robots)
   end
 end

data/db/migrate/00003_create_file_sources.rb

@@ -1,25 +1,38 @@
 # frozen_string_literal: true
 
-class CreateFileSources < ActiveRecord::Migration[7.1]
-  def change
-    create_table :file_sources, comment: 'Source file metadata for loaded documents' do |t|
-      t.text :file_path, null: false, comment: 'Absolute path to source file'
-      t.string :file_hash, limit: 64, comment: 'SHA-256 hash of file content'
-      t.timestamptz :mtime, comment: 'File modification time'
-      t.integer :file_size, comment: 'File size in bytes'
-      t.jsonb :frontmatter, default: {}, comment: 'Parsed YAML frontmatter'
-      t.timestamptz :last_synced_at, comment: 'When file was last synced to HTM'
-      t.timestamptz :created_at, default: -> { 'CURRENT_TIMESTAMP' }
-      t.timestamptz :updated_at, default: -> { 'CURRENT_TIMESTAMP' }
+require_relative '../../lib/htm/migration'
+
+class CreateFileSources < HTM::Migration
+  def up
+    create_table(:file_sources) do
+      primary_key :id
+      String :file_path, text: true, null: false
+      String :file_hash, size: 64
+      DateTime :mtime
+      Integer :file_size
+      column :frontmatter, :jsonb, default: Sequel.lit("'{}'::jsonb")
+      DateTime :last_synced_at
+      DateTime :created_at, default: Sequel::CURRENT_TIMESTAMP
+      DateTime :updated_at, default: Sequel::CURRENT_TIMESTAMP
     end
 
-    add_index :file_sources, :file_path, unique: true, name: 'idx_file_sources_path_unique'
-    add_index :file_sources, :file_hash, name: 'idx_file_sources_hash'
-    add_index :file_sources, :last_synced_at, name: 'idx_file_sources_last_synced'
+    add_index :file_sources, :file_path, unique: true, name: :idx_file_sources_path_unique
+    add_index :file_sources, :file_hash, name: :idx_file_sources_hash
+    add_index :file_sources, :last_synced_at, name: :idx_file_sources_last_synced
+
+    run "COMMENT ON TABLE file_sources IS 'Source file metadata for loaded documents'"
+    run "COMMENT ON COLUMN file_sources.file_path IS 'Absolute path to source file'"
+    run "COMMENT ON COLUMN file_sources.file_hash IS 'SHA-256 hash of file content'"
+    run "COMMENT ON COLUMN file_sources.mtime IS 'File modification time'"
+    run "COMMENT ON COLUMN file_sources.file_size IS 'File size in bytes'"
+    run "COMMENT ON COLUMN file_sources.frontmatter IS 'Parsed YAML frontmatter'"
+    run "COMMENT ON COLUMN file_sources.last_synced_at IS 'When file was last synced to HTM'"
 
     # LZ4 compression for better read performance on JSONB column
-    execute <<~SQL
-      ALTER TABLE file_sources ALTER COLUMN frontmatter SET COMPRESSION lz4;
-    SQL
+    run "ALTER TABLE file_sources ALTER COLUMN frontmatter SET COMPRESSION lz4"
+  end
+
+  def down
+    drop_table(:file_sources)
   end
 end

data/db/migrate/00004_create_nodes.rb

@@ -1,82 +1,94 @@
 # frozen_string_literal: true
 
-class CreateNodes < ActiveRecord::Migration[7.1]
-  def change
-    create_table :nodes, comment: 'Core memory storage for conversation messages and context' do |t|
-      t.text :content, null: false, comment: 'The conversation message/utterance content'
-      t.integer :access_count, default: 0, null: false, comment: 'Number of times this node has been accessed/retrieved'
-      t.timestamptz :created_at, default: -> { 'CURRENT_TIMESTAMP' }, comment: 'When this memory was created'
-      t.timestamptz :updated_at, default: -> { 'CURRENT_TIMESTAMP' }, comment: 'When this memory was last modified'
-      t.timestamptz :last_accessed, default: -> { 'CURRENT_TIMESTAMP' }, comment: 'When this memory was last accessed'
-      t.integer :token_count, comment: 'Number of tokens in the content (for context budget management)'
-      t.vector :embedding, limit: 2000, comment: 'Vector embedding (max 2000 dimensions) for semantic search'
-      t.integer :embedding_dimension, comment: 'Actual number of dimensions used in the embedding vector (max 2000)'
-      t.string :content_hash, limit: 64, comment: 'SHA-256 hash of content for deduplication'
-      t.timestamptz :deleted_at, comment: 'Soft delete timestamp - node is considered deleted when set'
-      t.bigint :source_id, comment: 'Reference to source file (for file-loaded nodes)'
-      t.integer :chunk_position, comment: 'Position within source file (0-indexed)'
-      t.jsonb :metadata, default: {}, null: false, comment: 'Flexible metadata storage (memory_type, importance, source, etc.)'
+require_relative '../../lib/htm/migration'
+
+class CreateNodes < HTM::Migration
+  def up
+    create_table(:nodes) do
+      primary_key :id
+      String :content, text: true, null: false
+      Integer :access_count, default: 0, null: false
+      DateTime :created_at, default: Sequel::CURRENT_TIMESTAMP
+      DateTime :updated_at, default: Sequel::CURRENT_TIMESTAMP
+      DateTime :last_accessed, default: Sequel::CURRENT_TIMESTAMP
+      Integer :token_count
+      column :embedding, 'vector(2000)'
+      Integer :embedding_dimension
+      String :content_hash, size: 64
+      DateTime :deleted_at
+      Bignum :source_id
+      Integer :chunk_position
+      column :metadata, :jsonb, default: Sequel.lit("'{}'::jsonb"), null: false
     end
 
     # Basic indexes for common queries
-    add_index :nodes, :created_at, name: 'idx_nodes_created_at'
-    add_index :nodes, :updated_at, name: 'idx_nodes_updated_at'
-    add_index :nodes, :last_accessed, name: 'idx_nodes_last_accessed'
-    add_index :nodes, :access_count, name: 'idx_nodes_access_count'
-    add_index :nodes, :content_hash, unique: true, name: 'idx_nodes_content_hash_unique'
-    add_index :nodes, :deleted_at, name: 'idx_nodes_deleted_at'
-    add_index :nodes, :source_id, name: 'idx_nodes_source_id'
-    add_index :nodes, [:source_id, :chunk_position], name: 'idx_nodes_source_chunk_position'
+    add_index :nodes, :created_at, name: :idx_nodes_created_at
+    add_index :nodes, :updated_at, name: :idx_nodes_updated_at
+    add_index :nodes, :last_accessed, name: :idx_nodes_last_accessed
+    add_index :nodes, :access_count, name: :idx_nodes_access_count
+    add_index :nodes, :content_hash, unique: true, name: :idx_nodes_content_hash_unique
+    add_index :nodes, :deleted_at, name: :idx_nodes_deleted_at
+    add_index :nodes, :source_id, name: :idx_nodes_source_id
+    add_index :nodes, [:source_id, :chunk_position], name: :idx_nodes_source_chunk_position
+
+    # Comments
+    run "COMMENT ON TABLE nodes IS 'Core memory storage for conversation messages and context'"
+    run "COMMENT ON COLUMN nodes.content IS 'The conversation message/utterance content'"
+    run "COMMENT ON COLUMN nodes.access_count IS 'Number of times this node has been accessed/retrieved'"
+    run "COMMENT ON COLUMN nodes.created_at IS 'When this memory was created'"
+    run "COMMENT ON COLUMN nodes.updated_at IS 'When this memory was last modified'"
+    run "COMMENT ON COLUMN nodes.last_accessed IS 'When this memory was last accessed'"
+    run "COMMENT ON COLUMN nodes.token_count IS 'Number of tokens in the content (for context budget management)'"
+    run "COMMENT ON COLUMN nodes.embedding IS 'Vector embedding (max 2000 dimensions) for semantic search'"
+    run "COMMENT ON COLUMN nodes.embedding_dimension IS 'Actual number of dimensions used in the embedding vector (max 2000)'"
+    run "COMMENT ON COLUMN nodes.content_hash IS 'SHA-256 hash of content for deduplication'"
+    run "COMMENT ON COLUMN nodes.deleted_at IS 'Soft delete timestamp - node is considered deleted when set'"
+    run "COMMENT ON COLUMN nodes.source_id IS 'Reference to source file (for file-loaded nodes)'"
+    run "COMMENT ON COLUMN nodes.chunk_position IS 'Position within source file (0-indexed)'"
+    run "COMMENT ON COLUMN nodes.metadata IS 'Flexible metadata storage (memory_type, importance, source, etc.)'"
 
     # Partial index for efficiently querying non-deleted nodes
-    add_index :nodes, :created_at, name: 'idx_nodes_not_deleted_created_at', where: 'deleted_at IS NULL'
+    run "CREATE INDEX idx_nodes_not_deleted_created_at ON nodes (created_at) WHERE deleted_at IS NULL"
 
     # GIN index for JSONB metadata queries
-    add_index :nodes, :metadata, using: :gin, name: 'idx_nodes_metadata'
+    run "CREATE INDEX idx_nodes_metadata ON nodes USING gin(metadata)"
 
     # Vector similarity search index (HNSW for better performance)
-    execute <<-SQL
+    run <<-SQL
       CREATE INDEX idx_nodes_embedding ON nodes
         USING hnsw (embedding vector_cosine_ops)
         WITH (m = 16, ef_construction = 64)
     SQL
 
     # Full-text search on conversation content
-    execute <<-SQL
-      CREATE INDEX idx_nodes_content_gin ON nodes
-        USING gin(to_tsvector('english', content))
-    SQL
+    run "CREATE INDEX idx_nodes_content_gin ON nodes USING gin(to_tsvector('english', content))"
 
     # Trigram indexes for fuzzy matching on conversation content
-    execute <<-SQL
-      CREATE INDEX idx_nodes_content_trgm ON nodes
-        USING gin(content gin_trgm_ops)
-    SQL
+    run "CREATE INDEX idx_nodes_content_trgm ON nodes USING gin(content gin_trgm_ops)"
 
     # Check constraint for embedding dimensions
-    execute <<-SQL
+    run <<-SQL
       ALTER TABLE nodes ADD CONSTRAINT check_embedding_dimension
         CHECK (embedding_dimension IS NULL OR (embedding_dimension > 0 AND embedding_dimension <= 2000))
     SQL
 
     # Partial index for active (non-deleted) node queries
-    add_index :nodes, :id,
-              name: 'idx_nodes_active',
-              where: 'deleted_at IS NULL'
+    run "CREATE INDEX idx_nodes_active ON nodes (id) WHERE deleted_at IS NULL"
 
     # Composite index for embedding-based searches on active nodes
-    execute <<-SQL
-      CREATE INDEX idx_nodes_active_with_embedding ON nodes (id)
-        WHERE deleted_at IS NULL AND embedding IS NOT NULL
-    SQL
+    run "CREATE INDEX idx_nodes_active_with_embedding ON nodes (id) WHERE deleted_at IS NULL AND embedding IS NOT NULL"
 
     # LZ4 compression for better read performance
-    execute <<-SQL
-      ALTER TABLE nodes ALTER COLUMN metadata SET COMPRESSION lz4;
-      ALTER TABLE nodes ALTER COLUMN content SET COMPRESSION lz4;
-    SQL
+    run "ALTER TABLE nodes ALTER COLUMN metadata SET COMPRESSION lz4"
+    run "ALTER TABLE nodes ALTER COLUMN content SET COMPRESSION lz4"
 
     # Foreign key to file_sources table
-    add_foreign_key :nodes, :file_sources, column: :source_id, on_delete: :nullify
+    alter_table(:nodes) do
+      add_foreign_key [:source_id], :file_sources, on_delete: :set_null
+    end
+  end
+
+  def down
+    drop_table(:nodes)
   end
 end

data/db/migrate/00005_create_tags.rb

@@ -1,20 +1,32 @@
 # frozen_string_literal: true
 
-class CreateTags < ActiveRecord::Migration[7.1]
-  def change
-    create_table :tags, comment: 'Unique tag names for categorization' do |t|
-      t.text :name, null: false, comment: 'Hierarchical tag in format: root:level1:level2 (e.g., database:postgresql:timescaledb)'
-      t.timestamptz :created_at, default: -> { 'CURRENT_TIMESTAMP' }, comment: 'When this tag was created'
-      t.timestamptz :deleted_at, comment: 'Soft delete timestamp'
+require_relative '../../lib/htm/migration'
+
+class CreateTags < HTM::Migration
+  def up
+    create_table(:tags) do
+      primary_key :id
+      String :name, text: true, null: false
+      DateTime :created_at, default: Sequel::CURRENT_TIMESTAMP
+      DateTime :deleted_at
     end
 
-    add_index :tags, :name, unique: true, name: 'idx_tags_name_unique'
-    add_index :tags, :name, using: :btree, opclass: :text_pattern_ops, name: 'idx_tags_name_pattern'
-    add_index :tags, :deleted_at, name: 'idx_tags_deleted_at'
+    add_index :tags, :name, unique: true, name: :idx_tags_name_unique
+    add_index :tags, :deleted_at, name: :idx_tags_deleted_at
+
+    # Pattern matching index for prefix queries
+    run "CREATE INDEX idx_tags_name_pattern ON tags USING btree (name text_pattern_ops)"
 
     # GIN trigram index for fuzzy search (typo-tolerant queries)
-    execute <<~SQL
-      CREATE INDEX idx_tags_name_trgm ON tags USING gin(name gin_trgm_ops);
-    SQL
+    run "CREATE INDEX idx_tags_name_trgm ON tags USING gin(name gin_trgm_ops)"
+
+    run "COMMENT ON TABLE tags IS 'Unique tag names for categorization'"
+    run "COMMENT ON COLUMN tags.name IS 'Hierarchical tag in format: root:level1:level2 (e.g., database:postgresql:timescaledb)'"
+    run "COMMENT ON COLUMN tags.created_at IS 'When this tag was created'"
+    run "COMMENT ON COLUMN tags.deleted_at IS 'Soft delete timestamp'"
+  end
+
+  def down
+    drop_table(:tags)
   end
 end