htm 0.0.14 → 0.0.17

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (140) hide show
  1. checksums.yaml +4 -4
  2. data/.envrc +1 -0
  3. data/CHANGELOG.md +100 -0
  4. data/README.md +107 -1412
  5. data/bin/htm_mcp +31 -0
  6. data/config/database.yml +7 -4
  7. data/db/migrate/00003_create_file_sources.rb +5 -0
  8. data/db/migrate/00004_create_nodes.rb +17 -0
  9. data/db/migrate/00005_create_tags.rb +7 -0
  10. data/db/migrate/00006_create_node_tags.rb +2 -0
  11. data/db/migrate/00007_create_robot_nodes.rb +7 -0
  12. data/db/schema.sql +41 -29
  13. data/docs/api/yard/HTM/Configuration.md +54 -0
  14. data/docs/api/yard/HTM/Database.md +13 -10
  15. data/docs/api/yard/HTM/EmbeddingService.md +5 -1
  16. data/docs/api/yard/HTM/LongTermMemory.md +18 -277
  17. data/docs/api/yard/HTM/PropositionError.md +18 -0
  18. data/docs/api/yard/HTM/PropositionService.md +66 -0
  19. data/docs/api/yard/HTM/QueryCache.md +88 -0
  20. data/docs/api/yard/HTM/RobotGroup.md +481 -0
  21. data/docs/api/yard/HTM/SqlBuilder.md +108 -0
  22. data/docs/api/yard/HTM/TagService.md +4 -0
  23. data/docs/api/yard/HTM/Telemetry/NullInstrument.md +13 -0
  24. data/docs/api/yard/HTM/Telemetry/NullMeter.md +15 -0
  25. data/docs/api/yard/HTM/Telemetry.md +109 -0
  26. data/docs/api/yard/HTM/WorkingMemoryChannel.md +176 -0
  27. data/docs/api/yard/HTM.md +11 -23
  28. data/docs/api/yard/index.csv +102 -25
  29. data/docs/api/yard-reference.md +8 -0
  30. data/docs/assets/images/multi-provider-failover.svg +51 -0
  31. data/docs/assets/images/robot-group-architecture.svg +65 -0
  32. data/docs/database/README.md +3 -3
  33. data/docs/database/public.file_sources.svg +29 -21
  34. data/docs/database/public.node_tags.md +2 -0
  35. data/docs/database/public.node_tags.svg +53 -41
  36. data/docs/database/public.nodes.md +2 -0
  37. data/docs/database/public.nodes.svg +52 -40
  38. data/docs/database/public.robot_nodes.md +2 -0
  39. data/docs/database/public.robot_nodes.svg +30 -22
  40. data/docs/database/public.robots.svg +16 -12
  41. data/docs/database/public.tags.md +3 -0
  42. data/docs/database/public.tags.svg +41 -33
  43. data/docs/database/schema.json +66 -0
  44. data/docs/database/schema.svg +60 -48
  45. data/docs/development/index.md +13 -0
  46. data/docs/development/rake-tasks.md +1068 -0
  47. data/docs/getting-started/installation.md +31 -11
  48. data/docs/getting-started/quick-start.md +144 -155
  49. data/docs/guides/adding-memories.md +2 -3
  50. data/docs/guides/context-assembly.md +185 -184
  51. data/docs/guides/getting-started.md +154 -148
  52. data/docs/guides/index.md +7 -0
  53. data/docs/guides/long-term-memory.md +60 -92
  54. data/docs/guides/mcp-server.md +1052 -0
  55. data/docs/guides/multi-robot.md +249 -345
  56. data/docs/guides/recalling-memories.md +153 -163
  57. data/docs/guides/robot-groups.md +604 -0
  58. data/docs/guides/search-strategies.md +61 -58
  59. data/docs/guides/working-memory.md +103 -136
  60. data/docs/index.md +30 -26
  61. data/docs/multi_framework_support.md +2 -2
  62. data/examples/mcp_client.rb +2 -2
  63. data/examples/rails_app/.gitignore +2 -0
  64. data/examples/rails_app/Gemfile +22 -0
  65. data/examples/rails_app/Gemfile.lock +438 -0
  66. data/examples/rails_app/Procfile.dev +1 -0
  67. data/examples/rails_app/README.md +98 -0
  68. data/examples/rails_app/Rakefile +5 -0
  69. data/examples/rails_app/app/assets/stylesheets/application.css +83 -0
  70. data/examples/rails_app/app/assets/stylesheets/inter-font.css +6 -0
  71. data/examples/rails_app/app/controllers/application_controller.rb +19 -0
  72. data/examples/rails_app/app/controllers/dashboard_controller.rb +27 -0
  73. data/examples/rails_app/app/controllers/files_controller.rb +205 -0
  74. data/examples/rails_app/app/controllers/memories_controller.rb +102 -0
  75. data/examples/rails_app/app/controllers/robots_controller.rb +44 -0
  76. data/examples/rails_app/app/controllers/search_controller.rb +46 -0
  77. data/examples/rails_app/app/controllers/tags_controller.rb +30 -0
  78. data/examples/rails_app/app/javascript/application.js +4 -0
  79. data/examples/rails_app/app/javascript/controllers/application.js +9 -0
  80. data/examples/rails_app/app/javascript/controllers/index.js +6 -0
  81. data/examples/rails_app/app/views/dashboard/index.html.erb +123 -0
  82. data/examples/rails_app/app/views/files/index.html.erb +108 -0
  83. data/examples/rails_app/app/views/files/new.html.erb +321 -0
  84. data/examples/rails_app/app/views/files/show.html.erb +130 -0
  85. data/examples/rails_app/app/views/layouts/application.html.erb +124 -0
  86. data/examples/rails_app/app/views/memories/_memory_card.html.erb +51 -0
  87. data/examples/rails_app/app/views/memories/deleted.html.erb +62 -0
  88. data/examples/rails_app/app/views/memories/edit.html.erb +35 -0
  89. data/examples/rails_app/app/views/memories/index.html.erb +81 -0
  90. data/examples/rails_app/app/views/memories/new.html.erb +71 -0
  91. data/examples/rails_app/app/views/memories/show.html.erb +126 -0
  92. data/examples/rails_app/app/views/robots/index.html.erb +106 -0
  93. data/examples/rails_app/app/views/robots/new.html.erb +36 -0
  94. data/examples/rails_app/app/views/robots/show.html.erb +79 -0
  95. data/examples/rails_app/app/views/search/index.html.erb +184 -0
  96. data/examples/rails_app/app/views/shared/_navbar.html.erb +52 -0
  97. data/examples/rails_app/app/views/shared/_stat_card.html.erb +52 -0
  98. data/examples/rails_app/app/views/tags/index.html.erb +131 -0
  99. data/examples/rails_app/app/views/tags/show.html.erb +67 -0
  100. data/examples/rails_app/bin/dev +8 -0
  101. data/examples/rails_app/bin/rails +4 -0
  102. data/examples/rails_app/bin/rake +4 -0
  103. data/examples/rails_app/config/application.rb +33 -0
  104. data/examples/rails_app/config/boot.rb +5 -0
  105. data/examples/rails_app/config/database.yml +15 -0
  106. data/examples/rails_app/config/environment.rb +5 -0
  107. data/examples/rails_app/config/importmap.rb +7 -0
  108. data/examples/rails_app/config/routes.rb +38 -0
  109. data/examples/rails_app/config/tailwind.config.js +35 -0
  110. data/examples/rails_app/config.ru +5 -0
  111. data/examples/rails_app/log/.keep +0 -0
  112. data/examples/rails_app/tmp/local_secret.txt +1 -0
  113. data/examples/robot_groups/robot_worker.rb +1 -2
  114. data/examples/robot_groups/same_process.rb +1 -4
  115. data/lib/htm/active_record_config.rb +2 -5
  116. data/lib/htm/configuration.rb +35 -2
  117. data/lib/htm/database.rb +3 -6
  118. data/lib/htm/mcp/cli.rb +333 -0
  119. data/lib/htm/mcp/group_tools.rb +476 -0
  120. data/lib/htm/mcp/resources.rb +89 -0
  121. data/lib/htm/mcp/server.rb +98 -0
  122. data/lib/htm/mcp/tools.rb +488 -0
  123. data/lib/htm/models/file_source.rb +5 -3
  124. data/lib/htm/railtie.rb +0 -4
  125. data/lib/htm/robot_group.rb +721 -0
  126. data/lib/htm/tasks.rb +7 -4
  127. data/lib/htm/version.rb +1 -1
  128. data/lib/htm/working_memory_channel.rb +250 -0
  129. data/lib/htm.rb +2 -0
  130. data/lib/tasks/htm.rake +6 -9
  131. data/mkdocs.yml +2 -0
  132. metadata +75 -11
  133. data/bin/htm_mcp.rb +0 -621
  134. data/db/migrate/00009_add_working_memory_to_robot_nodes.rb +0 -12
  135. data/db/migrate/00010_add_soft_delete_to_associations.rb +0 -29
  136. data/db/migrate/00011_add_performance_indexes.rb +0 -21
  137. data/db/migrate/00012_add_tags_trigram_index.rb +0 -18
  138. data/db/migrate/00013_enable_lz4_compression.rb +0 -43
  139. data/examples/robot_groups/lib/robot_group.rb +0 -419
  140. data/examples/robot_groups/lib/working_memory_channel.rb +0 -140
data/README.md CHANGED
@@ -1,1492 +1,187 @@
1
1
  <div align="center">
2
2
  <h1>HTM</h1>
3
- <!-- img src="docs/assets/images/htm.jpg" alt="HTM - Hierarchical Temporal Memory" width="600" -->
4
3
  <img src="docs/assets/images/htm_demo.gif" alt="Tree of Knowledge is Growing" width="400">
5
4
 
6
- <p>A hierarchical and temporal system for encoding, storing, and retrieving information—operating across varying levels of abstraction (from simple to detailed concepts and their relationships) and across time (from the present to the past).<br/>
5
+ <p><strong>Give your AI tools a shared, persistent memory.</strong></p>
6
+
7
+ <p>
8
+ <a href="https://rubygems.org/gems/htm"><img src="https://img.shields.io/gem/v/htm.svg" alt="Gem Version"></a>
9
+ <a href="https://github.com/madbomber/htm/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="License"></a>
7
10
  </p>
8
11
  </div>
9
12
 
10
- <br/><br/>
11
-
12
- > [!CAUTION]
13
- > This library is under active development and experimentation. APIs and features may change without notice. Not recommended for production use yet... and the documentation may lie to you!
14
- > <br /><br/>
15
- > Apologies to Jeff Hawkins for using his term in such a macro-superficial way.
16
-
17
-
18
- ## What does it mean?
19
-
20
- - **Hierarchical**: operates across multiple levels of abstraction, from simple concepts to detailed relationships
21
- - **Temporal**: functions across time, from the present moment to historical data
22
- - **Memory function**: encodes, stores, and retrieves information
23
-
24
- **HTM**: a hierarchical and temporal memory system that organizes and recalls information at multiple levels of detail over extended timeframes.
25
-
26
- ## Features
27
-
28
- - **Client-Side Embeddings**
29
- - Automatic embedding generation before database insertion
30
- - Uses the [ruby_llm](https://ruby_llm.com) gem for LLM access
31
- - Configurable embedding providers and models
13
+ <br/>
32
14
 
33
- - **Two-Tier Memory Architecture**
34
- - Working Memory: Token-limited active context for immediate LLM use
35
- - Long-term Memory: Durable PostgreSQL storage
15
+ ## The Problem
36
16
 
37
- - **Never Forgets (Unless Told)**
38
- - All memories persist in long-term storage
39
- - Only explicit `forget()` commands delete data
40
- - Working memory evicts to long-term, never deletes
17
+ Your AI assistants are brilliant but forgetful. Claude Desktop doesn't remember what Claude Code learned. Your custom chatbot can't recall what happened yesterday. Every tool starts fresh, every session.
41
18
 
42
- - **RAG-Based Retrieval**
43
- - Vector similarity search (pgvector)
44
- - Full-text search (PostgreSQL)
45
- - Hybrid search (combines both)
46
- - Temporal filtering ("last week", date ranges)
47
- - Variable embedding dimensions (384 to 3072)
19
+ **HTM fixes this.**
48
20
 
49
- - **Hive Mind**
50
- - All robots share global memory
51
- - Cross-robot context awareness
52
- - Track which robot said what
21
+ Connect all your MCP-enabled tools to the same long-term memory. What one robot learns, all robots remember. Context persists across sessions, tools, and time.
53
22
 
54
- - **LLM-Driven Tag Extraction**
55
- - Automatic hierarchical tag extraction from content
56
- - Tags in colon-delimited format (e.g., `database:postgresql:performance`)
57
- - LLM-powered asynchronous processing
58
- - Enables both structured navigation and semantic discovery
59
- - Complements vector embeddings (symbolic + sub-symbolic retrieval)
23
+ ## What is HTM?
60
24
 
61
- - **Knowledge Graph**
62
- - Tag-based categorization
63
- - Hierarchical tag structures
25
+ - **Hierarchical**: Organizes information from simple concepts to detailed relationships
26
+ - **Temporal**: Retrieves memories across time—from moments ago to months past
27
+ - **Memory**: Encodes, stores, and recalls information with semantic understanding
64
28
 
65
- - **File Loading**
66
- - Load markdown files into long-term memory
67
- - Automatic paragraph-based chunking
68
- - Source file tracking with re-sync support
69
- - YAML frontmatter extraction as metadata
29
+ HTM is a Ruby gem that provides durable, shared memory for LLM-powered applications.
70
30
 
71
- - **Telemetry (OpenTelemetry)**
72
- - Optional metrics collection via OpenTelemetry
73
- - Zero overhead when disabled (null object pattern)
74
- - Works with 50+ backends (Jaeger, Prometheus, Datadog, etc.)
75
- - Tracks job latency, search performance, and cache effectiveness
31
+ ## Key Capabilities
76
32
 
77
- ## Installation
78
-
79
- Add this line to your application's Gemfile:
80
-
81
- ```ruby
82
- gem 'htm'
83
- ```
84
-
85
- And then execute:
86
-
87
- ```bash
88
- bundle install
89
- ```
33
+ ### MCP Server — Connect Any AI Tool
90
34
 
91
- Or install it yourself as:
35
+ HTM includes a Model Context Protocol server with 23 tools for memory management. Connect Claude Desktop, Claude Code, AIA, or any MCP-compatible client:
92
36
 
93
- ```bash
94
- gem install htm
95
- ```
96
-
97
- ## Setup
98
-
99
- ### 1. Database Configuration
100
-
101
- HTM uses PostgreSQL for durable long-term memory storage. Set up your database connection via environment variables:
102
-
103
- ```bash
104
- # Preferred: Full connection URL
105
- export HTM_DBURL="postgresql://user:password@host:port/dbname?sslmode=require"
106
-
107
- # Alternative: Individual parameters
108
- export HTM_DBNAME="htm_production"
109
- export HTM_DBUSER="postgres"
110
- export HTM_DBPASS="your_password"
111
- export HTM_DBHOST="localhost"
112
- export HTM_DBPORT="5432"
113
- ```
114
-
115
- See the [Environment Variables](#environment-variables) section below for complete details.
116
-
117
- ### 2. Configure LLM Providers
118
-
119
- HTM uses LLM providers for embedding generation and tag extraction. By default, it uses Ollama (local).
120
-
121
- **Start Ollama (Default Configuration)**:
122
- ```bash
123
- # Install Ollama
124
- curl https://ollama.ai/install.sh | sh
125
-
126
- # Start Ollama and pull models
127
- ollama serve
128
- ollama pull nomic-embed-text # For embeddings (768 dimensions)
129
- ollama pull llama3 # For tag extraction
130
- ```
131
-
132
- **Configure HTM** (optional - uses defaults if not configured):
133
- ```ruby
134
- require 'htm'
135
-
136
- # Use defaults (Ollama with nomic-embed-text and llama3)
137
- HTM.configure
138
-
139
- # Or customize providers
140
- HTM.configure do |config|
141
- # Embedding configuration
142
- config.embedding_provider = :ollama # or :openai
143
- config.embedding_model = 'nomic-embed-text'
144
- config.embedding_dimensions = 768
145
- config.ollama_url = 'http://localhost:11434'
146
-
147
- # Tag extraction configuration
148
- config.tag_provider = :ollama # or :openai
149
- config.tag_model = 'llama3'
150
-
151
- # Logger configuration (optional)
152
- config.logger = Logger.new($stdout)
153
- config.logger.level = Logger::INFO
154
-
155
- # Custom embedding generator (advanced)
156
- config.embedding_generator = ->(text) {
157
- # Your custom implementation
158
- # Must return Array<Float>
159
- }
160
-
161
- # Custom tag extractor (advanced)
162
- config.tag_extractor = ->(text, ontology) {
163
- # Your custom implementation
164
- # Must return Array<String>
37
+ ```json
38
+ {
39
+ "mcpServers": {
40
+ "htm": {
41
+ "command": "htm_mcp",
42
+ "env": { "HTM_DBURL": "postgresql://localhost/htm" }
43
+ }
165
44
  }
166
-
167
- # Token counter (optional, defaults to Tiktoken)
168
- config.token_counter = ->(text) {
169
- # Your custom implementation
170
- # Must return Integer
171
- }
172
- end
173
- ```
174
-
175
- See the [Configuration](#configuration) section below for complete details.
176
-
177
- ### 3. Initialize Database Schema
178
-
179
- **Using Rake Tasks (Recommended)**:
180
-
181
- HTM provides comprehensive rake tasks for database management. Add this to your application's `Rakefile`:
182
-
183
- ```ruby
184
- require 'htm/tasks'
185
- ```
186
-
187
- Then use the tasks:
188
-
189
- ```bash
190
- # Set up database schema and run migrations
191
- rake htm:db:setup
192
-
193
- # Verify connection
194
- rake htm:db:test
195
-
196
- # Check database status
197
- rake htm:db:info
198
- ```
199
-
200
- **Or programmatically**:
201
-
202
- ```ruby
203
- require 'htm'
204
-
205
- # Run once to set up database schema
206
- HTM::Database.setup
207
- ```
208
-
209
- **Or from command line**:
210
-
211
- ```bash
212
- ruby -r ./lib/htm -e "HTM::Database.setup"
45
+ }
213
46
  ```
214
47
 
215
- See [Using HTM Rake Tasks in Your Application](docs/using_rake_tasks_in_your_app.md) for complete integration guide.
216
-
217
- ### 4. Verify Setup
48
+ The `htm_mcp` executable includes CLI commands for database management:
218
49
 
219
50
  ```bash
220
- rake htm:db:test # Using rake tasks
221
- # or
222
- ruby test_connection.rb
51
+ htm_mcp setup # Initialize database schema
52
+ htm_mcp verify # Check connection and migrations
53
+ htm_mcp stats # Show memory statistics
54
+ htm_mcp help # Full help with environment variables
55
+ htm_mcp # Start MCP server (default)
223
56
  ```
224
57
 
225
- See [docs/setup_local_database.md](docs/setup_local_database.md) for detailed local database setup instructions.
58
+ Now your AI assistant can remember and recall across sessions:
59
+ - Store decisions, preferences, and context
60
+ - Search memories with natural language
61
+ - Build knowledge that compounds over time
226
62
 
227
- ## Usage
63
+ ### Hive Mind — Shared Knowledge Across Robots
228
64
 
229
- ### Basic Example
65
+ All robots share the same global memory. What one learns, all can access:
230
66
 
231
67
  ```ruby
232
- require 'htm'
233
- require 'ruby_llm'
234
- require 'logger'
235
-
236
- # Configure HTM with RubyLLM for embeddings and tag extraction
237
- HTM.configure do |config|
238
- # Configure logger (optional - uses STDOUT at INFO level if not provided)
239
- config.logger = Logger.new($stdout)
240
- config.logger.level = Logger::INFO
241
-
242
- # Configure embedding generation using RubyLLM with Ollama
243
- config.embedding_provider = :ollama
244
- config.embedding_model = 'nomic-embed-text'
245
- config.embedding_dimensions = 768
246
- config.ollama_url = ENV['OLLAMA_URL'] || 'http://localhost:11434'
247
-
248
- # Configure tag extraction using RubyLLM with Ollama
249
- config.tag_provider = :ollama
250
- config.tag_model = 'llama3'
68
+ # Claude Desktop remembers a user preference
69
+ claude_desktop.remember("User prefers dark mode and concise responses")
251
70
 
252
- # Apply configuration (sets up default RubyLLM implementations)
253
- config.reset_to_defaults
254
- end
255
-
256
- # Initialize HTM for your robot
257
- htm = HTM.new(
258
- robot_name: "Code Helper",
259
- working_memory_size: 128_000 # tokens
260
- )
261
-
262
- # Remember information - embeddings and tags generated asynchronously
263
- node_id = htm.remember(
264
- "We decided to use PostgreSQL for HTM storage",
265
- source: "architect"
266
- )
267
-
268
- # Recall from the past (uses semantic search with embeddings)
269
- memories = htm.recall(
270
- "database decisions",
271
- timeframe: "last week",
272
- strategy: :hybrid # Combines vector + full-text search
273
- )
274
-
275
- # Forget (explicit deletion only)
276
- htm.forget(node_id, confirm: :confirmed)
71
+ # Later, Claude Code can recall it
72
+ claude_code.recall("user preferences")
73
+ # => "User prefers dark mode and concise responses"
277
74
  ```
278
75
 
279
- ### Loading Files
280
-
281
- HTM can load text-based files (currently markdown) into long-term memory with automatic chunking and source tracking.
76
+ Every tool builds on what came before. No more repeating yourself.
282
77
 
283
- ```ruby
284
- htm = HTM.new(robot_name: "Document Loader")
285
-
286
- # Load a single markdown file
287
- result = htm.load_file("docs/guide.md")
288
- # => { file_source_id: 1, chunks_created: 5, chunks_updated: 0, chunks_deleted: 0 }
78
+ ### Long-Term Memory — Never Forget
289
79
 
290
- # Load all markdown files in a directory
291
- results = htm.load_directory("docs/", pattern: "**/*.md")
80
+ Two-tier architecture ensures nothing is lost:
292
81
 
293
- # Get nodes from a specific file
294
- nodes = htm.nodes_from_file("docs/guide.md")
295
-
296
- # Unload a file (soft deletes chunks)
297
- htm.unload_file("docs/guide.md")
298
- ```
82
+ - **Working Memory**: Token-limited context for immediate use
83
+ - **Long-Term Memory**: Durable PostgreSQL storage with vector search
299
84
 
300
- **Features:**
301
- - **Paragraph chunking**: Text split by blank lines, code blocks preserved
302
- - **Source tracking**: Files tracked with mtime for automatic re-sync
303
- - **YAML frontmatter**: Extracted and stored as metadata
304
- - **Duplicate detection**: Content hash prevents duplicate nodes
85
+ Memories persist forever unless explicitly deleted. Working memory evicts to long-term storage—it never deletes.
305
86
 
306
- **Rake tasks:**
307
- ```bash
308
- rake 'htm:files:load[docs/guide.md]' # Load a single file
309
- rake 'htm:files:load_dir[docs/]' # Load all markdown files from directory
310
- rake htm:files:list # List all loaded file sources
311
- rake htm:files:sync # Sync all files (reload changed)
312
- rake htm:files:stats # Show file loading statistics
313
- ```
314
-
315
- ### Automatic Tag Extraction
316
-
317
- HTM automatically extracts hierarchical tags from content using LLM analysis. Tags are inferred from the content itself - you never specify them manually.
318
-
319
- **Example:**
320
87
  ```ruby
321
- htm.remember(
322
- "User prefers dark mode for all interfaces",
323
- source: "user"
324
- )
325
- # Tags like "preference", "ui", "dark-mode" may be auto-extracted by the LLM
326
- # The specific tags depend on the LLM's analysis of the content
327
- ```
88
+ htm.remember("PostgreSQL with pgvector handles our vector search")
328
89
 
329
- ### Async Job Processing
330
-
331
- HTM automatically generates embeddings and extracts tags asynchronously in background jobs. This avoids blocking the main request path.
332
-
333
- **Monitor job processing:**
334
- ```bash
335
- # Show statistics for nodes and async processing
336
- rake htm:jobs:stats
337
-
338
- # Output:
339
- # Total nodes: 150
340
- # Nodes with embeddings: 145 (96.7%)
341
- # Nodes without embeddings: 5 (3.3%)
342
- # Nodes with tags: 140 (93.3%)
343
- # Total tags in ontology: 42
344
- ```
345
-
346
- **Process pending jobs manually:**
347
- ```bash
348
- # Process all pending embedding jobs
349
- rake htm:jobs:process_embeddings
350
-
351
- # Process all pending tag extraction jobs
352
- rake htm:jobs:process_tags
353
-
354
- # Process both embeddings and tags
355
- rake htm:jobs:process_all
356
- ```
357
-
358
- **Reprocess all nodes (force regeneration):**
359
- ```bash
360
- # Regenerate embeddings for ALL nodes
361
- rake htm:jobs:reprocess_embeddings
362
-
363
- # WARNING: Prompts for confirmation
364
- ```
365
-
366
- **Show failed/stuck jobs:**
367
- ```bash
368
- # Show nodes that failed async processing (>1 hour old without embeddings/tags)
369
- rake htm:jobs:failed
370
- ```
371
-
372
- **Clear all for testing:**
373
- ```bash
374
- # Clear ALL embeddings and tags (development/testing only)
375
- rake htm:jobs:clear_all
376
-
377
- # WARNING: Prompts for confirmation
90
+ # Months later...
91
+ htm.recall("database decisions", timeframe: "last year")
92
+ # => Still there
378
93
  ```
379
94
 
380
- See `rake -T htm:jobs` for complete list of job management tasks.
381
-
382
- ## Telemetry (OpenTelemetry)
95
+ ### Robot Groups High Availability
383
96
 
384
- HTM includes optional OpenTelemetry-based metrics for production observability. Telemetry is **disabled by default** with zero overhead when off.
385
-
386
- ### Enabling Telemetry
97
+ Coordinate multiple robots with shared working memory and instant failover:
387
98
 
388
99
  ```ruby
389
- HTM.configure do |config|
390
- config.telemetry_enabled = true
391
- end
392
-
393
- # Or via environment variable
394
- # HTM_TELEMETRY_ENABLED=true
395
- ```
396
-
397
- ### Configuring the Destination
398
-
399
- HTM emits metrics via standard OpenTelemetry protocols. Configure your destination using environment variables:
400
-
401
- ```bash
402
- # Export to any OTLP-compatible backend
403
- export OTEL_METRICS_EXPORTER="otlp"
404
- export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
405
- ```
406
-
407
- ### Available Metrics
408
-
409
- | Metric | Type | Attributes | Description |
410
- |--------|------|------------|-------------|
411
- | `htm.jobs` | Counter | `job`, `status` | Job execution counts (embedding, tags) |
412
- | `htm.embedding.latency` | Histogram | `provider`, `status` | Embedding generation time (ms) |
413
- | `htm.tag.latency` | Histogram | `provider`, `status` | Tag extraction time (ms) |
414
- | `htm.search.latency` | Histogram | `strategy` | Search operation time (ms) |
415
- | `htm.cache.operations` | Counter | `operation` | Cache hits/misses |
416
-
417
- ### Compatible Backends
418
-
419
- HTM works with any OTLP-compatible observability platform:
420
-
421
- **Open Source:** Jaeger, Prometheus, Grafana Tempo/Mimir, SigNoz, Uptrace
422
-
423
- **Commercial:** Datadog, New Relic, Honeycomb, Splunk, Dynatrace, AWS X-Ray, Google Cloud Trace, Azure Monitor
424
-
425
- ### Optional Dependencies
426
-
427
- Users who want telemetry should add these gems:
428
-
429
- ```ruby
430
- gem 'opentelemetry-sdk'
431
- gem 'opentelemetry-metrics-sdk'
432
- gem 'opentelemetry-exporter-otlp' # For OTLP export
433
- ```
434
-
435
- ### Design
436
-
437
- HTM uses a **null object pattern** for telemetry. When disabled or when the SDK is not installed:
438
- - All metric operations are no-ops
439
- - Zero runtime overhead
440
- - No errors or exceptions
441
-
442
- See [docs/telemetry.md](docs/telemetry.md) for detailed configuration and usage examples.
443
-
444
- ## Configuration
445
-
446
- HTM uses dependency injection for LLM access, allowing you to configure embedding generation, tag extraction, logging, and token counting.
447
-
448
- ### Default Configuration
449
-
450
- By default, HTM uses:
451
- - **Embedding Provider**: Ollama (local) with `nomic-embed-text` model (768 dimensions)
452
- - **Tag Provider**: Ollama (local) with `llama3` model
453
- - **Logger**: Ruby's standard Logger to STDOUT at INFO level
454
- - **Token Counter**: Tiktoken with GPT-3.5-turbo encoding
455
-
456
- ```ruby
457
- require 'htm'
458
-
459
- # Use defaults
460
- HTM.configure # or omit this - configuration is lazy-loaded
461
- ```
462
-
463
- ### Custom Configuration
464
-
465
- Configure HTM before creating instances:
466
-
467
- #### Using Ollama (Default)
468
-
469
- ```ruby
470
- HTM.configure do |config|
471
- # Embedding configuration
472
- config.embedding_provider = :ollama
473
- config.embedding_model = 'nomic-embed-text' # 768 dimensions
474
- config.embedding_dimensions = 768
475
- config.ollama_url = 'http://localhost:11434'
476
-
477
- # Tag extraction configuration
478
- config.tag_provider = :ollama
479
- config.tag_model = 'llama3'
480
-
481
- # Logger configuration
482
- config.logger = Logger.new($stdout)
483
- config.logger.level = Logger::INFO
484
- end
485
- ```
486
-
487
- **Ollama Setup:**
488
- ```bash
489
- # Install Ollama
490
- curl https://ollama.ai/install.sh | sh
491
-
492
- # Pull models
493
- ollama pull nomic-embed-text # For embeddings
494
- ollama pull llama3 # For tag extraction
495
-
496
- # Verify Ollama is running
497
- curl http://localhost:11434/api/version
498
- ```
499
-
500
- #### Using OpenAI
501
-
502
- ```ruby
503
- HTM.configure do |config|
504
- # Embedding configuration (OpenAI)
505
- config.embedding_provider = :openai
506
- config.embedding_model = 'text-embedding-3-small' # 1536 dimensions
507
- config.embedding_dimensions = 1536
508
-
509
- # Tag extraction (can mix providers)
510
- config.tag_provider = :openai
511
- config.tag_model = 'gpt-4'
512
-
513
- # Logger
514
- config.logger = Rails.logger # Use Rails logger
515
- end
516
- ```
517
-
518
- **OpenAI Setup:**
519
- ```bash
520
- # Set your OpenAI API key
521
- export OPENAI_API_KEY='sk-your-api-key-here'
522
- ```
523
-
524
- **Important:** The database uses pgvector with HNSW indexing, which has a maximum dimension limit of 2000. Embeddings exceeding this limit will be automatically truncated with a warning.
525
-
526
- #### Custom Providers
527
-
528
- Provide your own LLM implementations:
529
-
530
- ```ruby
531
- HTM.configure do |config|
532
- # Custom embedding generator
533
- config.embedding_generator = ->(text) {
534
- # Call your custom LLM service
535
- response = MyEmbeddingService.generate(text)
536
-
537
- # Must return Array<Float>
538
- response[:embedding] # e.g., [0.123, -0.456, ...]
539
- }
540
-
541
- # Custom tag extractor
542
- config.tag_extractor = ->(text, existing_ontology) {
543
- # Call your custom LLM service for tag extraction
544
- # existing_ontology is Array<String> of recent tags for context
545
- response = MyTagService.extract(text, context: existing_ontology)
546
-
547
- # Must return Array<String> in hierarchical format
548
- # e.g., ["ai:llm:embeddings", "database:postgresql"]
549
- response[:tags]
550
- }
551
-
552
- # Custom token counter (optional)
553
- config.token_counter = ->(text) {
554
- # Your token counting implementation
555
- # Must return Integer
556
- MyTokenizer.count(text)
557
- }
558
- end
559
- ```
560
-
561
- #### Logger Configuration
562
-
563
- Customize logging behavior:
564
-
565
- ```ruby
566
- HTM.configure do |config|
567
- # Use custom logger
568
- config.logger = Logger.new('log/htm.log')
569
- config.logger.level = Logger::DEBUG
570
-
571
- # Or use Rails logger
572
- config.logger = Rails.logger
573
-
574
- # Or disable logging
575
- config.logger = Logger.new(IO::NULL)
576
- config.logger.level = Logger::FATAL
577
- end
578
-
579
- # Control log level via environment variable
580
- ENV['HTM_LOG_LEVEL'] = 'DEBUG' # or INFO, WARN, ERROR
581
- HTM.configure # Respects HTM_LOG_LEVEL
582
- ```
583
-
584
- #### Service Layer Architecture
585
-
586
- HTM uses a service layer to process LLM responses:
587
-
588
- - **EmbeddingService**: Calls your configured `embedding_generator`, validates responses, handles padding/truncation, and formats for storage
589
- - **TagService**: Calls your configured `tag_extractor`, parses responses (String or Array), validates format, and filters invalid tags
590
-
591
- This separation allows you to provide raw LLM access while HTM handles response processing, validation, and storage formatting.
592
-
593
- ### Recall Strategies
594
-
595
- HTM supports three retrieval strategies:
596
-
597
- ```ruby
598
- # Vector similarity search (semantic) - uses configured embedding provider
599
- memories = htm.recall(
600
- "HTM architecture",
601
- timeframe: "last week",
602
- strategy: :vector # Semantic similarity using embeddings
603
- )
604
-
605
- # Full-text search (keyword matching) - PostgreSQL full-text search with trigrams
606
- memories = htm.recall(
607
- "database performance",
608
- timeframe: "last month",
609
- strategy: :fulltext # Keyword-based matching
610
- )
611
-
612
- # Hybrid (combines both) - best of both worlds
613
- memories = htm.recall(
614
- "testing strategies",
615
- timeframe: "yesterday",
616
- strategy: :hybrid # Weighted combination of vector + full-text
617
- )
618
- ```
619
-
620
- **Strategy Details:**
621
- - `:vector` - Semantic search using pgvector cosine similarity (requires embeddings)
622
- - `:fulltext` - PostgreSQL tsvector with pg_trgm fuzzy matching
623
- - `:hybrid` - Combines both with weighted scoring (default: 70% vector, 30% full-text)
624
-
625
- ## API Reference
626
-
627
- HTM provides a minimal, focused API with only 3 core instance methods for memory operations:
628
-
629
- ### Core Memory Operations
630
-
631
- #### `remember(content, source: "", metadata: {})`
632
-
633
- Store information in memory. Embeddings and tags are automatically generated asynchronously.
634
-
635
- **Parameters:**
636
- - `content` (String, required) - The information to remember. Converted to string if nil. Returns ID of last node if empty.
637
- - `source` (String, optional) - Where the content came from (e.g., "user", "assistant", "system"). Defaults to empty string.
638
- - `metadata` (Hash, optional) - Arbitrary key-value metadata stored as JSONB. Keys must be strings or symbols. Defaults to `{}`.
639
-
640
- **Returns:** Integer - The node ID of the stored memory
641
-
642
- **Example:**
643
- ```ruby
644
- # Store with source
645
- node_id = htm.remember("PostgreSQL is excellent for vector search with pgvector", source: "architect")
646
-
647
- # Store without source (uses default empty string)
648
- node_id = htm.remember("HTM uses two-tier memory architecture")
649
-
650
- # Store with metadata
651
- node_id = htm.remember(
652
- "User prefers dark mode",
653
- metadata: { category: "preference", priority: "high", version: 2 }
100
+ group = HTM::RobotGroup.new(
101
+ name: 'support-ha',
102
+ active: ['primary'],
103
+ passive: ['standby']
654
104
  )
655
105
 
656
- # Nil/empty handling
657
- node_id = htm.remember(nil) # Returns ID of last node without creating duplicate
658
- node_id = htm.remember("") # Returns ID of last node without creating duplicate
106
+ # Primary fails? Standby takes over instantly
107
+ group.failover!
659
108
  ```
660
109
 
661
- ---
110
+ Real-time synchronization via PostgreSQL LISTEN/NOTIFY. Add or remove robots dynamically.
662
111
 
663
- #### `recall(topic, timeframe: nil, limit: 20, strategy: :vector, with_relevance: false, query_tags: [], metadata: {})`
112
+ ## Quick Start
664
113
 
665
- Retrieve memories using temporal filtering and semantic/keyword search.
666
-
667
- **Parameters:**
668
- - `topic` (String, required) - Query text for semantic/keyword matching (first positional argument)
669
- - `timeframe` (Range, String, optional) - Time range to search within. Default: "last 7 days"
670
- - Range: `(Time.now - 3600)..Time.now`
671
- - String: `"last hour"`, `"last week"`, `"yesterday"`
672
- - `limit` (Integer, optional) - Maximum number of results. Default: 20
673
- - `strategy` (Symbol, optional) - Search strategy. Default: `:vector`
674
- - `:vector` - Semantic search using embeddings (cosine similarity)
675
- - `:fulltext` - Keyword search using PostgreSQL full-text + trigrams
676
- - `:hybrid` - Weighted combination (70% vector, 30% full-text)
677
- - `with_relevance` (Boolean, optional) - Include dynamic relevance scores. Default: false
678
- - `query_tags` (Array<String>, optional) - Filter results by tags. Default: []
679
- - `metadata` (Hash, optional) - Filter results by metadata using JSONB containment (`@>`). Default: `{}`
680
-
681
- **Returns:** Array<Hash> - Matching memories with fields: `id`, `content`, `source`, `created_at`, `access_count`, `metadata`, (optionally `relevance`)
682
-
683
- **Example:**
684
- ```ruby
685
- # Basic recall with time range
686
- memories = htm.recall(
687
- "database architecture",
688
- timeframe: (Time.now - 86400)..Time.now
689
- )
690
-
691
- # Using human-readable timeframe
692
- memories = htm.recall(
693
- "PostgreSQL performance",
694
- timeframe: "last week",
695
- strategy: :hybrid,
696
- limit: 10
697
- )
698
-
699
- # With relevance scoring
700
- memories = htm.recall(
701
- "HTM design decisions",
702
- timeframe: "last month",
703
- with_relevance: true,
704
- query_tags: ["architecture"]
705
- )
706
- # => [{ "id" => 123, "content" => "...", "relevance" => 0.92, ... }, ...]
707
-
708
- # Filter by metadata
709
- memories = htm.recall(
710
- "user preferences",
711
- metadata: { category: "preference" }
712
- )
713
- # => Returns only nodes with metadata containing { category: "preference" }
714
-
715
- # Combine metadata with other filters
716
- memories = htm.recall(
717
- "settings",
718
- timeframe: "last month",
719
- strategy: :hybrid,
720
- metadata: { priority: "high", version: 2 }
721
- )
722
- ```
723
-
724
- ---
725
-
726
- #### `forget(node_id, confirm: false)`
727
-
728
- Permanently delete a memory from both working memory and long-term storage.
729
-
730
- **Parameters:**
731
- - `node_id` (Integer, required) - The ID of the node to delete
732
- - `confirm` (Symbol, Boolean, required) - Must be `:confirmed` or `true` to proceed
733
-
734
- **Returns:** Boolean - `true` if deleted, `false` if not found
735
-
736
- **Safety:** This is the ONLY way to delete memories. Requires explicit confirmation to prevent accidental deletion.
737
-
738
- **Example:**
739
- ```ruby
740
- # Delete with symbol confirmation (recommended)
741
- htm.forget(node_id, confirm: :confirmed)
742
-
743
- # Delete with boolean confirmation
744
- htm.forget(node_id, confirm: true)
745
-
746
- # Will raise error - confirmation required
747
- htm.forget(node_id) # ArgumentError: Must confirm deletion
748
- htm.forget(node_id, confirm: false) # ArgumentError: Must confirm deletion
749
- ```
750
-
751
- ---
752
-
753
- ### Complete Usage Example
754
-
755
- ```ruby
756
- require 'htm'
757
-
758
- # Configure once globally (optional - uses defaults if not called)
759
- HTM.configure do |config|
760
- config.embedding_provider = :ollama
761
- config.embedding_model = 'nomic-embed-text'
762
- config.tag_provider = :ollama
763
- config.tag_model = 'llama3'
764
- end
765
-
766
- # Initialize for your robot
767
- htm = HTM.new(robot_name: "Assistant", working_memory_size: 128_000)
768
-
769
- # Store information
770
- htm.remember("PostgreSQL with pgvector for vector search", source: "architect")
771
- htm.remember("User prefers dark mode", source: "user")
772
- htm.remember("Use debug_me for debugging, not puts", source: "system")
773
-
774
- # Retrieve by time + topic
775
- recent = htm.recall(
776
- "PostgreSQL",
777
- timeframe: "last week",
778
- strategy: :hybrid,
779
- limit: 5
780
- )
781
-
782
- # Delete if needed (requires node ID from remember or recall)
783
- htm.forget(node_id, confirm: :confirmed)
784
- ```
785
-
786
- ## Use with Rails
787
-
788
- HTM is designed to integrate seamlessly with Ruby on Rails applications. It uses ActiveRecord models and follows Rails conventions, making it easy to add AI memory capabilities to existing Rails apps.
789
-
790
- ### Why HTM Works Well with Rails
791
-
792
- **1. Uses ActiveRecord**
793
- - HTM models are standard ActiveRecord classes with associations, validations, and scopes
794
- - Follows Rails naming conventions and patterns
795
- - Models are namespaced under `HTM::Models::` to avoid conflicts
796
-
797
- **2. Standard Rails Configuration**
798
- - Uses `config/database.yml` (Rails convention)
799
- - Respects `RAILS_ENV` environment variable
800
- - Supports ERB in configuration files
801
-
802
- **3. Separate Database**
803
- - HTM uses its own PostgreSQL database connection
804
- - Won't interfere with your Rails app's database
805
- - Configured with `prepared_statements: false` and `advisory_locks: false` for compatibility
806
- - Thread-safe for Rails multi-threading
807
-
808
- **4. Connection Management**
809
- - Separate connection pool from your Rails app
810
- - Configurable pool size and timeouts
811
- - Proper cleanup and disconnection support
812
-
813
- ### Integration Steps
814
-
815
- #### 1. Add to Gemfile
816
-
817
- ```ruby
818
- gem 'htm'
819
- ```
820
-
821
- Then run:
822
- ```bash
823
- bundle install
824
- ```
825
-
826
- #### 2. Configure Environment Variables
827
-
828
- Add HTM database configuration to your Rails environment. You can use `.env` files (with `dotenv-rails`) or set them directly:
829
-
830
- ```ruby
831
- # .env or config/application.rb
832
- HTM_DBURL=postgresql://user:pass@localhost:5432/htm_production
833
- ```
834
-
835
- Or use individual variables:
836
- ```ruby
837
- HTM_DBHOST=localhost
838
- HTM_DBNAME=htm_production
839
- HTM_DBUSER=postgres
840
- HTM_DBPASS=password
841
- HTM_DBPORT=5432
842
- ```
843
-
844
- #### 3. Create Initializer
845
-
846
- Create `config/initializers/htm.rb`:
847
-
848
- ```ruby
849
- # config/initializers/htm.rb
850
-
851
- # Configure HTM
852
- HTM.configure do |config|
853
- # Use Rails logger
854
- config.logger = Rails.logger
855
-
856
- # Embedding configuration (optional - uses Ollama defaults if not set)
857
- config.embedding_provider = :ollama
858
- config.embedding_model = 'nomic-embed-text'
859
-
860
- # Tag extraction configuration (optional - uses Ollama defaults if not set)
861
- config.tag_provider = :ollama
862
- config.tag_model = 'llama3'
863
-
864
- # Custom providers (optional)
865
- # config.embedding_generator = ->(text) { YourEmbeddingService.call(text) }
866
- # config.tag_extractor = ->(text, ontology) { YourTagService.call(text, ontology) }
867
- end
868
-
869
- # Establish HTM database connection
870
- HTM::ActiveRecordConfig.establish_connection!
871
-
872
- # Verify required PostgreSQL extensions are installed
873
- HTM::ActiveRecordConfig.verify_extensions!
874
-
875
- Rails.logger.info "HTM initialized with database: #{HTM::Database.default_config[:database]}"
876
- ```
877
-
878
- #### 4. Set Up Database
879
-
880
- Run the HTM database setup:
881
-
882
- ```bash
883
- # Using HTM rake tasks
884
- rake htm:db:setup
885
-
886
- # Or manually in Rails console
887
- rails c
888
- > HTM::Database.setup
889
- ```
890
-
891
- #### 5. Use in Your Rails App
892
-
893
- **In Controllers:**
894
-
895
- ```ruby
896
- class ChatsController < ApplicationController
897
- before_action :initialize_memory
898
-
899
- def create
900
- # Add user message to memory (embeddings + tags auto-extracted asynchronously)
901
- @memory.remember(
902
- params[:message],
903
- source: "user-#{current_user.id}"
904
- )
905
-
906
- # Recall relevant context for the conversation
907
- context = @memory.create_context(strategy: :balanced)
908
-
909
- # Use context with your LLM to generate response
910
- response = generate_llm_response(context, params[:message])
911
-
912
- # Store assistant's response
913
- @memory.remember(
914
- response,
915
- source: "assistant"
916
- )
917
-
918
- render json: { response: response }
919
- end
920
-
921
- private
922
-
923
- def initialize_memory
924
- @memory = HTM.new(
925
- robot_name: "ChatBot-#{current_user.id}",
926
- working_memory_size: 128_000
927
- )
928
- end
929
- end
930
- ```
931
-
932
- **In Background Jobs:**
933
-
934
- ```ruby
935
- class ProcessDocumentJob < ApplicationJob
936
- queue_as :default
937
-
938
- def perform(document_id)
939
- document = Document.find(document_id)
940
-
941
- # Initialize HTM for this job
942
- memory = HTM.new(robot_name: "DocumentProcessor")
943
-
944
- # Store document content in memory (embeddings + tags auto-extracted asynchronously)
945
- memory.remember(
946
- document.content,
947
- source: "document-#{document.id}"
948
- )
949
-
950
- # Recall related documents (uses vector similarity)
951
- related = memory.recall(
952
- document.category,
953
- timeframe: "last month",
954
- strategy: :vector
955
- )
956
-
957
- # Process with context...
958
- end
959
- end
960
- ```
961
-
962
- **In Models (as a concern):**
963
-
964
- ```ruby
965
- # app/models/concerns/memorable.rb
966
- module Memorable
967
- extend ActiveSupport::Concern
968
-
969
- included do
970
- after_create :store_in_memory
971
- after_update :update_memory
972
- end
973
-
974
- def store_in_memory
975
- memory = HTM.new(robot_name: "Rails-#{Rails.env}")
976
-
977
- memory.remember(
978
- memory_content,
979
- source: "#{self.class.name}-#{id}"
980
- )
981
- end
982
-
983
- def update_memory
984
- # Update existing memory node
985
- store_in_memory
986
- end
987
-
988
- private
989
-
990
- def memory_content
991
- # Override in model to customize what gets stored
992
- attributes.to_json
993
- end
994
- end
995
-
996
- # Then in your model:
997
- class Article < ApplicationRecord
998
- include Memorable
999
-
1000
- private
1001
-
1002
- def memory_content
1003
- "Article: #{title}\n\n#{body}\n\nCategory: #{category}"
1004
- end
1005
- end
1006
- ```
1007
-
1008
- ### Multi-Database Configuration (Rails 6+)
1009
-
1010
- If you want to use Rails' built-in multi-database support, you can configure HTM in your `config/database.yml`:
1011
-
1012
- ```yaml
1013
- # config/database.yml
1014
- production:
1015
- primary:
1016
- <<: *default
1017
- database: myapp_production
1018
-
1019
- htm:
1020
- adapter: postgresql
1021
- encoding: unicode
1022
- database: <%= ENV['HTM_DBNAME'] || 'htm_production' %>
1023
- host: <%= ENV['HTM_DBHOST'] || 'localhost' %>
1024
- port: <%= ENV['HTM_DBPORT'] || 5432 %>
1025
- username: <%= ENV['HTM_DBUSER'] || 'postgres' %>
1026
- password: <%= ENV['HTM_DBPASS'] %>
1027
- pool: <%= ENV['HTM_DB_POOL_SIZE'] || 10 %>
1028
- ```
1029
-
1030
- Then update your HTM initializer:
1031
-
1032
- ```ruby
1033
- # config/initializers/htm.rb
1034
- HTM::ActiveRecordConfig.establish_connection!
1035
-
1036
- # Or use Rails' connection
1037
- # ActiveRecord::Base.connected_to(role: :writing, shard: :htm) do
1038
- # HTM::ActiveRecordConfig.verify_extensions!
1039
- # end
1040
- ```
1041
-
1042
- ### Testing with Rails
1043
-
1044
- **In RSpec:**
1045
-
1046
- ```ruby
1047
- # spec/rails_helper.rb
1048
- RSpec.configure do |config|
1049
- config.before(:suite) do
1050
- HTM::ActiveRecordConfig.establish_connection!
1051
- end
1052
-
1053
- config.after(:suite) do
1054
- HTM::ActiveRecordConfig.disconnect!
1055
- end
1056
- end
1057
-
1058
- # spec/features/chat_spec.rb
1059
- RSpec.describe "Chat with memory", type: :feature do
1060
- let(:memory) { HTM.new(robot_name: "TestBot") }
1061
-
1062
- before do
1063
- memory.remember("User prefers Ruby over Python", source: "user")
1064
- end
1065
-
1066
- it "recalls user preferences" do
1067
- context = memory.create_context(strategy: :balanced)
1068
- expect(context).to include("Ruby")
1069
- end
1070
- end
1071
- ```
1072
-
1073
- **In Minitest:**
1074
-
1075
- ```ruby
1076
- # test/test_helper.rb
1077
- class ActiveSupport::TestCase
1078
- setup do
1079
- HTM::ActiveRecordConfig.establish_connection! unless HTM::ActiveRecordConfig.connected?
1080
- end
1081
- end
1082
-
1083
- # test/integration/chat_test.rb
1084
- class ChatTest < ActionDispatch::IntegrationTest
1085
- test "stores chat messages in memory" do
1086
- memory = HTM.new(robot_name: "TestBot")
1087
-
1088
- post chats_path, params: { message: "Hello!" }
1089
-
1090
- assert_response :success
1091
-
1092
- # Verify message was stored
1093
- nodes = memory.recall("Hello", timeframe: "last hour")
1094
- assert_not_empty nodes
1095
- end
1096
- end
1097
- ```
1098
-
1099
- ### Production Considerations
1100
-
1101
- 1. **Connection Pooling**: Set appropriate pool size based on your Rails app's concurrency:
1102
- ```ruby
1103
- ENV['HTM_DB_POOL_SIZE'] = '20' # Match or exceed Rails pool
1104
- ```
1105
-
1106
- 2. **Separate Database Server**: For production, use a dedicated PostgreSQL instance for HTM
1107
-
1108
- 3. **Required Extensions**: Ensure `pgvector` and `pg_trgm` extensions are installed on your PostgreSQL server
1109
-
1110
- 4. **Memory Management**: HTM's working memory is per-instance. Consider using a singleton pattern or Rails cache for shared instances
1111
-
1112
- 5. **Background Processing**: Use Sidekiq or similar for embedding generation if processing large amounts of data
1113
-
1114
- ### Example Rails App Structure
1115
-
1116
- ```
1117
- app/
1118
- ├── controllers/
1119
- │ └── chats_controller.rb # Uses HTM for conversation memory
1120
- ├── jobs/
1121
- │ └── process_document_job.rb # Background HTM processing
1122
- ├── models/
1123
- │ ├── concerns/
1124
- │ │ └── memorable.rb # HTM integration concern
1125
- │ └── article.rb # Includes Memorable
1126
- └── services/
1127
- └── memory_service.rb # Centralized HTM access
1128
-
1129
- config/
1130
- ├── initializers/
1131
- │ └── htm.rb # HTM setup
1132
- └── database.yml # Optional: multi-database config
1133
-
1134
- .env
1135
- HTM_DBURL=postgresql://... # HTM database connection
1136
- OPENAI_API_KEY=sk-... # If using OpenAI embeddings
1137
- ```
1138
-
1139
- ## Environment Variables
1140
-
1141
- HTM uses environment variables for database and service configuration. These can be set in your shell profile, a `.env` file, or exported directly.
1142
-
1143
- ### Database Configuration
1144
-
1145
- #### HTM_DBURL (Recommended)
1146
-
1147
- The preferred method for database configuration is a single connection URL:
1148
-
1149
- ```bash
1150
- export HTM_DBURL="postgresql://username:password@host:port/dbname?sslmode=require"
1151
- ```
1152
-
1153
- **Format:** `postgresql://USER:PASSWORD@HOST:PORT/DATABASE?sslmode=MODE`
1154
-
1155
- **Example for Local PostgreSQL:**
1156
- ```bash
1157
- export HTM_DBURL="postgresql://postgres:postgres@localhost:5432/htm_dev?sslmode=disable"
1158
- ```
1159
-
1160
- #### Individual Database Parameters (Alternative)
1161
-
1162
- If `HTM_DBURL` is not set, HTM will use individual parameters:
1163
-
1164
- | Variable | Description | Default | Required |
1165
- |----------|-------------|---------|----------|
1166
- | `HTM_DBNAME` | Database name | None | Yes |
1167
- | `HTM_DBUSER` | Database username | None | Yes |
1168
- | `HTM_DBPASS` | Database password | None | Yes |
1169
- | `HTM_DBHOST` | Database hostname or IP | `localhost` | No |
1170
- | `HTM_DBPORT` | Database port | `5432` | No |
1171
- | `HTM_SERVICE_NAME` | Service identifier (informational only) | None | No |
1172
-
1173
- **Example:**
1174
114
  ```bash
1175
- export HTM_DBNAME="htm_production"
1176
- export HTM_DBUSER="postgres"
1177
- export HTM_DBPASS="secure_password_here"
1178
- export HTM_DBHOST="localhost"
1179
- export HTM_DBPORT="5432"
1180
- export HTM_SERVICE_NAME="production-db"
1181
- ```
1182
-
1183
- **Configuration Priority:**
1184
- 1. `HTM_DBURL` (if set, this takes precedence)
1185
- 2. Individual parameters (`HTM_DBNAME`, `HTM_DBUSER`, etc.)
1186
- 3. Direct configuration passed to `HTM.new(db_config: {...})`
1187
-
1188
- ### LLM Provider Configuration
1189
-
1190
- HTM configuration is done programmatically via `HTM.configure` (see [Configuration](#configuration) section). However, a few environment variables are still used:
1191
-
1192
- #### OPENAI_API_KEY
1193
-
1194
- Required when using OpenAI as your embedding or tag extraction provider:
1195
-
1196
- ```bash
1197
- export OPENAI_API_KEY="sk-proj-your-api-key-here"
115
+ gem install htm
1198
116
  ```
1199
117
 
1200
- This is required when you configure:
1201
118
  ```ruby
1202
- HTM.configure do |config|
1203
- config.embedding_provider = :openai
1204
- # or
1205
- config.tag_provider = :openai
1206
- end
1207
- ```
1208
-
1209
- #### OLLAMA_URL
1210
-
1211
- Custom Ollama server URL (optional):
1212
-
1213
- ```bash
1214
- export OLLAMA_URL="http://localhost:11434" # Default
1215
- export OLLAMA_URL="http://ollama-server:11434" # Custom
1216
- ```
1217
-
1218
- HTM defaults to `http://localhost:11434` if not set.
1219
-
1220
- #### HTM_LOG_LEVEL
1221
-
1222
- Control logging verbosity:
1223
-
1224
- ```bash
1225
- export HTM_LOG_LEVEL="DEBUG" # DEBUG, INFO, WARN, ERROR, FATAL
1226
- export HTM_LOG_LEVEL="INFO" # Default
1227
- ```
1228
-
1229
- This is used by the default logger when `HTM.configure` is called without a custom logger.
1230
-
1231
- #### HTM_TELEMETRY_ENABLED
1232
-
1233
- Enable OpenTelemetry metrics collection:
1234
-
1235
- ```bash
1236
- export HTM_TELEMETRY_ENABLED="true" # Enable telemetry
1237
- export HTM_TELEMETRY_ENABLED="false" # Disable (default)
1238
- ```
1239
-
1240
- When enabled, HTM emits metrics to configured OpenTelemetry collectors. See [Telemetry](#telemetry-opentelemetry) for details.
1241
-
1242
- ### Quick Setup Examples
1243
-
1244
- #### Local Development (PostgreSQL)
1245
-
1246
- ```bash
1247
- # Simple local setup
1248
- export HTM_DBURL="postgresql://postgres:postgres@localhost:5432/htm_dev?sslmode=disable"
1249
-
1250
- # With custom user
1251
- export HTM_DBURL="postgresql://myuser:mypass@localhost:5432/htm_dev"
1252
- ```
1253
-
1254
- #### With OpenAI Embeddings
1255
-
1256
- ```bash
1257
- # Database + OpenAI
1258
- export HTM_DBURL="postgresql://user:pass@host:port/db?sslmode=require"
1259
- export OPENAI_API_KEY="sk-proj-your-api-key-here"
1260
- ```
1261
-
1262
- ### Persistent Configuration
1263
-
1264
- To make environment variables permanent, add them to your shell profile:
1265
-
1266
- **For Bash (~/.bashrc or ~/.bash_profile):**
1267
- ```bash
1268
- echo 'export HTM_DBURL="postgresql://user:pass@host:port/db?sslmode=require"' >> ~/.bashrc
1269
- source ~/.bashrc
1270
- ```
1271
-
1272
- **For Zsh (~/.zshrc):**
1273
- ```bash
1274
- echo 'export HTM_DBURL="postgresql://user:pass@host:port/db?sslmode=require"' >> ~/.zshrc
1275
- source ~/.zshrc
1276
- ```
1277
-
1278
- **Or create a dedicated file (~/.bashrc__htm):**
1279
- ```bash
1280
- # ~/.bashrc__htm
1281
- export HTM_DBURL="postgresql://user:pass@host:port/db?sslmode=require"
1282
- export OPENAI_API_KEY="sk-proj-your-api-key"
1283
- export HTM_SERVICE_NAME="my-service"
1284
-
1285
- # Then source it from your main profile
1286
- echo 'source ~/.bashrc__htm' >> ~/.bashrc
1287
- ```
1288
-
1289
- ### Verification
1290
-
1291
- Verify your configuration:
1292
-
1293
- ```bash
1294
- # Check database connection
1295
- ruby test_connection.rb
1296
-
1297
- # Or in Ruby
1298
- irb -r ./lib/htm
1299
- > HTM::Database.default_config
1300
- > # Should return a hash with your connection parameters
1301
- ```
1302
-
1303
- ## Development
1304
-
1305
- After checking out the repo, run:
1306
-
1307
- ```bash
1308
- # Install dependencies
1309
- bundle install
1310
-
1311
- # Enable direnv (loads .envrc environment variables)
1312
- direnv allow
1313
-
1314
- # Run tests
1315
- rake test
1316
-
1317
- # Run example
1318
- ruby examples/basic_usage.rb
1319
- ```
1320
-
1321
- ### Database Management
1322
-
1323
- HTM provides comprehensive rake tasks under the `htm:db` namespace for managing the database:
1324
-
1325
- ```bash
1326
- # List all database tasks
1327
- rake -T htm:db
1328
-
1329
- # Set up database (create schema + run migrations)
1330
- rake htm:db:setup
1331
-
1332
- # Run pending migrations only
1333
- rake htm:db:migrate
1334
-
1335
- # Show migration status (which migrations are applied)
1336
- rake htm:db:status
1337
-
1338
- # Show database info (size, tables, extensions, row counts)
1339
- rake htm:db:info
1340
-
1341
- # Test database connection
1342
- rake htm:db:test
1343
-
1344
- # Open PostgreSQL console (interactive psql session)
1345
- rake htm:db:console
1346
-
1347
- # Seed database with sample data
1348
- rake htm:db:seed
1349
-
1350
- # Drop all HTM tables (WARNING: destructive!)
1351
- rake htm:db:drop
1352
-
1353
- # Drop and recreate database (WARNING: destructive!)
1354
- rake htm:db:reset
1355
- ```
1356
-
1357
- **Important**: Make sure `direnv allow` has been run once in the project directory to load database environment variables from `.envrc`. Alternatively, you can manually export the environment variables:
1358
-
1359
- ```bash
1360
- # Manually export HTM_DBURL
1361
- export HTM_DBURL="postgresql://user:password@host:port/dbname?sslmode=require"
1362
- ```
1363
-
1364
- **Common Workflows**:
1365
-
1366
- ```bash
1367
- # Initial setup (first time)
1368
- direnv allow
1369
- rake htm:db:setup # Creates schema and runs migrations
1370
-
1371
- # After pulling new migrations
1372
- rake htm:db:migrate # Run pending migrations
1373
-
1374
- # Check database state
1375
- rake htm:db:status # See migration status
1376
- rake htm:db:info # See database details
1377
-
1378
- # Reset database (development only!)
1379
- rake htm:db:reset # Drops and recreates everything
1380
-
1381
- # Open database console for debugging
1382
- rake htm:db:console # Opens psql
1383
- ```
1384
-
1385
- ### Async Job Management
1386
-
1387
- HTM provides rake tasks under the `htm:jobs` namespace for managing async embedding and tag extraction jobs:
1388
-
1389
- ```bash
1390
- # List all job management tasks
1391
- rake -T htm:jobs
1392
-
1393
- # Show statistics for async processing
1394
- rake htm:jobs:stats
1395
-
1396
- # Process pending jobs
1397
- rake htm:jobs:process_embeddings # Process nodes without embeddings
1398
- rake htm:jobs:process_tags # Process nodes without tags
1399
- rake htm:jobs:process_all # Process both
1400
-
1401
- # Reprocess all nodes (force regeneration)
1402
- rake htm:jobs:reprocess_embeddings # WARNING: Prompts for confirmation
1403
-
1404
- # Debugging and maintenance
1405
- rake htm:jobs:failed # Show stuck jobs (>1 hour old)
1406
- rake htm:jobs:clear_all # Clear all embeddings/tags (testing only)
1407
- ```
1408
-
1409
- **Common Workflows**:
1410
-
1411
- ```bash
1412
- # Monitor async processing
1413
- rake htm:jobs:stats
119
+ require 'htm'
1414
120
 
1415
- # Manually process pending jobs (if async job runner is not running)
1416
- rake htm:jobs:process_all
121
+ # Initialize
122
+ htm = HTM.new(robot_name: "MyAssistant")
1417
123
 
1418
- # Debug stuck jobs
1419
- rake htm:jobs:failed
1420
- rake htm:jobs:process_embeddings # Retry failed embeddings
124
+ # Remember
125
+ htm.remember("User's project uses Rails 7 with PostgreSQL")
1421
126
 
1422
- # Development/testing: force regeneration
1423
- rake htm:jobs:reprocess_embeddings # Regenerate all embeddings
1424
- rake htm:jobs:clear_all # Clear everything and start fresh
127
+ # Recall
128
+ memories = htm.recall("tech stack", timeframe: "last month")
1425
129
  ```
1426
130
 
1427
- See the [Async Job Processing](#async-job-processing) section for more details.
131
+ For MCP integration, database setup, and configuration options, see the [full documentation](https://madbomber.github.io/htm).
1428
132
 
1429
- ## Testing
133
+ ## Features at a Glance
1430
134
 
1431
- HTM uses Minitest:
135
+ | Feature | Description |
136
+ |---------|-------------|
137
+ | **MCP Server** | 23 tools for AI assistant integration |
138
+ | **Hive Mind** | Shared memory across all robots |
139
+ | **Vector Search** | Semantic retrieval with pgvector |
140
+ | **Hybrid Search** | Combines vector + full-text matching |
141
+ | **Temporal Queries** | "last week", "yesterday", date ranges |
142
+ | **Auto-Tagging** | LLM extracts hierarchical tags automatically |
143
+ | **Robot Groups** | High-availability with failover |
144
+ | **Rails Integration** | Auto-configures via Railtie, uses ActiveJob |
145
+ | **Telemetry** | Optional OpenTelemetry metrics |
1432
146
 
1433
- ```bash
1434
- # Run all tests
1435
- rake test
147
+ ## Requirements
1436
148
 
1437
- # Run specific test file
1438
- ruby test/htm_test.rb
1439
- ```
149
+ - Ruby 3.0+
150
+ - PostgreSQL with pgvector and pg_trgm extensions
151
+ - Ollama (default) or any local/private provider for embeddings and classifications
1440
152
 
1441
- ## Architecture
153
+ ## Documentation
1442
154
 
1443
- See [htm_teamwork.md](htm_teamwork.md) for detailed design documentation and planning notes.
155
+ **[https://madbomber.github.io/htm](https://madbomber.github.io/htm)**
1444
156
 
1445
- ### Key Components
157
+ - [Installation & Setup](https://madbomber.github.io/htm/getting-started/)
158
+ - [MCP Server Guide](https://madbomber.github.io/htm/guides/mcp-server/)
159
+ - [Rails Integration](https://madbomber.github.io/htm/guides/rails/)
160
+ - [API Reference](https://madbomber.github.io/htm/api/)
1446
161
 
1447
- - **HTM**: Main API class that coordinates all components and provides the primary interface
1448
- - **Configuration**: Dependency injection for LLM providers (embedding_generator, tag_extractor, token_counter, logger)
1449
- - **WorkingMemory**: In-memory, token-limited active context for immediate LLM use
1450
- - **LongTermMemory**: PostgreSQL-backed permanent storage with connection pooling
1451
- - **EmbeddingService**: Processing and validation layer for vector embeddings (calls configured `embedding_generator`)
1452
- - **TagService**: Processing and validation layer for hierarchical tags (calls configured `tag_extractor`)
1453
- - **Database**: Schema setup, migrations, and management
1454
- - **Background Jobs**: Async processing for embedding generation and tag extraction
162
+ ## Why "Robots" Instead of "Agents"?
1455
163
 
1456
- ### Database Schema
164
+ > "What's in a name? That which we call a rose
165
+ > By any other name would smell as sweet."
166
+ > — Shakespeare, *Romeo and Juliet*
1457
167
 
1458
- - `robots`: Robot registry for all LLM agents using HTM
1459
- - `nodes`: Main memory storage with vector embeddings (pgvector), full-text search (tsvector), JSONB metadata
1460
- - `tags`: Hierarchical tag ontology (format: `root:level1:level2:level3`)
1461
- - `node_tags`: Join table implementing many-to-many relationship between nodes and tags
168
+ Shakespeare argues names are arbitrary. In software, we respectfully disagree—names shape expectations and understanding.
1462
169
 
1463
- **Nodes Table Key Columns:**
1464
- - `content`: The memory content
1465
- - `embedding`: Vector embedding for semantic search (up to 2000 dimensions)
1466
- - `metadata`: JSONB column for arbitrary key-value data (filterable via `@>` containment operator)
1467
- - `content_hash`: SHA-256 hash for deduplication
170
+ HTM uses "robots" rather than the fashionable "agents" for several reasons:
1468
171
 
1469
- ### Service Architecture
172
+ - **Semantic clarity**: "Agent" is overloaded—user agents, software agents, real estate agents, secret agents. "Robot" is specific to automated workers.
1470
173
 
1471
- HTM uses a layered architecture for LLM integration:
174
+ - **Honest about capabilities**: "Agent" implies autonomy and genuine decision-making. These systems follow instructions—they don't have agency. "Robot" acknowledges what they actually are: tools.
1472
175
 
1473
- 1. **Configuration Layer** (`HTM.configure`): Provides raw LLM access via `embedding_generator` and `tag_extractor` callables
1474
- 2. **Service Layer** (`EmbeddingService`, `TagService`): Processes and validates LLM responses, handles padding/truncation, formats for storage
1475
- 3. **Consumer Layer** (`GenerateEmbeddingJob`, `GenerateTagsJob`, rake tasks): Uses services for async or synchronous processing
176
+ - **Avoiding the hype cycle**: "AI Agent" and "Agentic AI" became buzzwords in 2023-2024, often meaning nothing more than "LLM with a prompt." We prefer terminology that will age well.
1476
177
 
1477
- This separation allows you to provide any LLM implementation while HTM handles response processing, validation, and storage.
178
+ - **Many "agent" frameworks are prompt wrappers**: Look under the hood of popular agent libraries and you'll often find a single prompt in a for-loop. Calling that an "agent" sets false expectations.
1478
179
 
1479
- ## Roadmap
180
+ - **Rich heritage**: "Robot" comes from Karel Čapek's 1920 play *R.U.R.* (from Czech *robota*, meaning labor). Isaac Asimov gave us the Three Laws. There's decades of thoughtful writing about robot ethics and behavior. "Agent" has no comparable tradition.
1480
181
 
1481
- - [x] Phase 1: Foundation (basic two-tier memory)
1482
- - [x] Phase 2: RAG retrieval (semantic search)
1483
- - [x] Phase 3: Relationships & tags
1484
- - [x] Phase 4: Working memory management
1485
- - [x] Phase 5: Hive mind features
1486
- - [x] Phase 6: Operations & observability
1487
- - [x] Phase 7: Advanced features
1488
- - [x] Phase 8: Production-ready gem
182
+ - **Personality**: Robots are endearing—R2-D2, Wall-E, Bender. They have cultural weight that "agent" lacks.
1489
183
 
184
+ Honest terminology leads to clearer thinking. These are robots: tireless workers with perfect memory, executing instructions on our behalf. That's exactly what HTM helps them do better.
1490
185
 
1491
186
  ## Contributing
1492
187