htm 0.0.14 → 0.0.17
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.envrc +1 -0
- data/CHANGELOG.md +100 -0
- data/README.md +107 -1412
- data/bin/htm_mcp +31 -0
- data/config/database.yml +7 -4
- data/db/migrate/00003_create_file_sources.rb +5 -0
- data/db/migrate/00004_create_nodes.rb +17 -0
- data/db/migrate/00005_create_tags.rb +7 -0
- data/db/migrate/00006_create_node_tags.rb +2 -0
- data/db/migrate/00007_create_robot_nodes.rb +7 -0
- data/db/schema.sql +41 -29
- data/docs/api/yard/HTM/Configuration.md +54 -0
- data/docs/api/yard/HTM/Database.md +13 -10
- data/docs/api/yard/HTM/EmbeddingService.md +5 -1
- data/docs/api/yard/HTM/LongTermMemory.md +18 -277
- data/docs/api/yard/HTM/PropositionError.md +18 -0
- data/docs/api/yard/HTM/PropositionService.md +66 -0
- data/docs/api/yard/HTM/QueryCache.md +88 -0
- data/docs/api/yard/HTM/RobotGroup.md +481 -0
- data/docs/api/yard/HTM/SqlBuilder.md +108 -0
- data/docs/api/yard/HTM/TagService.md +4 -0
- data/docs/api/yard/HTM/Telemetry/NullInstrument.md +13 -0
- data/docs/api/yard/HTM/Telemetry/NullMeter.md +15 -0
- data/docs/api/yard/HTM/Telemetry.md +109 -0
- data/docs/api/yard/HTM/WorkingMemoryChannel.md +176 -0
- data/docs/api/yard/HTM.md +11 -23
- data/docs/api/yard/index.csv +102 -25
- data/docs/api/yard-reference.md +8 -0
- data/docs/assets/images/multi-provider-failover.svg +51 -0
- data/docs/assets/images/robot-group-architecture.svg +65 -0
- data/docs/database/README.md +3 -3
- data/docs/database/public.file_sources.svg +29 -21
- data/docs/database/public.node_tags.md +2 -0
- data/docs/database/public.node_tags.svg +53 -41
- data/docs/database/public.nodes.md +2 -0
- data/docs/database/public.nodes.svg +52 -40
- data/docs/database/public.robot_nodes.md +2 -0
- data/docs/database/public.robot_nodes.svg +30 -22
- data/docs/database/public.robots.svg +16 -12
- data/docs/database/public.tags.md +3 -0
- data/docs/database/public.tags.svg +41 -33
- data/docs/database/schema.json +66 -0
- data/docs/database/schema.svg +60 -48
- data/docs/development/index.md +13 -0
- data/docs/development/rake-tasks.md +1068 -0
- data/docs/getting-started/installation.md +31 -11
- data/docs/getting-started/quick-start.md +144 -155
- data/docs/guides/adding-memories.md +2 -3
- data/docs/guides/context-assembly.md +185 -184
- data/docs/guides/getting-started.md +154 -148
- data/docs/guides/index.md +7 -0
- data/docs/guides/long-term-memory.md +60 -92
- data/docs/guides/mcp-server.md +1052 -0
- data/docs/guides/multi-robot.md +249 -345
- data/docs/guides/recalling-memories.md +153 -163
- data/docs/guides/robot-groups.md +604 -0
- data/docs/guides/search-strategies.md +61 -58
- data/docs/guides/working-memory.md +103 -136
- data/docs/index.md +30 -26
- data/docs/multi_framework_support.md +2 -2
- data/examples/mcp_client.rb +2 -2
- data/examples/rails_app/.gitignore +2 -0
- data/examples/rails_app/Gemfile +22 -0
- data/examples/rails_app/Gemfile.lock +438 -0
- data/examples/rails_app/Procfile.dev +1 -0
- data/examples/rails_app/README.md +98 -0
- data/examples/rails_app/Rakefile +5 -0
- data/examples/rails_app/app/assets/stylesheets/application.css +83 -0
- data/examples/rails_app/app/assets/stylesheets/inter-font.css +6 -0
- data/examples/rails_app/app/controllers/application_controller.rb +19 -0
- data/examples/rails_app/app/controllers/dashboard_controller.rb +27 -0
- data/examples/rails_app/app/controllers/files_controller.rb +205 -0
- data/examples/rails_app/app/controllers/memories_controller.rb +102 -0
- data/examples/rails_app/app/controllers/robots_controller.rb +44 -0
- data/examples/rails_app/app/controllers/search_controller.rb +46 -0
- data/examples/rails_app/app/controllers/tags_controller.rb +30 -0
- data/examples/rails_app/app/javascript/application.js +4 -0
- data/examples/rails_app/app/javascript/controllers/application.js +9 -0
- data/examples/rails_app/app/javascript/controllers/index.js +6 -0
- data/examples/rails_app/app/views/dashboard/index.html.erb +123 -0
- data/examples/rails_app/app/views/files/index.html.erb +108 -0
- data/examples/rails_app/app/views/files/new.html.erb +321 -0
- data/examples/rails_app/app/views/files/show.html.erb +130 -0
- data/examples/rails_app/app/views/layouts/application.html.erb +124 -0
- data/examples/rails_app/app/views/memories/_memory_card.html.erb +51 -0
- data/examples/rails_app/app/views/memories/deleted.html.erb +62 -0
- data/examples/rails_app/app/views/memories/edit.html.erb +35 -0
- data/examples/rails_app/app/views/memories/index.html.erb +81 -0
- data/examples/rails_app/app/views/memories/new.html.erb +71 -0
- data/examples/rails_app/app/views/memories/show.html.erb +126 -0
- data/examples/rails_app/app/views/robots/index.html.erb +106 -0
- data/examples/rails_app/app/views/robots/new.html.erb +36 -0
- data/examples/rails_app/app/views/robots/show.html.erb +79 -0
- data/examples/rails_app/app/views/search/index.html.erb +184 -0
- data/examples/rails_app/app/views/shared/_navbar.html.erb +52 -0
- data/examples/rails_app/app/views/shared/_stat_card.html.erb +52 -0
- data/examples/rails_app/app/views/tags/index.html.erb +131 -0
- data/examples/rails_app/app/views/tags/show.html.erb +67 -0
- data/examples/rails_app/bin/dev +8 -0
- data/examples/rails_app/bin/rails +4 -0
- data/examples/rails_app/bin/rake +4 -0
- data/examples/rails_app/config/application.rb +33 -0
- data/examples/rails_app/config/boot.rb +5 -0
- data/examples/rails_app/config/database.yml +15 -0
- data/examples/rails_app/config/environment.rb +5 -0
- data/examples/rails_app/config/importmap.rb +7 -0
- data/examples/rails_app/config/routes.rb +38 -0
- data/examples/rails_app/config/tailwind.config.js +35 -0
- data/examples/rails_app/config.ru +5 -0
- data/examples/rails_app/log/.keep +0 -0
- data/examples/rails_app/tmp/local_secret.txt +1 -0
- data/examples/robot_groups/robot_worker.rb +1 -2
- data/examples/robot_groups/same_process.rb +1 -4
- data/lib/htm/active_record_config.rb +2 -5
- data/lib/htm/configuration.rb +35 -2
- data/lib/htm/database.rb +3 -6
- data/lib/htm/mcp/cli.rb +333 -0
- data/lib/htm/mcp/group_tools.rb +476 -0
- data/lib/htm/mcp/resources.rb +89 -0
- data/lib/htm/mcp/server.rb +98 -0
- data/lib/htm/mcp/tools.rb +488 -0
- data/lib/htm/models/file_source.rb +5 -3
- data/lib/htm/railtie.rb +0 -4
- data/lib/htm/robot_group.rb +721 -0
- data/lib/htm/tasks.rb +7 -4
- data/lib/htm/version.rb +1 -1
- data/lib/htm/working_memory_channel.rb +250 -0
- data/lib/htm.rb +2 -0
- data/lib/tasks/htm.rake +6 -9
- data/mkdocs.yml +2 -0
- metadata +75 -11
- data/bin/htm_mcp.rb +0 -621
- data/db/migrate/00009_add_working_memory_to_robot_nodes.rb +0 -12
- data/db/migrate/00010_add_soft_delete_to_associations.rb +0 -29
- data/db/migrate/00011_add_performance_indexes.rb +0 -21
- data/db/migrate/00012_add_tags_trigram_index.rb +0 -18
- data/db/migrate/00013_enable_lz4_compression.rb +0 -43
- data/examples/robot_groups/lib/robot_group.rb +0 -419
- data/examples/robot_groups/lib/working_memory_channel.rb +0 -140
data/README.md
CHANGED
|
@@ -1,1492 +1,187 @@
|
|
|
1
1
|
<div align="center">
|
|
2
2
|
<h1>HTM</h1>
|
|
3
|
-
<!-- img src="docs/assets/images/htm.jpg" alt="HTM - Hierarchical Temporal Memory" width="600" -->
|
|
4
3
|
<img src="docs/assets/images/htm_demo.gif" alt="Tree of Knowledge is Growing" width="400">
|
|
5
4
|
|
|
6
|
-
<p>
|
|
5
|
+
<p><strong>Give your AI tools a shared, persistent memory.</strong></p>
|
|
6
|
+
|
|
7
|
+
<p>
|
|
8
|
+
<a href="https://rubygems.org/gems/htm"><img src="https://img.shields.io/gem/v/htm.svg" alt="Gem Version"></a>
|
|
9
|
+
<a href="https://github.com/madbomber/htm/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="License"></a>
|
|
7
10
|
</p>
|
|
8
11
|
</div>
|
|
9
12
|
|
|
10
|
-
<br
|
|
11
|
-
|
|
12
|
-
> [!CAUTION]
|
|
13
|
-
> This library is under active development and experimentation. APIs and features may change without notice. Not recommended for production use yet... and the documentation may lie to you!
|
|
14
|
-
> <br /><br/>
|
|
15
|
-
> Apologies to Jeff Hawkins for using his term in such a macro-superficial way.
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
## What does it mean?
|
|
19
|
-
|
|
20
|
-
- **Hierarchical**: operates across multiple levels of abstraction, from simple concepts to detailed relationships
|
|
21
|
-
- **Temporal**: functions across time, from the present moment to historical data
|
|
22
|
-
- **Memory function**: encodes, stores, and retrieves information
|
|
23
|
-
|
|
24
|
-
**HTM**: a hierarchical and temporal memory system that organizes and recalls information at multiple levels of detail over extended timeframes.
|
|
25
|
-
|
|
26
|
-
## Features
|
|
27
|
-
|
|
28
|
-
- **Client-Side Embeddings**
|
|
29
|
-
- Automatic embedding generation before database insertion
|
|
30
|
-
- Uses the [ruby_llm](https://ruby_llm.com) gem for LLM access
|
|
31
|
-
- Configurable embedding providers and models
|
|
13
|
+
<br/>
|
|
32
14
|
|
|
33
|
-
|
|
34
|
-
- Working Memory: Token-limited active context for immediate LLM use
|
|
35
|
-
- Long-term Memory: Durable PostgreSQL storage
|
|
15
|
+
## The Problem
|
|
36
16
|
|
|
37
|
-
|
|
38
|
-
- All memories persist in long-term storage
|
|
39
|
-
- Only explicit `forget()` commands delete data
|
|
40
|
-
- Working memory evicts to long-term, never deletes
|
|
17
|
+
Your AI assistants are brilliant but forgetful. Claude Desktop doesn't remember what Claude Code learned. Your custom chatbot can't recall what happened yesterday. Every tool starts fresh, every session.
|
|
41
18
|
|
|
42
|
-
|
|
43
|
-
- Vector similarity search (pgvector)
|
|
44
|
-
- Full-text search (PostgreSQL)
|
|
45
|
-
- Hybrid search (combines both)
|
|
46
|
-
- Temporal filtering ("last week", date ranges)
|
|
47
|
-
- Variable embedding dimensions (384 to 3072)
|
|
19
|
+
**HTM fixes this.**
|
|
48
20
|
|
|
49
|
-
-
|
|
50
|
-
- All robots share global memory
|
|
51
|
-
- Cross-robot context awareness
|
|
52
|
-
- Track which robot said what
|
|
21
|
+
Connect all your MCP-enabled tools to the same long-term memory. What one robot learns, all robots remember. Context persists across sessions, tools, and time.
|
|
53
22
|
|
|
54
|
-
|
|
55
|
-
- Automatic hierarchical tag extraction from content
|
|
56
|
-
- Tags in colon-delimited format (e.g., `database:postgresql:performance`)
|
|
57
|
-
- LLM-powered asynchronous processing
|
|
58
|
-
- Enables both structured navigation and semantic discovery
|
|
59
|
-
- Complements vector embeddings (symbolic + sub-symbolic retrieval)
|
|
23
|
+
## What is HTM?
|
|
60
24
|
|
|
61
|
-
- **
|
|
62
|
-
|
|
63
|
-
|
|
25
|
+
- **Hierarchical**: Organizes information from simple concepts to detailed relationships
|
|
26
|
+
- **Temporal**: Retrieves memories across time—from moments ago to months past
|
|
27
|
+
- **Memory**: Encodes, stores, and recalls information with semantic understanding
|
|
64
28
|
|
|
65
|
-
-
|
|
66
|
-
- Load markdown files into long-term memory
|
|
67
|
-
- Automatic paragraph-based chunking
|
|
68
|
-
- Source file tracking with re-sync support
|
|
69
|
-
- YAML frontmatter extraction as metadata
|
|
29
|
+
HTM is a Ruby gem that provides durable, shared memory for LLM-powered applications.
|
|
70
30
|
|
|
71
|
-
|
|
72
|
-
- Optional metrics collection via OpenTelemetry
|
|
73
|
-
- Zero overhead when disabled (null object pattern)
|
|
74
|
-
- Works with 50+ backends (Jaeger, Prometheus, Datadog, etc.)
|
|
75
|
-
- Tracks job latency, search performance, and cache effectiveness
|
|
31
|
+
## Key Capabilities
|
|
76
32
|
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
Add this line to your application's Gemfile:
|
|
80
|
-
|
|
81
|
-
```ruby
|
|
82
|
-
gem 'htm'
|
|
83
|
-
```
|
|
84
|
-
|
|
85
|
-
And then execute:
|
|
86
|
-
|
|
87
|
-
```bash
|
|
88
|
-
bundle install
|
|
89
|
-
```
|
|
33
|
+
### MCP Server — Connect Any AI Tool
|
|
90
34
|
|
|
91
|
-
|
|
35
|
+
HTM includes a Model Context Protocol server with 23 tools for memory management. Connect Claude Desktop, Claude Code, AIA, or any MCP-compatible client:
|
|
92
36
|
|
|
93
|
-
```
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
HTM uses PostgreSQL for durable long-term memory storage. Set up your database connection via environment variables:
|
|
102
|
-
|
|
103
|
-
```bash
|
|
104
|
-
# Preferred: Full connection URL
|
|
105
|
-
export HTM_DBURL="postgresql://user:password@host:port/dbname?sslmode=require"
|
|
106
|
-
|
|
107
|
-
# Alternative: Individual parameters
|
|
108
|
-
export HTM_DBNAME="htm_production"
|
|
109
|
-
export HTM_DBUSER="postgres"
|
|
110
|
-
export HTM_DBPASS="your_password"
|
|
111
|
-
export HTM_DBHOST="localhost"
|
|
112
|
-
export HTM_DBPORT="5432"
|
|
113
|
-
```
|
|
114
|
-
|
|
115
|
-
See the [Environment Variables](#environment-variables) section below for complete details.
|
|
116
|
-
|
|
117
|
-
### 2. Configure LLM Providers
|
|
118
|
-
|
|
119
|
-
HTM uses LLM providers for embedding generation and tag extraction. By default, it uses Ollama (local).
|
|
120
|
-
|
|
121
|
-
**Start Ollama (Default Configuration)**:
|
|
122
|
-
```bash
|
|
123
|
-
# Install Ollama
|
|
124
|
-
curl https://ollama.ai/install.sh | sh
|
|
125
|
-
|
|
126
|
-
# Start Ollama and pull models
|
|
127
|
-
ollama serve
|
|
128
|
-
ollama pull nomic-embed-text # For embeddings (768 dimensions)
|
|
129
|
-
ollama pull llama3 # For tag extraction
|
|
130
|
-
```
|
|
131
|
-
|
|
132
|
-
**Configure HTM** (optional - uses defaults if not configured):
|
|
133
|
-
```ruby
|
|
134
|
-
require 'htm'
|
|
135
|
-
|
|
136
|
-
# Use defaults (Ollama with nomic-embed-text and llama3)
|
|
137
|
-
HTM.configure
|
|
138
|
-
|
|
139
|
-
# Or customize providers
|
|
140
|
-
HTM.configure do |config|
|
|
141
|
-
# Embedding configuration
|
|
142
|
-
config.embedding_provider = :ollama # or :openai
|
|
143
|
-
config.embedding_model = 'nomic-embed-text'
|
|
144
|
-
config.embedding_dimensions = 768
|
|
145
|
-
config.ollama_url = 'http://localhost:11434'
|
|
146
|
-
|
|
147
|
-
# Tag extraction configuration
|
|
148
|
-
config.tag_provider = :ollama # or :openai
|
|
149
|
-
config.tag_model = 'llama3'
|
|
150
|
-
|
|
151
|
-
# Logger configuration (optional)
|
|
152
|
-
config.logger = Logger.new($stdout)
|
|
153
|
-
config.logger.level = Logger::INFO
|
|
154
|
-
|
|
155
|
-
# Custom embedding generator (advanced)
|
|
156
|
-
config.embedding_generator = ->(text) {
|
|
157
|
-
# Your custom implementation
|
|
158
|
-
# Must return Array<Float>
|
|
159
|
-
}
|
|
160
|
-
|
|
161
|
-
# Custom tag extractor (advanced)
|
|
162
|
-
config.tag_extractor = ->(text, ontology) {
|
|
163
|
-
# Your custom implementation
|
|
164
|
-
# Must return Array<String>
|
|
37
|
+
```json
|
|
38
|
+
{
|
|
39
|
+
"mcpServers": {
|
|
40
|
+
"htm": {
|
|
41
|
+
"command": "htm_mcp",
|
|
42
|
+
"env": { "HTM_DBURL": "postgresql://localhost/htm" }
|
|
43
|
+
}
|
|
165
44
|
}
|
|
166
|
-
|
|
167
|
-
# Token counter (optional, defaults to Tiktoken)
|
|
168
|
-
config.token_counter = ->(text) {
|
|
169
|
-
# Your custom implementation
|
|
170
|
-
# Must return Integer
|
|
171
|
-
}
|
|
172
|
-
end
|
|
173
|
-
```
|
|
174
|
-
|
|
175
|
-
See the [Configuration](#configuration) section below for complete details.
|
|
176
|
-
|
|
177
|
-
### 3. Initialize Database Schema
|
|
178
|
-
|
|
179
|
-
**Using Rake Tasks (Recommended)**:
|
|
180
|
-
|
|
181
|
-
HTM provides comprehensive rake tasks for database management. Add this to your application's `Rakefile`:
|
|
182
|
-
|
|
183
|
-
```ruby
|
|
184
|
-
require 'htm/tasks'
|
|
185
|
-
```
|
|
186
|
-
|
|
187
|
-
Then use the tasks:
|
|
188
|
-
|
|
189
|
-
```bash
|
|
190
|
-
# Set up database schema and run migrations
|
|
191
|
-
rake htm:db:setup
|
|
192
|
-
|
|
193
|
-
# Verify connection
|
|
194
|
-
rake htm:db:test
|
|
195
|
-
|
|
196
|
-
# Check database status
|
|
197
|
-
rake htm:db:info
|
|
198
|
-
```
|
|
199
|
-
|
|
200
|
-
**Or programmatically**:
|
|
201
|
-
|
|
202
|
-
```ruby
|
|
203
|
-
require 'htm'
|
|
204
|
-
|
|
205
|
-
# Run once to set up database schema
|
|
206
|
-
HTM::Database.setup
|
|
207
|
-
```
|
|
208
|
-
|
|
209
|
-
**Or from command line**:
|
|
210
|
-
|
|
211
|
-
```bash
|
|
212
|
-
ruby -r ./lib/htm -e "HTM::Database.setup"
|
|
45
|
+
}
|
|
213
46
|
```
|
|
214
47
|
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
### 4. Verify Setup
|
|
48
|
+
The `htm_mcp` executable includes CLI commands for database management:
|
|
218
49
|
|
|
219
50
|
```bash
|
|
220
|
-
|
|
221
|
-
#
|
|
222
|
-
|
|
51
|
+
htm_mcp setup # Initialize database schema
|
|
52
|
+
htm_mcp verify # Check connection and migrations
|
|
53
|
+
htm_mcp stats # Show memory statistics
|
|
54
|
+
htm_mcp help # Full help with environment variables
|
|
55
|
+
htm_mcp # Start MCP server (default)
|
|
223
56
|
```
|
|
224
57
|
|
|
225
|
-
|
|
58
|
+
Now your AI assistant can remember and recall across sessions:
|
|
59
|
+
- Store decisions, preferences, and context
|
|
60
|
+
- Search memories with natural language
|
|
61
|
+
- Build knowledge that compounds over time
|
|
226
62
|
|
|
227
|
-
|
|
63
|
+
### Hive Mind — Shared Knowledge Across Robots
|
|
228
64
|
|
|
229
|
-
|
|
65
|
+
All robots share the same global memory. What one learns, all can access:
|
|
230
66
|
|
|
231
67
|
```ruby
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
require 'logger'
|
|
235
|
-
|
|
236
|
-
# Configure HTM with RubyLLM for embeddings and tag extraction
|
|
237
|
-
HTM.configure do |config|
|
|
238
|
-
# Configure logger (optional - uses STDOUT at INFO level if not provided)
|
|
239
|
-
config.logger = Logger.new($stdout)
|
|
240
|
-
config.logger.level = Logger::INFO
|
|
241
|
-
|
|
242
|
-
# Configure embedding generation using RubyLLM with Ollama
|
|
243
|
-
config.embedding_provider = :ollama
|
|
244
|
-
config.embedding_model = 'nomic-embed-text'
|
|
245
|
-
config.embedding_dimensions = 768
|
|
246
|
-
config.ollama_url = ENV['OLLAMA_URL'] || 'http://localhost:11434'
|
|
247
|
-
|
|
248
|
-
# Configure tag extraction using RubyLLM with Ollama
|
|
249
|
-
config.tag_provider = :ollama
|
|
250
|
-
config.tag_model = 'llama3'
|
|
68
|
+
# Claude Desktop remembers a user preference
|
|
69
|
+
claude_desktop.remember("User prefers dark mode and concise responses")
|
|
251
70
|
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
# Initialize HTM for your robot
|
|
257
|
-
htm = HTM.new(
|
|
258
|
-
robot_name: "Code Helper",
|
|
259
|
-
working_memory_size: 128_000 # tokens
|
|
260
|
-
)
|
|
261
|
-
|
|
262
|
-
# Remember information - embeddings and tags generated asynchronously
|
|
263
|
-
node_id = htm.remember(
|
|
264
|
-
"We decided to use PostgreSQL for HTM storage",
|
|
265
|
-
source: "architect"
|
|
266
|
-
)
|
|
267
|
-
|
|
268
|
-
# Recall from the past (uses semantic search with embeddings)
|
|
269
|
-
memories = htm.recall(
|
|
270
|
-
"database decisions",
|
|
271
|
-
timeframe: "last week",
|
|
272
|
-
strategy: :hybrid # Combines vector + full-text search
|
|
273
|
-
)
|
|
274
|
-
|
|
275
|
-
# Forget (explicit deletion only)
|
|
276
|
-
htm.forget(node_id, confirm: :confirmed)
|
|
71
|
+
# Later, Claude Code can recall it
|
|
72
|
+
claude_code.recall("user preferences")
|
|
73
|
+
# => "User prefers dark mode and concise responses"
|
|
277
74
|
```
|
|
278
75
|
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
HTM can load text-based files (currently markdown) into long-term memory with automatic chunking and source tracking.
|
|
76
|
+
Every tool builds on what came before. No more repeating yourself.
|
|
282
77
|
|
|
283
|
-
|
|
284
|
-
htm = HTM.new(robot_name: "Document Loader")
|
|
285
|
-
|
|
286
|
-
# Load a single markdown file
|
|
287
|
-
result = htm.load_file("docs/guide.md")
|
|
288
|
-
# => { file_source_id: 1, chunks_created: 5, chunks_updated: 0, chunks_deleted: 0 }
|
|
78
|
+
### Long-Term Memory — Never Forget
|
|
289
79
|
|
|
290
|
-
|
|
291
|
-
results = htm.load_directory("docs/", pattern: "**/*.md")
|
|
80
|
+
Two-tier architecture ensures nothing is lost:
|
|
292
81
|
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
296
|
-
# Unload a file (soft deletes chunks)
|
|
297
|
-
htm.unload_file("docs/guide.md")
|
|
298
|
-
```
|
|
82
|
+
- **Working Memory**: Token-limited context for immediate use
|
|
83
|
+
- **Long-Term Memory**: Durable PostgreSQL storage with vector search
|
|
299
84
|
|
|
300
|
-
|
|
301
|
-
- **Paragraph chunking**: Text split by blank lines, code blocks preserved
|
|
302
|
-
- **Source tracking**: Files tracked with mtime for automatic re-sync
|
|
303
|
-
- **YAML frontmatter**: Extracted and stored as metadata
|
|
304
|
-
- **Duplicate detection**: Content hash prevents duplicate nodes
|
|
85
|
+
Memories persist forever unless explicitly deleted. Working memory evicts to long-term storage—it never deletes.
|
|
305
86
|
|
|
306
|
-
**Rake tasks:**
|
|
307
|
-
```bash
|
|
308
|
-
rake 'htm:files:load[docs/guide.md]' # Load a single file
|
|
309
|
-
rake 'htm:files:load_dir[docs/]' # Load all markdown files from directory
|
|
310
|
-
rake htm:files:list # List all loaded file sources
|
|
311
|
-
rake htm:files:sync # Sync all files (reload changed)
|
|
312
|
-
rake htm:files:stats # Show file loading statistics
|
|
313
|
-
```
|
|
314
|
-
|
|
315
|
-
### Automatic Tag Extraction
|
|
316
|
-
|
|
317
|
-
HTM automatically extracts hierarchical tags from content using LLM analysis. Tags are inferred from the content itself - you never specify them manually.
|
|
318
|
-
|
|
319
|
-
**Example:**
|
|
320
87
|
```ruby
|
|
321
|
-
htm.remember(
|
|
322
|
-
"User prefers dark mode for all interfaces",
|
|
323
|
-
source: "user"
|
|
324
|
-
)
|
|
325
|
-
# Tags like "preference", "ui", "dark-mode" may be auto-extracted by the LLM
|
|
326
|
-
# The specific tags depend on the LLM's analysis of the content
|
|
327
|
-
```
|
|
88
|
+
htm.remember("PostgreSQL with pgvector handles our vector search")
|
|
328
89
|
|
|
329
|
-
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
|
|
333
|
-
**Monitor job processing:**
|
|
334
|
-
```bash
|
|
335
|
-
# Show statistics for nodes and async processing
|
|
336
|
-
rake htm:jobs:stats
|
|
337
|
-
|
|
338
|
-
# Output:
|
|
339
|
-
# Total nodes: 150
|
|
340
|
-
# Nodes with embeddings: 145 (96.7%)
|
|
341
|
-
# Nodes without embeddings: 5 (3.3%)
|
|
342
|
-
# Nodes with tags: 140 (93.3%)
|
|
343
|
-
# Total tags in ontology: 42
|
|
344
|
-
```
|
|
345
|
-
|
|
346
|
-
**Process pending jobs manually:**
|
|
347
|
-
```bash
|
|
348
|
-
# Process all pending embedding jobs
|
|
349
|
-
rake htm:jobs:process_embeddings
|
|
350
|
-
|
|
351
|
-
# Process all pending tag extraction jobs
|
|
352
|
-
rake htm:jobs:process_tags
|
|
353
|
-
|
|
354
|
-
# Process both embeddings and tags
|
|
355
|
-
rake htm:jobs:process_all
|
|
356
|
-
```
|
|
357
|
-
|
|
358
|
-
**Reprocess all nodes (force regeneration):**
|
|
359
|
-
```bash
|
|
360
|
-
# Regenerate embeddings for ALL nodes
|
|
361
|
-
rake htm:jobs:reprocess_embeddings
|
|
362
|
-
|
|
363
|
-
# WARNING: Prompts for confirmation
|
|
364
|
-
```
|
|
365
|
-
|
|
366
|
-
**Show failed/stuck jobs:**
|
|
367
|
-
```bash
|
|
368
|
-
# Show nodes that failed async processing (>1 hour old without embeddings/tags)
|
|
369
|
-
rake htm:jobs:failed
|
|
370
|
-
```
|
|
371
|
-
|
|
372
|
-
**Clear all for testing:**
|
|
373
|
-
```bash
|
|
374
|
-
# Clear ALL embeddings and tags (development/testing only)
|
|
375
|
-
rake htm:jobs:clear_all
|
|
376
|
-
|
|
377
|
-
# WARNING: Prompts for confirmation
|
|
90
|
+
# Months later...
|
|
91
|
+
htm.recall("database decisions", timeframe: "last year")
|
|
92
|
+
# => Still there
|
|
378
93
|
```
|
|
379
94
|
|
|
380
|
-
|
|
381
|
-
|
|
382
|
-
## Telemetry (OpenTelemetry)
|
|
95
|
+
### Robot Groups — High Availability
|
|
383
96
|
|
|
384
|
-
|
|
385
|
-
|
|
386
|
-
### Enabling Telemetry
|
|
97
|
+
Coordinate multiple robots with shared working memory and instant failover:
|
|
387
98
|
|
|
388
99
|
```ruby
|
|
389
|
-
HTM.
|
|
390
|
-
|
|
391
|
-
|
|
392
|
-
|
|
393
|
-
# Or via environment variable
|
|
394
|
-
# HTM_TELEMETRY_ENABLED=true
|
|
395
|
-
```
|
|
396
|
-
|
|
397
|
-
### Configuring the Destination
|
|
398
|
-
|
|
399
|
-
HTM emits metrics via standard OpenTelemetry protocols. Configure your destination using environment variables:
|
|
400
|
-
|
|
401
|
-
```bash
|
|
402
|
-
# Export to any OTLP-compatible backend
|
|
403
|
-
export OTEL_METRICS_EXPORTER="otlp"
|
|
404
|
-
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
|
|
405
|
-
```
|
|
406
|
-
|
|
407
|
-
### Available Metrics
|
|
408
|
-
|
|
409
|
-
| Metric | Type | Attributes | Description |
|
|
410
|
-
|--------|------|------------|-------------|
|
|
411
|
-
| `htm.jobs` | Counter | `job`, `status` | Job execution counts (embedding, tags) |
|
|
412
|
-
| `htm.embedding.latency` | Histogram | `provider`, `status` | Embedding generation time (ms) |
|
|
413
|
-
| `htm.tag.latency` | Histogram | `provider`, `status` | Tag extraction time (ms) |
|
|
414
|
-
| `htm.search.latency` | Histogram | `strategy` | Search operation time (ms) |
|
|
415
|
-
| `htm.cache.operations` | Counter | `operation` | Cache hits/misses |
|
|
416
|
-
|
|
417
|
-
### Compatible Backends
|
|
418
|
-
|
|
419
|
-
HTM works with any OTLP-compatible observability platform:
|
|
420
|
-
|
|
421
|
-
**Open Source:** Jaeger, Prometheus, Grafana Tempo/Mimir, SigNoz, Uptrace
|
|
422
|
-
|
|
423
|
-
**Commercial:** Datadog, New Relic, Honeycomb, Splunk, Dynatrace, AWS X-Ray, Google Cloud Trace, Azure Monitor
|
|
424
|
-
|
|
425
|
-
### Optional Dependencies
|
|
426
|
-
|
|
427
|
-
Users who want telemetry should add these gems:
|
|
428
|
-
|
|
429
|
-
```ruby
|
|
430
|
-
gem 'opentelemetry-sdk'
|
|
431
|
-
gem 'opentelemetry-metrics-sdk'
|
|
432
|
-
gem 'opentelemetry-exporter-otlp' # For OTLP export
|
|
433
|
-
```
|
|
434
|
-
|
|
435
|
-
### Design
|
|
436
|
-
|
|
437
|
-
HTM uses a **null object pattern** for telemetry. When disabled or when the SDK is not installed:
|
|
438
|
-
- All metric operations are no-ops
|
|
439
|
-
- Zero runtime overhead
|
|
440
|
-
- No errors or exceptions
|
|
441
|
-
|
|
442
|
-
See [docs/telemetry.md](docs/telemetry.md) for detailed configuration and usage examples.
|
|
443
|
-
|
|
444
|
-
## Configuration
|
|
445
|
-
|
|
446
|
-
HTM uses dependency injection for LLM access, allowing you to configure embedding generation, tag extraction, logging, and token counting.
|
|
447
|
-
|
|
448
|
-
### Default Configuration
|
|
449
|
-
|
|
450
|
-
By default, HTM uses:
|
|
451
|
-
- **Embedding Provider**: Ollama (local) with `nomic-embed-text` model (768 dimensions)
|
|
452
|
-
- **Tag Provider**: Ollama (local) with `llama3` model
|
|
453
|
-
- **Logger**: Ruby's standard Logger to STDOUT at INFO level
|
|
454
|
-
- **Token Counter**: Tiktoken with GPT-3.5-turbo encoding
|
|
455
|
-
|
|
456
|
-
```ruby
|
|
457
|
-
require 'htm'
|
|
458
|
-
|
|
459
|
-
# Use defaults
|
|
460
|
-
HTM.configure # or omit this - configuration is lazy-loaded
|
|
461
|
-
```
|
|
462
|
-
|
|
463
|
-
### Custom Configuration
|
|
464
|
-
|
|
465
|
-
Configure HTM before creating instances:
|
|
466
|
-
|
|
467
|
-
#### Using Ollama (Default)
|
|
468
|
-
|
|
469
|
-
```ruby
|
|
470
|
-
HTM.configure do |config|
|
|
471
|
-
# Embedding configuration
|
|
472
|
-
config.embedding_provider = :ollama
|
|
473
|
-
config.embedding_model = 'nomic-embed-text' # 768 dimensions
|
|
474
|
-
config.embedding_dimensions = 768
|
|
475
|
-
config.ollama_url = 'http://localhost:11434'
|
|
476
|
-
|
|
477
|
-
# Tag extraction configuration
|
|
478
|
-
config.tag_provider = :ollama
|
|
479
|
-
config.tag_model = 'llama3'
|
|
480
|
-
|
|
481
|
-
# Logger configuration
|
|
482
|
-
config.logger = Logger.new($stdout)
|
|
483
|
-
config.logger.level = Logger::INFO
|
|
484
|
-
end
|
|
485
|
-
```
|
|
486
|
-
|
|
487
|
-
**Ollama Setup:**
|
|
488
|
-
```bash
|
|
489
|
-
# Install Ollama
|
|
490
|
-
curl https://ollama.ai/install.sh | sh
|
|
491
|
-
|
|
492
|
-
# Pull models
|
|
493
|
-
ollama pull nomic-embed-text # For embeddings
|
|
494
|
-
ollama pull llama3 # For tag extraction
|
|
495
|
-
|
|
496
|
-
# Verify Ollama is running
|
|
497
|
-
curl http://localhost:11434/api/version
|
|
498
|
-
```
|
|
499
|
-
|
|
500
|
-
#### Using OpenAI
|
|
501
|
-
|
|
502
|
-
```ruby
|
|
503
|
-
HTM.configure do |config|
|
|
504
|
-
# Embedding configuration (OpenAI)
|
|
505
|
-
config.embedding_provider = :openai
|
|
506
|
-
config.embedding_model = 'text-embedding-3-small' # 1536 dimensions
|
|
507
|
-
config.embedding_dimensions = 1536
|
|
508
|
-
|
|
509
|
-
# Tag extraction (can mix providers)
|
|
510
|
-
config.tag_provider = :openai
|
|
511
|
-
config.tag_model = 'gpt-4'
|
|
512
|
-
|
|
513
|
-
# Logger
|
|
514
|
-
config.logger = Rails.logger # Use Rails logger
|
|
515
|
-
end
|
|
516
|
-
```
|
|
517
|
-
|
|
518
|
-
**OpenAI Setup:**
|
|
519
|
-
```bash
|
|
520
|
-
# Set your OpenAI API key
|
|
521
|
-
export OPENAI_API_KEY='sk-your-api-key-here'
|
|
522
|
-
```
|
|
523
|
-
|
|
524
|
-
**Important:** The database uses pgvector with HNSW indexing, which has a maximum dimension limit of 2000. Embeddings exceeding this limit will be automatically truncated with a warning.
|
|
525
|
-
|
|
526
|
-
#### Custom Providers
|
|
527
|
-
|
|
528
|
-
Provide your own LLM implementations:
|
|
529
|
-
|
|
530
|
-
```ruby
|
|
531
|
-
HTM.configure do |config|
|
|
532
|
-
# Custom embedding generator
|
|
533
|
-
config.embedding_generator = ->(text) {
|
|
534
|
-
# Call your custom LLM service
|
|
535
|
-
response = MyEmbeddingService.generate(text)
|
|
536
|
-
|
|
537
|
-
# Must return Array<Float>
|
|
538
|
-
response[:embedding] # e.g., [0.123, -0.456, ...]
|
|
539
|
-
}
|
|
540
|
-
|
|
541
|
-
# Custom tag extractor
|
|
542
|
-
config.tag_extractor = ->(text, existing_ontology) {
|
|
543
|
-
# Call your custom LLM service for tag extraction
|
|
544
|
-
# existing_ontology is Array<String> of recent tags for context
|
|
545
|
-
response = MyTagService.extract(text, context: existing_ontology)
|
|
546
|
-
|
|
547
|
-
# Must return Array<String> in hierarchical format
|
|
548
|
-
# e.g., ["ai:llm:embeddings", "database:postgresql"]
|
|
549
|
-
response[:tags]
|
|
550
|
-
}
|
|
551
|
-
|
|
552
|
-
# Custom token counter (optional)
|
|
553
|
-
config.token_counter = ->(text) {
|
|
554
|
-
# Your token counting implementation
|
|
555
|
-
# Must return Integer
|
|
556
|
-
MyTokenizer.count(text)
|
|
557
|
-
}
|
|
558
|
-
end
|
|
559
|
-
```
|
|
560
|
-
|
|
561
|
-
#### Logger Configuration
|
|
562
|
-
|
|
563
|
-
Customize logging behavior:
|
|
564
|
-
|
|
565
|
-
```ruby
|
|
566
|
-
HTM.configure do |config|
|
|
567
|
-
# Use custom logger
|
|
568
|
-
config.logger = Logger.new('log/htm.log')
|
|
569
|
-
config.logger.level = Logger::DEBUG
|
|
570
|
-
|
|
571
|
-
# Or use Rails logger
|
|
572
|
-
config.logger = Rails.logger
|
|
573
|
-
|
|
574
|
-
# Or disable logging
|
|
575
|
-
config.logger = Logger.new(IO::NULL)
|
|
576
|
-
config.logger.level = Logger::FATAL
|
|
577
|
-
end
|
|
578
|
-
|
|
579
|
-
# Control log level via environment variable
|
|
580
|
-
ENV['HTM_LOG_LEVEL'] = 'DEBUG' # or INFO, WARN, ERROR
|
|
581
|
-
HTM.configure # Respects HTM_LOG_LEVEL
|
|
582
|
-
```
|
|
583
|
-
|
|
584
|
-
#### Service Layer Architecture
|
|
585
|
-
|
|
586
|
-
HTM uses a service layer to process LLM responses:
|
|
587
|
-
|
|
588
|
-
- **EmbeddingService**: Calls your configured `embedding_generator`, validates responses, handles padding/truncation, and formats for storage
|
|
589
|
-
- **TagService**: Calls your configured `tag_extractor`, parses responses (String or Array), validates format, and filters invalid tags
|
|
590
|
-
|
|
591
|
-
This separation allows you to provide raw LLM access while HTM handles response processing, validation, and storage formatting.
|
|
592
|
-
|
|
593
|
-
### Recall Strategies
|
|
594
|
-
|
|
595
|
-
HTM supports three retrieval strategies:
|
|
596
|
-
|
|
597
|
-
```ruby
|
|
598
|
-
# Vector similarity search (semantic) - uses configured embedding provider
|
|
599
|
-
memories = htm.recall(
|
|
600
|
-
"HTM architecture",
|
|
601
|
-
timeframe: "last week",
|
|
602
|
-
strategy: :vector # Semantic similarity using embeddings
|
|
603
|
-
)
|
|
604
|
-
|
|
605
|
-
# Full-text search (keyword matching) - PostgreSQL full-text search with trigrams
|
|
606
|
-
memories = htm.recall(
|
|
607
|
-
"database performance",
|
|
608
|
-
timeframe: "last month",
|
|
609
|
-
strategy: :fulltext # Keyword-based matching
|
|
610
|
-
)
|
|
611
|
-
|
|
612
|
-
# Hybrid (combines both) - best of both worlds
|
|
613
|
-
memories = htm.recall(
|
|
614
|
-
"testing strategies",
|
|
615
|
-
timeframe: "yesterday",
|
|
616
|
-
strategy: :hybrid # Weighted combination of vector + full-text
|
|
617
|
-
)
|
|
618
|
-
```
|
|
619
|
-
|
|
620
|
-
**Strategy Details:**
|
|
621
|
-
- `:vector` - Semantic search using pgvector cosine similarity (requires embeddings)
|
|
622
|
-
- `:fulltext` - PostgreSQL tsvector with pg_trgm fuzzy matching
|
|
623
|
-
- `:hybrid` - Combines both with weighted scoring (default: 70% vector, 30% full-text)
|
|
624
|
-
|
|
625
|
-
## API Reference
|
|
626
|
-
|
|
627
|
-
HTM provides a minimal, focused API with only 3 core instance methods for memory operations:
|
|
628
|
-
|
|
629
|
-
### Core Memory Operations
|
|
630
|
-
|
|
631
|
-
#### `remember(content, source: "", metadata: {})`
|
|
632
|
-
|
|
633
|
-
Store information in memory. Embeddings and tags are automatically generated asynchronously.
|
|
634
|
-
|
|
635
|
-
**Parameters:**
|
|
636
|
-
- `content` (String, required) - The information to remember. Converted to string if nil. Returns ID of last node if empty.
|
|
637
|
-
- `source` (String, optional) - Where the content came from (e.g., "user", "assistant", "system"). Defaults to empty string.
|
|
638
|
-
- `metadata` (Hash, optional) - Arbitrary key-value metadata stored as JSONB. Keys must be strings or symbols. Defaults to `{}`.
|
|
639
|
-
|
|
640
|
-
**Returns:** Integer - The node ID of the stored memory
|
|
641
|
-
|
|
642
|
-
**Example:**
|
|
643
|
-
```ruby
|
|
644
|
-
# Store with source
|
|
645
|
-
node_id = htm.remember("PostgreSQL is excellent for vector search with pgvector", source: "architect")
|
|
646
|
-
|
|
647
|
-
# Store without source (uses default empty string)
|
|
648
|
-
node_id = htm.remember("HTM uses two-tier memory architecture")
|
|
649
|
-
|
|
650
|
-
# Store with metadata
|
|
651
|
-
node_id = htm.remember(
|
|
652
|
-
"User prefers dark mode",
|
|
653
|
-
metadata: { category: "preference", priority: "high", version: 2 }
|
|
100
|
+
group = HTM::RobotGroup.new(
|
|
101
|
+
name: 'support-ha',
|
|
102
|
+
active: ['primary'],
|
|
103
|
+
passive: ['standby']
|
|
654
104
|
)
|
|
655
105
|
|
|
656
|
-
#
|
|
657
|
-
|
|
658
|
-
node_id = htm.remember("") # Returns ID of last node without creating duplicate
|
|
106
|
+
# Primary fails? Standby takes over instantly
|
|
107
|
+
group.failover!
|
|
659
108
|
```
|
|
660
109
|
|
|
661
|
-
|
|
110
|
+
Real-time synchronization via PostgreSQL LISTEN/NOTIFY. Add or remove robots dynamically.
|
|
662
111
|
|
|
663
|
-
|
|
112
|
+
## Quick Start
|
|
664
113
|
|
|
665
|
-
Retrieve memories using temporal filtering and semantic/keyword search.
|
|
666
|
-
|
|
667
|
-
**Parameters:**
|
|
668
|
-
- `topic` (String, required) - Query text for semantic/keyword matching (first positional argument)
|
|
669
|
-
- `timeframe` (Range, String, optional) - Time range to search within. Default: "last 7 days"
|
|
670
|
-
- Range: `(Time.now - 3600)..Time.now`
|
|
671
|
-
- String: `"last hour"`, `"last week"`, `"yesterday"`
|
|
672
|
-
- `limit` (Integer, optional) - Maximum number of results. Default: 20
|
|
673
|
-
- `strategy` (Symbol, optional) - Search strategy. Default: `:vector`
|
|
674
|
-
- `:vector` - Semantic search using embeddings (cosine similarity)
|
|
675
|
-
- `:fulltext` - Keyword search using PostgreSQL full-text + trigrams
|
|
676
|
-
- `:hybrid` - Weighted combination (70% vector, 30% full-text)
|
|
677
|
-
- `with_relevance` (Boolean, optional) - Include dynamic relevance scores. Default: false
|
|
678
|
-
- `query_tags` (Array<String>, optional) - Filter results by tags. Default: []
|
|
679
|
-
- `metadata` (Hash, optional) - Filter results by metadata using JSONB containment (`@>`). Default: `{}`
|
|
680
|
-
|
|
681
|
-
**Returns:** Array<Hash> - Matching memories with fields: `id`, `content`, `source`, `created_at`, `access_count`, `metadata`, (optionally `relevance`)
|
|
682
|
-
|
|
683
|
-
**Example:**
|
|
684
|
-
```ruby
|
|
685
|
-
# Basic recall with time range
|
|
686
|
-
memories = htm.recall(
|
|
687
|
-
"database architecture",
|
|
688
|
-
timeframe: (Time.now - 86400)..Time.now
|
|
689
|
-
)
|
|
690
|
-
|
|
691
|
-
# Using human-readable timeframe
|
|
692
|
-
memories = htm.recall(
|
|
693
|
-
"PostgreSQL performance",
|
|
694
|
-
timeframe: "last week",
|
|
695
|
-
strategy: :hybrid,
|
|
696
|
-
limit: 10
|
|
697
|
-
)
|
|
698
|
-
|
|
699
|
-
# With relevance scoring
|
|
700
|
-
memories = htm.recall(
|
|
701
|
-
"HTM design decisions",
|
|
702
|
-
timeframe: "last month",
|
|
703
|
-
with_relevance: true,
|
|
704
|
-
query_tags: ["architecture"]
|
|
705
|
-
)
|
|
706
|
-
# => [{ "id" => 123, "content" => "...", "relevance" => 0.92, ... }, ...]
|
|
707
|
-
|
|
708
|
-
# Filter by metadata
|
|
709
|
-
memories = htm.recall(
|
|
710
|
-
"user preferences",
|
|
711
|
-
metadata: { category: "preference" }
|
|
712
|
-
)
|
|
713
|
-
# => Returns only nodes with metadata containing { category: "preference" }
|
|
714
|
-
|
|
715
|
-
# Combine metadata with other filters
|
|
716
|
-
memories = htm.recall(
|
|
717
|
-
"settings",
|
|
718
|
-
timeframe: "last month",
|
|
719
|
-
strategy: :hybrid,
|
|
720
|
-
metadata: { priority: "high", version: 2 }
|
|
721
|
-
)
|
|
722
|
-
```
|
|
723
|
-
|
|
724
|
-
---
|
|
725
|
-
|
|
726
|
-
#### `forget(node_id, confirm: false)`
|
|
727
|
-
|
|
728
|
-
Permanently delete a memory from both working memory and long-term storage.
|
|
729
|
-
|
|
730
|
-
**Parameters:**
|
|
731
|
-
- `node_id` (Integer, required) - The ID of the node to delete
|
|
732
|
-
- `confirm` (Symbol, Boolean, required) - Must be `:confirmed` or `true` to proceed
|
|
733
|
-
|
|
734
|
-
**Returns:** Boolean - `true` if deleted, `false` if not found
|
|
735
|
-
|
|
736
|
-
**Safety:** This is the ONLY way to delete memories. Requires explicit confirmation to prevent accidental deletion.
|
|
737
|
-
|
|
738
|
-
**Example:**
|
|
739
|
-
```ruby
|
|
740
|
-
# Delete with symbol confirmation (recommended)
|
|
741
|
-
htm.forget(node_id, confirm: :confirmed)
|
|
742
|
-
|
|
743
|
-
# Delete with boolean confirmation
|
|
744
|
-
htm.forget(node_id, confirm: true)
|
|
745
|
-
|
|
746
|
-
# Will raise error - confirmation required
|
|
747
|
-
htm.forget(node_id) # ArgumentError: Must confirm deletion
|
|
748
|
-
htm.forget(node_id, confirm: false) # ArgumentError: Must confirm deletion
|
|
749
|
-
```
|
|
750
|
-
|
|
751
|
-
---
|
|
752
|
-
|
|
753
|
-
### Complete Usage Example
|
|
754
|
-
|
|
755
|
-
```ruby
|
|
756
|
-
require 'htm'
|
|
757
|
-
|
|
758
|
-
# Configure once globally (optional - uses defaults if not called)
|
|
759
|
-
HTM.configure do |config|
|
|
760
|
-
config.embedding_provider = :ollama
|
|
761
|
-
config.embedding_model = 'nomic-embed-text'
|
|
762
|
-
config.tag_provider = :ollama
|
|
763
|
-
config.tag_model = 'llama3'
|
|
764
|
-
end
|
|
765
|
-
|
|
766
|
-
# Initialize for your robot
|
|
767
|
-
htm = HTM.new(robot_name: "Assistant", working_memory_size: 128_000)
|
|
768
|
-
|
|
769
|
-
# Store information
|
|
770
|
-
htm.remember("PostgreSQL with pgvector for vector search", source: "architect")
|
|
771
|
-
htm.remember("User prefers dark mode", source: "user")
|
|
772
|
-
htm.remember("Use debug_me for debugging, not puts", source: "system")
|
|
773
|
-
|
|
774
|
-
# Retrieve by time + topic
|
|
775
|
-
recent = htm.recall(
|
|
776
|
-
"PostgreSQL",
|
|
777
|
-
timeframe: "last week",
|
|
778
|
-
strategy: :hybrid,
|
|
779
|
-
limit: 5
|
|
780
|
-
)
|
|
781
|
-
|
|
782
|
-
# Delete if needed (requires node ID from remember or recall)
|
|
783
|
-
htm.forget(node_id, confirm: :confirmed)
|
|
784
|
-
```
|
|
785
|
-
|
|
786
|
-
## Use with Rails
|
|
787
|
-
|
|
788
|
-
HTM is designed to integrate seamlessly with Ruby on Rails applications. It uses ActiveRecord models and follows Rails conventions, making it easy to add AI memory capabilities to existing Rails apps.
|
|
789
|
-
|
|
790
|
-
### Why HTM Works Well with Rails
|
|
791
|
-
|
|
792
|
-
**1. Uses ActiveRecord**
|
|
793
|
-
- HTM models are standard ActiveRecord classes with associations, validations, and scopes
|
|
794
|
-
- Follows Rails naming conventions and patterns
|
|
795
|
-
- Models are namespaced under `HTM::Models::` to avoid conflicts
|
|
796
|
-
|
|
797
|
-
**2. Standard Rails Configuration**
|
|
798
|
-
- Uses `config/database.yml` (Rails convention)
|
|
799
|
-
- Respects `RAILS_ENV` environment variable
|
|
800
|
-
- Supports ERB in configuration files
|
|
801
|
-
|
|
802
|
-
**3. Separate Database**
|
|
803
|
-
- HTM uses its own PostgreSQL database connection
|
|
804
|
-
- Won't interfere with your Rails app's database
|
|
805
|
-
- Configured with `prepared_statements: false` and `advisory_locks: false` for compatibility
|
|
806
|
-
- Thread-safe for Rails multi-threading
|
|
807
|
-
|
|
808
|
-
**4. Connection Management**
|
|
809
|
-
- Separate connection pool from your Rails app
|
|
810
|
-
- Configurable pool size and timeouts
|
|
811
|
-
- Proper cleanup and disconnection support
|
|
812
|
-
|
|
813
|
-
### Integration Steps
|
|
814
|
-
|
|
815
|
-
#### 1. Add to Gemfile
|
|
816
|
-
|
|
817
|
-
```ruby
|
|
818
|
-
gem 'htm'
|
|
819
|
-
```
|
|
820
|
-
|
|
821
|
-
Then run:
|
|
822
|
-
```bash
|
|
823
|
-
bundle install
|
|
824
|
-
```
|
|
825
|
-
|
|
826
|
-
#### 2. Configure Environment Variables
|
|
827
|
-
|
|
828
|
-
Add HTM database configuration to your Rails environment. You can use `.env` files (with `dotenv-rails`) or set them directly:
|
|
829
|
-
|
|
830
|
-
```ruby
|
|
831
|
-
# .env or config/application.rb
|
|
832
|
-
HTM_DBURL=postgresql://user:pass@localhost:5432/htm_production
|
|
833
|
-
```
|
|
834
|
-
|
|
835
|
-
Or use individual variables:
|
|
836
|
-
```ruby
|
|
837
|
-
HTM_DBHOST=localhost
|
|
838
|
-
HTM_DBNAME=htm_production
|
|
839
|
-
HTM_DBUSER=postgres
|
|
840
|
-
HTM_DBPASS=password
|
|
841
|
-
HTM_DBPORT=5432
|
|
842
|
-
```
|
|
843
|
-
|
|
844
|
-
#### 3. Create Initializer
|
|
845
|
-
|
|
846
|
-
Create `config/initializers/htm.rb`:
|
|
847
|
-
|
|
848
|
-
```ruby
|
|
849
|
-
# config/initializers/htm.rb
|
|
850
|
-
|
|
851
|
-
# Configure HTM
|
|
852
|
-
HTM.configure do |config|
|
|
853
|
-
# Use Rails logger
|
|
854
|
-
config.logger = Rails.logger
|
|
855
|
-
|
|
856
|
-
# Embedding configuration (optional - uses Ollama defaults if not set)
|
|
857
|
-
config.embedding_provider = :ollama
|
|
858
|
-
config.embedding_model = 'nomic-embed-text'
|
|
859
|
-
|
|
860
|
-
# Tag extraction configuration (optional - uses Ollama defaults if not set)
|
|
861
|
-
config.tag_provider = :ollama
|
|
862
|
-
config.tag_model = 'llama3'
|
|
863
|
-
|
|
864
|
-
# Custom providers (optional)
|
|
865
|
-
# config.embedding_generator = ->(text) { YourEmbeddingService.call(text) }
|
|
866
|
-
# config.tag_extractor = ->(text, ontology) { YourTagService.call(text, ontology) }
|
|
867
|
-
end
|
|
868
|
-
|
|
869
|
-
# Establish HTM database connection
|
|
870
|
-
HTM::ActiveRecordConfig.establish_connection!
|
|
871
|
-
|
|
872
|
-
# Verify required PostgreSQL extensions are installed
|
|
873
|
-
HTM::ActiveRecordConfig.verify_extensions!
|
|
874
|
-
|
|
875
|
-
Rails.logger.info "HTM initialized with database: #{HTM::Database.default_config[:database]}"
|
|
876
|
-
```
|
|
877
|
-
|
|
878
|
-
#### 4. Set Up Database
|
|
879
|
-
|
|
880
|
-
Run the HTM database setup:
|
|
881
|
-
|
|
882
|
-
```bash
|
|
883
|
-
# Using HTM rake tasks
|
|
884
|
-
rake htm:db:setup
|
|
885
|
-
|
|
886
|
-
# Or manually in Rails console
|
|
887
|
-
rails c
|
|
888
|
-
> HTM::Database.setup
|
|
889
|
-
```
|
|
890
|
-
|
|
891
|
-
#### 5. Use in Your Rails App
|
|
892
|
-
|
|
893
|
-
**In Controllers:**
|
|
894
|
-
|
|
895
|
-
```ruby
|
|
896
|
-
class ChatsController < ApplicationController
|
|
897
|
-
before_action :initialize_memory
|
|
898
|
-
|
|
899
|
-
def create
|
|
900
|
-
# Add user message to memory (embeddings + tags auto-extracted asynchronously)
|
|
901
|
-
@memory.remember(
|
|
902
|
-
params[:message],
|
|
903
|
-
source: "user-#{current_user.id}"
|
|
904
|
-
)
|
|
905
|
-
|
|
906
|
-
# Recall relevant context for the conversation
|
|
907
|
-
context = @memory.create_context(strategy: :balanced)
|
|
908
|
-
|
|
909
|
-
# Use context with your LLM to generate response
|
|
910
|
-
response = generate_llm_response(context, params[:message])
|
|
911
|
-
|
|
912
|
-
# Store assistant's response
|
|
913
|
-
@memory.remember(
|
|
914
|
-
response,
|
|
915
|
-
source: "assistant"
|
|
916
|
-
)
|
|
917
|
-
|
|
918
|
-
render json: { response: response }
|
|
919
|
-
end
|
|
920
|
-
|
|
921
|
-
private
|
|
922
|
-
|
|
923
|
-
def initialize_memory
|
|
924
|
-
@memory = HTM.new(
|
|
925
|
-
robot_name: "ChatBot-#{current_user.id}",
|
|
926
|
-
working_memory_size: 128_000
|
|
927
|
-
)
|
|
928
|
-
end
|
|
929
|
-
end
|
|
930
|
-
```
|
|
931
|
-
|
|
932
|
-
**In Background Jobs:**
|
|
933
|
-
|
|
934
|
-
```ruby
|
|
935
|
-
class ProcessDocumentJob < ApplicationJob
|
|
936
|
-
queue_as :default
|
|
937
|
-
|
|
938
|
-
def perform(document_id)
|
|
939
|
-
document = Document.find(document_id)
|
|
940
|
-
|
|
941
|
-
# Initialize HTM for this job
|
|
942
|
-
memory = HTM.new(robot_name: "DocumentProcessor")
|
|
943
|
-
|
|
944
|
-
# Store document content in memory (embeddings + tags auto-extracted asynchronously)
|
|
945
|
-
memory.remember(
|
|
946
|
-
document.content,
|
|
947
|
-
source: "document-#{document.id}"
|
|
948
|
-
)
|
|
949
|
-
|
|
950
|
-
# Recall related documents (uses vector similarity)
|
|
951
|
-
related = memory.recall(
|
|
952
|
-
document.category,
|
|
953
|
-
timeframe: "last month",
|
|
954
|
-
strategy: :vector
|
|
955
|
-
)
|
|
956
|
-
|
|
957
|
-
# Process with context...
|
|
958
|
-
end
|
|
959
|
-
end
|
|
960
|
-
```
|
|
961
|
-
|
|
962
|
-
**In Models (as a concern):**
|
|
963
|
-
|
|
964
|
-
```ruby
|
|
965
|
-
# app/models/concerns/memorable.rb
|
|
966
|
-
module Memorable
|
|
967
|
-
extend ActiveSupport::Concern
|
|
968
|
-
|
|
969
|
-
included do
|
|
970
|
-
after_create :store_in_memory
|
|
971
|
-
after_update :update_memory
|
|
972
|
-
end
|
|
973
|
-
|
|
974
|
-
def store_in_memory
|
|
975
|
-
memory = HTM.new(robot_name: "Rails-#{Rails.env}")
|
|
976
|
-
|
|
977
|
-
memory.remember(
|
|
978
|
-
memory_content,
|
|
979
|
-
source: "#{self.class.name}-#{id}"
|
|
980
|
-
)
|
|
981
|
-
end
|
|
982
|
-
|
|
983
|
-
def update_memory
|
|
984
|
-
# Update existing memory node
|
|
985
|
-
store_in_memory
|
|
986
|
-
end
|
|
987
|
-
|
|
988
|
-
private
|
|
989
|
-
|
|
990
|
-
def memory_content
|
|
991
|
-
# Override in model to customize what gets stored
|
|
992
|
-
attributes.to_json
|
|
993
|
-
end
|
|
994
|
-
end
|
|
995
|
-
|
|
996
|
-
# Then in your model:
|
|
997
|
-
class Article < ApplicationRecord
|
|
998
|
-
include Memorable
|
|
999
|
-
|
|
1000
|
-
private
|
|
1001
|
-
|
|
1002
|
-
def memory_content
|
|
1003
|
-
"Article: #{title}\n\n#{body}\n\nCategory: #{category}"
|
|
1004
|
-
end
|
|
1005
|
-
end
|
|
1006
|
-
```
|
|
1007
|
-
|
|
1008
|
-
### Multi-Database Configuration (Rails 6+)
|
|
1009
|
-
|
|
1010
|
-
If you want to use Rails' built-in multi-database support, you can configure HTM in your `config/database.yml`:
|
|
1011
|
-
|
|
1012
|
-
```yaml
|
|
1013
|
-
# config/database.yml
|
|
1014
|
-
production:
|
|
1015
|
-
primary:
|
|
1016
|
-
<<: *default
|
|
1017
|
-
database: myapp_production
|
|
1018
|
-
|
|
1019
|
-
htm:
|
|
1020
|
-
adapter: postgresql
|
|
1021
|
-
encoding: unicode
|
|
1022
|
-
database: <%= ENV['HTM_DBNAME'] || 'htm_production' %>
|
|
1023
|
-
host: <%= ENV['HTM_DBHOST'] || 'localhost' %>
|
|
1024
|
-
port: <%= ENV['HTM_DBPORT'] || 5432 %>
|
|
1025
|
-
username: <%= ENV['HTM_DBUSER'] || 'postgres' %>
|
|
1026
|
-
password: <%= ENV['HTM_DBPASS'] %>
|
|
1027
|
-
pool: <%= ENV['HTM_DB_POOL_SIZE'] || 10 %>
|
|
1028
|
-
```
|
|
1029
|
-
|
|
1030
|
-
Then update your HTM initializer:
|
|
1031
|
-
|
|
1032
|
-
```ruby
|
|
1033
|
-
# config/initializers/htm.rb
|
|
1034
|
-
HTM::ActiveRecordConfig.establish_connection!
|
|
1035
|
-
|
|
1036
|
-
# Or use Rails' connection
|
|
1037
|
-
# ActiveRecord::Base.connected_to(role: :writing, shard: :htm) do
|
|
1038
|
-
# HTM::ActiveRecordConfig.verify_extensions!
|
|
1039
|
-
# end
|
|
1040
|
-
```
|
|
1041
|
-
|
|
1042
|
-
### Testing with Rails
|
|
1043
|
-
|
|
1044
|
-
**In RSpec:**
|
|
1045
|
-
|
|
1046
|
-
```ruby
|
|
1047
|
-
# spec/rails_helper.rb
|
|
1048
|
-
RSpec.configure do |config|
|
|
1049
|
-
config.before(:suite) do
|
|
1050
|
-
HTM::ActiveRecordConfig.establish_connection!
|
|
1051
|
-
end
|
|
1052
|
-
|
|
1053
|
-
config.after(:suite) do
|
|
1054
|
-
HTM::ActiveRecordConfig.disconnect!
|
|
1055
|
-
end
|
|
1056
|
-
end
|
|
1057
|
-
|
|
1058
|
-
# spec/features/chat_spec.rb
|
|
1059
|
-
RSpec.describe "Chat with memory", type: :feature do
|
|
1060
|
-
let(:memory) { HTM.new(robot_name: "TestBot") }
|
|
1061
|
-
|
|
1062
|
-
before do
|
|
1063
|
-
memory.remember("User prefers Ruby over Python", source: "user")
|
|
1064
|
-
end
|
|
1065
|
-
|
|
1066
|
-
it "recalls user preferences" do
|
|
1067
|
-
context = memory.create_context(strategy: :balanced)
|
|
1068
|
-
expect(context).to include("Ruby")
|
|
1069
|
-
end
|
|
1070
|
-
end
|
|
1071
|
-
```
|
|
1072
|
-
|
|
1073
|
-
**In Minitest:**
|
|
1074
|
-
|
|
1075
|
-
```ruby
|
|
1076
|
-
# test/test_helper.rb
|
|
1077
|
-
class ActiveSupport::TestCase
|
|
1078
|
-
setup do
|
|
1079
|
-
HTM::ActiveRecordConfig.establish_connection! unless HTM::ActiveRecordConfig.connected?
|
|
1080
|
-
end
|
|
1081
|
-
end
|
|
1082
|
-
|
|
1083
|
-
# test/integration/chat_test.rb
|
|
1084
|
-
class ChatTest < ActionDispatch::IntegrationTest
|
|
1085
|
-
test "stores chat messages in memory" do
|
|
1086
|
-
memory = HTM.new(robot_name: "TestBot")
|
|
1087
|
-
|
|
1088
|
-
post chats_path, params: { message: "Hello!" }
|
|
1089
|
-
|
|
1090
|
-
assert_response :success
|
|
1091
|
-
|
|
1092
|
-
# Verify message was stored
|
|
1093
|
-
nodes = memory.recall("Hello", timeframe: "last hour")
|
|
1094
|
-
assert_not_empty nodes
|
|
1095
|
-
end
|
|
1096
|
-
end
|
|
1097
|
-
```
|
|
1098
|
-
|
|
1099
|
-
### Production Considerations
|
|
1100
|
-
|
|
1101
|
-
1. **Connection Pooling**: Set appropriate pool size based on your Rails app's concurrency:
|
|
1102
|
-
```ruby
|
|
1103
|
-
ENV['HTM_DB_POOL_SIZE'] = '20' # Match or exceed Rails pool
|
|
1104
|
-
```
|
|
1105
|
-
|
|
1106
|
-
2. **Separate Database Server**: For production, use a dedicated PostgreSQL instance for HTM
|
|
1107
|
-
|
|
1108
|
-
3. **Required Extensions**: Ensure `pgvector` and `pg_trgm` extensions are installed on your PostgreSQL server
|
|
1109
|
-
|
|
1110
|
-
4. **Memory Management**: HTM's working memory is per-instance. Consider using a singleton pattern or Rails cache for shared instances
|
|
1111
|
-
|
|
1112
|
-
5. **Background Processing**: Use Sidekiq or similar for embedding generation if processing large amounts of data
|
|
1113
|
-
|
|
1114
|
-
### Example Rails App Structure
|
|
1115
|
-
|
|
1116
|
-
```
|
|
1117
|
-
app/
|
|
1118
|
-
├── controllers/
|
|
1119
|
-
│ └── chats_controller.rb # Uses HTM for conversation memory
|
|
1120
|
-
├── jobs/
|
|
1121
|
-
│ └── process_document_job.rb # Background HTM processing
|
|
1122
|
-
├── models/
|
|
1123
|
-
│ ├── concerns/
|
|
1124
|
-
│ │ └── memorable.rb # HTM integration concern
|
|
1125
|
-
│ └── article.rb # Includes Memorable
|
|
1126
|
-
└── services/
|
|
1127
|
-
└── memory_service.rb # Centralized HTM access
|
|
1128
|
-
|
|
1129
|
-
config/
|
|
1130
|
-
├── initializers/
|
|
1131
|
-
│ └── htm.rb # HTM setup
|
|
1132
|
-
└── database.yml # Optional: multi-database config
|
|
1133
|
-
|
|
1134
|
-
.env
|
|
1135
|
-
HTM_DBURL=postgresql://... # HTM database connection
|
|
1136
|
-
OPENAI_API_KEY=sk-... # If using OpenAI embeddings
|
|
1137
|
-
```
|
|
1138
|
-
|
|
1139
|
-
## Environment Variables
|
|
1140
|
-
|
|
1141
|
-
HTM uses environment variables for database and service configuration. These can be set in your shell profile, a `.env` file, or exported directly.
|
|
1142
|
-
|
|
1143
|
-
### Database Configuration
|
|
1144
|
-
|
|
1145
|
-
#### HTM_DBURL (Recommended)
|
|
1146
|
-
|
|
1147
|
-
The preferred method for database configuration is a single connection URL:
|
|
1148
|
-
|
|
1149
|
-
```bash
|
|
1150
|
-
export HTM_DBURL="postgresql://username:password@host:port/dbname?sslmode=require"
|
|
1151
|
-
```
|
|
1152
|
-
|
|
1153
|
-
**Format:** `postgresql://USER:PASSWORD@HOST:PORT/DATABASE?sslmode=MODE`
|
|
1154
|
-
|
|
1155
|
-
**Example for Local PostgreSQL:**
|
|
1156
|
-
```bash
|
|
1157
|
-
export HTM_DBURL="postgresql://postgres:postgres@localhost:5432/htm_dev?sslmode=disable"
|
|
1158
|
-
```
|
|
1159
|
-
|
|
1160
|
-
#### Individual Database Parameters (Alternative)
|
|
1161
|
-
|
|
1162
|
-
If `HTM_DBURL` is not set, HTM will use individual parameters:
|
|
1163
|
-
|
|
1164
|
-
| Variable | Description | Default | Required |
|
|
1165
|
-
|----------|-------------|---------|----------|
|
|
1166
|
-
| `HTM_DBNAME` | Database name | None | Yes |
|
|
1167
|
-
| `HTM_DBUSER` | Database username | None | Yes |
|
|
1168
|
-
| `HTM_DBPASS` | Database password | None | Yes |
|
|
1169
|
-
| `HTM_DBHOST` | Database hostname or IP | `localhost` | No |
|
|
1170
|
-
| `HTM_DBPORT` | Database port | `5432` | No |
|
|
1171
|
-
| `HTM_SERVICE_NAME` | Service identifier (informational only) | None | No |
|
|
1172
|
-
|
|
1173
|
-
**Example:**
|
|
1174
114
|
```bash
|
|
1175
|
-
|
|
1176
|
-
export HTM_DBUSER="postgres"
|
|
1177
|
-
export HTM_DBPASS="secure_password_here"
|
|
1178
|
-
export HTM_DBHOST="localhost"
|
|
1179
|
-
export HTM_DBPORT="5432"
|
|
1180
|
-
export HTM_SERVICE_NAME="production-db"
|
|
1181
|
-
```
|
|
1182
|
-
|
|
1183
|
-
**Configuration Priority:**
|
|
1184
|
-
1. `HTM_DBURL` (if set, this takes precedence)
|
|
1185
|
-
2. Individual parameters (`HTM_DBNAME`, `HTM_DBUSER`, etc.)
|
|
1186
|
-
3. Direct configuration passed to `HTM.new(db_config: {...})`
|
|
1187
|
-
|
|
1188
|
-
### LLM Provider Configuration
|
|
1189
|
-
|
|
1190
|
-
HTM configuration is done programmatically via `HTM.configure` (see [Configuration](#configuration) section). However, a few environment variables are still used:
|
|
1191
|
-
|
|
1192
|
-
#### OPENAI_API_KEY
|
|
1193
|
-
|
|
1194
|
-
Required when using OpenAI as your embedding or tag extraction provider:
|
|
1195
|
-
|
|
1196
|
-
```bash
|
|
1197
|
-
export OPENAI_API_KEY="sk-proj-your-api-key-here"
|
|
115
|
+
gem install htm
|
|
1198
116
|
```
|
|
1199
117
|
|
|
1200
|
-
This is required when you configure:
|
|
1201
118
|
```ruby
|
|
1202
|
-
|
|
1203
|
-
config.embedding_provider = :openai
|
|
1204
|
-
# or
|
|
1205
|
-
config.tag_provider = :openai
|
|
1206
|
-
end
|
|
1207
|
-
```
|
|
1208
|
-
|
|
1209
|
-
#### OLLAMA_URL
|
|
1210
|
-
|
|
1211
|
-
Custom Ollama server URL (optional):
|
|
1212
|
-
|
|
1213
|
-
```bash
|
|
1214
|
-
export OLLAMA_URL="http://localhost:11434" # Default
|
|
1215
|
-
export OLLAMA_URL="http://ollama-server:11434" # Custom
|
|
1216
|
-
```
|
|
1217
|
-
|
|
1218
|
-
HTM defaults to `http://localhost:11434` if not set.
|
|
1219
|
-
|
|
1220
|
-
#### HTM_LOG_LEVEL
|
|
1221
|
-
|
|
1222
|
-
Control logging verbosity:
|
|
1223
|
-
|
|
1224
|
-
```bash
|
|
1225
|
-
export HTM_LOG_LEVEL="DEBUG" # DEBUG, INFO, WARN, ERROR, FATAL
|
|
1226
|
-
export HTM_LOG_LEVEL="INFO" # Default
|
|
1227
|
-
```
|
|
1228
|
-
|
|
1229
|
-
This is used by the default logger when `HTM.configure` is called without a custom logger.
|
|
1230
|
-
|
|
1231
|
-
#### HTM_TELEMETRY_ENABLED
|
|
1232
|
-
|
|
1233
|
-
Enable OpenTelemetry metrics collection:
|
|
1234
|
-
|
|
1235
|
-
```bash
|
|
1236
|
-
export HTM_TELEMETRY_ENABLED="true" # Enable telemetry
|
|
1237
|
-
export HTM_TELEMETRY_ENABLED="false" # Disable (default)
|
|
1238
|
-
```
|
|
1239
|
-
|
|
1240
|
-
When enabled, HTM emits metrics to configured OpenTelemetry collectors. See [Telemetry](#telemetry-opentelemetry) for details.
|
|
1241
|
-
|
|
1242
|
-
### Quick Setup Examples
|
|
1243
|
-
|
|
1244
|
-
#### Local Development (PostgreSQL)
|
|
1245
|
-
|
|
1246
|
-
```bash
|
|
1247
|
-
# Simple local setup
|
|
1248
|
-
export HTM_DBURL="postgresql://postgres:postgres@localhost:5432/htm_dev?sslmode=disable"
|
|
1249
|
-
|
|
1250
|
-
# With custom user
|
|
1251
|
-
export HTM_DBURL="postgresql://myuser:mypass@localhost:5432/htm_dev"
|
|
1252
|
-
```
|
|
1253
|
-
|
|
1254
|
-
#### With OpenAI Embeddings
|
|
1255
|
-
|
|
1256
|
-
```bash
|
|
1257
|
-
# Database + OpenAI
|
|
1258
|
-
export HTM_DBURL="postgresql://user:pass@host:port/db?sslmode=require"
|
|
1259
|
-
export OPENAI_API_KEY="sk-proj-your-api-key-here"
|
|
1260
|
-
```
|
|
1261
|
-
|
|
1262
|
-
### Persistent Configuration
|
|
1263
|
-
|
|
1264
|
-
To make environment variables permanent, add them to your shell profile:
|
|
1265
|
-
|
|
1266
|
-
**For Bash (~/.bashrc or ~/.bash_profile):**
|
|
1267
|
-
```bash
|
|
1268
|
-
echo 'export HTM_DBURL="postgresql://user:pass@host:port/db?sslmode=require"' >> ~/.bashrc
|
|
1269
|
-
source ~/.bashrc
|
|
1270
|
-
```
|
|
1271
|
-
|
|
1272
|
-
**For Zsh (~/.zshrc):**
|
|
1273
|
-
```bash
|
|
1274
|
-
echo 'export HTM_DBURL="postgresql://user:pass@host:port/db?sslmode=require"' >> ~/.zshrc
|
|
1275
|
-
source ~/.zshrc
|
|
1276
|
-
```
|
|
1277
|
-
|
|
1278
|
-
**Or create a dedicated file (~/.bashrc__htm):**
|
|
1279
|
-
```bash
|
|
1280
|
-
# ~/.bashrc__htm
|
|
1281
|
-
export HTM_DBURL="postgresql://user:pass@host:port/db?sslmode=require"
|
|
1282
|
-
export OPENAI_API_KEY="sk-proj-your-api-key"
|
|
1283
|
-
export HTM_SERVICE_NAME="my-service"
|
|
1284
|
-
|
|
1285
|
-
# Then source it from your main profile
|
|
1286
|
-
echo 'source ~/.bashrc__htm' >> ~/.bashrc
|
|
1287
|
-
```
|
|
1288
|
-
|
|
1289
|
-
### Verification
|
|
1290
|
-
|
|
1291
|
-
Verify your configuration:
|
|
1292
|
-
|
|
1293
|
-
```bash
|
|
1294
|
-
# Check database connection
|
|
1295
|
-
ruby test_connection.rb
|
|
1296
|
-
|
|
1297
|
-
# Or in Ruby
|
|
1298
|
-
irb -r ./lib/htm
|
|
1299
|
-
> HTM::Database.default_config
|
|
1300
|
-
> # Should return a hash with your connection parameters
|
|
1301
|
-
```
|
|
1302
|
-
|
|
1303
|
-
## Development
|
|
1304
|
-
|
|
1305
|
-
After checking out the repo, run:
|
|
1306
|
-
|
|
1307
|
-
```bash
|
|
1308
|
-
# Install dependencies
|
|
1309
|
-
bundle install
|
|
1310
|
-
|
|
1311
|
-
# Enable direnv (loads .envrc environment variables)
|
|
1312
|
-
direnv allow
|
|
1313
|
-
|
|
1314
|
-
# Run tests
|
|
1315
|
-
rake test
|
|
1316
|
-
|
|
1317
|
-
# Run example
|
|
1318
|
-
ruby examples/basic_usage.rb
|
|
1319
|
-
```
|
|
1320
|
-
|
|
1321
|
-
### Database Management
|
|
1322
|
-
|
|
1323
|
-
HTM provides comprehensive rake tasks under the `htm:db` namespace for managing the database:
|
|
1324
|
-
|
|
1325
|
-
```bash
|
|
1326
|
-
# List all database tasks
|
|
1327
|
-
rake -T htm:db
|
|
1328
|
-
|
|
1329
|
-
# Set up database (create schema + run migrations)
|
|
1330
|
-
rake htm:db:setup
|
|
1331
|
-
|
|
1332
|
-
# Run pending migrations only
|
|
1333
|
-
rake htm:db:migrate
|
|
1334
|
-
|
|
1335
|
-
# Show migration status (which migrations are applied)
|
|
1336
|
-
rake htm:db:status
|
|
1337
|
-
|
|
1338
|
-
# Show database info (size, tables, extensions, row counts)
|
|
1339
|
-
rake htm:db:info
|
|
1340
|
-
|
|
1341
|
-
# Test database connection
|
|
1342
|
-
rake htm:db:test
|
|
1343
|
-
|
|
1344
|
-
# Open PostgreSQL console (interactive psql session)
|
|
1345
|
-
rake htm:db:console
|
|
1346
|
-
|
|
1347
|
-
# Seed database with sample data
|
|
1348
|
-
rake htm:db:seed
|
|
1349
|
-
|
|
1350
|
-
# Drop all HTM tables (WARNING: destructive!)
|
|
1351
|
-
rake htm:db:drop
|
|
1352
|
-
|
|
1353
|
-
# Drop and recreate database (WARNING: destructive!)
|
|
1354
|
-
rake htm:db:reset
|
|
1355
|
-
```
|
|
1356
|
-
|
|
1357
|
-
**Important**: Make sure `direnv allow` has been run once in the project directory to load database environment variables from `.envrc`. Alternatively, you can manually export the environment variables:
|
|
1358
|
-
|
|
1359
|
-
```bash
|
|
1360
|
-
# Manually export HTM_DBURL
|
|
1361
|
-
export HTM_DBURL="postgresql://user:password@host:port/dbname?sslmode=require"
|
|
1362
|
-
```
|
|
1363
|
-
|
|
1364
|
-
**Common Workflows**:
|
|
1365
|
-
|
|
1366
|
-
```bash
|
|
1367
|
-
# Initial setup (first time)
|
|
1368
|
-
direnv allow
|
|
1369
|
-
rake htm:db:setup # Creates schema and runs migrations
|
|
1370
|
-
|
|
1371
|
-
# After pulling new migrations
|
|
1372
|
-
rake htm:db:migrate # Run pending migrations
|
|
1373
|
-
|
|
1374
|
-
# Check database state
|
|
1375
|
-
rake htm:db:status # See migration status
|
|
1376
|
-
rake htm:db:info # See database details
|
|
1377
|
-
|
|
1378
|
-
# Reset database (development only!)
|
|
1379
|
-
rake htm:db:reset # Drops and recreates everything
|
|
1380
|
-
|
|
1381
|
-
# Open database console for debugging
|
|
1382
|
-
rake htm:db:console # Opens psql
|
|
1383
|
-
```
|
|
1384
|
-
|
|
1385
|
-
### Async Job Management
|
|
1386
|
-
|
|
1387
|
-
HTM provides rake tasks under the `htm:jobs` namespace for managing async embedding and tag extraction jobs:
|
|
1388
|
-
|
|
1389
|
-
```bash
|
|
1390
|
-
# List all job management tasks
|
|
1391
|
-
rake -T htm:jobs
|
|
1392
|
-
|
|
1393
|
-
# Show statistics for async processing
|
|
1394
|
-
rake htm:jobs:stats
|
|
1395
|
-
|
|
1396
|
-
# Process pending jobs
|
|
1397
|
-
rake htm:jobs:process_embeddings # Process nodes without embeddings
|
|
1398
|
-
rake htm:jobs:process_tags # Process nodes without tags
|
|
1399
|
-
rake htm:jobs:process_all # Process both
|
|
1400
|
-
|
|
1401
|
-
# Reprocess all nodes (force regeneration)
|
|
1402
|
-
rake htm:jobs:reprocess_embeddings # WARNING: Prompts for confirmation
|
|
1403
|
-
|
|
1404
|
-
# Debugging and maintenance
|
|
1405
|
-
rake htm:jobs:failed # Show stuck jobs (>1 hour old)
|
|
1406
|
-
rake htm:jobs:clear_all # Clear all embeddings/tags (testing only)
|
|
1407
|
-
```
|
|
1408
|
-
|
|
1409
|
-
**Common Workflows**:
|
|
1410
|
-
|
|
1411
|
-
```bash
|
|
1412
|
-
# Monitor async processing
|
|
1413
|
-
rake htm:jobs:stats
|
|
119
|
+
require 'htm'
|
|
1414
120
|
|
|
1415
|
-
#
|
|
1416
|
-
|
|
121
|
+
# Initialize
|
|
122
|
+
htm = HTM.new(robot_name: "MyAssistant")
|
|
1417
123
|
|
|
1418
|
-
#
|
|
1419
|
-
|
|
1420
|
-
rake htm:jobs:process_embeddings # Retry failed embeddings
|
|
124
|
+
# Remember
|
|
125
|
+
htm.remember("User's project uses Rails 7 with PostgreSQL")
|
|
1421
126
|
|
|
1422
|
-
#
|
|
1423
|
-
|
|
1424
|
-
rake htm:jobs:clear_all # Clear everything and start fresh
|
|
127
|
+
# Recall
|
|
128
|
+
memories = htm.recall("tech stack", timeframe: "last month")
|
|
1425
129
|
```
|
|
1426
130
|
|
|
1427
|
-
|
|
131
|
+
For MCP integration, database setup, and configuration options, see the [full documentation](https://madbomber.github.io/htm).
|
|
1428
132
|
|
|
1429
|
-
##
|
|
133
|
+
## Features at a Glance
|
|
1430
134
|
|
|
1431
|
-
|
|
135
|
+
| Feature | Description |
|
|
136
|
+
|---------|-------------|
|
|
137
|
+
| **MCP Server** | 23 tools for AI assistant integration |
|
|
138
|
+
| **Hive Mind** | Shared memory across all robots |
|
|
139
|
+
| **Vector Search** | Semantic retrieval with pgvector |
|
|
140
|
+
| **Hybrid Search** | Combines vector + full-text matching |
|
|
141
|
+
| **Temporal Queries** | "last week", "yesterday", date ranges |
|
|
142
|
+
| **Auto-Tagging** | LLM extracts hierarchical tags automatically |
|
|
143
|
+
| **Robot Groups** | High-availability with failover |
|
|
144
|
+
| **Rails Integration** | Auto-configures via Railtie, uses ActiveJob |
|
|
145
|
+
| **Telemetry** | Optional OpenTelemetry metrics |
|
|
1432
146
|
|
|
1433
|
-
|
|
1434
|
-
# Run all tests
|
|
1435
|
-
rake test
|
|
147
|
+
## Requirements
|
|
1436
148
|
|
|
1437
|
-
|
|
1438
|
-
|
|
1439
|
-
|
|
149
|
+
- Ruby 3.0+
|
|
150
|
+
- PostgreSQL with pgvector and pg_trgm extensions
|
|
151
|
+
- Ollama (default) or any local/private provider for embeddings and classifications
|
|
1440
152
|
|
|
1441
|
-
##
|
|
153
|
+
## Documentation
|
|
1442
154
|
|
|
1443
|
-
|
|
155
|
+
**[https://madbomber.github.io/htm](https://madbomber.github.io/htm)**
|
|
1444
156
|
|
|
1445
|
-
|
|
157
|
+
- [Installation & Setup](https://madbomber.github.io/htm/getting-started/)
|
|
158
|
+
- [MCP Server Guide](https://madbomber.github.io/htm/guides/mcp-server/)
|
|
159
|
+
- [Rails Integration](https://madbomber.github.io/htm/guides/rails/)
|
|
160
|
+
- [API Reference](https://madbomber.github.io/htm/api/)
|
|
1446
161
|
|
|
1447
|
-
|
|
1448
|
-
- **Configuration**: Dependency injection for LLM providers (embedding_generator, tag_extractor, token_counter, logger)
|
|
1449
|
-
- **WorkingMemory**: In-memory, token-limited active context for immediate LLM use
|
|
1450
|
-
- **LongTermMemory**: PostgreSQL-backed permanent storage with connection pooling
|
|
1451
|
-
- **EmbeddingService**: Processing and validation layer for vector embeddings (calls configured `embedding_generator`)
|
|
1452
|
-
- **TagService**: Processing and validation layer for hierarchical tags (calls configured `tag_extractor`)
|
|
1453
|
-
- **Database**: Schema setup, migrations, and management
|
|
1454
|
-
- **Background Jobs**: Async processing for embedding generation and tag extraction
|
|
162
|
+
## Why "Robots" Instead of "Agents"?
|
|
1455
163
|
|
|
1456
|
-
|
|
164
|
+
> "What's in a name? That which we call a rose
|
|
165
|
+
> By any other name would smell as sweet."
|
|
166
|
+
> — Shakespeare, *Romeo and Juliet*
|
|
1457
167
|
|
|
1458
|
-
|
|
1459
|
-
- `nodes`: Main memory storage with vector embeddings (pgvector), full-text search (tsvector), JSONB metadata
|
|
1460
|
-
- `tags`: Hierarchical tag ontology (format: `root:level1:level2:level3`)
|
|
1461
|
-
- `node_tags`: Join table implementing many-to-many relationship between nodes and tags
|
|
168
|
+
Shakespeare argues names are arbitrary. In software, we respectfully disagree—names shape expectations and understanding.
|
|
1462
169
|
|
|
1463
|
-
|
|
1464
|
-
- `content`: The memory content
|
|
1465
|
-
- `embedding`: Vector embedding for semantic search (up to 2000 dimensions)
|
|
1466
|
-
- `metadata`: JSONB column for arbitrary key-value data (filterable via `@>` containment operator)
|
|
1467
|
-
- `content_hash`: SHA-256 hash for deduplication
|
|
170
|
+
HTM uses "robots" rather than the fashionable "agents" for several reasons:
|
|
1468
171
|
|
|
1469
|
-
|
|
172
|
+
- **Semantic clarity**: "Agent" is overloaded—user agents, software agents, real estate agents, secret agents. "Robot" is specific to automated workers.
|
|
1470
173
|
|
|
1471
|
-
|
|
174
|
+
- **Honest about capabilities**: "Agent" implies autonomy and genuine decision-making. These systems follow instructions—they don't have agency. "Robot" acknowledges what they actually are: tools.
|
|
1472
175
|
|
|
1473
|
-
|
|
1474
|
-
2. **Service Layer** (`EmbeddingService`, `TagService`): Processes and validates LLM responses, handles padding/truncation, formats for storage
|
|
1475
|
-
3. **Consumer Layer** (`GenerateEmbeddingJob`, `GenerateTagsJob`, rake tasks): Uses services for async or synchronous processing
|
|
176
|
+
- **Avoiding the hype cycle**: "AI Agent" and "Agentic AI" became buzzwords in 2023-2024, often meaning nothing more than "LLM with a prompt." We prefer terminology that will age well.
|
|
1476
177
|
|
|
1477
|
-
|
|
178
|
+
- **Many "agent" frameworks are prompt wrappers**: Look under the hood of popular agent libraries and you'll often find a single prompt in a for-loop. Calling that an "agent" sets false expectations.
|
|
1478
179
|
|
|
1479
|
-
|
|
180
|
+
- **Rich heritage**: "Robot" comes from Karel Čapek's 1920 play *R.U.R.* (from Czech *robota*, meaning labor). Isaac Asimov gave us the Three Laws. There's decades of thoughtful writing about robot ethics and behavior. "Agent" has no comparable tradition.
|
|
1480
181
|
|
|
1481
|
-
-
|
|
1482
|
-
- [x] Phase 2: RAG retrieval (semantic search)
|
|
1483
|
-
- [x] Phase 3: Relationships & tags
|
|
1484
|
-
- [x] Phase 4: Working memory management
|
|
1485
|
-
- [x] Phase 5: Hive mind features
|
|
1486
|
-
- [x] Phase 6: Operations & observability
|
|
1487
|
-
- [x] Phase 7: Advanced features
|
|
1488
|
-
- [x] Phase 8: Production-ready gem
|
|
182
|
+
- **Personality**: Robots are endearing—R2-D2, Wall-E, Bender. They have cultural weight that "agent" lacks.
|
|
1489
183
|
|
|
184
|
+
Honest terminology leads to clearer thinking. These are robots: tireless workers with perfect memory, executing instructions on our behalf. That's exactly what HTM helps them do better.
|
|
1490
185
|
|
|
1491
186
|
## Contributing
|
|
1492
187
|
|