htm 0.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (155) hide show
  1. checksums.yaml +7 -0
  2. data/.architecture/decisions/adrs/001-use-postgresql-timescaledb-storage.md +227 -0
  3. data/.architecture/decisions/adrs/002-two-tier-memory-architecture.md +322 -0
  4. data/.architecture/decisions/adrs/003-ollama-default-embedding-provider.md +339 -0
  5. data/.architecture/decisions/adrs/004-multi-robot-shared-memory-hive-mind.md +374 -0
  6. data/.architecture/decisions/adrs/005-rag-based-retrieval-with-hybrid-search.md +443 -0
  7. data/.architecture/decisions/adrs/006-context-assembly-strategies.md +444 -0
  8. data/.architecture/decisions/adrs/007-working-memory-eviction-strategy.md +461 -0
  9. data/.architecture/decisions/adrs/008-robot-identification-system.md +550 -0
  10. data/.architecture/decisions/adrs/009-never-forget-explicit-deletion-only.md +570 -0
  11. data/.architecture/decisions/adrs/010-redis-working-memory-rejected.md +323 -0
  12. data/.architecture/decisions/adrs/011-database-side-embedding-generation-with-pgai.md +585 -0
  13. data/.architecture/decisions/adrs/012-llm-driven-ontology-topic-extraction.md +583 -0
  14. data/.architecture/decisions/adrs/013-activerecord-orm-and-many-to-many-tagging.md +299 -0
  15. data/.architecture/decisions/adrs/014-client-side-embedding-generation-workflow.md +569 -0
  16. data/.architecture/decisions/adrs/015-hierarchical-tag-ontology-and-llm-extraction.md +701 -0
  17. data/.architecture/decisions/adrs/016-async-embedding-and-tag-generation.md +694 -0
  18. data/.architecture/members.yml +144 -0
  19. data/.architecture/reviews/2025-10-29-llm-configuration-and-async-processing-review.md +1137 -0
  20. data/.architecture/reviews/initial-system-analysis.md +330 -0
  21. data/.envrc +32 -0
  22. data/.irbrc +145 -0
  23. data/CHANGELOG.md +150 -0
  24. data/COMMITS.md +196 -0
  25. data/LICENSE +21 -0
  26. data/README.md +1347 -0
  27. data/Rakefile +51 -0
  28. data/SETUP.md +268 -0
  29. data/config/database.yml +67 -0
  30. data/db/migrate/20250101000001_enable_extensions.rb +14 -0
  31. data/db/migrate/20250101000002_create_robots.rb +14 -0
  32. data/db/migrate/20250101000003_create_nodes.rb +42 -0
  33. data/db/migrate/20250101000005_create_tags.rb +38 -0
  34. data/db/migrate/20250101000007_add_node_vector_indexes.rb +30 -0
  35. data/db/schema.sql +473 -0
  36. data/db/seed_data/README.md +100 -0
  37. data/db/seed_data/presidents.md +136 -0
  38. data/db/seed_data/states.md +151 -0
  39. data/db/seeds.rb +208 -0
  40. data/dbdoc/README.md +173 -0
  41. data/dbdoc/public.node_stats.md +48 -0
  42. data/dbdoc/public.node_stats.svg +41 -0
  43. data/dbdoc/public.node_tags.md +40 -0
  44. data/dbdoc/public.node_tags.svg +112 -0
  45. data/dbdoc/public.nodes.md +54 -0
  46. data/dbdoc/public.nodes.svg +118 -0
  47. data/dbdoc/public.nodes_tags.md +39 -0
  48. data/dbdoc/public.nodes_tags.svg +112 -0
  49. data/dbdoc/public.ontology_structure.md +48 -0
  50. data/dbdoc/public.ontology_structure.svg +38 -0
  51. data/dbdoc/public.operations_log.md +42 -0
  52. data/dbdoc/public.operations_log.svg +130 -0
  53. data/dbdoc/public.relationships.md +39 -0
  54. data/dbdoc/public.relationships.svg +41 -0
  55. data/dbdoc/public.robot_activity.md +46 -0
  56. data/dbdoc/public.robot_activity.svg +35 -0
  57. data/dbdoc/public.robots.md +35 -0
  58. data/dbdoc/public.robots.svg +90 -0
  59. data/dbdoc/public.schema_migrations.md +29 -0
  60. data/dbdoc/public.schema_migrations.svg +26 -0
  61. data/dbdoc/public.tags.md +35 -0
  62. data/dbdoc/public.tags.svg +60 -0
  63. data/dbdoc/public.topic_relationships.md +45 -0
  64. data/dbdoc/public.topic_relationships.svg +32 -0
  65. data/dbdoc/schema.json +1437 -0
  66. data/dbdoc/schema.svg +154 -0
  67. data/docs/api/database.md +806 -0
  68. data/docs/api/embedding-service.md +532 -0
  69. data/docs/api/htm.md +797 -0
  70. data/docs/api/index.md +259 -0
  71. data/docs/api/long-term-memory.md +1096 -0
  72. data/docs/api/working-memory.md +665 -0
  73. data/docs/architecture/adrs/001-postgresql-timescaledb.md +314 -0
  74. data/docs/architecture/adrs/002-two-tier-memory.md +411 -0
  75. data/docs/architecture/adrs/003-ollama-embeddings.md +421 -0
  76. data/docs/architecture/adrs/004-hive-mind.md +437 -0
  77. data/docs/architecture/adrs/005-rag-retrieval.md +531 -0
  78. data/docs/architecture/adrs/006-context-assembly.md +496 -0
  79. data/docs/architecture/adrs/007-eviction-strategy.md +645 -0
  80. data/docs/architecture/adrs/008-robot-identification.md +625 -0
  81. data/docs/architecture/adrs/009-never-forget.md +648 -0
  82. data/docs/architecture/adrs/010-redis-working-memory-rejected.md +323 -0
  83. data/docs/architecture/adrs/011-pgai-integration.md +494 -0
  84. data/docs/architecture/adrs/index.md +215 -0
  85. data/docs/architecture/hive-mind.md +736 -0
  86. data/docs/architecture/index.md +351 -0
  87. data/docs/architecture/overview.md +538 -0
  88. data/docs/architecture/two-tier-memory.md +873 -0
  89. data/docs/assets/css/custom.css +83 -0
  90. data/docs/assets/images/htm-core-components.svg +63 -0
  91. data/docs/assets/images/htm-database-schema.svg +93 -0
  92. data/docs/assets/images/htm-hive-mind-architecture.svg +125 -0
  93. data/docs/assets/images/htm-importance-scoring-framework.svg +83 -0
  94. data/docs/assets/images/htm-layered-architecture.svg +71 -0
  95. data/docs/assets/images/htm-long-term-memory-architecture.svg +115 -0
  96. data/docs/assets/images/htm-working-memory-architecture.svg +120 -0
  97. data/docs/assets/images/htm.jpg +0 -0
  98. data/docs/assets/images/htm_demo.gif +0 -0
  99. data/docs/assets/js/mathjax.js +18 -0
  100. data/docs/assets/videos/htm_video.mp4 +0 -0
  101. data/docs/database_rake_tasks.md +322 -0
  102. data/docs/development/contributing.md +787 -0
  103. data/docs/development/index.md +336 -0
  104. data/docs/development/schema.md +596 -0
  105. data/docs/development/setup.md +719 -0
  106. data/docs/development/testing.md +819 -0
  107. data/docs/guides/adding-memories.md +824 -0
  108. data/docs/guides/context-assembly.md +1009 -0
  109. data/docs/guides/getting-started.md +577 -0
  110. data/docs/guides/index.md +118 -0
  111. data/docs/guides/long-term-memory.md +941 -0
  112. data/docs/guides/multi-robot.md +866 -0
  113. data/docs/guides/recalling-memories.md +927 -0
  114. data/docs/guides/search-strategies.md +953 -0
  115. data/docs/guides/working-memory.md +717 -0
  116. data/docs/index.md +214 -0
  117. data/docs/installation.md +477 -0
  118. data/docs/multi_framework_support.md +519 -0
  119. data/docs/quick-start.md +655 -0
  120. data/docs/setup_local_database.md +302 -0
  121. data/docs/using_rake_tasks_in_your_app.md +383 -0
  122. data/examples/basic_usage.rb +93 -0
  123. data/examples/cli_app/README.md +317 -0
  124. data/examples/cli_app/htm_cli.rb +270 -0
  125. data/examples/custom_llm_configuration.rb +183 -0
  126. data/examples/example_app/Rakefile +71 -0
  127. data/examples/example_app/app.rb +206 -0
  128. data/examples/sinatra_app/Gemfile +21 -0
  129. data/examples/sinatra_app/app.rb +335 -0
  130. data/lib/htm/active_record_config.rb +113 -0
  131. data/lib/htm/configuration.rb +342 -0
  132. data/lib/htm/database.rb +594 -0
  133. data/lib/htm/embedding_service.rb +115 -0
  134. data/lib/htm/errors.rb +34 -0
  135. data/lib/htm/job_adapter.rb +154 -0
  136. data/lib/htm/jobs/generate_embedding_job.rb +65 -0
  137. data/lib/htm/jobs/generate_tags_job.rb +82 -0
  138. data/lib/htm/long_term_memory.rb +965 -0
  139. data/lib/htm/models/node.rb +109 -0
  140. data/lib/htm/models/node_tag.rb +33 -0
  141. data/lib/htm/models/robot.rb +52 -0
  142. data/lib/htm/models/tag.rb +76 -0
  143. data/lib/htm/railtie.rb +76 -0
  144. data/lib/htm/sinatra.rb +157 -0
  145. data/lib/htm/tag_service.rb +135 -0
  146. data/lib/htm/tasks.rb +38 -0
  147. data/lib/htm/version.rb +5 -0
  148. data/lib/htm/working_memory.rb +182 -0
  149. data/lib/htm.rb +400 -0
  150. data/lib/tasks/db.rake +19 -0
  151. data/lib/tasks/htm.rake +147 -0
  152. data/lib/tasks/jobs.rake +312 -0
  153. data/mkdocs.yml +190 -0
  154. data/scripts/install_local_database.sh +309 -0
  155. metadata +341 -0
@@ -0,0 +1,115 @@
1
+ <svg viewBox="0 0 950 700" xmlns="http://www.w3.org/2000/svg" style="background: transparent;">
2
+ <defs>
3
+ <style>
4
+ .ltm-box { fill: rgba(33, 150, 243, 0.2); stroke: #2196F3; stroke-width: 3; }
5
+ .table-box { fill: rgba(76, 175, 80, 0.2); stroke: #4CAF50; stroke-width: 2; }
6
+ .index-box { fill: rgba(255, 152, 0, 0.2); stroke: #FF9800; stroke-width: 2; }
7
+ .hypertable { fill: rgba(156, 39, 176, 0.2); stroke: #9C27B0; stroke-width: 2; }
8
+ .text-header { fill: #E0E0E0; font-size: 18px; font-weight: bold; }
9
+ .text-label { fill: #E0E0E0; font-size: 14px; font-weight: bold; }
10
+ .text-small { fill: #B0B0B0; font-size: 12px; }
11
+ .text-tiny { fill: #A0A0A0; font-size: 10px; }
12
+ </style>
13
+ </defs>
14
+
15
+ <!-- Title -->
16
+ <text x="475" y="30" text-anchor="middle" class="text-header">Long-term Memory Architecture</text>
17
+ <text x="475" y="50" text-anchor="middle" class="text-small">PostgreSQL</text>
18
+
19
+ <!-- Main Container -->
20
+ <rect x="50" y="75" width="850" height="590" class="ltm-box" rx="5"/>
21
+
22
+ <!-- Tables Section -->
23
+ <text x="70" y="105" class="text-label">Database Tables</text>
24
+
25
+ <!-- nodes table -->
26
+ <rect x="70" y="120" width="250" height="180" class="table-box" rx="3"/>
27
+ <text x="195" y="140" text-anchor="middle" class="text-label" fill="#4CAF50">nodes</text>
28
+ <text x="80" y="165" class="text-small">Memories + Embeddings</text>
29
+ <text x="80" y="185" class="text-tiny">• id, key, value</text>
30
+ <text x="80" y="200" class="text-tiny">• type, category, importance</text>
31
+ <text x="80" y="215" class="text-tiny">• token_count, robot_id</text>
32
+ <text x="80" y="230" class="text-tiny">• embedding vector(1536)</text>
33
+ <text x="80" y="245" class="text-tiny">• created_at, last_accessed</text>
34
+ <text x="80" y="260" class="text-tiny">• in_working_memory flag</text>
35
+ <text x="80" y="280" class="text-small" fill="#66BB6A">Primary storage</text>
36
+
37
+ <!-- relationships table -->
38
+ <rect x="340" y="120" width="250" height="100" class="table-box" rx="3"/>
39
+ <text x="465" y="140" text-anchor="middle" class="text-label" fill="#4CAF50">relationships</text>
40
+ <text x="350" y="165" class="text-small">Knowledge Graph</text>
41
+ <text x="350" y="185" class="text-tiny">• from_node_id → to_node_id</text>
42
+ <text x="350" y="200" class="text-tiny">• relationship_type, strength</text>
43
+
44
+ <!-- tags table -->
45
+ <rect x="610" y="120" width="250" height="100" class="table-box" rx="3"/>
46
+ <text x="735" y="140" text-anchor="middle" class="text-label" fill="#4CAF50">tags</text>
47
+ <text x="620" y="165" class="text-small">Categorization</text>
48
+ <text x="620" y="185" class="text-tiny">• node_id, tag</text>
49
+ <text x="620" y="200" class="text-tiny">• Flexible tagging system</text>
50
+
51
+ <!-- robots table -->
52
+ <rect x="340" y="240" width="250" height="100" class="table-box" rx="3"/>
53
+ <text x="465" y="260" text-anchor="middle" class="text-label" fill="#4CAF50">robots</text>
54
+ <text x="350" y="285" class="text-small">Identity Registry</text>
55
+ <text x="350" y="305" class="text-tiny">• id, name, created_at</text>
56
+ <text x="350" y="320" class="text-tiny">• last_active, metadata</text>
57
+
58
+ <!-- operations_log table (hypertable) -->
59
+ <rect x="610" y="240" width="250" height="100" class="hypertable" rx="3"/>
60
+ <text x="735" y="260" text-anchor="middle" class="text-label" fill="#9C27B0">operations_log</text>
61
+ <text x="620" y="285" class="text-small">Audit Trail (Hypertable)</text>
62
+ <text x="620" y="305" class="text-tiny">• timestamp, operation</text>
63
+ <text x="620" y="320" class="text-tiny">• node_id, robot_id, details</text>
64
+
65
+ <!-- Indexes Section -->
66
+ <text x="70" y="375" class="text-label">Indexes &amp; Optimization</text>
67
+
68
+ <!-- HNSW Vector Index -->
69
+ <rect x="70" y="390" width="250" height="120" class="index-box" rx="3"/>
70
+ <text x="195" y="410" text-anchor="middle" class="text-label" fill="#FF9800">HNSW Vector Index</text>
71
+ <text x="80" y="435" class="text-small">Semantic Similarity</text>
72
+ <text x="80" y="455" class="text-tiny">• idx_nodes_embedding</text>
73
+ <text x="80" y="470" class="text-tiny">• vector_cosine_ops</text>
74
+ <text x="80" y="485" class="text-tiny">• m=16, ef_construction=64</text>
75
+ <text x="80" y="500" class="text-tiny">⚡ Fast nearest-neighbor search</text>
76
+
77
+ <!-- GIN Full-text Index -->
78
+ <rect x="340" y="390" width="250" height="120" class="index-box" rx="3"/>
79
+ <text x="465" y="410" text-anchor="middle" class="text-label" fill="#FF9800">GIN Full-text Index</text>
80
+ <text x="350" y="435" class="text-small">Keyword Search</text>
81
+ <text x="350" y="455" class="text-tiny">• idx_nodes_fulltext</text>
82
+ <text x="350" y="470" class="text-tiny">• to_tsvector('english')</text>
83
+ <text x="350" y="485" class="text-tiny">• Supports ts_rank scoring</text>
84
+ <text x="350" y="500" class="text-tiny">⚡ Fast keyword matching</text>
85
+
86
+ <!-- PostgreSQL Chunks -->
87
+ <rect x="610" y="390" width="250" height="120" class="hypertable" rx="3"/>
88
+ <text x="735" y="410" text-anchor="middle" class="text-label" fill="#9C27B0">PostgreSQL Chunks</text>
89
+ <text x="620" y="435" class="text-small">Time-series Partitioning</text>
90
+ <text x="620" y="455" class="text-tiny">• Auto-partitioning by time</text>
91
+ <text x="620" y="470" class="text-tiny">• Compression policies</text>
92
+ <text x="620" y="485" class="text-tiny">• Retention policies</text>
93
+ <text x="620" y="500" class="text-tiny">⚡ Fast time-range queries</text>
94
+
95
+ <!-- Features Section -->
96
+ <text x="70" y="545" class="text-label">Key Features</text>
97
+
98
+ <rect x="70" y="560" width="250" height="90" fill="rgba(33, 150, 243, 0.15)" stroke="#2196F3" stroke-width="1.5" rx="3"/>
99
+ <text x="80" y="585" class="text-small">✓ Permanent storage</text>
100
+ <text x="80" y="605" class="text-small">✓ ACID compliant</text>
101
+ <text x="80" y="625" class="text-small">✓ Never forgets</text>
102
+ <text x="80" y="645" class="text-small">✓ Unlimited capacity</text>
103
+
104
+ <rect x="340" y="560" width="250" height="90" fill="rgba(33, 150, 243, 0.15)" stroke="#2196F3" stroke-width="1.5" rx="3"/>
105
+ <text x="350" y="585" class="text-small">✓ Vector similarity search</text>
106
+ <text x="350" y="605" class="text-small">✓ Full-text search</text>
107
+ <text x="350" y="625" class="text-small">✓ Hybrid retrieval</text>
108
+ <text x="350" y="645" class="text-small">✓ RAG-optimized</text>
109
+
110
+ <rect x="610" y="560" width="250" height="90" fill="rgba(33, 150, 243, 0.15)" stroke="#2196F2" stroke-width="1.5" rx="3"/>
111
+ <text x="620" y="585" class="text-small">✓ Multi-robot support</text>
112
+ <text x="620" y="605" class="text-small">✓ Relationship graphs</text>
113
+ <text x="620" y="625" class="text-small">✓ Tag-based filtering</text>
114
+ <text x="620" y="645" class="text-small">✓ Complete audit trail</text>
115
+ </svg>
@@ -0,0 +1,120 @@
1
+ <svg viewBox="0 0 900 650" xmlns="http://www.w3.org/2000/svg" style="background: transparent;">
2
+ <defs>
3
+ <style>
4
+ .wm-box { fill: rgba(33, 150, 243, 0.2); stroke: #2196F3; stroke-width: 3; }
5
+ .ltm-box { fill: rgba(76, 175, 80, 0.2); stroke: #4CAF50; stroke-width: 3; }
6
+ .memory-item { fill: rgba(255, 152, 0, 0.3); stroke: #FF9800; stroke-width: 1.5; }
7
+ .evicted-item { fill: rgba(244, 67, 54, 0.3); stroke: #F44336; stroke-width: 1.5; }
8
+ .text-header { fill: #E0E0E0; font-size: 18px; font-weight: bold; }
9
+ .text-label { fill: #E0E0E0; font-size: 13px; }
10
+ .text-small { fill: #B0B0B0; font-size: 11px; }
11
+ .arrow { stroke: #4A9EFF; stroke-width: 2; fill: none; marker-end: url(#arrowhead); }
12
+ .arrow-evict { stroke: #F44336; stroke-width: 2; fill: none; marker-end: url(#arrowhead-red); stroke-dasharray: 5,3; }
13
+ .arrow-recall { stroke: #4CAF50; stroke-width: 2; fill: none; marker-end: url(#arrowhead-green); stroke-dasharray: 5,3; }
14
+ </style>
15
+ <marker id="arrowhead" markerWidth="10" markerHeight="10" refX="9" refY="3" orient="auto">
16
+ <polygon points="0 0, 10 3, 0 6" fill="#4A9EFF" />
17
+ </marker>
18
+ <marker id="arrowhead-red" markerWidth="10" markerHeight="10" refX="9" refY="3" orient="auto">
19
+ <polygon points="0 0, 10 3, 0 6" fill="#F44336" />
20
+ </marker>
21
+ <marker id="arrowhead-green" markerWidth="10" markerHeight="10" refX="9" refY="3" orient="auto">
22
+ <polygon points="0 0, 10 3, 0 6" fill="#4CAF50" />
23
+ </marker>
24
+ </defs>
25
+
26
+ <!-- Title -->
27
+ <text x="450" y="30" text-anchor="middle" class="text-header">Working Memory Architecture</text>
28
+
29
+ <!-- Working Memory Box -->
30
+ <rect x="50" y="70" width="800" height="220" class="wm-box" rx="5"/>
31
+ <text x="450" y="100" text-anchor="middle" class="text-header">Working Memory</text>
32
+ <text x="450" y="120" text-anchor="middle" class="text-small">(In-memory, Token-limited)</text>
33
+
34
+ <!-- Capacity bar -->
35
+ <rect x="70" y="140" width="760" height="25" fill="rgba(255, 255, 255, 0.1)" stroke="#666" stroke-width="1" rx="3"/>
36
+ <rect x="70" y="140" width="608" height="25" fill="rgba(76, 175, 80, 0.4)" rx="3"/>
37
+ <text x="450" y="157" text-anchor="middle" class="text-small">102,400 / 128,000 tokens (80% full)</text>
38
+
39
+ <!-- Memory items in working memory -->
40
+ <rect x="70" y="180" width="150" height="40" class="memory-item" rx="3"/>
41
+ <text x="80" y="197" class="text-label">Critical Decision</text>
42
+ <text x="210" y="197" text-anchor="end" class="text-small">imp: 10.0</text>
43
+ <text x="210" y="212" text-anchor="end" class="text-small">2000 tok</text>
44
+
45
+ <rect x="235" y="180" width="150" height="40" class="memory-item" rx="3"/>
46
+ <text x="245" y="197" class="text-label">User Preference</text>
47
+ <text x="375" y="197" text-anchor="end" class="text-small">imp: 8.0</text>
48
+ <text x="375" y="212" text-anchor="end" class="text-small">500 tok</text>
49
+
50
+ <rect x="400" y="180" width="150" height="40" class="memory-item" rx="3"/>
51
+ <text x="410" y="197" class="text-label">Recent Context</text>
52
+ <text x="540" y="197" text-anchor="end" class="text-small">imp: 6.0</text>
53
+ <text x="540" y="212" text-anchor="end" class="text-small">1500 tok</text>
54
+
55
+ <rect x="565" y="180" width="150" height="40" class="evicted-item" rx="3"/>
56
+ <text x="575" y="197" class="text-label">Old Note</text>
57
+ <text x="705" y="197" text-anchor="end" class="text-small">imp: 2.0</text>
58
+ <text x="705" y="212" text-anchor="end" class="text-small">800 tok</text>
59
+ <text x="640" y="235" text-anchor="middle" class="text-small" fill="#F44336">← Being evicted</text>
60
+
61
+ <!-- Features list -->
62
+ <text x="70" y="260" class="text-small">✓ Fast access</text>
63
+ <text x="250" y="260" class="text-small">✓ Token-based capacity</text>
64
+ <text x="450" y="260" class="text-small">✓ Automatic eviction</text>
65
+ <text x="650" y="260" class="text-small">✓ Importance-based retention</text>
66
+
67
+ <!-- Arrow down (Eviction) -->
68
+ <path d="M 640 290 L 640 350" class="arrow-evict"/>
69
+ <text x="660" y="325" class="text-small" fill="#F44336">Eviction</text>
70
+ <text x="660" y="340" class="text-small" fill="#F44336">(when full)</text>
71
+
72
+ <!-- Arrow up (Recall) -->
73
+ <path d="M 260 350 L 260 290" class="arrow-recall"/>
74
+ <text x="280" y="315" class="text-small" fill="#4CAF50">Recall</text>
75
+ <text x="280" y="330" class="text-small" fill="#4CAF50">(RAG retrieval)</text>
76
+
77
+ <!-- Long-term Memory Box -->
78
+ <rect x="50" y="370" width="800" height="240" class="ltm-box" rx="5"/>
79
+ <text x="450" y="400" text-anchor="middle" class="text-header">Long-term Memory</text>
80
+ <text x="450" y="420" text-anchor="middle" class="text-small">(PostgreSQL, Unlimited)</text>
81
+
82
+ <!-- Database tables visualization -->
83
+ <rect x="70" y="440" width="180" height="150" fill="rgba(76, 175, 80, 0.1)" stroke="#4CAF50" stroke-width="1" rx="3"/>
84
+ <text x="160" y="460" text-anchor="middle" class="text-label">nodes</text>
85
+ <text x="80" y="480" class="text-small">All memories</text>
86
+ <text x="80" y="495" class="text-small">Vector embeddings</text>
87
+ <text x="80" y="510" class="text-small">Full-text search</text>
88
+ <text x="80" y="525" class="text-small">Timestamps</text>
89
+ <text x="80" y="540" class="text-small">Importance scores</text>
90
+ <text x="80" y="555" class="text-small">Robot attribution</text>
91
+ <text x="80" y="570" class="text-small">in_working_memory flag</text>
92
+
93
+ <rect x="265" y="440" width="180" height="70" fill="rgba(76, 175, 80, 0.1)" stroke="#4CAF50" stroke-width="1" rx="3"/>
94
+ <text x="355" y="460" text-anchor="middle" class="text-label">relationships</text>
95
+ <text x="275" y="480" class="text-small">Knowledge graph</text>
96
+ <text x="275" y="495" class="text-small">Node connections</text>
97
+
98
+ <rect x="460" y="440" width="180" height="70" fill="rgba(76, 175, 80, 0.1)" stroke="#4CAF50" stroke-width="1" rx="3"/>
99
+ <text x="550" y="460" text-anchor="middle" class="text-label">tags</text>
100
+ <text x="470" y="480" class="text-small">Categorization</text>
101
+ <text x="470" y="495" class="text-small">Flexible tagging</text>
102
+
103
+ <rect x="655" y="440" width="180" height="70" fill="rgba(76, 175, 80, 0.1)" stroke="#4CAF50" stroke-width="1" rx="3"/>
104
+ <text x="745" y="460" text-anchor="middle" class="text-label">operations_log</text>
105
+ <text x="665" y="480" class="text-small">Audit trail</text>
106
+ <text x="665" y="495" class="text-small">PostgreSQL table</text>
107
+
108
+ <!-- Evicted item in long-term memory -->
109
+ <rect x="265" y="520" width="380" height="70" class="evicted-item" rx="3"/>
110
+ <text x="275" y="540" class="text-label">Evicted: Old Note</text>
111
+ <text x="635" y="540" text-anchor="end" class="text-small">in_working_memory: false</text>
112
+ <text x="275" y="560" class="text-small">Still stored, can be recalled when needed</text>
113
+ <text x="275" y="575" class="text-small">Full history preserved • Never forgotten</text>
114
+
115
+ <!-- Features list -->
116
+ <text x="70" y="335" class="text-small" fill="#4CAF50">✓ Durable storage</text>
117
+ <text x="270" y="335" class="text-small" fill="#4CAF50">✓ Full history</text>
118
+ <text x="450" y="335" class="text-small" fill="#4CAF50">✓ RAG retrieval</text>
119
+ <text x="610" y="335" class="text-small" fill="#4CAF50">✓ Never forgets</text>
120
+ </svg>
Binary file
Binary file
@@ -0,0 +1,18 @@
1
+ /* MathJax Configuration for HTM Documentation */
2
+
3
+ window.MathJax = {
4
+ tex: {
5
+ inlineMath: [["\\(", "\\)"]],
6
+ displayMath: [["\\[", "\\]"]],
7
+ processEscapes: true,
8
+ processEnvironments: true
9
+ },
10
+ options: {
11
+ ignoreHtmlClass: ".*|",
12
+ processHtmlClass: "arithmatex"
13
+ }
14
+ };
15
+
16
+ document$.subscribe(() => {
17
+ MathJax.typesetPromise()
18
+ })
Binary file
@@ -0,0 +1,322 @@
1
+ # HTM Database Rake Tasks Reference
2
+
3
+ Complete reference for HTM database management tasks.
4
+
5
+ ## Quick Start
6
+
7
+ ```bash
8
+ # First time setup
9
+ cd /path/to/htm
10
+ direnv allow # Load environment variables from .envrc
11
+ rake htm:db:setup # Create schema and run migrations
12
+ ```
13
+
14
+ ## Available Tasks
15
+
16
+ ### Setup and Schema
17
+
18
+ #### `rake htm:db:setup`
19
+ Sets up the HTM database schema and runs all migrations.
20
+
21
+ **What it does:**
22
+ - Verifies required extensions (timescaledb, pgvector, pg_trgm)
23
+ - Creates all HTM tables (nodes, tags, robots, operations_log)
24
+ - Runs all pending migrations
25
+ - Sets up hypertables for time-series optimization
26
+
27
+ **When to use:** First-time setup or after dropping the database
28
+
29
+ ```bash
30
+ $ rake htm:db:setup
31
+ ✓ TimescaleDB version: 2.22.1
32
+ ✓ pgvector version: 0.8.1
33
+ Creating HTM schema...
34
+ ✓ Schema created
35
+ Running migration: 001_support_variable_dimensions
36
+ ✓ Migration 001_support_variable_dimensions applied
37
+ Running migration: 002_ontology_topic_extraction
38
+ ✓ Migration 002_ontology_topic_extraction applied
39
+ ✓ Created hypertable for operations_log
40
+ ✓ HTM database schema created successfully
41
+ ```
42
+
43
+ ---
44
+
45
+ ### Migrations
46
+
47
+ #### `rake htm:db:migrate`
48
+ Runs pending database migrations only.
49
+
50
+ **What it does:**
51
+ - Checks which migrations have been applied
52
+ - Runs any new migrations in `sql/migrations/`
53
+ - Updates the `schema_migrations` table
54
+
55
+ **When to use:** After pulling new code with migrations
56
+
57
+ ```bash
58
+ $ rake htm:db:migrate
59
+ Running migration: 002_ontology_topic_extraction
60
+ ✓ Migration 002_ontology_topic_extraction applied
61
+ ✓ Database migrations completed
62
+ ```
63
+
64
+ #### `rake htm:db:status`
65
+ Shows which migrations have been applied and which are pending.
66
+
67
+ **Example output:**
68
+ ```bash
69
+ $ rake htm:db:status
70
+
71
+ Migration Status
72
+ ================================================================================
73
+ ✓ 001_support_variable_dimensions (applied: 2025-10-26 04:27:15.428951+00)
74
+ ✓ 002_ontology_topic_extraction (applied: 2025-10-27 03:44:15.012345+00)
75
+
76
+ Summary: 2 applied, 0 pending
77
+ ================================================================================
78
+ ```
79
+
80
+ ---
81
+
82
+ ### Information
83
+
84
+ #### `rake htm:db:info`
85
+ Shows comprehensive database information.
86
+
87
+ **Example output:**
88
+ ```bash
89
+ $ rake htm:db:info
90
+
91
+ HTM Database Information
92
+ ================================================================================
93
+
94
+ Connection:
95
+ Host: cw7rxj91bm.srbbwwxn56.tsdb.cloud.timescale.com
96
+ Port: 37807
97
+ Database: tsdb
98
+ User: tsdbadmin
99
+
100
+ PostgreSQL Version:
101
+ PostgreSQL 17.6 (Ubuntu 17.6-2.pgdg22.04+1) on aarch64-unknown-linux-gnu
102
+
103
+ Extensions:
104
+ ai (0.11.2)
105
+ pg_stat_statements (1.11)
106
+ pg_trgm (1.6)
107
+ plpgsql (1.0)
108
+ timescaledb (2.22.1)
109
+ timescaledb_toolkit (1.21.0)
110
+ vector (0.8.1)
111
+ vectorscale (0.8.0)
112
+
113
+ HTM Tables:
114
+ nodes: 42 rows
115
+ tags: 156 rows
116
+ robots: 3 rows
117
+ operations_log: 289 rows
118
+ schema_migrations: 2 rows
119
+
120
+ Database Size: 14 MB
121
+ ================================================================================
122
+ ```
123
+
124
+ #### `rake htm:db:test`
125
+ Tests database connection by running `test_connection.rb`.
126
+
127
+ **Example output:**
128
+ ```bash
129
+ $ rake htm:db:test
130
+ Connecting to TimescaleDB...
131
+ ✓ Connected successfully!
132
+ ✓ TimescaleDB Extension: Version 2.22.1
133
+ ✓ pgvector Extension: Version 0.8.1
134
+ ✓ pg_trgm Extension: Version 1.6
135
+ ```
136
+
137
+ ---
138
+
139
+ ### Utilities
140
+
141
+ #### `rake htm:db:console`
142
+ Opens an interactive PostgreSQL console (psql).
143
+
144
+ **What it does:**
145
+ - Launches `psql` connected to your HTM database
146
+ - Uses connection parameters from `HTM_DBURL` or `.envrc`
147
+ - Allows you to run SQL queries directly
148
+
149
+ **Example:**
150
+ ```bash
151
+ $ rake htm:db:console
152
+ psql (17.6)
153
+ SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, compression: off, ALPN: none)
154
+ Type "help" for help.
155
+
156
+ tsdb=> SELECT COUNT(*) FROM nodes;
157
+ count
158
+ -------
159
+ 42
160
+ (1 row)
161
+
162
+ tsdb=> \d nodes
163
+ tsdb=> \q
164
+ ```
165
+
166
+ #### `rake htm:db:seed`
167
+ Seeds the database with sample data.
168
+
169
+ **What it does:**
170
+ - Creates a sample robot
171
+ - Adds 3 sample nodes with different topics
172
+ - Useful for testing or demos
173
+
174
+ **Example:**
175
+ ```bash
176
+ $ rake htm:db:seed
177
+ Seeding database with sample data...
178
+ Creating sample nodes...
179
+ ✓ Database seeded with 3 sample nodes
180
+ ```
181
+
182
+ ---
183
+
184
+ ### Destructive Operations
185
+
186
+ ⚠️ **WARNING**: These tasks delete data and cannot be undone!
187
+
188
+ #### `rake htm:db:drop`
189
+ Drops all HTM tables, functions, triggers, and views.
190
+
191
+ **What it does:**
192
+ - Drops tables: nodes, tags, robots, operations_log, schema_migrations
193
+ - Drops ontology functions and triggers
194
+ - Drops views: ontology_structure, topic_relationships
195
+
196
+ **Safety:** Prompts for confirmation ("yes" must be typed)
197
+
198
+ ```bash
199
+ $ rake htm:db:drop
200
+ Are you sure you want to drop all tables? This cannot be undone! (yes/no): yes
201
+ Dropping HTM tables...
202
+ ✓ Dropped nodes
203
+ ✓ Dropped tags
204
+ ✓ Dropped robots
205
+ ✓ Dropped operations_log
206
+ ✓ Dropped schema_migrations
207
+ ✓ Dropped ontology functions and triggers
208
+ ✓ Dropped ontology views
209
+ ✓ All HTM tables dropped
210
+ ```
211
+
212
+ #### `rake htm:db:reset`
213
+ Drops and recreates the entire database (equivalent to `drop` + `setup`).
214
+
215
+ **When to use:** Development only, to start fresh
216
+
217
+ ```bash
218
+ $ rake htm:db:reset
219
+ # Runs drop (with confirmation) then setup
220
+ ```
221
+
222
+ ---
223
+
224
+ ## Environment Variables
225
+
226
+ All tasks require database configuration via environment variables. Use one of these methods:
227
+
228
+ ### Method 1: direnv (Recommended)
229
+ ```bash
230
+ # One-time setup
231
+ cd /path/to/htm
232
+ direnv allow
233
+
234
+ # Variables are automatically loaded from .envrc
235
+ rake htm:db:info
236
+ ```
237
+
238
+ ### Method 2: Manual Export
239
+ ```bash
240
+ export HTM_DBURL="postgresql://user:password@host:port/dbname?sslmode=require"
241
+ rake htm:db:info
242
+ ```
243
+
244
+ ### Method 3: Source Tiger Credentials
245
+ ```bash
246
+ source ~/.bashrc__tiger # If using TimescaleDB Cloud
247
+ rake htm:db:info
248
+ ```
249
+
250
+ ---
251
+
252
+ ## Common Workflows
253
+
254
+ ### Initial Project Setup
255
+ ```bash
256
+ cd /path/to/htm
257
+ direnv allow
258
+ bundle install
259
+ rake htm:db:setup
260
+ rake htm:db:seed # Optional: add sample data
261
+ rake htm:db:info # Verify setup
262
+ ```
263
+
264
+ ### After Pulling New Code
265
+ ```bash
266
+ git pull
267
+ rake htm:db:status # Check for new migrations
268
+ rake htm:db:migrate # Run pending migrations
269
+ ```
270
+
271
+ ### Development Reset
272
+ ```bash
273
+ rake htm:db:reset # Drop and recreate (type 'yes' to confirm)
274
+ rake htm:db:seed # Re-add sample data
275
+ ```
276
+
277
+ ### Debugging
278
+ ```bash
279
+ rake htm:db:info # Check database state
280
+ rake htm:db:status # Check migration status
281
+ rake htm:db:console # Open psql for SQL queries
282
+ ```
283
+
284
+ ### Production Deployment
285
+ ```bash
286
+ # NEVER use reset or drop in production!
287
+ rake htm:db:migrate # Run new migrations only
288
+ ```
289
+
290
+ ---
291
+
292
+ ## Troubleshooting
293
+
294
+ ### "Database configuration not found"
295
+ - Run `direnv allow` in the project directory
296
+ - Or manually export `HTM_DBURL`
297
+ - Verify: `echo $HTM_DBURL`
298
+
299
+ ### "Connection refused"
300
+ - Check database is running
301
+ - Verify host/port in `HTM_DBURL`
302
+ - Test: `rake htm:db:test`
303
+
304
+ ### "Extension not found"
305
+ - Ensure TimescaleDB Cloud instance has required extensions
306
+ - Check with: `rake htm:db:info`
307
+ - Extensions needed: timescaledb, pgvector, pg_trgm
308
+
309
+ ### Migrations not running
310
+ - Check migration files exist: `ls -la sql/migrations/`
311
+ - Verify migrations table: `rake htm:db:console` then `SELECT * FROM schema_migrations;`
312
+ - Force re-run: `rake htm:db:drop` then `rake htm:db:setup`
313
+
314
+ ---
315
+
316
+ ## Legacy Tasks (Deprecated)
317
+
318
+ These tasks still work but will be removed in a future version:
319
+
320
+ - `rake db_setup` → Use `rake htm:db:setup`
321
+ - `rake db_test` → Use `rake htm:db:test`
322
+