htm 0.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.architecture/decisions/adrs/001-use-postgresql-timescaledb-storage.md +227 -0
- data/.architecture/decisions/adrs/002-two-tier-memory-architecture.md +322 -0
- data/.architecture/decisions/adrs/003-ollama-default-embedding-provider.md +339 -0
- data/.architecture/decisions/adrs/004-multi-robot-shared-memory-hive-mind.md +374 -0
- data/.architecture/decisions/adrs/005-rag-based-retrieval-with-hybrid-search.md +443 -0
- data/.architecture/decisions/adrs/006-context-assembly-strategies.md +444 -0
- data/.architecture/decisions/adrs/007-working-memory-eviction-strategy.md +461 -0
- data/.architecture/decisions/adrs/008-robot-identification-system.md +550 -0
- data/.architecture/decisions/adrs/009-never-forget-explicit-deletion-only.md +570 -0
- data/.architecture/decisions/adrs/010-redis-working-memory-rejected.md +323 -0
- data/.architecture/decisions/adrs/011-database-side-embedding-generation-with-pgai.md +585 -0
- data/.architecture/decisions/adrs/012-llm-driven-ontology-topic-extraction.md +583 -0
- data/.architecture/decisions/adrs/013-activerecord-orm-and-many-to-many-tagging.md +299 -0
- data/.architecture/decisions/adrs/014-client-side-embedding-generation-workflow.md +569 -0
- data/.architecture/decisions/adrs/015-hierarchical-tag-ontology-and-llm-extraction.md +701 -0
- data/.architecture/decisions/adrs/016-async-embedding-and-tag-generation.md +694 -0
- data/.architecture/members.yml +144 -0
- data/.architecture/reviews/2025-10-29-llm-configuration-and-async-processing-review.md +1137 -0
- data/.architecture/reviews/initial-system-analysis.md +330 -0
- data/.envrc +32 -0
- data/.irbrc +145 -0
- data/CHANGELOG.md +150 -0
- data/COMMITS.md +196 -0
- data/LICENSE +21 -0
- data/README.md +1347 -0
- data/Rakefile +51 -0
- data/SETUP.md +268 -0
- data/config/database.yml +67 -0
- data/db/migrate/20250101000001_enable_extensions.rb +14 -0
- data/db/migrate/20250101000002_create_robots.rb +14 -0
- data/db/migrate/20250101000003_create_nodes.rb +42 -0
- data/db/migrate/20250101000005_create_tags.rb +38 -0
- data/db/migrate/20250101000007_add_node_vector_indexes.rb +30 -0
- data/db/schema.sql +473 -0
- data/db/seed_data/README.md +100 -0
- data/db/seed_data/presidents.md +136 -0
- data/db/seed_data/states.md +151 -0
- data/db/seeds.rb +208 -0
- data/dbdoc/README.md +173 -0
- data/dbdoc/public.node_stats.md +48 -0
- data/dbdoc/public.node_stats.svg +41 -0
- data/dbdoc/public.node_tags.md +40 -0
- data/dbdoc/public.node_tags.svg +112 -0
- data/dbdoc/public.nodes.md +54 -0
- data/dbdoc/public.nodes.svg +118 -0
- data/dbdoc/public.nodes_tags.md +39 -0
- data/dbdoc/public.nodes_tags.svg +112 -0
- data/dbdoc/public.ontology_structure.md +48 -0
- data/dbdoc/public.ontology_structure.svg +38 -0
- data/dbdoc/public.operations_log.md +42 -0
- data/dbdoc/public.operations_log.svg +130 -0
- data/dbdoc/public.relationships.md +39 -0
- data/dbdoc/public.relationships.svg +41 -0
- data/dbdoc/public.robot_activity.md +46 -0
- data/dbdoc/public.robot_activity.svg +35 -0
- data/dbdoc/public.robots.md +35 -0
- data/dbdoc/public.robots.svg +90 -0
- data/dbdoc/public.schema_migrations.md +29 -0
- data/dbdoc/public.schema_migrations.svg +26 -0
- data/dbdoc/public.tags.md +35 -0
- data/dbdoc/public.tags.svg +60 -0
- data/dbdoc/public.topic_relationships.md +45 -0
- data/dbdoc/public.topic_relationships.svg +32 -0
- data/dbdoc/schema.json +1437 -0
- data/dbdoc/schema.svg +154 -0
- data/docs/api/database.md +806 -0
- data/docs/api/embedding-service.md +532 -0
- data/docs/api/htm.md +797 -0
- data/docs/api/index.md +259 -0
- data/docs/api/long-term-memory.md +1096 -0
- data/docs/api/working-memory.md +665 -0
- data/docs/architecture/adrs/001-postgresql-timescaledb.md +314 -0
- data/docs/architecture/adrs/002-two-tier-memory.md +411 -0
- data/docs/architecture/adrs/003-ollama-embeddings.md +421 -0
- data/docs/architecture/adrs/004-hive-mind.md +437 -0
- data/docs/architecture/adrs/005-rag-retrieval.md +531 -0
- data/docs/architecture/adrs/006-context-assembly.md +496 -0
- data/docs/architecture/adrs/007-eviction-strategy.md +645 -0
- data/docs/architecture/adrs/008-robot-identification.md +625 -0
- data/docs/architecture/adrs/009-never-forget.md +648 -0
- data/docs/architecture/adrs/010-redis-working-memory-rejected.md +323 -0
- data/docs/architecture/adrs/011-pgai-integration.md +494 -0
- data/docs/architecture/adrs/index.md +215 -0
- data/docs/architecture/hive-mind.md +736 -0
- data/docs/architecture/index.md +351 -0
- data/docs/architecture/overview.md +538 -0
- data/docs/architecture/two-tier-memory.md +873 -0
- data/docs/assets/css/custom.css +83 -0
- data/docs/assets/images/htm-core-components.svg +63 -0
- data/docs/assets/images/htm-database-schema.svg +93 -0
- data/docs/assets/images/htm-hive-mind-architecture.svg +125 -0
- data/docs/assets/images/htm-importance-scoring-framework.svg +83 -0
- data/docs/assets/images/htm-layered-architecture.svg +71 -0
- data/docs/assets/images/htm-long-term-memory-architecture.svg +115 -0
- data/docs/assets/images/htm-working-memory-architecture.svg +120 -0
- data/docs/assets/images/htm.jpg +0 -0
- data/docs/assets/images/htm_demo.gif +0 -0
- data/docs/assets/js/mathjax.js +18 -0
- data/docs/assets/videos/htm_video.mp4 +0 -0
- data/docs/database_rake_tasks.md +322 -0
- data/docs/development/contributing.md +787 -0
- data/docs/development/index.md +336 -0
- data/docs/development/schema.md +596 -0
- data/docs/development/setup.md +719 -0
- data/docs/development/testing.md +819 -0
- data/docs/guides/adding-memories.md +824 -0
- data/docs/guides/context-assembly.md +1009 -0
- data/docs/guides/getting-started.md +577 -0
- data/docs/guides/index.md +118 -0
- data/docs/guides/long-term-memory.md +941 -0
- data/docs/guides/multi-robot.md +866 -0
- data/docs/guides/recalling-memories.md +927 -0
- data/docs/guides/search-strategies.md +953 -0
- data/docs/guides/working-memory.md +717 -0
- data/docs/index.md +214 -0
- data/docs/installation.md +477 -0
- data/docs/multi_framework_support.md +519 -0
- data/docs/quick-start.md +655 -0
- data/docs/setup_local_database.md +302 -0
- data/docs/using_rake_tasks_in_your_app.md +383 -0
- data/examples/basic_usage.rb +93 -0
- data/examples/cli_app/README.md +317 -0
- data/examples/cli_app/htm_cli.rb +270 -0
- data/examples/custom_llm_configuration.rb +183 -0
- data/examples/example_app/Rakefile +71 -0
- data/examples/example_app/app.rb +206 -0
- data/examples/sinatra_app/Gemfile +21 -0
- data/examples/sinatra_app/app.rb +335 -0
- data/lib/htm/active_record_config.rb +113 -0
- data/lib/htm/configuration.rb +342 -0
- data/lib/htm/database.rb +594 -0
- data/lib/htm/embedding_service.rb +115 -0
- data/lib/htm/errors.rb +34 -0
- data/lib/htm/job_adapter.rb +154 -0
- data/lib/htm/jobs/generate_embedding_job.rb +65 -0
- data/lib/htm/jobs/generate_tags_job.rb +82 -0
- data/lib/htm/long_term_memory.rb +965 -0
- data/lib/htm/models/node.rb +109 -0
- data/lib/htm/models/node_tag.rb +33 -0
- data/lib/htm/models/robot.rb +52 -0
- data/lib/htm/models/tag.rb +76 -0
- data/lib/htm/railtie.rb +76 -0
- data/lib/htm/sinatra.rb +157 -0
- data/lib/htm/tag_service.rb +135 -0
- data/lib/htm/tasks.rb +38 -0
- data/lib/htm/version.rb +5 -0
- data/lib/htm/working_memory.rb +182 -0
- data/lib/htm.rb +400 -0
- data/lib/tasks/db.rake +19 -0
- data/lib/tasks/htm.rake +147 -0
- data/lib/tasks/jobs.rake +312 -0
- data/mkdocs.yml +190 -0
- data/scripts/install_local_database.sh +309 -0
- metadata +341 -0
|
@@ -0,0 +1,115 @@
|
|
|
1
|
+
<svg viewBox="0 0 950 700" xmlns="http://www.w3.org/2000/svg" style="background: transparent;">
|
|
2
|
+
<defs>
|
|
3
|
+
<style>
|
|
4
|
+
.ltm-box { fill: rgba(33, 150, 243, 0.2); stroke: #2196F3; stroke-width: 3; }
|
|
5
|
+
.table-box { fill: rgba(76, 175, 80, 0.2); stroke: #4CAF50; stroke-width: 2; }
|
|
6
|
+
.index-box { fill: rgba(255, 152, 0, 0.2); stroke: #FF9800; stroke-width: 2; }
|
|
7
|
+
.hypertable { fill: rgba(156, 39, 176, 0.2); stroke: #9C27B0; stroke-width: 2; }
|
|
8
|
+
.text-header { fill: #E0E0E0; font-size: 18px; font-weight: bold; }
|
|
9
|
+
.text-label { fill: #E0E0E0; font-size: 14px; font-weight: bold; }
|
|
10
|
+
.text-small { fill: #B0B0B0; font-size: 12px; }
|
|
11
|
+
.text-tiny { fill: #A0A0A0; font-size: 10px; }
|
|
12
|
+
</style>
|
|
13
|
+
</defs>
|
|
14
|
+
|
|
15
|
+
<!-- Title -->
|
|
16
|
+
<text x="475" y="30" text-anchor="middle" class="text-header">Long-term Memory Architecture</text>
|
|
17
|
+
<text x="475" y="50" text-anchor="middle" class="text-small">PostgreSQL</text>
|
|
18
|
+
|
|
19
|
+
<!-- Main Container -->
|
|
20
|
+
<rect x="50" y="75" width="850" height="590" class="ltm-box" rx="5"/>
|
|
21
|
+
|
|
22
|
+
<!-- Tables Section -->
|
|
23
|
+
<text x="70" y="105" class="text-label">Database Tables</text>
|
|
24
|
+
|
|
25
|
+
<!-- nodes table -->
|
|
26
|
+
<rect x="70" y="120" width="250" height="180" class="table-box" rx="3"/>
|
|
27
|
+
<text x="195" y="140" text-anchor="middle" class="text-label" fill="#4CAF50">nodes</text>
|
|
28
|
+
<text x="80" y="165" class="text-small">Memories + Embeddings</text>
|
|
29
|
+
<text x="80" y="185" class="text-tiny">• id, key, value</text>
|
|
30
|
+
<text x="80" y="200" class="text-tiny">• type, category, importance</text>
|
|
31
|
+
<text x="80" y="215" class="text-tiny">• token_count, robot_id</text>
|
|
32
|
+
<text x="80" y="230" class="text-tiny">• embedding vector(1536)</text>
|
|
33
|
+
<text x="80" y="245" class="text-tiny">• created_at, last_accessed</text>
|
|
34
|
+
<text x="80" y="260" class="text-tiny">• in_working_memory flag</text>
|
|
35
|
+
<text x="80" y="280" class="text-small" fill="#66BB6A">Primary storage</text>
|
|
36
|
+
|
|
37
|
+
<!-- relationships table -->
|
|
38
|
+
<rect x="340" y="120" width="250" height="100" class="table-box" rx="3"/>
|
|
39
|
+
<text x="465" y="140" text-anchor="middle" class="text-label" fill="#4CAF50">relationships</text>
|
|
40
|
+
<text x="350" y="165" class="text-small">Knowledge Graph</text>
|
|
41
|
+
<text x="350" y="185" class="text-tiny">• from_node_id → to_node_id</text>
|
|
42
|
+
<text x="350" y="200" class="text-tiny">• relationship_type, strength</text>
|
|
43
|
+
|
|
44
|
+
<!-- tags table -->
|
|
45
|
+
<rect x="610" y="120" width="250" height="100" class="table-box" rx="3"/>
|
|
46
|
+
<text x="735" y="140" text-anchor="middle" class="text-label" fill="#4CAF50">tags</text>
|
|
47
|
+
<text x="620" y="165" class="text-small">Categorization</text>
|
|
48
|
+
<text x="620" y="185" class="text-tiny">• node_id, tag</text>
|
|
49
|
+
<text x="620" y="200" class="text-tiny">• Flexible tagging system</text>
|
|
50
|
+
|
|
51
|
+
<!-- robots table -->
|
|
52
|
+
<rect x="340" y="240" width="250" height="100" class="table-box" rx="3"/>
|
|
53
|
+
<text x="465" y="260" text-anchor="middle" class="text-label" fill="#4CAF50">robots</text>
|
|
54
|
+
<text x="350" y="285" class="text-small">Identity Registry</text>
|
|
55
|
+
<text x="350" y="305" class="text-tiny">• id, name, created_at</text>
|
|
56
|
+
<text x="350" y="320" class="text-tiny">• last_active, metadata</text>
|
|
57
|
+
|
|
58
|
+
<!-- operations_log table (hypertable) -->
|
|
59
|
+
<rect x="610" y="240" width="250" height="100" class="hypertable" rx="3"/>
|
|
60
|
+
<text x="735" y="260" text-anchor="middle" class="text-label" fill="#9C27B0">operations_log</text>
|
|
61
|
+
<text x="620" y="285" class="text-small">Audit Trail (Hypertable)</text>
|
|
62
|
+
<text x="620" y="305" class="text-tiny">• timestamp, operation</text>
|
|
63
|
+
<text x="620" y="320" class="text-tiny">• node_id, robot_id, details</text>
|
|
64
|
+
|
|
65
|
+
<!-- Indexes Section -->
|
|
66
|
+
<text x="70" y="375" class="text-label">Indexes & Optimization</text>
|
|
67
|
+
|
|
68
|
+
<!-- HNSW Vector Index -->
|
|
69
|
+
<rect x="70" y="390" width="250" height="120" class="index-box" rx="3"/>
|
|
70
|
+
<text x="195" y="410" text-anchor="middle" class="text-label" fill="#FF9800">HNSW Vector Index</text>
|
|
71
|
+
<text x="80" y="435" class="text-small">Semantic Similarity</text>
|
|
72
|
+
<text x="80" y="455" class="text-tiny">• idx_nodes_embedding</text>
|
|
73
|
+
<text x="80" y="470" class="text-tiny">• vector_cosine_ops</text>
|
|
74
|
+
<text x="80" y="485" class="text-tiny">• m=16, ef_construction=64</text>
|
|
75
|
+
<text x="80" y="500" class="text-tiny">⚡ Fast nearest-neighbor search</text>
|
|
76
|
+
|
|
77
|
+
<!-- GIN Full-text Index -->
|
|
78
|
+
<rect x="340" y="390" width="250" height="120" class="index-box" rx="3"/>
|
|
79
|
+
<text x="465" y="410" text-anchor="middle" class="text-label" fill="#FF9800">GIN Full-text Index</text>
|
|
80
|
+
<text x="350" y="435" class="text-small">Keyword Search</text>
|
|
81
|
+
<text x="350" y="455" class="text-tiny">• idx_nodes_fulltext</text>
|
|
82
|
+
<text x="350" y="470" class="text-tiny">• to_tsvector('english')</text>
|
|
83
|
+
<text x="350" y="485" class="text-tiny">• Supports ts_rank scoring</text>
|
|
84
|
+
<text x="350" y="500" class="text-tiny">⚡ Fast keyword matching</text>
|
|
85
|
+
|
|
86
|
+
<!-- PostgreSQL Chunks -->
|
|
87
|
+
<rect x="610" y="390" width="250" height="120" class="hypertable" rx="3"/>
|
|
88
|
+
<text x="735" y="410" text-anchor="middle" class="text-label" fill="#9C27B0">PostgreSQL Chunks</text>
|
|
89
|
+
<text x="620" y="435" class="text-small">Time-series Partitioning</text>
|
|
90
|
+
<text x="620" y="455" class="text-tiny">• Auto-partitioning by time</text>
|
|
91
|
+
<text x="620" y="470" class="text-tiny">• Compression policies</text>
|
|
92
|
+
<text x="620" y="485" class="text-tiny">• Retention policies</text>
|
|
93
|
+
<text x="620" y="500" class="text-tiny">⚡ Fast time-range queries</text>
|
|
94
|
+
|
|
95
|
+
<!-- Features Section -->
|
|
96
|
+
<text x="70" y="545" class="text-label">Key Features</text>
|
|
97
|
+
|
|
98
|
+
<rect x="70" y="560" width="250" height="90" fill="rgba(33, 150, 243, 0.15)" stroke="#2196F3" stroke-width="1.5" rx="3"/>
|
|
99
|
+
<text x="80" y="585" class="text-small">✓ Permanent storage</text>
|
|
100
|
+
<text x="80" y="605" class="text-small">✓ ACID compliant</text>
|
|
101
|
+
<text x="80" y="625" class="text-small">✓ Never forgets</text>
|
|
102
|
+
<text x="80" y="645" class="text-small">✓ Unlimited capacity</text>
|
|
103
|
+
|
|
104
|
+
<rect x="340" y="560" width="250" height="90" fill="rgba(33, 150, 243, 0.15)" stroke="#2196F3" stroke-width="1.5" rx="3"/>
|
|
105
|
+
<text x="350" y="585" class="text-small">✓ Vector similarity search</text>
|
|
106
|
+
<text x="350" y="605" class="text-small">✓ Full-text search</text>
|
|
107
|
+
<text x="350" y="625" class="text-small">✓ Hybrid retrieval</text>
|
|
108
|
+
<text x="350" y="645" class="text-small">✓ RAG-optimized</text>
|
|
109
|
+
|
|
110
|
+
<rect x="610" y="560" width="250" height="90" fill="rgba(33, 150, 243, 0.15)" stroke="#2196F2" stroke-width="1.5" rx="3"/>
|
|
111
|
+
<text x="620" y="585" class="text-small">✓ Multi-robot support</text>
|
|
112
|
+
<text x="620" y="605" class="text-small">✓ Relationship graphs</text>
|
|
113
|
+
<text x="620" y="625" class="text-small">✓ Tag-based filtering</text>
|
|
114
|
+
<text x="620" y="645" class="text-small">✓ Complete audit trail</text>
|
|
115
|
+
</svg>
|
|
@@ -0,0 +1,120 @@
|
|
|
1
|
+
<svg viewBox="0 0 900 650" xmlns="http://www.w3.org/2000/svg" style="background: transparent;">
|
|
2
|
+
<defs>
|
|
3
|
+
<style>
|
|
4
|
+
.wm-box { fill: rgba(33, 150, 243, 0.2); stroke: #2196F3; stroke-width: 3; }
|
|
5
|
+
.ltm-box { fill: rgba(76, 175, 80, 0.2); stroke: #4CAF50; stroke-width: 3; }
|
|
6
|
+
.memory-item { fill: rgba(255, 152, 0, 0.3); stroke: #FF9800; stroke-width: 1.5; }
|
|
7
|
+
.evicted-item { fill: rgba(244, 67, 54, 0.3); stroke: #F44336; stroke-width: 1.5; }
|
|
8
|
+
.text-header { fill: #E0E0E0; font-size: 18px; font-weight: bold; }
|
|
9
|
+
.text-label { fill: #E0E0E0; font-size: 13px; }
|
|
10
|
+
.text-small { fill: #B0B0B0; font-size: 11px; }
|
|
11
|
+
.arrow { stroke: #4A9EFF; stroke-width: 2; fill: none; marker-end: url(#arrowhead); }
|
|
12
|
+
.arrow-evict { stroke: #F44336; stroke-width: 2; fill: none; marker-end: url(#arrowhead-red); stroke-dasharray: 5,3; }
|
|
13
|
+
.arrow-recall { stroke: #4CAF50; stroke-width: 2; fill: none; marker-end: url(#arrowhead-green); stroke-dasharray: 5,3; }
|
|
14
|
+
</style>
|
|
15
|
+
<marker id="arrowhead" markerWidth="10" markerHeight="10" refX="9" refY="3" orient="auto">
|
|
16
|
+
<polygon points="0 0, 10 3, 0 6" fill="#4A9EFF" />
|
|
17
|
+
</marker>
|
|
18
|
+
<marker id="arrowhead-red" markerWidth="10" markerHeight="10" refX="9" refY="3" orient="auto">
|
|
19
|
+
<polygon points="0 0, 10 3, 0 6" fill="#F44336" />
|
|
20
|
+
</marker>
|
|
21
|
+
<marker id="arrowhead-green" markerWidth="10" markerHeight="10" refX="9" refY="3" orient="auto">
|
|
22
|
+
<polygon points="0 0, 10 3, 0 6" fill="#4CAF50" />
|
|
23
|
+
</marker>
|
|
24
|
+
</defs>
|
|
25
|
+
|
|
26
|
+
<!-- Title -->
|
|
27
|
+
<text x="450" y="30" text-anchor="middle" class="text-header">Working Memory Architecture</text>
|
|
28
|
+
|
|
29
|
+
<!-- Working Memory Box -->
|
|
30
|
+
<rect x="50" y="70" width="800" height="220" class="wm-box" rx="5"/>
|
|
31
|
+
<text x="450" y="100" text-anchor="middle" class="text-header">Working Memory</text>
|
|
32
|
+
<text x="450" y="120" text-anchor="middle" class="text-small">(In-memory, Token-limited)</text>
|
|
33
|
+
|
|
34
|
+
<!-- Capacity bar -->
|
|
35
|
+
<rect x="70" y="140" width="760" height="25" fill="rgba(255, 255, 255, 0.1)" stroke="#666" stroke-width="1" rx="3"/>
|
|
36
|
+
<rect x="70" y="140" width="608" height="25" fill="rgba(76, 175, 80, 0.4)" rx="3"/>
|
|
37
|
+
<text x="450" y="157" text-anchor="middle" class="text-small">102,400 / 128,000 tokens (80% full)</text>
|
|
38
|
+
|
|
39
|
+
<!-- Memory items in working memory -->
|
|
40
|
+
<rect x="70" y="180" width="150" height="40" class="memory-item" rx="3"/>
|
|
41
|
+
<text x="80" y="197" class="text-label">Critical Decision</text>
|
|
42
|
+
<text x="210" y="197" text-anchor="end" class="text-small">imp: 10.0</text>
|
|
43
|
+
<text x="210" y="212" text-anchor="end" class="text-small">2000 tok</text>
|
|
44
|
+
|
|
45
|
+
<rect x="235" y="180" width="150" height="40" class="memory-item" rx="3"/>
|
|
46
|
+
<text x="245" y="197" class="text-label">User Preference</text>
|
|
47
|
+
<text x="375" y="197" text-anchor="end" class="text-small">imp: 8.0</text>
|
|
48
|
+
<text x="375" y="212" text-anchor="end" class="text-small">500 tok</text>
|
|
49
|
+
|
|
50
|
+
<rect x="400" y="180" width="150" height="40" class="memory-item" rx="3"/>
|
|
51
|
+
<text x="410" y="197" class="text-label">Recent Context</text>
|
|
52
|
+
<text x="540" y="197" text-anchor="end" class="text-small">imp: 6.0</text>
|
|
53
|
+
<text x="540" y="212" text-anchor="end" class="text-small">1500 tok</text>
|
|
54
|
+
|
|
55
|
+
<rect x="565" y="180" width="150" height="40" class="evicted-item" rx="3"/>
|
|
56
|
+
<text x="575" y="197" class="text-label">Old Note</text>
|
|
57
|
+
<text x="705" y="197" text-anchor="end" class="text-small">imp: 2.0</text>
|
|
58
|
+
<text x="705" y="212" text-anchor="end" class="text-small">800 tok</text>
|
|
59
|
+
<text x="640" y="235" text-anchor="middle" class="text-small" fill="#F44336">← Being evicted</text>
|
|
60
|
+
|
|
61
|
+
<!-- Features list -->
|
|
62
|
+
<text x="70" y="260" class="text-small">✓ Fast access</text>
|
|
63
|
+
<text x="250" y="260" class="text-small">✓ Token-based capacity</text>
|
|
64
|
+
<text x="450" y="260" class="text-small">✓ Automatic eviction</text>
|
|
65
|
+
<text x="650" y="260" class="text-small">✓ Importance-based retention</text>
|
|
66
|
+
|
|
67
|
+
<!-- Arrow down (Eviction) -->
|
|
68
|
+
<path d="M 640 290 L 640 350" class="arrow-evict"/>
|
|
69
|
+
<text x="660" y="325" class="text-small" fill="#F44336">Eviction</text>
|
|
70
|
+
<text x="660" y="340" class="text-small" fill="#F44336">(when full)</text>
|
|
71
|
+
|
|
72
|
+
<!-- Arrow up (Recall) -->
|
|
73
|
+
<path d="M 260 350 L 260 290" class="arrow-recall"/>
|
|
74
|
+
<text x="280" y="315" class="text-small" fill="#4CAF50">Recall</text>
|
|
75
|
+
<text x="280" y="330" class="text-small" fill="#4CAF50">(RAG retrieval)</text>
|
|
76
|
+
|
|
77
|
+
<!-- Long-term Memory Box -->
|
|
78
|
+
<rect x="50" y="370" width="800" height="240" class="ltm-box" rx="5"/>
|
|
79
|
+
<text x="450" y="400" text-anchor="middle" class="text-header">Long-term Memory</text>
|
|
80
|
+
<text x="450" y="420" text-anchor="middle" class="text-small">(PostgreSQL, Unlimited)</text>
|
|
81
|
+
|
|
82
|
+
<!-- Database tables visualization -->
|
|
83
|
+
<rect x="70" y="440" width="180" height="150" fill="rgba(76, 175, 80, 0.1)" stroke="#4CAF50" stroke-width="1" rx="3"/>
|
|
84
|
+
<text x="160" y="460" text-anchor="middle" class="text-label">nodes</text>
|
|
85
|
+
<text x="80" y="480" class="text-small">All memories</text>
|
|
86
|
+
<text x="80" y="495" class="text-small">Vector embeddings</text>
|
|
87
|
+
<text x="80" y="510" class="text-small">Full-text search</text>
|
|
88
|
+
<text x="80" y="525" class="text-small">Timestamps</text>
|
|
89
|
+
<text x="80" y="540" class="text-small">Importance scores</text>
|
|
90
|
+
<text x="80" y="555" class="text-small">Robot attribution</text>
|
|
91
|
+
<text x="80" y="570" class="text-small">in_working_memory flag</text>
|
|
92
|
+
|
|
93
|
+
<rect x="265" y="440" width="180" height="70" fill="rgba(76, 175, 80, 0.1)" stroke="#4CAF50" stroke-width="1" rx="3"/>
|
|
94
|
+
<text x="355" y="460" text-anchor="middle" class="text-label">relationships</text>
|
|
95
|
+
<text x="275" y="480" class="text-small">Knowledge graph</text>
|
|
96
|
+
<text x="275" y="495" class="text-small">Node connections</text>
|
|
97
|
+
|
|
98
|
+
<rect x="460" y="440" width="180" height="70" fill="rgba(76, 175, 80, 0.1)" stroke="#4CAF50" stroke-width="1" rx="3"/>
|
|
99
|
+
<text x="550" y="460" text-anchor="middle" class="text-label">tags</text>
|
|
100
|
+
<text x="470" y="480" class="text-small">Categorization</text>
|
|
101
|
+
<text x="470" y="495" class="text-small">Flexible tagging</text>
|
|
102
|
+
|
|
103
|
+
<rect x="655" y="440" width="180" height="70" fill="rgba(76, 175, 80, 0.1)" stroke="#4CAF50" stroke-width="1" rx="3"/>
|
|
104
|
+
<text x="745" y="460" text-anchor="middle" class="text-label">operations_log</text>
|
|
105
|
+
<text x="665" y="480" class="text-small">Audit trail</text>
|
|
106
|
+
<text x="665" y="495" class="text-small">PostgreSQL table</text>
|
|
107
|
+
|
|
108
|
+
<!-- Evicted item in long-term memory -->
|
|
109
|
+
<rect x="265" y="520" width="380" height="70" class="evicted-item" rx="3"/>
|
|
110
|
+
<text x="275" y="540" class="text-label">Evicted: Old Note</text>
|
|
111
|
+
<text x="635" y="540" text-anchor="end" class="text-small">in_working_memory: false</text>
|
|
112
|
+
<text x="275" y="560" class="text-small">Still stored, can be recalled when needed</text>
|
|
113
|
+
<text x="275" y="575" class="text-small">Full history preserved • Never forgotten</text>
|
|
114
|
+
|
|
115
|
+
<!-- Features list -->
|
|
116
|
+
<text x="70" y="335" class="text-small" fill="#4CAF50">✓ Durable storage</text>
|
|
117
|
+
<text x="270" y="335" class="text-small" fill="#4CAF50">✓ Full history</text>
|
|
118
|
+
<text x="450" y="335" class="text-small" fill="#4CAF50">✓ RAG retrieval</text>
|
|
119
|
+
<text x="610" y="335" class="text-small" fill="#4CAF50">✓ Never forgets</text>
|
|
120
|
+
</svg>
|
|
Binary file
|
|
Binary file
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
/* MathJax Configuration for HTM Documentation */
|
|
2
|
+
|
|
3
|
+
window.MathJax = {
|
|
4
|
+
tex: {
|
|
5
|
+
inlineMath: [["\\(", "\\)"]],
|
|
6
|
+
displayMath: [["\\[", "\\]"]],
|
|
7
|
+
processEscapes: true,
|
|
8
|
+
processEnvironments: true
|
|
9
|
+
},
|
|
10
|
+
options: {
|
|
11
|
+
ignoreHtmlClass: ".*|",
|
|
12
|
+
processHtmlClass: "arithmatex"
|
|
13
|
+
}
|
|
14
|
+
};
|
|
15
|
+
|
|
16
|
+
document$.subscribe(() => {
|
|
17
|
+
MathJax.typesetPromise()
|
|
18
|
+
})
|
|
Binary file
|
|
@@ -0,0 +1,322 @@
|
|
|
1
|
+
# HTM Database Rake Tasks Reference
|
|
2
|
+
|
|
3
|
+
Complete reference for HTM database management tasks.
|
|
4
|
+
|
|
5
|
+
## Quick Start
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
# First time setup
|
|
9
|
+
cd /path/to/htm
|
|
10
|
+
direnv allow # Load environment variables from .envrc
|
|
11
|
+
rake htm:db:setup # Create schema and run migrations
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
## Available Tasks
|
|
15
|
+
|
|
16
|
+
### Setup and Schema
|
|
17
|
+
|
|
18
|
+
#### `rake htm:db:setup`
|
|
19
|
+
Sets up the HTM database schema and runs all migrations.
|
|
20
|
+
|
|
21
|
+
**What it does:**
|
|
22
|
+
- Verifies required extensions (timescaledb, pgvector, pg_trgm)
|
|
23
|
+
- Creates all HTM tables (nodes, tags, robots, operations_log)
|
|
24
|
+
- Runs all pending migrations
|
|
25
|
+
- Sets up hypertables for time-series optimization
|
|
26
|
+
|
|
27
|
+
**When to use:** First-time setup or after dropping the database
|
|
28
|
+
|
|
29
|
+
```bash
|
|
30
|
+
$ rake htm:db:setup
|
|
31
|
+
✓ TimescaleDB version: 2.22.1
|
|
32
|
+
✓ pgvector version: 0.8.1
|
|
33
|
+
Creating HTM schema...
|
|
34
|
+
✓ Schema created
|
|
35
|
+
Running migration: 001_support_variable_dimensions
|
|
36
|
+
✓ Migration 001_support_variable_dimensions applied
|
|
37
|
+
Running migration: 002_ontology_topic_extraction
|
|
38
|
+
✓ Migration 002_ontology_topic_extraction applied
|
|
39
|
+
✓ Created hypertable for operations_log
|
|
40
|
+
✓ HTM database schema created successfully
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
---
|
|
44
|
+
|
|
45
|
+
### Migrations
|
|
46
|
+
|
|
47
|
+
#### `rake htm:db:migrate`
|
|
48
|
+
Runs pending database migrations only.
|
|
49
|
+
|
|
50
|
+
**What it does:**
|
|
51
|
+
- Checks which migrations have been applied
|
|
52
|
+
- Runs any new migrations in `sql/migrations/`
|
|
53
|
+
- Updates the `schema_migrations` table
|
|
54
|
+
|
|
55
|
+
**When to use:** After pulling new code with migrations
|
|
56
|
+
|
|
57
|
+
```bash
|
|
58
|
+
$ rake htm:db:migrate
|
|
59
|
+
Running migration: 002_ontology_topic_extraction
|
|
60
|
+
✓ Migration 002_ontology_topic_extraction applied
|
|
61
|
+
✓ Database migrations completed
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
#### `rake htm:db:status`
|
|
65
|
+
Shows which migrations have been applied and which are pending.
|
|
66
|
+
|
|
67
|
+
**Example output:**
|
|
68
|
+
```bash
|
|
69
|
+
$ rake htm:db:status
|
|
70
|
+
|
|
71
|
+
Migration Status
|
|
72
|
+
================================================================================
|
|
73
|
+
✓ 001_support_variable_dimensions (applied: 2025-10-26 04:27:15.428951+00)
|
|
74
|
+
✓ 002_ontology_topic_extraction (applied: 2025-10-27 03:44:15.012345+00)
|
|
75
|
+
|
|
76
|
+
Summary: 2 applied, 0 pending
|
|
77
|
+
================================================================================
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
---
|
|
81
|
+
|
|
82
|
+
### Information
|
|
83
|
+
|
|
84
|
+
#### `rake htm:db:info`
|
|
85
|
+
Shows comprehensive database information.
|
|
86
|
+
|
|
87
|
+
**Example output:**
|
|
88
|
+
```bash
|
|
89
|
+
$ rake htm:db:info
|
|
90
|
+
|
|
91
|
+
HTM Database Information
|
|
92
|
+
================================================================================
|
|
93
|
+
|
|
94
|
+
Connection:
|
|
95
|
+
Host: cw7rxj91bm.srbbwwxn56.tsdb.cloud.timescale.com
|
|
96
|
+
Port: 37807
|
|
97
|
+
Database: tsdb
|
|
98
|
+
User: tsdbadmin
|
|
99
|
+
|
|
100
|
+
PostgreSQL Version:
|
|
101
|
+
PostgreSQL 17.6 (Ubuntu 17.6-2.pgdg22.04+1) on aarch64-unknown-linux-gnu
|
|
102
|
+
|
|
103
|
+
Extensions:
|
|
104
|
+
ai (0.11.2)
|
|
105
|
+
pg_stat_statements (1.11)
|
|
106
|
+
pg_trgm (1.6)
|
|
107
|
+
plpgsql (1.0)
|
|
108
|
+
timescaledb (2.22.1)
|
|
109
|
+
timescaledb_toolkit (1.21.0)
|
|
110
|
+
vector (0.8.1)
|
|
111
|
+
vectorscale (0.8.0)
|
|
112
|
+
|
|
113
|
+
HTM Tables:
|
|
114
|
+
nodes: 42 rows
|
|
115
|
+
tags: 156 rows
|
|
116
|
+
robots: 3 rows
|
|
117
|
+
operations_log: 289 rows
|
|
118
|
+
schema_migrations: 2 rows
|
|
119
|
+
|
|
120
|
+
Database Size: 14 MB
|
|
121
|
+
================================================================================
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
#### `rake htm:db:test`
|
|
125
|
+
Tests database connection by running `test_connection.rb`.
|
|
126
|
+
|
|
127
|
+
**Example output:**
|
|
128
|
+
```bash
|
|
129
|
+
$ rake htm:db:test
|
|
130
|
+
Connecting to TimescaleDB...
|
|
131
|
+
✓ Connected successfully!
|
|
132
|
+
✓ TimescaleDB Extension: Version 2.22.1
|
|
133
|
+
✓ pgvector Extension: Version 0.8.1
|
|
134
|
+
✓ pg_trgm Extension: Version 1.6
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
---
|
|
138
|
+
|
|
139
|
+
### Utilities
|
|
140
|
+
|
|
141
|
+
#### `rake htm:db:console`
|
|
142
|
+
Opens an interactive PostgreSQL console (psql).
|
|
143
|
+
|
|
144
|
+
**What it does:**
|
|
145
|
+
- Launches `psql` connected to your HTM database
|
|
146
|
+
- Uses connection parameters from `HTM_DBURL` or `.envrc`
|
|
147
|
+
- Allows you to run SQL queries directly
|
|
148
|
+
|
|
149
|
+
**Example:**
|
|
150
|
+
```bash
|
|
151
|
+
$ rake htm:db:console
|
|
152
|
+
psql (17.6)
|
|
153
|
+
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, compression: off, ALPN: none)
|
|
154
|
+
Type "help" for help.
|
|
155
|
+
|
|
156
|
+
tsdb=> SELECT COUNT(*) FROM nodes;
|
|
157
|
+
count
|
|
158
|
+
-------
|
|
159
|
+
42
|
|
160
|
+
(1 row)
|
|
161
|
+
|
|
162
|
+
tsdb=> \d nodes
|
|
163
|
+
tsdb=> \q
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
#### `rake htm:db:seed`
|
|
167
|
+
Seeds the database with sample data.
|
|
168
|
+
|
|
169
|
+
**What it does:**
|
|
170
|
+
- Creates a sample robot
|
|
171
|
+
- Adds 3 sample nodes with different topics
|
|
172
|
+
- Useful for testing or demos
|
|
173
|
+
|
|
174
|
+
**Example:**
|
|
175
|
+
```bash
|
|
176
|
+
$ rake htm:db:seed
|
|
177
|
+
Seeding database with sample data...
|
|
178
|
+
Creating sample nodes...
|
|
179
|
+
✓ Database seeded with 3 sample nodes
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
---
|
|
183
|
+
|
|
184
|
+
### Destructive Operations
|
|
185
|
+
|
|
186
|
+
⚠️ **WARNING**: These tasks delete data and cannot be undone!
|
|
187
|
+
|
|
188
|
+
#### `rake htm:db:drop`
|
|
189
|
+
Drops all HTM tables, functions, triggers, and views.
|
|
190
|
+
|
|
191
|
+
**What it does:**
|
|
192
|
+
- Drops tables: nodes, tags, robots, operations_log, schema_migrations
|
|
193
|
+
- Drops ontology functions and triggers
|
|
194
|
+
- Drops views: ontology_structure, topic_relationships
|
|
195
|
+
|
|
196
|
+
**Safety:** Prompts for confirmation ("yes" must be typed)
|
|
197
|
+
|
|
198
|
+
```bash
|
|
199
|
+
$ rake htm:db:drop
|
|
200
|
+
Are you sure you want to drop all tables? This cannot be undone! (yes/no): yes
|
|
201
|
+
Dropping HTM tables...
|
|
202
|
+
✓ Dropped nodes
|
|
203
|
+
✓ Dropped tags
|
|
204
|
+
✓ Dropped robots
|
|
205
|
+
✓ Dropped operations_log
|
|
206
|
+
✓ Dropped schema_migrations
|
|
207
|
+
✓ Dropped ontology functions and triggers
|
|
208
|
+
✓ Dropped ontology views
|
|
209
|
+
✓ All HTM tables dropped
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
#### `rake htm:db:reset`
|
|
213
|
+
Drops and recreates the entire database (equivalent to `drop` + `setup`).
|
|
214
|
+
|
|
215
|
+
**When to use:** Development only, to start fresh
|
|
216
|
+
|
|
217
|
+
```bash
|
|
218
|
+
$ rake htm:db:reset
|
|
219
|
+
# Runs drop (with confirmation) then setup
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
---
|
|
223
|
+
|
|
224
|
+
## Environment Variables
|
|
225
|
+
|
|
226
|
+
All tasks require database configuration via environment variables. Use one of these methods:
|
|
227
|
+
|
|
228
|
+
### Method 1: direnv (Recommended)
|
|
229
|
+
```bash
|
|
230
|
+
# One-time setup
|
|
231
|
+
cd /path/to/htm
|
|
232
|
+
direnv allow
|
|
233
|
+
|
|
234
|
+
# Variables are automatically loaded from .envrc
|
|
235
|
+
rake htm:db:info
|
|
236
|
+
```
|
|
237
|
+
|
|
238
|
+
### Method 2: Manual Export
|
|
239
|
+
```bash
|
|
240
|
+
export HTM_DBURL="postgresql://user:password@host:port/dbname?sslmode=require"
|
|
241
|
+
rake htm:db:info
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
### Method 3: Source Tiger Credentials
|
|
245
|
+
```bash
|
|
246
|
+
source ~/.bashrc__tiger # If using TimescaleDB Cloud
|
|
247
|
+
rake htm:db:info
|
|
248
|
+
```
|
|
249
|
+
|
|
250
|
+
---
|
|
251
|
+
|
|
252
|
+
## Common Workflows
|
|
253
|
+
|
|
254
|
+
### Initial Project Setup
|
|
255
|
+
```bash
|
|
256
|
+
cd /path/to/htm
|
|
257
|
+
direnv allow
|
|
258
|
+
bundle install
|
|
259
|
+
rake htm:db:setup
|
|
260
|
+
rake htm:db:seed # Optional: add sample data
|
|
261
|
+
rake htm:db:info # Verify setup
|
|
262
|
+
```
|
|
263
|
+
|
|
264
|
+
### After Pulling New Code
|
|
265
|
+
```bash
|
|
266
|
+
git pull
|
|
267
|
+
rake htm:db:status # Check for new migrations
|
|
268
|
+
rake htm:db:migrate # Run pending migrations
|
|
269
|
+
```
|
|
270
|
+
|
|
271
|
+
### Development Reset
|
|
272
|
+
```bash
|
|
273
|
+
rake htm:db:reset # Drop and recreate (type 'yes' to confirm)
|
|
274
|
+
rake htm:db:seed # Re-add sample data
|
|
275
|
+
```
|
|
276
|
+
|
|
277
|
+
### Debugging
|
|
278
|
+
```bash
|
|
279
|
+
rake htm:db:info # Check database state
|
|
280
|
+
rake htm:db:status # Check migration status
|
|
281
|
+
rake htm:db:console # Open psql for SQL queries
|
|
282
|
+
```
|
|
283
|
+
|
|
284
|
+
### Production Deployment
|
|
285
|
+
```bash
|
|
286
|
+
# NEVER use reset or drop in production!
|
|
287
|
+
rake htm:db:migrate # Run new migrations only
|
|
288
|
+
```
|
|
289
|
+
|
|
290
|
+
---
|
|
291
|
+
|
|
292
|
+
## Troubleshooting
|
|
293
|
+
|
|
294
|
+
### "Database configuration not found"
|
|
295
|
+
- Run `direnv allow` in the project directory
|
|
296
|
+
- Or manually export `HTM_DBURL`
|
|
297
|
+
- Verify: `echo $HTM_DBURL`
|
|
298
|
+
|
|
299
|
+
### "Connection refused"
|
|
300
|
+
- Check database is running
|
|
301
|
+
- Verify host/port in `HTM_DBURL`
|
|
302
|
+
- Test: `rake htm:db:test`
|
|
303
|
+
|
|
304
|
+
### "Extension not found"
|
|
305
|
+
- Ensure TimescaleDB Cloud instance has required extensions
|
|
306
|
+
- Check with: `rake htm:db:info`
|
|
307
|
+
- Extensions needed: timescaledb, pgvector, pg_trgm
|
|
308
|
+
|
|
309
|
+
### Migrations not running
|
|
310
|
+
- Check migration files exist: `ls -la sql/migrations/`
|
|
311
|
+
- Verify migrations table: `rake htm:db:console` then `SELECT * FROM schema_migrations;`
|
|
312
|
+
- Force re-run: `rake htm:db:drop` then `rake htm:db:setup`
|
|
313
|
+
|
|
314
|
+
---
|
|
315
|
+
|
|
316
|
+
## Legacy Tasks (Deprecated)
|
|
317
|
+
|
|
318
|
+
These tasks still work but will be removed in a future version:
|
|
319
|
+
|
|
320
|
+
- `rake db_setup` → Use `rake htm:db:setup`
|
|
321
|
+
- `rake db_test` → Use `rake htm:db:test`
|
|
322
|
+
|