htm 0.0.10 → 0.0.14

data/notes/tag_ontology_enhancement_ideas.md

@@ -1,222 +0,0 @@
-# Tag Ontology Enhancement Ideas
-
-## Problem Statement
-
-HTM builds a dynamic hierarchical tag ontology but doesn't fully leverage it for retrieval. The tag extraction is sophisticated (LLM-based, ontology-consistent, hierarchical), but the retrieval capabilities are limited.
-
-## Current State Analysis
-
-### What's Working Well
-- Sophisticated LLM-based tag extraction with ontology consistency
-- Hierarchical format (`database:postgresql:indexes`)
-- Beautiful visualization (tree, Mermaid, SVG)
-- Tag relationship analysis (co-occurrence)
-- Async generation via `GenerateTagsJob`
-
-### What's Missing
-1. **`search_by_tags()` and `nodes_by_topic()` exist in LongTermMemory but aren't exposed in the public HTM API**
-2. `recall()` accepts `query_tags:` parameter but **ignores it** except for relevance scoring
-3. No hierarchical traversal (parent/child/ancestor queries)
-4. No tag-only retrieval (without text query)
-5. No query expansion using tag relationships
-6. No faceted navigation
-7. No tag-based result grouping
-
----
-
-## Proposed Enhancements
-
-### 1. Expose Tag-Based Retrieval in HTM API
-
-**Priority: High** - These methods already exist, just need to be exposed.
-
-```ruby
-# Browse by topic path (hierarchical navigation)
-htm.browse("database:postgresql")  # All nodes under this branch
-
-# Filter recall with tags
-htm.recall("query", tags: ["database:postgresql"], match_all: false)
-
-# Tag-only retrieval (no text query required)
-htm.by_tags(["database:postgresql", "ai:embeddings"])
-```
-
-**Implementation:**
-- Add `browse(topic_path, exact: false, limit: 20)` to HTM class
-- Add `by_tags(tags, match_all: false, limit: 20)` to HTM class
-- Wire `query_tags:` parameter through to actual filtering in `recall()`
-
----
-
-### 2. Hierarchical Query Expansion
-
-**Priority: High** - Leverages the hierarchical structure.
-
-```ruby
-# Searching "database:postgresql" should optionally include children:
-#   database:postgresql:indexes
-#   database:postgresql:extensions
-#   database:postgresql:partitioning
-
-htm.recall("indexes", tags: ["database:postgresql"], expand_children: true)
-
-# Or expand upward to include parent context
-htm.recall("specific query", tags: ["database:postgresql:indexes"], expand_ancestors: true)
-```
-
-**Implementation:**
-- Add `expand_children:` and `expand_ancestors:` options
-- Query `tags.name LIKE 'database:postgresql:%'` for children
-- Parse tag path and query ancestors for upward expansion
-
----
-
-### 3. Faceted Search / Tag Aggregation
-
-**Priority: Medium** - Enables discovery and navigation.
-
-```ruby
-# "What topics are represented in my search results?"
-results = htm.recall("machine learning")
-facets = results.facet_by_tags
-# => { "ai:ml" => 15, "ai:llm" => 8, "database:vector" => 5 }
-
-# Or as a standalone method
-htm.tag_facets(query: "machine learning", limit: 10)
-```
-
-**Implementation:**
-- Return tag counts grouped by topic
-- Weight by specificity (deeper tags = more specific)
-- Consider returning hierarchical facet structure
-
----
-
-### 4. Semantic Tag Matching in Recall
-
-**Priority: Medium** - Auto-extract tags from query.
-
-```ruby
-# Auto-extract tags from query and use them to boost results
-htm.recall("PostgreSQL vector search", auto_tag: true)
-# Internally: extracts ["database:postgresql", "database:vector-search"]
-# and uses these to boost relevant results
-```
-
-**Implementation:**
-- Use `find_query_matching_tags()` (already exists in LongTermMemory)
-- Boost results that match extracted tags
-- Make this opt-in or opt-out via configuration
-
----
-
-### 5. Tag-Based Context Assembly
-
-**Priority: Medium** - Topic-focused context building.
-
-```ruby
-# Assemble context prioritizing specific topic branches
-htm.assemble_context(
-  token_budget: 4000,
-  focus_topics: ["database:postgresql", "ai:embeddings"],
-  strategy: :topic_balanced  # New strategy
-)
-```
-
-**Implementation:**
-- Add `:topic_balanced` strategy to WorkingMemory
-- Weight nodes by tag overlap with focus topics
-- Ensure diverse topic coverage within budget
-
----
-
-### 6. Ontology-Aware Related Memories
-
-**Priority: Low** - Nice-to-have for exploration.
-
-```ruby
-# Find related memories via shared tags (not just vector similarity)
-htm.related_by_topic(node_id)
-# Returns nodes sharing the most tags, weighted by specificity
-
-# Compare to existing vector-based similarity
-htm.similar(node_id)  # Vector similarity
-htm.related_by_topic(node_id)  # Tag-based similarity
-```
-
-**Implementation:**
-- Count shared tags between nodes
-- Weight by tag depth (more specific = higher weight)
-- Combine with vector similarity for hybrid relatedness
-
----
-
-### 7. Tag Model Enhancements
-
-**Priority: Medium** - Better hierarchy navigation.
-
-```ruby
-# Add to HTM::Models::Tag
-tag.parent          # Parent tag (e.g., "database:postgresql" -> "database")
-tag.children        # Child tags (e.g., "database" -> ["database:postgresql", "database:mysql"])
-tag.siblings        # Same-level tags under same parent
-tag.ancestors       # All ancestors up to root
-tag.descendants     # All descendants (recursive)
-
-# Class methods
-Tag.roots           # All root-level tags
-Tag.at_depth(2)     # All tags at specific depth
-Tag.under("database")  # All tags in this branch
-```
-
-**Implementation:**
-- Parse colon-separated paths
-- Use SQL LIKE queries for efficient hierarchy traversal
-- Consider materialized path or nested set for performance at scale
-
----
-
-### 8. Tag-Based Grouping in Results
-
-**Priority: Low** - Organizational feature.
-
-```ruby
-# Group results by their primary tag
-results = htm.recall("query", group_by_tag: true)
-# => {
-#   "database:postgresql" => [node1, node2],
-#   "ai:embeddings" => [node3, node4],
-#   "uncategorized" => [node5]
-# }
-```
-
----
-
-## Implementation Priority
-
-### Phase 1 (High Value, Low Effort)
-1. Expose existing `search_by_tags()` and `nodes_by_topic()` in HTM API
-2. Wire `query_tags:` parameter through `recall()` for actual filtering
-
-### Phase 2 (High Value, Medium Effort)
-3. Add hierarchical query expansion (`expand_children:`, `expand_ancestors:`)
-4. Add Tag model hierarchy methods (parent, children, ancestors, descendants)
-
-### Phase 3 (Medium Value, Medium Effort)
-5. Faceted search / tag aggregation
-6. Semantic tag matching in recall (auto-extract from query)
-7. Tag-based context assembly strategy
-
-### Phase 4 (Nice to Have)
-8. Ontology-aware related memories
-9. Tag-based result grouping
-
----
-
-## Notes
-
-- All enhancements should maintain backward compatibility
-- Consider adding configuration options for default behaviors
-- Tag operations should be efficient (indexed queries)
-- Consider caching popular tag queries
-- Document new methods with YARD comments

data/notes/timescaledb_removal_summary.md

@@ -1,200 +0,0 @@
-# TimescaleDB Removal Summary
-
-**Date:** 2025-10-28
-**Decision:** Remove TimescaleDB extension from HTM gem as it does not add sufficient value
-
-## Overview
-
-TimescaleDB was originally included in the HTM gem for time-series optimization capabilities. However, analysis revealed that:
-
-1. **No hypertables were actually created** - The `setup_hypertables` method in `lib/htm/database.rb` was essentially a no-op with a comment stating "All tables use simple PRIMARY KEY (id), no hypertable conversions"
-2. **Time-range queries use standard indexed columns** - The `created_at` column on `nodes` and `timestamp` column on `operations_log` are indexed using standard PostgreSQL B-tree indexes
-3. **No compression policies were used** - Despite documentation mentioning compression, no actual compression was implemented
-4. **Additional dependency overhead** - Required users to have TimescaleDB available even though it provided no actual benefit
-
-## Code Files Modified
-
-The following code files were modified to remove TimescaleDB:
-
-### 1. `lib/htm/active_record_config.rb`
-**Lines modified:** 71-74
-
-**Changes:**
-- Removed `'timescaledb' => 'TimescaleDB extension'` from the `required_extensions` hash in `verify_extensions!` method
-
-**Before:**
-```ruby
-required_extensions = {
-  'timescaledb' => 'TimescaleDB extension',
-  'vector' => 'pgvector extension',
-  'pg_trgm' => 'PostgreSQL trigram extension'
-}
-```
-
-**After:**
-```ruby
-required_extensions = {
-  'vector' => 'pgvector extension',
-  'pg_trgm' => 'PostgreSQL trigram extension'
-}
-```
-
-### 2. `lib/htm/database.rb`
-**Multiple sections modified**
-
-**Changes:**
-- **Line 9:** Updated class documentation comment from "Handles schema creation and TimescaleDB hypertable setup" to "Handles schema creation and database initialization"
-- **Lines 31-39:** Removed entire hypertable conversion block that called `setup_hypertables(conn)`
-- **Lines 342-347:** Removed TimescaleDB version check from `verify_extensions` method
-- **Lines 432-437:** Removed entire `setup_hypertables` method definition
-
-**Impact:** The Database class now only handles standard PostgreSQL schema setup without any TimescaleDB-specific code.
-
-### 3. `db/README.md`
-**Lines modified:** 29, 127
-
-**Changes:**
-- **Line 29:** Changed "Vector similarity search (pgvector on TimescaleDB Cloud)" to "Vector similarity search (pgvector)"
-- **Line 127:** Removed "**TimescaleDB** extension (optional, for hypertables)" from Database Requirements section
-
-### 4. `lib/tasks/htm.rake`
-**Lines modified:** 67-73
-
-**Changes:**
-- Removed TimescaleDB version check from the `htm:db:test` rake task
-
-**Before:**
-```ruby
-# Check TimescaleDB
-timescale = conn.exec("SELECT extversion FROM pg_extension WHERE extname='timescaledb'").first
-if timescale
-  puts "  ✓ TimescaleDB version: #{timescale['extversion']}"
-else
-  puts "  ⚠ Warning: TimescaleDB extension not found"
-end
-
-# Check pgvector
-```
-
-**After:**
-```ruby
-# Check pgvector
-```
-
-## Other Potentially Impacted Files
-
-A codebase-wide search revealed **114 total files** containing references to "TimescaleDB", "timescaledb", or "hypertable". These fall into the following categories:
-
-### Documentation Files
-- `README.md` - Main project documentation
-- `CLAUDE.md` - AI assistant context documentation
-- `.architecture/` directory - Architecture Decision Records (ADRs) and reviews
-- `dbdoc/` directory - Auto-generated database documentation (120+ files)
-
-### Test Files
-- `test/` directory - Unit and integration tests may reference TimescaleDB in comments or mock data
-
-### Example Files
-- `examples/` directory - Example code may mention TimescaleDB in documentation
-
-### Migration Files
-- `db/migrate/` directory - Migration files may have comments referencing TimescaleDB optimization
-
-## Recommended Follow-up Actions
-
-### High Priority
-1. **Update README.md** - Remove TimescaleDB from installation instructions and feature descriptions
-2. **Update CLAUDE.md** - Remove TimescaleDB references from project overview and architecture descriptions
-3. **Review ADRs** - Update or create new ADR documenting the decision to remove TimescaleDB
-
-### Medium Priority
-4. **Update test files** - Remove TimescaleDB references from test comments and documentation
-5. **Update example code** - Remove TimescaleDB mentions from example documentation
-6. **Regenerate dbdoc/** - Run `tbls` again to regenerate database documentation without TimescaleDB references
-
-### Low Priority
-7. **Update migration comments** - Clean up any comments in migration files that reference TimescaleDB optimization
-8. **Review dependencies** - Verify that `Gemfile` or gemspec doesn't list TimescaleDB as a requirement (not found in initial search)
-
-## Benefits of Removal
-
-1. **Simplified deployment** - Users no longer need TimescaleDB-enabled PostgreSQL instances
-2. **Reduced complexity** - One less extension to manage and verify
-3. **Broader compatibility** - Works with any PostgreSQL 12+ installation (not just TimescaleDB Cloud or self-hosted TimescaleDB)
-4. **Clearer documentation** - Removes confusion about TimescaleDB's role (since it wasn't actually used)
-5. **Honest architecture** - Codebase now accurately reflects what it actually uses
-
-## No Loss of Functionality
-
-Removing TimescaleDB results in **zero loss of functionality** because:
-
-- No hypertables were created
-- No compression policies were used
-- Time-range queries already use standard B-tree indexes
-- All existing queries continue to work identically
-- Performance characteristics remain unchanged
-
-## Database Requirements After Removal
-
-The HTM gem now requires:
-
-- **PostgreSQL** 12+
-- **vector** extension (pgvector) - for embedding similarity search
-- **pg_trgm** extension - for fuzzy text matching
-
-No TimescaleDB required.
-
-## Testing Verification
-
-After these changes, verify:
-
-1. **Database setup works:**
-   ```bash
-   rake htm:db:setup
-   ```
-
-2. **Database connection test works:**
-   ```bash
-   rake htm:db:test
-   ```
-
-3. **All tests pass:**
-   ```bash
-   rake test
-   ```
-
-4. **Example code runs:**
-   ```bash
-   rake example
-   ```
-
-## Git Commit Message Suggestion
-
-```
-refactor!: remove TimescaleDB extension dependency
-
-BREAKING CHANGE: TimescaleDB is no longer required or checked for.
-
-TimescaleDB was originally included for time-series optimization
-but was never actually used (no hypertables were created, no
-compression policies configured). Time-range queries use standard
-PostgreSQL B-tree indexes on timestamp columns.
-
-This change:
-- Removes TimescaleDB from required extensions check
-- Removes verify_extensions and setup_hypertables methods
-- Updates documentation to reflect PostgreSQL-only requirements
-- Simplifies deployment by removing unnecessary dependency
-
-No functionality is lost as TimescaleDB features were not being used.
-
-Modified files:
-- lib/htm/active_record_config.rb
-- lib/htm/database.rb
-- db/README.md
-- lib/tasks/htm.rake
-```
-
-## Conclusion
-
-The removal of TimescaleDB from the HTM gem is a **low-risk refactoring** that simplifies the architecture and deployment requirements without any loss of functionality. All code changes have been completed in the core library files, with follow-up documentation updates recommended.