htm 0.0.18 → 0.0.30

data/docs/telemetry.md

@@ -1,391 +0,0 @@
-# Telemetry (OpenTelemetry Metrics)
-
-HTM includes optional OpenTelemetry-based metrics for production observability. This document provides detailed configuration and usage information.
-
-## Overview
-
-HTM uses [OpenTelemetry](https://opentelemetry.io/) for metrics collection, the industry-standard observability framework. This provides:
-
-- **Universal compatibility**: Works with 50+ observability backends
-- **Zero vendor lock-in**: Emit standard OTLP, route anywhere
-- **Zero overhead when disabled**: Null object pattern ensures no performance impact
-
-## Quick Start
-
-### 1. Install Dependencies
-
-Add the OpenTelemetry gems to your Gemfile:
-
-```ruby
-gem 'opentelemetry-sdk'
-gem 'opentelemetry-metrics-sdk'
-gem 'opentelemetry-exporter-otlp'  # For OTLP export
-```
-
-### 2. Enable Telemetry
-
-```ruby
-HTM.configure do |config|
-  config.telemetry_enabled = true
-end
-```
-
-Or via environment variable:
-
-```bash
-export HTM_TELEMETRY_ENABLED="true"
-```
-
-### 3. Configure Destination
-
-Set standard OpenTelemetry environment variables:
-
-```bash
-export OTEL_METRICS_EXPORTER="otlp"
-export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
-```
-
-## Configuration Options
-
-### HTM Configuration
-
-| Option | Type | Default | Description |
-|--------|------|---------|-------------|
-| `telemetry_enabled` | Boolean | `false` | Enable/disable telemetry |
-
-```ruby
-HTM.configure do |config|
-  config.telemetry_enabled = true
-end
-```
-
-### Environment Variables
-
-#### HTM-specific
-
-| Variable | Default | Description |
-|----------|---------|-------------|
-| `HTM_TELEMETRY_ENABLED` | `false` | Enable telemetry (`true`/`false`) |
-
-#### OpenTelemetry Standard
-
-These are standard OpenTelemetry environment variables (not HTM-specific):
-
-| Variable | Example | Description |
-|----------|---------|-------------|
-| `OTEL_SERVICE_NAME` | `htm` | Service name in telemetry data |
-| `OTEL_METRICS_EXPORTER` | `otlp` | Metrics exporter type |
-| `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://localhost:4318` | OTLP collector endpoint |
-| `OTEL_EXPORTER_OTLP_HEADERS` | `Authorization=Bearer token` | Headers for OTLP requests |
-
-## Available Metrics
-
-### htm.jobs (Counter)
-
-Counts job executions by type and outcome.
-
-**Attributes:**
-- `job`: Job type (`embedding`, `tags`)
-- `status`: Outcome (`success`, `error`, `circuit_open`)
-
-**Example queries (PromQL):**
-```promql
-# Jobs per minute by type
-rate(htm_jobs_total[1m])
-
-# Error rate for embedding jobs
-rate(htm_jobs_total{job="embedding",status="error"}[5m])
-  / rate(htm_jobs_total{job="embedding"}[5m])
-```
-
-### htm.embedding.latency (Histogram)
-
-Measures embedding generation time in milliseconds.
-
-**Attributes:**
-- `provider`: LLM provider (`ollama`, `openai`, etc.)
-- `status`: Outcome (`success`, `error`)
-
-**Example queries (PromQL):**
-```promql
-# p95 embedding latency
-histogram_quantile(0.95, rate(htm_embedding_latency_bucket[5m]))
-
-# Average latency by provider
-rate(htm_embedding_latency_sum[5m]) / rate(htm_embedding_latency_count[5m])
-```
-
-### htm.tag.latency (Histogram)
-
-Measures tag extraction time in milliseconds.
-
-**Attributes:**
-- `provider`: LLM provider (`ollama`, `openai`, etc.)
-- `status`: Outcome (`success`, `error`)
-
-**Example queries (PromQL):**
-```promql
-# p99 tag extraction latency
-histogram_quantile(0.99, rate(htm_tag_latency_bucket[5m]))
-```
-
-### htm.search.latency (Histogram)
-
-Measures search operation time in milliseconds.
-
-**Attributes:**
-- `strategy`: Search strategy (`vector`, `fulltext`, `hybrid`)
-
-**Example queries (PromQL):**
-```promql
-# Search latency by strategy
-histogram_quantile(0.95, rate(htm_search_latency_bucket[5m])) by (strategy)
-
-# Hybrid search avg latency
-rate(htm_search_latency_sum{strategy="hybrid"}[5m])
-  / rate(htm_search_latency_count{strategy="hybrid"}[5m])
-```
-
-### htm.cache.operations (Counter)
-
-Counts cache hits and misses.
-
-**Attributes:**
-- `operation`: Operation type (`hit`, `miss`)
-
-**Example queries (PromQL):**
-```promql
-# Cache hit rate
-rate(htm_cache_operations_total{operation="hit"}[5m])
-  / (rate(htm_cache_operations_total{operation="hit"}[5m])
-     + rate(htm_cache_operations_total{operation="miss"}[5m]))
-```
-
-## Compatible Backends
-
-HTM metrics work with any OTLP-compatible platform:
-
-### Open Source / Self-Hosted
-
-| Platform | OTLP Support | Notes |
-|----------|--------------|-------|
-| **Jaeger** | Native (v1.35+) | Distributed tracing with metrics |
-| **Prometheus** | Via receiver | Popular metrics platform |
-| **Grafana Tempo** | Native | Traces with metrics correlation |
-| **Grafana Mimir** | Native | Scalable Prometheus-compatible |
-| **SigNoz** | Native | Full-stack observability |
-| **Uptrace** | Native | Open source APM |
-
-### Commercial / SaaS
-
-| Platform | OTLP Support | Notes |
-|----------|--------------|-------|
-| **Datadog** | Native | Full-stack observability |
-| **New Relic** | Native | APM and infrastructure |
-| **Honeycomb** | Native (preferred) | High-cardinality observability |
-| **Splunk APM** | Native | Enterprise observability |
-| **Dynatrace** | Native | AI-powered monitoring |
-| **AWS X-Ray** | Via ADOT | AWS native tracing |
-| **Google Cloud Trace** | Native | GCP native tracing |
-| **Azure Monitor** | Native | Azure native monitoring |
-| **Grafana Cloud** | Native | Managed Grafana stack |
-
-## Setup Examples
-
-### Jaeger (Local Development)
-
-```bash
-# Start Jaeger with OTLP support
-docker run -d --name jaeger \
-  -p 4318:4318 \
-  -p 16686:16686 \
-  jaegertracing/all-in-one:latest
-
-# Configure HTM
-export HTM_TELEMETRY_ENABLED="true"
-export OTEL_METRICS_EXPORTER="otlp"
-export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
-```
-
-### Prometheus (via OpenTelemetry Collector)
-
-```yaml
-# otel-collector-config.yaml
-receivers:
-  otlp:
-    protocols:
-      grpc:
-      http:
-
-exporters:
-  prometheus:
-    endpoint: "0.0.0.0:8889"
-
-service:
-  pipelines:
-    metrics:
-      receivers: [otlp]
-      exporters: [prometheus]
-```
-
-```bash
-docker run -d \
-  -p 4317:4317 \
-  -p 4318:4318 \
-  -p 8889:8889 \
-  -v $(pwd)/otel-collector-config.yaml:/etc/otel-collector-config.yaml \
-  otel/opentelemetry-collector:latest \
-  --config=/etc/otel-collector-config.yaml
-```
-
-### Datadog
-
-```bash
-export HTM_TELEMETRY_ENABLED="true"
-export OTEL_METRICS_EXPORTER="otlp"
-export OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp.datadoghq.com"
-export OTEL_EXPORTER_OTLP_HEADERS="DD-API-KEY=your-api-key"
-```
-
-### Honeycomb
-
-```bash
-export HTM_TELEMETRY_ENABLED="true"
-export OTEL_METRICS_EXPORTER="otlp"
-export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.honeycomb.io"
-export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key"
-```
-
-### New Relic
-
-```bash
-export HTM_TELEMETRY_ENABLED="true"
-export OTEL_METRICS_EXPORTER="otlp"
-export OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp.nr-data.net"
-export OTEL_EXPORTER_OTLP_HEADERS="api-key=your-license-key"
-```
-
-## Architecture
-
-### Null Object Pattern
-
-HTM uses a null object pattern for telemetry. When disabled:
-
-```ruby
-# This code runs identically whether telemetry is enabled or not
-HTM::Telemetry.job_counter.add(1, attributes: { 'job' => 'embedding' })
-```
-
-When telemetry is disabled or the SDK is not installed:
-- `HTM::Telemetry.meter` returns a `NullMeter`
-- All instruments return `NullInstrument` instances
-- All operations (`add`, `record`) are no-ops returning `nil`
-- Zero memory allocation, zero CPU overhead
-
-### Code Flow
-
-![HTM Telemetry Architecture](images/telemetry-architecture.svg)
-
-### Instrumentation Points
-
-HTM instruments these key operations:
-
-| Component | Metrics Recorded |
-|-----------|-----------------|
-| `GenerateEmbeddingJob` | `htm.jobs`, `htm.embedding.latency` |
-| `GenerateTagsJob` | `htm.jobs`, `htm.tag.latency` |
-| `VectorSearch#search` | `htm.search.latency` (strategy: vector) |
-| `FulltextSearch#search_fulltext` | `htm.search.latency` (strategy: fulltext) |
-| `HybridSearch#search_hybrid` | `htm.search.latency` (strategy: hybrid) |
-| `QueryCache#fetch` | `htm.cache.operations` |
-
-## Testing
-
-When writing tests, reset telemetry state between tests:
-
-```ruby
-def setup
-  HTM::Telemetry.reset!
-  HTM.configuration.telemetry_enabled = false
-end
-
-def teardown
-  HTM::Telemetry.reset!
-end
-```
-
-### Verifying Metrics in Tests
-
-```ruby
-def test_records_embedding_latency
-  # Create a mock histogram
-  recorded = []
-  mock_histogram = Object.new
-  mock_histogram.define_singleton_method(:record) do |value, **kwargs|
-    recorded << { value: value, attributes: kwargs[:attributes] }
-  end
-
-  # Inject mock (requires test helper)
-  HTM::Telemetry.instance_variable_set(:@embedding_latency, mock_histogram)
-
-  # Run operation
-  HTM::Jobs::GenerateEmbeddingJob.perform(node_id: node.id)
-
-  # Verify
-  assert recorded.any? { |r| r[:attributes]['status'] == 'success' }
-end
-```
-
-## Troubleshooting
-
-### Metrics Not Appearing
-
-1. **Check telemetry is enabled:**
-   ```ruby
-   puts HTM.configuration.telemetry_enabled  # Should be true
-   puts HTM::Telemetry.enabled?              # Should be true
-   ```
-
-2. **Check SDK is installed:**
-   ```ruby
-   puts HTM::Telemetry.sdk_available?  # Should be true
-   ```
-
-3. **Check environment variables:**
-   ```bash
-   echo $OTEL_METRICS_EXPORTER
-   echo $OTEL_EXPORTER_OTLP_ENDPOINT
-   ```
-
-4. **Check collector is running:**
-   ```bash
-   curl http://localhost:4318/v1/metrics  # Should not 404
-   ```
-
-### High Cardinality Issues
-
-Avoid high-cardinality attributes in metrics. HTM uses low-cardinality attributes by design:
-
-- `job`: 2-3 values (embedding, tags, propositions)
-- `status`: 3 values (success, error, circuit_open)
-- `provider`: ~10 values (ollama, openai, etc.)
-- `strategy`: 3 values (vector, fulltext, hybrid)
-- `operation`: 2 values (hit, miss)
-
-### Performance Concerns
-
-With telemetry disabled (default), there is zero overhead:
-- No gem loading
-- No object allocation
-- No method calls beyond the null check
-
-With telemetry enabled, overhead is minimal:
-- ~1-5μs per metric recording
-- Memory for histogram buckets (~1KB per histogram)
-
-## References
-
-- [OpenTelemetry Ruby SDK](https://github.com/open-telemetry/opentelemetry-ruby)
-- [OpenTelemetry Metrics SDK](https://github.com/open-telemetry/opentelemetry-ruby/tree/main/metrics_sdk)
-- [OpenTelemetry Specification](https://opentelemetry.io/docs/specs/otel/)
-- [OTLP Protocol](https://opentelemetry.io/docs/specs/otlp/)

data/examples/rails_app/app/controllers/tags_controller.rb

@@ -1,30 +0,0 @@
-# frozen_string_literal: true
-
-class TagsController < ApplicationController
-  def index
-    @view_type = params[:view] || 'list'
-
-    case @view_type
-    when 'tree'
-      @tree_string = HTM::Models::Tag.all.tree_string
-      @tree_svg = HTM::Models::Tag.all.tree_svg(title: 'HTM Tag Hierarchy')
-    when 'mermaid'
-      @tree_mermaid = HTM::Models::Tag.all.tree_mermaid
-    else
-      @tags = HTM::Models::Tag
-              .left_joins(:nodes)
-              .group('tags.id')
-              .select('tags.*, COUNT(nodes.id) as node_count')
-              .order(:name)
-
-      if params[:prefix].present?
-        @tags = @tags.where('tags.name LIKE ?', "#{params[:prefix]}%")
-      end
-    end
-  end
-
-  def show
-    @tag = HTM::Models::Tag.find(params[:id])
-    @memories = @tag.nodes.active.includes(:tags).order(created_at: :desc)
-  end
-end

data/examples/sinatra_app/Gemfile.lock

@@ -1,166 +0,0 @@
-PATH
-  remote: ../..
-  specs:
-    htm (0.0.2)
-      activerecord
-      lru_redux
-      neighbor
-      pg (>= 1.5.0)
-      ruby_llm
-      tiktoken_ruby
-
-GEM
-  remote: https://rubygems.org/
-  specs:
-    activemodel (8.1.1)
-      activesupport (= 8.1.1)
-    activerecord (8.1.1)
-      activemodel (= 8.1.1)
-      activesupport (= 8.1.1)
-      timeout (>= 0.4.0)
-    activesupport (8.1.1)
-      base64
-      bigdecimal
-      concurrent-ruby (~> 1.0, >= 1.3.1)
-      connection_pool (>= 2.2.5)
-      drb
-      i18n (>= 1.6, < 2)
-      json
-      logger (>= 1.4.2)
-      minitest (>= 5.1)
-      securerandom (>= 0.3)
-      tzinfo (~> 2.0, >= 2.0.5)
-      uri (>= 0.13.1)
-    base64 (0.3.0)
-    bigdecimal (3.3.1)
-    concurrent-ruby (1.3.5)
-    connection_pool (2.5.5)
-    drb (2.2.3)
-    event_stream_parser (1.0.0)
-    faraday (2.14.0)
-      faraday-net_http (>= 2.0, < 3.5)
-      json
-      logger
-    faraday-multipart (1.1.1)
-      multipart-post (~> 2.0)
-    faraday-net_http (3.4.2)
-      net-http (~> 0.5)
-    faraday-retry (2.3.2)
-      faraday (~> 2.0)
-    ffi (1.17.2-aarch64-linux-gnu)
-    ffi (1.17.2-aarch64-linux-musl)
-    ffi (1.17.2-arm-linux-gnu)
-    ffi (1.17.2-arm-linux-musl)
-    ffi (1.17.2-arm64-darwin)
-    ffi (1.17.2-x86_64-darwin)
-    ffi (1.17.2-x86_64-linux-gnu)
-    ffi (1.17.2-x86_64-linux-musl)
-    i18n (1.14.7)
-      concurrent-ruby (~> 1.0)
-    json (2.16.0)
-    listen (3.9.0)
-      rb-fsevent (~> 0.10, >= 0.10.3)
-      rb-inotify (~> 0.9, >= 0.9.10)
-    logger (1.7.0)
-    lru_redux (1.1.0)
-    marcel (1.1.0)
-    minitest (5.26.2)
-    multi_json (1.17.0)
-    multipart-post (2.4.1)
-    mustermann (3.0.4)
-      ruby2_keywords (~> 0.0.1)
-    neighbor (0.6.0)
-      activerecord (>= 7.1)
-    net-http (0.8.0)
-      uri (>= 0.11.1)
-    nio4r (2.7.5)
-    pg (1.6.2)
-    pg (1.6.2-aarch64-linux)
-    pg (1.6.2-aarch64-linux-musl)
-    pg (1.6.2-arm64-darwin)
-    pg (1.6.2-x86_64-darwin)
-    pg (1.6.2-x86_64-linux)
-    pg (1.6.2-x86_64-linux-musl)
-    puma (6.6.1)
-      nio4r (~> 2.0)
-    rack (2.2.21)
-    rack-protection (3.2.0)
-      base64 (>= 0.1.0)
-      rack (~> 2.2, >= 2.2.4)
-    rb-fsevent (0.11.2)
-    rb-inotify (0.11.1)
-      ffi (~> 1.0)
-    redis (5.4.1)
-      redis-client (>= 0.22.0)
-    redis-client (0.26.1)
-      connection_pool
-    rerun (0.14.0)
-      listen (~> 3.0)
-    ruby2_keywords (0.0.5)
-    ruby_llm (1.9.1)
-      base64
-      event_stream_parser (~> 1)
-      faraday (>= 1.10.0)
-      faraday-multipart (>= 1)
-      faraday-net_http (>= 1)
-      faraday-retry (>= 1)
-      marcel (~> 1.0)
-      ruby_llm-schema (~> 0.2.1)
-      zeitwerk (~> 2)
-    ruby_llm-schema (0.2.5)
-    securerandom (0.4.1)
-    sidekiq (7.3.9)
-      base64
-      connection_pool (>= 2.3.0)
-      logger
-      rack (>= 2.2.4)
-      redis-client (>= 0.22.2)
-    sinatra (3.2.0)
-      mustermann (~> 3.0)
-      rack (~> 2.2, >= 2.2.4)
-      rack-protection (= 3.2.0)
-      tilt (~> 2.0)
-    sinatra-contrib (3.2.0)
-      multi_json (>= 0.0.2)
-      mustermann (~> 3.0)
-      rack-protection (= 3.2.0)
-      sinatra (= 3.2.0)
-      tilt (~> 2.0)
-    tiktoken_ruby (0.0.13-aarch64-linux)
-    tiktoken_ruby (0.0.13-arm-linux)
-    tiktoken_ruby (0.0.13-arm64-darwin)
-    tiktoken_ruby (0.0.13-x86_64-darwin)
-    tiktoken_ruby (0.0.13-x86_64-linux)
-    tiktoken_ruby (0.0.13-x86_64-linux-musl)
-    tilt (2.6.1)
-    timeout (0.4.4)
-    tzinfo (2.0.6)
-      concurrent-ruby (~> 1.0)
-    uri (1.1.1)
-    zeitwerk (2.7.3)
-
-PLATFORMS
-  aarch64-linux
-  aarch64-linux-gnu
-  aarch64-linux-musl
-  arm-linux
-  arm-linux-gnu
-  arm-linux-musl
-  arm64-darwin
-  x86_64-darwin
-  x86_64-linux
-  x86_64-linux-gnu
-  x86_64-linux-musl
-
-DEPENDENCIES
-  htm!
-  multi_json
-  puma (~> 6.0)
-  redis (~> 5.0)
-  rerun
-  sidekiq (~> 7.0)
-  sinatra (~> 3.0)
-  sinatra-contrib (~> 3.0)
-
-BUNDLED WITH
-   2.7.2

data/lib/htm/active_record_config.rb

@@ -1,104 +0,0 @@
-# frozen_string_literal: true
-
-require 'active_record'
-require 'pg'
-require 'neighbor'
-
-class HTM
-  # ActiveRecord database configuration and model loading
-  #
-  # Uses HTM::Config for database settings. Configuration can come from:
-  # - Environment variables (HTM_DATABASE__URL, HTM_DATABASE__HOST, etc.)
-  # - Programmatic configuration via HTM.configure
-  #
-  class ActiveRecordConfig
-    class << self
-      # Establish database connection from HTM::Config
-      def establish_connection!
-        config = load_database_config
-
-        ActiveRecord::Base.establish_connection(config)
-
-        # Set search path
-        ActiveRecord::Base.connection.execute("SET search_path TO public")
-
-        # Load models after connection is established
-        require_models
-
-        true
-      end
-
-      # Load database configuration from HTM::Config
-      #
-      # @return [Hash] ActiveRecord-compatible configuration hash
-      #
-      def load_database_config
-        HTM.config.database_config
-      end
-
-      # Check if connection is established and active
-      def connected?
-        return false unless defined?(ActiveRecord::Base)
-        return false unless ActiveRecord::Base.connection_handler.connection_pool_list.any?
-
-        ActiveRecord::Base.connected? && ActiveRecord::Base.connection.active?
-      rescue ActiveRecord::ConnectionNotDefined, ActiveRecord::ConnectionNotEstablished
-        false
-      rescue StandardError
-        false
-      end
-
-      # Close all database connections
-      def disconnect!
-        ActiveRecord::Base.connection_pool.disconnect!
-      end
-
-      # Verify required extensions are available
-      def verify_extensions!
-        conn = ActiveRecord::Base.connection
-
-        required_extensions = {
-          'vector' => 'pgvector extension',
-          'pg_trgm' => 'PostgreSQL trigram extension'
-        }
-
-        missing = []
-        required_extensions.each do |ext, name|
-          result = conn.select_value(
-            "SELECT COUNT(*) FROM pg_extension WHERE extname = '#{ext}'"
-          )
-          missing << name if result.to_i.zero?
-        end
-
-        if missing.any?
-          raise "Missing required PostgreSQL extensions: #{missing.join(', ')}"
-        end
-
-        true
-      end
-
-      # Get connection pool statistics
-      def connection_stats
-        pool = ActiveRecord::Base.connection_pool
-        {
-          size: pool.size,
-          connections: pool.connections.size,
-          in_use: pool.connections.count(&:in_use?),
-          available: pool.connections.count { |c| !c.in_use? }
-        }
-      end
-
-      private
-
-      # Require all model files
-      def require_models
-        require_relative 'models/robot'
-        require_relative 'models/node'
-        require_relative 'models/robot_node'
-        require_relative 'models/tag'
-        require_relative 'models/node_tag'
-        require_relative 'models/file_source'
-      end
-    end
-  end
-end