vectra-client 0.3.0 → 0.3.1

data/docs/guides/runbooks/pool-exhausted.md

@@ -0,0 +1,216 @@
+---
+layout: page
+title: "Runbook: Pool Exhaustion"
+permalink: /guides/runbooks/pool-exhausted/
+---
+
+# Runbook: Pool Exhaustion
+
+**Alert:** `VectraPoolExhausted`  
+**Severity:** Critical  
+**Threshold:** 0 available connections for 1 minute
+
+## Symptoms
+
+- `Vectra::Pool::TimeoutError` exceptions
+- Requests timing out waiting for connections
+- Application threads blocked
+
+## Quick Diagnosis
+
+```ruby
+# Check pool stats
+client = Vectra::Client.new(provider: :pgvector, host: ENV['DATABASE_URL'])
+puts client.provider.pool_stats
+# => { available: 0, checked_out: 10, size: 10 }
+```
+
+```bash
+# Check PostgreSQL connections
+psql -c "SELECT count(*) FROM pg_stat_activity WHERE application_name LIKE '%vectra%';"
+```
+
+## Investigation Steps
+
+### 1. Check Current Pool State
+
+```ruby
+stats = client.provider.pool_stats
+puts "Available: #{stats[:available]}"
+puts "Checked out: #{stats[:checked_out]}"
+puts "Total size: #{stats[:size]}"
+puts "Shutdown: #{stats[:shutdown]}"
+```
+
+### 2. Identify Connection Leaks
+
+```ruby
+# Look for connections not being returned
+# Common causes:
+# - Missing ensure blocks
+# - Exceptions before checkin
+# - Long-running operations
+
+# Bad:
+conn = pool.checkout
+do_something(conn)  # If this raises, connection is leaked!
+pool.checkin(conn)
+
+# Good:
+pool.with_connection do |conn|
+  do_something(conn)
+end  # Always returns connection
+```
+
+### 3. Check for Long-Running Queries
+
+```sql
+-- PostgreSQL: Find long-running queries
+SELECT pid, now() - pg_stat_activity.query_start AS duration, query
+FROM pg_stat_activity
+WHERE state != 'idle'
+AND query NOT LIKE '%pg_stat_activity%'
+ORDER BY duration DESC;
+
+-- Kill long-running query if needed
+SELECT pg_terminate_backend(pid);
+```
+
+### 4. Check Application Thread Count
+
+```ruby
+# If using Puma/Sidekiq
+# Ensure pool_size >= max_threads
+puts "Thread count: #{Thread.list.count}"
+puts "Pool size: #{client.config.pool_size}"
+```
+
+## Resolution Steps
+
+### Immediate: Restart Connection Pool
+
+```ruby
+# Force pool restart
+client.provider.shutdown_pool
+# Pool will be recreated on next operation
+```
+
+### Increase Pool Size
+
+```ruby
+Vectra.configure do |config|
+  config.provider = :pgvector
+  config.host = ENV['DATABASE_URL']
+  config.pool_size = 20      # Increase from default 5
+  config.pool_timeout = 10   # Increase timeout
+end
+```
+
+### Fix Connection Leaks
+
+```ruby
+# Always use with_connection block
+client.provider.with_pooled_connection do |conn|
+  # Your code here
+  # Connection automatically returned
+end
+
+# Or ensure checkin in rescue
+begin
+  conn = pool.checkout
+  do_work(conn)
+ensure
+  pool.checkin(conn) if conn
+end
+```
+
+### Reduce Connection Hold Time
+
+```ruby
+# Break up long operations
+large_dataset.each_slice(100) do |batch|
+  client.provider.with_pooled_connection do |conn|
+    process_batch(batch, conn)
+  end
+  # Connection returned between batches
+end
+```
+
+### Add Connection Warmup
+
+```ruby
+# In application initializer
+client = Vectra::Client.new(provider: :pgvector, host: ENV['DATABASE_URL'])
+client.provider.warmup_pool(5)  # Pre-create 5 connections
+```
+
+## Prevention
+
+### 1. Right-size Pool
+
+```ruby
+# Formula: pool_size = (max_threads * 1.5) + background_workers
+# Example: Puma with 5 threads, 3 Sidekiq workers
+pool_size = (5 * 1.5) + 3  # = 10.5, round to 12
+```
+
+### 2. Monitor Pool Usage
+
+```promql
+# Alert when pool is >80% utilized
+vectra_pool_connections{state="checked_out"} 
+/ vectra_pool_connections{state="available"} > 0.8
+```
+
+### 3. Implement Connection Timeout
+
+```ruby
+Vectra.configure do |config|
+  config.pool_timeout = 5  # Fail fast instead of hanging
+end
+```
+
+### 4. Use Connection Pool Metrics
+
+```ruby
+# Log pool stats periodically
+every(60.seconds) do
+  stats = client.provider.pool_stats
+  logger.info "Pool: avail=#{stats[:available]} out=#{stats[:checked_out]}"
+end
+```
+
+## PostgreSQL-Specific
+
+### Check max_connections
+
+```sql
+SHOW max_connections;  -- Default: 100
+
+-- Increase if needed (requires restart)
+ALTER SYSTEM SET max_connections = 200;
+```
+
+### Monitor Connection Usage
+
+```sql
+SELECT 
+  count(*) as total,
+  count(*) FILTER (WHERE state = 'active') as active,
+  count(*) FILTER (WHERE state = 'idle') as idle
+FROM pg_stat_activity;
+```
+
+## Escalation
+
+| Time | Action |
+|------|--------|
+| 1 min | Restart pool, page on-call |
+| 5 min | Increase pool size, restart app |
+| 15 min | Check for connection leaks |
+| 30 min | Escalate to DBA |
+
+## Related
+
+- [High Error Rate Runbook]({{ site.baseurl }}/guides/runbooks/high-error-rate)
+- [Performance Guide]({{ site.baseurl }}/guides/performance)

data/lib/vectra/circuit_breaker.rb

@@ -0,0 +1,336 @@
+# frozen_string_literal: true
+
+module Vectra
+  # Circuit Breaker pattern for handling provider failures
+  #
+  # Prevents cascading failures by temporarily stopping requests to a failing provider.
+  # The circuit has three states:
+  # - :closed - Normal operation, requests pass through
+  # - :open - Requests fail immediately without calling provider
+  # - :half_open - Limited requests allowed to test if provider recovered
+  #
+  # @example Basic usage
+  #   breaker = Vectra::CircuitBreaker.new(
+  #     failure_threshold: 5,
+  #     recovery_timeout: 30
+  #   )
+  #
+  #   breaker.call do
+  #     client.query(index: "my-index", vector: vec, top_k: 10)
+  #   end
+  #
+  # @example With fallback
+  #   breaker.call(fallback: -> { cached_results }) do
+  #     client.query(...)
+  #   end
+  #
+  # @example Per-provider circuit breakers
+  #   breakers = {
+  #     pinecone: Vectra::CircuitBreaker.new(name: "pinecone"),
+  #     qdrant: Vectra::CircuitBreaker.new(name: "qdrant")
+  #   }
+  #
+  class CircuitBreaker
+    STATES = [:closed, :open, :half_open].freeze
+
+    # Error raised when circuit is open
+    class OpenCircuitError < Vectra::Error
+      attr_reader :circuit_name, :failures, :opened_at
+
+      def initialize(circuit_name:, failures:, opened_at:)
+        @circuit_name = circuit_name
+        @failures = failures
+        @opened_at = opened_at
+        super("Circuit '#{circuit_name}' is open after #{failures} failures")
+      end
+    end
+
+    attr_reader :name, :state, :failure_count, :success_count,
+                :last_failure_at, :opened_at
+
+    # Initialize a new circuit breaker
+    #
+    # @param name [String] Circuit name for logging/metrics
+    # @param failure_threshold [Integer] Failures before opening circuit (default: 5)
+    # @param success_threshold [Integer] Successes in half-open to close (default: 3)
+    # @param recovery_timeout [Integer] Seconds before trying half-open (default: 30)
+    # @param monitored_errors [Array<Class>] Errors that count as failures
+    def initialize(
+      name: "default",
+      failure_threshold: 5,
+      success_threshold: 3,
+      recovery_timeout: 30,
+      monitored_errors: nil
+    )
+      @name = name
+      @failure_threshold = failure_threshold
+      @success_threshold = success_threshold
+      @recovery_timeout = recovery_timeout
+      @monitored_errors = monitored_errors || default_monitored_errors
+
+      @state = :closed
+      @failure_count = 0
+      @success_count = 0
+      @last_failure_at = nil
+      @opened_at = nil
+      @mutex = Mutex.new
+    end
+
+    # Execute block through circuit breaker
+    #
+    # @param fallback [Proc, nil] Fallback to call when circuit is open
+    # @yield The operation to execute
+    # @return [Object] Result of block or fallback
+    # @raise [OpenCircuitError] If circuit is open and no fallback provided
+    def call(fallback: nil, &)
+      check_state!
+
+      if open?
+        return handle_open_circuit(fallback)
+      end
+
+      execute_with_monitoring(&)
+    rescue *@monitored_errors => e
+      record_failure(e)
+      raise
+    end
+
+    # Force circuit to closed state (manual reset)
+    #
+    # @return [void]
+    def reset!
+      @mutex.synchronize do
+        transition_to(:closed)
+        @failure_count = 0
+        @success_count = 0
+        @last_failure_at = nil
+        @opened_at = nil
+      end
+    end
+
+    # Force circuit to open state (manual trip)
+    #
+    # @return [void]
+    def trip!
+      @mutex.synchronize do
+        transition_to(:open)
+        @opened_at = Time.now
+      end
+    end
+
+    # Check if circuit is closed (normal operation)
+    #
+    # @return [Boolean]
+    def closed?
+      state == :closed
+    end
+
+    # Check if circuit is open (blocking requests)
+    #
+    # @return [Boolean]
+    def open?
+      state == :open
+    end
+
+    # Check if circuit is half-open (testing recovery)
+    #
+    # @return [Boolean]
+    def half_open?
+      state == :half_open
+    end
+
+    # Get circuit statistics
+    #
+    # @return [Hash]
+    def stats
+      {
+        name: name,
+        state: state,
+        failure_count: failure_count,
+        success_count: success_count,
+        failure_threshold: @failure_threshold,
+        success_threshold: @success_threshold,
+        recovery_timeout: @recovery_timeout,
+        last_failure_at: last_failure_at,
+        opened_at: opened_at
+      }
+    end
+
+    private
+
+    def default_monitored_errors
+      [
+        Vectra::ServerError,
+        Vectra::ConnectionError,
+        Vectra::TimeoutError
+      ]
+    end
+
+    def check_state!
+      @mutex.synchronize do
+        # Check if we should transition from open to half-open
+        if open? && recovery_timeout_elapsed?
+          transition_to(:half_open)
+          @success_count = 0
+        end
+      end
+    end
+
+    def recovery_timeout_elapsed?
+      return false unless opened_at
+
+      Time.now - opened_at >= @recovery_timeout
+    end
+
+    def handle_open_circuit(fallback)
+      if fallback
+        log_fallback
+        fallback.call
+      else
+        raise OpenCircuitError.new(
+          circuit_name: name,
+          failures: failure_count,
+          opened_at: opened_at
+        )
+      end
+    end
+
+    def execute_with_monitoring
+      result = yield
+      record_success
+      result
+    end
+
+    def record_success
+      @mutex.synchronize do
+        @success_count += 1
+
+        # In half-open, check if we should close
+        if half_open? && @success_count >= @success_threshold
+          transition_to(:closed)
+          @failure_count = 0
+          log_circuit_closed
+        end
+      end
+    end
+
+    def record_failure(error)
+      @mutex.synchronize do
+        @failure_count += 1
+        @last_failure_at = Time.now
+
+        # In half-open, immediately open again
+        if half_open?
+          transition_to(:open)
+          @opened_at = Time.now
+          log_circuit_reopened(error)
+          return
+        end
+
+        # In closed, check threshold
+        if closed? && @failure_count >= @failure_threshold
+          transition_to(:open)
+          @opened_at = Time.now
+          log_circuit_opened(error)
+        end
+      end
+    end
+
+    def transition_to(new_state)
+      @state = new_state
+    end
+
+    def log_circuit_opened(error)
+      logger&.error(
+        "[Vectra::CircuitBreaker] Circuit '#{name}' opened after #{failure_count} failures. " \
+        "Last error: #{error.class} - #{error.message}"
+      )
+    end
+
+    def log_circuit_closed
+      logger&.info(
+        "[Vectra::CircuitBreaker] Circuit '#{name}' closed after #{success_count} successes"
+      )
+    end
+
+    def log_circuit_reopened(error)
+      logger&.warn(
+        "[Vectra::CircuitBreaker] Circuit '#{name}' reopened. " \
+        "Recovery failed: #{error.class} - #{error.message}"
+      )
+    end
+
+    def log_fallback
+      logger&.info(
+        "[Vectra::CircuitBreaker] Circuit '#{name}' open, using fallback"
+      )
+    end
+
+    def logger
+      Vectra.configuration.logger
+    end
+  end
+
+  # Circuit breaker registry for managing multiple circuits
+  #
+  # @example
+  #   Vectra::CircuitBreakerRegistry.register(:pinecone, failure_threshold: 3)
+  #   Vectra::CircuitBreakerRegistry.register(:qdrant, failure_threshold: 5)
+  #
+  #   Vectra::CircuitBreakerRegistry[:pinecone].call { ... }
+  #
+  module CircuitBreakerRegistry
+    class << self
+      # Get or create a circuit breaker
+      #
+      # @param name [Symbol, String] Circuit name
+      # @return [CircuitBreaker]
+      def [](name)
+        circuits[name.to_sym]
+      end
+
+      # Register a new circuit breaker
+      #
+      # @param name [Symbol, String] Circuit name
+      # @param options [Hash] CircuitBreaker options
+      # @return [CircuitBreaker]
+      def register(name, **options)
+        circuits[name.to_sym] = CircuitBreaker.new(name: name.to_s, **options)
+      end
+
+      # Get all registered circuits
+      #
+      # @return [Hash<Symbol, CircuitBreaker>]
+      def all
+        circuits.dup
+      end
+
+      # Reset all circuits
+      #
+      # @return [void]
+      def reset_all!
+        circuits.each_value(&:reset!)
+      end
+
+      # Get stats for all circuits
+      #
+      # @return [Hash<Symbol, Hash>]
+      def stats
+        circuits.transform_values(&:stats)
+      end
+
+      # Clear all registered circuits
+      #
+      # @return [void]
+      def clear!
+        @circuits = {}
+      end
+
+      private
+
+      def circuits
+        @circuits ||= {}
+      end
+    end
+  end
+end

data/lib/vectra/client.rb

@@ -25,6 +25,8 @@ module Vectra
   #   )
   #
   class Client
+    include HealthCheck
+
     attr_reader :config, :provider
 
     # Initialize a new Client