vectra-client 0.3.3 → 0.4.0

data/examples/grafana-setup.md

@@ -0,0 +1,340 @@
+# Grafana Dashboard Setup for Vectra
+
+Complete guide to set up a beautiful Grafana dashboard for monitoring Vectra vector database operations.
+
+## Prerequisites
+
+1. **Grafana Account** - Sign up at [grafana.com](https://grafana.com) or use self-hosted Grafana
+2. **Prometheus** - For metrics collection (or use Grafana Cloud's Prometheus)
+3. **Vectra with Instrumentation** - Enable metrics in your application
+
+## Quick Setup for Screenshots (3 minutes)
+
+**Perfect for creating dashboard screenshots!**
+
+### Option 1: Use Demo Exporter (Easiest)
+
+1. **Start Prometheus Exporter:**
+   ```bash
+   ruby examples/prometheus-exporter.rb
+   ```
+   This generates demo metrics automatically.
+
+2. **Setup Grafana Cloud (or local):**
+   - Sign up at [grafana.com](https://grafana.com) (free tier available)
+   - Create a Prometheus data source pointing to `http://localhost:9394`
+   - Or use Grafana Cloud's built-in Prometheus
+
+3. **Import Dashboard:**
+   - Go to Dashboards → Import
+   - Upload `examples/grafana-dashboard.json`
+   - Select your Prometheus data source
+   - Click "Import"
+
+4. **Take Screenshots:**
+   - Dashboard will populate with demo data
+   - Wait 1-2 minutes for metrics to accumulate
+   - Use Grafana's built-in screenshot feature or browser screenshot
+
+### Option 2: Full Production Setup
+
+## Full Setup (5 minutes)
+
+### Step 1: Enable Vectra Metrics
+
+Add to your Rails initializer or application:
+
+```ruby
+# config/initializers/vectra_metrics.rb
+require "prometheus/client"
+
+module VectraMetrics
+  REGISTRY = Prometheus::Client.registry
+
+  REQUESTS_TOTAL = REGISTRY.counter(
+    :vectra_requests_total,
+    docstring: "Total Vectra requests",
+    labels: [:provider, :operation, :status]
+  )
+
+  REQUEST_DURATION = REGISTRY.histogram(
+    :vectra_request_duration_seconds,
+    docstring: "Request duration in seconds",
+    labels: [:provider, :operation],
+    buckets: [0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
+  )
+
+  VECTORS_PROCESSED = REGISTRY.counter(
+    :vectra_vectors_processed_total,
+    docstring: "Total vectors processed",
+    labels: [:provider, :operation]
+  )
+
+  CACHE_HITS = REGISTRY.counter(
+    :vectra_cache_hits_total,
+    docstring: "Cache hit count"
+  )
+
+  CACHE_MISSES = REGISTRY.counter(
+    :vectra_cache_misses_total,
+    docstring: "Cache miss count"
+  )
+
+  ERRORS_TOTAL = REGISTRY.counter(
+    :vectra_errors_total,
+    docstring: "Total errors",
+    labels: [:provider, :error_type]
+  )
+
+  POOL_SIZE = REGISTRY.gauge(
+    :vectra_pool_connections,
+    docstring: "Connection pool size",
+    labels: [:state]
+  )
+end
+
+# Register instrumentation
+Vectra::Instrumentation.register(:prometheus) do |event|
+  labels = {
+    provider: event[:provider],
+    operation: event[:operation]
+  }
+
+  status = event[:error] ? "error" : "success"
+  VectraMetrics::REQUESTS_TOTAL.increment(labels: labels.merge(status: status))
+
+  if event[:duration]
+    VectraMetrics::REQUEST_DURATION.observe(event[:duration], labels: labels)
+  end
+
+  if event[:metadata]&.dig(:vector_count)
+    VectraMetrics::VECTORS_PROCESSED.increment(
+      by: event[:metadata][:vector_count],
+      labels: labels
+    )
+  end
+
+  if event[:error]
+    VectraMetrics::ERRORS_TOTAL.increment(
+      labels: labels.merge(error_type: event[:error].class.name)
+    )
+  end
+end
+```
+
+### Step 2: Expose Prometheus Endpoint
+
+For Rails, add to `config/routes.rb`:
+
+```ruby
+# config/routes.rb
+require "rack/prometheus"
+
+Rails.application.routes.draw do
+  # ... your routes ...
+  
+  # Prometheus metrics endpoint
+  get "/metrics", to: proc { |env|
+    [
+      200,
+      { "Content-Type" => "text/plain" },
+      [Prometheus::Client::Formats::Text.marshal(VectraMetrics::REGISTRY)]
+    ]
+  }
+end
+```
+
+Or use `rack-prometheus` gem:
+
+```ruby
+# Gemfile
+gem "rack-prometheus"
+
+# config/application.rb
+config.middleware.use Rack::Prometheus::Middleware
+```
+
+### Step 3: Configure Prometheus
+
+Create `prometheus.yml`:
+
+```yaml
+global:
+  scrape_interval: 15s
+  evaluation_interval: 15s
+
+scrape_configs:
+  - job_name: "vectra"
+    static_configs:
+      - targets: ["localhost:3000"]  # Your Rails app
+    metrics_path: "/metrics"
+```
+
+### Step 4: Import Dashboard to Grafana
+
+1. **Login to Grafana** (grafana.com or your instance)
+
+2. **Add Prometheus Data Source:**
+   - Go to Configuration → Data Sources
+   - Add Prometheus
+   - URL: `http://localhost:9090` (or your Prometheus URL)
+   - Click "Save & Test"
+
+3. **Import Dashboard:**
+   - Go to Dashboards → Import
+   - Click "Upload JSON file"
+   - Select `grafana-dashboard.json`
+   - Select your Prometheus data source
+   - Click "Import"
+
+4. **View Dashboard:**
+   - Dashboard will appear with all panels
+   - Run your Vectra demo to generate metrics
+   - Watch real-time data populate!
+
+## Dashboard Panels
+
+The dashboard includes 12 panels:
+
+### Top Row (Stats)
+1. **Total Requests** - Requests per second
+2. **Error Rate** - Error percentage with color coding
+3. **P95 Latency** - 95th percentile latency
+4. **Cache Hit Ratio** - Cache performance
+
+### Middle Row (Time Series)
+5. **Request Rate by Operation** - Query, upsert, delete, etc.
+6. **Latency Distribution** - P50, P95, P99 percentiles
+7. **Vectors Processed** - Throughput by operation
+8. **Errors by Type** - Error breakdown
+
+### Bottom Row (Visualizations)
+9. **Connection Pool Status** - pgvector pool metrics
+10. **Request Rate by Provider** - Bar chart by provider
+11. **Operations Distribution** - Pie chart
+12. **Provider Distribution** - Pie chart
+
+## Generating Demo Data
+
+Run the comprehensive demo to generate metrics:
+
+```bash
+# Run demo (generates metrics)
+bundle exec ruby examples/comprehensive_demo.rb
+
+# Or run your application
+rails server
+```
+
+## Screenshot Tips for Social Media
+
+### Best Panels for Screenshots
+
+1. **Request Rate by Operation** ⭐
+   - Shows activity over time
+   - Clean, professional look
+   - Perfect for Twitter/LinkedIn
+
+2. **Latency Distribution (P50, P95, P99)** ⭐
+   - Shows performance metrics
+   - Multiple lines = impressive
+   - Great for technical posts
+
+3. **Operations Distribution (Pie Chart)** ⭐
+   - Clean, colorful visualization
+   - Easy to understand
+   - Perfect for overview posts
+
+4. **Top Row Stats** ⭐
+   - 4 stat panels side-by-side
+   - Shows key metrics at a glance
+   - Great for hero images
+
+### Time Ranges for Screenshots
+
+- **Last 15 minutes** - Shows recent activity (best for demos)
+- **Last 1 hour** - Shows trends (good for posts)
+- **Last 6 hours** - Shows daily patterns (for analysis posts)
+
+### Grafana Screenshot Features
+
+1. **Built-in Screenshot:**
+   - Click panel → Share → Direct link rendered image
+   - Or use browser screenshot (Cmd+Shift+4 on Mac)
+
+2. **Full Dashboard Screenshot:**
+   - Use browser developer tools
+   - Or Grafana's export feature
+
+3. **Panel Screenshots:**
+   - Right-click panel → Inspect
+   - Use browser screenshot tool
+
+### Customization for Better Screenshots
+
+1. **Change Theme:**
+   - Settings → Preferences → Theme
+   - Dark theme looks more professional
+
+2. **Adjust Time Range:**
+   - Use "Last 15 minutes" for demo screenshots
+   - Shows active data
+
+3. **Hide Legend (if needed):**
+   - Panel → Options → Legend → Hide
+   - Cleaner look for some panels
+
+4. **Add Title/Description:**
+   - Panel → Title → Add description
+   - Makes screenshots self-explanatory
+
+## Troubleshooting
+
+### No Data Showing
+
+1. **Check Prometheus is scraping:**
+   ```bash
+   curl http://localhost:9090/api/v1/targets
+   ```
+
+2. **Check metrics endpoint:**
+   ```bash
+   curl http://localhost:3000/metrics | grep vectra
+   ```
+
+3. **Verify instrumentation:**
+   ```ruby
+   # In Rails console
+   Vectra::Instrumentation.enabled?  # Should be true
+   ```
+
+### Missing Metrics
+
+- Ensure `config.instrumentation = true` in Vectra config
+- Check Prometheus is scraping your app
+- Verify labels match dashboard queries
+
+## Advanced: Grafana Cloud Setup
+
+If using Grafana Cloud:
+
+1. **Create Prometheus data source:**
+   - Use Grafana Cloud's Prometheus URL
+   - Add API key for authentication
+
+2. **Push metrics:**
+   ```ruby
+   # Use prometheus/client_ruby with push gateway
+   require "prometheus/client/push"
+   
+   Prometheus::Client::Push.new(
+     job: "vectra",
+     gateway: "https://prometheus-us-central1.grafana.net"
+   ).add(VectraMetrics::REGISTRY)
+   ```
+
+## Next Steps
+
+- [Monitoring Guide](../docs/guides/monitoring.md) - Full monitoring setup
+- [Performance Guide](../docs/guides/performance.md) - Optimization tips
+- [Examples](../examples/) - More demo code

data/examples/prometheus-exporter.rb

@@ -0,0 +1,229 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+# Simple Prometheus Exporter for Vectra Demo
+#
+# This script exposes Prometheus metrics endpoint for Grafana dashboard.
+# Run this alongside your demo to generate metrics.
+#
+# Prerequisites:
+#   gem install prometheus-client
+#
+# Usage:
+#   ruby examples/prometheus-exporter.rb
+#   # Then run: bundle exec ruby examples/comprehensive_demo.rb
+#
+# Access metrics:
+#   curl http://localhost:9394/metrics
+
+begin
+  require "webrick"
+  require "prometheus/client"
+  require "json"
+rescue LoadError => e
+  if e.message.include?("prometheus")
+    puts "❌ Error: prometheus-client gem not found"
+    puts
+    puts "Install it with:"
+    puts "  gem install prometheus-client"
+    puts
+    puts "Or add to Gemfile:"
+    puts "  gem 'prometheus-client'"
+    exit 1
+  else
+    raise
+  end
+end
+
+# Create Prometheus registry
+REGISTRY = Prometheus::Client.registry
+
+# Define metrics
+REQUESTS_TOTAL = REGISTRY.counter(
+  :vectra_requests_total,
+  docstring: "Total Vectra requests",
+  labels: [:provider, :operation, :status]
+)
+
+REQUEST_DURATION = REGISTRY.histogram(
+  :vectra_request_duration_seconds,
+  docstring: "Request duration in seconds",
+  labels: [:provider, :operation],
+  buckets: [0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
+)
+
+VECTORS_PROCESSED = REGISTRY.counter(
+  :vectra_vectors_processed_total,
+  docstring: "Total vectors processed",
+  labels: [:provider, :operation]
+)
+
+CACHE_HITS = REGISTRY.counter(
+  :vectra_cache_hits_total,
+  docstring: "Cache hit count"
+)
+
+CACHE_MISSES = REGISTRY.counter(
+  :vectra_cache_misses_total,
+  docstring: "Cache miss count"
+)
+
+ERRORS_TOTAL = REGISTRY.counter(
+  :vectra_errors_total,
+  docstring: "Total errors",
+  labels: [:provider, :error_type]
+)
+
+POOL_SIZE = REGISTRY.gauge(
+  :vectra_pool_connections,
+  docstring: "Connection pool size",
+  labels: [:state]
+)
+
+# Simulate metrics for demo (in production, these come from Vectra)
+def generate_demo_metrics
+  providers = %w[pinecone qdrant pgvector]
+  operations = %w[query upsert fetch delete update]
+  
+  loop do
+    provider = providers.sample
+    operation = operations.sample
+    
+    # Simulate requests
+    REQUESTS_TOTAL.increment(
+      labels: {
+        provider: provider,
+        operation: operation,
+        status: rand > 0.05 ? "success" : "error"
+      }
+    )
+    
+    # Simulate latency
+    duration = case operation
+               when "query"
+                 rand(0.01..0.1)
+               when "upsert"
+                 rand(0.05..0.3)
+               else
+                 rand(0.01..0.2)
+               end
+    
+    REQUEST_DURATION.observe(
+      duration,
+      labels: { provider: provider, operation: operation }
+    )
+    
+    # Simulate vectors processed
+    if %w[upsert query].include?(operation)
+      VECTORS_PROCESSED.increment(
+        by: rand(1..100),
+        labels: { provider: provider, operation: operation }
+      )
+    end
+    
+    # Simulate cache hits/misses
+    if rand > 0.3
+      CACHE_HITS.increment
+    else
+      CACHE_MISSES.increment
+    end
+    
+    # Simulate errors
+    if rand < 0.05
+      ERRORS_TOTAL.increment(
+        labels: {
+          provider: provider,
+          error_type: %w[RateLimitError ServerError ConnectionError].sample
+        }
+      )
+    end
+    
+    # Simulate pool metrics (pgvector)
+    if provider == "pgvector"
+      POOL_SIZE.set(
+        rand(3..8),
+        labels: { state: "available" }
+      )
+      POOL_SIZE.set(
+        rand(1..3),
+        labels: { state: "checked_out" }
+      )
+    end
+    
+    sleep(rand(0.5..2.0))
+  end
+end
+
+# Start metrics generation in background
+Thread.new { generate_demo_metrics }
+
+# Create HTTP server
+server = WEBrick::HTTPServer.new(Port: 9394)
+
+server.mount_proc("/metrics") do |_req, res|
+  res.content_type = "text/plain; version=0.0.4"
+  res.body = Prometheus::Client::Formats::Text.marshal(REGISTRY)
+end
+
+server.mount_proc("/") do |_req, res|
+  res.content_type = "text/html"
+  res.body = <<~HTML
+    <!DOCTYPE html>
+    <html>
+    <head>
+      <title>Vectra Prometheus Exporter</title>
+      <style>
+        body { font-family: system-ui; max-width: 800px; margin: 50px auto; padding: 20px; }
+        h1 { color: #05df72; }
+        .status { background: #f0f0f0; padding: 15px; border-radius: 5px; margin: 20px 0; }
+        code { background: #f5f5f5; padding: 2px 6px; border-radius: 3px; }
+        a { color: #05df72; text-decoration: none; }
+        a:hover { text-decoration: underline; }
+      </style>
+    </head>
+    <body>
+      <h1>🚀 Vectra Prometheus Exporter</h1>
+      <div class="status">
+        <p><strong>Status:</strong> Running</p>
+        <p><strong>Metrics Endpoint:</strong> <a href="/metrics"><code>/metrics</code></a></p>
+        <p><strong>Port:</strong> 9394</p>
+      </div>
+      <h2>Available Metrics</h2>
+      <ul>
+        <li><code>vectra_requests_total</code> - Total requests by provider/operation</li>
+        <li><code>vectra_request_duration_seconds</code> - Request latency histogram</li>
+        <li><code>vectra_vectors_processed_total</code> - Vectors processed</li>
+        <li><code>vectra_cache_hits_total</code> - Cache hits</li>
+        <li><code>vectra_cache_misses_total</code> - Cache misses</li>
+        <li><code>vectra_errors_total</code> - Error counts</li>
+        <li><code>vectra_pool_connections</code> - Connection pool status</li>
+      </ul>
+      <h2>Next Steps</h2>
+      <ol>
+        <li>Configure Prometheus to scrape <code>http://localhost:9394/metrics</code></li>
+        <li>Import Grafana dashboard from <code>examples/grafana-dashboard.json</code></li>
+        <li>Run your Vectra demo to generate real metrics</li>
+      </ol>
+      <p><a href="/metrics">View Metrics →</a></p>
+    </body>
+    </html>
+  HTML
+end
+
+trap("INT") { server.shutdown }
+trap("TERM") { server.shutdown }
+
+puts "=" * 60
+puts "🚀 Vectra Prometheus Exporter"
+puts "=" * 60
+puts
+puts "📊 Metrics endpoint: http://localhost:9394/metrics"
+puts "🌐 Web interface: http://localhost:9394"
+puts
+puts "💡 This exporter generates demo metrics."
+puts "   In production, use Vectra instrumentation instead."
+puts
+puts "Press Ctrl+C to stop..."
+puts
+
+server.start

data/lib/vectra/batch.rb

@@ -17,6 +17,17 @@ module Vectra
   #   )
   #   puts "Upserted: #{result[:upserted_count]}"
   #
+  # @example With progress tracking
+  #   batch.upsert_async(
+  #     index: 'docs',
+  #     vectors: large_array,
+  #     on_progress: ->(stats) {
+  #       puts "Progress: #{stats[:percentage]}% (#{stats[:processed]}/#{stats[:total]})"
+  #       puts "  Chunk #{stats[:current_chunk] + 1}/#{stats[:total_chunks]}"
+  #       puts "  Success: #{stats[:success_count]}, Failed: #{stats[:failed_count]}"
+  #     }
+  #   )
+  #
   class Batch
     DEFAULT_CONCURRENCY = 4
     DEFAULT_CHUNK_SIZE = 100
@@ -38,12 +49,23 @@ module Vectra
     # @param vectors [Array<Hash>] vectors to upsert
     # @param namespace [String, nil] optional namespace
     # @param chunk_size [Integer] vectors per chunk (default: 100)
+    # @param on_progress [Proc, nil] optional callback called after each chunk completes
+    #   Callback receives hash with: processed, total, percentage, current_chunk, total_chunks, success_count, failed_count
     # @return [Hash] aggregated result with :upserted_count, :chunks, :errors
-    def upsert_async(index:, vectors:, namespace: nil, chunk_size: DEFAULT_CHUNK_SIZE)
+    #
+    # @example With progress callback
+    #   batch.upsert_async(
+    #     index: 'docs',
+    #     vectors: large_array,
+    #     on_progress: ->(stats) {
+    #       puts "Progress: #{stats[:percentage]}% (#{stats[:processed]}/#{stats[:total]})"
+    #     }
+    #   )
+    def upsert_async(index:, vectors:, namespace: nil, chunk_size: DEFAULT_CHUNK_SIZE, on_progress: nil)
       chunks = vectors.each_slice(chunk_size).to_a
       return { upserted_count: 0, chunks: 0, errors: [] } if chunks.empty?
 
-      results = process_chunks_concurrently(chunks) do |chunk|
+      results = process_chunks_concurrently(chunks, total_items: vectors.size, on_progress: on_progress) do |chunk|
         client.upsert(index: index, vectors: chunk, namespace: namespace)
       end
 
@@ -56,12 +78,14 @@ module Vectra
     # @param ids [Array<String>] IDs to delete
     # @param namespace [String, nil] optional namespace
     # @param chunk_size [Integer] IDs per chunk (default: 100)
+    # @param on_progress [Proc, nil] optional callback called after each chunk completes
+    #   Callback receives hash with: processed, total, percentage, current_chunk, total_chunks, success_count, failed_count
     # @return [Hash] aggregated result
-    def delete_async(index:, ids:, namespace: nil, chunk_size: DEFAULT_CHUNK_SIZE)
+    def delete_async(index:, ids:, namespace: nil, chunk_size: DEFAULT_CHUNK_SIZE, on_progress: nil)
       chunks = ids.each_slice(chunk_size).to_a
       return { deleted_count: 0, chunks: 0, errors: [] } if chunks.empty?
 
-      results = process_chunks_concurrently(chunks) do |chunk|
+      results = process_chunks_concurrently(chunks, total_items: ids.size, on_progress: on_progress) do |chunk|
         client.delete(index: index, ids: chunk, namespace: namespace)
       end
 
@@ -74,12 +98,14 @@ module Vectra
     # @param ids [Array<String>] IDs to fetch
     # @param namespace [String, nil] optional namespace
     # @param chunk_size [Integer] IDs per chunk (default: 100)
+    # @param on_progress [Proc, nil] optional callback called after each chunk completes
+    #   Callback receives hash with: processed, total, percentage, current_chunk, total_chunks, success_count, failed_count
     # @return [Hash<String, Vector>] merged results
-    def fetch_async(index:, ids:, namespace: nil, chunk_size: DEFAULT_CHUNK_SIZE)
+    def fetch_async(index:, ids:, namespace: nil, chunk_size: DEFAULT_CHUNK_SIZE, on_progress: nil)
       chunks = ids.each_slice(chunk_size).to_a
       return {} if chunks.empty?
 
-      results = process_chunks_concurrently(chunks) do |chunk|
+      results = process_chunks_concurrently(chunks, total_items: ids.size, on_progress: on_progress) do |chunk|
         client.fetch(index: index, ids: chunk, namespace: namespace)
       end
 
@@ -88,15 +114,43 @@ module Vectra
 
     private
 
-    def process_chunks_concurrently(chunks)
+    # rubocop:disable Metrics/AbcSize, Metrics/MethodLength, Metrics/BlockLength
+    def process_chunks_concurrently(chunks, total_items: nil, on_progress: nil)
       pool = Concurrent::FixedThreadPool.new(concurrency)
       futures = []
+      progress_mutex = Mutex.new
+      completed_count = Concurrent::AtomicFixnum.new(0)
+      success_count = Concurrent::AtomicFixnum.new(0)
+      failed_count = Concurrent::AtomicFixnum.new(0)
 
       chunks.each_with_index do |chunk, index|
         futures << Concurrent::Future.execute(executor: pool) do
-          { index: index, result: yield(chunk), error: nil }
+          result = yield(chunk)
+          success_count.increment
+          { index: index, result: result, error: nil }
         rescue StandardError => e
+          failed_count.increment
           { index: index, result: nil, error: e }
+        ensure
+          # Call progress callback when chunk completes
+          if on_progress
+            completed = completed_count.increment
+            total_size = chunks.size * chunks.first.size
+            processed = [completed * chunks.first.size, total_items || total_size].min
+            percentage = total_items ? (processed.to_f / total_items * 100).round(2) : (completed.to_f / chunks.size * 100).round(2)
+
+            progress_mutex.synchronize do
+              on_progress.call(
+                processed: processed,
+                total: total_items || total_size,
+                percentage: percentage,
+                current_chunk: completed - 1,
+                total_chunks: chunks.size,
+                success_count: success_count.value,
+                failed_count: failed_count.value
+              )
+            end
+          end
         end
       end
 
@@ -107,6 +161,7 @@ module Vectra
 
       results.sort_by { |r| r[:index] }
     end
+    # rubocop:enable Metrics/AbcSize, Metrics/MethodLength, Metrics/BlockLength
 
     def aggregate_results(results, total_vectors)
       errors = results.select { |r| r[:error] }.map { |r| r[:error] }