vectra-client 0.2.1 → 0.3.0

data/NEW_FEATURES_v0.2.0.md

@@ -1,459 +0,0 @@
-# 🚀 NEW FEATURES in v0.2.0
-
-## Overview
-
-Version 0.2.0 adds **enterprise-grade features** for production use:
-
-- 📊 **Instrumentation & Monitoring** - New Relic, Datadog, custom handlers
-- 🎨 **Rails Generator** - `rails g vectra:install`
-- 💎 **ActiveRecord Integration** - `has_vector` DSL for seamless Rails integration
-- 🔄 **Automatic Retry Logic** - Resilience for transient database errors
-- ⚡ **Performance Benchmarks** - Measure and optimize your setup
-
----
-
-## 📊 Instrumentation & Monitoring
-
-Track all vector operations in production.
-
-### Quick Start
-
-```ruby
-# config/initializers/vectra.rb
-Vectra.configure do |config|
-  config.instrumentation = true
-end
-
-# Custom handler
-Vectra.on_operation do |event|
-  Rails.logger.info "Vectra: #{event.operation} took #{event.duration}ms"
-
-  # Send to monitoring
-  StatsD.timing("vectra.#{event.operation}", event.duration)
-  StatsD.increment("vectra.#{event.success? ? 'success' : 'error'}")
-end
-```
-
-### New Relic Integration
-
-```ruby
-require 'vectra/instrumentation/new_relic'
-
-Vectra.configure { |c| c.instrumentation = true }
-Vectra::Instrumentation::NewRelic.setup!
-```
-
-Automatically tracks:
-- `Custom/Vectra/pgvector/query/duration`
-- `Custom/Vectra/pgvector/upsert/success`
-- And more...
-
-### Datadog Integration
-
-```ruby
-require 'vectra/instrumentation/datadog'
-
-Vectra.configure { |c| c.instrumentation = true }
-Vectra::Instrumentation::Datadog.setup!(
-  host: ENV['DD_AGENT_HOST'],
-  port: 8125
-)
-```
-
-Metrics:
-- `vectra.operation.duration` (timing)
-- `vectra.operation.count` (counter)
-- `vectra.operation.error` (counter)
-
-### Event API
-
-```ruby
-Vectra.on_operation do |event|
-  event.operation    # :query, :upsert, :fetch, :update, :delete
-  event.provider     # :pgvector, :pinecone
-  event.index        # 'documents'
-  event.duration     # 123.45 (milliseconds)
-  event.metadata     # { vector_count: 10, top_k: 5, ... }
-  event.success?     # true/false
-  event.error        # Exception or nil
-end
-```
-
----
-
-## 🎨 Rails Generator
-
-One command to set up Vectra in Rails.
-
-### Usage
-
-```bash
-rails generate vectra:install --provider=pgvector --instrumentation=true
-```
-
-### What it creates:
-
-1. **config/initializers/vectra.rb** - Configuration with smart defaults
-2. **db/migrate/XXX_enable_pgvector_extension.rb** - Enables pgvector (if using PostgreSQL)
-3. **Setup instructions** - Provider-specific next steps
-
-### Options
-
-```bash
---provider=NAME        # pinecone, pgvector, qdrant, weaviate (default: pgvector)
---database-url=URL     # PostgreSQL connection URL (for pgvector)
---api-key=KEY          # API key for the provider
---instrumentation      # Enable instrumentation
-```
-
-### Example
-
-```bash
-rails g vectra:install --provider=pgvector --instrumentation=true
-rails db:migrate
-
-# Creates initializer with:
-# - Connection pooling (10 connections)
-# - Batch operations (100 vectors/batch)
-# - Instrumentation enabled
-# - Logging to Rails.logger
-```
-
----
-
-## 💎 ActiveRecord Integration
-
-Add vector search to any Rails model.
-
-### Quick Start
-
-```ruby
-class Document < ApplicationRecord
-  include Vectra::ActiveRecord
-
-  has_vector :embedding,
-             dimension: 384,
-             provider: :pgvector,
-             index: 'documents',
-             auto_index: true,
-             metadata_fields: [:title, :category, :status]
-
-  # Generate embeddings (use OpenAI, Cohere, etc.)
-  before_validation :generate_embedding, if: :content_changed?
-
-  def generate_embedding
-    self.embedding = OpenAI::Client.new.embeddings(
-      parameters: { model: 'text-embedding-3-small', input: content }
-    ).dig('data', 0, 'embedding')
-  end
-end
-```
-
-### Usage
-
-```ruby
-# Create (automatically indexed)
-doc = Document.create!(
-  title: 'Getting Started',
-  content: 'Learn how to...',
-  category: 'tutorial'
-)
-
-# Vector search
-query_vector = generate_embedding('how to get started')
-results = Document.vector_search(query_vector, limit: 10)
-
-results.each do |doc|
-  puts "#{doc.title} - Score: #{doc.vector_score}"
-end
-
-# Find similar documents
-similar = doc.similar(limit: 5)
-
-# Search with filters
-results = Document.vector_search(
-  query_vector,
-  limit: 10,
-  filter: { category: 'tutorial', status: 'published' }
-)
-
-# Manual control
-doc.index_vector!   # Force indexing
-doc.delete_vector!  # Remove from index
-```
-
-### Features
-
-- ✅ **Auto-indexing** - On create/update
-- ✅ **Auto-deletion** - On destroy
-- ✅ **Metadata sync** - Specified fields included in vector metadata
-- ✅ **AR object loading** - Search returns ActiveRecord objects, not just vectors
-- ✅ **Score access** - `doc.vector_score` available on results
-- ✅ **Background jobs** - Disable auto-index for async processing
-
-### Background Indexing
-
-```ruby
-class Document < ApplicationRecord
-  include Vectra::ActiveRecord
-
-  has_vector :embedding, dimension: 384, auto_index: false
-
-  after_commit :index_async, on: [:create, :update]
-
-  def index_async
-    IndexVectorJob.perform_later(id)
-  end
-end
-
-# Job
-class IndexVectorJob < ApplicationJob
-  def perform(document_id)
-    doc = Document.find(document_id)
-    doc.index_vector!
-  end
-end
-```
-
----
-
-## 🔄 Automatic Retry Logic
-
-Resilience for transient database errors.
-
-### What it does
-
-Automatically retries operations on:
-- `PG::ConnectionBad`
-- `PG::UnableToSend`
-- `PG::TooManyConnections`
-- `PG::SerializationFailure`
-- `PG::DeadlockDetected`
-- `ConnectionPool::TimeoutError`
-- Any error with "timeout" or "connection" in message
-
-### Configuration
-
-```ruby
-Vectra.configure do |config|
-  config.max_retries = 3      # Default: 3
-  config.retry_delay = 1.0    # Initial delay: 1 second
-end
-```
-
-### How it works
-
-- **Exponential backoff**: 1s, 2s, 4s, 8s, ...
-- **Jitter**: ±25% randomness to prevent thundering herd
-- **Max delay**: Capped at 30 seconds
-- **Logging**: Each retry logged with delay and reason
-
-### Already integrated
-
-Retry logic is automatically used in:
-- pgvector `execute()` method
-- All database operations
-
-No code changes needed - it just works!
-
----
-
-## ⚡ Performance Benchmarks
-
-Measure and optimize your setup.
-
-### Run benchmarks
-
-```bash
-# Batch operations
-DATABASE_URL=postgres://localhost/vectra_bench \
-  ruby benchmarks/batch_operations_benchmark.rb
-
-# Connection pooling
-DATABASE_URL=postgres://localhost/vectra_bench \
-  ruby benchmarks/connection_pooling_benchmark.rb
-```
-
-### What they measure
-
-**Batch Operations:**
-- Vector counts: 100, 500, 1K, 5K, 10K
-- Batch sizes: 50, 100, 250, 500
-- Metrics: avg time, vectors/sec, batch count
-
-**Connection Pooling:**
-- Pool sizes: 5, 10, 20
-- Thread counts: 1, 2, 5, 10, 20
-- Metrics: total time, ops/sec, pool availability
-
-### Example output
-
-```
-1000 vectors:
-  Batch size  50: 2.134s avg (468 vectors/sec, 20 batches)
-  Batch size 100: 1.892s avg (529 vectors/sec, 10 batches)
-  Batch size 250: 1.645s avg (608 vectors/sec, 4 batches)
-  Batch size 500: 1.523s avg (657 vectors/sec, 2 batches)
-
-Pool Size: 10
-   1 threads: 5.21s total (9.6 ops/sec) Pool: 9/10 available
-   5 threads: 5.34s total (46.8 ops/sec) Pool: 5/10 available
-  10 threads: 5.67s total (88.2 ops/sec) Pool: 0/10 available
-```
-
-### Recommendations
-
-Based on benchmarks:
-- **Batch size**: 250-500 for best performance
-- **Pool size**: Match your app server thread count
-- **Monitoring**: Use instrumentation to track in production
-
----
-
-## 📚 Documentation
-
-New comprehensive guides:
-
-- **USAGE_EXAMPLES.md** - 10 practical examples
-  - E-commerce semantic search
-  - RAG chatbot
-  - Duplicate detection
-  - Rails integration
-  - Performance tips
-
-- **IMPLEMENTATION_GUIDE.md** - Developer guide
-  - Feature implementation
-  - Testing strategies
-  - Customization examples
-  - Troubleshooting
-
----
-
-## 🚀 Migration Guide
-
-### From v0.1.x to v0.2.0
-
-**No breaking changes!** v0.2.0 is fully backward compatible.
-
-### New configuration options
-
-```ruby
-Vectra.configure do |config|
-  # Existing options still work
-  config.provider = :pgvector
-  config.host = 'postgres://...'
-
-  # NEW: Instrumentation (optional)
-  config.instrumentation = false  # default
-
-  # NEW: Already had these, now documented
-  config.pool_size = 5           # Connection pool size
-  config.pool_timeout = 5        # Seconds to wait for connection
-  config.batch_size = 100        # Vectors per batch
-  config.max_retries = 3         # Retry attempts
-  config.retry_delay = 1         # Initial retry delay
-end
-```
-
-### Enabling new features
-
-```ruby
-# 1. Enable instrumentation
-Vectra.configure { |c| c.instrumentation = true }
-
-# 2. Add handler (optional)
-Vectra.on_operation { |event| puts event.operation }
-
-# 3. Use ActiveRecord integration (opt-in)
-class Document < ApplicationRecord
-  include Vectra::ActiveRecord
-  has_vector :embedding, dimension: 384
-end
-```
-
-### No action required
-
-These features work automatically:
-- ✅ Retry logic (already integrated)
-- ✅ Performance improvements (transparent)
-
----
-
-## 📊 Performance Improvements
-
-### Before v0.2.0
-
-```ruby
-# 10,000 individual operations
-10_000.times { client.upsert(index: 'docs', vectors: [vec]) }
-# ~50 seconds
-```
-
-### After v0.2.0
-
-```ruby
-# Batch operations (automatic)
-client.upsert(index: 'docs', vectors: 10_000_vectors)
-# ~5 seconds (10x faster!)
-```
-
-### Connection Pooling
-
-```ruby
-# Before: New connection per request
-# After: Reuse from pool (5-10x faster in multi-threaded apps)
-
-Vectra.configure do |config|
-  config.pool_size = 20  # Match your app threads
-end
-```
-
----
-
-## 🎯 Next Steps
-
-1. **Update your gem:**
-   ```bash
-   bundle update vectra
-   ```
-
-2. **Enable instrumentation:**
-   ```ruby
-   Vectra.configure { |c| c.instrumentation = true }
-   ```
-
-3. **Try ActiveRecord integration:**
-   ```ruby
-   rails g vectra:install --provider=pgvector
-   ```
-
-4. **Run benchmarks:**
-   ```bash
-   ruby benchmarks/batch_operations_benchmark.rb
-   ```
-
-5. **Read examples:**
-   - See USAGE_EXAMPLES.md for 10 practical examples
-   - See IMPLEMENTATION_GUIDE.md for detailed docs
-
----
-
-## 🤝 Contributing
-
-We welcome contributions! See:
-- CONTRIBUTING.md - Contribution guide
-- IMPLEMENTATION_GUIDE.md - Feature implementation guide
-
----
-
-## 📝 Changelog
-
-See CHANGELOG.md for complete version history.
-
----
-
-## ❓ Questions?
-
-- GitHub Issues: https://github.com/stokry/vectra/issues
-- Documentation: See README.md, USAGE_EXAMPLES.md
-- Examples: See examples/ directory