vectra-client 0.3.4 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: bcc42d23052b076d9efcb085b9490b3d21b0e1b101a8edb3d9fca500d72ac2cd
-  data.tar.gz: 5abe33ef210d55e3cda6ffa8f80ca1ecadcea8632c46eb9ce78c824f35684c68
+  metadata.gz: f9965925f32b1b497e306ba25776f06edaba86b3c8abf8b750c8a449fe68ba5a
+  data.tar.gz: 1622d500398e1fb95b146d2792c0f29abdde93888e7b74af8fd1dc674c9bee58
 SHA512:
-  metadata.gz: 40aaec0556a2de07028db2e75ad2ebf463f51728336fc508676a567837a45d1be00174ed679601a1aa5f976a05605c10c86817dd99616bc38b63350fa1ed9e63
-  data.tar.gz: 1a3bfad7f776a429be42f591af896c3c9e9ba55a45eb58dffda8b9fa50190d5248552bfab87038ab53e04183d2cbe5fcade506bf0cc99580b1435c50a30b64b9
+  metadata.gz: 1911cce768648f9c48e9c94e13a7d0c51f39e81eb381a0defcbc581eb27b4cfacb56d41e1cfb776f6a2ef4522e344c32d842a550c79d5a7584bb1a84a637e605
+  data.tar.gz: bfb7cc7174a739591a8061436f47963dab2c33809db3437558f5addf9cf4dff2b15b93bc72934f1dd04821f2f93884b07f0c9f5886b4d618d9e3eff15ce907a6

data/CHANGELOG.md

@@ -1,13 +1,63 @@
 # Changelog
 
+## [v1.0.0](https://github.com/stokry/vectra/tree/v1.0.0) (2026-01-12)
+
+### Added
+- Hybrid search functionality for Qdrant and Weaviate providers
+- Enhanced provider capabilities and error handling
+- Support for advanced filtering and namespace operations
+- Improved vector search performance
+
+### Changed
+- Major API refinements and provider implementations
+- Enhanced test coverage and documentation
+
+[Full Changelog](https://github.com/stokry/vectra/compare/v0.4.0...v1.0.0)
+
+## [v0.4.0](https://github.com/stokry/vectra/tree/v0.4.0) (2026-01-12)
+
+[Full Changelog](https://github.com/stokry/vectra/compare/v0.3.4...v0.4.0)
+
+### Added
+- **Hybrid Search** - Combine semantic (vector) and keyword (text) search across all providers
+  - Full support for Qdrant (prefetch + rescore API)
+  - Full support for Weaviate (hybrid GraphQL with BM25)
+  - Full support for pgvector (vector similarity + PostgreSQL full-text search)
+  - Partial support for Pinecone (requires sparse vectors for true hybrid search)
+  - Alpha parameter (0.0 = pure keyword, 1.0 = pure semantic) for fine-tuning balance
+- **Batch Progress Callbacks** - Real-time visibility into batch operations
+  - `on_progress` callback with detailed statistics (processed, total, percentage, chunk info)
+  - Thread-safe progress tracking with `Concurrent::AtomicFixnum`
+  - Support for `upsert_async`, `delete_async`, and `fetch_async` methods
+- **Vector Normalization Helper** - Improve cosine similarity results
+  - `Vector.normalize!` instance method (L2 and L1 normalization)
+  - `Vector.normalize` class method for non-mutating normalization
+  - Automatic handling of zero vectors
+- **Dimension Validation** - Automatic validation of vector dimension consistency
+  - Validates all vectors in a batch have the same dimension
+  - Detailed error messages with index and expected/actual dimensions
+  - Works with both Vector objects and hash vectors
+- **Better Error Messages** - Enhanced error context and debugging
+  - Includes error details, field-specific errors, and context
+  - Improved error message format: "Main message (details) [Fields: field1, field2]"
+- **Connection Health Check** - Simple health monitoring methods
+  - `healthy?` method for quick boolean health check
+  - `ping` method with latency measurement and detailed status
+  - Automatic error logging when logger is configured
+
+### Changed
+- Improved error handling with more context in error messages
+- Enhanced batch operations with progress tracking capabilities
+
+### Documentation
+- Added comprehensive hybrid search examples and provider support matrix
+- Updated getting started guide with normalization, health checks, and dimension validation
+- Added real-world examples demonstrating new features
+
 ## [v0.3.4](https://github.com/stokry/vectra/tree/v0.3.4) (2026-01-12)
 
 [Full Changelog](https://github.com/stokry/vectra/compare/v0.3.3...v0.3.4)
 
-### Fixed
-- Fixed Weaviate provider DELETE request WebMock stub to include query parameters
-- Fixed RuboCop offenses: empty string interpolation, redundant else, ambiguous block associations, and style issues
-
 ## [v0.3.3](https://github.com/stokry/vectra/tree/v0.3.3) (2026-01-09)
 
 [Full Changelog](https://github.com/stokry/vectra/compare/v0.3.2...v0.3.3)

data/README.md

@@ -17,6 +17,7 @@
 | **Qdrant** | Open Source | ✅ Supported |
 | **Weaviate** | Open Source | ✅ Supported |
 | **pgvector** | PostgreSQL | ✅ Supported |
+| **Memory** | In-Memory | ✅ Testing only |
 
 ## Installation
 
@@ -48,12 +49,47 @@ client.upsert(
   ]
 )
 
-# Search
+# Search (classic API)
 results = client.query(vector: [0.1, 0.2, 0.3], top_k: 5)
 results.each { |match| puts "#{match.id}: #{match.score}" }
 
+# Search (chainable Query Builder)
+results = client
+  .query('docs')
+  .vector([0.1, 0.2, 0.3])
+  .top_k(5)
+  .with_metadata
+  .execute
+
+results.each do |match|
+  puts "#{match.id}: #{match.score}"
+end
+
+# Normalize embeddings (for better cosine similarity)
+embedding = openai_response['data'][0]['embedding']
+normalized = Vectra::Vector.normalize(embedding)
+client.upsert(vectors: [{ id: 'doc-1', values: normalized }])
+
 # Delete
 client.delete(ids: ['doc-1', 'doc-2'])
+
+# Health check
+if client.healthy?
+  puts "Connection is healthy"
+end
+
+# Ping with latency
+status = client.ping
+puts "Provider: #{status[:provider]}, Latency: #{status[:latency_ms]}ms"
+
+# Hybrid search (semantic + keyword)
+# Supported by: Qdrant, Weaviate, Pinecone, pgvector
+results = client.hybrid_search(
+  index: 'docs',
+  vector: embedding,
+  text: 'ruby programming',
+  alpha: 0.7  # 70% semantic, 30% keyword
+)
 ```
 
 ## Provider Examples
@@ -69,10 +105,16 @@ client = Vectra.qdrant(host: 'http://localhost:6333')
 client = Vectra.qdrant(host: 'https://your-cluster.qdrant.io', api_key: ENV['QDRANT_API_KEY'])
 
 # Weaviate
-client = Vectra.weaviate(host: 'http://localhost:8080', api_key: ENV['WEAVIATE_API_KEY'])
+client = Vectra.weaviate(
+  api_key: ENV['WEAVIATE_API_KEY'],
+  host: 'https://your-weaviate-instance'
+)
 
 # pgvector (PostgreSQL)
 client = Vectra.pgvector(connection_url: 'postgres://user:pass@localhost/mydb')
+
+# Memory (in-memory, testing only)
+client = Vectra.memory
 ```
 
 ## Features

data/docs/api/overview.md

@@ -111,6 +111,98 @@ Get index statistics.
 }
 ```
 
+### `hybrid_search(index:, vector:, text:, alpha:, top_k:)`
+
+Combine semantic (vector) and keyword (text) search.
+
+**Parameters:**
+- `index` (String) - Index/collection name
+- `vector` (Array) - Query vector for semantic search
+- `text` (String) - Text query for keyword search
+- `alpha` (Float) - Balance between semantic and keyword (0.0 = pure keyword, 1.0 = pure semantic)
+- `top_k` (Integer) - Number of results (default: 10)
+- `namespace` (String, optional) - Namespace
+- `filter` (Hash, optional) - Metadata filter
+- `include_values` (Boolean) - Include vector values (default: false)
+- `include_metadata` (Boolean) - Include metadata (default: true)
+
+**Example:**
+```ruby
+results = client.hybrid_search(
+  index: 'docs',
+  vector: embedding,
+  text: 'ruby programming',
+  alpha: 0.7  # 70% semantic, 30% keyword
+)
+```
+
+**Provider Support:** Qdrant ✅, Weaviate ✅, pgvector ✅, Pinecone ⚠️
+
+### `healthy?`
+
+Quick health check - returns true if provider connection is healthy.
+
+**Returns:** Boolean
+
+**Example:**
+```ruby
+if client.healthy?
+  client.upsert(...)
+end
+```
+
+### `ping`
+
+Ping provider and get connection health status with latency.
+
+**Returns:**
+```ruby
+{
+  healthy: true,
+  provider: :pinecone,
+  latency_ms: 45.23
+}
+```
+
+**Example:**
+```ruby
+status = client.ping
+puts "Latency: #{status[:latency_ms]}ms"
+```
+
+### `Vector.normalize(vector, type: :l2)`
+
+Normalize a vector array (non-mutating).
+
+**Parameters:**
+- `vector` (Array) - Vector values to normalize
+- `type` (Symbol) - Normalization type: `:l2` (default) or `:l1`
+
+**Returns:** Array of normalized values
+
+**Example:**
+```ruby
+embedding = openai_response['data'][0]['embedding']
+normalized = Vectra::Vector.normalize(embedding)
+client.upsert(vectors: [{ id: 'doc-1', values: normalized }])
+```
+
+### `vector.normalize!(type: :l2)`
+
+Normalize vector in-place (mutates the vector).
+
+**Parameters:**
+- `type` (Symbol) - Normalization type: `:l2` (default) or `:l1`
+
+**Returns:** Self (for method chaining)
+
+**Example:**
+```ruby
+vector = Vectra::Vector.new(id: 'doc-1', values: embedding)
+vector.normalize!  # L2 normalization
+client.upsert(vectors: [vector])
+```
+
 ## Error Handling
 
 ```ruby

data/docs/examples/real-world.md

@@ -37,6 +37,8 @@ class ProductSearchService
 
   def search(query:, category: nil, price_range: nil, limit: 20)
     query_embedding = generate_embedding(query)
+    # Normalize for better cosine similarity
+    query_embedding = Vectra::Vector.normalize(query_embedding)
     
     filter = {}
     filter[:category] = category if category
@@ -68,9 +70,12 @@ class ProductSearchService
 
   def generate_embedding(text)
     # Use your embedding model (OpenAI, sentence-transformers, etc.)
-    OpenAI::Client.new.embeddings(
+    embedding = OpenAI::Client.new.embeddings(
       parameters: { model: "text-embedding-ada-002", input: text }
     )["data"][0]["embedding"]
+    
+    # Normalize embeddings before storing
+    Vectra::Vector.normalize(embedding)
   end
 
   def fallback_search(query, category)
@@ -257,14 +262,14 @@ class TenantDocumentService
     query_embedding = generate_embedding(query)
     
     # Ensure tenant isolation via namespace
-    results = @client.query(
-      index: "documents",
-      vector: query_embedding,
-      top_k: limit,
-      namespace: "tenant-#{@tenant_id}",
-      filter: { tenant_id: @tenant_id }, # Double protection
-      include_metadata: true
-    )
+    results = @client
+      .query("documents")
+      .vector(query_embedding)
+      .top_k(limit)
+      .namespace("tenant-#{@tenant_id}")
+      .filter(tenant_id: @tenant_id) # Double protection
+      .with_metadata
+      .execute
     
     # Audit log
     @audit.log_access(
@@ -325,9 +330,11 @@ class DocumentIndexer
 
   def index_large_dataset(documents, concurrency: 4)
     total = documents.size
-    processed = 0
     errors = []
     
+    # Create batch client with specified concurrency
+    batch_client = Vectra::Batch.new(@client, concurrency: concurrency)
+    
     # Convert to vectors
     vectors = documents.map do |doc|
       {
@@ -337,25 +344,27 @@ class DocumentIndexer
       }
     end
     
-    # Process in async batches
-    result = @batch_client.upsert_async(
+    # Process in async batches with progress tracking
+    result = batch_client.upsert_async(
       index: "documents",
       vectors: vectors,
-      concurrency: concurrency,
-      on_progress: proc { |success, failed, total|
-        processed = success + failed
-        progress = (processed.to_f / total * 100).round(1)
+      chunk_size: 100,
+      on_progress: proc { |stats|
+        progress = stats[:percentage]
+        processed = stats[:processed]
+        total = stats[:total]
+        chunk = stats[:current_chunk] + 1
+        total_chunks = stats[:total_chunks]
+        
         puts "Progress: #{progress}% (#{processed}/#{total})"
-      },
-      on_error: proc { |error, vector|
-        errors << { id: vector[:id], error: error.message }
+        puts "  Chunk #{chunk}/#{total_chunks} | Success: #{stats[:success_count]}, Failed: #{stats[:failed_count]}"
       }
     )
     
     {
-      success: result[:success],
-      failed: result[:failed],
-      errors: errors,
+      success: result[:upserted_count],
+      failed: result[:errors].size,
+      errors: result[:errors],
       total: total
     }
   end
@@ -475,6 +484,37 @@ module VectraHelper
 end
 ```
 
+## Testing with the Memory Provider
+
+For fast, deterministic tests you can run Vectra entirely in memory without any external services:
+
+```ruby
+# config/initializers/vectra.rb (test environment)
+Vectra.configure do |config|
+  config.provider = :memory if Rails.env.test?
+end
+
+RSpec.describe ProductSearchService do
+  let(:client) { Vectra::Client.new } # uses memory provider in test
+
+  before do
+    client.provider.clear! if client.provider.respond_to?(:clear!)
+
+    client.upsert(
+      index: "products",
+      vectors: [
+        { id: "p1", values: [0.1, 0.2], metadata: { name: "Test Product" } }
+      ]
+    )
+  end
+
+  it "returns relevant products" do
+    results = client.query(index: "products", vector: [0.1, 0.2], top_k: 5)
+    expect(results.ids).to include("p1")
+  end
+end
+```
+
 ## Best Practices
 
 ### 1. Always Use Caching for Frequent Queries

data/docs/guides/getting-started.md

@@ -43,17 +43,49 @@ client.upsert(
 ### Query (Search)
 
 ```ruby
+# Classic API
 results = client.query(
   vector: [0.1, 0.2, 0.3],
   top_k: 5,
   include_metadata: true
 )
 
-results.matches.each do |match|
-  puts "ID: #{match['id']}, Score: #{match['score']}"
+results.each do |match|
+  puts "ID: #{match.id}, Score: #{match.score}"
+end
+
+# Chainable Query Builder
+results = client
+  .query("my-index")
+  .vector([0.1, 0.2, 0.3])
+  .top_k(5)
+  .with_metadata
+  .execute
+
+results.each do |match|
+  puts "ID: #{match.id}, Score: #{match.score}"
 end
 ```
 
+### Normalize Embeddings
+
+For better cosine similarity results, normalize your embeddings before upserting:
+
+```ruby
+# Normalize OpenAI embeddings (recommended)
+embedding = openai_response['data'][0]['embedding']
+normalized = Vectra::Vector.normalize(embedding)
+client.upsert(vectors: [{ id: 'doc-1', values: normalized }])
+
+# Or normalize in-place
+vector = Vectra::Vector.new(id: 'doc-1', values: embedding)
+vector.normalize!  # L2 normalization (default, unit vector)
+client.upsert(vectors: [vector])
+
+# L1 normalization (sum of absolute values = 1)
+vector.normalize!(type: :l1)
+```
+
 ### Delete Vectors
 
 ```ruby
@@ -68,6 +100,88 @@ puts "Index dimension: #{stats['dimension']}"
 puts "Vector count: #{stats['vector_count']}"
 ```
 
+### Health Check & Ping
+
+```ruby
+# Quick health check
+if client.healthy?
+  client.upsert(...)
+else
+  handle_unhealthy_connection
+end
+
+# Ping with latency measurement
+status = client.ping
+puts "Provider: #{status[:provider]}"
+puts "Healthy: #{status[:healthy]}"
+puts "Latency: #{status[:latency_ms]}ms"
+
+if status[:error]
+  puts "Error: #{status[:error_message]}"
+end
+```
+
+### Hybrid Search (Semantic + Keyword)
+
+Combine the best of both worlds: semantic understanding from vectors and exact keyword matching:
+
+```ruby
+# Hybrid search with 70% semantic, 30% keyword
+results = client.hybrid_search(
+  index: 'docs',
+  vector: embedding,           # Semantic search
+  text: 'ruby programming',  # Keyword search
+  alpha: 0.7,                 # 0.0 = pure keyword, 1.0 = pure semantic
+  top_k: 10
+)
+
+results.each do |match|
+  puts "#{match.id}: #{match.score}"
+end
+
+# Pure semantic (alpha = 1.0)
+results = client.hybrid_search(
+  index: 'docs',
+  vector: embedding,
+  text: 'ruby',
+  alpha: 1.0
+)
+
+# Pure keyword (alpha = 0.0)
+results = client.hybrid_search(
+  index: 'docs',
+  vector: embedding,
+  text: 'ruby programming',
+  alpha: 0.0
+)
+```
+
+**Provider Support:**
+- **Qdrant**: ✅ Full support (prefetch + rescore API)
+- **Weaviate**: ✅ Full support (hybrid GraphQL with BM25)
+- **Pinecone**: ⚠️ Partial support (requires sparse vectors for true hybrid search)
+- **pgvector**: ✅ Full support (combines vector similarity + PostgreSQL full-text search)
+
+**Note for pgvector:** Your table needs a text column with a tsvector index:
+```sql
+CREATE INDEX idx_content_fts ON my_index USING gin(to_tsvector('english', content));
+```
+
+### Dimension Validation
+
+Vectra automatically validates that all vectors in a batch have the same dimension:
+
+```ruby
+# This will raise ValidationError
+vectors = [
+  { id: "vec1", values: [0.1, 0.2, 0.3] }, # 3 dimensions
+  { id: "vec2", values: [0.4, 0.5] }        # 2 dimensions - ERROR!
+]
+
+client.upsert(vectors: vectors)
+# => ValidationError: Inconsistent vector dimensions at index 1: expected 3, got 2
+```
+
 ## Configuration
 
 Create a configuration file (Rails: `config/initializers/vectra.rb`):

data/docs/guides/performance.md

@@ -26,13 +26,47 @@ vectors = 10_000.times.map { |i| { id: "vec_#{i}", values: Array.new(384) { rand
 result = batch.upsert_async(
   index: 'my-index',
   vectors: vectors,
-  chunk_size: 100
+  chunk_size: 100,
+  on_progress: proc { |stats|
+    progress = stats[:percentage]
+    processed = stats[:processed]
+    total = stats[:total]
+    chunk = stats[:current_chunk] + 1
+    total_chunks = stats[:total_chunks]
+    
+    puts "Progress: #{progress}% (#{processed}/#{total})"
+    puts "  Chunk #{chunk}/#{total_chunks} | Success: #{stats[:success_count]}, Failed: #{stats[:failed_count]}"
+  }
 )
 
 puts "Upserted: #{result[:upserted_count]} vectors in #{result[:chunks]} chunks"
 puts "Errors: #{result[:errors].size}" if result[:errors].any?
 ```
 
+### Progress Tracking
+
+Monitor batch operations in real-time with progress callbacks:
+
+```ruby
+batch.upsert_async(
+  index: 'my-index',
+  vectors: large_vector_array,
+  chunk_size: 100,
+  on_progress: proc { |stats|
+    # stats contains:
+    # - processed: number of processed vectors
+    # - total: total number of vectors
+    # - percentage: progress percentage (0-100)
+    # - current_chunk: current chunk index (0-based)
+    # - total_chunks: total number of chunks
+    # - success_count: number of successful chunks
+    # - failed_count: number of failed chunks
+    
+    puts "Progress: #{stats[:percentage]}% (#{stats[:processed]}/#{stats[:total]})"
+  }
+)
+```
+
 ### Batch Delete
 
 ```ruby

data/docs/providers/index.md

@@ -16,6 +16,7 @@ Vectra supports multiple vector database providers. Choose the one that best fit
 | [**Qdrant**]({{ site.baseurl }}/providers/qdrant) | Open Source | Self-hosted, Performance |
 | [**Weaviate**]({{ site.baseurl }}/providers/weaviate) | Open Source | Semantic search, GraphQL |
 | [**pgvector**]({{ site.baseurl }}/providers/pgvector) | PostgreSQL | SQL integration, ACID |
+| [**Memory**]({{ site.baseurl }}/providers/memory) | In-Memory | Testing, CI, local dev |
 
 ## Quick Comparison
 
@@ -75,6 +76,17 @@ client.upsert(vectors: [...])
 results = client.query(vector: [...], top_k: 5)
 ```
 
+For **tests and CI** you can use the in-memory provider:
+
+```ruby
+# config/initializers/vectra.rb (test environment)
+Vectra.configure do |config|
+  config.provider = :memory if Rails.env.test?
+end
+
+client = Vectra::Client.new
+```
+
 ## Next Steps
 
 - [Getting Started Guide]({{ site.baseurl }}/guides/getting-started)