vectra-client 0.3.4 → 1.0.0

data/docs/providers/memory.md

@@ -0,0 +1,145 @@
+---
+layout: page
+title: Memory Provider
+permalink: /providers/memory/
+---
+
+# Memory Provider
+
+The **Memory provider** is an in-memory vector store built into Vectra.
+It is designed **exclusively for testing, local development, and CI** – no external database required.
+
+> Not for production use. All data lives in process memory and is lost when the process exits.
+
+## When to Use
+
+- ✅ RSpec / Minitest suites (fast, isolated tests)
+- ✅ Local development without provisioning Pinecone/Qdrant/Weaviate/pgvector
+- ✅ CI pipelines where external services are not available
+- ❌ Not suitable for production workloads
+
+## Setup
+
+### Global Configuration (Rails Example)
+
+```ruby
+# config/initializers/vectra.rb
+Vectra.configure do |config|
+  config.provider = :memory if Rails.env.test?
+end
+```
+
+Then in your application code:
+
+```ruby
+client = Vectra::Client.new
+```
+
+### Direct Construction
+
+```ruby
+require "vectra"
+
+client = Vectra.memory
+```
+
+No `host`, `api_key`, or environment configuration is required.
+
+## Features
+
+- ✅ `upsert` – store vectors in memory
+- ✅ `query` – cosine similarity search (with optional metadata filtering)
+- ✅ `fetch` – retrieve vectors by ID
+- ✅ `update` – merge metadata for existing vectors
+- ✅ `delete` – delete by IDs, namespace, filter, or `delete_all`
+- ✅ `list_indexes` / `describe_index` – basic index metadata (dimension, metric)
+- ✅ `stats` – vector counts per namespace
+- ✅ `clear!` – wipe all data between tests
+
+## Basic Usage
+
+```ruby
+client = Vectra.memory
+
+# Upsert
+client.upsert(
+  index: "documents",
+  vectors: [
+    {
+      id: "doc-1",
+      values: [0.1, 0.2, 0.3],
+      metadata: { title: "Hello", category: "docs" }
+    }
+  ],
+  namespace: "test-suite"
+)
+
+# Query
+results = client.query(
+  index: "documents",
+  vector: [0.1, 0.2, 0.3],
+  top_k: 5,
+  namespace: "test-suite",
+  filter: { category: "docs" },
+  include_metadata: true
+)
+
+results.each do |match|
+  puts "#{match.id}: #{match.metadata["title"]} (score=#{match.score.round(3)})"
+end
+```
+
+## Metadata Filtering
+
+The Memory provider supports a subset of the same filter operators as other providers:
+
+```ruby
+results = client.query(
+  index: "products",
+  vector: query_embedding,
+  top_k: 10,
+  filter: {
+    status: ["active", "preview"],         # IN
+    price: { "$gte" => 10, "$lte" => 100 },# range
+    brand: { "$ne" => "Acme" }             # not equal
+  },
+  include_metadata: true
+)
+```
+
+Supported operators:
+
+- Equality: `{ field: "value" }` or `{ field: { "$eq" => "value" } }`
+- Inequality: `{ field: { "$ne" => "value" } }`
+- Ranges: `{ field: { "$gt" => 10 } }`, `{ "$gte" => 10 }`, `{ "$lt" => 20 }`, `{ "$lte" => 20 }`
+- Arrays / IN: `{ field: ["a", "b"] }` or `{ field: { "$in" => ["a", "b"] } }`
+
+## Resetting State Between Tests
+
+The Memory provider exposes a `clear!` method to reset all in-memory state:
+
+```ruby
+RSpec.describe "MyService" do
+  let(:client) { Vectra.memory }
+
+  before do
+    client.provider.clear! if client.provider.respond_to?(:clear!)
+  end
+
+  # your tests...
+end
+```
+
+You can also call `clear!` on the provider obtained from a regular `Vectra::Client`:
+
+```ruby
+client = Vectra::Client.new(provider: :memory)
+client.provider.clear!
+```
+
+## Limitations
+
+- Not distributed – data lives only in the current Ruby process.
+- No persistence – all vectors are lost when the process exits or `clear!` is called.
+- Intended for **testing and development only**, not for production traffic.
+

data/docs/providers/pgvector.md

@@ -43,6 +43,7 @@ client = Vectra::Client.new(
 - ✅ ACID transactions
 - ✅ Complex queries
 - ✅ Rails ActiveRecord integration
+- ✅ Hybrid search (vector + full-text search)
 
 ## Example
 
@@ -63,6 +64,17 @@ client.upsert(
 
 # Search using cosine distance
 results = client.query(vector: [0.1, 0.2, 0.3], top_k: 5)
+
+# Hybrid search (requires text column with tsvector index)
+# First, create the index:
+# CREATE INDEX idx_content_fts ON my_index USING gin(to_tsvector('english', content));
+results = client.hybrid_search(
+  index: 'my_index',
+  vector: embedding,
+  text: 'ruby programming',
+  alpha: 0.7,
+  text_column: 'content'  # default: 'content'
+)
 ```
 
 ## ActiveRecord Integration

data/docs/providers/pinecone.md

@@ -32,6 +32,7 @@ client = Vectra::Client.new(
 - ✅ Index statistics
 - ✅ Metadata filtering
 - ✅ Namespace support
+- ⚠️ Hybrid search (partial - requires sparse vectors)
 
 ## Example
 
@@ -56,6 +57,15 @@ results = client.query(vector: [0.1, 0.2, 0.3], top_k: 5)
 results.matches.each do |match|
   puts "#{match['id']}: #{match['score']}"
 end
+
+# Hybrid search (note: requires sparse vectors for true hybrid search)
+# For now, this uses dense vector search only
+results = client.hybrid_search(
+  index: 'my-index',
+  vector: embedding,
+  text: 'ruby programming',
+  alpha: 0.7
+)
 ```
 
 ## Configuration Options

data/docs/providers/qdrant.md

@@ -56,6 +56,14 @@ client.upsert(
 
 # Search
 results = client.query(vector: [0.1, 0.2, 0.3], top_k: 10)
+
+# Hybrid search (semantic + keyword)
+results = client.hybrid_search(
+  index: 'my-collection',
+  vector: embedding,
+  text: 'ruby programming',
+  alpha: 0.7  # 70% semantic, 30% keyword
+)
 ```
 
 ## Configuration Options

data/docs/providers/weaviate.md

@@ -18,53 +18,122 @@ docker run -p 8080:8080 semitechnologies/weaviate:latest
 
 ### Connect with Vectra
 
+You can either use the convenience constructor:
+
+```ruby
+client = Vectra.weaviate(
+  api_key: ENV["WEAVIATE_API_KEY"], # optional for local / required for cloud
+  host: "http://localhost:8080"     # or your cloud endpoint
+)
+```
+
+or configure via `Vectra::Client`:
+
 ```ruby
 client = Vectra::Client.new(
   provider: :weaviate,
-  host: 'localhost',
-  port: 8080,
-  class_name: 'Document'
+  api_key: ENV["WEAVIATE_API_KEY"],
+  host: "https://your-weaviate-instance"
 )
 ```
 
 ## Features
 
-- ✅ Upsert vectors
-- ✅ Query/search
-- ✅ Delete vectors
-- ✅ Class management
-- ✅ Metadata filtering
-- ✅ Semantic search
+- ✅ Upsert vectors into Weaviate classes (per-index `class`)
+- ✅ Vector similarity search via GraphQL `nearVector`
+- ✅ Fetch by IDs
+- ✅ Metadata filtering (exact match, ranges, arrays)
+- ✅ Namespace support via `_namespace` property
+- ✅ Delete by IDs or filter
+- ✅ List and describe classes
+- ✅ Basic stats via GraphQL `Aggregate`
+- ✅ Hybrid search (BM25 + vector similarity)
 
-## Example
+## Basic Example
 
 ```ruby
-# Initialize client
-client = Vectra::Client.new(
-  provider: :weaviate,
-  host: 'localhost',
-  port: 8080
+require "vectra"
+
+client = Vectra.weaviate(
+  api_key: ENV["WEAVIATE_API_KEY"],
+  host: "https://your-weaviate-instance"
 )
 
+index = "Document" # Weaviate class name
+
 # Upsert vectors
 client.upsert(
+  index: index,
   vectors: [
-    { id: 'doc-1', values: [0.1, 0.2, 0.3], metadata: { category: 'news' } }
-  ]
+    {
+      id: "doc-1",
+      values: [0.1, 0.2, 0.3],
+      metadata: {
+        title: "Getting started with Vectra",
+        category: "docs"
+      }
+    }
+  ],
+  namespace: "prod" # stored as _namespace property
+)
+
+# Query with metadata filter
+results = client.query(
+  index: index,
+  vector: [0.1, 0.2, 0.3],
+  top_k: 5,
+  namespace: "prod",
+  filter: { category: "docs" },
+  include_metadata: true
 )
 
-# Search
-results = client.query(vector: [0.1, 0.2, 0.3], top_k: 5)
+results.each do |match|
+  puts "#{match.id} (score=#{match.score.round(3)}): #{match.metadata["title"]}"
+end
+
+# Hybrid search (BM25 + vector)
+results = client.hybrid_search(
+  index: index,
+  vector: embedding,
+  text: 'ruby programming',
+  alpha: 0.7,  # 70% semantic, 30% keyword
+  top_k: 10
+)
 ```
 
+## Advanced Filtering
+
+Vectra maps simple Ruby hashes to Weaviate `where` filters:
+
+```ruby
+results = client.query(
+  index: "Document",
+  vector: query_embedding,
+  top_k: 10,
+  filter: {
+    category: "blog",
+    views: { "$gt" => 1000 },
+    tags: ["ruby", "vectra"] # ContainsAny
+  },
+  include_metadata: true
+)
+```
+
+Supported operators:
+
+- Equality: `{ field: "value" }` or `{ field: { "$eq" => "value" } }`
+- Inequality: `{ field: { "$ne" => "value" } }`
+- Ranges: `{ field: { "$gt" => 10 } }`, `{ "$gte" => 10 }`, `{ "$lt" => 20 }`, `{ "$lte" => 20 }`
+- Arrays: `{ field: ["a", "b"] }` (contains any), `{ field: { "$in" => ["a", "b"] } }`
+
 ## Configuration Options
 
-| Option | Type | Required | Description |
-|--------|------|----------|-------------|
-| `host` | String | Yes | Weaviate host address |
-| `port` | Integer | Yes | Weaviate port (default: 8080) |
-| `class_name` | String | No | Class name for vectors |
-| `api_key` | String | No | API key if auth is enabled |
+| Option   | Type   | Required | Description                                   |
+|----------|--------|----------|-----------------------------------------------|
+| `host`   | String | Yes      | Weaviate base URL (`http://` or `https://`)  |
+| `api_key`| String | No*      | API key if auth is enabled / cloud instances |
+
+> `api_key` is optional for local, unsecured Weaviate; required for managed/cloud deployments.
 
 ## Documentation
 

data/examples/README.md

@@ -261,6 +261,18 @@ The comprehensive demo now includes **4 additional sections** demonstrating prod
 - **Error Tracking**: Sentry and Honeybadger integration
 - Production monitoring setup
 
+In addition to the classic API, the demo now also showcases the **chainable Query Builder** style:
+
+```ruby
+results = client
+  .query("documents")
+  .vector(embedding)
+  .top_k(10)
+  .filter(category: "ruby")
+  .with_metadata
+  .execute
+```
+
 ## Tips for Production
 
 ### Performance

data/lib/vectra/batch.rb

@@ -17,6 +17,17 @@ module Vectra
   #   )
   #   puts "Upserted: #{result[:upserted_count]}"
   #
+  # @example With progress tracking
+  #   batch.upsert_async(
+  #     index: 'docs',
+  #     vectors: large_array,
+  #     on_progress: ->(stats) {
+  #       puts "Progress: #{stats[:percentage]}% (#{stats[:processed]}/#{stats[:total]})"
+  #       puts "  Chunk #{stats[:current_chunk] + 1}/#{stats[:total_chunks]}"
+  #       puts "  Success: #{stats[:success_count]}, Failed: #{stats[:failed_count]}"
+  #     }
+  #   )
+  #
   class Batch
     DEFAULT_CONCURRENCY = 4
     DEFAULT_CHUNK_SIZE = 100
@@ -38,12 +49,23 @@ module Vectra
     # @param vectors [Array<Hash>] vectors to upsert
     # @param namespace [String, nil] optional namespace
     # @param chunk_size [Integer] vectors per chunk (default: 100)
+    # @param on_progress [Proc, nil] optional callback called after each chunk completes
+    #   Callback receives hash with: processed, total, percentage, current_chunk, total_chunks, success_count, failed_count
     # @return [Hash] aggregated result with :upserted_count, :chunks, :errors
-    def upsert_async(index:, vectors:, namespace: nil, chunk_size: DEFAULT_CHUNK_SIZE)
+    #
+    # @example With progress callback
+    #   batch.upsert_async(
+    #     index: 'docs',
+    #     vectors: large_array,
+    #     on_progress: ->(stats) {
+    #       puts "Progress: #{stats[:percentage]}% (#{stats[:processed]}/#{stats[:total]})"
+    #     }
+    #   )
+    def upsert_async(index:, vectors:, namespace: nil, chunk_size: DEFAULT_CHUNK_SIZE, on_progress: nil)
       chunks = vectors.each_slice(chunk_size).to_a
       return { upserted_count: 0, chunks: 0, errors: [] } if chunks.empty?
 
-      results = process_chunks_concurrently(chunks) do |chunk|
+      results = process_chunks_concurrently(chunks, total_items: vectors.size, on_progress: on_progress) do |chunk|
         client.upsert(index: index, vectors: chunk, namespace: namespace)
       end
 
@@ -56,12 +78,14 @@ module Vectra
     # @param ids [Array<String>] IDs to delete
     # @param namespace [String, nil] optional namespace
     # @param chunk_size [Integer] IDs per chunk (default: 100)
+    # @param on_progress [Proc, nil] optional callback called after each chunk completes
+    #   Callback receives hash with: processed, total, percentage, current_chunk, total_chunks, success_count, failed_count
     # @return [Hash] aggregated result
-    def delete_async(index:, ids:, namespace: nil, chunk_size: DEFAULT_CHUNK_SIZE)
+    def delete_async(index:, ids:, namespace: nil, chunk_size: DEFAULT_CHUNK_SIZE, on_progress: nil)
       chunks = ids.each_slice(chunk_size).to_a
       return { deleted_count: 0, chunks: 0, errors: [] } if chunks.empty?
 
-      results = process_chunks_concurrently(chunks) do |chunk|
+      results = process_chunks_concurrently(chunks, total_items: ids.size, on_progress: on_progress) do |chunk|
         client.delete(index: index, ids: chunk, namespace: namespace)
       end
 
@@ -74,12 +98,14 @@ module Vectra
     # @param ids [Array<String>] IDs to fetch
     # @param namespace [String, nil] optional namespace
     # @param chunk_size [Integer] IDs per chunk (default: 100)
+    # @param on_progress [Proc, nil] optional callback called after each chunk completes
+    #   Callback receives hash with: processed, total, percentage, current_chunk, total_chunks, success_count, failed_count
     # @return [Hash<String, Vector>] merged results
-    def fetch_async(index:, ids:, namespace: nil, chunk_size: DEFAULT_CHUNK_SIZE)
+    def fetch_async(index:, ids:, namespace: nil, chunk_size: DEFAULT_CHUNK_SIZE, on_progress: nil)
       chunks = ids.each_slice(chunk_size).to_a
       return {} if chunks.empty?
 
-      results = process_chunks_concurrently(chunks) do |chunk|
+      results = process_chunks_concurrently(chunks, total_items: ids.size, on_progress: on_progress) do |chunk|
         client.fetch(index: index, ids: chunk, namespace: namespace)
       end
 
@@ -88,15 +114,43 @@ module Vectra
 
     private
 
-    def process_chunks_concurrently(chunks)
+    # rubocop:disable Metrics/AbcSize, Metrics/MethodLength, Metrics/BlockLength
+    def process_chunks_concurrently(chunks, total_items: nil, on_progress: nil)
       pool = Concurrent::FixedThreadPool.new(concurrency)
       futures = []
+      progress_mutex = Mutex.new
+      completed_count = Concurrent::AtomicFixnum.new(0)
+      success_count = Concurrent::AtomicFixnum.new(0)
+      failed_count = Concurrent::AtomicFixnum.new(0)
 
       chunks.each_with_index do |chunk, index|
         futures << Concurrent::Future.execute(executor: pool) do
-          { index: index, result: yield(chunk), error: nil }
+          result = yield(chunk)
+          success_count.increment
+          { index: index, result: result, error: nil }
         rescue StandardError => e
+          failed_count.increment
           { index: index, result: nil, error: e }
+        ensure
+          # Call progress callback when chunk completes
+          if on_progress
+            completed = completed_count.increment
+            total_size = chunks.size * chunks.first.size
+            processed = [completed * chunks.first.size, total_items || total_size].min
+            percentage = total_items ? (processed.to_f / total_items * 100).round(2) : (completed.to_f / chunks.size * 100).round(2)
+
+            progress_mutex.synchronize do
+              on_progress.call(
+                processed: processed,
+                total: total_items || total_size,
+                percentage: percentage,
+                current_chunk: completed - 1,
+                total_chunks: chunks.size,
+                success_count: success_count.value,
+                failed_count: failed_count.value
+              )
+            end
+          end
         end
       end
 
@@ -107,6 +161,7 @@ module Vectra
 
       results.sort_by { |r| r[:index] }
     end
+    # rubocop:enable Metrics/AbcSize, Metrics/MethodLength, Metrics/BlockLength
 
     def aggregate_results(results, total_vectors)
       errors = results.select { |r| r[:error] }.map { |r| r[:error] }