vectra-client 1.0.6 → 1.0.8

data/docs/guides/testing.md

@@ -0,0 +1,211 @@
+---
+layout: page
+title: Testing Guide
+permalink: /guides/testing/
+---
+
+# Testing Guide
+
+How to test code that uses Vectra without running a real vector database.
+
+Vectra ships with a **Memory provider** – an in-memory vector store designed for **RSpec/Minitest, local dev, and CI**.
+
+> Not for production use. All data is stored in memory and lost when the process exits.
+
+---
+
+## 1. Configure Vectra for Tests
+
+### Option A: Global Config (Rails)
+
+```ruby
+# config/initializers/vectra.rb
+require 'vectra'
+
+Vectra.configure do |config|
+  if Rails.env.test?
+    config.provider = :memory
+  else
+    config.provider = :qdrant
+    config.host     = ENV.fetch('QDRANT_HOST', 'http://localhost:6333')
+    config.api_key  = ENV['QDRANT_API_KEY']
+  end
+end
+```
+
+Then in your application code and tests:
+
+```ruby
+client = Vectra::Client.new
+```
+
+### Option B: Direct Construction in Tests
+
+```ruby
+require 'vectra'
+
+RSpec.describe 'My vector code' do
+  let(:client) { Vectra.memory }
+
+  it 'searches using in-memory provider' do
+    client.upsert(
+      index: 'documents',
+      vectors: [
+        { id: 'doc-1', values: [0.1, 0.2, 0.3], metadata: { title: 'Hello' } }
+      ]
+    )
+
+    results = client.query(index: 'documents', vector: [0.1, 0.2, 0.3], top_k: 5)
+    expect(results.ids).to include('doc-1')
+  end
+end
+```
+
+---
+
+## 2. Reset State Between Tests
+
+The Memory provider exposes `clear!` to wipe all in-memory data.
+
+```ruby
+# spec/support/vectra_memory.rb
+RSpec.configure do |config|
+  config.before(:each, vectra: :memory) do
+    Vectra::Providers::Memory.instance.clear!
+  end
+end
+```
+
+Usage:
+
+```ruby
+RSpec.describe SearchService, vectra: :memory do
+  it 'returns expected results' do
+    # state is clean here
+  end
+end
+```
+
+If you use global `config.provider = :memory` in test env, this hook ensures tests are isolated.
+
+---
+
+## 3. Testing Application Code (Service Example)
+
+Assume you have a simple service that wraps Vectra:
+
+```ruby
+# app/services/search_service.rb
+class SearchService
+  def self.search(query:, limit: 10)
+    client = Vectra::Client.new
+
+    embedding = EmbeddingService.generate(query)
+
+    client.query(
+      index: 'documents',
+      vector: embedding,
+      top_k: limit,
+      include_metadata: true
+    )
+  end
+end
+```
+
+You can test it **without** any external DB:
+
+```ruby
+# spec/services/search_service_spec.rb
+require 'rails_helper'
+
+RSpec.describe SearchService, vectra: :memory do
+  let(:client) { Vectra.memory }
+
+  before do
+    # Seed in-memory vectors
+    client.upsert(
+      index: 'documents',
+      vectors: [
+        { id: 'doc-1', values: [1.0, 0.0, 0.0], metadata: { title: 'Ruby Vectors' } },
+        { id: 'doc-2', values: [0.0, 1.0, 0.0], metadata: { title: 'PostgreSQL' } }
+      ]
+    )
+
+    # Stub client used inside SearchService to our memory client
+    allow(Vectra::Client).to receive(:new).and_return(client)
+
+    # Stub embedding service to return a deterministic vector
+    allow(EmbeddingService).to receive(:generate).and_return([1.0, 0.0, 0.0])
+  end
+
+  it 'returns most relevant document first' do
+    results = SearchService.search(query: 'ruby vectors', limit: 2)
+
+    expect(results.first.metadata['title']).to eq('Ruby Vectors')
+  end
+end
+```
+
+---
+
+## 4. Testing `has_vector` Models
+
+Example model:
+
+```ruby
+# app/models/document.rb
+class Document < ApplicationRecord
+  include Vectra::ActiveRecord
+
+  has_vector :embedding,
+    provider: :memory,
+    index: 'documents',
+    dimension: 3,
+    metadata_fields: [:title]
+end
+```
+
+### Model Spec
+
+```ruby
+# spec/models/document_spec.rb
+require 'rails_helper'
+
+RSpec.describe Document, vectra: :memory do
+  it 'indexes on create and can be searched' do
+    doc = Document.create!(
+      title: 'Hello',
+      embedding: [1.0, 0.0, 0.0]
+    )
+
+    results = Document.vector_search(
+      embedding: [1.0, 0.0, 0.0],
+      limit: 5
+    )
+
+    expect(results.map(&:id)).to include(doc.id)
+  end
+end
+```
+
+---
+
+## 5. CI Considerations
+
+- ✅ No external services needed (no Docker, no cloud credentials)
+- ✅ Fast and deterministic tests
+- ✅ Same Vectra API as real providers
+
+Recommended CI pattern:
+
+1. Use `provider: :memory` for the majority of tests.
+2. Have a **small set of integration tests** (separate job) that hit a real provider (e.g. Qdrant in Docker) if needed.
+
+---
+
+## 6. Useful Links
+
+- [Memory Provider Docs](/providers/memory/)
+- [Recipes & Patterns](/guides/recipes/)
+- [Rails Integration Guide](/guides/rails-integration/)
+- [API Cheatsheet](/api/cheatsheet/)

data/docs/providers/selection.md

@@ -0,0 +1,215 @@
+---
+layout: page
+title: Provider Selection Guide
+permalink: /providers/selection/
+---
+
+# Provider Selection Guide
+
+Kratki vodič kako odabrati pravog providera za tvoj use-case.
+
+Vectra podržava 5 providera:
+
+- **Pinecone** – managed cloud
+- **Qdrant** – open source, self-host + cloud
+- **Weaviate** – AI-native, GraphQL, open source
+- **pgvector** – PostgreSQL ekstenzija
+- **Memory** – in-memory, samo za testing
+
+---
+
+## Brzi "decision tree"
+
+### 1. Već koristiš PostgreSQL i želiš minimalne promjene
+
+**Preporuka:** `pgvector`
+
+Koristi pgvector ako:
+
+- Sve ti je već u Postgresu
+- Ne želiš dodatni servis u infrastrukturi
+- Hoćeš **SQL + ACID** i transakcije
+- Dataset je **srednji** (deseci / stotine tisuća do par milijuna vektora)
+
+```ruby
+# Gemfile
+# pg + pgvector extension u bazi
+
+Vectra.configure do |config|
+  config.provider = :pgvector
+  config.host     = ENV['DATABASE_URL']
+end
+
+client = Vectra::Client.new
+```
+
+**Plusevi:**
+- Nema dodatne baze → manje ops-a
+- Možeš raditi JOIN-ove, transakcije, migrations kao i inače
+
+**Minusi:**
+- Nije "pure" vektorska baza (manje specijaliziranih featurea)
+- Scaling je vezan uz Postgres
+
+---
+
+### 2. Želiš open source i mogućnost self-hosta
+
+**Preporuka:** `Qdrant` (ili `Weaviate` ako ti treba GraphQL i AI-native featurei)
+
+Koristi **Qdrant** ako:
+
+- Želiš **OSS** i full kontrolu
+- Hoćeš odličan performance i dobar filter engine
+- Možeš vrtiti Docker/Kubernetes ili koristiti Qdrant Cloud
+
+```ruby
+Vectra.configure do |config|
+  config.provider = :qdrant
+  config.host     = ENV.fetch('QDRANT_HOST', 'http://localhost:6333')
+  config.api_key  = ENV['QDRANT_API_KEY'] # opcionalno za lokalni
+end
+
+client = Vectra::Client.new
+```
+
+Koristi **Weaviate** ako:
+
+- Želiš **GraphQL API** i bogat schema model
+- Hoćeš AI-native feature (ugrađeni vektorizatori, hybrid search, cross-references)
+
+```ruby
+Vectra.configure do |config|
+  config.provider = :weaviate
+  config.host     = ENV['WEAVIATE_HOST']
+  config.api_key  = ENV['WEAVIATE_API_KEY']
+end
+```
+
+---
+
+### 3. Želiš managed cloud i "zero ops"
+
+**Preporuka:** `Pinecone`
+
+Koristi Pinecone ako:
+
+- Ne želiš brinuti o indexima, sharding-u, backupima
+- Imaš veće volumene i trebaš stabilan cloud servis
+- Hoćeš multi-region, SLA, enterprise podršku
+
+```ruby
+Vectra.configure do |config|
+  config.provider   = :pinecone
+  config.api_key    = ENV['PINECONE_API_KEY']
+  config.environment = ENV['PINECONE_ENVIRONMENT'] # npr. 'us-west-4'
+end
+
+client = Vectra::Client.new
+```
+
+**Plusevi:**
+- Najmanje ops-a
+- Dobri performance i scaling out-of-the-box
+
+**Minusi:**
+- Vezan si na cloud provider
+- Plaćeni servis
+
+---
+
+### 4. Samo želiš nešto što radi lokalno za prototipiranje / testing
+
+**Preporuka:** `Memory` ili lokalni `Qdrant`
+
+- **Memory provider** (`:memory`):
+  - Super za RSpec / Minitest i CI
+  - Nema vanjskih ovisnosti
+
+```ruby
+Vectra.configure do |config|
+  config.provider = :memory if Rails.env.test?
+end
+
+client = Vectra::Client.new
+```
+
+- **Lokalni Qdrant**:
+  - Pokreneš `docker run qdrant/qdrant`
+  - Koristiš pravi vektorski engine i lokalni disk
+
+```ruby
+Vectra.configure do |config|
+  config.provider = :qdrant
+  config.host     = 'http://localhost:6333'
+end
+```
+
+---
+
+## Tipični scenariji
+
+### E-commerce (1000–1M proizvoda)
+
+- **Ako već koristiš Postgres** → `pgvector`
+- **Ako želiš dedicated vektorsku bazu** → `Qdrant`
+
+Za Rails primjer pogledaj [Rails Integration Guide](/guides/rails-integration/) i [Recipes & Patterns](/guides/recipes/).
+
+---
+
+### SaaS aplikacija s multi-tenant podrškom
+
+- **Qdrant** ili **Weaviate** zbog dobrog filteringa i fleksibilnosti
+- **Pgvector** ako je sve već u jednoj Postgres bazi
+
+Koristi **namespace per tenant** + metadata `tenant_id` (primjer u [Recipes](/guides/recipes/#multi-tenant-saas-namespace-isolation)).
+
+---
+
+### RAG chatbot / dokumentacija / knowledge base
+
+- **Qdrant** ili **Weaviate** zbog dobrog rada s tekstom i filterima
+- **Pinecone** ako želiš managed cloud bez brige
+
+Važno:
+
+- Čunkaj dokumente u manje dijelove (npr. 200–500 tokena)
+- Spremi `document_id`, `chunk_index`, `source_url` u metadata
+
+Primjer: [RAG Chatbot recipe](/guides/recipes/#rag-chatbot-context-retrieval).
+
+---
+
+### Interni alati / reporting / dashboards
+
+- **Pgvector** je često najbolji izbor:
+  - Podaci već u Postgresu
+  - Možeš kombinirati vektorske upite i klasične SQL joinove
+
+---
+
+## Što ako želim promijeniti providera kasnije?
+
+To je upravo najveća prednost Vectre 🙂
+
+1. U konfiguraciji promijeniš `config.provider` + relevantne kredencijale
+2. Opcionalno pokreneš migraciju podataka (dual-write ili batch migracija)
+3. Tvoj aplikacijski kod (`client.upsert`, `client.query`, `has_vector`) ostaje isti
+
+Za detaljan primjer pogledaj:
+
+- [Zero-Downtime Provider Migration](/guides/recipes/#zero-downtime-provider-migration)
+
+---
+
+## Sažetak preporuka
+
+- **Samo Postgres, minimalne promjene:** `pgvector`
+- **OSS + self-host, jak filter engine:** `Qdrant`
+- **AI-native, GraphQL:** `Weaviate`
+- **Managed cloud, zero ops:** `Pinecone`
+- **Testing / CI:** `Memory`
+
+Sve ove opcije koristiš kroz **isti Vectra API**, pa kasnije možeš mijenjati providera s minimalnim promjenama u kodu.
+

data/lib/vectra/active_record.rb

@@ -2,8 +2,9 @@
 
 require "active_support/concern"
 
-# Ensure Client and Providers are loaded (for Rails autoloading compatibility)
+# Ensure Client and supporting classes are loaded (for Rails autoloading compatibility)
 require_relative "client" unless defined?(Vectra::Client)
+require_relative "batch" unless defined?(Vectra::Batch)
 
 module Vectra
   # ActiveRecord integration for vector embeddings
@@ -26,6 +27,7 @@ module Vectra
   #   # Search similar documents
   #   results = Document.vector_search([0.1, 0.2, ...], limit: 10)
   #
+  # rubocop:disable Metrics/ModuleLength
   module ActiveRecord
     extend ActiveSupport::Concern
 
@@ -86,6 +88,54 @@ module Vectra
         end
       end
 
+      # Reindex all vectors for this model using current configuration.
+      #
+      # @param scope [ActiveRecord::Relation] records to reindex (default: all)
+      # @param batch_size [Integer] number of records per batch
+      # @param on_progress [Proc, nil] optional callback called after each batch
+      #   Receives a hash with :processed and :total keys (and any other stats from Batch)
+      #
+      # @return [Integer] number of records processed
+      def reindex_vectors(scope: all, batch_size: 1_000, on_progress: nil)
+        config = _vectra_config
+        client = vectra_client
+        batch = Vectra::Batch.new(client)
+
+        processed = 0
+
+        scope.in_batches(of: batch_size).each do |relation|
+          records = relation.to_a
+
+          vectors = records.map do |record|
+            vector = record.send(config[:attribute])
+            next if vector.nil?
+
+            metadata = config[:metadata_fields].each_with_object({}) do |field, hash|
+              hash[field.to_s] = record.send(field) if record.respond_to?(field)
+            end
+
+            {
+              id: "#{config[:index]}_#{record.id}",
+              values: vector,
+              metadata: metadata
+            }
+          end.compact
+
+          next if vectors.empty?
+
+          batch.upsert_async(
+            index: config[:index],
+            vectors: vectors,
+            namespace: nil,
+            on_progress: on_progress
+          )
+
+          processed += vectors.size
+        end
+
+        processed
+      end
+
       # Search vectors
       #
       # @api private
@@ -195,4 +245,5 @@ module Vectra
       "#{self.class._vectra_config[:index]}_#{id}"
     end
   end
+  # rubocop:enable Metrics/ModuleLength
 end

data/lib/vectra/batch.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require "concurrent"
+require_relative "query_result" unless defined?(Vectra::QueryResult)
 
 module Vectra
   # Batch operations with concurrent processing
@@ -112,6 +113,74 @@ module Vectra
       merge_fetch_results(results)
     end
 
+    # Perform async batch query with concurrent requests
+    #
+    # Useful for finding similar items for multiple vectors at once (e.g., recommendation engine).
+    #
+    # @param index [String] the index name
+    # @param vectors [Array<Array<Float>>] query vectors
+    # @param top_k [Integer] number of results per query (default: 10)
+    # @param namespace [String, nil] optional namespace
+    # @param filter [Hash, nil] metadata filter
+    # @param include_values [Boolean] include vector values in response
+    # @param include_metadata [Boolean] include metadata in response
+    # @param chunk_size [Integer] queries per chunk for progress tracking (default: 10)
+    # @param on_progress [Proc, nil] optional callback called after each chunk completes
+    #   Callback receives hash with: processed, total, percentage, current_chunk, total_chunks, success_count, failed_count
+    # @return [Array<QueryResult>] array of query results, one per input vector
+    #
+    # @example Find similar items for multiple products
+    #   product_embeddings = products.map(&:embedding)
+    #   results = batch.query_async(
+    #     index: 'products',
+    #     vectors: product_embeddings,
+    #     top_k: 5,
+    #     on_progress: ->(stats) {
+    #       puts "Processed #{stats[:processed]}/#{stats[:total]} queries"
+    #     }
+    #   )
+    #   results.each_with_index do |result, i|
+    #     puts "Similar to product #{i}: #{result.ids}"
+    #   end
+    #
+    def query_async(index:, vectors:, top_k: 10, namespace: nil, filter: nil,
+                    include_values: false, include_metadata: true,
+                    chunk_size: 10, on_progress: nil)
+      return [] if vectors.empty?
+
+      # Process queries in chunks for progress tracking
+      chunks = vectors.each_slice(chunk_size).to_a
+      results = process_chunks_concurrently(chunks, total_items: vectors.size, on_progress: on_progress) do |chunk|
+        # Execute queries sequentially within chunk (each query is already fast)
+        chunk.map do |vector|
+          client.query(
+            index: index,
+            vector: vector,
+            top_k: top_k,
+            namespace: namespace,
+            filter: filter,
+            include_values: include_values,
+            include_metadata: include_metadata
+          )
+        end
+      end
+
+      # Flatten results and handle errors
+      all_results = []
+      results.each_with_index do |chunk_result, chunk_index|
+        if chunk_result[:error]
+          # On error, return empty QueryResult for each vector in chunk
+          # Use actual chunk size (last chunk might be smaller)
+          actual_chunk_size = chunk_index < chunks.size ? chunks[chunk_index].size : chunk_size
+          actual_chunk_size.times { all_results << QueryResult.new(matches: []) }
+        else
+          all_results.concat(chunk_result[:result] || [])
+        end
+      end
+
+      all_results
+    end
+
     private
 
     # rubocop:disable Metrics/AbcSize, Metrics/MethodLength, Metrics/BlockLength

data/lib/vectra/cache.rb

@@ -258,4 +258,53 @@ module Vectra
       "#{index}:f:#{id}:#{namespace || 'default'}"
     end
   end
+
+  # Helper for caching embeddings based on model, record ID and input text.
+  #
+  # @example
+  #   cache = Vectra::Cache.new(ttl: 600, max_size: 1000)
+  #
+  #   embedding = Vectra::Embeddings.fetch(
+  #     cache: cache,
+  #     model_name: "Product",
+  #     id: product.id,
+  #     input: product.description,
+  #     field: :description
+  #   ) do
+  #     EmbeddingService.generate(product.description)
+  #   end
+  #
+  module Embeddings
+    module_function
+
+    # Build a stable cache key for an embedding.
+    #
+    # @param model_name [String] model class name (e.g. "Product")
+    # @param id [Integer, String] record ID
+    # @param input [String] raw input used for embedding
+    # @param field [Symbol, String, nil] optional field name
+    #
+    # @return [String] cache key
+    def cache_key(model_name:, id:, input:, field: nil)
+      field_part = field ? field.to_s : "default"
+      base = "#{model_name}:#{field_part}:#{id}:#{input}"
+      digest = Digest::SHA256.hexdigest(base)[0, 32]
+      "emb:#{model_name}:#{field_part}:#{digest}"
+    end
+
+    # Fetch an embedding from cache or compute and store it.
+    #
+    # @param cache [Vectra::Cache] cache instance
+    # @param model_name [String] model class name
+    # @param id [Integer, String] record ID
+    # @param input [String] input used for embedding
+    # @param field [Symbol, String, nil] optional field name
+    #
+    # @yield block that computes the embedding when not cached
+    # @return [Object] cached or computed embedding
+    def fetch(cache:, model_name:, id:, input:, field: nil, &block)
+      key = cache_key(model_name: model_name, id: id, input: input, field: field)
+      cache.fetch(key, &block)
+    end
+  end
 end