vectra-client 1.0.7 → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 316a75b282cab4d293dfdb3ab9f7b2220a58078008a6887e7e47e7caf6172211
-  data.tar.gz: d4a9c10c5e6862194e7fb028c254b19d63b0d2fb6fa5ef5f45b59b6a9da1a317
+  metadata.gz: 70730299dab8475b05688f7017dccd1965b87a49589c96be7633a356af8d57f2
+  data.tar.gz: 14999ebde62586578b444ba45d396cb38f1a4dd5dcbf2fd126e042ebb1c6fae5
 SHA512:
-  metadata.gz: ca06a9e2a961f6130aaa06cf42caafd900ef9fb38dc902e42c3bccb649962abe3d3cd64a50af84605df273682013f0ae90476c5096f4a3efee1ac67d3d1dc672
-  data.tar.gz: e1fb369da70479f0b3505be4c15d1b6ee7089d0f6c14bb6b58d1ae02c15b9cbb92bfb448e88697726a769d01d5ac7b9569f6085c04ee420e39e55b2ee754013c
+  metadata.gz: 69c8fa722ee4abfe3ddf6b19f8bd12f46d09edefee50612021142c3951600666ed854018e8a5c1bc33249895fec42ae18ab1d821beffa989d2700da99c28bcf4
+  data.tar.gz: f27a0df4bbcf618659297b1376ebe977588d32d5dfef76fe7c8f5f38c8780b37e055a0e8d91f6bc3ae8eb7cad88606c760d4ed5882c61756ac1495dedfc339a5

data/CHANGELOG.md

@@ -1,5 +1,9 @@
 # Changelog
 
+## [v1.0.8](https://github.com/stokry/vectra/tree/v1.0.8) (2026-01-14)
+
+[Full Changelog](https://github.com/stokry/vectra/compare/v1.0.7...v1.0.8)
+
 ## [v1.0.7](https://github.com/stokry/vectra/tree/v1.0.7) (2026-01-14)
 
 [Full Changelog](https://github.com/stokry/vectra/compare/v1.0.6...v1.0.7)

data/README.md

@@ -242,6 +242,59 @@ Real-world patterns for common use cases:
 
 👉 **[Browse all Recipes & Patterns](https://vectra-docs.netlify.app/guides/recipes/)**
 
+## Middleware System
+
+Vectra features a powerful **Rack-style middleware system** that allows you to extend functionality without modifying core code.
+
+### Quick Start
+
+```ruby
+# Global middleware (applies to all clients)
+Vectra::Client.use Vectra::Middleware::Logging
+Vectra::Client.use Vectra::Middleware::Retry, max_attempts: 5
+Vectra::Client.use Vectra::Middleware::CostTracker
+
+# Per-client middleware
+client = Vectra::Client.new(
+  provider: :qdrant,
+  middleware: [
+    Vectra::Middleware::PIIRedaction,
+    Vectra::Middleware::Instrumentation
+  ]
+)
+```
+
+### Built-in Middleware
+
+- **Logging** - Request/response logging with timing
+- **Retry** - Automatic retries with exponential backoff
+- **Instrumentation** - Metrics and monitoring integration
+- **PIIRedaction** - Automatic PII (email, phone, SSN) redaction
+- **CostTracker** - Track API usage costs per operation
+
+### Custom Middleware
+
+```ruby
+class MyAuditMiddleware < Vectra::Middleware::Base
+  def before(request)
+    # Called before the operation
+    AuditLog.create!(action: request.operation, user: Current.user)
+  end
+
+  def after(request, response)
+    # Called after successful operation
+    puts "Duration: #{response.metadata[:duration_ms]}ms"
+  end
+
+  def on_error(request, error)
+    # Called when an error occurs
+    ErrorTracker.notify(error, context: { operation: request.operation })
+  end
+end
+
+Vectra::Client.use MyAuditMiddleware
+```
+
 ## Production Patterns
 
 Vectra includes 7 production-ready patterns out of the box:

data/docs/_layouts/home.html

@@ -55,7 +55,7 @@
   <!-- Hero Section -->
   <section class="tma-hero">
     <div class="tma-hero__container">
-      <span class="tma-hero__badge">v1.0.1 — Hybrid Search & Rails Generator</span>
+      <span class="tma-hero__badge">v1.1.0 — Hybrid Search, Rails Generator & Middleware</span>
       <h1 class="tma-hero__title">
         Vector Databases,<br>
         <span class="tma-hero__title-gradient">Unified for Ruby.</span>

data/docs/api/cheatsheet.md

@@ -34,6 +34,21 @@ client = Vectra.pgvector(connection_url: ENV['DATABASE_URL'])
 client = Vectra.memory # In-memory (testing only)
 ```
 
+You can also set a **default index and namespace**:
+
+```ruby
+client = Vectra::Client.new(
+  provider: :qdrant,
+  host: 'http://localhost:6333',
+  index: 'products',
+  namespace: 'tenant-1'
+)
+
+# Now index and namespace can be omitted
+client.upsert(vectors: [...])
+client.query(vector: query_embedding, top_k: 10)
+```
+
 ### Upsert
 
 ```ruby
@@ -225,6 +240,22 @@ vector.normalize! # Mutates values
 client.upsert(index: 'documents', vectors: [vector])
 ```
 
+### Embedding Cache Helper
+
+```ruby
+cache = Vectra::Cache.new(ttl: 600, max_size: 1000)
+
+embedding = Vectra::Embeddings.fetch(
+  cache: cache,
+  model_name: "Product",
+  id: product.id,
+  input: product.description,
+  field: :description
+) do
+  EmbeddingService.generate(product.description)
+end
+```
+
 ---
 
 ## Batch Operations
@@ -318,6 +349,18 @@ results.each do |doc|
 end
 ```
 
+### Reindex All Records
+
+```ruby
+# Reindex all documents that already have embeddings
+processed = Document.reindex_vectors(
+  scope: Document.where.not(embedding: nil),
+  batch_size: 500
+)
+
+puts "Reindexed #{processed} documents"
+```
+
 ---
 
 ## Error Handling

data/docs/api/methods.md

@@ -43,7 +43,7 @@ client = Vectra::Client.new(
 Upsert vectors into an index. If a vector with the same ID exists, it will be updated.
 
 **Parameters:**
-- `index` (String) - Index/collection name
+- `index` (String) - Index/collection name (uses client's default index when omitted)
 - `vectors` (Array<Hash, Vector>) - Array of vector hashes or Vector objects
 - `namespace` (String, optional) - Namespace
 
@@ -77,7 +77,7 @@ result = client.upsert(
 Search for similar vectors using cosine similarity.
 
 **Parameters:**
-- `index` (String) - Index/collection name
+- `index` (String) - Index/collection name (uses client's default index when omitted)
 - `vector` (Array<Float>) - Query vector
 - `top_k` (Integer) - Number of results (default: 10)
 - `namespace` (String, optional) - Namespace
@@ -152,7 +152,7 @@ results = client.hybrid_search(
 Fetch vectors by their IDs.
 
 **Parameters:**
-- `index` (String) - Index/collection name
+- `index` (String) - Index/collection name (uses client's default index when omitted)
 - `ids` (Array<String>) - Array of vector IDs
 - `namespace` (String, optional) - Namespace
 
@@ -176,7 +176,7 @@ vectors['doc-1'].metadata # => { 'title' => 'Hello' }
 Update a vector's metadata or values.
 
 **Parameters:**
-- `index` (String) - Index/collection name
+- `index` (String) - Index/collection name (uses client's default index when omitted)
 - `id` (String) - Vector ID
 - `metadata` (Hash, optional) - New metadata (merged with existing)
 - `values` (Array<Float>, optional) - New vector values
@@ -202,7 +202,7 @@ client.update(
 Delete vectors.
 
 **Parameters:**
-- `index` (String) - Index/collection name
+- `index` (String) - Index/collection name (uses client's default index when omitted)
 - `ids` (Array<String>, optional) - Vector IDs to delete
 - `namespace` (String, optional) - Namespace
 - `filter` (Hash, optional) - Delete by metadata filter
@@ -231,7 +231,7 @@ client.delete(index: 'documents', delete_all: true)
 Get index statistics.
 
 **Parameters:**
-- `index` (String) - Index/collection name
+- `index` (String) - Index/collection name (uses client's default index when omitted)
 - `namespace` (String, optional) - Namespace
 
 **Returns:** `Hash` with statistics:
@@ -571,6 +571,30 @@ end
 
 ---
 
+### `Model.reindex_vectors(scope: all, batch_size: 1000, on_progress: nil)`
+
+Reindex all records for a model into the configured vector index.
+
+**Parameters:**
+- `scope` (ActiveRecord::Relation) - Records to reindex (default: `Model.all`)
+- `batch_size` (Integer) - Number of records per batch (default: 1000)
+- `on_progress` (Proc, optional) - Progress callback, receives a hash with `:processed` and `:total`
+
+**Returns:** `Integer` - Number of records processed
+
+**Example:**
+```ruby
+# Reindex all products with embeddings
+processed = Product.reindex_vectors(
+  scope: Product.where.not(embedding: nil),
+  batch_size: 500
+)
+
+puts "Reindexed #{processed} products"
+```
+
+---
+
 ## Error Handling
 
 Vectra defines specific error types:

data/docs/api/overview.md

@@ -216,3 +216,9 @@ end
 ```
 
 See [Complete API Methods Reference]({{ site.baseurl }}/api/methods/) for detailed method documentation.
+
+## Next Steps
+
+- [Providers Guide]({{ site.baseurl }}/providers/)
+- [Rails Integration Guide]({{ site.baseurl }}/guides/rails-integration/)
+- [Middleware System]({{ site.baseurl }}/guides/middleware/)

data/docs/guides/middleware.md

@@ -0,0 +1,324 @@
+---
+layout: page
+title: Middleware System
+permalink: /guides/middleware/
+---
+
+# Middleware System
+
+Vectra includes a **Rack-style middleware stack** that lets you extend the client
+without forking the gem or patching providers.
+
+You can:
+
+- Add **logging, metrics, retries, PII redaction, cost tracking**.
+- Inject **custom behaviour** before/after every operation.
+- Enable features **globally** or **per client**, same kao Faraday/Sidekiq.
+
+---
+
+## Core Concepts
+
+### Request / Response
+
+- **`Vectra::Middleware::Request`**
+  - `operation` – npr. `:upsert`, `:query`, `:fetch`, `:delete`, `:stats`, `:hybrid_search`
+  - `index` – ime indeksa (može biti `nil` ako koristiš default)
+  - `namespace` – namespace (može biti `nil`)
+  - `params` – originalni keyword parametri koje je `Vectra::Client` proslijedio
+  - `provider` – ime providera (`:pinecone`, `:qdrant`, `:pgvector`, `:memory`, …)
+  - helperi: `write_operation?`, `read_operation?`
+
+- **`Vectra::Middleware::Response`**
+  - `result` – što god provider vrati (npr. hash, `QueryResult`, itd.)
+  - `error` – iznimka ako je došlo do greške
+  - `metadata` – slobodan hash za dodatne informacije (trajanje, cost, retry_count…)
+  - helperi: `success?`, `failure?`, `raise_if_error!`, `value!`
+
+### Base i Stack
+
+- **`Vectra::Middleware::Base`**
+  - Hookovi koje možeš override-ati:
+    - `before(request)` – prije poziva providera / sljedećeg middlewarea
+    - `after(request, response)` – nakon uspješnog poziva
+    - `on_error(request, error)` – kad dođe do iznimke (error se zatim re-raise-a)
+
+- **`Vectra::Middleware::Stack`**
+  - Gradi chain oko konkretnog providera:
+    ```ruby
+    stack = Vectra::Middleware::Stack.new(provider, [Logging.new, Retry.new])
+    result = stack.call(:upsert, index: "docs", vectors: [...], provider: :qdrant)
+    ```
+  - `Stack` interno:
+    - kreira `Request`,
+    - kroz sve middlewares propagira isti `Request`,
+    - na kraju zove `provider.public_send(request.operation, **provider_params)`,
+    - vraća `Response` (s `result` ili `error`).
+
+---
+
+## Enabling Middleware
+
+### Global Middleware
+
+Primjenjuje se na **sve** `Vectra::Client` instance.
+
+```ruby
+require "vectra"
+
+# Global logging + retry + cost tracking
+Vectra::Client.use Vectra::Middleware::Logging
+Vectra::Client.use Vectra::Middleware::Retry, max_attempts: 5
+Vectra::Client.use Vectra::Middleware::CostTracker
+```
+
+Sve sljedeće `Vectra::Client.new(...)` instance će koristiti ovaj globalni stack.
+
+### Per‑Client Middleware
+
+Dodatni ili prilagođeni middleware samo za jedan klijent:
+
+```ruby
+client = Vectra::Client.new(
+  provider: :qdrant,
+  index: "products",
+  middleware: [
+    Vectra::Middleware::PIIRedaction,
+    Vectra::Middleware::Instrumentation
+  ]
+)
+```
+
+- Per‑client middleware se izvodi **nakon** globalnog, u istom chainu.
+- Redoslijed u arrayu definira redoslijed ekzekucije (zadnji je najunutarnji, tik do providera).
+
+---
+
+## Koje operacije prolaze kroz stack?
+
+Sve standardne operacije `Vectra::Client`a koriste middleware stack:
+
+- `upsert`
+- `query`
+- `fetch`
+- `update`
+- `delete`
+- `list_indexes`
+- `describe_index`
+- `stats`
+- `hybrid_search`
+
+To znači da middleware može:
+
+- logirati / instrumentirati **sve pozive** prema provideru,
+- raditi **PII redakciju** na `upsert` zahtjevima,
+- brojati i retry-ati i **read** i **write** operacije,
+- računati trošak po operaciji (npr. za billing / budžete).
+
+---
+
+## Built‑in Middleware
+
+### Logging (`Vectra::Middleware::Logging`)
+
+**Što radi:**
+- logira početak i kraj svake operacije (`operation`, `index`, `namespace`),
+- mjeri trajanje i sprema ga u `response.metadata[:duration_ms]`.
+
+**Konfiguracija:**
+
+```ruby
+# Globalno
+Vectra::Client.use Vectra::Middleware::Logging
+
+# S custom loggerom
+logger = Logger.new($stdout)
+Vectra::Client.use Vectra::Middleware::Logging, logger: logger
+```
+
+**Tipična upotreba:** debugiranje, audit logovi, korelacija s HTTP logovima.
+
+---
+
+### Retry (`Vectra::Middleware::Retry`)
+
+**Što radi:**
+- automatski retry-a transient greške:
+  - `Vectra::RateLimitError`
+  - `Vectra::ConnectionError`
+  - `Vectra::TimeoutError`
+  - `Vectra::ServerError`
+- koristi exponential ili linear backoff,
+- upisuje broj retry-a u `response.metadata[:retry_count]`.
+
+**Konfiguracija:**
+
+```ruby
+# 3 pokušaja, exponential backoff (default)
+Vectra::Client.use Vectra::Middleware::Retry
+
+# 5 pokušaja, linearni backoff
+Vectra::Client.use Vectra::Middleware::Retry,
+  max_attempts: 5,
+  backoff: :linear
+
+# Fiksni delay 1.0s
+Vectra::Client.use Vectra::Middleware::Retry,
+  max_attempts: 3,
+  backoff: 1.0
+```
+
+**Tipična upotreba:** zaštita od povremenih mrežnih problema i rate‑limit grešaka.
+
+---
+
+### Instrumentation (`Vectra::Middleware::Instrumentation`)
+
+**Što radi:**
+- emitira događaje preko postojećeg `Vectra::Instrumentation` sustava,
+- prikuplja trajanje, status, error class, dodatni `metadata`.
+
+**Primjer:**
+
+```ruby
+Vectra::Client.use Vectra::Middleware::Instrumentation
+
+Vectra.on_operation do |event|
+  # event[:operation], event[:provider], event[:duration_ms], event[:success], ...
+  StatsD.timing("vectra.#{event[:operation]}", event[:duration_ms])
+end
+```
+
+**Tipična upotreba:** integracija s Prometheus/Grafana, Datadog, New Relic…
+
+---
+
+### PII Redaction (`Vectra::Middleware::PIIRedaction`)
+
+**Što radi:**
+- prije `upsert` operacija prolazi kroz `vectors[:metadata]`,
+- prepoznaje PII pattern-e (email, phone, SSN, credit card) i zamjenjuje ih placeholderom
+  npr. `[REDACTED_EMAIL]`, `[REDACTED_PHONE]`, itd.
+
+**Primjer:**
+
+```ruby
+Vectra::Client.use Vectra::Middleware::PIIRedaction
+
+client.upsert(
+  index: "sensitive",
+  vectors: [
+    {
+      id: "user-1",
+      values: [0.1, 0.2, 0.3],
+      metadata: {
+        email: "user@example.com",
+        phone: "555-1234",
+        note:  "Contact at user@example.com"
+      }
+    }
+  ]
+)
+```
+
+Nakon `upsert`‑a, provider će vidjeti već **redaktirani** metadata.
+
+**Custom patterni:**
+
+```ruby
+patterns = {
+  credit_card: /\b\d{4}[-\s]?\d{4}[-\s]?\d{4}[-\s]?\d{4}\b/,
+  api_key: /sk-[a-zA-Z0-9]{32}/
+}
+
+Vectra::Client.use Vectra::Middleware::PIIRedaction, patterns: patterns
+```
+
+**Tipična upotreba:** GDPR, SOC2, PCI‑DSS okruženja gdje je zabranjen PII u vektor bazi.
+
+---
+
+### CostTracker (`Vectra::Middleware::CostTracker`)
+
+**Što radi:**
+- procjenjuje trošak po operaciji na temelju providera i tipa operacije (`read` / `write`),
+- upisuje trošak u `response.metadata[:cost_usd]`,
+- opcionalno zove `on_cost` callback za real‑time praćenje.
+
+**Primjer:**
+
+```ruby
+Vectra::Client.use Vectra::Middleware::CostTracker,
+  on_cost: ->(event) {
+    puts "💰 Cost: $#{event[:cost_usd].round(6)} for #{event[:operation]} (#{event[:provider]})"
+  }
+```
+
+**Custom pricing:**
+
+```ruby
+pricing = {
+  pinecone: { read: 0.0001, write: 0.0002 },
+  qdrant:   { read: 0.00005, write: 0.0001 }
+}
+
+Vectra::Client.use Vectra::Middleware::CostTracker, pricing: pricing
+```
+
+**Tipična upotreba:** unutarnji billing, budget guardrails, cost dashboards.
+
+---
+
+## Custom Middleware
+
+Najjednostavniji način je naslijediti `Vectra::Middleware::Base` i override-ati hookove:
+
+```ruby
+class MyAuditMiddleware < Vectra::Middleware::Base
+  def before(request)
+    AuditLog.create!(
+      operation: request.operation,
+      index:     request.index,
+      namespace: request.namespace,
+      provider:  request.provider
+    )
+  end
+
+  def after(_request, response)
+    puts "Duration: #{response.metadata[:duration_ms]}ms"
+  end
+
+  def on_error(request, error)
+    ErrorTracker.notify(error, context: { operation: request.operation })
+  end
+end
+
+Vectra::Client.use MyAuditMiddleware
+```
+
+**Savjeti:**
+
+- Ne mijenjaj strukturu `request.params` na način koji provider ne očekuje.
+- Svoje pomoćne podatke stavljaj u `response.metadata` ili `request.metadata`.
+- Ako hvataš greške u `on_error`, **nemoj ih gutati** – middleware stack će ih ponovo baciti.
+
+---
+
+## Primjer: `examples/middleware_demo.rb`
+
+U repozitoriju imaš kompletan demo:
+
+- konfigurira globalni stack (`Logging`, `Retry`, `CostTracker`),
+- pokazuje per‑client PII redaction,
+- definira custom `TimingMiddleware`,
+- demonstrira kako izgleda output u konzoli.
+
+Pokretanje:
+
+```bash
+bundle exec ruby examples/middleware_demo.rb
+```
+
+Ovaj demo je dobar “živi” primjer kako kombinirati više middleware-a u praksi.
+
+

data/examples/middleware_demo.rb

@@ -0,0 +1,103 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+# Middleware System Demo
+#
+# This script demonstrates the new middleware system in Vectra.
+# Run with: ruby examples/middleware_demo.rb
+
+require_relative "../lib/vectra"
+
+# Configure global middleware
+puts "🎯 Configuring global middleware..."
+Vectra::Client.use Vectra::Middleware::Logging
+Vectra::Client.use Vectra::Middleware::Retry, max_attempts: 3
+Vectra::Client.use Vectra::Middleware::CostTracker, on_cost: ->(event) {
+  puts "💰 Cost: $#{event[:cost_usd].round(6)} for #{event[:operation]}"
+}
+
+# Create client
+puts "\n📦 Creating client with Memory provider..."
+client = Vectra::Client.new(
+  provider: :memory,
+  index: "demo"
+)
+
+# Example 1: Upsert with middleware
+puts "\n🔄 Example 1: Upsert with middleware stack"
+puts "=" * 50
+client.upsert(
+  index: "demo",
+  vectors: [
+    { id: "doc-1", values: [0.1, 0.2, 0.3], metadata: { title: "Ruby" } },
+    { id: "doc-2", values: [0.4, 0.5, 0.6], metadata: { title: "Python" } }
+  ]
+)
+
+# Example 2: Query with middleware
+puts "\n🔍 Example 2: Query with middleware stack"
+puts "=" * 50
+results = client.query(
+  index: "demo",
+  vector: [0.1, 0.2, 0.3],
+  top_k: 2
+)
+puts "Found #{results.size} results"
+
+# Example 3: Per-client middleware
+puts "\n🎨 Example 3: Per-client middleware (PII Redaction)"
+puts "=" * 50
+pii_client = Vectra::Client.new(
+  provider: :memory,
+  index: "sensitive",
+  middleware: [Vectra::Middleware::PIIRedaction]
+)
+
+pii_client.upsert(
+  index: "sensitive",
+  vectors: [
+    {
+      id: "user-1",
+      values: [0.1, 0.2, 0.3],
+      metadata: {
+        email: "user@example.com",
+        phone: "555-1234",
+        note: "Contact at user@example.com"
+      }
+    }
+  ]
+)
+
+# Fetch to see redacted data
+fetched = pii_client.fetch(index: "sensitive", ids: ["user-1"])
+puts "Original email: user@example.com"
+puts "Redacted: #{fetched["user-1"].metadata[:email]}"
+puts "Redacted note: #{fetched["user-1"].metadata[:note]}"
+
+# Example 4: Custom middleware
+puts "\n🛠️  Example 4: Custom middleware"
+puts "=" * 50
+
+class TimingMiddleware < Vectra::Middleware::Base
+  def before(request)
+    puts "⏱️  Starting #{request.operation}..."
+  end
+
+  def after(request, response)
+    duration = response.metadata[:duration_ms] || 0
+    puts "✅ Completed in #{duration.round(2)}ms"
+  end
+end
+
+custom_client = Vectra::Client.new(
+  provider: :memory,
+  index: "custom",
+  middleware: [TimingMiddleware]
+)
+
+custom_client.upsert(
+  index: "custom",
+  vectors: [{ id: "test", values: [1, 2, 3] }]
+)
+
+puts "\n✨ Demo complete!"