vectra-client 1.0.8 → 1.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '05778206f66d4e2ead830f5b7c5885f6336b46d36af696f3df5a56eb26163c5f'
-  data.tar.gz: 1ae5814cd10df006ecf5568f34e457400b072ecc8a06dbf85a4e4f5b94120fb9
+  metadata.gz: 0f1fd06b5874c1bc1da1244fb05321cda4a4759e234d9175159c5a6ba7ed8d40
+  data.tar.gz: 9d69e983f4ef5ed4d6bdd58e7a003e06045f3570ba1b3329587f82bcf07902a3
 SHA512:
-  metadata.gz: d25889a4626c9caa9edf7ccc285c849799de87cf38a8a7a6f56a49edb46da592b28bc0232960e2612fd0d9199e990c21bd5fe6b88f15351d8ff38c73bd7afc84
-  data.tar.gz: d4ef355089814ed4caf8c61e216b9554f23035df12be72a65c778b3230bd2a32f16ecf4c50de3bc33e40dc323f1aee83d7a891ad962bec105585c8bc6b30902b
+  metadata.gz: 7c1911470f96d83470dd98cdc6bb3e6c438a11e71b7317265388d22ae4c3792cea3e595661c927a42587b6008fc891940d511815932b3da478bfeb056ccf8c30
+  data.tar.gz: a3a17643736f8b9a19b92c81a87ce8749a4e55798085e474ef1bc2f13ab9f8c688b2bc7537467a851e59a31cf06390444a064a9d525cff8efb175dbae1de7c7c

data/CHANGELOG.md

@@ -1,5 +1,42 @@
 # Changelog
 
+## [v1.1.0](https://github.com/stokry/vectra/tree/v1.1.0) (2026-01-15)
+
+[Full Changelog](https://github.com/stokry/vectra/compare/v1.0.8...v1.1.0)
+
+### 🎉 Major Feature: Middleware System
+
+This release introduces a **Rack-style middleware system** for all vector database operations.
+
+#### Added
+
+- **Middleware Stack** - All client operations now route through a composable middleware pipeline
+- **5 Built-in Middleware**:
+  - `Vectra::Middleware::Logging` - Structured logs with timing for all operations
+  - `Vectra::Middleware::Retry` - Automatic retry with exponential/linear backoff for transient errors
+  - `Vectra::Middleware::Instrumentation` - Hooks for metrics and APM integration
+  - `Vectra::Middleware::PIIRedaction` - Automatic PII redaction (email, phone, SSN, credit cards)
+  - `Vectra::Middleware::CostTracker` - Track API costs per operation with callbacks
+- **Request/Response Objects** - Type-safe objects with metadata attachment
+- **Extensible Framework** - Create custom middleware by extending `Vectra::Middleware::Base`
+- **Global & Per-Client Middleware** - Apply middleware globally (`Client.use`) or per-instance (`new(middleware: [...])`)
+
+#### Changed
+
+- All client operations (`upsert`, `query`, `fetch`, `update`, `delete`, `stats`, `list_indexes`, etc.) now route through middleware stack for consistency
+- Middleware has complete visibility into all client operations
+
+#### Documentation
+
+- Added comprehensive middleware section to README
+- Created `examples/middleware_demo.rb` demonstrating all 5 built-in middleware
+- Full YARD documentation for all middleware classes
+- Published [middleware guide](https://dev.to/stokry/rack-style-middleware-for-vector-databases-in-ruby-vectra-client-110-2jh3) on Dev.to
+
+#### Migration Notes
+
+No breaking changes. Middleware is opt-in - existing code works without modification.
+
 ## [v1.0.8](https://github.com/stokry/vectra/tree/v1.0.8) (2026-01-14)
 
 [Full Changelog](https://github.com/stokry/vectra/compare/v1.0.7...v1.0.8)

data/README.md

@@ -109,6 +109,14 @@ results = client.hybrid_search(
   text: 'ruby programming',
   alpha: 0.7  # 70% semantic, 30% keyword
 )
+
+# Text-only search (keyword search without embeddings)
+# Supported by: Qdrant, Weaviate, pgvector
+results = client.text_search(
+  index: 'products',
+  text: 'iPhone 15 Pro',
+  top_k: 10
+)
 ```
 
 ## Provider Examples
@@ -242,6 +250,59 @@ Real-world patterns for common use cases:
 
 👉 **[Browse all Recipes & Patterns](https://vectra-docs.netlify.app/guides/recipes/)**
 
+## Middleware System
+
+Vectra features a powerful **Rack-style middleware system** that allows you to extend functionality without modifying core code.
+
+### Quick Start
+
+```ruby
+# Global middleware (applies to all clients)
+Vectra::Client.use Vectra::Middleware::Logging
+Vectra::Client.use Vectra::Middleware::Retry, max_attempts: 5
+Vectra::Client.use Vectra::Middleware::CostTracker
+
+# Per-client middleware
+client = Vectra::Client.new(
+  provider: :qdrant,
+  middleware: [
+    Vectra::Middleware::PIIRedaction,
+    Vectra::Middleware::Instrumentation
+  ]
+)
+```
+
+### Built-in Middleware
+
+- **Logging** - Request/response logging with timing
+- **Retry** - Automatic retries with exponential backoff
+- **Instrumentation** - Metrics and monitoring integration
+- **PIIRedaction** - Automatic PII (email, phone, SSN) redaction
+- **CostTracker** - Track API usage costs per operation
+
+### Custom Middleware
+
+```ruby
+class MyAuditMiddleware < Vectra::Middleware::Base
+  def before(request)
+    # Called before the operation
+    AuditLog.create!(action: request.operation, user: Current.user)
+  end
+
+  def after(request, response)
+    # Called after successful operation
+    puts "Duration: #{response.metadata[:duration_ms]}ms"
+  end
+
+  def on_error(request, error)
+    # Called when an error occurs
+    ErrorTracker.notify(error, context: { operation: request.operation })
+  end
+end
+
+Vectra::Client.use MyAuditMiddleware
+```
+
 ## Production Patterns
 
 Vectra includes 7 production-ready patterns out of the box:
@@ -254,6 +315,20 @@ Vectra includes 7 production-ready patterns out of the box:
 - **Health Checks** - `healthy?`, `ping`, and `health_check` methods
 - **Instrumentation** - Datadog, New Relic, Sentry, Honeybadger support
 
+## Roadmap
+
+High-level roadmap for `vectra-client`:
+
+- **1.x (near term)**
+  - Reranking middleware built on top of the existing Rack-style middleware stack.
+  - Additional middleware building blocks (sampling, tracing, score normalization).
+  - Smoother Rails UX for multi-tenant setups and larger demos (e‑commerce, RAG, recommendations).
+- **Mid term**
+  - Additional providers where it makes sense and stays maintainable.
+  - Deeper documentation and recipes around reranking and hybrid search.
+
+For a more detailed, always-up-to-date version, see the online roadmap: https://vectra-docs.netlify.app/guides/roadmap/
+
 ## Development
 
 ```bash

data/docs/_layouts/home.html

@@ -55,7 +55,7 @@
   <!-- Hero Section -->
   <section class="tma-hero">
     <div class="tma-hero__container">
-      <span class="tma-hero__badge">v1.0.1 — Hybrid Search & Rails Generator</span>
+      <span class="tma-hero__badge">v1.1.0 — Hybrid Search, Rails Generator & Middleware</span>
       <h1 class="tma-hero__title">
         Vector Databases,<br>
         <span class="tma-hero__title-gradient">Unified for Ruby.</span>

data/docs/_layouts/page.html

@@ -91,6 +91,13 @@
           <li><a href="https://github.com/stokry/vectra/issues" class="tma-sidebar__link" target="_blank">Report Issue ↗</a></li>
         </ul>
       </div>
+
+      <div class="tma-sidebar__section">
+        <h3 class="tma-sidebar__title">Resources</h3>
+        <ul class="tma-sidebar__list">
+          <li><a href="{{ site.baseurl }}/guides/roadmap" class="tma-sidebar__link {% if page.url == '/guides/roadmap/' %}tma-sidebar__link--active{% endif %}">Roadmap</a></li>
+        </ul>
+      </div>
     </aside>
 
     <!-- Main Content -->

data/docs/api/cheatsheet.md

@@ -98,6 +98,23 @@ results = client.hybrid_search(
 
 Supported providers: Qdrant ✅, Weaviate ✅, pgvector ✅, Pinecone ⚠️
 
+### Text Search (keyword-only, no embeddings)
+
+```ruby
+results = client.text_search(
+  index: 'products',
+  text: 'iPhone 15 Pro',
+  top_k: 10,
+  filter: { category: 'electronics' }
+)
+
+results.each do |match|
+  puts "#{match.id} (score=#{match.score.round(3)}): #{match.metadata['title']}"
+end
+```
+
+Supported providers: Qdrant ✅ (BM25), Weaviate ✅ (BM25), pgvector ✅ (PostgreSQL full-text)
+
 ### Fetch
 
 ```ruby

data/docs/api/methods.md

@@ -147,6 +147,51 @@ results = client.hybrid_search(
 
 ---
 
+### `client.text_search(index:, text:, top_k: 10, namespace: nil, filter: nil, include_values: false, include_metadata: true)`
+
+Text-only search (keyword search without requiring embeddings).
+
+**Parameters:**
+- `index` (String) - Index/collection name (uses client's default index when omitted)
+- `text` (String) - Text query for keyword search
+- `top_k` (Integer) - Number of results (default: 10)
+- `namespace` (String, optional) - Namespace
+- `filter` (Hash, optional) - Metadata filter
+- `include_values` (Boolean) - Include vector values (default: false)
+- `include_metadata` (Boolean) - Include metadata (default: true)
+
+**Returns:** `Vectra::QueryResult`
+
+**Provider Support:**
+- ✅ Qdrant (BM25)
+- ✅ Weaviate (BM25)
+- ✅ pgvector (PostgreSQL full-text search)
+- ✅ Memory (simple keyword matching - for testing only)
+- ❌ Pinecone (not supported - use sparse vectors instead)
+
+**Example:**
+```ruby
+# Keyword search for exact matches
+results = client.text_search(
+  index: 'products',
+  text: 'iPhone 15 Pro',
+  top_k: 10,
+  filter: { category: 'electronics' }
+)
+
+results.each do |match|
+  puts "#{match.id}: #{match.score} - #{match.metadata['title']}"
+end
+```
+
+**Use Cases:**
+- Product name search (exact matches)
+- Function/class name search in documentation
+- Keyword-based filtering when semantic search is not needed
+- Faster search when embeddings are not available
+
+---
+
 ### `client.fetch(index:, ids:, namespace: nil)`
 
 Fetch vectors by their IDs.

data/docs/api/overview.md

@@ -216,3 +216,9 @@ end
 ```
 
 See [Complete API Methods Reference]({{ site.baseurl }}/api/methods/) for detailed method documentation.
+
+## Next Steps
+
+- [Providers Guide]({{ site.baseurl }}/providers/)
+- [Rails Integration Guide]({{ site.baseurl }}/guides/rails-integration/)
+- [Middleware System]({{ site.baseurl }}/guides/middleware/)

data/docs/guides/middleware.md

@@ -0,0 +1,324 @@
+---
+layout: page
+title: Middleware System
+permalink: /guides/middleware/
+---
+
+# Middleware System
+
+Vectra includes a **Rack-style middleware stack** that lets you extend the client
+without forking the gem or patching providers.
+
+You can:
+
+- Add **logging, metrics, retries, PII redaction, cost tracking**.
+- Inject **custom behaviour** before/after every operation.
+- Enable features **globally** or **per client**, same kao Faraday/Sidekiq.
+
+---
+
+## Core Concepts
+
+### Request / Response
+
+- **`Vectra::Middleware::Request`**
+  - `operation` – npr. `:upsert`, `:query`, `:fetch`, `:delete`, `:stats`, `:hybrid_search`
+  - `index` – ime indeksa (može biti `nil` ako koristiš default)
+  - `namespace` – namespace (može biti `nil`)
+  - `params` – originalni keyword parametri koje je `Vectra::Client` proslijedio
+  - `provider` – ime providera (`:pinecone`, `:qdrant`, `:pgvector`, `:memory`, …)
+  - helperi: `write_operation?`, `read_operation?`
+
+- **`Vectra::Middleware::Response`**
+  - `result` – što god provider vrati (npr. hash, `QueryResult`, itd.)
+  - `error` – iznimka ako je došlo do greške
+  - `metadata` – slobodan hash za dodatne informacije (trajanje, cost, retry_count…)
+  - helperi: `success?`, `failure?`, `raise_if_error!`, `value!`
+
+### Base i Stack
+
+- **`Vectra::Middleware::Base`**
+  - Hookovi koje možeš override-ati:
+    - `before(request)` – prije poziva providera / sljedećeg middlewarea
+    - `after(request, response)` – nakon uspješnog poziva
+    - `on_error(request, error)` – kad dođe do iznimke (error se zatim re-raise-a)
+
+- **`Vectra::Middleware::Stack`**
+  - Gradi chain oko konkretnog providera:
+    ```ruby
+    stack = Vectra::Middleware::Stack.new(provider, [Logging.new, Retry.new])
+    result = stack.call(:upsert, index: "docs", vectors: [...], provider: :qdrant)
+    ```
+  - `Stack` interno:
+    - kreira `Request`,
+    - kroz sve middlewares propagira isti `Request`,
+    - na kraju zove `provider.public_send(request.operation, **provider_params)`,
+    - vraća `Response` (s `result` ili `error`).
+
+---
+
+## Enabling Middleware
+
+### Global Middleware
+
+Primjenjuje se na **sve** `Vectra::Client` instance.
+
+```ruby
+require "vectra"
+
+# Global logging + retry + cost tracking
+Vectra::Client.use Vectra::Middleware::Logging
+Vectra::Client.use Vectra::Middleware::Retry, max_attempts: 5
+Vectra::Client.use Vectra::Middleware::CostTracker
+```
+
+Sve sljedeće `Vectra::Client.new(...)` instance će koristiti ovaj globalni stack.
+
+### Per‑Client Middleware
+
+Dodatni ili prilagođeni middleware samo za jedan klijent:
+
+```ruby
+client = Vectra::Client.new(
+  provider: :qdrant,
+  index: "products",
+  middleware: [
+    Vectra::Middleware::PIIRedaction,
+    Vectra::Middleware::Instrumentation
+  ]
+)
+```
+
+- Per‑client middleware se izvodi **nakon** globalnog, u istom chainu.
+- Redoslijed u arrayu definira redoslijed ekzekucije (zadnji je najunutarnji, tik do providera).
+
+---
+
+## Koje operacije prolaze kroz stack?
+
+Sve standardne operacije `Vectra::Client`a koriste middleware stack:
+
+- `upsert`
+- `query`
+- `fetch`
+- `update`
+- `delete`
+- `list_indexes`
+- `describe_index`
+- `stats`
+- `hybrid_search`
+
+To znači da middleware može:
+
+- logirati / instrumentirati **sve pozive** prema provideru,
+- raditi **PII redakciju** na `upsert` zahtjevima,
+- brojati i retry-ati i **read** i **write** operacije,
+- računati trošak po operaciji (npr. za billing / budžete).
+
+---
+
+## Built‑in Middleware
+
+### Logging (`Vectra::Middleware::Logging`)
+
+**Što radi:**
+- logira početak i kraj svake operacije (`operation`, `index`, `namespace`),
+- mjeri trajanje i sprema ga u `response.metadata[:duration_ms]`.
+
+**Konfiguracija:**
+
+```ruby
+# Globalno
+Vectra::Client.use Vectra::Middleware::Logging
+
+# S custom loggerom
+logger = Logger.new($stdout)
+Vectra::Client.use Vectra::Middleware::Logging, logger: logger
+```
+
+**Tipična upotreba:** debugiranje, audit logovi, korelacija s HTTP logovima.
+
+---
+
+### Retry (`Vectra::Middleware::Retry`)
+
+**Što radi:**
+- automatski retry-a transient greške:
+  - `Vectra::RateLimitError`
+  - `Vectra::ConnectionError`
+  - `Vectra::TimeoutError`
+  - `Vectra::ServerError`
+- koristi exponential ili linear backoff,
+- upisuje broj retry-a u `response.metadata[:retry_count]`.
+
+**Konfiguracija:**
+
+```ruby
+# 3 pokušaja, exponential backoff (default)
+Vectra::Client.use Vectra::Middleware::Retry
+
+# 5 pokušaja, linearni backoff
+Vectra::Client.use Vectra::Middleware::Retry,
+  max_attempts: 5,
+  backoff: :linear
+
+# Fiksni delay 1.0s
+Vectra::Client.use Vectra::Middleware::Retry,
+  max_attempts: 3,
+  backoff: 1.0
+```
+
+**Tipična upotreba:** zaštita od povremenih mrežnih problema i rate‑limit grešaka.
+
+---
+
+### Instrumentation (`Vectra::Middleware::Instrumentation`)
+
+**Što radi:**
+- emitira događaje preko postojećeg `Vectra::Instrumentation` sustava,
+- prikuplja trajanje, status, error class, dodatni `metadata`.
+
+**Primjer:**
+
+```ruby
+Vectra::Client.use Vectra::Middleware::Instrumentation
+
+Vectra.on_operation do |event|
+  # event[:operation], event[:provider], event[:duration_ms], event[:success], ...
+  StatsD.timing("vectra.#{event[:operation]}", event[:duration_ms])
+end
+```
+
+**Tipična upotreba:** integracija s Prometheus/Grafana, Datadog, New Relic…
+
+---
+
+### PII Redaction (`Vectra::Middleware::PIIRedaction`)
+
+**Što radi:**
+- prije `upsert` operacija prolazi kroz `vectors[:metadata]`,
+- prepoznaje PII pattern-e (email, phone, SSN, credit card) i zamjenjuje ih placeholderom
+  npr. `[REDACTED_EMAIL]`, `[REDACTED_PHONE]`, itd.
+
+**Primjer:**
+
+```ruby
+Vectra::Client.use Vectra::Middleware::PIIRedaction
+
+client.upsert(
+  index: "sensitive",
+  vectors: [
+    {
+      id: "user-1",
+      values: [0.1, 0.2, 0.3],
+      metadata: {
+        email: "user@example.com",
+        phone: "555-1234",
+        note:  "Contact at user@example.com"
+      }
+    }
+  ]
+)
+```
+
+Nakon `upsert`‑a, provider će vidjeti već **redaktirani** metadata.
+
+**Custom patterni:**
+
+```ruby
+patterns = {
+  credit_card: /\b\d{4}[-\s]?\d{4}[-\s]?\d{4}[-\s]?\d{4}\b/,
+  api_key: /sk-[a-zA-Z0-9]{32}/
+}
+
+Vectra::Client.use Vectra::Middleware::PIIRedaction, patterns: patterns
+```
+
+**Tipična upotreba:** GDPR, SOC2, PCI‑DSS okruženja gdje je zabranjen PII u vektor bazi.
+
+---
+
+### CostTracker (`Vectra::Middleware::CostTracker`)
+
+**Što radi:**
+- procjenjuje trošak po operaciji na temelju providera i tipa operacije (`read` / `write`),
+- upisuje trošak u `response.metadata[:cost_usd]`,
+- opcionalno zove `on_cost` callback za real‑time praćenje.
+
+**Primjer:**
+
+```ruby
+Vectra::Client.use Vectra::Middleware::CostTracker,
+  on_cost: ->(event) {
+    puts "💰 Cost: $#{event[:cost_usd].round(6)} for #{event[:operation]} (#{event[:provider]})"
+  }
+```
+
+**Custom pricing:**
+
+```ruby
+pricing = {
+  pinecone: { read: 0.0001, write: 0.0002 },
+  qdrant:   { read: 0.00005, write: 0.0001 }
+}
+
+Vectra::Client.use Vectra::Middleware::CostTracker, pricing: pricing
+```
+
+**Tipična upotreba:** unutarnji billing, budget guardrails, cost dashboards.
+
+---
+
+## Custom Middleware
+
+Najjednostavniji način je naslijediti `Vectra::Middleware::Base` i override-ati hookove:
+
+```ruby
+class MyAuditMiddleware < Vectra::Middleware::Base
+  def before(request)
+    AuditLog.create!(
+      operation: request.operation,
+      index:     request.index,
+      namespace: request.namespace,
+      provider:  request.provider
+    )
+  end
+
+  def after(_request, response)
+    puts "Duration: #{response.metadata[:duration_ms]}ms"
+  end
+
+  def on_error(request, error)
+    ErrorTracker.notify(error, context: { operation: request.operation })
+  end
+end
+
+Vectra::Client.use MyAuditMiddleware
+```
+
+**Savjeti:**
+
+- Ne mijenjaj strukturu `request.params` na način koji provider ne očekuje.
+- Svoje pomoćne podatke stavljaj u `response.metadata` ili `request.metadata`.
+- Ako hvataš greške u `on_error`, **nemoj ih gutati** – middleware stack će ih ponovo baciti.
+
+---
+
+## Primjer: `examples/middleware_demo.rb`
+
+U repozitoriju imaš kompletan demo:
+
+- konfigurira globalni stack (`Logging`, `Retry`, `CostTracker`),
+- pokazuje per‑client PII redaction,
+- definira custom `TimingMiddleware`,
+- demonstrira kako izgleda output u konzoli.
+
+Pokretanje:
+
+```bash
+bundle exec ruby examples/middleware_demo.rb
+```
+
+Ovaj demo je dobar “živi” primjer kako kombinirati više middleware-a u praksi.
+
+

data/docs/guides/roadmap.md

@@ -0,0 +1,53 @@
+---
+layout: page
+title: Roadmap
+permalink: /guides/roadmap/
+---
+
+# Vectra Roadmap
+
+This page outlines the high-level roadmap for **vectra-client**, the unified Ruby client for vector databases.
+
+The roadmap is intentionally focused on **production features** that make AI workloads reliable, observable, and easy to operate in Ruby.
+
+## Near Term (1.x)
+
+- **Reranking middleware**
+  - Middleware that can call external rerankers (e.g., Cohere, Jina, custom HTTP) and reorder search results after a `query`.
+  - Pluggable providers, configurable `top_n`, and safe fallbacks when reranking fails.
+- **More middleware building blocks**
+  - Request sampling / tracing for debugging complex production issues.
+  - Response shaping (e.g., score normalization, custom thresholds) as reusable middleware.
+- **Rails UX improvements**
+  - Convenience generators and helpers for multi-tenant setups.
+  - Better defaults and examples for 1k+ records demos (e‑commerce, blogs, RAG, recommendations).
+
+## Mid Term
+
+- **Additional providers**
+  - Support for more hosted / self-hosted vector solutions where it makes sense and stays maintainable.
+- **First-class reranking guides**
+  - End-to-end documentation for combining vectra-client with external LLMs / rerankers.
+- **More recipes & patterns**
+  - Deeper recipes for analytics, recommendations, and hybrid search in large Rails apps.
+
+## Long Term Vision
+
+Keep **vectra-client** the most **production-ready Ruby toolkit** for vector databases:
+
+- Strong guarantees around retries, circuit breakers, and backpressure.
+- Excellent observability out of the box.
+- Stable, provider-agnostic API that lets you change infra without rewriting your app.
+
+If you have ideas or needs that fit this direction, please open an issue on GitHub so we can prioritise the roadmap around real-world use cases.
+
+{
+  "cells": [],
+  "metadata": {
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 2
+}