vectra-client 0.1.2

data/README.md

@@ -0,0 +1,456 @@
+# Vectra
+
+[![Gem Version](https://badge.fury.io/rb/vectra.svg)](https://badge.fury.io/rb/vectra)
+[![CI](https://github.com/stokry/vectra/actions/workflows/ci.yml/badge.svg)](https://github.com/stokry/vectra/actions)
+[![codecov](https://codecov.io/gh/stokry/vectra/branch/main/graph/badge.svg)](https://codecov.io/gh/stokry/vectra)
+[![Ruby Style Guide](https://img.shields.io/badge/code_style-rubocop-brightgreen.svg)](https://github.com/rubocop/rubocop)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-2.0-4baaaa.svg)](CODE_OF_CONDUCT.md)
+
+**Vectra** is a unified Ruby client for vector databases. Write once, switch providers seamlessly.
+
+## Features
+
+- 🔌 **Unified API** - One interface for multiple vector databases
+- 🚀 **Modern Ruby** - Built for Ruby 3.2+ with modern patterns
+- 🔄 **Automatic Retries** - Built-in retry logic with exponential backoff
+- 📊 **Rich Results** - Enumerable query results with filtering capabilities
+- 🛡️ **Type Safety** - Comprehensive validation and meaningful errors
+- 📝 **Well Documented** - Extensive YARD documentation
+
+## Supported Providers
+
+| Provider | Status | Version |
+|----------|--------|---------|
+| [Pinecone](https://pinecone.io) | ✅ Fully Supported | v0.1.0 |
+| [PostgreSQL + pgvector](https://github.com/pgvector/pgvector) | ✅ Fully Supported | v0.1.1 |
+| [Qdrant](https://qdrant.tech) | 🚧 Planned | v0.2.0 |
+| [Weaviate](https://weaviate.io) | 🚧 Planned | v0.3.0 |
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'vectra'
+```
+
+And then execute:
+
+```bash
+bundle install
+```
+
+Or install it yourself:
+
+```bash
+gem install vectra
+```
+
+### Provider-Specific Dependencies
+
+For **pgvector** support, add the `pg` gem:
+
+```ruby
+gem 'pg', '~> 1.5'
+```
+
+## Quick Start
+
+### Configuration
+
+```ruby
+require 'vectra'
+
+# Global configuration
+Vectra.configure do |config|
+  config.provider = :pinecone
+  config.api_key = ENV['PINECONE_API_KEY']
+  config.environment = 'us-east-1'  # or config.host = 'your-index-host.pinecone.io'
+end
+
+# Create a client
+client = Vectra::Client.new
+```
+
+Or use per-client configuration:
+
+```ruby
+# Shortcut for Pinecone
+client = Vectra.pinecone(
+  api_key: ENV['PINECONE_API_KEY'],
+  environment: 'us-east-1'
+)
+
+# Shortcut for pgvector (PostgreSQL)
+client = Vectra.pgvector(
+  connection_url: 'postgres://user:password@localhost/mydb'
+)
+
+# Generic client with options
+client = Vectra::Client.new(
+  provider: :pinecone,
+  api_key: ENV['PINECONE_API_KEY'],
+  environment: 'us-east-1',
+  timeout: 60,
+  max_retries: 5
+)
+```
+
+### Basic Operations
+
+#### Upsert Vectors
+
+```ruby
+client.upsert(
+  index: 'my-index',
+  vectors: [
+    { id: 'vec1', values: [0.1, 0.2, 0.3], metadata: { text: 'Hello world' } },
+    { id: 'vec2', values: [0.4, 0.5, 0.6], metadata: { text: 'Ruby is great' } }
+  ]
+)
+# => { upserted_count: 2 }
+```
+
+#### Query Vectors
+
+```ruby
+results = client.query(
+  index: 'my-index',
+  vector: [0.1, 0.2, 0.3],
+  top_k: 5,
+  include_metadata: true
+)
+
+# Iterate over results
+results.each do |match|
+  puts "ID: #{match.id}, Score: #{match.score}"
+  puts "Metadata: #{match.metadata}"
+end
+
+# Access specific results
+results.first          # First match
+results.ids            # All matching IDs
+results.scores         # All scores
+results.max_score      # Highest score
+
+# Filter by score
+high_quality = results.above_score(0.8)
+```
+
+#### Query with Filters
+
+```ruby
+results = client.query(
+  index: 'my-index',
+  vector: [0.1, 0.2, 0.3],
+  top_k: 10,
+  filter: { category: 'programming', language: 'ruby' }
+)
+```
+
+#### Fetch Vectors by ID
+
+```ruby
+vectors = client.fetch(index: 'my-index', ids: ['vec1', 'vec2'])
+
+vectors['vec1'].values    # [0.1, 0.2, 0.3]
+vectors['vec1'].metadata  # { 'text' => 'Hello world' }
+```
+
+#### Update Vector Metadata
+
+```ruby
+client.update(
+  index: 'my-index',
+  id: 'vec1',
+  metadata: { category: 'updated', processed: true }
+)
+```
+
+#### Delete Vectors
+
+```ruby
+# Delete by IDs
+client.delete(index: 'my-index', ids: ['vec1', 'vec2'])
+
+# Delete by filter
+client.delete(index: 'my-index', filter: { category: 'old' })
+
+# Delete all (use with caution!)
+client.delete(index: 'my-index', delete_all: true)
+```
+
+### Working with Vectors
+
+```ruby
+# Create a Vector object
+vector = Vectra::Vector.new(
+  id: 'my-vector',
+  values: [0.1, 0.2, 0.3],
+  metadata: { text: 'Example' }
+)
+
+vector.dimension        # => 3
+vector.metadata?        # => true
+vector.to_h             # Convert to hash
+
+# Calculate similarity
+other = Vectra::Vector.new(id: 'other', values: [0.1, 0.2, 0.3])
+vector.cosine_similarity(other)    # => 1.0 (identical)
+vector.euclidean_distance(other)   # => 0.0
+```
+
+### Index Management
+
+```ruby
+# List all indexes
+indexes = client.list_indexes
+indexes.each { |idx| puts idx[:name] }
+
+# Describe an index
+info = client.describe_index(index: 'my-index')
+puts info[:dimension]  # => 384
+puts info[:metric]     # => "cosine"
+
+# Get index statistics
+stats = client.stats(index: 'my-index')
+puts stats[:total_vector_count]
+```
+
+### Namespaces
+
+Namespaces allow you to partition vectors within an index:
+
+```ruby
+# Upsert to a namespace
+client.upsert(
+  index: 'my-index',
+  namespace: 'production',
+  vectors: [...]
+)
+
+# Query within a namespace
+client.query(
+  index: 'my-index',
+  namespace: 'production',
+  vector: [0.1, 0.2, 0.3],
+  top_k: 5
+)
+```
+
+### pgvector (PostgreSQL)
+
+pgvector uses PostgreSQL tables as indexes. Each "index" is a table with a vector column.
+
+#### Setup PostgreSQL with pgvector
+
+```bash
+# Using Docker
+docker run -d --name pgvector \
+  -e POSTGRES_PASSWORD=password \
+  -p 5432:5432 \
+  pgvector/pgvector:pg16
+```
+
+#### Create an Index (Table)
+
+```ruby
+client = Vectra.pgvector(connection_url: 'postgres://postgres:password@localhost/postgres')
+
+# Create a new index with cosine similarity
+client.provider.create_index(
+  name: 'documents',
+  dimension: 384,
+  metric: 'cosine'  # or 'euclidean', 'inner_product'
+)
+```
+
+#### Supported Metrics
+
+| Metric | Description | pgvector Operator |
+|--------|-------------|-------------------|
+| `cosine` | Cosine similarity (default) | `<=>` |
+| `euclidean` | Euclidean distance | `<->` |
+| `inner_product` | Inner product / dot product | `<#>` |
+
+#### Table Structure
+
+Vectra creates tables with the following structure:
+
+```sql
+CREATE TABLE documents (
+  id TEXT PRIMARY KEY,
+  embedding vector(384),
+  metadata JSONB DEFAULT '{}',
+  namespace TEXT DEFAULT '',
+  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+);
+
+-- IVFFlat index for fast similarity search
+CREATE INDEX ON documents USING ivfflat (embedding vector_cosine_ops);
+```
+
+## Configuration Options
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `provider` | Vector database provider (`:pinecone`, `:pgvector`, `:qdrant`, `:weaviate`) | Required |
+| `api_key` | API key for authentication (password for pgvector) | Required* |
+| `environment` | Environment/region (Pinecone) | - |
+| `host` | Direct host URL or PostgreSQL connection URL | - |
+| `timeout` | Request timeout in seconds | 30 |
+| `open_timeout` | Connection timeout in seconds | 10 |
+| `max_retries` | Maximum retry attempts | 3 |
+| `retry_delay` | Initial retry delay in seconds | 1 |
+| `logger` | Logger instance for debugging | nil |
+
+*For pgvector, `api_key` is used as the PostgreSQL password.
+
+## Error Handling
+
+Vectra provides specific error classes for different failure scenarios:
+
+```ruby
+begin
+  client.query(index: 'my-index', vector: [0.1, 0.2], top_k: 5)
+rescue Vectra::AuthenticationError => e
+  puts "Authentication failed: #{e.message}"
+rescue Vectra::RateLimitError => e
+  puts "Rate limited. Retry after #{e.retry_after} seconds"
+rescue Vectra::NotFoundError => e
+  puts "Resource not found: #{e.message}"
+rescue Vectra::ValidationError => e
+  puts "Invalid request: #{e.message}"
+rescue Vectra::ServerError => e
+  puts "Server error (#{e.status_code}): #{e.message}"
+rescue Vectra::Error => e
+  puts "General error: #{e.message}"
+end
+```
+
+## Logging
+
+Enable debug logging to see request details:
+
+```ruby
+require 'logger'
+
+Vectra.configure do |config|
+  config.provider = :pinecone
+  config.api_key = ENV['PINECONE_API_KEY']
+  config.environment = 'us-east-1'
+  config.logger = Logger.new($stdout)
+end
+```
+
+## Best Practices
+
+### Batch Upserts
+
+For large datasets, batch your upserts:
+
+```ruby
+vectors = large_dataset.each_slice(100).map do |batch|
+  client.upsert(index: 'my-index', vectors: batch)
+end
+```
+
+### Connection Reuse
+
+Create a single client instance and reuse it:
+
+```ruby
+# Good: Reuse the client
+client = Vectra::Client.new(...)
+client.query(...)
+client.upsert(...)
+
+# Avoid: Creating new clients for each operation
+Vectra::Client.new(...).query(...)
+Vectra::Client.new(...).upsert(...)
+```
+
+### Error Recovery
+
+Implement retry logic for transient failures:
+
+```ruby
+def query_with_retry(client, **params, retries: 3)
+  client.query(**params)
+rescue Vectra::RateLimitError => e
+  if retries > 0
+    sleep(e.retry_after || 1)
+    retry(retries: retries - 1)
+  else
+    raise
+  end
+end
+```
+
+## Development
+
+After checking out the repo:
+
+```bash
+# Install dependencies
+bundle install
+
+# Run tests
+bundle exec rspec
+
+# Run linter
+bundle exec rubocop
+
+# Generate documentation
+bundle exec rake docs
+```
+
+## Roadmap
+
+### v0.1.0
+- ✅ Pinecone provider
+- ✅ Basic CRUD operations
+- ✅ Configuration system
+- ✅ Error handling with retries
+- ✅ Comprehensive tests
+
+### v0.1.1 (Current)
+- ✅ pgvector (PostgreSQL) provider
+- ✅ Multiple similarity metrics (cosine, euclidean, inner product)
+- ✅ Namespace support for pgvector
+- ✅ IVFFlat index creation
+
+### v0.2.0
+- 🚧 Qdrant provider
+- 🚧 Enhanced error handling
+- 🚧 Connection pooling
+
+### v0.3.0
+- 🚧 Weaviate provider
+- 🚧 Batch operations
+- 🚧 Performance optimizations
+
+### v1.0.0
+- 🚧 Rails integration
+- 🚧 ActiveRecord-like DSL
+- 🚧 Background job support
+- 🚧 Full documentation
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/stokry/vectra.
+
+1. Fork it
+2. Create your feature branch (`git checkout -b feature/my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin feature/my-new-feature`)
+5. Create a new Pull Request
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+
+## Acknowledgments
+
+Inspired by the simplicity of Ruby database gems and the need for a unified vector database interface.

data/Rakefile

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+require "rubocop/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+RuboCop::RakeTask.new
+
+task default: %i[spec rubocop]
+
+namespace :spec do
+  desc "Run unit tests only"
+  RSpec::Core::RakeTask.new(:unit) do |t|
+    t.pattern = "spec/vectra/**/*_spec.rb"
+  end
+
+  desc "Run integration tests only"
+  RSpec::Core::RakeTask.new(:integration) do |t|
+    t.pattern = "spec/integration/**/*_spec.rb"
+  end
+end
+
+desc "Generate documentation"
+task :docs do
+  require "yard"
+  YARD::CLI::Yardoc.run("--output-dir", "doc", "lib/**/*.rb", "-", "README.md", "CHANGELOG.md")
+end
+
+desc "Generate CHANGELOG.md"
+task :changelog do
+  puts "Generating CHANGELOG.md..."
+  system("github_changelog_generator") || puts("Install with: gem install github_changelog_generator")
+end

data/SECURITY.md

@@ -0,0 +1,196 @@
+# Security Policy
+
+## Supported Versions
+
+We release patches for security vulnerabilities for the following versions:
+
+| Version | Supported          |
+| ------- | ------------------ |
+| 0.1.x   | :white_check_mark: |
+| < 0.1   | :x:                |
+
+## Reporting a Vulnerability
+
+We take the security of Vectra seriously. If you believe you have found a security vulnerability, please report it to us as described below.
+
+### Where to Report
+
+**Please do NOT report security vulnerabilities through public GitHub issues.**
+
+Instead, please report them via email to: **mijo@mijokristo.com**
+
+### What to Include
+
+Please include the following information in your report:
+
+- Type of vulnerability (e.g., authentication bypass, SQL injection, credential exposure)
+- Full paths of source file(s) related to the manifestation of the vulnerability
+- The location of the affected source code (tag/branch/commit or direct URL)
+- Step-by-step instructions to reproduce the issue
+- Proof-of-concept or exploit code (if possible)
+- Impact of the issue, including how an attacker might exploit it
+
+### Response Timeline
+
+- **Initial Response**: Within 48 hours
+- **Status Update**: Within 7 days
+- **Fix Timeline**: Depends on severity, typically 30-90 days
+
+We will:
+1. Confirm the receipt of your vulnerability report
+2. Provide an estimated timeline for a fix
+3. Notify you when the vulnerability is fixed
+4. Credit you in the security advisory (unless you prefer to remain anonymous)
+
+## Security Best Practices for Users
+
+### API Key Management
+
+- **Never commit API keys** to version control
+- Store API keys in environment variables or secure vaults
+- Use different API keys for development, staging, and production
+- Rotate API keys regularly
+- Limit API key permissions to minimum required access
+
+```ruby
+# ✅ Good - Use environment variables
+Vectra.configure do |config|
+  config.api_key = ENV['PINECONE_API_KEY']
+end
+
+# ❌ Bad - Hardcoded API key
+Vectra.configure do |config|
+  config.api_key = "pk-123456789"  # Never do this!
+end
+```
+
+### Network Security
+
+- Always use HTTPS for API connections (enforced by default)
+- Verify SSL certificates (enabled by default)
+- Use VPN or private networks when possible
+- Monitor API usage for unusual patterns
+
+### Data Security
+
+- **Sanitize input data** before upserting to vector databases
+- **Validate vector dimensions** match your index configuration
+- **Review metadata** for sensitive information before upserting
+- **Implement access controls** at the application level
+- **Encrypt sensitive metadata** before storage if needed
+
+```ruby
+# Example: Sanitizing metadata
+def sanitize_metadata(metadata)
+  metadata.reject { |k, _| k.to_s.match?(/password|secret|token/i) }
+end
+
+vectors = [{
+  id: "vec1",
+  values: embedding,
+  metadata: sanitize_metadata(user_data)
+}]
+
+client.upsert(index: "my-index", vectors: vectors)
+```
+
+### Dependency Security
+
+- Keep Vectra and its dependencies up to date
+- Run `bundle audit` regularly to check for known vulnerabilities
+- Review dependency changes in updates
+
+```bash
+# Check for vulnerabilities
+gem install bundler-audit
+bundle audit --update
+```
+
+### Rate Limiting
+
+- Implement application-level rate limiting
+- Handle `RateLimitError` exceptions appropriately
+- Use exponential backoff for retries
+
+```ruby
+def safe_query_with_backoff(client, **params, max_retries: 3)
+  retries = 0
+
+  begin
+    client.query(**params)
+  rescue Vectra::RateLimitError => e
+    retries += 1
+    if retries <= max_retries
+      sleep_time = e.retry_after || (2 ** retries)
+      sleep(sleep_time)
+      retry
+    else
+      raise
+    end
+  end
+end
+```
+
+### Logging and Monitoring
+
+- **Do not log API keys** or sensitive data
+- Monitor for authentication failures
+- Track unusual query patterns
+- Set up alerts for rate limit violations
+
+```ruby
+# ❌ Bad - Logs API key
+logger.info("Using API key: #{config.api_key}")
+
+# ✅ Good - Logs without sensitive data
+logger.info("Initializing Vectra client for #{config.provider}")
+```
+
+## Known Security Considerations
+
+### API Key Exposure
+
+API keys are transmitted in HTTP headers. While connections use HTTPS, ensure:
+- API keys are never logged or exposed in error messages
+- API keys are not included in client-side code
+- Development/test API keys are separate from production
+
+### Metadata Privacy
+
+Metadata stored with vectors may contain sensitive information:
+- Review metadata fields before upserting
+- Consider encryption for sensitive fields
+- Implement data retention policies
+- Follow GDPR/privacy regulations for user data
+
+### Dependency Chain
+
+Vectra depends on:
+- `faraday` - HTTP client library
+- `faraday-retry` - Retry middleware
+
+We monitor these dependencies for security issues and update promptly.
+
+## Security Updates
+
+Security updates will be released as patch versions (e.g., 0.1.1) and announced:
+- On GitHub Security Advisories
+- In the CHANGELOG.md
+- Via RubyGems security notifications
+
+Subscribe to GitHub releases to be notified of security updates.
+
+## Compliance
+
+Vectra is designed to work with various vector database providers. Ensure your usage complies with:
+- Your provider's security requirements
+- Data protection regulations (GDPR, CCPA, etc.)
+- Industry-specific compliance (HIPAA, PCI-DSS, etc.)
+
+## Questions?
+
+If you have questions about security that are not covered here, please email: mijo@mijokristo.com
+
+## Attribution
+
+We appreciate responsible disclosure and will acknowledge security researchers who help improve Vectra's security.