aigroup-workflow 2.1.1 → 2.2.0

package/skills/rag-architect/references/vector-databases.md

@@ -1,589 +0,0 @@
-# Vector Databases
-
----
-
-## Database Comparison Matrix
-
-| Feature | Pinecone | Weaviate | Qdrant | Chroma | pgvector |
-|---------|----------|----------|--------|--------|----------|
-| **Hosting** | Managed only | Managed + Self-hosted | Managed + Self-hosted | Self-hosted (cloud beta) | Self-hosted |
-| **Hybrid Search** | Yes (sparse-dense) | Yes (BM25 + vector) | Yes (sparse vectors) | Limited | Manual (+ pg_trgm) |
-| **Filtering** | Excellent | Excellent | Excellent | Basic | SQL-native |
-| **Max Dimensions** | 20,000 | Unlimited | 65,535 | Unlimited | 2,000 |
-| **Pricing Model** | Per-vector/query | Per-node | Per-node | Free (OSS) | Free (extension) |
-| **Multi-tenancy** | Namespaces | Multi-tenant class | Collections + payloads | Collections | Schema/RLS |
-| **Best For** | Enterprise SaaS | Semantic apps | High-performance | Prototyping | Postgres shops |
-
-## When to Use Each
-
-### Pinecone
-```
-Best For:
-- Enterprise RAG with strict SLAs
-- Teams wanting zero infrastructure management
-- Applications needing sparse-dense hybrid search
-- High-volume production with predictable costs
-
-When to Avoid:
-- Cost-sensitive projects (expensive at scale)
-- Need for self-hosting or data residency
-- Complex filtering requirements beyond metadata
-- Wanting to avoid vendor lock-in
-```
-
-### Weaviate
-```
-Best For:
-- Semantic search with built-in vectorization
-- Multi-modal (text, image) applications
-- GraphQL-native teams
-- Hybrid BM25 + vector search requirements
-
-When to Avoid:
-- Simple embedding storage only
-- Memory-constrained environments
-- Teams unfamiliar with GraphQL
-```
-
-### Qdrant
-```
-Best For:
-- High-performance, low-latency requirements
-- Complex filtering with payload indexes
-- Rust/performance-focused teams
-- Self-hosted with full control
-
-When to Avoid:
-- Teams wanting fully managed simplicity
-- GraphQL preference (REST/gRPC only)
-```
-
-### Chroma
-```
-Best For:
-- Local development and prototyping
-- LangChain/LlamaIndex integration
-- Simple RAG proof-of-concepts
-- Educational projects
-
-When to Avoid:
-- Production workloads at scale
-- Multi-tenant applications
-- High availability requirements
-```
-
-### pgvector
-```
-Best For:
-- Existing PostgreSQL infrastructure
-- Transactional + vector in same DB
-- SQL-native teams
-- Cost optimization (no new infra)
-
-When to Avoid:
-- Vectors > 2000 dimensions
-- Billions of vectors (scaling limits)
-- Sub-millisecond latency requirements
-```
-
----
-
-## Pinecone Setup
-
-```python
-from pinecone import Pinecone, ServerlessSpec
-
-# Initialize client
-pc = Pinecone(api_key="your-api-key")
-
-# Create index with serverless
-pc.create_index(
-    name="rag-index",
-    dimension=1536,  # OpenAI ada-002
-    metric="cosine",
-    spec=ServerlessSpec(
-        cloud="aws",
-        region="us-east-1"
-    )
-)
-
-# Get index reference
-index = pc.Index("rag-index")
-
-# Upsert vectors with metadata
-index.upsert(
-    vectors=[
-        {
-            "id": "doc-1",
-            "values": embedding_vector,
-            "metadata": {
-                "source": "manual.pdf",
-                "page": 42,
-                "section": "installation",
-                "tenant_id": "acme-corp"
-            }
-        }
-    ],
-    namespace="production"
-)
-
-# Query with metadata filter
-results = index.query(
-    vector=query_embedding,
-    top_k=10,
-    include_metadata=True,
-    namespace="production",
-    filter={
-        "tenant_id": {"$eq": "acme-corp"},
-        "section": {"$in": ["installation", "setup"]}
-    }
-)
-
-# Hybrid search (sparse-dense)
-from pinecone_text.sparse import BM25Encoder
-
-bm25 = BM25Encoder()
-bm25.fit(corpus)  # Fit on your documents
-
-results = index.query(
-    vector=dense_embedding,
-    sparse_vector=bm25.encode_queries(query_text),
-    top_k=10,
-    alpha=0.5  # Balance dense vs sparse
-)
-```
-
----
-
-## Weaviate Setup
-
-```python
-import weaviate
-from weaviate.classes.config import Configure, Property, DataType
-
-# Connect to Weaviate Cloud
-client = weaviate.connect_to_weaviate_cloud(
-    cluster_url="https://your-cluster.weaviate.network",
-    auth_credentials=weaviate.auth.AuthApiKey("your-api-key")
-)
-
-# Or self-hosted
-client = weaviate.connect_to_local(
-    host="localhost",
-    port=8080
-)
-
-# Create collection with vectorizer
-client.collections.create(
-    name="Document",
-    vectorizer_config=Configure.Vectorizer.text2vec_openai(
-        model="text-embedding-3-small"
-    ),
-    properties=[
-        Property(name="content", data_type=DataType.TEXT),
-        Property(name="source", data_type=DataType.TEXT),
-        Property(name="page", data_type=DataType.INT),
-        Property(name="tenant_id", data_type=DataType.TEXT, index_filterable=True)
-    ]
-)
-
-# Insert with auto-vectorization
-documents = client.collections.get("Document")
-documents.data.insert(
-    properties={
-        "content": "Installation guide content...",
-        "source": "manual.pdf",
-        "page": 42,
-        "tenant_id": "acme-corp"
-    }
-)
-
-# Or with pre-computed vector
-documents.data.insert(
-    properties={"content": "...", "source": "..."},
-    vector=precomputed_embedding
-)
-
-# Hybrid search (BM25 + vector)
-from weaviate.classes.query import MetadataQuery, Filter
-
-results = documents.query.hybrid(
-    query="how to install",
-    alpha=0.5,  # 0=BM25 only, 1=vector only
-    limit=10,
-    filters=Filter.by_property("tenant_id").equal("acme-corp"),
-    return_metadata=MetadataQuery(score=True, explain_score=True)
-)
-
-for obj in results.objects:
-    print(f"Score: {obj.metadata.score}, Content: {obj.properties['content'][:100]}")
-
-client.close()
-```
-
----
-
-## Qdrant Setup
-
-```python
-from qdrant_client import QdrantClient
-from qdrant_client.models import (
-    Distance, VectorParams, PointStruct,
-    Filter, FieldCondition, MatchValue,
-    PayloadSchemaType
-)
-
-# Connect to Qdrant Cloud
-client = QdrantClient(
-    url="https://your-cluster.qdrant.io",
-    api_key="your-api-key"
-)
-
-# Or local
-client = QdrantClient(host="localhost", port=6333)
-
-# Create collection
-client.create_collection(
-    collection_name="documents",
-    vectors_config=VectorParams(
-        size=1536,
-        distance=Distance.COSINE
-    )
-)
-
-# Create payload index for fast filtering
-client.create_payload_index(
-    collection_name="documents",
-    field_name="tenant_id",
-    field_schema=PayloadSchemaType.KEYWORD
-)
-
-# Upsert points
-client.upsert(
-    collection_name="documents",
-    points=[
-        PointStruct(
-            id="doc-1",
-            vector=embedding_vector,
-            payload={
-                "content": "Installation guide...",
-                "source": "manual.pdf",
-                "page": 42,
-                "tenant_id": "acme-corp"
-            }
-        )
-    ]
-)
-
-# Search with filter
-results = client.search(
-    collection_name="documents",
-    query_vector=query_embedding,
-    limit=10,
-    query_filter=Filter(
-        must=[
-            FieldCondition(
-                key="tenant_id",
-                match=MatchValue(value="acme-corp")
-            )
-        ]
-    ),
-    with_payload=True
-)
-
-# Batch upsert for large datasets
-from qdrant_client.models import Batch
-
-client.upsert(
-    collection_name="documents",
-    points=Batch(
-        ids=ids_list,
-        vectors=vectors_list,
-        payloads=payloads_list
-    )
-)
-```
-
----
-
-## Chroma Setup
-
-```python
-import chromadb
-from chromadb.config import Settings
-
-# Persistent local storage
-client = chromadb.PersistentClient(
-    path="./chroma_data",
-    settings=Settings(anonymized_telemetry=False)
-)
-
-# Create collection with custom embedding function
-from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction
-
-embedding_fn = OpenAIEmbeddingFunction(
-    api_key="your-openai-key",
-    model_name="text-embedding-3-small"
-)
-
-collection = client.get_or_create_collection(
-    name="documents",
-    embedding_function=embedding_fn,
-    metadata={"hnsw:space": "cosine"}
-)
-
-# Add documents (auto-embeds)
-collection.add(
-    ids=["doc-1", "doc-2"],
-    documents=["Installation guide...", "Configuration steps..."],
-    metadatas=[
-        {"source": "manual.pdf", "page": 42},
-        {"source": "manual.pdf", "page": 43}
-    ]
-)
-
-# Or with pre-computed embeddings
-collection.add(
-    ids=["doc-3"],
-    embeddings=[precomputed_vector],
-    metadatas=[{"source": "guide.pdf"}],
-    documents=["Original text for reference"]
-)
-
-# Query
-results = collection.query(
-    query_texts=["how to install"],
-    n_results=10,
-    where={"source": "manual.pdf"},
-    include=["documents", "metadatas", "distances"]
-)
-
-# Update existing document
-collection.update(
-    ids=["doc-1"],
-    documents=["Updated installation guide..."],
-    metadatas=[{"source": "manual_v2.pdf", "page": 42}]
-)
-```
-
----
-
-## pgvector Setup
-
-```sql
--- Enable extension
-CREATE EXTENSION IF NOT EXISTS vector;
-
--- Create table with vector column
-CREATE TABLE documents (
-    id SERIAL PRIMARY KEY,
-    content TEXT NOT NULL,
-    embedding vector(1536),  -- OpenAI dimensions
-    source VARCHAR(255),
-    page INTEGER,
-    tenant_id VARCHAR(100),
-    created_at TIMESTAMP DEFAULT NOW()
-);
-
--- Create HNSW index (recommended for most cases)
-CREATE INDEX ON documents
-USING hnsw (embedding vector_cosine_ops)
-WITH (m = 16, ef_construction = 64);
-
--- Or IVFFlat for very large datasets
-CREATE INDEX ON documents
-USING ivfflat (embedding vector_cosine_ops)
-WITH (lists = 100);
-
--- Create index on filter columns
-CREATE INDEX ON documents (tenant_id);
-```
-
-```python
-import psycopg2
-from pgvector.psycopg2 import register_vector
-
-conn = psycopg2.connect("postgresql://localhost/ragdb")
-register_vector(conn)
-
-# Insert with embedding
-cur = conn.cursor()
-cur.execute(
-    """
-    INSERT INTO documents (content, embedding, source, page, tenant_id)
-    VALUES (%s, %s, %s, %s, %s)
-    RETURNING id
-    """,
-    ("Installation guide...", embedding_vector, "manual.pdf", 42, "acme-corp")
-)
-
-# Similarity search with filter
-cur.execute(
-    """
-    SELECT id, content, source, page,
-           1 - (embedding <=> %s) AS similarity
-    FROM documents
-    WHERE tenant_id = %s
-    ORDER BY embedding <=> %s
-    LIMIT 10
-    """,
-    (query_embedding, "acme-corp", query_embedding)
-)
-
-results = cur.fetchall()
-
-# Hybrid search with pg_trgm
-cur.execute(
-    """
-    SELECT id, content,
-           (0.5 * (1 - (embedding <=> %s))) +
-           (0.5 * similarity(content, %s)) AS hybrid_score
-    FROM documents
-    WHERE tenant_id = %s
-      AND content %% %s  -- Trigram similarity threshold
-    ORDER BY hybrid_score DESC
-    LIMIT 10
-    """,
-    (query_embedding, query_text, "acme-corp", query_text)
-)
-```
-
----
-
-## Index Tuning Guide
-
-### HNSW Parameters
-
-| Parameter | Description | Trade-off |
-|-----------|-------------|-----------|
-| `m` | Connections per node | Higher = better recall, more memory |
-| `ef_construction` | Build-time search width | Higher = better index, slower build |
-| `ef_search` | Query-time search width | Higher = better recall, slower query |
-
-```python
-# Qdrant HNSW tuning
-client.update_collection(
-    collection_name="documents",
-    hnsw_config=HnswConfigDiff(
-        m=16,                    # Default: 16, increase for better recall
-        ef_construct=100,        # Default: 100, higher for better index
-        full_scan_threshold=10000  # Use brute force below this size
-    )
-)
-
-# Query-time ef adjustment
-results = client.search(
-    collection_name="documents",
-    query_vector=query_embedding,
-    limit=10,
-    search_params=SearchParams(hnsw_ef=128)  # Higher for better recall
-)
-```
-
-### Quantization for Scale
-
-```python
-# Qdrant scalar quantization (4x memory reduction)
-from qdrant_client.models import ScalarQuantization, ScalarQuantizationConfig
-
-client.update_collection(
-    collection_name="documents",
-    quantization_config=ScalarQuantization(
-        scalar=ScalarQuantizationConfig(
-            type="int8",
-            quantile=0.99,
-            always_ram=True
-        )
-    )
-)
-```
-
----
-
-## Multi-Tenancy Patterns
-
-### Namespace Isolation (Pinecone)
-```python
-# Tenant data in separate namespaces
-index.upsert(vectors=[...], namespace="tenant-acme")
-index.upsert(vectors=[...], namespace="tenant-globex")
-
-# Query within tenant namespace
-results = index.query(
-    vector=query_embedding,
-    namespace="tenant-acme",
-    top_k=10
-)
-```
-
-### Metadata Filtering (Qdrant/Weaviate)
-```python
-# Add tenant_id to all documents
-point = PointStruct(
-    id="doc-1",
-    vector=embedding,
-    payload={"tenant_id": "acme", "content": "..."}
-)
-
-# Always filter by tenant
-results = client.search(
-    collection_name="documents",
-    query_vector=query_embedding,
-    query_filter=Filter(
-        must=[FieldCondition(key="tenant_id", match=MatchValue(value="acme"))]
-    )
-)
-```
-
-### Collection per Tenant (High Isolation)
-```python
-# Create tenant-specific collection
-client.create_collection(
-    collection_name=f"docs_{tenant_id}",
-    vectors_config=VectorParams(size=1536, distance=Distance.COSINE)
-)
-```
-
----
-
-## Decision Flowchart
-
-```
-Start
-  │
-  ├─ Need managed service with zero ops?
-  │   └─ Yes → Pinecone
-  │
-  ├─ Have existing PostgreSQL?
-  │   └─ Yes → pgvector (if vectors < 2000 dims)
-  │
-  ├─ Need built-in vectorization?
-  │   └─ Yes → Weaviate
-  │
-  ├─ Need maximum performance + self-host?
-  │   └─ Yes → Qdrant
-  │
-  ├─ Prototyping / local development?
-  │   └─ Yes → Chroma
-  │
-  └─ Default recommendation → Qdrant (balance of features/performance)
-```
-
----
-
-## Quick Reference
-
-| Task | Pinecone | Weaviate | Qdrant | pgvector |
-|------|----------|----------|--------|----------|
-| Create index/collection | `create_index()` | `collections.create()` | `create_collection()` | `CREATE TABLE` |
-| Insert | `upsert()` | `data.insert()` | `upsert()` | `INSERT` |
-| Search | `query()` | `query.near_vector()` | `search()` | `ORDER BY <=>` |
-| Filter | `filter={}` | `Filter.by_property()` | `query_filter=Filter()` | `WHERE` |
-| Delete | `delete()` | `data.delete_by_id()` | `delete()` | `DELETE` |
-| Hybrid | sparse_vector param | `query.hybrid()` | sparse vectors | Manual |
-
-## Related Skills
-
-- **Database Optimizer** - Index tuning and query performance
-- **Cloud Architect** - Infrastructure decisions for vector DB hosting
-- **Python Pro** - Implementation patterns with async clients