@cubis/foundry 0.3.82 → 0.3.84

package/workflows/workflows/agent-environment-setup/platforms/gemini/skills/rag-patterns/examples/02-hybrid-search-support-kb.md

@@ -1,264 +0,0 @@
-# Example: Hybrid Search for Customer Support Knowledge Base
-
-## Context
-
-A SaaS company's support team uses an internal knowledge base with 8,000 articles. The current dense-only search fails on queries containing:
-- Error codes: "ERR-4502", "TIMEOUT-3001"
-- Product SKUs: "PRO-TEAM-50", "ENT-UNLIMITED"
-- Configuration keys: "max_retry_count", "ssl_verify_mode"
-
-These exact strings have no semantic meaning -- they are identifiers that must be matched lexically. The goal is to implement hybrid search that combines semantic understanding with exact keyword matching.
-
-## Problem Analysis
-
-Dense embeddings map text to a continuous vector space based on semantic meaning. The string "ERR-4502" has no inherent semantic content -- it is an arbitrary identifier. When a user searches for "ERR-4502 payment failed," the dense embedding captures "payment failed" but the error code gets mapped to a generic region of the embedding space, losing its specificity.
-
-**Evidence from retrieval eval:**
-
-| Query type | Dense Recall@5 | Dense MRR@5 |
-|------------|---------------|-------------|
-| Conceptual ("how to set up SSO") | 0.85 | 0.72 |
-| Error code ("ERR-4502 payment") | 0.31 | 0.18 |
-| SKU-specific ("PRO-TEAM-50 limits") | 0.28 | 0.15 |
-| Config key ("max_retry_count") | 0.22 | 0.12 |
-
-Dense search performs well on conceptual queries but fails dramatically on identifier-heavy queries.
-
-## Hybrid Search Architecture
-
-```
-User Query: "ERR-4502 payment failed"
-    |
-    +---> [Dense Path]
-    |       Embed query --> Qdrant dense search --> top-50
-    |
-    +---> [Sparse Path]
-    |       BM25 tokenize --> Qdrant sparse search --> top-50
-    |
-    v
-[Reciprocal Rank Fusion] --> merge top-50 + top-50 --> ranked list
-    |
-    v
-[Cross-Encoder Re-ranker] --> top-5
-    |
-    v
-[LLM Generation with context]
-```
-
-## Qdrant Configuration
-
-Qdrant supports named vectors, allowing both dense and sparse vectors on the same point.
-
-```python
-from qdrant_client import QdrantClient, models
-
-client = QdrantClient(url="http://localhost:6333")
-
-# Create collection with both dense and sparse vectors
-client.create_collection(
-    collection_name="support_kb",
-    vectors_config={
-        "dense": models.VectorParams(
-            size=1536,
-            distance=models.Distance.COSINE
-        )
-    },
-    sparse_vectors_config={
-        "sparse": models.SparseVectorParams(
-            modifier=models.Modifier.IDF  # TF-IDF weighting
-        )
-    }
-)
-
-# Create payload index for metadata filtering
-client.create_payload_index(
-    collection_name="support_kb",
-    field_name="doc_type",
-    field_schema=models.PayloadSchemaType.KEYWORD
-)
-client.create_payload_index(
-    collection_name="support_kb",
-    field_name="product",
-    field_schema=models.PayloadSchemaType.KEYWORD
-)
-```
-
-## Indexing Pipeline
-
-```python
-from fastembed import SparseTextEmbedding, TextEmbedding
-
-dense_model = TextEmbedding("BAAI/bge-small-en-v1.5")
-sparse_model = SparseTextEmbedding("Qdrant/bm25")
-
-def index_document(doc: dict) -> None:
-    chunks = chunk_document(doc)
-
-    for chunk in chunks:
-        # Generate dense embedding
-        dense_vector = list(dense_model.embed([chunk["text"]]))[0].tolist()
-
-        # Generate sparse embedding (BM25)
-        sparse_result = list(sparse_model.embed([chunk["text"]]))[0]
-        sparse_vector = models.SparseVector(
-            indices=sparse_result.indices.tolist(),
-            values=sparse_result.values.tolist()
-        )
-
-        # Upsert with both vector types
-        client.upsert(
-            collection_name="support_kb",
-            points=[
-                models.PointStruct(
-                    id=chunk["id"],
-                    vector={
-                        "dense": dense_vector,
-                        "sparse": sparse_vector
-                    },
-                    payload={
-                        "text": chunk["text"],
-                        "doc_type": chunk["metadata"]["doc_type"],
-                        "product": chunk["metadata"]["product"],
-                        "title": chunk["metadata"]["title"],
-                        "url": chunk["metadata"]["url"]
-                    }
-                )
-            ]
-        )
-```
-
-## Query Pipeline
-
-```python
-def hybrid_search(
-    query: str,
-    top_k: int = 50,
-    dense_weight: float = 0.6,
-    sparse_weight: float = 0.4,
-    rerank_top_n: int = 5
-) -> list[dict]:
-    # Dense retrieval
-    dense_vector = list(dense_model.embed([query]))[0].tolist()
-    dense_results = client.query_points(
-        collection_name="support_kb",
-        query=dense_vector,
-        using="dense",
-        limit=top_k,
-        with_payload=True
-    )
-
-    # Sparse retrieval
-    sparse_result = list(sparse_model.embed([query]))[0]
-    sparse_vector = models.SparseVector(
-        indices=sparse_result.indices.tolist(),
-        values=sparse_result.values.tolist()
-    )
-    sparse_results = client.query_points(
-        collection_name="support_kb",
-        query=sparse_vector,
-        using="sparse",
-        limit=top_k,
-        with_payload=True
-    )
-
-    # Reciprocal Rank Fusion
-    fused = reciprocal_rank_fusion(
-        dense_results=dense_results.points,
-        sparse_results=sparse_results.points,
-        dense_weight=dense_weight,
-        sparse_weight=sparse_weight
-    )
-
-    # Cross-encoder re-ranking
-    reranked = rerank_with_cross_encoder(query, fused[:20], top_n=rerank_top_n)
-
-    return reranked
-```
-
-## Reciprocal Rank Fusion (RRF)
-
-RRF merges two ranked lists by assigning each result a score based on its rank position, then summing scores across lists.
-
-```python
-def reciprocal_rank_fusion(
-    dense_results: list,
-    sparse_results: list,
-    dense_weight: float = 0.6,
-    sparse_weight: float = 0.4,
-    k: int = 60
-) -> list[dict]:
-    """
-    RRF score = sum(weight / (k + rank)) for each list the document appears in.
-    k=60 is the standard constant that prevents high-ranked documents from dominating.
-    """
-    scores = {}
-
-    for rank, result in enumerate(dense_results):
-        doc_id = result.id
-        rrf_score = dense_weight / (k + rank + 1)
-        scores[doc_id] = scores.get(doc_id, {"score": 0, "payload": result.payload})
-        scores[doc_id]["score"] += rrf_score
-
-    for rank, result in enumerate(sparse_results):
-        doc_id = result.id
-        rrf_score = sparse_weight / (k + rank + 1)
-        scores[doc_id] = scores.get(doc_id, {"score": 0, "payload": result.payload})
-        scores[doc_id]["score"] += rrf_score
-
-    # Sort by fused score descending
-    fused = sorted(scores.items(), key=lambda x: x[1]["score"], reverse=True)
-    return [{"id": doc_id, **data} for doc_id, data in fused]
-```
-
-## Weight Tuning
-
-The dense/sparse weight balance depends on query distribution. Use a retrieval eval set to find the optimal weights.
-
-```python
-def tune_weights(
-    eval_queries: list[dict],
-    weight_range: list[float] = [0.3, 0.4, 0.5, 0.6, 0.7]
-) -> dict:
-    """Grid search over dense weights. Sparse weight = 1 - dense weight."""
-    best_mrr = 0
-    best_dense_weight = 0.5
-
-    for dw in weight_range:
-        sw = 1.0 - dw
-        mrr_scores = []
-
-        for q in eval_queries:
-            results = hybrid_search(q["query"], dense_weight=dw, sparse_weight=sw)
-            mrr = compute_mrr(results, q["relevant_doc_ids"])
-            mrr_scores.append(mrr)
-
-        mean_mrr = statistics.mean(mrr_scores)
-        if mean_mrr > best_mrr:
-            best_mrr = mean_mrr
-            best_dense_weight = dw
-
-    return {
-        "best_dense_weight": best_dense_weight,
-        "best_sparse_weight": 1.0 - best_dense_weight,
-        "best_mrr": best_mrr
-    }
-```
-
-## Results After Hybrid Search
-
-| Query type | Dense Recall@5 | Hybrid Recall@5 | Improvement |
-|------------|---------------|-----------------|-------------|
-| Conceptual | 0.85 | 0.87 | +2% |
-| Error code | 0.31 | 0.82 | +51% |
-| SKU-specific | 0.28 | 0.79 | +51% |
-| Config key | 0.22 | 0.76 | +54% |
-| **Overall** | **0.52** | **0.82** | **+30%** |
-
-The dramatic improvement on identifier queries comes from BM25 matching exact strings. The slight improvement on conceptual queries comes from BM25 catching important keywords that dense search sometimes misses (e.g., specific product names).
-
-## Key Takeaways
-
-1. **Dense search is not enough.** Any corpus with identifiers, codes, or proper nouns needs a keyword matching component.
-2. **RRF is robust.** Reciprocal Rank Fusion with k=60 works well out of the box and is less sensitive to weight tuning than linear combination.
-3. **Weight tuning matters for edge cases.** The default 0.6/0.4 split was optimal overall, but error-code-heavy query distributions benefited from 0.4/0.6 (more sparse weight).
-4. **Re-ranking is the precision layer.** Hybrid search improved recall dramatically. The cross-encoder re-ranker then improved MRR by reordering the top results for precision.

package/workflows/workflows/agent-environment-setup/platforms/gemini/skills/rag-patterns/references/chunking.md

@@ -1,217 +0,0 @@
-# Chunking Strategies
-
-## Overview
-
-Chunking transforms raw documents into retrieval-ready units. The quality of chunking directly determines retrieval precision: good chunks map one concept to one embedding, while bad chunks mix unrelated content or split coherent ideas across boundaries. This reference covers strategies by document type, sizing heuristics, and implementation patterns.
-
-## Core Principles
-
-### One Chunk, One Concept
-
-The ideal chunk contains exactly the information needed to answer one type of question. A chunk that covers authentication AND rate limiting will match queries about either topic but dilute the relevance signal for both.
-
-### Semantic Boundaries Over Token Counts
-
-Split at structural boundaries first (headings, paragraphs, function definitions), then enforce token budgets as a secondary constraint. A 300-token chunk that ends at a heading is better than a 512-token chunk that cuts a paragraph in half.
-
-### Preserve Context
-
-A chunk that starts with "Additionally, the third parameter controls..." is useless without the preceding context. Include enough context (the section heading, the parent topic) so the chunk is self-contained.
-
-## Strategies by Document Type
-
-### Markdown / Technical Documentation
-
-```python
-import re
-from typing import Optional
-
-def chunk_markdown(
-    text: str,
-    max_tokens: int = 512,
-    overlap_tokens: int = 64,
-    heading_level: int = 2
-) -> list[dict]:
-    """Split markdown by heading level, then by paragraph within long sections."""
-    pattern = rf'^({"#" * heading_level}\s+.+)$'
-    sections = re.split(pattern, text, flags=re.MULTILINE)
-
-    chunks = []
-    current_heading = "Introduction"
-
-    for part in sections:
-        if re.match(pattern, part):
-            current_heading = part.strip("# ").strip()
-            continue
-
-        tokens = count_tokens(part)
-        if tokens <= max_tokens:
-            chunks.append({"heading": current_heading, "text": part.strip()})
-        else:
-            sub_chunks = split_by_paragraph(
-                part, max_tokens=max_tokens, overlap_tokens=overlap_tokens
-            )
-            for i, sub in enumerate(sub_chunks):
-                chunks.append({
-                    "heading": f"{current_heading} (part {i+1})",
-                    "text": sub
-                })
-
-    return chunks
-```
-
-### Source Code
-
-```python
-import ast
-
-def chunk_python_file(source: str, file_path: str) -> list[dict]:
-    """Chunk Python files by top-level definitions."""
-    tree = ast.parse(source)
-    chunks = []
-
-    for node in ast.iter_child_nodes(tree):
-        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
-            start_line = node.lineno - 1
-            end_line = node.end_lineno
-            chunk_text = "\n".join(source.splitlines()[start_line:end_line])
-
-            # Include the docstring as searchable content
-            docstring = ast.get_docstring(node) or ""
-
-            chunks.append({
-                "text": chunk_text,
-                "metadata": {
-                    "type": type(node).__name__,
-                    "name": node.name,
-                    "file": file_path,
-                    "docstring": docstring,
-                    "start_line": start_line + 1,
-                    "end_line": end_line
-                }
-            })
-
-    return chunks
-```
-
-### Conversational Data
-
-```python
-def chunk_conversation(
-    messages: list[dict],
-    max_turns: int = 6,
-    overlap_turns: int = 1
-) -> list[dict]:
-    """Chunk conversations by turn windows with overlap."""
-    chunks = []
-    for i in range(0, len(messages), max_turns - overlap_turns):
-        window = messages[i:i + max_turns]
-        chunk_text = "\n".join(
-            f"{m['role'].upper()}: {m['content']}" for m in window
-        )
-        chunks.append({
-            "text": chunk_text,
-            "metadata": {
-                "start_turn": i,
-                "end_turn": i + len(window) - 1,
-                "turn_count": len(window)
-            }
-        })
-    return chunks
-```
-
-### Tabular Data
-
-Tables should be chunked row-by-row or section-by-section with the header repeated in each chunk.
-
-```python
-def chunk_table(header: str, rows: list[str], rows_per_chunk: int = 10) -> list[dict]:
-    """Chunk a table by row groups, repeating the header in each chunk."""
-    chunks = []
-    for i in range(0, len(rows), rows_per_chunk):
-        chunk_rows = rows[i:i + rows_per_chunk]
-        chunk_text = header + "\n" + "\n".join(chunk_rows)
-        chunks.append({
-            "text": chunk_text,
-            "metadata": {
-                "row_start": i,
-                "row_end": i + len(chunk_rows) - 1
-            }
-        })
-    return chunks
-```
-
-## Chunk Sizing Heuristics
-
-| Embedding Model | Recommended Chunk Size | Max Context Window |
-|----------------|----------------------|-------------------|
-| text-embedding-3-small | 256-512 tokens | 8,191 tokens |
-| text-embedding-3-large | 256-512 tokens | 8,191 tokens |
-| voyage-3 | 256-512 tokens | 32,000 tokens |
-| BAAI/bge-small-en-v1.5 | 256-512 tokens | 512 tokens |
-| Cohere embed-v3 | 256-512 tokens | 512 tokens |
-
-**Why 256-512 tokens?** Empirical studies consistently show that chunks in this range produce the best retrieval quality. Smaller chunks lose context; larger chunks dilute the embedding with unrelated content.
-
-**Exception:** For models with large context windows (voyage-3, jina-v2), chunks up to 1024 tokens can work if the document structure demands it (e.g., long code functions).
-
-## Overlap Configuration
-
-Overlap ensures that content at chunk boundaries is captured by at least one chunk's embedding.
-
-| Document type | Recommended overlap | Rationale |
-|---------------|-------------------|-----------|
-| Technical docs | 10-15% (50-75 tokens) | Paragraphs sometimes span concepts across section boundaries. |
-| Source code | 0% | Functions and classes are self-contained. Overlap wastes index space. |
-| Conversations | 1-2 turns | Context from the previous exchange helps ground the current window. |
-| Legal documents | 15-20% (75-100 tokens) | Legal clauses reference adjacent clauses frequently. |
-
-## Code Block Preservation
-
-Code blocks must never be split. A half-function is worse than no function.
-
-```python
-def split_preserving_code_blocks(
-    text: str,
-    max_tokens: int = 512,
-    overlap_tokens: int = 64
-) -> list[str]:
-    """Split text into chunks, keeping code blocks atomic."""
-    code_block_pattern = r"```[\s\S]*?```"
-    parts = re.split(f"({code_block_pattern})", text)
-
-    chunks = []
-    current_chunk = ""
-
-    for part in parts:
-        is_code = part.startswith("```")
-        part_tokens = count_tokens(part)
-
-        if is_code and part_tokens > max_tokens:
-            # Code block exceeds limit -- keep it as its own chunk
-            if current_chunk.strip():
-                chunks.append(current_chunk.strip())
-                current_chunk = ""
-            chunks.append(part)
-        elif count_tokens(current_chunk + part) <= max_tokens:
-            current_chunk += part
-        else:
-            if current_chunk.strip():
-                chunks.append(current_chunk.strip())
-            current_chunk = get_overlap_text(chunks[-1], overlap_tokens) + part if chunks else part
-
-    if current_chunk.strip():
-        chunks.append(current_chunk.strip())
-
-    return chunks
-```
-
-## Quality Checks
-
-After chunking, validate the output:
-
-1. **No empty chunks.** Filter out chunks with fewer than 20 tokens.
-2. **No giant chunks.** Flag chunks exceeding the embedding model's context window.
-3. **Metadata completeness.** Every chunk must have the required metadata fields.
-4. **Code block integrity.** Verify that no code fences are opened without closing.
-5. **Distribution check.** Plot the token-count histogram. A healthy distribution is unimodal around the target size. Bimodal distributions indicate a chunking rule that produces many tiny and many oversized chunks.

package/workflows/workflows/agent-environment-setup/platforms/gemini/skills/rag-patterns/references/embedding-models.md

@@ -1,208 +0,0 @@
-# Embedding Models
-
-## Overview
-
-The embedding model maps text to dense vectors that capture semantic meaning. Model selection determines retrieval quality, cost, latency, and storage requirements. This reference covers model comparison, selection criteria, fine-tuning, and operational considerations.
-
-## Model Comparison (2025)
-
-| Model | Provider | Dimensions | Max Tokens | MTEB Avg | Cost (per 1M tokens) | Latency (batch) |
-|-------|----------|------------|------------|----------|----------------------|-----------------|
-| text-embedding-3-large | OpenAI | 3072 (configurable) | 8,191 | 64.6 | $0.13 | ~1.5s/100 items |
-| text-embedding-3-small | OpenAI | 1536 | 8,191 | 62.3 | $0.02 | ~0.8s/100 items |
-| voyage-3 | Voyage AI | 1024 | 32,000 | 67.1 | $0.06 | ~1.2s/100 items |
-| voyage-3-lite | Voyage AI | 512 | 32,000 | 63.2 | $0.02 | ~0.6s/100 items |
-| embed-v3.0 | Cohere | 1024 | 512 | 64.5 | $0.10 | ~1.0s/100 items |
-| bge-large-en-v1.5 | BAAI | 1024 | 512 | 63.9 | Self-hosted | Depends on GPU |
-| jina-embeddings-v3 | Jina AI | 1024 | 8,192 | 65.5 | $0.02 | ~1.0s/100 items |
-| nomic-embed-text-v1.5 | Nomic | 768 | 8,192 | 62.8 | Free (open-source) | Self-hosted |
-
-**Note:** MTEB scores are general-purpose benchmarks. Domain-specific performance can differ by 10-20 points. Always benchmark on your own data.
-
-## Selection Criteria
-
-### Quality vs. Cost Trade-off
-
-For most production RAG systems, the embedding model is NOT the bottleneck for answer quality. The gap between the best and worst models on this table is ~5 MTEB points, while chunking strategy and retrieval pipeline design have much larger impact.
-
-**Recommendation:** Start with `text-embedding-3-small` (cheapest, strong quality) and only upgrade if retrieval eval shows the embedding model is the limiting factor.
-
-### Dimension Count and Storage
-
-Higher dimensions capture more semantic nuance but increase storage and query cost:
-
-| Dimensions | Vector size (bytes) | 1M vectors storage | Query latency impact |
-|------------|--------------------|--------------------|---------------------|
-| 384 | 1,536 | ~1.5 GB | Fastest |
-| 768 | 3,072 | ~3.0 GB | Fast |
-| 1024 | 4,096 | ~4.0 GB | Moderate |
-| 1536 | 6,144 | ~6.0 GB | Moderate |
-| 3072 | 12,288 | ~12.0 GB | Slower |
-
-**OpenAI dimension reduction:** `text-embedding-3-large` supports `dimensions` parameter to reduce from 3072 to any lower value. Testing shows 1536 retains 98% of quality at half the storage.
-
-```python
-response = client.embeddings.create(
-    model="text-embedding-3-large",
-    input=["example text"],
-    dimensions=1536  # Reduce from 3072
-)
-```
-
-### Context Window
-
-Models with large context windows (voyage-3 at 32K, jina-v3 at 8K) can embed entire documents or large chunks. However, larger inputs produce embeddings that average over more content, which can dilute topic-specific signal.
-
-**Guideline:** Even with large-context models, keep chunks in the 256-512 token range for optimal retrieval. Use the large context window for late-interaction models or when you need to embed full documents for classification (not retrieval).
-
-## Domain-Specific Benchmarking
-
-### Building a Retrieval Eval Set
-
-```python
-def build_eval_set(queries_with_relevance: list[dict]) -> list[dict]:
-    """
-    Each item: {
-        "query": "How do I authenticate?",
-        "relevant_chunk_ids": ["chunk-auth-001", "chunk-auth-002"],
-        "irrelevant_chunk_ids": ["chunk-billing-001"]  # optional hard negatives
-    }
-    """
-    return queries_with_relevance
-
-def evaluate_model(
-    model_fn,
-    eval_set: list[dict],
-    chunk_embeddings: dict[str, list[float]],
-    k: int = 5
-) -> dict:
-    """Compute Recall@K and MRR@K for a given embedding model."""
-    recall_scores = []
-    mrr_scores = []
-
-    for item in eval_set:
-        query_embedding = model_fn(item["query"])
-        ranked = rank_by_similarity(query_embedding, chunk_embeddings, top_k=k)
-        ranked_ids = [r["id"] for r in ranked]
-
-        relevant = set(item["relevant_chunk_ids"])
-        hits = [1 if rid in relevant else 0 for rid in ranked_ids]
-
-        recall_scores.append(sum(hits) / len(relevant))
-        mrr = next((1.0 / (i + 1) for i, h in enumerate(hits) if h), 0.0)
-        mrr_scores.append(mrr)
-
-    return {
-        "recall_at_k": statistics.mean(recall_scores),
-        "mrr_at_k": statistics.mean(mrr_scores)
-    }
-```
-
-### Interpretation
-
-| Metric | Threshold | Action |
-|--------|-----------|--------|
-| Recall@5 > 0.85 | Good | Embedding model is sufficient. Focus optimization elsewhere. |
-| Recall@5 0.70-0.85 | Acceptable | Try hybrid search or re-ranking before changing the model. |
-| Recall@5 < 0.70 | Poor | Consider a larger model, domain fine-tuning, or hybrid search. |
-
-## Fine-Tuning Embeddings
-
-Fine-tuning adapts a pre-trained embedding model to your domain's vocabulary and similarity patterns. It is most valuable when:
-- Your domain has specialized terminology (medical, legal, financial).
-- General-purpose models confuse domain-specific concepts.
-- You have labeled retrieval pairs (query, relevant_document).
-
-### Synthetic Training Data
-
-If you lack labeled pairs, generate them from your corpus:
-
-```python
-def generate_training_pairs(chunks: list[dict], llm) -> list[dict]:
-    """Generate (question, relevant_chunk) pairs using an LLM."""
-    pairs = []
-    for chunk in chunks:
-        prompt = f"""Given this text passage, generate 3 questions that this passage would answer.
-Return only the questions, one per line.
-
-Passage: {chunk['text']}"""
-        questions = llm.generate(prompt).strip().split("\n")
-        for q in questions:
-            pairs.append({
-                "query": q.strip(),
-                "positive": chunk["text"],
-                "negative": random.choice([c["text"] for c in chunks if c["id"] != chunk["id"]])
-            })
-    return pairs
-```
-
-### Fine-Tuning with Sentence Transformers
-
-```python
-from sentence_transformers import SentenceTransformer, InputExample, losses
-from torch.utils.data import DataLoader
-
-model = SentenceTransformer("BAAI/bge-small-en-v1.5")
-
-train_examples = [
-    InputExample(texts=[pair["query"], pair["positive"], pair["negative"]])
-    for pair in training_pairs
-]
-
-train_dataloader = DataLoader(train_examples, shuffle=True, batch_size=16)
-train_loss = losses.TripletLoss(model=model)
-
-model.fit(
-    train_objectives=[(train_dataloader, train_loss)],
-    epochs=3,
-    warmup_steps=100,
-    output_path="models/domain-tuned-bge"
-)
-```
-
-## Operational Considerations
-
-### Embedding Versioning
-
-When you change the embedding model, ALL existing vectors become incompatible. You must re-index the entire corpus.
-
-**Strategy:** Include the model name in the vector store index/collection name:
-
-```
-support_kb_bge_small_v1  --> current
-support_kb_voyage3_v1    --> migration target
-```
-
-Run both indices in parallel during migration, verify retrieval quality on the new index, then switch over.
-
-### Batching and Rate Limits
-
-```python
-import asyncio
-from tenacity import retry, wait_exponential, stop_after_attempt
-
-@retry(wait=wait_exponential(min=1, max=60), stop=stop_after_attempt(5))
-async def embed_batch(texts: list[str], batch_size: int = 100) -> list[list[float]]:
-    """Embed texts in batches with retry and rate limit handling."""
-    all_embeddings = []
-
-    for i in range(0, len(texts), batch_size):
-        batch = texts[i:i + batch_size]
-        response = await client.embeddings.create(
-            model="text-embedding-3-small",
-            input=batch
-        )
-        all_embeddings.extend([d.embedding for d in response.data])
-
-    return all_embeddings
-```
-
-### Cost Estimation
-
-| Corpus size (tokens) | text-embedding-3-small | text-embedding-3-large | voyage-3 |
-|---------------------|----------------------|----------------------|----------|
-| 10M | $0.20 | $1.30 | $0.60 |
-| 100M | $2.00 | $13.00 | $6.00 |
-| 1B | $20.00 | $130.00 | $60.00 |
-
-These are one-time ingestion costs. Query embedding costs are negligible (single queries are fractions of a cent).