@cubis/foundry 0.3.82 → 0.3.84

package/workflows/workflows/agent-environment-setup/platforms/copilot/skills/rag-patterns/SKILL.md

@@ -1,80 +0,0 @@
----
-name: rag-patterns
-description: "Use when designing or optimizing Retrieval-Augmented Generation systems: chunking strategies, embedding model selection, vector store configuration, retrieval optimization, hybrid search, re-ranking, and production RAG pipelines."
-license: MIT
-metadata:
-  author: cubis-foundry
-  version: "3.0"
-compatibility: Claude Code, Codex, GitHub Copilot
----
-# RAG Patterns
-
-## Purpose
-
-Guide the design, implementation, and optimization of production-grade Retrieval-Augmented Generation pipelines. Every instruction ensures that documents are chunked for retrieval quality rather than convenience, embeddings capture semantic meaning, retrieval returns the most relevant context, and the generation model produces grounded answers.
-
-## When to Use
-
-- Designing a RAG pipeline for a new knowledge base or document corpus.
-- Choosing a chunking strategy for a specific document type (code, legal, technical docs).
-- Selecting and configuring embedding models for a domain.
-- Setting up or migrating a vector store (Pinecone, Weaviate, Qdrant, pgvector, Chroma).
-- Optimizing retrieval quality with re-ranking, hybrid search, or query expansion.
-- Debugging RAG systems that produce hallucinated or irrelevant answers.
-- Scaling RAG pipelines for latency, throughput, or cost constraints.
-
-## Instructions
-
-1. **Map the document corpus before choosing a chunking strategy** because the optimal chunk size and boundary rules depend on document structure. Code files need function-level chunks, legal documents need section-level, and conversations need turn-level. A universal chunk size wastes retrieval precision.
-
-2. **Chunk at semantic boundaries, not fixed token counts** because splitting mid-sentence or mid-paragraph destroys the semantic coherence that embeddings rely on. Use headings, paragraph breaks, or AST nodes as natural boundaries, then enforce a token budget as a secondary constraint.
-
-3. **Include overlap between adjacent chunks** because queries often match concepts that span a chunk boundary. 10-15% overlap ensures that boundary-spanning content appears in at least one chunk's embedding.
-
-4. **Attach metadata to every chunk at ingestion time** because retrieval without filtering is a brute-force search. Metadata (source document, section title, date, author, document type) enables filtered retrieval that narrows the search space before vector similarity runs.
-
-5. **Select embedding models based on domain benchmarks, not general leaderboards** because MTEB scores on Wikipedia do not predict performance on your legal contracts or API documentation. Run a retrieval eval on 50-100 representative queries from your domain before committing to a model.
-
-6. **Normalize embedding dimensions when using cosine similarity** because unnormalized vectors distort similarity scores. Most vector stores handle this automatically, but verify the configuration to avoid silent precision loss.
-
-7. **Choose the vector store based on operational requirements, not feature lists** because every store handles millions of vectors but they differ on filtering speed, multi-tenancy support, managed vs. self-hosted operations, and cost at scale. Match the store to your deployment model.
-
-8. **Implement hybrid search combining dense vectors and sparse keyword matching** because dense embeddings excel at semantic similarity but miss exact keyword matches (product codes, error messages, proper nouns). BM25 or sparse vectors catch what dense search misses.
-
-9. **Add a re-ranker between retrieval and generation** because the initial retrieval (top-50 by cosine similarity) optimizes for recall while the generation context window needs precision. A cross-encoder re-ranker reorders by relevance and lets you send only the top-5 to the LLM.
-
-10. **Limit the context window to the most relevant chunks** because stuffing 20 chunks into the prompt dilutes the signal and increases latency and cost. Fewer, higher-quality chunks produce better answers than more, lower-quality ones.
-
-11. **Include source citations in the generation prompt** because grounded answers with traceable citations allow users to verify claims and build trust. Instruct the LLM to reference chunk IDs or document titles in its response.
-
-12. **Build a retrieval evaluation set before optimizing** because retrieval improvements without measurement are guesswork. A set of 50-100 queries with known-relevant documents enables Recall@K and MRR metrics that quantify every change.
-
-13. **Monitor retrieval quality in production with user feedback signals** because offline evals do not capture distribution shift. Track "thumbs down" rates, query reformulations, and zero-result queries to detect degradation.
-
-14. **Cache frequent queries and their retrieval results** because identical or near-identical queries hitting the vector store on every request waste latency and compute. Semantic caching (embedding the query and checking cache similarity) reduces redundant retrievals.
-
-15. **Version your ingestion pipeline alongside the embedding model** because re-indexing is expensive and a model change without full re-indexing creates a mixed-embedding index where similarity scores are meaningless across model versions.
-
-16. **Plan for re-indexing from day one** because embedding models improve, chunking strategies evolve, and metadata schemas change. A pipeline that cannot re-index the full corpus within a maintenance window becomes a legacy liability.
-
-## Output Format
-
-Provide architecture diagrams as ASCII or Mermaid, configuration snippets for vector stores and embedding models, chunking implementation code, retrieval pipeline code, and evaluation results as metric tables.
-
-## References
-
-| File | Load when |
-| --- | --- |
-| `references/chunking.md` | Designing chunking strategies for specific document types. |
-| `references/embedding-models.md` | Selecting, configuring, or fine-tuning embedding models. |
-| `references/vector-stores.md` | Choosing, configuring, or migrating vector stores. |
-| `references/retrieval-optimization.md` | Improving retrieval quality with re-ranking, query expansion, or filtering. |
-| `references/hybrid-search.md` | Implementing hybrid dense + sparse search pipelines. |
-
-## Copilot Platform Notes
-
-- Skill files are stored under `.github/prompts/` (prompt files) and `.github/instructions/` (instruction files).
-- Copilot does not support subagent spawning — all skill guidance executes within the current conversation context.
-- User arguments are provided as natural language input in the prompt, not through a `$ARGUMENTS` variable.
-- Frontmatter keys `context`, `agent`, and `allowed-tools` are not supported; guidance is advisory only.
-- Reference files can be included via `#file:references/<name>.md` syntax in Copilot Chat.

package/workflows/workflows/agent-environment-setup/platforms/copilot/skills/rag-patterns/evals/assertions.md

@@ -1,49 +0,0 @@
-# RAG Patterns Eval Assertions
-
-## Eval 1: Chunking Strategy for Technical Docs
-
-This eval tests the skill's ability to design a document-type-aware chunking strategy with proper semantic boundaries, overlap, metadata, and size constraints.
-
-### Assertions
-
-| # | Type | What it checks | Why it matters |
-|---|------|----------------|----------------|
-| 1 | contains | `semantic boundary` -- structure-aware chunking | Fixed-size chunking destroys document structure. Technical docs have headings, sections, and endpoints that define natural chunk boundaries. |
-| 2 | contains | `overlap` -- inter-chunk overlap | Queries matching concepts at chunk boundaries return no results without overlap. 10-15% overlap is standard for retrieval quality. |
-| 3 | contains | `metadata` -- chunk metadata schema | Metadata enables filtered retrieval (e.g., search only API references, filter by version). Without metadata, every query searches the full index. |
-| 4 | contains | `code block` -- code-specific handling | Code examples are atomic units. A function split across two chunks is useless for developer Q&A. The strategy must keep code blocks intact. |
-| 5 | contains | `token` -- token-based size constraints | Character-based limits do not account for tokenizer behavior. Embedding models have token limits, and the LLM context window is measured in tokens. |
-
-### What a passing response looks like
-
-- Three distinct chunking strategies for the three document types (API refs, tutorials, changelogs).
-- API references chunked by endpoint (one chunk per operation with description, parameters, and response schema).
-- Tutorials chunked by section with heading-based boundaries, code blocks kept intact, and 10-15% overlap.
-- Changelogs chunked by release version.
-- A metadata schema with fields: doc_type, title, section, version, api_endpoint (nullable), language (for code).
-- Chunk size target of 256-512 tokens with a hard ceiling and soft floor.
-- A code-block preservation rule: if a code block would push the chunk over the limit, the code block becomes its own chunk with the preceding paragraph as context.
-
----
-
-## Eval 2: Hybrid Search Implementation
-
-This eval tests the skill's ability to implement a complete hybrid search pipeline with dense + sparse retrieval, fusion scoring, and re-ranking.
-
-### Assertions
-
-| # | Type | What it checks | Why it matters |
-|---|------|----------------|----------------|
-| 1 | contains | `BM25` -- sparse retrieval component | Dense embeddings encode meaning but miss exact lexical matches. Error codes like "ERR-4502" and SKUs need keyword matching. |
-| 2 | contains | `fusion` -- result merging strategy | Dense and sparse retrievers return separate ranked lists. Without a principled fusion strategy, the pipeline cannot combine their strengths. |
-| 3 | contains | `re-rank` -- cross-encoder re-ranking | Initial retrieval optimizes for recall (find everything that might be relevant). Re-ranking optimizes for precision (put the best results first). |
-| 4 | contains | `Qdrant` -- vector store implementation | The eval specifies Qdrant. The implementation must use Qdrant's sparse vector support or payload indexing, not a generic abstraction. |
-| 5 | contains | `weight` -- configurable fusion weights | The optimal dense/sparse balance varies by query type. Keyword-heavy queries need more sparse weight; conceptual queries need more dense weight. |
-
-### What a passing response looks like
-
-- An indexing pipeline that stores both dense vectors and sparse representations (BM25 or SPLADE) in Qdrant.
-- A query pipeline that: (a) embeds the query with the dense model, (b) generates a sparse query representation, (c) retrieves top-50 from each, (d) fuses results with RRF or weighted scoring, (e) re-ranks top-20 with a cross-encoder, (f) returns top-5 to the LLM.
-- Qdrant-specific configuration showing named vectors or sparse vector fields.
-- Configurable weights with a sensible default (e.g., 0.6 dense, 0.4 sparse) and guidance on tuning.
-- A worked example showing how "error code ERR-4502 payment failed" benefits from hybrid search vs. dense-only.

package/workflows/workflows/agent-environment-setup/platforms/copilot/skills/rag-patterns/evals/evals.json

@@ -1,66 +0,0 @@
-[
-  {
-    "name": "chunking-strategy-for-technical-docs",
-    "description": "Validate that the skill produces a well-reasoned chunking strategy for technical documentation, covering semantic boundaries, overlap, metadata, and size constraints appropriate for the document type.",
-    "input": "Design a chunking strategy for a technical documentation corpus that includes API references (OpenAPI specs), tutorial guides (markdown with code blocks), and changelog entries. The corpus has 5,000 documents totaling 50 million tokens. The target vector store is Pinecone with a 512-dimension embedding model. Chunks will be used for a developer-facing Q&A chatbot. Provide the chunking logic, metadata schema, and recommended chunk sizes.",
-    "assertions": [
-      {
-        "type": "contains",
-        "value": "semantic boundary",
-        "description": "The strategy must chunk at semantic boundaries (headings, sections, function definitions) rather than fixed token counts, because technical docs have natural structure that should be preserved."
-      },
-      {
-        "type": "contains",
-        "value": "overlap",
-        "description": "The strategy must include chunk overlap (10-15%) to handle queries that match concepts spanning adjacent chunks in tutorials and guides."
-      },
-      {
-        "type": "contains",
-        "value": "metadata",
-        "description": "The strategy must define a metadata schema attached to each chunk (document type, section title, API endpoint, version) to enable filtered retrieval."
-      },
-      {
-        "type": "contains",
-        "value": "code block",
-        "description": "The strategy must address code blocks specifically, because splitting a code example across chunks destroys its utility for developer Q&A."
-      },
-      {
-        "type": "contains",
-        "value": "token",
-        "description": "The strategy must specify chunk size constraints in tokens (not just characters) to align with the embedding model's context window and the LLM's generation budget."
-      }
-    ]
-  },
-  {
-    "name": "hybrid-search-implementation",
-    "description": "Validate that the skill produces a hybrid search implementation combining dense vector search with sparse keyword matching, including fusion scoring, re-ranking, and configuration for a production deployment.",
-    "input": "Implement a hybrid search pipeline for a customer support knowledge base. The system must handle queries like 'error code ERR-4502 payment failed' where both semantic meaning and exact keyword matching matter. The current dense-only retrieval misses exact error codes and product SKUs. Use Qdrant as the vector store. Provide the indexing pipeline, query pipeline, and fusion scoring logic.",
-    "assertions": [
-      {
-        "type": "contains",
-        "value": "BM25",
-        "description": "The implementation must include BM25 or equivalent sparse retrieval to capture exact keyword matches that dense embeddings miss (error codes, SKUs, proper nouns)."
-      },
-      {
-        "type": "contains",
-        "value": "fusion",
-        "description": "The pipeline must include a fusion step (RRF, weighted combination, or similar) that merges dense and sparse results into a single ranked list."
-      },
-      {
-        "type": "contains",
-        "value": "re-rank",
-        "description": "The pipeline must include a re-ranking step after fusion to refine the top-K results using a cross-encoder or similar model before sending context to the LLM."
-      },
-      {
-        "type": "contains",
-        "value": "Qdrant",
-        "description": "The implementation must use Qdrant as specified and demonstrate its sparse vector or payload indexing capabilities for the keyword matching component."
-      },
-      {
-        "type": "contains",
-        "value": "weight",
-        "description": "The fusion scoring must include configurable weights for dense vs. sparse components, because the optimal balance depends on query distribution and must be tunable."
-      }
-    ]
-  }
-]

package/workflows/workflows/agent-environment-setup/platforms/copilot/skills/rag-patterns/examples/01-developer-docs-rag.md

@@ -1,256 +0,0 @@
-# Example: RAG Pipeline for Developer Documentation
-
-## Context
-
-A developer tools company wants to build a Q&A chatbot over their documentation corpus:
-- 2,000 markdown files covering API references, tutorials, and conceptual guides.
-- Users ask questions like "How do I authenticate with OAuth2?" and "What are the parameters for the /users endpoint?"
-- The chatbot must cite specific documentation pages in its answers.
-- Target latency: < 3 seconds end-to-end.
-
-## Architecture
-
-```
-User Query
-    |
-    v
-[Query Preprocessing] --> expand abbreviations, detect intent
-    |
-    v
-[Embedding Model] --> text-embedding-3-small (1536-dim)
-    |
-    v
-[Vector Store: Pinecone] --> top-50 by cosine similarity
-    |                          filtered by doc_type metadata
-    v
-[Re-ranker: Cohere rerank-v3] --> top-5 by cross-encoder relevance
-    |
-    v
-[Prompt Assembly] --> system prompt + top-5 chunks + user query
-    |
-    v
-[LLM: Claude Sonnet] --> grounded answer with citations
-    |
-    v
-[Response + Source Links]
-```
-
-## Chunking Strategy
-
-### API References
-
-API reference pages follow a consistent structure: endpoint URL, method, parameters table, request/response examples.
-
-```python
-def chunk_api_reference(doc: dict) -> list[dict]:
-    """One chunk per API endpoint operation."""
-    chunks = []
-    for endpoint in doc["endpoints"]:
-        chunk_text = f"""## {endpoint['method']} {endpoint['path']}
-
-{endpoint['description']}
-
-### Parameters
-{format_params_table(endpoint['parameters'])}
-
-### Request Example
-```json
-{json.dumps(endpoint['request_example'], indent=2)}
-```
-
-### Response Example
-```json
-{json.dumps(endpoint['response_example'], indent=2)}
-```"""
-        chunks.append({
-            "text": chunk_text,
-            "metadata": {
-                "doc_type": "api_reference",
-                "endpoint": endpoint["path"],
-                "method": endpoint["method"],
-                "title": doc["title"],
-                "version": doc["api_version"]
-            }
-        })
-    return chunks
-```
-
-**Rationale:** Each endpoint is a self-contained unit that answers "what does this endpoint do?" Splitting an endpoint across chunks would force the retrieval to return multiple chunks for a single question.
-
-### Tutorial Guides
-
-Tutorials are long-form markdown with headings, prose, and code blocks.
-
-```python
-def chunk_tutorial(markdown: str, doc_metadata: dict) -> list[dict]:
-    """Chunk by H2 sections with code-block preservation."""
-    sections = split_by_heading(markdown, level=2)
-    chunks = []
-
-    for section in sections:
-        tokens = count_tokens(section["text"])
-
-        if tokens <= 512:
-            # Section fits in one chunk
-            chunks.append({
-                "text": section["text"],
-                "metadata": {
-                    **doc_metadata,
-                    "doc_type": "tutorial",
-                    "section": section["heading"]
-                }
-            })
-        else:
-            # Split by paragraphs, keeping code blocks atomic
-            sub_chunks = split_preserving_code_blocks(
-                section["text"],
-                max_tokens=512,
-                overlap_tokens=64
-            )
-            for i, sub in enumerate(sub_chunks):
-                chunks.append({
-                    "text": sub,
-                    "metadata": {
-                        **doc_metadata,
-                        "doc_type": "tutorial",
-                        "section": f"{section['heading']} (part {i+1})"
-                    }
-                })
-
-    return chunks
-```
-
-**Rationale:** H2 sections are the natural semantic boundary in tutorials. Code blocks are kept atomic because a split function definition is useless. Overlap of ~64 tokens (roughly 10-15% of 512) ensures boundary-spanning concepts are captured.
-
-### Chunk Size Analysis
-
-| Document type | Chunk count | Median tokens | p95 tokens | Max tokens |
-|---------------|-------------|---------------|------------|------------|
-| API reference | 3,200 | 280 | 450 | 510 |
-| Tutorial | 8,500 | 340 | 500 | 512 |
-| Conceptual | 4,100 | 310 | 480 | 512 |
-| **Total** | **15,800** | **320** | **490** | **512** |
-
-## Embedding Configuration
-
-```python
-from openai import OpenAI
-
-client = OpenAI()
-
-def embed_chunks(chunks: list[dict]) -> list[dict]:
-    """Embed chunks in batches of 100."""
-    batch_size = 100
-    for i in range(0, len(chunks), batch_size):
-        batch = chunks[i:i + batch_size]
-        texts = [c["text"] for c in batch]
-
-        response = client.embeddings.create(
-            model="text-embedding-3-small",
-            input=texts,
-            dimensions=1536
-        )
-
-        for j, embedding_data in enumerate(response.data):
-            batch[j]["embedding"] = embedding_data.embedding
-
-    return chunks
-```
-
-**Model choice:** `text-embedding-3-small` at 1536 dimensions provides strong retrieval quality for technical documentation at low cost ($0.02/1M tokens). For this 50M token corpus, initial embedding costs approximately $1.00.
-
-## Retrieval Pipeline
-
-```python
-from pinecone import Pinecone
-
-pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"])
-index = pc.Index("developer-docs")
-
-def retrieve(query: str, doc_type: str = None, top_k: int = 50) -> list[dict]:
-    query_embedding = embed_query(query)
-
-    filter_dict = {}
-    if doc_type:
-        filter_dict["doc_type"] = {"$eq": doc_type}
-
-    results = index.query(
-        vector=query_embedding,
-        top_k=top_k,
-        include_metadata=True,
-        filter=filter_dict if filter_dict else None
-    )
-
-    return [
-        {
-            "text": match["metadata"]["text"],
-            "score": match["score"],
-            "metadata": match["metadata"]
-        }
-        for match in results["matches"]
-    ]
-```
-
-## Re-ranking
-
-```python
-import cohere
-
-co = cohere.Client(os.environ["COHERE_API_KEY"])
-
-def rerank(query: str, documents: list[dict], top_n: int = 5) -> list[dict]:
-    results = co.rerank(
-        model="rerank-english-v3.0",
-        query=query,
-        documents=[d["text"] for d in documents],
-        top_n=top_n,
-        return_documents=True
-    )
-
-    reranked = []
-    for result in results.results:
-        original = documents[result.index]
-        reranked.append({
-            **original,
-            "rerank_score": result.relevance_score
-        })
-
-    return reranked
-```
-
-## Prompt Assembly
-
-```python
-def build_prompt(query: str, chunks: list[dict]) -> str:
-    context_parts = []
-    for i, chunk in enumerate(chunks):
-        source = chunk["metadata"].get("title", "Unknown")
-        section = chunk["metadata"].get("section", "")
-        context_parts.append(f"[Source {i+1}: {source} > {section}]\n{chunk['text']}")
-
-    context = "\n\n---\n\n".join(context_parts)
-
-    return f"""Answer the developer's question using ONLY the provided documentation context.
-If the context does not contain enough information, say so explicitly.
-Cite sources using [Source N] notation.
-
-## Documentation Context
-
-{context}
-
-## Question
-
-{query}"""
-```
-
-## Evaluation Results
-
-After building the pipeline, a retrieval eval on 100 representative queries produced:
-
-| Metric | Dense only | Dense + Rerank | Hybrid + Rerank |
-|--------|-----------|----------------|-----------------|
-| Recall@5 | 0.72 | 0.72 | 0.84 |
-| MRR@5 | 0.58 | 0.71 | 0.79 |
-| Latency p95 | 180ms | 420ms | 480ms |
-
-The re-ranker improved MRR by 13 points (better ordering) without affecting recall. Adding BM25 hybrid search improved recall by 12 points by catching exact API endpoint names and error codes that dense embeddings missed.