@musashishao/agent-kit 1.2.2 → 1.4.0

package/.agent/skills/rag-engineering/contextual-retrieval.md

@@ -0,0 +1,261 @@
+# Contextual Retrieval
+
+> Anthropic's technique that reduces retrieval failures by 49%.
+
+---
+
+## The Problem
+
+Standard chunks lack context:
+
+```typescript
+// This chunk is meaningless in isolation
+return amount * rate;
+```
+
+Questions the AI can't answer:
+- What is `amount`?
+- Where does `rate` come from?
+- What calls this code?
+
+---
+
+## The Solution
+
+Add a **context summary** before embedding:
+
+```
+[CONTEXT]
+File: src/utils/tax.ts
+Function: calculateTax
+Purpose: Calculates tax amount for customer billing
+Called by: ShippingService.calculateShipping(), InvoiceGenerator.generate()
+Dependencies: TAX_RATE constant from ./constants.ts
+[/CONTEXT]
+
+function calculateTax(amount: number): number {
+  const rate = TAX_RATE;
+  return amount * rate;
+}
+```
+
+Now the embedding captures:
+- File location
+- Function purpose
+- Callers and dependencies
+- Business context
+
+---
+
+## Implementation
+
+### Step 1: Gather Context Sources
+
+```python
+def gather_context(chunk: dict, project_info: dict, graph: dict) -> dict:
+    """Collect all context sources for a chunk."""
+    
+    file_path = chunk['metadata']['file_path']
+    chunk_name = chunk['metadata'].get('name', 'unknown')
+    
+    return {
+        'file_path': file_path,
+        'chunk_name': chunk_name,
+        'file_purpose': get_file_purpose(file_path, project_info),
+        'dependencies': graph.get_dependencies(file_path),
+        'dependents': graph.get_dependents(file_path),
+        'project_context': project_info.get('summary', ''),
+    }
+```
+
+### Step 2: Generate Context Summary
+
+Option A: **Template-based** (Fast, deterministic)
+
+```python
+def generate_context_template(ctx: dict) -> str:
+    """Generate context using templates."""
+    
+    return f"""[CONTEXT]
+File: {ctx['file_path']}
+Name: {ctx['chunk_name']}
+Purpose: {ctx['file_purpose']}
+Imported by: {', '.join(ctx['dependents'][:5]) or 'None'}
+Imports: {', '.join(ctx['dependencies'][:5]) or 'None'}
+[/CONTEXT]"""
+```
+
+Option B: **LLM-generated** (Rich, semantic)
+
+```python
+def generate_context_llm(chunk: str, ctx: dict) -> str:
+    """Generate context using LLM."""
+    
+    prompt = f"""Analyze this code chunk and write a 2-3 sentence summary.
+
+File: {ctx['file_path']}
+Project: {ctx['project_context']}
+Dependencies: {ctx['dependencies']}
+Used by: {ctx['dependents']}
+
+Code:
+```
+{chunk}
+```
+
+Summary (2-3 sentences explaining what this code does and its role):"""
+    
+    return llm.generate(prompt, max_tokens=100)
+```
+
+### Step 3: Combine Context + Content
+
+```python
+def create_contextual_chunk(chunk: dict, context: str) -> dict:
+    """Create final chunk with context prepended."""
+    
+    contextual_content = f"{context}\n\n{chunk['content']}"
+    
+    return {
+        'id': chunk['id'],
+        'content': contextual_content,
+        'original_content': chunk['content'],
+        'context': context,
+        'metadata': chunk['metadata']
+    }
+```
+
+---
+
+## Context Sources (Priority Order)
+
+| Source | Information | Priority |
+|--------|-------------|----------|
+| **File Path** | Location in project | P0 (Always) |
+| **AGENTS.md** | Project overview | P0 (Always) |
+| **Knowledge Graph** | Dependencies | P1 (If available) |
+| **Docstrings/Comments** | Developer intent | P1 (If available) |
+| **Git History** | Change context | P2 (Optional) |
+
+---
+
+## When to Use LLM vs Template
+
+| Scenario | Approach | Reason |
+|----------|----------|--------|
+| Initial indexing | Template | Speed |
+| High-value files | LLM | Quality |
+| Simple utilities | Template | Overkill for LLM |
+| Complex business logic | LLM | Need semantic understanding |
+
+---
+
+## Cost Optimization
+
+LLM context generation is expensive. Optimize:
+
+1. **Cache aggressively** - Context rarely changes
+2. **Batch processing** - Send multiple chunks per request
+3. **Use smaller models** - Claude Haiku, GPT-3.5-turbo
+4. **Template for simple cases** - 80% template, 20% LLM
+
+### Cost Estimation
+
+| Codebase Size | Chunks | LLM Calls | Est. Cost |
+|---------------|--------|-----------|-----------|
+| 10k lines | ~200 | 200 | ~$0.50 |
+| 100k lines | ~2,000 | 2,000 | ~$5.00 |
+| 1M lines | ~20,000 | 20,000 | ~$50.00 |
+
+---
+
+## Quality Measurement
+
+### Good Context Indicators
+
+- [ ] Mentions file purpose
+- [ ] Lists key dependencies
+- [ ] Describes what code does (not just syntax)
+- [ ] Under 100 tokens (concise)
+
+### Bad Context Examples
+
+❌ Too vague:
+```
+[CONTEXT]
+This is code from the project.
+[/CONTEXT]
+```
+
+❌ Too long:
+```
+[CONTEXT]
+This function is located in src/utils/tax.ts which is part of the utilities 
+folder that contains various helper functions used throughout the application 
+for different purposes including but not limited to calculations, formatting,
+validation, and data transformation. The specific function calculateTax...
+(200 more tokens)
+[/CONTEXT]
+```
+
+✅ Just right:
+```
+[CONTEXT]
+File: src/utils/tax.ts
+Function calculateTax computes tax for billing. Used by ShippingService 
+and InvoiceGenerator. Depends on TAX_RATE from constants.
+[/CONTEXT]
+```
+
+---
+
+## Integration with graph-mapper
+
+Use Knowledge Graph data for richer context:
+
+```python
+from graph_mapper import load_graph
+
+def enrich_context_with_graph(chunk: dict, graph_path: str) -> dict:
+    """Add dependency info from graph."""
+    
+    graph = load_graph(graph_path)
+    file_id = chunk['metadata']['file_path']
+    
+    # Get direct dependencies
+    imports = graph.get_imports(file_id)
+    
+    # Get files that import this
+    imported_by = graph.get_importers(file_id)
+    
+    # Calculate impact score
+    impact = graph.calculate_impact(file_id)
+    
+    return {
+        'imports': imports,
+        'imported_by': imported_by,
+        'impact_score': impact['score'],
+        'impact_files': impact['files'][:5]
+    }
+```
+
+---
+
+## Retrieval Impact
+
+### Without Contextual Retrieval
+
+Query: "How is tax calculated for shipping?"
+
+Results: Random code mentioning "tax" or "shipping"
+- ❌ `const TAX = 0.1;` (irrelevant constant)
+- ❌ `// TODO: add tax` (comment)
+- ⚠️ `calculateTax(...)` (correct but no context)
+
+### With Contextual Retrieval
+
+Results: Properly contextualized chunks
+- ✅ `[calculateTax used by ShippingService] function calculateTax...`
+- ✅ `[ShippingService imports calculateTax] class ShippingService...`
+
+**Improvement: 35-49% reduction in failed retrievals** (Anthropic research)

package/.agent/skills/rag-engineering/hybrid-search.md

@@ -0,0 +1,356 @@
+# Hybrid Search
+
+> Combine semantic (vector) and keyword (BM25) search for best results.
+
+---
+
+## Why Hybrid?
+
+| Query Type | Vector Search | BM25 Search | Winner |
+|------------|---------------|-------------|--------|
+| "How to handle user authentication" | ✅ Great | ⚠️ Okay | Vector |
+| "Error 503" | ❌ Poor | ✅ Great | BM25 |
+| "calculateTax function" | ⚠️ Okay | ✅ Great | BM25 |
+| "similar to login flow" | ✅ Great | ❌ Poor | Vector |
+
+**Hybrid = Best of both worlds**
+
+---
+
+## Architecture
+
+```
+                    ┌─────────────────┐
+                    │     Query       │
+                    └────────┬────────┘
+                             │
+              ┌──────────────┴──────────────┐
+              │                             │
+              ▼                             ▼
+    ┌─────────────────┐           ┌─────────────────┐
+    │  Vector Search  │           │   BM25 Search   │
+    │   (Semantic)    │           │   (Keyword)     │
+    └────────┬────────┘           └────────┬────────┘
+             │                             │
+             │   Top 50 results each       │
+             │                             │
+             └──────────────┬──────────────┘
+                            │
+                            ▼
+                  ┌─────────────────┐
+                  │ Fusion (RRF)    │
+                  │ Combine ranks   │
+                  └────────┬────────┘
+                           │
+                           ▼
+                  ┌─────────────────┐
+                  │   Reranker      │
+                  │   Top 10        │
+                  └────────┬────────┘
+                           │
+                           ▼
+                  ┌─────────────────┐
+                  │  Final Results  │
+                  └─────────────────┘
+```
+
+---
+
+## Implementation
+
+### Step 1: Set Up Both Indexes
+
+```python
+import chromadb
+from rank_bm25 import BM25Okapi
+
+class HybridSearchEngine:
+    def __init__(self, chunks: list[dict]):
+        # Vector index (ChromaDB)
+        self.chroma_client = chromadb.Client()
+        self.collection = self.chroma_client.create_collection("code_chunks")
+        
+        # Add chunks to vector DB
+        self.collection.add(
+            documents=[c['content'] for c in chunks],
+            ids=[c['id'] for c in chunks],
+            metadatas=[c['metadata'] for c in chunks]
+        )
+        
+        # BM25 index
+        tokenized_chunks = [c['content'].split() for c in chunks]
+        self.bm25 = BM25Okapi(tokenized_chunks)
+        self.chunk_ids = [c['id'] for c in chunks]
+        self.chunks = {c['id']: c for c in chunks}
+```
+
+### Step 2: Implement Search Methods
+
+```python
+def vector_search(self, query: str, top_k: int = 50) -> list[tuple]:
+    """Semantic search using embeddings."""
+    results = self.collection.query(
+        query_texts=[query],
+        n_results=top_k
+    )
+    
+    # Return [(id, score), ...]
+    return list(zip(
+        results['ids'][0],
+        results['distances'][0]
+    ))
+
+def bm25_search(self, query: str, top_k: int = 50) -> list[tuple]:
+    """Keyword search using BM25."""
+    tokenized_query = query.split()
+    scores = self.bm25.get_scores(tokenized_query)
+    
+    # Get top k
+    top_indices = sorted(
+        range(len(scores)),
+        key=lambda i: scores[i],
+        reverse=True
+    )[:top_k]
+    
+    return [(self.chunk_ids[i], scores[i]) for i in top_indices]
+```
+
+### Step 3: Reciprocal Rank Fusion
+
+```python
+def reciprocal_rank_fusion(
+    self,
+    results_list: list[list[tuple]],
+    k: int = 60
+) -> list[tuple]:
+    """
+    Combine multiple ranked lists using RRF.
+    
+    Formula: score(d) = Σ 1/(k + rank(d))
+    
+    Args:
+        results_list: List of [(id, score), ...] for each search method
+        k: Constant to prevent high scores (default 60)
+    """
+    scores = {}
+    
+    for results in results_list:
+        for rank, (doc_id, _) in enumerate(results):
+            if doc_id not in scores:
+                scores[doc_id] = 0
+            scores[doc_id] += 1 / (k + rank + 1)
+    
+    # Sort by combined score
+    sorted_results = sorted(
+        scores.items(),
+        key=lambda x: x[1],
+        reverse=True
+    )
+    
+    return sorted_results
+```
+
+### Step 4: Hybrid Search Method
+
+```python
+def hybrid_search(
+    self,
+    query: str,
+    top_k: int = 20,
+    vector_weight: float = 0.5
+) -> list[dict]:
+    """
+    Perform hybrid search combining vector and BM25.
+    
+    Args:
+        query: Search query
+        top_k: Number of results to return
+        vector_weight: Weight for vector search (0-1)
+    """
+    # Get results from both methods
+    vector_results = self.vector_search(query, top_k=50)
+    bm25_results = self.bm25_search(query, top_k=50)
+    
+    # Fuse results
+    fused = self.reciprocal_rank_fusion([vector_results, bm25_results])
+    
+    # Get top k chunks
+    top_ids = [doc_id for doc_id, score in fused[:top_k]]
+    
+    return [self.chunks[doc_id] for doc_id in top_ids]
+```
+
+---
+
+## Weighting Strategies
+
+### Fixed Weights
+
+```python
+# Equal weight
+vector_weight = 0.5
+bm25_weight = 0.5
+
+# Semantic-heavy (for conceptual queries)
+vector_weight = 0.7
+bm25_weight = 0.3
+
+# Keyword-heavy (for exact matches)
+vector_weight = 0.3
+bm25_weight = 0.7
+```
+
+### Dynamic Weights
+
+```python
+def determine_weights(query: str) -> tuple[float, float]:
+    """Dynamically adjust weights based on query type."""
+    
+    # Check for exact identifiers (function names, error codes)
+    has_identifier = bool(re.search(r'[A-Z][a-z]+[A-Z]|Error\s*\d+|_\w+_', query))
+    
+    # Check for conceptual language
+    conceptual_words = {'how', 'what', 'why', 'similar', 'like', 'related'}
+    is_conceptual = any(word in query.lower() for word in conceptual_words)
+    
+    if has_identifier:
+        return (0.3, 0.7)  # Favor BM25
+    elif is_conceptual:
+        return (0.7, 0.3)  # Favor Vector
+    else:
+        return (0.5, 0.5)  # Equal
+```
+
+---
+
+## Alternative: Weighted Score Fusion
+
+Instead of RRF, use weighted scores directly:
+
+```python
+def weighted_score_fusion(
+    vector_results: list[tuple],
+    bm25_results: list[tuple],
+    vector_weight: float = 0.5
+) -> list[tuple]:
+    """Combine using normalized weighted scores."""
+    
+    # Normalize scores to 0-1 range
+    def normalize(results):
+        if not results:
+            return {}
+        scores = [s for _, s in results]
+        min_s, max_s = min(scores), max(scores)
+        range_s = max_s - min_s or 1
+        return {
+            doc_id: (score - min_s) / range_s
+            for doc_id, score in results
+        }
+    
+    vector_norm = normalize(vector_results)
+    bm25_norm = normalize(bm25_results)
+    
+    # Combine weighted scores
+    all_ids = set(vector_norm.keys()) | set(bm25_norm.keys())
+    combined = {}
+    
+    for doc_id in all_ids:
+        v_score = vector_norm.get(doc_id, 0) * vector_weight
+        b_score = bm25_norm.get(doc_id, 0) * (1 - vector_weight)
+        combined[doc_id] = v_score + b_score
+    
+    return sorted(combined.items(), key=lambda x: x[1], reverse=True)
+```
+
+---
+
+## Performance Optimization
+
+### Caching
+
+```python
+from functools import lru_cache
+
+@lru_cache(maxsize=1000)
+def cached_search(query: str, top_k: int = 20) -> tuple:
+    """Cache frequent queries."""
+    results = hybrid_search(query, top_k)
+    return tuple(r['id'] for r in results)
+```
+
+### Batch Queries
+
+```python
+def batch_hybrid_search(queries: list[str], top_k: int = 20) -> list[list[dict]]:
+    """Process multiple queries efficiently."""
+    
+    # Batch vector search
+    vector_results = collection.query(
+        query_texts=queries,
+        n_results=50
+    )
+    
+    # BM25 for each (can't batch easily)
+    bm25_results = [bm25_search(q, 50) for q in queries]
+    
+    # Fuse each
+    return [
+        reciprocal_rank_fusion([vector_results[i], bm25_results[i]])[:top_k]
+        for i in range(len(queries))
+    ]
+```
+
+---
+
+## Evaluation Metrics
+
+### Hit Rate @ K
+
+```python
+def hit_rate(queries: list, ground_truth: list, k: int = 10) -> float:
+    """Percentage of queries where correct answer is in top K."""
+    hits = 0
+    for query, expected in zip(queries, ground_truth):
+        results = hybrid_search(query, top_k=k)
+        result_ids = [r['id'] for r in results]
+        if expected in result_ids:
+            hits += 1
+    return hits / len(queries)
+```
+
+### Mean Reciprocal Rank (MRR)
+
+```python
+def mrr(queries: list, ground_truth: list) -> float:
+    """Average of 1/rank of first correct result."""
+    reciprocal_ranks = []
+    for query, expected in zip(queries, ground_truth):
+        results = hybrid_search(query, top_k=100)
+        for rank, result in enumerate(results, 1):
+            if result['id'] == expected:
+                reciprocal_ranks.append(1 / rank)
+                break
+        else:
+            reciprocal_ranks.append(0)
+    return sum(reciprocal_ranks) / len(reciprocal_ranks)
+```
+
+---
+
+## Quick Setup (Copy-Paste Ready)
+
+```python
+# Install dependencies
+# pip install chromadb rank-bm25
+
+import chromadb
+from rank_bm25 import BM25Okapi
+
+def create_hybrid_engine(chunks):
+    """One-liner to create hybrid search engine."""
+    return HybridSearchEngine(chunks)
+
+# Usage
+engine = create_hybrid_engine(my_chunks)
+results = engine.hybrid_search("how to calculate tax", top_k=10)
+```