@musashishao/agent-kit 1.2.2 → 1.3.0

package/.agent/skills/rag-engineering/SKILL.md

@@ -0,0 +1,342 @@
+---
+name: rag-engineering
+description: Advanced RAG pipeline engineering for codebases. Contextual chunking, semantic embeddings, hybrid search, and reranking strategies. Build production-grade retrieval systems.
+allowed-tools: Read, Write, Edit, Bash, Glob, Grep
+skills:
+  - database-design
+  - architecture
+---
+
+# RAG Engineering
+
+> Build **Production-Grade Retrieval-Augmented Generation** pipelines for code.
+
+## 🎯 Purpose
+
+This skill teaches you to build RAG systems that actually work for large codebases (200k+ files). Standard RAG fails for code because:
+- Random chunking breaks functions mid-logic
+- Semantic search misses exact identifiers (`Error 503`)
+- No context = AI retrieves wrong code
+
+---
+
+## 🔧 Quick Reference
+
+| File | Purpose |
+|------|---------|
+| [chunking-strategies.md](chunking-strategies.md) | Smart chunking by logical boundaries |
+| [contextual-retrieval.md](contextual-retrieval.md) | Add context before embedding (Anthropic method) |
+| [hybrid-search.md](hybrid-search.md) | Vector + BM25 combination |
+| [scripts/chunk_code.py](scripts/chunk_code.py) | Code chunking script |
+| [scripts/embed_chunks.py](scripts/embed_chunks.py) | Embedding generation script |
+
+---
+
+## 📊 The RAG Pipeline
+
+```
+┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐
+│   Ingest    │───►│   Chunk     │───►│   Embed     │───►│   Store     │
+│  (Files)    │    │  (Smart)    │    │ (Semantic)  │    │ (Vector DB) │
+└─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘
+                                                                │
+┌─────────────┐    ┌─────────────┐    ┌─────────────┐           │
+│   Output    │◄───│  Rerank     │◄───│  Retrieve   │◄──────────┘
+│  (Context)  │    │  (Top K)    │    │  (Hybrid)   │
+└─────────────┘    └─────────────┘    └─────────────┘
+```
+
+---
+
+## 1. Smart Chunking (The Foundation)
+
+### ❌ Bad: Random Chunking
+
+```
+Chunk 1: "function calculateTax(amount) { const rate = 0.1; retu"
+Chunk 2: "rn amount * rate; } function calculateTotal(items) {"
+```
+→ AI gets broken, meaningless code
+
+### ✅ Good: Logical Boundary Chunking
+
+```
+Chunk 1: "function calculateTax(amount) { const rate = 0.1; return amount * rate; }"
+Chunk 2: "function calculateTotal(items) { ... }"
+```
+→ AI gets complete, understandable units
+
+### Chunking Rules
+
+| Language | Boundaries | Tool |
+|----------|------------|------|
+| TypeScript/JS | Function, Class, Module | tree-sitter, regex |
+| Python | Function, Class, Module | ast module |
+| Go | Function, Type, Package | go/parser |
+
+### Chunk Metadata
+
+Always attach metadata to chunks:
+
+```json
+{
+  "content": "function calculateTax(amount) { ... }",
+  "metadata": {
+    "file_path": "src/utils/tax.ts",
+    "type": "function",
+    "name": "calculateTax",
+    "start_line": 15,
+    "end_line": 20,
+    "exports": ["calculateTax"],
+    "imports": ["TAX_RATE from ./constants"]
+  }
+}
+```
+
+---
+
+## 2. Contextual Retrieval (The Secret Sauce)
+
+> **Anthropic's technique that reduces retrieval errors by 49%.**
+
+### Problem
+A chunk like `return amount * rate;` is meaningless without context.
+
+### Solution
+Before embedding, prepend a context summary:
+
+```
+[CONTEXT]
+This chunk is from file `src/utils/tax.ts`.
+It contains the function `calculateTax` which calculates tax for billing.
+This function is used by the ShippingService and InvoiceGenerator.
+[/CONTEXT]
+
+function calculateTax(amount: number): number {
+  const rate = TAX_RATE;
+  return amount * rate;
+}
+```
+
+### How to Generate Context
+
+Use LLM to generate summaries:
+
+```python
+def generate_context(chunk: str, file_path: str, project_summary: str) -> str:
+    prompt = f"""
+    Given this code chunk from {file_path}:
+    ```
+    {chunk}
+    ```
+    
+    Project context: {project_summary}
+    
+    Write a 2-3 sentence summary explaining:
+    1. What this code does
+    2. Where it fits in the project
+    3. What depends on it
+    """
+    return llm.generate(prompt)
+```
+
+---
+
+## 3. Hybrid Search (Best of Both Worlds)
+
+### Why Hybrid?
+
+| Search Type | Strength | Weakness |
+|-------------|----------|----------|
+| **Vector (Semantic)** | Finds similar meaning | Misses exact terms |
+| **BM25 (Keyword)** | Finds exact matches | Misses synonyms |
+| **Hybrid** | Both! | Slightly more complex |
+
+### Implementation Strategy
+
+```python
+def hybrid_search(query: str, top_k: int = 20) -> List[Chunk]:
+    # 1. Vector search (semantic)
+    vector_results = vector_db.search(embed(query), top_k=top_k)
+    
+    # 2. BM25 search (keyword)
+    bm25_results = bm25_index.search(query, top_k=top_k)
+    
+    # 3. Reciprocal Rank Fusion (RRF)
+    combined = reciprocal_rank_fusion(vector_results, bm25_results)
+    
+    return combined[:top_k]
+```
+
+### Reciprocal Rank Fusion (RRF)
+
+```python
+def reciprocal_rank_fusion(results_list: List[List], k: int = 60) -> List:
+    """Combine multiple result lists using RRF."""
+    scores = {}
+    
+    for results in results_list:
+        for rank, item in enumerate(results):
+            if item.id not in scores:
+                scores[item.id] = 0
+            scores[item.id] += 1 / (k + rank)
+    
+    # Sort by combined score
+    return sorted(scores.items(), key=lambda x: x[1], reverse=True)
+```
+
+---
+
+## 4. Reranking (The Final Filter)
+
+### Why Rerank?
+
+Initial retrieval gets top 100 candidates. Reranking picks the best 10-20.
+
+### Reranking Strategies
+
+| Strategy | Accuracy | Speed | Cost |
+|----------|----------|-------|------|
+| **Cross-encoder** | Highest | Slow | High |
+| **ColBERT** | High | Medium | Medium |
+| **LLM-based** | High | Slow | High |
+| **Cohere Rerank** | High | Fast | API cost |
+
+### Simple Reranker Implementation
+
+```python
+def rerank(query: str, chunks: List[Chunk], top_k: int = 10) -> List[Chunk]:
+    """Rerank chunks using cross-encoder or LLM."""
+    
+    scored_chunks = []
+    for chunk in chunks:
+        # Use LLM to score relevance
+        score = llm.score_relevance(query, chunk.content)
+        scored_chunks.append((chunk, score))
+    
+    # Sort by score and return top k
+    scored_chunks.sort(key=lambda x: x[1], reverse=True)
+    return [c for c, s in scored_chunks[:top_k]]
+```
+
+---
+
+## 5. Vector Database Selection
+
+### For Agent Kit (Local/Lightweight)
+
+| Database | Best For | Setup |
+|----------|----------|-------|
+| **ChromaDB** | Local development | `pip install chromadb` |
+| **SQLite + sqlite-vec** | Embedded apps | Single file |
+| **LanceDB** | Large local datasets | `pip install lancedb` |
+
+### For Production
+
+| Database | Best For | Notes |
+|----------|----------|-------|
+| **Pinecone** | Managed, scalable | Free tier available |
+| **Weaviate** | Self-hosted + managed | GraphQL API |
+| **Qdrant** | High performance | Rust-based |
+| **pgvector** | PostgreSQL users | Familiar stack |
+
+---
+
+## 6. Embedding Models
+
+### For Code (Recommended)
+
+| Model | Dimensions | Best For |
+|-------|------------|----------|
+| **Voyage Code 2** | 1536 | Code-specific |
+| **text-embedding-3-large** | 3072 | General + Code |
+| **Gemini Text Embedding** | 768 | Google ecosystem |
+| **CodeBERT** | 768 | Open source |
+
+### Embedding Best Practices
+
+1. **Normalize embeddings** for cosine similarity
+2. **Batch processing** for efficiency
+3. **Cache embeddings** - don't re-embed unchanged files
+4. **Dimension reduction** if storage is concern (PCA)
+
+---
+
+## 7. Integration with Agent Kit
+
+### Workflow
+
+1. **On Project Init (`/create`):**
+   - Run chunking script
+   - Generate embeddings
+   - Store in local vector DB
+
+2. **On Code Change (Pre-commit hook):**
+   - Detect changed files
+   - Re-chunk and re-embed only changes
+   - Update vector DB
+
+3. **On AI Query:**
+   - Hybrid search for relevant chunks
+   - Rerank results
+   - Inject top chunks into context
+
+### File Structure
+
+```
+project-root/
+├── .agent/
+│   ├── AGENTS.md          # Layer 1: Map
+│   ├── graph.json         # Layer 2: Knowledge Graph
+│   └── rag/               # Layer 3: RAG Data
+│       ├── chunks.json    # Chunked code
+│       ├── embeddings.npy # Vector embeddings
+│       └── index.db       # Vector index (ChromaDB/SQLite)
+```
+
+---
+
+## 📋 Implementation Checklist
+
+### Phase 1: Basic RAG
+- [ ] Implement logical chunking (by function/class)
+- [ ] Choose embedding model
+- [ ] Set up local vector DB (ChromaDB)
+- [ ] Basic semantic search
+
+### Phase 2: Enhanced RAG
+- [ ] Add contextual summaries to chunks
+- [ ] Implement BM25 for hybrid search
+- [ ] Add RRF fusion
+- [ ] Implement basic reranking
+
+### Phase 3: Production RAG
+- [ ] Incremental indexing (only changed files)
+- [ ] Caching layer
+- [ ] Evaluation metrics (hit rate, MRR)
+- [ ] Monitoring and logging
+
+---
+
+## ⚠️ Common Pitfalls
+
+| Pitfall | Solution |
+|---------|----------|
+| Chunking breaks code | Use AST-based chunking |
+| Retrieving wrong file | Add file path to chunk text |
+| Missing exact matches | Add BM25/keyword search |
+| Too many irrelevant results | Implement reranking |
+| Slow embedding | Batch + cache |
+| Stale index | Automate re-indexing |
+
+---
+
+## 🔗 Related Skills
+
+- `graph-mapper` - Knowledge Graph for dependencies
+- `database-design` - Vector DB selection
+- `mcp-builder` - Build MCP server for RAG queries
+
+---
+
+> **Remember:** RAG is not magic. It's engineering. Measure, iterate, improve.

package/.agent/skills/rag-engineering/chunking-strategies.md

@@ -0,0 +1,229 @@
+# Chunking Strategies for Code
+
+> Break code into meaningful, complete units - not random fragments.
+
+---
+
+## Golden Rule
+
+> **A chunk should be a complete, understandable unit of code.**
+
+If you read a chunk in isolation, you should understand what it does.
+
+---
+
+## Strategy 1: Function-Level Chunking
+
+Best for: Most codebases
+
+```typescript
+// ✅ One chunk = One function
+function calculateTax(amount: number): number {
+  const rate = 0.1;
+  return amount * rate;
+}
+```
+
+### Implementation
+
+```python
+import re
+
+def chunk_by_functions_ts(code: str) -> list[dict]:
+    """Chunk TypeScript/JavaScript by functions."""
+    
+    # Pattern for function declarations
+    pattern = r'''
+        (?:export\s+)?                    # Optional export
+        (?:async\s+)?                     # Optional async
+        (?:function\s+(\w+)|              # function name() or
+           (?:const|let|var)\s+(\w+)\s*=\s*(?:async\s+)?(?:\([^)]*\)|[^=])\s*=>)  # arrow function
+        [^{]*\{                           # Opening brace
+    '''
+    
+    chunks = []
+    # Use AST parser for robust extraction (recommended)
+    # This regex is simplified example
+    
+    return chunks
+```
+
+---
+
+## Strategy 2: Class-Level Chunking
+
+Best for: OOP-heavy codebases
+
+```typescript
+// ✅ One chunk = One class (with all methods)
+class UserService {
+  constructor(private db: Database) {}
+  
+  async getUser(id: string): Promise<User> {
+    return this.db.findById(id);
+  }
+  
+  async createUser(data: UserData): Promise<User> {
+    return this.db.create(data);
+  }
+}
+```
+
+### When to Split Classes
+
+If class > 500 lines:
+- Chunk class definition + constructor
+- Chunk each method separately
+- Link via metadata
+
+---
+
+## Strategy 3: Module-Level Chunking
+
+Best for: Utility files, configs
+
+```typescript
+// ✅ One chunk = Entire file (if small)
+// constants.ts
+export const API_URL = 'https://api.example.com';
+export const MAX_RETRIES = 3;
+export const TIMEOUT_MS = 5000;
+```
+
+### Size Limits
+
+| File Size | Strategy |
+|-----------|----------|
+| < 100 lines | Entire file as one chunk |
+| 100-500 lines | Split by function/class |
+| > 500 lines | Split by function + subdivide large functions |
+
+---
+
+## Strategy 4: Semantic Chunking
+
+Best for: Mixed content (code + docs)
+
+Split by semantic meaning rather than syntax:
+
+1. **Imports section** → One chunk
+2. **Type definitions** → One chunk
+3. **Main logic** → Multiple chunks
+4. **Exports** → One chunk
+
+---
+
+## Chunk Overlap
+
+### Why Overlap?
+
+Context at boundaries gets lost without overlap.
+
+### Recommended Overlap
+
+```python
+CHUNK_SIZE = 1500  # tokens
+OVERLAP = 200      # tokens
+
+# Result: Each chunk shares 200 tokens with neighbors
+```
+
+### Implementation
+
+```python
+def chunk_with_overlap(text: str, chunk_size: int, overlap: int) -> list[str]:
+    tokens = tokenize(text)
+    chunks = []
+    
+    start = 0
+    while start < len(tokens):
+        end = min(start + chunk_size, len(tokens))
+        chunks.append(detokenize(tokens[start:end]))
+        start += chunk_size - overlap
+    
+    return chunks
+```
+
+---
+
+## Metadata: The Critical Addition
+
+Every chunk MUST have metadata:
+
+```json
+{
+  "id": "chunk_abc123",
+  "content": "function calculateTax(amount) { ... }",
+  "metadata": {
+    "file_path": "src/utils/tax.ts",
+    "file_type": "typescript",
+    "chunk_type": "function",
+    "name": "calculateTax",
+    "start_line": 15,
+    "end_line": 20,
+    "tokens": 45,
+    "hash": "sha256:...",
+    "dependencies": ["TAX_RATE"],
+    "dependents": ["ShippingService", "InvoiceGenerator"],
+    "last_modified": "2025-01-24T10:00:00Z"
+  }
+}
+```
+
+---
+
+## Language-Specific Parsers
+
+### TypeScript/JavaScript
+
+```bash
+# Recommended: tree-sitter
+npm install tree-sitter tree-sitter-typescript
+
+# Alternative: @babel/parser
+npm install @babel/parser
+```
+
+### Python
+
+```python
+import ast
+
+def extract_functions(code: str) -> list[dict]:
+    tree = ast.parse(code)
+    functions = []
+    
+    for node in ast.walk(tree):
+        if isinstance(node, ast.FunctionDef):
+            functions.append({
+                'name': node.name,
+                'start_line': node.lineno,
+                'end_line': node.end_lineno,
+                'code': ast.get_source_segment(code, node)
+            })
+    
+    return functions
+```
+
+### Go
+
+```go
+// Use go/parser package
+import (
+    "go/parser"
+    "go/token"
+)
+```
+
+---
+
+## Quality Checklist
+
+Before chunking:
+
+- [ ] Chunk size is appropriate (500-1500 tokens)
+- [ ] Overlap is configured (10-20%)
+- [ ] Metadata is captured
+- [ ] Complete units (no broken functions)
+- [ ] Imports are preserved or referenced
+- [ ] Comments are included with code