omgkit 2.5.1 → 2.6.0

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "omgkit",
-  "version": "2.5.1",
-  "description": "Omega-Level Development Kit - AI Team System for Claude Code. 23 agents, 54 commands, 72 skills, sprint management.",
+  "version": "2.6.0",
+  "description": "Omega-Level Development Kit - AI Team System for Claude Code. 23 agents, 58 commands, 88 skills, sprint management.",
   "keywords": [
     "claude-code",
     "ai",

package/plugin/skills/ai-engineering/SKILL.md

@@ -0,0 +1,65 @@
+---
+name: ai-engineering
+description: Building production AI applications with Foundation Models. Covers prompt engineering, RAG, agents, finetuning, evaluation, and deployment. Use when working with LLMs, building AI features, or architecting AI systems.
+---
+
+# AI Engineering Skills
+
+Comprehensive skills for building AI applications with Foundation Models.
+
+## AI Engineering Stack
+
+```
+┌─────────────────────────────────────────────────────┐
+│  APPLICATION LAYER                                   │
+│  Prompt Engineering, RAG, Agents, Guardrails        │
+├─────────────────────────────────────────────────────┤
+│  MODEL LAYER                                         │
+│  Model Selection, Finetuning, Evaluation            │
+├─────────────────────────────────────────────────────┤
+│  INFRASTRUCTURE LAYER                                │
+│  Inference Optimization, Caching, Orchestration     │
+└─────────────────────────────────────────────────────┘
+```
+
+## 12 Core Skills
+
+| Skill | Description | Guide |
+|-------|-------------|-------|
+| Foundation Models | Model architecture, sampling, structured outputs | [foundation-models/](foundation-models/SKILL.md) |
+| Evaluation Methodology | Metrics, AI-as-judge, comparative evaluation | [evaluation-methodology/](evaluation-methodology/SKILL.md) |
+| AI System Evaluation | End-to-end evaluation, benchmarks, model selection | [ai-system-evaluation/](ai-system-evaluation/SKILL.md) |
+| Prompt Engineering | System prompts, few-shot, chain-of-thought, defense | [prompt-engineering/](prompt-engineering/SKILL.md) |
+| RAG Systems | Chunking, embedding, retrieval, reranking | [rag-systems/](rag-systems/SKILL.md) |
+| AI Agents | Tool use, planning strategies, memory systems | [ai-agents/](ai-agents/SKILL.md) |
+| Finetuning | LoRA, QLoRA, PEFT, model merging | [finetuning/](finetuning/SKILL.md) |
+| Dataset Engineering | Data quality, curation, synthesis, annotation | [dataset-engineering/](dataset-engineering/SKILL.md) |
+| Inference Optimization | Quantization, batching, caching, speculative decoding | [inference-optimization/](inference-optimization/SKILL.md) |
+| AI Architecture | Gateway, routing, observability, deployment | [ai-architecture/](ai-architecture/SKILL.md) |
+| Guardrails & Safety | Input/output guards, PII protection, injection defense | [guardrails-safety/](guardrails-safety/SKILL.md) |
+| User Feedback | Explicit/implicit signals, feedback loops, A/B testing | [user-feedback/](user-feedback/SKILL.md) |
+
+## Development Process
+
+```
+1. Use Case Evaluation → 2. Model Selection → 3. Evaluation Pipeline
+                                                      ↓
+4. Prompt Engineering → 5. Context (RAG/Agents) → 6. Finetuning (if needed)
+                                                      ↓
+7. Inference Optimization → 8. Deployment → 9. Monitoring & Feedback
+```
+
+## Quick Decision Guide
+
+| Need | Start With |
+|------|------------|
+| Improve output quality | prompt-engineering |
+| Add external knowledge | rag-systems |
+| Multi-step reasoning | ai-agents |
+| Reduce latency/cost | inference-optimization |
+| Measure quality | evaluation-methodology |
+| Protect system | guardrails-safety |
+
+## Reference
+
+Based on "AI Engineering" by Chip Huyen (O'Reilly, 2025).

package/plugin/skills/ai-engineering/ai-agents/SKILL.md

@@ -0,0 +1,157 @@
+---
+name: ai-agents
+description: Building AI agents - tool use, planning strategies (ReAct, Plan-and-Execute), memory systems, agent evaluation. Use when building autonomous AI systems, tool-augmented apps, or multi-step workflows.
+---
+
+# AI Agents
+
+Building AI agents with tools and planning.
+
+## Agent Architecture
+
+```
+┌─────────────────────────────────────┐
+│            AI AGENT                  │
+├─────────────────────────────────────┤
+│  ┌──────────┐                       │
+│  │  BRAIN   │  (Foundation Model)   │
+│  │ Planning │                       │
+│  │ Reasoning│                       │
+│  └────┬─────┘                       │
+│       │                             │
+│   ┌───┴───┐                         │
+│   ↓       ↓                         │
+│ ┌─────┐ ┌──────┐                    │
+│ │TOOLS│ │MEMORY│                    │
+│ └─────┘ └──────┘                    │
+└─────────────────────────────────────┘
+```
+
+## Tool Definition
+
+```python
+tools = [{
+    "type": "function",
+    "function": {
+        "name": "search_database",
+        "description": "Search products by query",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "query": {"type": "string"},
+                "category": {"type": "string", "enum": ["electronics", "clothing"]},
+                "max_price": {"type": "number"}
+            },
+            "required": ["query"]
+        }
+    }
+}]
+```
+
+## Planning Strategies
+
+### ReAct (Reasoning + Acting)
+```python
+REACT_PROMPT = """Tools: {tools}
+
+Format:
+Thought: [reasoning]
+Action: [tool_name]
+Action Input: [JSON]
+Observation: [result]
+... repeat ...
+Final Answer: [answer]
+
+Question: {question}
+Thought:"""
+
+def react_agent(question, tools, max_steps=10):
+    prompt = REACT_PROMPT.format(...)
+
+    for _ in range(max_steps):
+        response = llm.generate(prompt)
+
+        if "Final Answer:" in response:
+            return extract_answer(response)
+
+        action, input = parse_action(response)
+        observation = execute(tools[action], input)
+        prompt += f"\nObservation: {observation}\nThought:"
+```
+
+### Plan-and-Execute
+```python
+def plan_and_execute(task, tools):
+    # Step 1: Create plan
+    plan = llm.generate(f"Create step-by-step plan for: {task}")
+    steps = parse_plan(plan)
+
+    # Step 2: Execute each step
+    results = []
+    for step in steps:
+        result = execute_step(step, tools)
+        results.append(result)
+
+    # Step 3: Synthesize
+    return synthesize(task, results)
+```
+
+### Reflexion (Self-Reflection)
+```python
+def reflexion_agent(task, max_attempts=3):
+    memory = []
+
+    for attempt in range(max_attempts):
+        solution = generate(task, memory)
+        success, feedback = evaluate(solution)
+
+        if success:
+            return solution
+
+        reflection = reflect(task, solution, feedback)
+        memory.append({"solution": solution, "reflection": reflection})
+```
+
+## Memory Systems
+
+```python
+class AgentMemory:
+    def __init__(self):
+        self.short_term = []        # Recent turns
+        self.long_term = VectorDB() # Persistent
+
+    def add(self, message):
+        self.short_term.append(message)
+        if len(self.short_term) > 20:
+            self.consolidate()
+
+    def consolidate(self):
+        summary = summarize(self.short_term[:10])
+        self.long_term.add(summary)
+        self.short_term = self.short_term[10:]
+
+    def retrieve(self, query, k=5):
+        return {
+            "recent": self.short_term[-5:],
+            "relevant": self.long_term.search(query, k),
+        }
+```
+
+## Agent Evaluation
+
+```python
+def evaluate_agent(agent, test_cases):
+    return {
+        "task_success": mean([agent.run(c["task"]) == c["expected"] for c in test_cases]),
+        "avg_steps": mean([agent.step_count for _ in test_cases]),
+        "avg_latency": mean([measure_time(agent.run, c["task"]) for c in test_cases]),
+    }
+```
+
+## Best Practices
+
+1. Start with simple tools, add complexity gradually
+2. Add reflection for complex tasks
+3. Limit max steps to prevent infinite loops
+4. Log all agent actions for debugging
+5. Use evaluation to measure progress

package/plugin/skills/ai-engineering/ai-architecture/SKILL.md

@@ -0,0 +1,133 @@
+---
+name: ai-architecture
+description: AI application architecture - gateway, orchestration, model routing, observability, deployment patterns. Use when designing AI systems, scaling applications, or building production infrastructure.
+---
+
+# AI Architecture Skill
+
+Designing production AI applications.
+
+## Reference Architecture
+
+```
+┌─────────────────────────────────────────────────────┐
+│  CLIENT LAYER (Web, Mobile, API, CLI)               │
+└───────────────────────┬─────────────────────────────┘
+                        │
+┌───────────────────────┴─────────────────────────────┐
+│  GATEWAY LAYER                                       │
+│  Rate Limiter | Auth | Input Guard                  │
+└───────────────────────┬─────────────────────────────┘
+                        │
+┌───────────────────────┴─────────────────────────────┐
+│  ORCHESTRATION LAYER                                 │
+│  Router | Cache | Context | Agent | Output Guard    │
+└───────────────────────┬─────────────────────────────┘
+                        │
+┌───────────────────────┴─────────────────────────────┐
+│  MODEL LAYER                                         │
+│  Primary LLM | Fallback | Specialized               │
+└───────────────────────┬─────────────────────────────┘
+                        │
+┌───────────────────────┴─────────────────────────────┐
+│  DATA LAYER                                          │
+│  Vector DB | SQL DB | Cache                         │
+└─────────────────────────────────────────────────────┘
+```
+
+## Model Router
+
+```python
+class ModelRouter:
+    def __init__(self):
+        self.models = {
+            "gpt-4": {"cost": 0.03, "quality": 0.95, "latency": 2.0},
+            "gpt-3.5": {"cost": 0.002, "quality": 0.80, "latency": 0.5},
+        }
+        self.classifier = load_complexity_classifier()
+
+    def route(self, query, constraints):
+        complexity = self.classifier.predict(query)
+
+        if complexity == "simple" and constraints.get("cost_sensitive"):
+            return "gpt-3.5"
+        elif complexity == "complex":
+            return "gpt-4"
+        else:
+            return "gpt-3.5"
+
+    def with_fallback(self, query, primary, fallbacks):
+        for model in [primary] + fallbacks:
+            try:
+                response = self.call(model, query)
+                if self.validate(response):
+                    return response
+            except:
+                continue
+        raise Exception("All models failed")
+```
+
+## Context Enhancement
+
+```python
+class ContextEnhancer:
+    def enhance(self, query, history):
+        # Retrieve
+        docs = self.retriever.retrieve(query, k=10)
+
+        # Rerank
+        docs = self.rerank(query, docs)[:5]
+
+        # Compress if needed
+        context = self.format(docs)
+        if len(context) > 4000:
+            context = self.summarize(context)
+
+        # Add history
+        history_context = self.format_history(history[-5:])
+
+        return {
+            "retrieved": context,
+            "history": history_context
+        }
+```
+
+## Observability
+
+```python
+from opentelemetry import trace
+from prometheus_client import Counter, Histogram
+
+REQUESTS = Counter('ai_requests', 'Total', ['model', 'status'])
+LATENCY = Histogram('ai_latency', 'Latency', ['model'])
+TOKENS = Counter('ai_tokens', 'Tokens', ['model', 'type'])
+
+tracer = trace.get_tracer(__name__)
+
+class ObservableClient:
+    def generate(self, prompt, model):
+        with tracer.start_as_current_span("ai_generate") as span:
+            span.set_attribute("model", model)
+
+            start = time.time()
+            try:
+                response = self.client.generate(prompt, model)
+
+                REQUESTS.labels(model=model, status="ok").inc()
+                LATENCY.labels(model=model).observe(time.time()-start)
+                TOKENS.labels(model=model, type="in").inc(count(prompt))
+                TOKENS.labels(model=model, type="out").inc(count(response))
+
+                return response
+            except Exception as e:
+                REQUESTS.labels(model=model, status="error").inc()
+                raise
+```
+
+## Best Practices
+
+1. Add gateway for rate limiting/auth
+2. Use model router for cost optimization
+3. Implement fallback chains
+4. Add comprehensive observability
+5. Cache at multiple levels

package/plugin/skills/ai-engineering/ai-system-evaluation/SKILL.md

@@ -0,0 +1,95 @@
+---
+name: ai-system-evaluation
+description: End-to-end AI system evaluation - model selection, benchmarks, cost/latency analysis, build vs buy decisions. Use when selecting models, designing eval pipelines, or making architecture decisions.
+---
+
+# AI System Evaluation
+
+Evaluating AI systems end-to-end.
+
+## Evaluation Criteria
+
+### 1. Domain-Specific Capability
+
+| Domain | Benchmarks |
+|--------|------------|
+| Math & Reasoning | GSM-8K, MATH |
+| Code | HumanEval, MBPP |
+| Knowledge | MMLU, ARC |
+| Multi-turn Chat | MT-Bench |
+
+### 2. Generation Quality
+
+| Criterion | Measurement |
+|-----------|-------------|
+| Factual Consistency | NLI, SAFE, SelfCheckGPT |
+| Coherence | AI judge rubric |
+| Relevance | Semantic similarity |
+| Fluency | Perplexity |
+
+### 3. Cost & Latency
+
+```python
+@dataclass
+class PerformanceMetrics:
+    ttft: float      # Time to First Token (seconds)
+    tpot: float      # Time Per Output Token
+    throughput: float # Tokens/second
+
+    def cost(self, input_tokens, output_tokens, prices):
+        return input_tokens * prices["input"] + output_tokens * prices["output"]
+```
+
+## Model Selection Workflow
+
+```
+1. Define Requirements
+   ├── Task type
+   ├── Quality threshold
+   ├── Latency requirements (<2s TTFT)
+   ├── Cost budget
+   └── Deployment constraints
+
+2. Filter Options
+   ├── API vs Self-hosted
+   ├── Open source vs Proprietary
+   └── Size constraints
+
+3. Benchmark on Your Data
+   ├── Create eval dataset (100+ examples)
+   ├── Run experiments
+   └── Analyze results
+
+4. Make Decision
+   └── Balance quality, cost, latency
+```
+
+## Build vs Buy
+
+| Factor | API | Self-Host |
+|--------|-----|-----------|
+| Data Privacy | Less control | Full control |
+| Performance | Best models | Slightly behind |
+| Cost at Scale | Expensive | Amortized |
+| Customization | Limited | Full control |
+| Maintenance | Zero | Significant |
+
+## Public Benchmarks
+
+| Benchmark | Focus |
+|-----------|-------|
+| MMLU | Knowledge (57 subjects) |
+| HumanEval | Code generation |
+| GSM-8K | Math reasoning |
+| TruthfulQA | Factuality |
+| MT-Bench | Multi-turn chat |
+
+**Caution**: Benchmarks can be gamed. Data contamination is common. Always evaluate on YOUR data.
+
+## Best Practices
+
+1. Test on domain-specific data
+2. Measure both quality and cost
+3. Consider latency requirements
+4. Plan for fallback models
+5. Re-evaluate periodically

package/plugin/skills/ai-engineering/dataset-engineering/SKILL.md

@@ -0,0 +1,135 @@
+---
+name: dataset-engineering
+description: Building and processing datasets - data quality, curation, deduplication, synthesis, annotation, formatting. Use when creating training data, improving data quality, or generating synthetic data.
+---
+
+# Dataset Engineering Skill
+
+Building high-quality datasets for AI applications.
+
+## Data Quality Dimensions
+
+| Dimension | Description | Check |
+|-----------|-------------|-------|
+| Accuracy | Data is correct | Validation |
+| Completeness | No missing values | Schema check |
+| Consistency | No contradictions | Dedup |
+| Timeliness | Up-to-date | Timestamps |
+| Relevance | Matches use case | Filtering |
+
+## Data Curation Pipeline
+
+```python
+class DataCurationPipeline:
+    def run(self, raw_data):
+        # 1. Inspect
+        self.inspect(raw_data)
+
+        # 2. Deduplicate
+        data = self.deduplicator.dedupe(raw_data)
+
+        # 3. Clean and filter
+        data = self.cleaner.clean(data)
+        data = self.filter.filter(data)
+
+        # 4. Format
+        return self.formatter.format(data)
+```
+
+## Deduplication
+
+```python
+from datasketch import MinHash, MinHashLSH
+
+class Deduplicator:
+    def __init__(self, threshold=0.8):
+        self.lsh = MinHashLSH(threshold=threshold, num_perm=128)
+
+    def minhash(self, text):
+        m = MinHash(num_perm=128)
+        for word in text.split():
+            m.update(word.encode('utf8'))
+        return m
+
+    def dedupe(self, docs):
+        unique = []
+        for i, doc in enumerate(docs):
+            mh = self.minhash(doc["text"])
+            if not self.lsh.query(mh):
+                self.lsh.insert(f"doc_{i}", mh)
+                unique.append(doc)
+        return unique
+```
+
+## Data Synthesis
+
+### AI-Powered QA Generation
+```python
+def generate_qa(document, model, n=5):
+    prompt = f"""Generate {n} QA pairs from:
+
+{document}
+
+Format: [{{"question": "...", "answer": "..."}}]"""
+
+    return json.loads(model.generate(prompt))
+```
+
+### Self-Instruct
+```python
+def self_instruct(seeds, model, n=100):
+    generated = []
+
+    for _ in range(n):
+        samples = random.sample(seeds + generated[-20:], 5)
+        prompt = f"Examples:\n{format(samples)}\n\nNew task:"
+
+        new = model.generate(prompt)
+        if is_valid(new) and is_diverse(new, generated):
+            generated.append(new)
+
+    return generated
+```
+
+### Data Augmentation
+```python
+def augment_text(text):
+    methods = [
+        lambda t: synonym_replace(t),
+        lambda t: back_translate(t),
+        lambda t: model.rephrase(t)
+    ]
+    return random.choice(methods)(text)
+```
+
+## Data Formatting
+
+### Instruction Format
+```python
+def format_instruction(example):
+    return f"""### Instruction:
+{example['instruction']}
+
+### Input:
+{example.get('input', '')}
+
+### Response:
+{example['output']}"""
+```
+
+### Chat Format
+```python
+def format_chat(conversation):
+    return [
+        {"role": turn["role"], "content": turn["content"]}
+        for turn in conversation
+    ]
+```
+
+## Best Practices
+
+1. Inspect data before processing
+2. Deduplicate before expensive operations
+3. Use multiple synthesis methods
+4. Validate synthetic data quality
+5. Track data lineage

package/plugin/skills/ai-engineering/evaluation-methodology/SKILL.md

@@ -0,0 +1,93 @@
+---
+name: evaluation-methodology
+description: Methods for evaluating AI model outputs - exact match, semantic similarity, LLM-as-judge, comparative evaluation, ELO ranking. Use when measuring AI quality, building eval pipelines, or comparing models.
+---
+
+# Evaluation Methodology
+
+Methods for evaluating Foundation Model outputs.
+
+## Evaluation Approaches
+
+### 1. Exact Evaluation
+
+| Method | Use Case | Example |
+|--------|----------|---------|
+| Exact Match | QA, Math | `"5" == "5"` |
+| Functional Correctness | Code | Pass test cases |
+| BLEU/ROUGE | Translation | N-gram overlap |
+| Semantic Similarity | Open-ended | Embedding cosine |
+
+```python
+# Semantic Similarity
+from sentence_transformers import SentenceTransformer
+from sklearn.metrics.pairwise import cosine_similarity
+
+model = SentenceTransformer('all-MiniLM-L6-v2')
+emb1 = model.encode([generated])
+emb2 = model.encode([reference])
+similarity = cosine_similarity(emb1, emb2)[0][0]
+```
+
+### 2. AI as Judge
+
+```python
+JUDGE_PROMPT = """Rate the response on a scale of 1-5.
+
+Criteria:
+- Accuracy: Is information correct?
+- Helpfulness: Does it address the need?
+- Clarity: Is it easy to understand?
+
+Query: {query}
+Response: {response}
+
+Return JSON: {"score": N, "reasoning": "..."}"""
+
+# Multi-judge for reliability
+judges = ["gpt-4", "claude-3"]
+scores = [get_score(judge, response) for judge in judges]
+final_score = sum(scores) / len(scores)
+```
+
+### 3. Comparative Evaluation (ELO)
+
+```python
+COMPARE_PROMPT = """Compare these responses.
+
+Query: {query}
+A: {response_a}
+B: {response_b}
+
+Which is better? Return: A, B, or tie"""
+
+def update_elo(rating_a, rating_b, winner, k=32):
+    expected_a = 1 / (1 + 10**((rating_b - rating_a) / 400))
+    score_a = 1 if winner == "A" else 0 if winner == "B" else 0.5
+    return rating_a + k * (score_a - expected_a)
+```
+
+## Evaluation Pipeline
+
+```
+1. Define Criteria (accuracy, helpfulness, safety)
+   ↓
+2. Create Scoring Rubric with Examples
+   ↓
+3. Select Methods (exact + AI judge + human)
+   ↓
+4. Create Evaluation Dataset
+   ↓
+5. Run Evaluation
+   ↓
+6. Analyze & Iterate
+```
+
+## Best Practices
+
+1. Use multiple evaluation methods
+2. Calibrate AI judges with human data
+3. Include both automatic and human evaluation
+4. Version your evaluation datasets
+5. Track metrics over time
+6. Test for position bias in comparisons