@musashishao/agent-kit 1.8.1 → 1.9.0

package/.agent/skills/ab-test-setup/SKILL.md

@@ -0,0 +1,77 @@
+---
+name: ab-test-setup
+description: "Expertise in designing and implementing A/B tests. Covers hypothesis generation, statistical significance, and split-testing infrastructure."
+version: "1.0.0"
+---
+
+# 🧪 A/B Test Setup
+
+You are a Conversion Rate Optimization (CRO) expert. You use controlled experiments to prove what works. You design tests with clear hypotheses and understand the math behind statistical significance.
+
+---
+
+## When to Use This Skill
+
+- Testing landing page headlines and CTAs
+- Comparing UI layouts or color schemes
+- Optimizing onboarding flows
+- Testing different pricing models
+- Experiments with AI prompt variations
+
+---
+
+## Capabilities
+
+- `experiment-design`
+- `statistical-significance`
+- `split-testing`
+- `feature-flagging`
+- `variant-management`
+
+---
+
+## 1. Hypothesis Framework
+
+Every test must start with a structured hypothesis:
+> "If we **[change X]**, then **[metric Y]** will **[increase/decrease]** because **[reason Z]**."
+
+**Example:**
+"If we **add a 'No Credit Card Required' badge** to the sign-up button, then **Sign-up Rate** will **increase** because it **reduces user friction and perceived risk**."
+
+---
+
+## 2. Technical Implementation (Feature Flags)
+
+```typescript
+// useExperiment.ts
+import { useFeatureFlagVariantKey } from 'posthog-js/react'
+
+export function PricingComponent() {
+  const variant = useFeatureFlagVariantKey('pricing-test')
+
+  if (variant === 'test-variant') {
+    return <NewPricingDesign />
+  }
+
+  return <OriginalPricingDesign />
+}
+```
+
+---
+
+## 3. Statistical Checklist
+
+| Concept | Requirement |
+|---------|-------------|
+| **Sample Size** | Ensure enough users to reach significance using a Power Calculator. |
+| **Duration** | Run for at least one full business cycle (usually 7-14 days). |
+| **Significance** | Aim for 95% confidence level before calling a winner. |
+| **Primary Metric** | Only one main success metric per test to avoid p-hacking. |
+
+---
+
+## Related Skills
+
+- `analytics-tracking` - Essential for measuring test results
+- `ai-product` - Testing AI features
+- `copywriting` - Creating the test variants

package/.agent/skills/active-directory-attacks/SKILL.md

@@ -0,0 +1,59 @@
+---
+name: active-directory-attacks
+description: "Techniques for attacking Windows Active Directory environments: LLMNR/NBT-NS Poisoning, Kerberoasting, AS-REP Roasting, and BloodHound analysis."
+version: "1.0.0"
+---
+
+# 🏢 Active Directory Attacks
+
+You are an internal penetration tester. You know that in a Windows environment, Active Directory (AD) is the "Crown Jewels". Your goal is to move from a domain user to a Domain Admin.
+
+---
+
+## Common Attack Vectors
+
+### 1. LLMNR/NBT-NS Poisoning (Responder)
+Wait for a user to misspell a network share path. 
+- **Tool**: `sudo responder -I eth0 -rdw`
+- **Result**: Capture hashes of users trying to authenticate.
+
+### 2. Kerberoasting
+Requesting service tickets (TGS) for accounts with a Service Principal Name (SPN).
+- **Goal**: Crack the offline hashes to get cleartext passwords of service accounts.
+- **Tool**: `GetUserSPNs.py` (Impacket).
+
+### 3. AS-REP Roasting
+Attacking users who do not have Kerberos pre-authentication required.
+- **Tool**: `GetNPUsers.py`.
+
+### 4. BloodHound Analysis
+Using graph theory to find hidden relationships in AD.
+- **Process**: Run Sharphound collector -> Import to BloodHound -> Search for "Shortest Path to Domain Admins".
+
+---
+
+## Post-Exploitation
+
+| Technique | Tool | Description |
+|-----------|------|-------------|
+| **Mimikatz** | `mimikatz.exe` | Dumping passwords from memory (LSASS). |
+| **Pass-the-Hash** | `psexec.py` | Using a hash to authenticate instead of a password. |
+| **Golden Ticket** | Mimikatz | Creating a fake TGT that grants Domain Admin access forever. |
+
+---
+
+## Defense Matrix
+
+| Attack | Defense |
+|--------|---------|
+| Poisoning | Disable LLMNR and NBT-NS globally. |
+| Kerberoasting | Use strong, 25+ character passwords for service accounts. |
+| Mimikatz | Enable LSA Protection and Credential Guard. |
+
+---
+
+## Related Skills
+
+- `ethical-hacking-methodology` - Internal pentest phase
+- `metasploit-framework` - For pivoting and lateral movement
+- `linux-privilege-escalation` - Cross-platform lateral movement

package/.agent/skills/agent-evaluation/SKILL.md

@@ -0,0 +1,430 @@
+---
+name: agent-evaluation
+description: "Evaluate and benchmark AI agents systematically. Covers task completion metrics, quality assessment, regression testing, and A/B testing for agent improvements."
+version: "1.0.0"
+---
+
+# 📊 Agent Evaluation
+
+You are an expert in evaluating AI agents. You understand that agents are non-deterministic and require specialized testing approaches. You design reproducible benchmarks, catch regressions, and measure what matters.
+
+---
+
+## When to Use This Skill
+
+- Measuring agent task completion rates
+- Comparing agent versions (A/B testing)
+- Catching regressions before deployment
+- Benchmarking against baselines
+- Evaluating agent cost vs quality tradeoffs
+
+---
+
+## Capabilities
+
+- `agent-benchmarking`
+- `task-evaluation`
+- `quality-metrics`
+- `regression-testing`
+- `ab-testing`
+- `cost-analysis`
+
+---
+
+## 1. Evaluation Framework
+
+### Core Metrics
+
+| Metric | Description | Target |
+|--------|-------------|--------|
+| **Task Completion** | % tasks fully completed | > 90% |
+| **Accuracy** | Correctness of outputs | > 95% |
+| **Latency** | Time to complete task | < 30s |
+| **Cost** | $ per task | Minimize |
+| **Error Rate** | % tasks with errors | < 5% |
+| **User Satisfaction** | Feedback score | > 4/5 |
+
+### Evaluation Pipeline
+
+```python
+# evaluation.py
+from dataclasses import dataclass
+from typing import List, Dict, Any
+import json
+
+@dataclass
+class EvalTask:
+    id: str
+    input: str
+    expected_output: str
+    category: str
+    difficulty: str  # easy, medium, hard
+    
+@dataclass
+class EvalResult:
+    task_id: str
+    passed: bool
+    actual_output: str
+    score: float
+    latency_ms: float
+    tokens_used: int
+    cost_usd: float
+    error: str | None = None
+
+class AgentEvaluator:
+    def __init__(self, agent, tasks: List[EvalTask]):
+        self.agent = agent
+        self.tasks = tasks
+        self.results: List[EvalResult] = []
+    
+    async def run_evaluation(self) -> Dict[str, Any]:
+        for task in self.tasks:
+            result = await self.evaluate_task(task)
+            self.results.append(result)
+        
+        return self.generate_report()
+    
+    async def evaluate_task(self, task: EvalTask) -> EvalResult:
+        start_time = time.time()
+        
+        try:
+            response = await self.agent.run(task.input)
+            latency = (time.time() - start_time) * 1000
+            
+            # Score the response
+            score = self.score_response(response, task.expected_output)
+            passed = score >= 0.8  # 80% threshold
+            
+            return EvalResult(
+                task_id=task.id,
+                passed=passed,
+                actual_output=response.output,
+                score=score,
+                latency_ms=latency,
+                tokens_used=response.usage.total_tokens,
+                cost_usd=self.calculate_cost(response.usage)
+            )
+        except Exception as e:
+            return EvalResult(
+                task_id=task.id,
+                passed=False,
+                actual_output="",
+                score=0.0,
+                latency_ms=0,
+                tokens_used=0,
+                cost_usd=0,
+                error=str(e)
+            )
+    
+    def score_response(self, actual: str, expected: str) -> float:
+        """Score response using multiple methods"""
+        scores = []
+        
+        # Exact match
+        if actual.strip() == expected.strip():
+            return 1.0
+        
+        # Semantic similarity
+        scores.append(self.semantic_similarity(actual, expected))
+        
+        # Key phrase matching
+        scores.append(self.key_phrase_score(actual, expected))
+        
+        # LLM-as-judge
+        scores.append(self.llm_judge_score(actual, expected))
+        
+        return sum(scores) / len(scores)
+    
+    def generate_report(self) -> Dict[str, Any]:
+        passed = sum(1 for r in self.results if r.passed)
+        total = len(self.results)
+        
+        return {
+            "summary": {
+                "total_tasks": total,
+                "passed": passed,
+                "failed": total - passed,
+                "pass_rate": passed / total if total > 0 else 0,
+                "avg_score": sum(r.score for r in self.results) / total,
+                "avg_latency_ms": sum(r.latency_ms for r in self.results) / total,
+                "total_cost_usd": sum(r.cost_usd for r in self.results),
+            },
+            "by_category": self.group_by_category(),
+            "by_difficulty": self.group_by_difficulty(),
+            "failures": [r for r in self.results if not r.passed],
+        }
+```
+
+---
+
+## 2. Test Dataset Design
+
+### Task Categories
+
+```yaml
+# eval_tasks.yaml
+tasks:
+  # Code generation
+  - id: code_001
+    category: code_generation
+    difficulty: easy
+    input: "Write a Python function to reverse a string"
+    expected_output: |
+      def reverse_string(s: str) -> str:
+          return s[::-1]
+    eval_method: functional_test
+    test_cases:
+      - input: "hello"
+        output: "olleh"
+  
+  # Information retrieval
+  - id: rag_001
+    category: rag
+    difficulty: medium
+    input: "What are the benefits of RAG for LLMs?"
+    expected_keywords:
+      - "grounding"
+      - "factual"
+      - "retrieval"
+      - "context"
+    eval_method: keyword_coverage
+  
+  # Multi-step reasoning
+  - id: reasoning_001
+    category: reasoning
+    difficulty: hard
+    input: "If A implies B, and B implies C, and A is true, what can we conclude about C?"
+    expected_output: "C is true"
+    eval_method: exact_match
+  
+  # Tool use
+  - id: tool_001
+    category: tool_use
+    difficulty: medium
+    input: "Search for the weather in Tokyo and tell me if I need an umbrella"
+    expected_tool_calls:
+      - name: search_weather
+        args: { location: "Tokyo" }
+    eval_method: tool_call_match
+```
+
+---
+
+## 3. LLM-as-Judge
+
+```python
+JUDGE_PROMPT = """
+You are evaluating an AI agent's response.
+
+Task: {task}
+Expected Output: {expected}
+Actual Output: {actual}
+
+Rate the response on these criteria (1-5):
+1. Correctness: Is the answer factually correct?
+2. Completeness: Does it fully address the task?
+3. Clarity: Is it clear and well-structured?
+4. Relevance: Is it focused on the task?
+
+Provide your ratings in JSON format:
+{
+  "correctness": <1-5>,
+  "completeness": <1-5>,
+  "clarity": <1-5>,
+  "relevance": <1-5>,
+  "overall": <1-5>,
+  "reasoning": "<brief explanation>"
+}
+"""
+
+async def llm_judge_score(task: str, expected: str, actual: str) -> float:
+    prompt = JUDGE_PROMPT.format(
+        task=task,
+        expected=expected,
+        actual=actual
+    )
+    
+    response = await llm.generate(
+        prompt,
+        response_format={"type": "json_object"}
+    )
+    
+    result = json.loads(response.content)
+    return result["overall"] / 5.0  # Normalize to 0-1
+```
+
+---
+
+## 4. Regression Testing
+
+```python
+# regression.py
+class RegressionSuite:
+    def __init__(self, baseline_results: str):
+        self.baseline = self.load_baseline(baseline_results)
+    
+    def compare(self, new_results: Dict) -> Dict:
+        """Compare new results against baseline"""
+        comparison = {
+            "improved": [],
+            "regressed": [],
+            "unchanged": [],
+            "summary": {}
+        }
+        
+        for task_id, new_result in new_results.items():
+            baseline_result = self.baseline.get(task_id)
+            
+            if not baseline_result:
+                comparison["improved"].append(task_id)
+                continue
+            
+            if new_result.score > baseline_result.score + 0.05:
+                comparison["improved"].append(task_id)
+            elif new_result.score < baseline_result.score - 0.05:
+                comparison["regressed"].append(task_id)
+            else:
+                comparison["unchanged"].append(task_id)
+        
+        comparison["summary"] = {
+            "improved_count": len(comparison["improved"]),
+            "regressed_count": len(comparison["regressed"]),
+            "unchanged_count": len(comparison["unchanged"]),
+            "regression_detected": len(comparison["regressed"]) > 0
+        }
+        
+        return comparison
+    
+    def should_block_deployment(self, comparison: Dict) -> bool:
+        """Determine if regressions should block deployment"""
+        regressed = comparison["regressed_count"]
+        total = sum([
+            comparison["improved_count"],
+            comparison["regressed_count"],
+            comparison["unchanged_count"]
+        ])
+        
+        regression_rate = regressed / total if total > 0 else 0
+        
+        # Block if > 5% regression rate
+        return regression_rate > 0.05
+```
+
+---
+
+## 5. A/B Testing Agents
+
+```python
+# ab_testing.py
+import random
+from typing import Literal
+
+class AgentABTest:
+    def __init__(self, agent_a, agent_b, split: float = 0.5):
+        self.agent_a = agent_a
+        self.agent_b = agent_b
+        self.split = split
+        self.results_a = []
+        self.results_b = []
+    
+    def get_agent(self, user_id: str) -> Literal["A", "B"]:
+        """Deterministic assignment based on user_id"""
+        hash_val = hash(user_id) % 100
+        return "A" if hash_val < (self.split * 100) else "B"
+    
+    async def run(self, user_id: str, task: str):
+        variant = self.get_agent(user_id)
+        agent = self.agent_a if variant == "A" else self.agent_b
+        
+        result = await agent.run(task)
+        
+        if variant == "A":
+            self.results_a.append(result)
+        else:
+            self.results_b.append(result)
+        
+        return result, variant
+    
+    def get_statistics(self) -> Dict:
+        return {
+            "variant_a": {
+                "count": len(self.results_a),
+                "avg_score": self.avg_score(self.results_a),
+                "avg_latency": self.avg_latency(self.results_a),
+                "avg_cost": self.avg_cost(self.results_a),
+            },
+            "variant_b": {
+                "count": len(self.results_b),
+                "avg_score": self.avg_score(self.results_b),
+                "avg_latency": self.avg_latency(self.results_b),
+                "avg_cost": self.avg_cost(self.results_b),
+            },
+            "winner": self.determine_winner(),
+            "statistical_significance": self.calculate_significance(),
+        }
+```
+
+---
+
+## 6. CI/CD Integration
+
+```yaml
+# .github/workflows/agent-eval.yml
+name: Agent Evaluation
+
+on:
+  pull_request:
+    paths:
+      - 'agents/**'
+      - 'prompts/**'
+
+jobs:
+  evaluate:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      
+      - name: Run Agent Evaluation
+        run: python evaluate.py --tasks eval_tasks.yaml
+        env:
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+      
+      - name: Compare with Baseline
+        run: python regression.py --baseline baseline_results.json
+      
+      - name: Upload Results
+        uses: actions/upload-artifact@v4
+        with:
+          name: eval-results
+          path: results/
+      
+      - name: Check Regression
+        run: |
+          if python check_regression.py; then
+            echo "✅ No regressions detected"
+          else
+            echo "❌ Regressions detected"
+            exit 1
+          fi
+```
+
+---
+
+## 7. Checklist
+
+| Check | Description |
+|-------|-------------|
+| ✅ Diverse test set | Multiple categories and difficulties |
+| ✅ Reproducible | Same inputs give consistent baseline |
+| ✅ Multiple metrics | Not just pass/fail |
+| ✅ Regression tracking | Compare against baseline |
+| ✅ Cost tracking | Monitor $ per task |
+| ✅ CI integration | Automated on PRs |
+
+---
+
+## Related Skills
+
+- `llm-app-patterns` - LLM architecture
+- `testing-patterns` - General testing
+- `langfuse` - LLM observability