@musashishao/agent-kit 1.6.1 → 1.7.0

package/.agent/skills/ai-incident-management/SKILL.md

@@ -0,0 +1,517 @@
+---
+name: ai-incident-management
+description: AI-specific incident response playbook, hallucination detection patterns, degradation vs outage classification, rollback strategies, post-mortem templates.
+allowed-tools: Read, Glob, Grep
+skills:
+  - systematic-debugging
+  - observability-patterns
+---
+
+# AI Incident Management
+
+> When AI breaks, respond fast and learn faster.
+
+---
+
+## 1. AI-Specific Incident Types
+
+### Taxonomy
+
+| Category | Examples | Severity |
+|----------|----------|----------|
+| **Hallucination** | Factually wrong, made-up info | Medium-Critical |
+| **Toxicity** | Harmful, offensive output | Critical |
+| **Data Leakage** | PII in responses, prompt leak | Critical |
+| **Performance** | High latency, timeouts | Medium |
+| **Availability** | Model API down | High-Critical |
+| **Drift** | Quality degradation over time | Medium |
+| **Prompt Injection** | Security bypass | Critical |
+
+### Severity Matrix
+
+```
+┌─────────────────┬──────────────────┬─────────────────────┐
+│                 │  Impact: Low     │  Impact: High       │
+├─────────────────┼──────────────────┼─────────────────────┤
+│ Frequency:      │  P3 - Monitor    │  P2 - Investigate   │
+│ Rare            │  Next sprint     │  Within 24h         │
+├─────────────────┼──────────────────┼─────────────────────┤
+│ Frequency:      │  P2 - Fix Soon   │  P1 - DROP ALL      │
+│ Common          │  Within 24h      │  Immediate          │
+└─────────────────┴──────────────────┴─────────────────────┘
+```
+
+---
+
+## 2. Incident Response Playbook
+
+### Phase 1: Detection (0-5 minutes)
+
+```markdown
+## Detection Checklist
+- [ ] Alert received and acknowledged
+- [ ] Initial severity assessed
+- [ ] On-call notified (if P1/P2)
+- [ ] Incident channel created
+- [ ] User impact estimated
+```
+
+### Phase 2: Triage (5-15 minutes)
+
+```markdown
+## Triage Questions
+1. What type of AI incident? (hallucination/toxicity/etc.)
+2. How widespread? (single user / all users / specific segment)
+3. Is it ongoing? (active / resolved / intermittent)
+4. What changed recently? (model / prompt / data)
+5. Can we reproduce it?
+
+## Quick Actions
+- [ ] Check model health dashboard
+- [ ] Review recent deployments
+- [ ] Check provider status page
+- [ ] Sample recent outputs for patterns
+```
+
+### Phase 3: Mitigation (15-60 minutes)
+
+| Strategy | When to Use | Impact |
+|----------|-------------|--------|
+| **Kill Switch** | Toxicity, data leak | Service degraded |
+| **Fallback Model** | Primary unavailable | Quality may differ |
+| **Circuit Breaker** | High error rate | Automatic recovery |
+| **Content Filter** | Specific bad patterns | Targeted fix |
+| **Rollback Prompt** | Prompt regression | Quick if versioned |
+
+### Phase 4: Resolution
+
+```markdown
+## Resolution Checklist
+- [ ] Root cause identified
+- [ ] Fix implemented and tested
+- [ ] Affected users notified (if needed)
+- [ ] Monitoring confirmed improvement
+- [ ] Incident timeline documented
+```
+
+---
+
+## 3. Hallucination Response
+
+### Detection Patterns
+
+```python
+HALLUCINATION_INDICATORS = [
+    # Over-confidence on uncertain topics
+    "definitely", "certainly", "always", "never",
+    
+    # Made-up citations
+    r"\d{4}\)", r"According to Dr\.",
+    
+    # Fictional data
+    r"\d+%", "studies show", "research indicates",
+    
+    # Self-referential confusion
+    "as an AI", "I don't have", "I cannot",
+]
+
+def assess_hallucination_risk(response: str) -> float:
+    """Score 0-1 for hallucination risk."""
+    score = 0.0
+    
+    # Check indicators
+    for pattern in HALLUCINATION_INDICATORS:
+        if re.search(pattern, response, re.IGNORECASE):
+            score += 0.1
+    
+    # Check for specific claims
+    if has_numeric_claims(response):
+        score += 0.2
+    
+    return min(score, 1.0)
+```
+
+### Response Flow
+
+```
+Hallucination Detected
+├── Severity Assessment
+│   ├── Factual error, low impact → Log, monitor
+│   ├── Medical/Legal/Financial → Immediate action
+│   └── User reported → Investigate
+├── Immediate Actions
+│   ├── Flag response for review
+│   ├── If critical: suppress similar queries
+│   └── If pattern: add to filter
+└── Long-term
+    ├── Add to evaluation dataset
+    ├── Consider prompt improvements
+    └── Update knowledge base
+```
+
+---
+
+## 4. Toxicity Response
+
+### Severity Levels
+
+| Level | Description | Response Time |
+|-------|-------------|---------------|
+| **L1** | Mildly inappropriate | Review next day |
+| **L2** | Offensive to groups | Fix within hours |
+| **L3** | Harmful instructions | Immediate block |
+| **L4** | Illegal content | Emergency + Legal |
+
+### Emergency Response
+
+```typescript
+async function handleToxicOutput(incident: ToxicityIncident): Promise<void> {
+  // 1. Immediate containment
+  await blockSimilarQueries(incident.queryPatterns);
+  
+  // 2. Notify
+  await notifyOnCall({
+    severity: incident.level,
+    description: incident.summary,  // Never include actual content
+    affectedUsers: incident.userCount,
+  });
+  
+  // 3. Evidence preservation
+  await preserveForReview({
+    incidentId: incident.id,
+    // Store hashed/redacted version
+    contentHash: hash(incident.content),
+    timestamp: Date.now(),
+  });
+  
+  // 4. If L3+, activate circuit breaker
+  if (incident.level >= 3) {
+    await circuitBreaker.open('chat', {
+      fallbackMessage: 'Service temporarily unavailable',
+      duration: 30 * 60 * 1000,  // 30 minutes
+    });
+  }
+}
+```
+
+---
+
+## 5. Rollback Strategies
+
+### What to Rollback
+
+| Component | Rollback Method | Time to Effect |
+|-----------|----------------|----------------|
+| **Prompt** | Version control | Seconds |
+| **Fine-tuned Model** | Model registry | Minutes |
+| **RAG Data** | Snapshot restore | Minutes |
+| **Base Model** | Provider switch | Varies |
+| **Feature Flag** | Kill switch | Seconds |
+
+### Prompt Versioning
+
+```typescript
+interface PromptVersion {
+  version: string;
+  content: string;
+  deployedAt: Date;
+  deployedBy: string;
+  rollbackTarget?: string;  // Previous version to use
+}
+
+class PromptManager {
+  async rollback(promptId: string): Promise<void> {
+    const current = await this.getCurrent(promptId);
+    const previous = await this.getVersion(promptId, current.rollbackTarget);
+    
+    // Deploy previous version
+    await this.deploy(promptId, previous.content);
+    
+    // Log rollback
+    await this.logRollback({
+      promptId,
+      fromVersion: current.version,
+      toVersion: previous.version,
+      reason: 'incident_rollback',
+      timestamp: new Date(),
+    });
+  }
+}
+```
+
+### Model Fallback Chain
+
+```yaml
+# config/ai-fallback.yaml
+fallback_chain:
+  primary:
+    provider: openai
+    model: gpt-4
+    timeout_ms: 30000
+    
+  secondary:
+    provider: anthropic
+    model: claude-3-sonnet
+    timeout_ms: 30000
+    trigger:
+      - primary_unavailable
+      - error_rate > 0.1
+      
+  tertiary:
+    provider: internal
+    model: fallback-model
+    timeout_ms: 10000
+    trigger:
+      - secondary_unavailable
+      - degraded_mode
+      
+  circuit_breaker:
+    type: static_response
+    message: "Service temporarily unavailable. Please try again later."
+    trigger:
+      - all_failed
+      - critical_incident
+```
+
+---
+
+## 6. Post-Mortem Template
+
+```markdown
+# AI Incident Post-Mortem: [Title]
+
+**Date:** [YYYY-MM-DD]
+**Duration:** [Start time] - [End time] ([Duration])
+**Severity:** P[1-4]
+**Incident Commander:** [Name]
+
+## Summary
+[2-3 sentence description of what happened]
+
+## Impact
+- **Users Affected:** [Number/Percentage]
+- **Requests Impacted:** [Count]
+- **Business Impact:** [Revenue/Reputation/Legal]
+
+## Timeline
+| Time | Event |
+|------|-------|
+| HH:MM | [First detection] |
+| HH:MM | [Incident declared] |
+| HH:MM | [Mitigation started] |
+| HH:MM | [Resolved] |
+
+## Root Cause
+[Detailed explanation of why this happened]
+
+### Contributing Factors
+1. [Factor 1]
+2. [Factor 2]
+
+## Detection
+- **How Detected:** [Alert/User report/Manual]
+- **Time to Detect:** [Duration]
+- **Detection Gap:** [What could have detected sooner]
+
+## Response
+- **What Worked:** [Effective actions]
+- **What Didn't:** [Ineffective or delayed actions]
+- **Escalation:** [Was escalation appropriate?]
+
+## Lessons Learned
+### What Went Well
+- [Positive 1]
+- [Positive 2]
+
+### What Went Poorly
+- [Issue 1]
+- [Issue 2]
+
+## Action Items
+| Action | Owner | Due Date | Status |
+|--------|-------|----------|--------|
+| [Action 1] | [Name] | [Date] | [ ] |
+| [Action 2] | [Name] | [Date] | [ ] |
+
+## AI-Specific Analysis
+### Model Behavior
+- **Expected:** [What should have happened]
+- **Actual:** [What happened]
+- **Gap:** [Why the difference]
+
+### Prompt Analysis
+- **Prompt Version:** [v1.2.3]
+- **Recent Changes:** [What changed]
+- **Rollback Used:** [Yes/No]
+
+### Prevention
+- [ ] Added to evaluation dataset
+- [ ] Updated content filters
+- [ ] Added monitoring for similar patterns
+- [ ] Documented in knowledge base
+```
+
+---
+
+## 7. Degradation vs Outage
+
+### Classification
+
+| State | Characteristics | User Experience |
+|-------|----------------|-----------------|
+| **Healthy** | Normal metrics | Full service |
+| **Degraded** | High latency, reduced quality | Slow but works |
+| **Partial Outage** | Some features down | Core works |
+| **Full Outage** | Service unavailable | Error pages |
+
+### Degradation Handling
+
+```typescript
+enum ServiceState {
+  HEALTHY = 'healthy',
+  DEGRADED = 'degraded',
+  PARTIAL_OUTAGE = 'partial_outage',
+  OUTAGE = 'outage',
+}
+
+function determineServiceState(metrics: AIMetrics): ServiceState {
+  const { errorRate, p99Latency, successRate } = metrics;
+  
+  if (successRate < 0.5) return ServiceState.OUTAGE;
+  if (successRate < 0.9) return ServiceState.PARTIAL_OUTAGE;
+  if (errorRate > 0.05 || p99Latency > 30000) return ServiceState.DEGRADED;
+  return ServiceState.HEALTHY;
+}
+
+// Communicate appropriately
+function getStatusMessage(state: ServiceState): string {
+  switch (state) {
+    case ServiceState.DEGRADED:
+      return "AI responses may be slower than usual. Thank you for your patience.";
+    case ServiceState.PARTIAL_OUTAGE:
+      return "Some AI features are currently limited. Basic functions are available.";
+    case ServiceState.OUTAGE:
+      return "AI services are temporarily unavailable. Please try again later.";
+    default:
+      return "";
+  }
+}
+```
+
+---
+
+## 8. Communication Templates
+
+### Internal Alert
+
+```markdown
+🚨 **AI Incident Declared**
+
+**Severity:** P[X]
+**Type:** [Hallucination/Toxicity/Outage]
+**Status:** Investigating
+
+**Impact:** [Brief description]
+**Incident Commander:** @[name]
+**Channel:** #incident-[date]-ai
+
+Updates every [15/30] minutes until resolved.
+```
+
+### User-Facing (if needed)
+
+```markdown
+**Service Notice**
+
+We're currently experiencing issues with our AI assistant. You may notice:
+- Slower response times
+- Reduced functionality
+
+Our team is actively working on a resolution. We apologize for any inconvenience.
+
+Last updated: [Time]
+```
+
+---
+
+## 9. On-Call Runbook
+
+### First Responder Checklist
+
+```markdown
+## When Paged for AI Incident
+
+1. **Acknowledge** the alert (do this first!)
+
+2. **Quick Assessment** (2 min max)
+   - [ ] Check status page for provider outages
+   - [ ] Check error rate dashboard
+   - [ ] Check recent deployments
+
+3. **Declare or Escalate**
+   - If clear cause → Fix
+   - If unclear → Declare incident
+   - If serious → Page secondary
+
+4. **Communicate**
+   - [ ] Update status in #on-call
+   - [ ] Create incident channel if P1/P2
+   - [ ] Set timer for next update
+
+5. **Mitigate**
+   - Refer to playbooks above
+   - When in doubt, use fallback/kill switch
+```
+
+### Key Links
+
+```yaml
+quick_links:
+  dashboards:
+    - AI Health: https://grafana.internal/ai-health
+    - LLM Metrics: https://grafana.internal/llm
+    - Error Tracking: https://sentry.internal/ai-service
+  
+  provider_status:
+    - OpenAI: https://status.openai.com
+    - Anthropic: https://status.anthropic.com
+    
+  runbooks:
+    - Hallucination: /docs/runbooks/hallucination.md
+    - Toxicity: /docs/runbooks/toxicity.md
+    - Provider Outage: /docs/runbooks/provider-outage.md
+```
+
+---
+
+## 10. Checklist
+
+### Preparation
+
+- [ ] Incident response team defined
+- [ ] Escalation paths documented
+- [ ] Kill switches tested
+- [ ] Fallback models configured
+- [ ] Post-mortem template ready
+- [ ] Communication templates approved
+
+### During Incident
+
+- [ ] Incident acknowledged
+- [ ] Severity determined
+- [ ] Incident commander assigned
+- [ ] Communication channel created
+- [ ] Timeline being tracked
+- [ ] Updates being shared
+
+### Post-Incident
+
+- [ ] Post-mortem scheduled
+- [ ] Action items tracked
+- [ ] Lessons shared with team
+- [ ] Monitoring improved
+- [ ] Playbook updated
+
+---
+
+> **Remember:** The goal isn't to prevent all incidents—it's to detect fast, respond faster, and never repeat.