specweave 0.1.0

package/src/agents/sre/playbooks/01-high-cpu-usage.md

@@ -0,0 +1,204 @@
+# Playbook: High CPU Usage
+
+## Symptoms
+
+- CPU usage at 80-100%
+- Applications slow or unresponsive
+- Server lag, SSH slow
+- Monitoring alert: "CPU usage >80% for 5 minutes"
+
+## Severity
+
+- **SEV2** if application degraded but functional
+- **SEV1** if application unresponsive
+
+## Diagnosis
+
+### Step 1: Identify Top CPU Process
+
+```bash
+# Current CPU usage
+top -bn1 | head -20
+
+# Top CPU processes
+ps aux | sort -nrk 3,3 | head -10
+
+# CPU per thread
+top -H -p <PID>
+```
+
+**What to look for**:
+- Single process using >80% CPU
+- Multiple processes all high (system-wide issue)
+- System CPU vs user CPU (iowait = disk issue)
+
+---
+
+### Step 2: Identify Process Type
+
+**Application process** (node, java, python):
+```bash
+# Check application logs
+tail -100 /var/log/application.log
+
+# Check for infinite loops, heavy computation
+# Check APM for slow endpoints
+```
+
+**System process** (kernel, systemd):
+```bash
+# Check system logs
+dmesg | tail -50
+journalctl -xe
+
+# Check for hardware issues
+```
+
+**Unknown/suspicious process**:
+```bash
+# Check process details
+ps aux | grep <PID>
+lsof -p <PID>
+
+# Could be malware (crypto mining)
+# See security-incidents.md
+```
+
+---
+
+### Step 3: Check If Disk-Related
+
+```bash
+# Check iowait
+iostat -x 1 5
+
+# If iowait >20%, disk is bottleneck
+# See infrastructure.md for disk I/O troubleshooting
+```
+
+---
+
+## Mitigation
+
+### Immediate (Now - 5 min)
+
+**Option A: Lower Process Priority**
+```bash
+# Reduce CPU priority
+renice +10 <PID>
+
+# Impact: Process gets less CPU time
+# Risk: Low (process still runs, just slower)
+```
+
+**Option B: Kill Process** (if application)
+```bash
+# Graceful shutdown
+kill -TERM <PID>
+
+# Force kill (last resort)
+kill -KILL <PID>
+
+# Restart service
+systemctl restart <service>
+
+# Impact: Process restarts, CPU normalizes
+# Risk: Medium (brief downtime)
+```
+
+**Option C: Scale Horizontally** (cloud)
+```bash
+# Add more instances to distribute load
+# AWS: Auto Scaling Group
+# Azure: Scale Set
+# Kubernetes: Horizontal Pod Autoscaler
+
+# Impact: Load distributed across instances
+# Risk: Low (no downtime)
+```
+
+---
+
+### Short-term (5 min - 1 hour)
+
+**Option A: Optimize Code** (if application bug)
+```bash
+# Profile application
+# Node.js: node --prof
+# Java: jstack, jvisualvm
+# Python: py-spy
+
+# Identify hot path
+# Fix infinite loop, heavy computation
+```
+
+**Option B: Add Caching**
+```javascript
+// Cache expensive computation
+const cache = new Map();
+
+function expensiveOperation(input) {
+  if (cache.has(input)) {
+    return cache.get(input);
+  }
+
+  const result = /* heavy computation */;
+  cache.set(input, result);
+  return result;
+}
+```
+
+**Option C: Scale Vertically** (cloud)
+```bash
+# Resize to larger instance type
+# AWS: Change instance type (t3.medium → t3.large)
+# Azure: Resize VM
+# Impact: More CPU capacity
+# Risk: Medium (brief downtime during resize)
+```
+
+---
+
+### Long-term (1 hour+)
+
+- [ ] Add CPU monitoring alert (>70% for 5 min)
+- [ ] Optimize application code (reduce computation)
+- [ ] Use worker threads for heavy tasks (Node.js)
+- [ ] Implement auto-scaling (cloud)
+- [ ] Add APM for performance profiling
+- [ ] Review architecture (async processing, job queues)
+
+---
+
+## Escalation
+
+**Escalate to developer if**:
+- Application code causing issue
+- Requires code fix or optimization
+
+**Escalate to security-agent if**:
+- Unknown/suspicious process
+- Potential malware or crypto mining
+
+**Escalate to infrastructure if**:
+- Hardware issue (kernel errors)
+- Cloud infrastructure problem
+
+---
+
+## Related Runbooks
+
+- [03-memory-leak.md](03-memory-leak.md) - If memory also high
+- [04-slow-api-response.md](04-slow-api-response.md) - If API slow due to CPU
+- [../modules/infrastructure.md](../modules/infrastructure.md) - Infrastructure diagnostics
+
+---
+
+## Post-Incident
+
+After resolving:
+- [ ] Create post-mortem (if SEV1/SEV2)
+- [ ] Identify root cause
+- [ ] Add monitoring/alerting
+- [ ] Update this runbook if needed
+- [ ] Add regression test (if code bug)

package/src/agents/sre/playbooks/02-database-deadlock.md

@@ -0,0 +1,241 @@
+# Playbook: Database Deadlock
+
+## Symptoms
+
+- "Deadlock detected" errors in application
+- API returning 500 errors
+- Transactions timing out
+- Database connection pool exhausted
+- Monitoring alert: "Deadlock count >0"
+
+## Severity
+
+- **SEV2** if isolated to specific endpoint
+- **SEV1** if affecting all database operations
+
+## Diagnosis
+
+### Step 1: Confirm Deadlock (PostgreSQL)
+
+```sql
+-- Check for currently locked queries
+SELECT
+  blocked_locks.pid AS blocked_pid,
+  blocked_activity.usename AS blocked_user,
+  blocking_locks.pid AS blocking_pid,
+  blocking_activity.usename AS blocking_user,
+  blocked_activity.query AS blocked_statement,
+  blocking_activity.query AS blocking_statement
+FROM pg_catalog.pg_locks blocked_locks
+JOIN pg_catalog.pg_stat_activity blocked_activity
+  ON blocked_activity.pid = blocked_locks.pid
+JOIN pg_catalog.pg_locks blocking_locks
+  ON blocking_locks.locktype = blocked_locks.locktype
+  AND blocking_locks.database IS NOT DISTINCT FROM blocked_locks.database
+  AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation
+  AND blocking_locks.pid != blocked_locks.pid
+JOIN pg_catalog.pg_stat_activity blocking_activity
+  ON blocking_activity.pid = blocking_locks.pid
+WHERE NOT blocked_locks.granted;
+
+-- Check deadlock log
+SELECT * FROM pg_stat_database WHERE datname = 'your_database';
+```
+
+### Step 2: Confirm Deadlock (MySQL)
+
+```sql
+-- Show InnoDB status (includes deadlock info)
+SHOW ENGINE INNODB STATUS\G
+
+-- Look for "LATEST DETECTED DEADLOCK" section
+-- Shows which transactions were involved
+```
+
+---
+
+### Step 3: Identify Deadlock Pattern
+
+**Common Pattern 1: Lock Order Mismatch**
+```
+Transaction A: Locks row 1, then row 2
+Transaction B: Locks row 2, then row 1
+→ DEADLOCK
+```
+
+**Common Pattern 2: Gap Locks**
+```
+Transaction A: SELECT ... FOR UPDATE WHERE id BETWEEN 1 AND 10
+Transaction B: INSERT INTO table (id) VALUES (5)
+→ DEADLOCK
+```
+
+**Common Pattern 3: Foreign Key Deadlock**
+```
+Transaction A: Updates parent table
+Transaction B: Inserts into child table
+→ DEADLOCK (foreign key check locks)
+```
+
+---
+
+## Mitigation
+
+### Immediate (Now - 5 min)
+
+**Option A: Kill Blocking Query** (PostgreSQL)
+```sql
+-- Terminate blocking process
+SELECT pg_terminate_backend(<blocking_pid>);
+
+-- Verify deadlock cleared
+SELECT count(*) FROM pg_locks WHERE NOT granted;
+-- Should return 0
+```
+
+**Option B: Kill Blocking Query** (MySQL)
+```sql
+-- Show process list
+SHOW PROCESSLIST;
+
+-- Kill blocking query
+KILL <process_id>;
+```
+
+**Option C: Kill Idle Transactions** (PostgreSQL)
+```sql
+-- Find idle transactions (>5 min)
+SELECT pg_terminate_backend(pid)
+FROM pg_stat_activity
+WHERE state = 'idle in transaction'
+AND state_change < NOW() - INTERVAL '5 minutes';
+
+-- Impact: Frees up locks
+-- Risk: Low (transactions are idle)
+```
+
+---
+
+### Short-term (5 min - 1 hour)
+
+**Option A: Add Transaction Timeout** (PostgreSQL)
+```sql
+-- Set statement timeout (30 seconds)
+ALTER DATABASE your_database SET statement_timeout = '30s';
+
+-- Or in application:
+SET statement_timeout = '30s';
+
+-- Impact: Prevents long-running transactions
+-- Risk: Low (transactions should be fast)
+```
+
+**Option B: Add Transaction Timeout** (MySQL)
+```sql
+-- Set lock wait timeout
+SET GLOBAL innodb_lock_wait_timeout = 30;
+
+-- Impact: Transactions fail instead of waiting forever
+-- Risk: Low (application should handle errors)
+```
+
+**Option C: Fix Lock Order in Application**
+```javascript
+// BAD: Inconsistent lock order
+async function transferMoney(fromId, toId, amount) {
+  await db.query('UPDATE accounts SET balance = balance - ? WHERE id = ?', [amount, fromId]);
+  await db.query('UPDATE accounts SET balance = balance + ? WHERE id = ?', [amount, toId]);
+}
+
+// GOOD: Consistent lock order
+async function transferMoney(fromId, toId, amount) {
+  const firstId = Math.min(fromId, toId);
+  const secondId = Math.max(fromId, toId);
+
+  await db.query('UPDATE accounts SET balance = balance - ? WHERE id = ?', [amount, firstId]);
+  await db.query('UPDATE accounts SET balance = balance + ? WHERE id = ?', [amount, secondId]);
+}
+```
+
+---
+
+### Long-term (1 hour+)
+
+**Option A: Reduce Transaction Scope**
+```javascript
+// BAD: Long transaction
+BEGIN;
+const user = await db.query('SELECT * FROM users WHERE id = ? FOR UPDATE', [userId]);
+await sendEmail(user.email); // External call (slow!)
+await db.query('UPDATE users SET last_email_sent = NOW() WHERE id = ?', [userId]);
+COMMIT;
+
+// GOOD: Short transaction
+const user = await db.query('SELECT * FROM users WHERE id = ?', [userId]);
+await sendEmail(user.email); // Outside transaction
+await db.query('UPDATE users SET last_email_sent = NOW() WHERE id = ?', [userId]);
+```
+
+**Option B: Use Optimistic Locking**
+```sql
+-- Add version column
+ALTER TABLE accounts ADD COLUMN version INT DEFAULT 0;
+
+-- Update with version check
+UPDATE accounts
+SET balance = balance - 100, version = version + 1
+WHERE id = 1 AND version = <current_version>;
+
+-- If 0 rows updated, retry with new version
+```
+
+**Option C: Review Isolation Level**
+```sql
+-- PostgreSQL default: READ COMMITTED
+-- Most cases: READ COMMITTED is fine
+-- Rare cases: REPEATABLE READ or SERIALIZABLE
+
+-- Lower isolation = less locking = fewer deadlocks
+SET TRANSACTION ISOLATION LEVEL READ COMMITTED;
+```
+
+---
+
+## Escalation
+
+**Escalate to developer if**:
+- Application code causing deadlock
+- Requires code refactoring
+
+**Escalate to DBA if**:
+- Database configuration issue
+- Foreign key constraint problem
+
+---
+
+## Prevention
+
+- [ ] Always lock in same order
+- [ ] Keep transactions short
+- [ ] Use timeout (statement_timeout, lock_wait_timeout)
+- [ ] Use optimistic locking when possible
+- [ ] Add deadlock monitoring alert
+- [ ] Review isolation level (lower = fewer deadlocks)
+
+---
+
+## Related Runbooks
+
+- [04-slow-api-response.md](04-slow-api-response.md) - If API slow due to deadlock
+- [../modules/database-diagnostics.md](../modules/database-diagnostics.md) - Database troubleshooting
+
+---
+
+## Post-Incident
+
+After resolving:
+- [ ] Create post-mortem
+- [ ] Identify which queries deadlocked
+- [ ] Fix lock order in application code
+- [ ] Add regression test
+- [ ] Update this runbook if needed

package/src/agents/sre/playbooks/03-memory-leak.md

@@ -0,0 +1,252 @@
+# Playbook: Memory Leak
+
+## Symptoms
+
+- Memory usage increasing continuously over time
+- Application crashes with OutOfMemoryError (Java) or "JavaScript heap out of memory" (Node.js)
+- Performance degrades over time
+- High swap usage
+- Monitoring alert: "Memory usage >90%"
+
+## Severity
+
+- **SEV2** if memory increasing but not yet critical
+- **SEV1** if application crashed or unresponsive
+
+## Diagnosis
+
+### Step 1: Confirm Memory Leak
+
+```bash
+# Monitor memory over time (5 minute intervals)
+watch -n 300 'ps aux | grep <process> | awk "{print \$4, \$5, \$6}"'
+
+# Check if memory continuously increasing
+# Leak: 20% → 30% → 40% → 50% (linear growth)
+# Normal: 30% → 32% → 31% → 30% (stable)
+```
+
+---
+
+### Step 2: Get Memory Snapshot
+
+**Java (Heap Dump)**:
+```bash
+# Get heap dump
+jmap -dump:format=b,file=heap.bin <PID>
+
+# Analyze with jhat or VisualVM
+jhat heap.bin
+# Open http://localhost:7000
+
+# Or use Eclipse Memory Analyzer
+```
+
+**Node.js (Heap Snapshot)**:
+```bash
+# Start with --inspect
+node --inspect index.js
+
+# Chrome DevTools → Memory → Take heap snapshot
+
+# Or use heapdump module
+const heapdump = require('heapdump');
+heapdump.writeSnapshot('/tmp/heap-' + Date.now() + '.heapsnapshot');
+```
+
+**Python (Memory Profiler)**:
+```bash
+# Install memory_profiler
+pip install memory_profiler
+
+# Profile function
+python -m memory_profiler script.py
+```
+
+---
+
+### Step 3: Identify Leak Source
+
+**Look for**:
+- Large arrays/objects growing over time
+- Detached DOM nodes (if browser/UI)
+- Event listeners not removed
+- Timers/intervals not cleared
+- Closures holding references
+- Cache without eviction policy
+
+**Common patterns**:
+```javascript
+// 1. Global cache growing forever
+global.cache = {}; // Never cleared
+
+// 2. Event listeners not removed
+emitter.on('event', handler); // Never removed
+
+// 3. Timers not cleared
+setInterval(() => { /* ... */ }, 1000); // Never cleared
+
+// 4. Closures
+function createHandler() {
+  const largeData = new Array(1000000);
+  return () => {
+    // Closure keeps largeData in memory
+  };
+}
+```
+
+---
+
+## Mitigation
+
+### Immediate (Now - 5 min)
+
+**Option A: Restart Application**
+```bash
+# Restart to free memory
+systemctl restart application
+
+# Impact: Memory usage returns to baseline
+# Risk: Low (brief downtime)
+# NOTE: This is temporary, leak will recur!
+```
+
+**Option B: Increase Memory Limit** (temporary)
+```bash
+# Java
+java -Xmx4G -jar application.jar  # Was 2G
+
+# Node.js
+node --max-old-space-size=4096 index.js  # Was 2048
+
+# Impact: Buys time to find root cause
+# Risk: Low (but doesn't fix leak)
+```
+
+**Option C: Scale Horizontally** (cloud)
+```bash
+# Add more instances
+# Use load balancer to rotate traffic
+# Restart instances on schedule (e.g., every 6 hours)
+
+# Impact: Distributes load, restarts prevent OOM
+# Risk: Low (but doesn't fix root cause)
+```
+
+---
+
+### Short-term (5 min - 1 hour)
+
+**Analyze heap dump** and identify leak source
+
+**Common Fixes**:
+
+**1. Add LRU Cache**
+```javascript
+// BAD: Unbounded cache
+const cache = {};
+
+// GOOD: LRU cache with size limit
+const LRU = require('lru-cache');
+const cache = new LRU({ max: 1000 });
+```
+
+**2. Remove Event Listeners**
+```javascript
+// Add listener
+const handler = () => { /* ... */ };
+emitter.on('event', handler);
+
+// CRITICAL: Remove later
+emitter.off('event', handler);
+
+// React/Vue: cleanup in componentWillUnmount/onUnmounted
+```
+
+**3. Clear Timers**
+```javascript
+// Set timer
+const intervalId = setInterval(() => { /* ... */ }, 1000);
+
+// CRITICAL: Clear later
+clearInterval(intervalId);
+
+// React: cleanup in useEffect return
+useEffect(() => {
+  const id = setInterval(() => { /* ... */ }, 1000);
+  return () => clearInterval(id);
+}, []);
+```
+
+**4. Close Connections**
+```javascript
+// BAD: Connection leak
+const conn = await db.connect();
+await conn.query(/* ... */);
+// Connection never closed!
+
+// GOOD: Always close
+const conn = await db.connect();
+try {
+  await conn.query(/* ... */);
+} finally {
+  await conn.close(); // CRITICAL
+}
+```
+
+---
+
+### Long-term (1 hour+)
+
+- [ ] Add memory monitoring (alert if >80% and increasing)
+- [ ] Add memory profiling to CI/CD (detect leaks early)
+- [ ] Use WeakMap for caches (auto garbage collected)
+- [ ] Review closure usage (avoid holding large data)
+- [ ] Add automated restart (every N hours, if leak can't be fixed immediately)
+- [ ] Load test to reproduce leak in test environment
+- [ ] Fix root cause in code
+
+---
+
+## Escalation
+
+**Escalate to developer if**:
+- Application code causing leak
+- Requires code fix
+
+**Escalate to platform team if**:
+- Platform/framework bug
+- Requires upgrade or workaround
+
+---
+
+## Prevention Checklist
+
+- [ ] Use LRU cache (not unbounded)
+- [ ] Remove event listeners in cleanup
+- [ ] Clear timers/intervals
+- [ ] Close database connections (use `finally`)
+- [ ] Avoid closures holding large data
+- [ ] Use WeakMap for temporary caches
+- [ ] Profile memory in development
+- [ ] Load test before production
+
+---
+
+## Related Runbooks
+
+- [01-high-cpu-usage.md](01-high-cpu-usage.md) - If CPU also high
+- [07-service-down.md](07-service-down.md) - If OOM crashed service
+- [../modules/backend-diagnostics.md](../modules/backend-diagnostics.md) - Backend troubleshooting
+
+---
+
+## Post-Incident
+
+After resolving:
+- [ ] Create post-mortem
+- [ ] Identify leak source from heap dump
+- [ ] Fix code
+- [ ] Add regression test (memory profiling)
+- [ ] Add monitoring alert
+- [ ] Update this runbook if needed