mumucc 0.4.4 → 0.4.10

package/defaults/plugin-seed/marketplaces/claude-code-workflows/plugins/incident-response/skills/on-call-handoff-patterns/SKILL.md

@@ -0,0 +1,331 @@
+---
+name: on-call-handoff-patterns
+description: Master on-call shift handoffs with context transfer, escalation procedures, and documentation. Use this skill when transitioning on-call responsibilities between engineers and ensuring the incoming responder has full situational awareness, when writing a shift summary that captures active incidents, ongoing investigations, and recent changes, when handing off mid-incident so a fresh engineer can take over the incident commander role without losing context, when onboarding a new engineer to the on-call rotation for the first time, or when auditing and improving the quality of existing handoff processes across teams.
+---
+
+# On-Call Handoff Patterns
+
+Effective patterns for on-call shift transitions, ensuring continuity, context transfer, and reliable incident response across shifts.
+
+## When to Use This Skill
+
+- Transitioning on-call responsibilities
+- Writing shift handoff summaries
+- Documenting ongoing investigations
+- Establishing on-call rotation procedures
+- Improving handoff quality
+- Onboarding new on-call engineers
+
+## Core Concepts
+
+### 1. Handoff Components
+
+| Component                  | Purpose                 |
+| -------------------------- | ----------------------- |
+| **Active Incidents**       | What's currently broken |
+| **Ongoing Investigations** | Issues being debugged   |
+| **Recent Changes**         | Deployments, configs    |
+| **Known Issues**           | Workarounds in place    |
+| **Upcoming Events**        | Maintenance, releases   |
+
+### 2. Handoff Timing
+
+```
+Recommended: 30 min overlap between shifts
+
+Outgoing:
+├── 15 min: Write handoff document
+└── 15 min: Sync call with incoming
+
+Incoming:
+├── 15 min: Review handoff document
+├── 15 min: Sync call with outgoing
+└── 5 min: Verify alerting setup
+```
+
+## Templates
+
+### Template 1: Shift Handoff Document
+
+````markdown
+# On-Call Handoff: Platform Team
+
+**Outgoing**: @alice (2024-01-15 to 2024-01-22)
+**Incoming**: @bob (2024-01-22 to 2024-01-29)
+**Handoff Time**: 2024-01-22 09:00 UTC
+
+---
+
+## 🔴 Active Incidents
+
+### None currently active
+
+No active incidents at handoff time.
+
+---
+
+## 🟡 Ongoing Investigations
+
+### 1. Intermittent API Timeouts (ENG-1234)
+
+**Status**: Investigating
+**Started**: 2024-01-20
+**Impact**: ~0.1% of requests timing out
+
+**Context**:
+
+- Timeouts correlate with database backup window (02:00-03:00 UTC)
+- Suspect backup process causing lock contention
+- Added extra logging in PR #567 (deployed 01/21)
+
+**Next Steps**:
+
+- [ ] Review new logs after tonight's backup
+- [ ] Consider moving backup window if confirmed
+
+**Resources**:
+
+- Dashboard: [API Latency](https://grafana/d/api-latency)
+- Thread: #platform-eng (01/20, 14:32)
+
+---
+
+### 2. Memory Growth in Auth Service (ENG-1235)
+
+**Status**: Monitoring
+**Started**: 2024-01-18
+**Impact**: None yet (proactive)
+
+**Context**:
+
+- Memory usage growing ~5% per day
+- No memory leak found in profiling
+- Suspect connection pool not releasing properly
+
+**Next Steps**:
+
+- [ ] Review heap dump from 01/21
+- [ ] Consider restart if usage > 80%
+
+**Resources**:
+
+- Dashboard: [Auth Service Memory](https://grafana/d/auth-memory)
+- Analysis doc: [Memory Investigation](https://docs/eng-1235)
+
+---
+
+## 🟢 Resolved This Shift
+
+### Payment Service Outage (2024-01-19)
+
+- **Duration**: 23 minutes
+- **Root Cause**: Database connection exhaustion
+- **Resolution**: Rolled back v2.3.4, increased pool size
+- **Postmortem**: [POSTMORTEM-89](https://docs/postmortem-89)
+- **Follow-up tickets**: ENG-1230, ENG-1231
+
+---
+
+## 📋 Recent Changes
+
+### Deployments
+
+| Service      | Version | Time        | Notes                      |
+| ------------ | ------- | ----------- | -------------------------- |
+| api-gateway  | v3.2.1  | 01/21 14:00 | Bug fix for header parsing |
+| user-service | v2.8.0  | 01/20 10:00 | New profile features       |
+| auth-service | v4.1.2  | 01/19 16:00 | Security patch             |
+
+### Configuration Changes
+
+- 01/21: Increased API rate limit from 1000 to 1500 RPS
+- 01/20: Updated database connection pool max from 50 to 75
+
+### Infrastructure
+
+- 01/20: Added 2 nodes to Kubernetes cluster
+- 01/19: Upgraded Redis from 6.2 to 7.0
+
+---
+
+## ⚠️ Known Issues & Workarounds
+
+### 1. Slow Dashboard Loading
+
+**Issue**: Grafana dashboards slow on Monday mornings
+**Workaround**: Wait 5 min after 08:00 UTC for cache warm-up
+**Ticket**: OPS-456 (P3)
+
+### 2. Flaky Integration Test
+
+**Issue**: `test_payment_flow` fails intermittently in CI
+**Workaround**: Re-run failed job (usually passes on retry)
+**Ticket**: ENG-1200 (P2)
+
+---
+
+## 📅 Upcoming Events
+
+| Date        | Event                | Impact              | Contact       |
+| ----------- | -------------------- | ------------------- | ------------- |
+| 01/23 02:00 | Database maintenance | 5 min read-only     | @dba-team     |
+| 01/24 14:00 | Major release v5.0   | Monitor closely     | @release-team |
+| 01/25       | Marketing campaign   | 2x traffic expected | @platform     |
+
+---
+
+## 📞 Escalation Reminders
+
+| Issue Type      | First Escalation     | Second Escalation |
+| --------------- | -------------------- | ----------------- |
+| Payment issues  | @payments-oncall     | @payments-manager |
+| Auth issues     | @auth-oncall         | @security-team    |
+| Database issues | @dba-team            | @infra-manager    |
+| Unknown/severe  | @engineering-manager | @vp-engineering   |
+
+---
+
+## 🔧 Quick Reference
+
+### Common Commands
+
+```bash
+# Check service health
+kubectl get pods -A | grep -v Running
+
+# Recent deployments
+kubectl get events --sort-by='.lastTimestamp' | tail -20
+
+# Database connections
+psql -c "SELECT count(*) FROM pg_stat_activity;"
+
+# Clear cache (emergency only)
+redis-cli FLUSHDB
+```
+````
+
+### Important Links
+
+- [Runbooks](https://wiki/runbooks)
+- [Service Catalog](https://wiki/services)
+- [Incident Slack](https://slack.com/incidents)
+- [PagerDuty](https://pagerduty.com/schedules)
+
+---
+
+## Handoff Checklist
+
+### Outgoing Engineer
+
+- [x] Document active incidents
+- [x] Document ongoing investigations
+- [x] List recent changes
+- [x] Note known issues
+- [x] Add upcoming events
+- [x] Sync with incoming engineer
+
+### Incoming Engineer
+
+- [ ] Read this document
+- [ ] Join sync call
+- [ ] Verify PagerDuty is routing to you
+- [ ] Verify Slack notifications working
+- [ ] Check VPN/access working
+- [ ] Review critical dashboards
+
+````
+
+### Template 2: Quick Handoff (Async)
+
+```markdown
+# Quick Handoff: @alice → @bob
+
+## TL;DR
+- No active incidents
+- 1 investigation ongoing (API timeouts, see ENG-1234)
+- Major release tomorrow (01/24) - be ready for issues
+
+## Watch List
+1. API latency around 02:00-03:00 UTC (backup window)
+2. Auth service memory (restart if > 80%)
+
+## Recent
+- Deployed api-gateway v3.2.1 yesterday (stable)
+- Increased rate limits to 1500 RPS
+
+## Coming Up
+- 01/23 02:00 - DB maintenance (5 min read-only)
+- 01/24 14:00 - v5.0 release
+
+## Questions?
+I'll be available on Slack until 17:00 today.
+````
+
+### Template 3: Incident Handoff (Mid-Incident)
+
+```markdown
+# INCIDENT HANDOFF: Payment Service Degradation
+
+**Incident Start**: 2024-01-22 08:15 UTC
+**Current Status**: Mitigating
+**Severity**: SEV2
+
+---
+
+## Current State
+
+- Error rate: 15% (down from 40%)
+- Mitigation in progress: scaling up pods
+- ETA to resolution: ~30 min
+
+## What We Know
+
+1. Root cause: Memory pressure on payment-service pods
+2. Triggered by: Unusual traffic spike (3x normal)
+3. Contributing: Inefficient query in checkout flow
+
+## What We've Done
+
+- Scaled payment-service from 5 → 15 pods
+- Enabled rate limiting on checkout endpoint
+- Disabled non-critical features
+
+## What Needs to Happen
+
+1. Monitor error rate - should reach <1% in ~15 min
+2. If not improving, escalate to @payments-manager
+3. Once stable, begin root cause investigation
+
+## Key People
+
+- Incident Commander: @alice (handing off)
+- Comms Lead: @charlie
+- Technical Lead: @bob (incoming)
+
+## Communication
+
+- Status page: Updated at 08:45
+- Customer support: Notified
+- Exec team: Aware
+
+## Troubleshooting
+
+**Incoming engineer misses a critical issue because the handoff document was incomplete.**
+Use the outgoing checklist as a gate: do not mark handoff complete until every section has at least one entry (or an explicit "none"). Make incomplete handoffs a blameless postmortem action item.
+
+**A 30-minute sync call is not possible due to timezone gaps.**
+Fall back to the async quick handoff template (Template 2). Supplement with a short Loom or voice memo walking through the watch list. Ensure the incoming engineer has a direct contact method if they have follow-up questions.
+
+**The incoming engineer inherits a mid-incident and is immediately overwhelmed.**
+Use the incident handoff template (Template 3) specifically. The outgoing engineer should remain available on Slack for 15 minutes after handoff, even if off-call, to answer clarifying questions.
+
+**On-call handoff documents are inconsistently formatted across teams.**
+Adopt the shift handoff template organization-wide and store completed handoffs in a shared location (wiki, Notion, Confluence). Link each handoff from the on-call schedule entry in PagerDuty.
+
+**Incoming engineer cannot verify their alerting is working before the outgoing engineer logs off.**
+Add a standard step: outgoing engineer fires a test alert and confirms incoming engineer receives it in PagerDuty and Slack before ending the overlap window.
+
+## Related Skills
+
+- [incident-classification](../../skills/incident-classification/SKILL.md) — Classify and prioritize incidents that need to be included in the handoff document
+- [postmortem-facilitation](../../skills/postmortem-facilitation/SKILL.md) — Turn resolved incidents from the shift into structured postmortems

package/defaults/plugin-seed/marketplaces/claude-code-workflows/plugins/incident-response/skills/postmortem-writing/SKILL.md

@@ -0,0 +1,390 @@
+---
+name: postmortem-writing
+description: Write effective blameless postmortems with root cause analysis, timelines, and action items. Use when conducting incident reviews, writing postmortem documents, or improving incident response processes.
+---
+
+# Postmortem Writing
+
+Comprehensive guide to writing effective, blameless postmortems that drive organizational learning and prevent incident recurrence.
+
+## When to Use This Skill
+
+- Conducting post-incident reviews
+- Writing postmortem documents
+- Facilitating blameless postmortem meetings
+- Identifying root causes and contributing factors
+- Creating actionable follow-up items
+- Building organizational learning culture
+
+## Core Concepts
+
+### 1. Blameless Culture
+
+| Blame-Focused            | Blameless                         |
+| ------------------------ | --------------------------------- |
+| "Who caused this?"       | "What conditions allowed this?"   |
+| "Someone made a mistake" | "The system allowed this mistake" |
+| Punish individuals       | Improve systems                   |
+| Hide information         | Share learnings                   |
+| Fear of speaking up      | Psychological safety              |
+
+### 2. Postmortem Triggers
+
+- SEV1 or SEV2 incidents
+- Customer-facing outages > 15 minutes
+- Data loss or security incidents
+- Near-misses that could have been severe
+- Novel failure modes
+- Incidents requiring unusual intervention
+
+## Quick Start
+
+### Postmortem Timeline
+
+```
+Day 0: Incident occurs
+Day 1-2: Draft postmortem document
+Day 3-5: Postmortem meeting
+Day 5-7: Finalize document, create tickets
+Week 2+: Action item completion
+Quarterly: Review patterns across incidents
+```
+
+## Templates
+
+### Template 1: Standard Postmortem
+
+```markdown
+# Postmortem: [Incident Title]
+
+**Date**: 2024-01-15
+**Authors**: @alice, @bob
+**Status**: Draft | In Review | Final
+**Incident Severity**: SEV2
+**Incident Duration**: 47 minutes
+
+## Executive Summary
+
+On January 15, 2024, the payment processing service experienced a 47-minute outage affecting approximately 12,000 customers. The root cause was a database connection pool exhaustion triggered by a configuration change in deployment v2.3.4. The incident was resolved by rolling back to v2.3.3 and increasing connection pool limits.
+
+**Impact**:
+
+- 12,000 customers unable to complete purchases
+- Estimated revenue loss: $45,000
+- 847 support tickets created
+- No data loss or security implications
+
+## Timeline (All times UTC)
+
+| Time  | Event                                           |
+| ----- | ----------------------------------------------- |
+| 14:23 | Deployment v2.3.4 completed to production       |
+| 14:31 | First alert: `payment_error_rate > 5%`          |
+| 14:33 | On-call engineer @alice acknowledges alert      |
+| 14:35 | Initial investigation begins, error rate at 23% |
+| 14:41 | Incident declared SEV2, @bob joins              |
+| 14:45 | Database connection exhaustion identified       |
+| 14:52 | Decision to rollback deployment                 |
+| 14:58 | Rollback to v2.3.3 initiated                    |
+| 15:10 | Rollback complete, error rate dropping          |
+| 15:18 | Service fully recovered, incident resolved      |
+
+## Root Cause Analysis
+
+### What Happened
+
+The v2.3.4 deployment included a change to the database query pattern that inadvertently removed connection pooling for a frequently-called endpoint. Each request opened a new database connection instead of reusing pooled connections.
+
+### Why It Happened
+
+1. **Proximate Cause**: Code change in `PaymentRepository.java` replaced pooled `DataSource` with direct `DriverManager.getConnection()` calls.
+
+2. **Contributing Factors**:
+   - Code review did not catch the connection handling change
+   - No integration tests specifically for connection pool behavior
+   - Staging environment has lower traffic, masking the issue
+   - Database connection metrics alert threshold was too high (90%)
+
+3. **5 Whys Analysis**:
+   - Why did the service fail? → Database connections exhausted
+   - Why were connections exhausted? → Each request opened new connection
+   - Why did each request open new connection? → Code bypassed connection pool
+   - Why did code bypass connection pool? → Developer unfamiliar with codebase patterns
+   - Why was developer unfamiliar? → No documentation on connection management patterns
+
+### System Diagram
+```
+
+[Client] → [Load Balancer] → [Payment Service] → [Database]
+↓
+Connection Pool (broken)
+↓
+Direct connections (cause)
+
+```
+
+## Detection
+
+### What Worked
+- Error rate alert fired within 8 minutes of deployment
+- Grafana dashboard clearly showed connection spike
+- On-call response was swift (2 minute acknowledgment)
+
+### What Didn't Work
+- Database connection metric alert threshold too high
+- No deployment-correlated alerting
+- Canary deployment would have caught this earlier
+
+### Detection Gap
+The deployment completed at 14:23, but the first alert didn't fire until 14:31 (8 minutes). A deployment-aware alert could have detected the issue faster.
+
+## Response
+
+### What Worked
+- On-call engineer quickly identified database as the issue
+- Rollback decision was made decisively
+- Clear communication in incident channel
+
+### What Could Be Improved
+- Took 10 minutes to correlate issue with recent deployment
+- Had to manually check deployment history
+- Rollback took 12 minutes (could be faster)
+
+## Impact
+
+### Customer Impact
+- 12,000 unique customers affected
+- Average impact duration: 35 minutes
+- 847 support tickets (23% of affected users)
+- Customer satisfaction score dropped 12 points
+
+### Business Impact
+- Estimated revenue loss: $45,000
+- Support cost: ~$2,500 (agent time)
+- Engineering time: ~8 person-hours
+
+### Technical Impact
+- Database primary experienced elevated load
+- Some replica lag during incident
+- No permanent damage to systems
+
+## Lessons Learned
+
+### What Went Well
+1. Alerting detected the issue before customer reports
+2. Team collaborated effectively under pressure
+3. Rollback procedure worked smoothly
+4. Communication was clear and timely
+
+### What Went Wrong
+1. Code review missed critical change
+2. Test coverage gap for connection pooling
+3. Staging environment doesn't reflect production traffic
+4. Alert thresholds were not tuned properly
+
+### Where We Got Lucky
+1. Incident occurred during business hours with full team available
+2. Database handled the load without failing completely
+3. No other incidents occurred simultaneously
+
+## Action Items
+
+| Priority | Action | Owner | Due Date | Ticket |
+|----------|--------|-------|----------|--------|
+| P0 | Add integration test for connection pool behavior | @alice | 2024-01-22 | ENG-1234 |
+| P0 | Lower database connection alert threshold to 70% | @bob | 2024-01-17 | OPS-567 |
+| P1 | Document connection management patterns | @alice | 2024-01-29 | DOC-89 |
+| P1 | Implement deployment-correlated alerting | @bob | 2024-02-05 | OPS-568 |
+| P2 | Evaluate canary deployment strategy | @charlie | 2024-02-15 | ENG-1235 |
+| P2 | Load test staging with production-like traffic | @dave | 2024-02-28 | QA-123 |
+
+## Appendix
+
+### Supporting Data
+
+#### Error Rate Graph
+[Link to Grafana dashboard snapshot]
+
+#### Database Connection Graph
+[Link to metrics]
+
+### Related Incidents
+- 2023-11-02: Similar connection issue in User Service (POSTMORTEM-42)
+
+### References
+- [Connection Pool Best Practices](internal-wiki/connection-pools)
+- [Deployment Runbook](internal-wiki/deployment-runbook)
+```
+
+### Template 2: 5 Whys Analysis
+
+```markdown
+# 5 Whys Analysis: [Incident]
+
+## Problem Statement
+
+Payment service experienced 47-minute outage due to database connection exhaustion.
+
+## Analysis
+
+### Why #1: Why did the service fail?
+
+**Answer**: Database connections were exhausted, causing all new requests to fail.
+
+**Evidence**: Metrics showed connection count at 100/100 (max), with 500+ pending requests.
+
+---
+
+### Why #2: Why were database connections exhausted?
+
+**Answer**: Each incoming request opened a new database connection instead of using the connection pool.
+
+**Evidence**: Code diff shows direct `DriverManager.getConnection()` instead of pooled `DataSource`.
+
+---
+
+### Why #3: Why did the code bypass the connection pool?
+
+**Answer**: A developer refactored the repository class and inadvertently changed the connection acquisition method.
+
+**Evidence**: PR #1234 shows the change, made while fixing a different bug.
+
+---
+
+### Why #4: Why wasn't this caught in code review?
+
+**Answer**: The reviewer focused on the functional change (the bug fix) and didn't notice the infrastructure change.
+
+**Evidence**: Review comments only discuss business logic.
+
+---
+
+### Why #5: Why isn't there a safety net for this type of change?
+
+**Answer**: We lack automated tests that verify connection pool behavior and lack documentation about our connection patterns.
+
+**Evidence**: Test suite has no tests for connection handling; wiki has no article on database connections.
+
+## Root Causes Identified
+
+1. **Primary**: Missing automated tests for infrastructure behavior
+2. **Secondary**: Insufficient documentation of architectural patterns
+3. **Tertiary**: Code review checklist doesn't include infrastructure considerations
+
+## Systemic Improvements
+
+| Root Cause    | Improvement                       | Type       |
+| ------------- | --------------------------------- | ---------- |
+| Missing tests | Add infrastructure behavior tests | Prevention |
+| Missing docs  | Document connection patterns      | Prevention |
+| Review gaps   | Update review checklist           | Detection  |
+| No canary     | Implement canary deployments      | Mitigation |
+```
+
+### Template 3: Quick Postmortem (Minor Incidents)
+
+```markdown
+# Quick Postmortem: [Brief Title]
+
+**Date**: 2024-01-15 | **Duration**: 12 min | **Severity**: SEV3
+
+## What Happened
+
+API latency spiked to 5s due to cache miss storm after cache flush.
+
+## Timeline
+
+- 10:00 - Cache flush initiated for config update
+- 10:02 - Latency alerts fire
+- 10:05 - Identified as cache miss storm
+- 10:08 - Enabled cache warming
+- 10:12 - Latency normalized
+
+## Root Cause
+
+Full cache flush for minor config update caused thundering herd.
+
+## Fix
+
+- Immediate: Enabled cache warming
+- Long-term: Implement partial cache invalidation (ENG-999)
+
+## Lessons
+
+Don't full-flush cache in production; use targeted invalidation.
+```
+
+## Facilitation Guide
+
+### Running a Postmortem Meeting
+
+```markdown
+## Meeting Structure (60 minutes)
+
+### 1. Opening (5 min)
+
+- Remind everyone of blameless culture
+- "We're here to learn, not to blame"
+- Review meeting norms
+
+### 2. Timeline Review (15 min)
+
+- Walk through events chronologically
+- Ask clarifying questions
+- Identify gaps in timeline
+
+### 3. Analysis Discussion (20 min)
+
+- What failed?
+- Why did it fail?
+- What conditions allowed this?
+- What would have prevented it?
+
+### 4. Action Items (15 min)
+
+- Brainstorm improvements
+- Prioritize by impact and effort
+- Assign owners and due dates
+
+### 5. Closing (5 min)
+
+- Summarize key learnings
+- Confirm action item owners
+- Schedule follow-up if needed
+
+## Facilitation Tips
+
+- Keep discussion on track
+- Redirect blame to systems
+- Encourage quiet participants
+- Document dissenting views
+- Time-box tangents
+```
+
+## Anti-Patterns to Avoid
+
+| Anti-Pattern            | Problem                    | Better Approach                 |
+| ----------------------- | -------------------------- | ------------------------------- |
+| **Blame game**          | Shuts down learning        | Focus on systems                |
+| **Shallow analysis**    | Doesn't prevent recurrence | Ask "why" 5 times               |
+| **No action items**     | Waste of time              | Always have concrete next steps |
+| **Unrealistic actions** | Never completed            | Scope to achievable tasks       |
+| **No follow-up**        | Actions forgotten          | Track in ticketing system       |
+
+## Best Practices
+
+### Do's
+
+- **Start immediately** - Memory fades fast
+- **Be specific** - Exact times, exact errors
+- **Include graphs** - Visual evidence
+- **Assign owners** - No orphan action items
+- **Share widely** - Organizational learning
+
+### Don'ts
+
+- **Don't name and shame** - Ever
+- **Don't skip small incidents** - They reveal patterns
+- **Don't make it a blame doc** - That kills learning
+- **Don't create busywork** - Actions should be meaningful
+- **Don't skip follow-up** - Verify actions completed