@shaykec/bridge 0.4.25 → 0.4.26

package/modules/incident-response/content.md

@@ -0,0 +1,124 @@
+# Incident Response — From Alert to Postmortem
+
+<!-- hint:slides topic="Incident response lifecycle: severity levels, roles (IC, Comms Lead), triage, mitigation, resolution, and blameless postmortem" slides="6" -->
+
+## Incident Severity Levels
+
+Define severity so everyone knows how to respond. Common tiers:
+
+| Severity | Impact | Example | Response Time |
+|----------|--------|---------|---------------|
+| **SEV1** | Critical: full outage, data loss, security breach | Site down, payment broken | Immediate (e.g., 5 min) |
+| **SEV2** | Major: significant degradation, key feature broken | Search down, 50% errors | Within 15–30 min |
+| **SEV3** | Minor: limited impact, workaround exists | One region slow, non-critical bug | Within hours |
+| **SEV4** | Low: cosmetic or edge case | UI glitch, rare error | Next business day |
+
+Customize thresholds for your system. Document them in your runbooks so on-call knows when to escalate.
+
+## Roles During Incidents
+
+| Role | Responsibility |
+|------|----------------|
+| **Incident Commander (IC)** | Owns the response; coordinates, decides, delegates. Does not fix—orchestrates. |
+| **Comms Lead** | Updates stakeholders, status page, and internal channels. Frees IC to focus on resolution. |
+| **Responders** | Engineers debugging, rolling back, or applying fixes. Report progress to IC. |
+
+IC and Comms Lead should be different people when possible. IC stays focused on technical decisions; Comms keeps everyone informed.
+
+## The Incident Lifecycle
+
+```mermaid
+flowchart LR
+    A[Detect] --> B[Triage]
+    B --> C[Respond]
+    C --> D[Mitigate]
+    D --> E[Resolve]
+    E --> F[Postmortem]
+```
+
+```mermaid
+flowchart TD
+  A[Detect] --> B[Triage]
+  B --> C[Mitigate]
+  C --> D[Resolve]
+  D --> E[Postmortem]
+  E --> F[Action Items]
+
+  A -.->|Alert, user report| A
+  B -.->|Severity, scope| B
+  C -.->|Rollback, fix, workaround| C
+  D -.->|Verify, all-clear| D
+  E -.->|Blameless, 5 whys| E
+  F -.->|Track, prevent recurrence| F
+```
+
+1. **Detect** — Alert fires, user reports, or monitoring catches the issue.
+2. **Triage** — Assess severity, scope, and impact. Assign IC and responders.
+3. **Mitigate** — Roll back, apply fix, or implement workaround. Goal: reduce impact.
+4. **Resolve** — Verify the fix, confirm stability, declare incident closed.
+5. **Postmortem** — Blameless analysis: what happened, why, what we'll do differently.
+6. **Action Items** — Track improvements (runbooks, monitoring, code changes).
+
+## Communication Templates
+
+**Internal (Slack, email):**
+> **Incident: [Brief title]**  
+> **Severity:** SEV[1–4]  
+> **Impact:** [Who/what is affected]  
+> **Status:** Investigating / Mitigating / Resolved  
+> **ETA:** [If known]  
+> **Incident Commander:** [Name]
+
+**External (status page, customers):**
+> **We're aware of [issue].** Impact: [description]. We're investigating and will update within [time]. ETA: [if known].
+
+**Update cadence:** Every 15–30 min for SEV1/2; don't leave people guessing. "No update" is an update—say "Still investigating, next update in 15 min."
+
+## Blameless Postmortems
+
+**Goal:** Learn, not blame. Focus on systems and process, not individuals.
+
+**Structure:**
+1. **Summary** — What happened, in plain language.
+2. **Timeline** — Key events with timestamps.
+3. **Root cause** — Use "5 whys" or similar. Go past symptoms to contributing factors.
+4. **Impact** — Users affected, duration, business impact.
+5. **Action items** — What we'll change (runbooks, monitoring, code, process). Owner and due date.
+6. **Lessons learned** — What went well, what didn't.
+
+**Rule:** No names in the "who messed up" sense. "The deploy went out" not "Alice deployed the bad code." Discuss the decision-making and systems that allowed the failure.
+
+## On-Call Best Practices
+
+- **Runbooks** — Step-by-step guides for common incidents. "If X, do Y."
+- **Escalation paths** — Who gets paged when: primary → secondary → manager. Define SLA.
+- **Rotation** — Fair distribution; avoid burnout. Tools: PagerDuty, OpsGenie, VictorOps.
+- **Compensation** — On-call pay, time off, or flexibility. Make it sustainable.
+- **Training** — Shadow new on-call; run game days (simulated incidents).
+
+## Preventing Incident Fatigue
+
+- Limit consecutive weeks on primary.
+- Handoff meetings: what’s in flight, recent changes.
+- Post-incident rest: no meetings for IC for a few hours after SEV1/2.
+- Track pages: if someone is woken up constantly, fix the alerts or the system.
+
+## Learning from Incidents — SLOs and Error Budgets
+
+**SLO (Service Level Objective):** "99.9% of requests succeed."  
+**Error budget:** The allowed failure (e.g., 0.1% = ~43 min downtime/month). When you're within budget, you can ship; when you're over, focus on reliability.
+
+Incidents consume the error budget. Use postmortems to decide: was this a one-off or a systemic problem? Invest in fixes that prevent recurrence and protect the budget.
+
+## Incident Lifecycle Flow (Simplified)
+
+```mermaid
+flowchart LR
+  A[Alert] --> B[Tri age]
+  B --> C[Mitigate]
+  C --> D[Resolve]
+  D --> E[Postmortem]
+  E --> A
+```
+
+Continuous improvement: each incident makes the system more resilient if we learn from it.

package/modules/incident-response/exercises.md

@@ -0,0 +1,82 @@
+# Incident Response — Exercises
+
+## Exercise 1: Severity Classification
+
+**Task:** Classify these incidents as SEV1–4 and justify: (A) Payment processing returns 500 for 10% of users. (B) Typo on "About" page. (C) Full site returns 502 for all users. (D) Search is slow (3s vs 1s) for one region.
+
+**Validation:**
+- [ ] Each has a severity
+- [ ] Justification references impact and scope
+- [ ] Response time is implied or stated
+
+**Hints:**
+1. Full site down → SEV1
+2. Payment 500 for 10% → SEV2 (major, key feature)
+3. Search slow, one region → SEV3 (degradation, workaround: wait or retry)
+4. Typo → SEV4
+
+---
+
+## Exercise 2: Communication Template in Action
+
+**Task:** A database failover fails at 2 p.m.; the site is down. Write the internal announcement and external status update you'd post within 5 minutes. Then write the "Resolved" update for when you fix it 45 minutes later.
+
+**Validation:**
+- [ ] Internal has: severity, impact, status, IC, ETA
+- [ ] External is customer-friendly, no jargon
+- [ ] Resolved update thanks users and summarizes what happened
+
+**Hints:**
+1. Internal: "SEV1, full outage, mitigating, IC: [name], ETA: investigating"
+2. External: "We're experiencing an outage. Our team is on it. Next update in 15 min."
+3. Resolved: "Service restored. We'll share a postmortem within 48 hours."
+
+---
+
+## Exercise 3: Blameless Postmortem Draft
+
+**Task:** A deploy at 10 a.m. introduced a bug; it wasn't caught by CI. Incident resolved at 11:30 a.m. Draft the "Root Cause" section using 5 whys. Use blameless language—focus on process, not people.
+
+**Validation:**
+- [ ] 5 whys go from symptom to systemic cause
+- [ ] No blame ("the process allowed" not "Alice didn't test")
+- [ ] At least one action item is implied
+
+**Hints:**
+1. Why did the bug reach prod? → CI didn't catch it
+2. Why didn't CI catch it? → No test for this case
+3. Why no test? → Gap in test coverage / requirements
+4. Keep going to process: how do we close the gap?
+
+---
+
+## Exercise 4: Runbook Entry
+
+**Task:** Write a runbook entry for "Database replica lag exceeds 30 seconds." Include: how to detect, how to confirm, mitigation steps (2–4), escalation path, and where to log the incident.
+
+**Validation:**
+- [ ] Detection is specific (metric, dashboard, alert)
+- [ ] Mitigation steps are ordered and actionable
+- [ ] Escalation path is clear
+
+**Hints:**
+1. Detect: CloudWatch/graph, PagerDuty alert
+2. Confirm: Check replica lag metric, verify impact
+3. Mitigate: Identify slow query, kill if safe, scale read replicas, failover if critical
+4. Escalate: DB team, IC if no progress in 15 min
+
+---
+
+## Exercise 5: On-Call Rotation Design
+
+**Task:** Design an on-call rotation for a 5-person team. Include: primary and secondary, rotation cadence (e.g., weekly), handoff process, and one policy to prevent fatigue (e.g., no back-to-back primaries).
+
+**Validation:**
+- [ ] Primary and secondary defined
+- [ ] Cadence and handoff are specified
+- [ ] At least one fatigue-prevention policy
+
+**Hints:**
+1. Weekly primary, weekly secondary (different person)
+2. Handoff: 15-min call, share recent incidents, in-flight work
+3. Policy: No one does primary 2 weeks in a row; comp time after SEV1

package/modules/incident-response/game.yaml

@@ -0,0 +1,132 @@
+games:
+  - type: scenario
+    title: "Incident Commander"
+    startHealth: 5
+    steps:
+      - id: start
+        situation: "It's 2 AM. PagerDuty wakes you — the payment service is down. Alerts are firing: 503s on checkout, payment gateway timeouts. You're the incident commander. What do you do first?"
+        choices:
+          - text: "Declare SEV1 immediately and page the full team."
+            consequence: "You acted decisively. SEV1 is correct for customer-facing payment outages. The team is assembling."
+            health: 1
+            next: triage
+          - text: "Wait 5 minutes to see if it self-recovers before paging anyone."
+            consequence: "Every minute of payment downtime costs revenue and trust. SEV1 incidents need immediate response."
+            health: -2
+            next: triage
+          - text: "Post in Slack and ask if anyone has seen this before."
+            consequence: "Slack is too slow for payment outages. You need to formally declare severity and page. Time is critical."
+            health: -1
+            next: triage
+      - id: triage
+        situation: "The team is joining. You confirm: checkout is returning 503, payment gateway is timing out. One engineer says the gateway provider's status page shows 'degraded.' What's your next move?"
+        choices:
+          - text: "Create a shared doc, assign roles (comms, technical lead, scribe), and have tech lead investigate root cause."
+            consequence: "Good incident hygiene. Clear roles prevent chaos. The technical lead can focus on debugging while you coordinate."
+            health: 1
+            next: communicate
+          - text: "Have everyone start debugging in parallel."
+            consequence: "Too many cooks. Without a single technical lead and clear assignments, people duplicate work and step on each other."
+            health: -1
+            next: communicate
+          - text: "Pause and run a full architecture review before touching anything."
+            consequence: "Architecture reviews are for postmortems. During an incident, you need fast triage and mitigation, not deep analysis."
+            health: -2
+            next: communicate
+      - id: communicate
+        situation: "Stakeholders are asking for updates. Support is getting flooded with tickets. The technical lead is still investigating. What do you prioritize?"
+        choices:
+          - text: "Send a brief status to stakeholders — 'We're investigating payment gateway issues, ETA for next update in 15 min' — then let the tech lead work."
+            consequence: "Right balance. Stakeholders get reassurance; the team isn't distracted. Time-boxed updates manage expectations."
+            health: 1
+            next: debug_vs_comms
+          - text: "Ignore stakeholders until you have a root cause."
+            consequence: "Stakeholders need to know you're on it. Radio silence increases anxiety and can trigger escalations."
+            health: -1
+            next: debug_vs_comms
+          - text: "Pull the technical lead off investigation to draft a detailed customer-facing post."
+            consequence: "Investigation should take priority. Detailed posts can wait. Brief internal updates are enough for now."
+            health: -2
+            next: debug_vs_comms
+      - id: debug_vs_comms
+        situation: "The tech lead suspects a bad deploy from 2 hours ago. Rolling back would take ~20 minutes. The payment gateway provider's status still says 'degraded.' Do you rollback or wait for more data?"
+        choices:
+          - text: "Rollback now. Payment is critical; the deploy is the most likely culprit and we can't afford to wait."
+            consequence: "Pragmatic. For payment, speed matters. If the rollback fixes it, you're done. If not, you've ruled out a major variable."
+            health: 1
+            next: rollback_result
+          - text: "Wait for the gateway provider to confirm their status before deciding."
+            consequence: "External status pages can lag. Your own deploy is something you control. Delaying the rollback extends customer impact."
+            health: -1
+            next: rollback_result
+          - text: "Run A/B tests to confirm the deploy is the cause."
+            consequence: "A/B tests take time. During an incident, you need fast, reversible actions. Rollback is low risk and high signal."
+            health: -2
+            next: rollback_result
+      - id: rollback_result
+        situation: "You rolled back. Checkout recovers. The incident is resolved. Now the CEO asks for a postmortem by end of week. How do you approach it?"
+        choices:
+          - text: "Schedule a blameless postmortem with everyone involved. Focus on what we'll change (process, tooling, checks), not who screwed up."
+            consequence: "Blameless postmortems build learning culture. People share more, and you get better preventative measures."
+            health: 1
+            next: postmortem
+          - text: "Assign it to the engineer who pushed the deploy."
+            consequence: "That creates blame and discourages future transparency. Postmortems should be collaborative and blameless."
+            health: -2
+            next: postmortem
+          - text: "Skip the postmortem — we fixed it, let's move on."
+            consequence: "Every incident is a learning opportunity. Skipping postmortems means the same failures repeat."
+            health: -1
+            next: postmortem
+      - id: postmortem
+        situation: "In the postmortem, the team identifies: no canary for the payment service, and deploy went out without a staging gate. What should you document?"
+        choices:
+          - text: "Write action items: add canary deployment, require staging validation for payment service, set up alerts for gateway timeouts."
+            consequence: "Well done, Incident Commander. Clear, actionable items. You triaged severity, coordinated the team, communicated with stakeholders, made a timely rollback decision, and ran a blameless postmortem."
+            health: 1
+            next: end
+          - text: "Document that 'we'll be more careful next time.'"
+            consequence: "Vague. 'Be more careful' doesn't change systems. You need concrete process or tooling changes."
+            health: -1
+            next: end
+          - text: "Blame the engineer who merged without sufficient review."
+            consequence: "Blameless means focusing on system failures, not individuals. Blame stops people from participating honestly."
+            health: -2
+            next: end
+
+  - type: classify
+    title: "Severity Sorter"
+    categories:
+      - name: "SEV1 - Critical"
+        color: "#f85149"
+      - name: "SEV2 - Major"
+        color: "#d29922"
+      - name: "SEV3 - Minor"
+        color: "#58a6ff"
+      - name: "SEV4 - Low"
+        color: "#8b949e"
+    items:
+      - text: "All checkouts failing, payment service returning 503. Revenue impact."
+        category: "SEV1 - Critical"
+      - text: "Authentication service down — no one can log in."
+        category: "SEV1 - Critical"
+      - text: "Database primary unreachable; replicas serving read-only. Writes failing."
+        category: "SEV1 - Critical"
+      - text: "Search is slow (2–3s) but returning results. Some users complaining."
+        category: "SEV2 - Major"
+      - text: "CDN cache miss rate up 20%. Page loads slightly slower."
+        category: "SEV2 - Major"
+      - text: "Admin dashboard export fails for reports over 10k rows."
+        category: "SEV2 - Major"
+      - text: "Non-critical feature flag not applying for a small segment."
+        category: "SEV3 - Minor"
+      - text: "Email notifications delayed by 5–10 minutes."
+        category: "SEV3 - Minor"
+      - text: "Minor UI glitch on settings page in Safari only."
+        category: "SEV3 - Minor"
+      - text: "Spelling error in FAQ page."
+        category: "SEV4 - Low"
+      - text: "Deprecated API endpoint returns 410 as expected; one client not updated."
+        category: "SEV4 - Low"
+      - text: "Analytics pipeline backlog of 1 hour; dashboards slightly stale."
+        category: "SEV4 - Low"

package/modules/incident-response/module.yaml

@@ -0,0 +1,45 @@
+slug: incident-response
+title: "Incident Response — From Alert to Postmortem"
+version: 1.0.0
+description: "Handle production incidents: severity levels, roles, lifecycle, communication, blameless postmortems, and on-call best practices."
+category: leadership
+tags: [incident-response, on-call, postmortem, reliability, sre, blameless]
+difficulty: intermediate
+
+xp:
+  read: 15
+  walkthrough: 40
+  exercise: 25
+  quiz: 20
+  quiz-perfect-bonus: 10
+  game: 25
+  game-perfect-bonus: 15
+
+time:
+  quick: 5
+  read: 20
+  guided: 50
+
+prerequisites: [code-review]
+related: [risk-management, debugging-systematically, stakeholder-communication]
+
+triggers:
+  - "How do I handle a production incident?"
+  - "What is a blameless postmortem?"
+  - "How do I set up an on-call rotation?"
+  - "How do I run an incident response process?"
+
+visuals:
+  diagrams: [diagram-flow, diagram-mermaid]
+  quiz-types: [quiz-drag-order, quiz-timed-choice]
+  game-types: [scenario, classify]
+  playground: bash
+  slides: true
+
+sources:
+  - url: "https://sre.google/sre-book/"
+    label: "Google SRE Book"
+    type: docs
+  - url: "https://response.pagerduty.com"
+    label: "PagerDuty Incident Response Guide"
+    type: docs

package/modules/incident-response/quick-ref.md

@@ -0,0 +1,53 @@
+# Incident Response — Quick Reference
+
+## Severity Levels
+
+| SEV | Impact | Response |
+|-----|--------|----------|
+| 1 | Critical: full outage, data loss | Immediate (~5 min) |
+| 2 | Major: key feature broken | 15–30 min |
+| 3 | Minor: limited impact, workaround | Hours |
+| 4 | Low: cosmetic, edge case | Next business day |
+
+## Roles
+
+| Role | Responsibility |
+|------|----------------|
+| Incident Commander | Coordinate, decide, delegate |
+| Comms Lead | Stakeholder updates, status page |
+| Responders | Debug, rollback, fix |
+
+## Lifecycle
+
+1. **Detect** — Alert, user report
+2. **Triage** — Severity, scope, assign IC
+3. **Mitigate** — Rollback, fix, workaround
+4. **Resolve** — Verify, all-clear
+5. **Postmortem** — Blameless, 5 whys
+6. **Action Items** — Track, prevent recurrence
+
+## Communication Template
+
+**Internal:** Severity, impact, status, IC, ETA  
+**External:** "We're aware of X. Impact: Y. ETA: Z."  
+**Cadence:** Every 15–30 min for SEV1/2
+
+## Blameless Postmortem
+
+- Summary, Timeline, Root Cause (5 whys), Impact, Action Items, Lessons
+- No blame: focus on systems and process
+- Action items: owner, due date
+
+## On-Call Best Practices
+
+- Runbooks for common incidents
+- Escalation path: primary → secondary → manager
+- Rotation: fair, no back-to-back primary
+- Post-incident rest for IC
+- Compensation / flexibility
+
+## Error Budgets
+
+- SLO: e.g., 99.9% success
+- Error budget: allowed failure (0.1% ≈ 43 min/month)
+- Incidents consume budget; postmortems guide investment in reliability

package/modules/incident-response/quiz.md

@@ -0,0 +1,103 @@
+# Incident Response — Quiz
+
+## Question 1
+
+What is the primary role of the Incident Commander?
+
+A) Fix the bug
+B) Coordinate the response, make decisions, delegate—not necessarily fix
+C) Update the status page
+D) Write the postmortem
+
+<!-- ANSWER: B -->
+<!-- EXPLANATION: The IC owns the response and orchestrates. They coordinate responders, decide on approach, and delegate. They may not be the one typing the fix—their job is to ensure the team responds effectively. Fixing is the responders' job. -->
+
+## Question 2
+
+What makes a postmortem "blameless"?
+
+A) No one is mentioned
+B) Focus on systems and process, not individuals; we learn, not punish
+C) Only positive feedback
+D) No root cause is identified
+
+<!-- ANSWER: B -->
+<!-- EXPLANATION: Blameless means we analyze what happened in terms of systems, process, and decisions—not "who messed up." The goal is learning and preventing recurrence, not assigning fault. People need to feel safe to contribute honestly. -->
+
+## Question 3
+
+Which severity typically requires immediate (e.g., 5 min) response?
+
+A) SEV4
+B) SEV3
+C) SEV2
+D) SEV1
+
+<!-- ANSWER: D -->
+<!-- EXPLANATION: SEV1 is critical—full outage, data loss, security breach. Response time is immediate (e.g., 5 min). SEV2 is major (15–30 min); SEV3/4 are slower. -->
+
+## Question 4
+
+What should the external status update include?
+
+A) Technical details of the failure
+B) We're aware of X, impact is Y, ETA is Z (or "investigating")
+C) Names of engineers working on it
+D) Root cause analysis
+
+<!-- ANSWER: B -->
+<!-- EXPLANATION: External updates should be clear and customer-friendly: we're aware, here's the impact, here's our ETA (or that we're investigating). No technical jargon, no blame, no internal details. -->
+
+## Question 5
+
+The "5 whys" in a postmortem are used to:
+
+A) Assign blame
+B) Trace from symptom to systemic/root cause
+C) List five people involved
+D) Count how many things went wrong
+
+<!-- ANSWER: B -->
+<!-- EXPLANATION: 5 whys is a technique to dig from the surface symptom to deeper causes. "Why did X happen?" → "Because Y." "Why did Y happen?" → Repeat. You reach contributing factors in process and design, not just the immediate trigger. -->
+
+## Question 6
+
+To prevent incident fatigue, a good practice is:
+
+A) Have one person always on-call
+B) Limit consecutive weeks on primary; offer post-incident rest
+C) Only page during business hours
+D) Skip postmortems to save time
+
+<!-- ANSWER: B -->
+<!-- EXPLANATION: Limit consecutive primary weeks, rotate fairly, give IC rest after SEV1/2 (no meetings for a few hours). One person always on-call and skipping rest leads to burnout. -->
+
+## Question 7
+
+<!-- VISUAL: quiz-drag-order -->
+
+Put these incident lifecycle phases in the correct order:
+
+A) Mitigation (contain and fix)
+B) Detection and triage
+C) Post-incident review
+D) Recovery (restore service)
+E) Preparation (runbooks, on-call)
+
+<!-- ANSWER: E,B,A,D,C -->
+<!-- EXPLANATION: Prepare first (runbooks, on-call). When an incident occurs, detect and triage, then mitigate (contain and fix), recover (restore service), and finally conduct a post-incident review to learn. -->
+
+## Question 8
+
+<!-- VISUAL: quiz-drag-order -->
+
+Put these postmortem sections in a logical order for the document:
+
+A) Timeline of events
+B) Root cause analysis
+C) Impact and severity
+D) Action items
+E) Summary
+
+<!-- ANSWER: E,C,A,B,D -->
+<!-- EXPLANATION: A postmortem typically opens with a summary, then impact/severity, a chronological timeline, root cause analysis (5 whys, etc.), and closes with action items to prevent recurrence. -->

package/modules/incident-response/resources.md

@@ -0,0 +1,40 @@
+# Incident Response — Resources
+
+## Official Guides
+
+- [Google SRE Book](https://sre.google/sre-book/) — Free online. Covers incident response, postmortems, SLOs, and error budgets. Essential reading.
+- [Google SRE Book — Managing Incidents](https://sre.google/sre-book/managing-incidents/) — Direct chapter on incident management.
+- [PagerDuty Incident Response Guide](https://response.pagerduty.com) — Roles, workflows, communication, and best practices.
+- [PagerDuty — Postmortem Best Practices](https://response.pagerduty.com/before/incident-response/postmortem/) — Blameless postmortems.
+
+## Articles
+
+- [Blameless PostMortems and a Just Culture](https://codeascraft.com/2013/11/14/blameless-postmortems-and-a-just-culture/) — Etsy (Code as Craft). Why blameless matters and how to build the culture.
+- [Writing an Incident Postmortem](https://www.atlassian.com/incident-management/postmortem) — Atlassian. Template and examples.
+- [The Five Whys](https://www.atlassian.com/incident-management/postmortem/blameless) — Using 5 whys in postmortems.
+- [Incident Communication Best Practices](https://www.pagerduty.com/blog/incident-communication-best-practices/) — PagerDuty. Status updates and stakeholder communication.
+- [On-Call Best Practices](https://www.pagerduty.com/resources/learn/on-call-best-practices/) — PagerDuty. Rotation, runbooks, fatigue prevention.
+
+## Books
+
+- **Site Reliability Engineering** (O'Reilly) — Google SRE book in print. Covers incidents, SLOs, and more.
+- **The Phoenix Project** by Gene Kim et al. — Novel about IT ops and incident response; illustrates principles.
+- **An Elegant Puzzle** by Will Larson — Includes on-call, incident process, and team reliability.
+
+## Tools
+
+- [PagerDuty](https://www.pagerduty.com/) — Incident alerting, on-call scheduling, escalation.
+- [OpsGenie](https://www.atlassian.com/software/opsgenie) — Alerting and on-call management.
+- [Statuspage](https://www.atlassian.com/software/statuspage) — Public status pages.
+- [Rootly](https://www.rootly.com/) — Incident management and runbooks.
+- [Incident.io](https://incident.io/) — Incident response workflow and automation.
+
+## Videos
+
+- [Google SRE — How We Do It](https://www.youtube.com/results?search_query=google+sre+incident) — Search for talks on incident response.
+- [Blameless Postmortems at Etsy](https://www.youtube.com/results?search_query=blameless+postmortem+etsy) — Culture and process.
+
+## Podcasts
+
+- [Engineering Culture by InfoQ](https://www.infoq.com/podcasts/engineering-culture/) — Episodes on reliability and incidents.
+- [SRE Weekly](https://sreweekly.com/) — Newsletter; often covers incident and postmortem content.

package/modules/incident-response/walkthrough.md

@@ -0,0 +1,82 @@
+# Incident Response Walkthrough — Learn by Doing
+
+## Before We Begin
+
+Incidents follow a lifecycle: detect, triage, communicate, mitigate, resolve, and learn. How you handle each phase—and who does what—determines whether you fix things quickly and improve, or repeat the same mistakes.
+
+**Diagnostic question:** Have you (or your team) ever had something break in production? What went well? What was chaotic—alerts, communication, or knowing who owned what?
+
+**Checkpoint:** You can name at least two phases of the incident lifecycle and one thing that often goes wrong.
+
+---
+
+## Step 1: Define Severity Levels
+
+<!-- hint:diagram mermaid-type="flowchart" topic="Incident lifecycle from detect to postmortem" -->
+<!-- hint:list style="cards" -->
+
+**Task:** For your system (or a hypothetical one), define SEV1–4 with concrete examples. Include: impact description, example scenario, and target response time for each.
+
+**Question:** How would someone know in the moment which severity applies? What's ambiguous?
+
+**Checkpoint:** Each severity has a clear example and response time.
+
+---
+
+## Step 2: Draft a Communication Template
+
+**Task:** Write two templates: (1) internal incident announcement (Slack/email) and (2) external status page update. Include placeholders for severity, impact, status, and ETA.
+
+**Question:** What do stakeholders need to know immediately vs. what can wait? How often should you update?
+
+**Checkpoint:** Templates are copy-pastable with clear placeholders.
+
+---
+
+## Step 3: Outline Incident Roles
+
+**Task:** Define the roles for your team's incident response: Incident Commander, Comms Lead, Responders. For each, write 2–3 responsibilities. Who would fill each role in a real incident?
+
+**Question:** What happens if the usual IC is unavailable? How do you rotate?
+
+**Checkpoint:** Roles are defined; you know who does what and who backs up whom.
+
+---
+
+## Step 4: Write a Postmortem Template
+
+**Task:** Create a blameless postmortem template. Include: Summary, Timeline, Root Cause (5 whys), Impact, Action Items (with owners), and Lessons Learned. Add 1–2 "blameless language" guidelines (how to describe what happened without naming individuals negatively).
+
+**Question:** How do you make it safe for people to contribute honestly? What would discourage participation?
+
+**Checkpoint:** Template is complete; language guidelines support blameless culture.
+
+---
+
+## Step 5: Map the Incident Lifecycle
+
+**Task:** Draw or describe your team's incident flow from Detect → Resolve → Postmortem. Include: how alerts fire, who gets paged, how triage happens, where updates are posted, and when postmortem is scheduled.
+
+**Question:** Where are the gaps? What would fail in a real 3 a.m. incident?
+
+**Checkpoint:** Flow is documented; at least one gap is identified.
+
+---
+
+## Step 6: Design an On-Call Runbook Entry
+
+**Task:** Pick one realistic incident type (e.g., "High error rate on API"). Write a runbook entry: symptoms, how to confirm, steps to mitigate, who to escalate to, and where to document.
+
+**Question:** Could someone unfamiliar with the system follow this at 2 a.m.? What's missing?
+
+**Checkpoint:** Runbook is actionable; an on-call engineer could follow it.
+
+---
+
+## Step 7: Plan a Post-Incident Review
+
+**Task:** Schedule a blameless postmortem for a past incident (real or hypothetical). Write the agenda: timebox per section, who facilitates, how action items are tracked. Include one "retrospective" question: "What would we do differently in the response itself?"
+
+**Question:** How do you avoid making the postmortem feel punitive? How do you ensure action items get done?
+
+**Checkpoint:** Agenda is timeboxed; facilitation and follow-through are planned.