@brunosps00/dev-workflow 0.13.0 → 0.15.0

package/scaffold/skills/dw-incident-response/references/communication-templates.md

@@ -0,0 +1,107 @@
+# Communication templates
+
+Two messages get drafted during Phase 4: an initial notification (sent during the incident) and a resolution notification (sent at all-clear). Both go through the same channels — status page, support team, internal Slack/Teams, customer email if applicable.
+
+## Initial notification
+
+Sent as soon as the incident is acknowledged. Update every 30 min for SEV-1/2 until resolved.
+
+```markdown
+## Incident: <Title>
+
+**Severity:** SEV-<X>
+**Status:** Investigating | Mitigated | Resolved
+**Impact:** <What users experience — be specific>
+**Started:** YYYY-MM-DD HH:MM UTC
+**Next update:** HH:MM UTC
+
+We are aware of <symptom> affecting <surface>. The team is actively investigating.
+<Optional: known workaround if any.>
+
+We will post the next update by HH:MM UTC.
+```
+
+### Update cadence
+
+- **SEV-1:** every 15 min until mitigated, then every 30 min until resolved.
+- **SEV-2:** every 30 min throughout.
+- **SEV-3:** every 2 hours.
+- **SEV-4:** initial + resolution; no interim updates.
+
+### Status progression
+
+The `Status` field moves through three values:
+1. **Investigating** — we know it's happening; cause unknown.
+2. **Mitigated** — symptoms are reduced or contained; root cause may still be unresolved.
+3. **Resolved** — symptoms gone, monitoring shows baseline, no follow-up expected.
+
+### Tone
+
+- Direct. No "we're experiencing a slight disruption."
+- Specific. Say which feature; don't generalize.
+- Honest. If you don't know the cause, say so.
+- No blame. Don't mention which team owns the area in public-facing comms.
+
+## Resolution notification
+
+Sent when Phase 3 confirms recovery.
+
+```markdown
+## Resolved: <Title>
+
+**Duration:** Xh Ym (from HH:MM to HH:MM UTC)
+**Root cause:** <1-2 sentences, technical but accessible>
+**Fix applied:** <what was done — link to deploy / PR if public-facing isn't a concern>
+**Postmortem:** Will be published within 48 hours at <link>
+
+We apologize for the disruption. Customers affected by <specific issue> can <specific recovery action if applicable>.
+
+Questions: <support@example.com> or <#incident-channel>.
+```
+
+### What to include
+
+- Specific duration in minutes/hours.
+- Root cause that's accurate but readable by non-engineers.
+- Concrete remedy (refund window, retry-now action, etc.) if customers need to do something.
+- Postmortem link or commitment to publish.
+
+### What NOT to include
+
+- Names of individuals.
+- Detailed stack traces or internal tool names.
+- Speculation on prevention before the postmortem is done.
+- Apologies that promise "this will never happen again" — incidents do recur; promise to learn.
+
+## Internal-only versions
+
+For internal tools without external users, drop the apology section and add:
+
+```markdown
+## Internal incident — <Title>
+
+**Severity:** SEV-X
+**Affected team(s):** <list>
+**Duration:** Xh Ym
+**Cause:** <technical>
+**Fix:** <commit/deploy>
+**Action items:** see postmortem at <link>
+```
+
+Shorter, more technical, no need to be customer-friendly.
+
+## Status page updates
+
+If using a status page (Statuspage, Cachet, Better Uptime, etc.):
+
+- Component-level granularity: mark which specific service is degraded, not "platform down."
+- Match the severity to the visual indicator (degraded vs partial outage vs major outage).
+- Close the incident on the status page only after the resolution notification confirms recovery.
+
+## When the communication discipline bends
+
+- **Internal-only tools:** initial + resolution only; no 30-min updates needed.
+- **Security incidents:** legal/compliance may need to approve every external message. Build that handoff into Phase 4.
+- **Customer-specific incidents** (affects 1-2 enterprise customers, not the whole user base): direct contact to those customers replaces public status page.
+
+Document the deviation in `04-communication.md`.

package/scaffold/skills/dw-incident-response/references/postmortem-template.md

@@ -0,0 +1,133 @@
+# Postmortem template — blameless, action-oriented
+
+Use this template for the Phase 5 output (`05-postmortem.md`). Every field is mandatory unless explicitly marked optional.
+
+The discipline behind the template lives in `blameless-discipline.md` — read both.
+
+## Template
+
+```markdown
+# Postmortem: <Incident title>
+
+**Date:** YYYY-MM-DD
+**Duration:** Xh Ym (from detection to all-clear)
+**Severity:** SEV-X
+**Authors:** @name1, @name2
+**Status:** Draft | Reviewed | Published
+
+## Summary
+
+[2-3 sentences. What happened? What was the user-visible impact? How was it resolved?
+No blame, no causes yet — just the observable narrative.]
+
+## Timeline
+
+All times UTC. Aim for ±5 min granularity; tighter for SEV-1/2.
+
+| Time (UTC) | Event |
+|------------|-------|
+| HH:MM | First alert fired (or first user report) |
+| HH:MM | On-call acknowledged |
+| HH:MM | War room opened / incident channel created |
+| HH:MM | Initial triage complete; severity confirmed |
+| HH:MM | Hypothesis: <X> |
+| HH:MM | Hypothesis rejected — observed <Y> |
+| HH:MM | Root cause identified |
+| HH:MM | Fix applied: <commit SHA / deploy>
+| HH:MM | Verified recovery (health checks green, error rate baseline)
+| HH:MM | All-clear confirmed; communications sent |
+
+## Root cause
+
+[Technical explanation. Walk through the chain:
+1. What was the proximate cause? (the immediate trigger)
+2. What was the underlying assumption that broke?
+3. Why didn't existing safeguards catch it?
+
+No blame. No "X did Y wrong." Frame as: "the system allowed Z because A was assumed."]
+
+## Impact
+
+- **Users affected:** N (counting method: <how we estimated>)
+- **Geography:** [if applicable]
+- **Revenue impact:** $X estimated (calculation: <method>)
+- **Error budget consumed:** X% of monthly SLO
+- **Data impact:** [data lost / corrupted? recoverable?]
+
+## Detection
+
+- **How was it detected?** [alert / user report / scheduled check]
+- **Time to detection (TTD):** X minutes
+- **Was the detection signal adequate?** [yes/no — and what would improve it]
+
+## What went well
+
+Concrete things to keep:
+- [item — specific, not "we worked as a team"]
+- [item]
+- [item]
+
+## What went wrong
+
+Process / tooling / assumptions that failed:
+- [item — specific failure mode, no blame]
+- [item]
+- [item]
+
+## Where we got lucky
+
+[Optional. What worked by accident that we shouldn't rely on next time?
+e.g., "The bug only fired during business hours because of <X>; could have been overnight and worse."]
+
+## Action items
+
+Quality bar: each item has **owner**, **due date**, and **measurable outcome**. "Improve monitoring" does NOT count. Read `blameless-discipline.md` for the bar.
+
+| Action | Owner | Due date | Priority | Tracking |
+|--------|-------|----------|----------|----------|
+| Add Datadog SLO alert at p99 > 800ms with PagerDuty routing | @bruno | 2026-06-01 | P1 | Linear ENG-1234 |
+| Add idempotency key to /api/orders POST | @maria | 2026-05-25 | P1 | GitHub #5678 |
+| Update runbook with new mitigation steps | @bruno | 2026-05-19 | P2 | Linear ENG-1235 |
+| Constitution principle: every state-changing endpoint requires audit log | @team | 2026-06-15 | P3 | ADR-042 |
+
+## Constitution implications
+
+[Did this incident reveal a principle the team should commit to? If yes, an ADR
+should be opened via `/dw-adr`. Examples:
+- "Every write to billing tables requires audit log entry"
+- "Database migrations must run in a separate maintenance window"
+- "External API calls must have circuit breakers"
+
+Link the ADR slug here.]
+
+## Cross-incident pattern
+
+[Have we seen this category of incident before? If yes, list the prior incidents.
+3+ similar incidents = structural problem; needs design review, not just an action item.]
+
+## References
+
+- Incident channel: [Slack link]
+- Diagnostic dashboards: [Grafana / Datadog links]
+- Affected commits: [SHA range]
+- Related ADRs: [list]
+- Related runbook: `<path>` (updated in this incident if applicable)
+```
+
+## Quality checklist before publishing
+
+- [ ] Summary is 2-3 sentences — not a wall of text.
+- [ ] Timeline events have specific times (not "around lunch time").
+- [ ] Root cause section explains the ASSUMPTION, not just the trigger.
+- [ ] Every action item has owner + due date + measurable outcome.
+- [ ] Action items are filed in the actual tracker (Linear/Jira/GitHub) with links.
+- [ ] No names appear in "what went wrong" — only systems, processes, and assumptions.
+- [ ] Cross-incident pattern section reviewed against `.dw/incidents/` for prior occurrences.
+- [ ] If a constitution principle emerged, ADR opened via `/dw-adr` and linked.
+
+## Publishing
+
+- Postmortems for SEV-1/2 are shared with stakeholders within 48 hours.
+- Postmortems for SEV-3 are reviewed in the next team retro.
+- Action items track in the project's normal tooling — not in the postmortem file.
+- The postmortem file in `.dw/incidents/` is the canonical write-up; everything else (Slack threads, status pages) eventually points back here.

package/scaffold/skills/dw-incident-response/references/runbook-templates.md

@@ -0,0 +1,169 @@
+# Runbook templates + on-call handoff
+
+Two contexts where this reference applies:
+1. **Generating a runbook** (entry-mode 3 in the skill — no live incident, just producing operational docs).
+2. **On-call handoff** at the end of a shift.
+
+## Runbook: service outage template
+
+```markdown
+## Runbook — <Service Name> outage
+
+### Quick diagnosis (< 5 min)
+
+1. Health check: `curl -sf https://<service>/health | jq .`
+2. Pod status (K8s): `kubectl get pods -l app=<service>`
+3. Recent deploys: `kubectl rollout history deployment/<service>`
+4. Recent logs: `kubectl logs -l app=<service> --tail=50 --since=5m`
+5. Metrics dashboard: <Grafana / Datadog link>
+
+### Common failure modes
+
+| Symptom | Likely cause | Fix |
+|---------|--------------|-----|
+| OOMKilled | Memory leak or under-sized pod | Scale up replicas; increase memory limit; investigate leak |
+| CrashLoopBackOff | Config error, missing env var | Check logs for startup errors; verify ConfigMap/Secret |
+| ImagePullBackOff | Bad image tag or registry auth | Verify tag exists; check registry credentials |
+| 503s from healthy pods | Downstream dependency down | Check the dependency (DB, cache, queue) |
+| Slow responses (high p99) | DB connection saturation; N+1 query; cold cache | Check DB connections; tail slow query log; check cache hit rate |
+| Latency spike after deploy | New code path slower than expected | Roll back; profile the new code path |
+
+### Rollback procedure
+
+```bash
+# K8s
+kubectl rollout undo deployment/<service>
+
+# Confirm
+kubectl rollout status deployment/<service>
+```
+
+### Escalation chain
+
+- **L1 (on-call engineer):** <name / PagerDuty schedule>
+- **L2 (team lead):** <name>
+- **L3 (infrastructure / platform team):** <team channel>
+- **L4 (CTO / VP Eng):** <name> (SEV-1 only)
+
+### Known dependencies
+
+- Database: <DB name + version>
+- Cache: <Redis / Memcached>
+- Queue: <SQS / RabbitMQ / Kafka>
+- External APIs: <list with SLAs>
+
+If a dependency is down, the service may degrade gracefully — see `<degraded-mode>` section in the architecture doc.
+
+### Related runbooks
+
+- `<dependency-A>` runbook
+- `<related-service-B>` runbook
+```
+
+## Runbook: database incident template
+
+```markdown
+## Runbook — Database (<engine, version>) incident
+
+### Quick diagnosis (Postgres examples; adapt for MySQL/etc.)
+
+```sql
+-- Active connections vs baseline
+SELECT count(*) FROM pg_stat_activity WHERE state = 'active';
+-- Expected baseline: <N>; alert if > <2N>
+
+-- Long-running queries
+SELECT pid, now() - query_start AS duration, query
+FROM pg_stat_activity
+WHERE state = 'active' AND now() - query_start > interval '30 seconds'
+ORDER BY duration DESC;
+
+-- Lock waits
+SELECT * FROM pg_locks WHERE NOT granted;
+
+-- Replication lag (if applicable)
+SELECT * FROM pg_stat_replication;
+```
+
+### Common DB failure modes
+
+| Symptom | Likely cause | Mitigation |
+|---------|--------------|------------|
+| Connection pool exhaustion | App leak; spike in traffic | Restart app pods to release connections; scale app; investigate leak |
+| Lock contention | Long-running transaction blocking writers | Identify holder via pg_stat_activity; terminate if safe (`pg_terminate_backend`) |
+| Disk full | Unbounded growth; WAL retention | Free space; review retention; vacuum |
+| Replication lag | Network or replica overload | Check replica health; review primary write rate |
+| Slow query (p99 spike) | Missing index; bad plan after stats change | `EXPLAIN ANALYZE` the slow query; check `pg_stat_user_indexes` |
+
+### Emergency procedures
+
+- **Terminate a stuck query:** `SELECT pg_terminate_backend(<pid>);` — only after confirming it's safe (it won't roll back distributed transactions cleanly).
+- **Failover to replica:** `<runbook-specific commands>`.
+- **Restore from snapshot:** see `<DR runbook>` — only for data corruption / loss.
+
+### Escalation
+
+- **DB on-call:** <name / schedule>
+- **DBA team:** <channel>
+- **Vendor support:** <ticket portal> (RDS, CloudSQL, etc.)
+```
+
+## On-call handoff template
+
+End of every shift. Sent in the team's on-call channel + emailed to the incoming engineer.
+
+```markdown
+## On-call handoff — <Date>
+
+**Outgoing:** @<name>
+**Incoming:** @<name>
+
+### Active incidents
+
+- [None] OR list with status, severity, channel link
+
+### Ongoing investigations (carried over)
+
+- **<service/area>:** <one-line status, what's been tried, next step>
+- **<issue>:** <what's blocking>
+
+### Recent changes (last 24h)
+
+- **<service>:** deploy at HH:MM UTC — <commit/PR>
+- **<config>:** change at HH:MM — <what>
+- **<infra>:** change at HH:MM — <what>
+
+### Known issues (workarounds active)
+
+- **<symptom>:** workaround: <action>. Tracking: <issue link>.
+
+### Upcoming events
+
+- **<date HH:MM UTC>:** scheduled maintenance — <service> — owner <name>
+- **<date>:** traffic spike expected (launch, marketing campaign, etc.)
+
+### Notes for the new shift
+
+- [free-form: anything the incoming engineer should know that didn't fit above]
+```
+
+## Generating runbooks proactively
+
+When the skill is invoked in entry-mode 3 (runbook generation, no live incident):
+
+1. Ask: "Which service / area is the runbook for?"
+2. Look up the service in `.dw/intel/` — pull the stack, dependencies, recent deploys.
+3. Use the appropriate template above (service outage / DB / on-call handoff).
+4. Fill in concrete commands, dashboard links, on-call info.
+5. Save to `.dw/runbooks/<service-name>.md`.
+6. Suggest adding the runbook path to the relevant `.dw/rules/<service>.md`.
+
+Runbooks should be **executable** — every command listed must work without modification when copy-pasted by a tired engineer at 3am.
+
+## Maintenance cadence
+
+- Runbooks reviewed quarterly.
+- Updated immediately after any incident where the runbook was wrong or missing.
+- Owner per runbook = the team that owns the service.
+
+A stale runbook is worse than no runbook (false confidence). If you can't keep one current, delete it and rely on the incident-response workflow alone.

package/scaffold/skills/dw-incident-response/references/severity-and-triage.md

@@ -0,0 +1,186 @@
+# Severity classification + triage commands
+
+## Severity criteria — extended version
+
+### SEV-1 (Critical)
+
+**Trigger:** any of
+- Production service fully down or returning >50% errors.
+- Data loss in progress or already occurred.
+- Security breach detected (active credential leak, RCE, unauthorized data access).
+- Compliance violation (PCI, HIPAA, GDPR exposure).
+
+**Response:** page on-call immediately. CEO/CTO notified within 30 min if customer-facing. War room opened.
+
+### SEV-2 (Major)
+
+**Trigger:** any of
+- Significant feature degradation affecting >25% of users.
+- Latency 10× normal or higher.
+- Background jobs queue backing up beyond SLA.
+- Partial outage of a critical dependency (auth, payments, search).
+
+**Response:** on-call investigates within 30 min. Stakeholders notified within 1 hour.
+
+### SEV-3 (Minor)
+
+**Trigger:** any of
+- Single endpoint returning errors with a workaround available.
+- Performance regression affecting <25% of users.
+- Non-critical feature broken.
+
+**Response:** investigated within 4 hours. Communicated in standup/Slack channel.
+
+### SEV-4 (Low)
+
+**Trigger:** any of
+- Cosmetic issue.
+- Non-user-facing bug.
+- Minor UX papercut.
+
+**Response:** logged as a normal bug. Fixed in next routine deploy.
+
+## Blast radius assessment
+
+Before declaring severity, answer:
+1. **Which services** are affected?
+2. **How many users** are seeing the issue? Estimate from error rate × DAU.
+3. **What's the revenue impact** per hour (if customer-facing)?
+4. **What downstream systems** depend on the affected service?
+5. **Is the issue spreading** or contained?
+
+Document each in `01-triage.md`. The numbers go into the postmortem.
+
+## Triage commands by stack
+
+### Kubernetes
+
+```bash
+# Find non-running pods
+kubectl get pods -A | grep -v -E 'Running|Completed'
+
+# Pods consuming most memory
+kubectl top pods --sort-by=memory --all-namespaces | head -20
+
+# Recent logs from a specific service
+kubectl logs -l app=<service> --tail=100 --since=10m
+
+# Recent deploys
+kubectl rollout history deployment/<service>
+
+# Rollback the most recent deploy
+kubectl rollout undo deployment/<service>
+```
+
+### Docker / Docker Compose
+
+```bash
+# Containers that exited
+docker ps -a --filter "status=exited" --format "table {{.Names}}\t{{.Status}}"
+
+# Recent container logs
+docker logs <container> --tail=100 --since=10m
+
+# Container resource usage
+docker stats --no-stream
+
+# Recent compose deploys (assumes versioned compose files)
+git log --oneline --since="24 hours ago" -- docker-compose.yml
+```
+
+### Generic (any service with HTTP health endpoint)
+
+```bash
+# Health check
+curl -sf https://<host>/health | jq .
+
+# Compare healthy vs unhealthy timing
+time curl -sf https://<host>/health
+time curl -sf https://<host>/api/<critical-endpoint>
+
+# Recent application deploys (any git-tracked deployment)
+git log --oneline --since="2 hours ago"
+
+# Recent infra changes
+git -C /path/to/infra log --oneline --since="24 hours ago"
+```
+
+### Database (Postgres)
+
+```sql
+-- Active connections (compare against normal baseline)
+SELECT count(*) FROM pg_stat_activity WHERE state = 'active';
+
+-- Long-running queries (>30s)
+SELECT pid, now() - query_start AS duration, query, state
+FROM pg_stat_activity
+WHERE state = 'active' AND now() - query_start > interval '30 seconds'
+ORDER BY duration DESC;
+
+-- Lock contention
+SELECT blocked_locks.pid AS blocked_pid,
+       blocking_locks.pid AS blocking_pid,
+       blocked_activity.query AS blocked_query,
+       blocking_activity.query AS blocking_query
+FROM pg_catalog.pg_locks blocked_locks
+JOIN pg_catalog.pg_stat_activity blocked_activity ON blocked_activity.pid = blocked_locks.pid
+JOIN pg_catalog.pg_locks blocking_locks ON blocking_locks.locktype = blocked_locks.locktype
+  AND blocking_locks.database IS NOT DISTINCT FROM blocked_locks.database
+  AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation
+  AND blocking_locks.pid != blocked_locks.pid
+JOIN pg_catalog.pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid
+WHERE NOT blocked_locks.granted;
+
+-- Terminate a stuck query (last resort; investigate first)
+SELECT pg_terminate_backend(<pid>);
+```
+
+### Generic application metrics
+
+If your APM provides these (Datadog, New Relic, Sentry):
+- Error rate per endpoint over the last hour vs the prior week's baseline.
+- p50/p95/p99 latency per endpoint.
+- Throughput (requests per second).
+- Saturation (queue depth, connection pool usage).
+
+The Google SRE "Four Golden Signals" (latency, errors, traffic, saturation) cover most cases.
+
+## Immediate mitigation patterns
+
+Before debugging root cause, can you reduce blast radius now? Try in order:
+
+1. **Rollback the most recent deploy** if timing matches.
+2. **Toggle feature flag** for the affected feature.
+3. **Redirect traffic** away from the unhealthy instance / region.
+4. **Rate-limit** the abusive client or queue depth.
+5. **Scale up** if it's resource starvation (CPU, memory, connection pool).
+
+If none apply, escalate to investigation phase. Don't burn time forcing mitigation that doesn't fit.
+
+## What to record in `01-triage.md`
+
+```markdown
+# Triage — <incident title>
+
+## Detected
+- **Time:** YYYY-MM-DD HH:MM UTC
+- **Detected by:** alert / user report / monitoring dashboard / customer support
+- **Severity:** SEV-X
+
+## Symptoms
+- [observable behavior]
+- [error rates, latency, etc.]
+
+## Blast radius
+- Services: [list]
+- Users affected: [estimate]
+- Revenue impact: [if known]
+- Downstream impact: [systems that depend on this]
+
+## Initial mitigation
+- [what was tried]
+- [what worked / didn't]
+
+## Next steps
+Proceeding to Phase 2 (investigation) after user confirmation.
+```