aigroup-workflow 1.0.0

package/skills/jarvis/engineering-team/engineering-team/incident-commander/references/sla-management-guide.md

@@ -0,0 +1,566 @@
+# SLA Management Guide
+
+> Comprehensive reference for Service Level Agreements, Objectives, and Indicators.
+> Designed for incident commanders who must understand, protect, and communicate SLA status during and after incidents.
+
+---
+
+## 1. Definitions & Relationships
+
+### Service Level Indicator (SLI)
+
+An SLI is the quantitative measurement of a specific aspect of service quality. SLIs are the raw data that feed everything above them. They must be precisely defined, automatically collected, and unambiguous.
+
+**Common SLI types by service:**
+
+| Service Type | SLI | Measurement Method |
+|---|---|---|
+| Web Application | Request latency (p50, p95, p99) | Server-side histogram |
+| Web Application | Availability (successful responses / total requests) | Load balancer logs |
+| REST API | Error rate (5xx responses / total responses) | API gateway metrics |
+| REST API | Throughput (requests per second) | Counter metric |
+| Database | Query latency (p99) | Slow query log + APM |
+| Database | Replication lag (seconds) | Replica monitoring |
+| Message Queue | End-to-end delivery latency | Timestamp comparison |
+| Message Queue | Message loss rate | Producer vs consumer counts |
+| Storage | Durability (objects lost / objects stored) | Integrity checksums |
+| CDN | Cache hit ratio | Edge server logs |
+
+**SLI specification formula:**
+
+```
+SLI = (good events / total events) x 100
+```
+
+For availability: `SLI = (successful requests / total requests) x 100`
+For latency: `SLI = (requests faster than threshold / total requests) x 100`
+
+### Service Level Objective (SLO)
+
+An SLO is the target value or range for an SLI. It defines the acceptable level of reliability. SLOs are internal goals that engineering teams commit to.
+
+**Setting meaningful SLOs:**
+
+1. Measure the current baseline over 30 days minimum
+2. Subtract a safety margin (typically 0.05%-0.1% below actual performance)
+3. Validate against user expectations and business requirements
+4. Never set an SLO higher than what the system can sustain without heroics
+
+**Common pitfall:** Setting 99.99% availability when 99.9% meets every user need. The jump from 99.9% to 99.99% is a 10x reduction in allowed downtime and typically requires 3-5x the engineering investment.
+
+**SLO examples:**
+
+- `99.9% of HTTP requests return a non-5xx response within each calendar month`
+- `95% of API requests complete in under 200ms (p95 latency)`
+- `99.95% of messages are delivered within 30 seconds of production`
+
+### Service Level Agreement (SLA)
+
+An SLA is a formal contract between a service provider and its customers that specifies consequences for failing to meet defined service levels. SLAs must always be looser than SLOs to provide a buffer zone.
+
+**Rule of thumb:** If your SLO is 99.95%, your SLA should be 99.9% or lower. The gap between SLO and SLA is your safety margin.
+
+### The Hierarchy
+
+```
+  SLA (99.9%)     ← Contract with customers, financial penalties
+    ↑ backs
+  SLO (99.95%)    ← Internal target, triggers error budget policy
+    ↑ targets
+  SLI (measured)  ← Raw metric: actual uptime = 99.97% this month
+```
+
+**Standard combinations by tier:**
+
+| Tier | SLI (Metric) | SLO (Target) | SLA (Contract) | Allowed Downtime/Month |
+|---|---|---|---|---|
+| Critical (payments) | Availability | 99.99% | 99.95% | SLO: 4.38 min / SLA: 21.9 min |
+| High (core API) | Availability | 99.95% | 99.9% | SLO: 21.9 min / SLA: 43.8 min |
+| Standard (dashboard) | Availability | 99.9% | 99.5% | SLO: 43.8 min / SLA: 3.65 hrs |
+| Low (internal tools) | Availability | 99.5% | 99.0% | SLO: 3.65 hrs / SLA: 7.3 hrs |
+
+---
+
+## 2. Error Budget Policy
+
+### What Is an Error Budget
+
+An error budget is the maximum amount of unreliability a service can have within a given period while still meeting its SLO. It is calculated as:
+
+```
+Error Budget = 1 - SLO target
+```
+
+For a 99.9% SLO over a 30-day month (43,200 minutes):
+
+```
+Error Budget = 1 - 0.999 = 0.001 = 0.1%
+Allowed Downtime = 43,200 x 0.001 = 43.2 minutes
+```
+
+### Downtime Allowances by SLO
+
+| SLO | Error Budget | Monthly Downtime | Quarterly Downtime | Annual Downtime |
+|---|---|---|---|---|
+| 99.0% | 1.0% | 7 hrs 18 min | 21 hrs 54 min | 3 days 15 hrs |
+| 99.5% | 0.5% | 3 hrs 39 min | 10 hrs 57 min | 1 day 19 hrs |
+| 99.9% | 0.1% | 43.8 min | 2 hrs 11 min | 8 hrs 46 min |
+| 99.95% | 0.05% | 21.9 min | 1 hr 6 min | 4 hrs 23 min |
+| 99.99% | 0.01% | 4.38 min | 13.1 min | 52.6 min |
+| 99.999% | 0.001% | 26.3 sec | 78.9 sec | 5.26 min |
+
+### Error Budget Consumption Tracking
+
+Track budget consumption as a percentage of the total budget used so far in the current window:
+
+```
+Budget Consumed (%) = (actual bad minutes / allowed bad minutes) x 100
+```
+
+Example: SLO is 99.9% (43.8 min budget/month). On day 10, you have had 15 minutes of downtime.
+
+```
+Budget Consumed = (15 / 43.8) x 100 = 34.2%
+Expected consumption at day 10 = (10/30) x 100 = 33.3%
+Status: Slightly over pace (34.2% consumed at 33.3% of month elapsed)
+```
+
+### Burn Rate
+
+Burn rate measures how fast the error budget is being consumed relative to the steady-state rate:
+
+```
+Burn Rate = (error rate observed / error rate allowed by SLO)
+```
+
+A burn rate of 1.0 means the budget will be exactly exhausted by the end of the window. A burn rate of 10 means the budget will be exhausted in 1/10th of the window.
+
+**Burn rate to time-to-exhaustion (30-day month):**
+
+| Burn Rate | Budget Exhausted In | Urgency |
+|---|---|---|
+| 1x | 30 days | On pace, monitoring only |
+| 2x | 15 days | Elevated attention |
+| 6x | 5 days | Active investigation required |
+| 14.4x | 2.08 days (~50 hours) | Immediate page |
+| 36x | 20 hours | Critical, all-hands |
+| 720x | 1 hour | Total outage scenario |
+
+### Error Budget Exhaustion Policy
+
+When the error budget is consumed, the following actions trigger based on threshold:
+
+**Tier 1 - Budget at 75% consumed (Yellow):**
+- Notify service team lead via automated alert
+- Freeze non-critical deployments to the affected service
+- Conduct pre-emptive review of upcoming changes for risk
+- Increase monitoring sensitivity (lower alert thresholds)
+
+**Tier 2 - Budget at 100% consumed (Orange):**
+- Hard feature freeze on the affected service
+- Mandatory reliability sprint: all engineering effort redirected to reliability
+- Daily status updates to engineering leadership
+- Postmortem required for the incidents that consumed the budget
+- Freeze lasts until budget replenishes to 50% or systemic fixes are verified
+
+**Tier 3 - Budget at 150% consumed / SLA breach imminent (Red):**
+- Escalation to VP Engineering and CTO
+- Cross-team war room if dependencies are involved
+- Customer communication prepared and staged
+- Legal and finance teams briefed on potential SLA credit obligations
+- Recovery plan with specific milestones required within 24 hours
+
+### Error Budget Policy Template
+
+```
+SERVICE: [service-name]
+SLO: [target]% availability over [rolling 30-day / calendar month] window
+ERROR BUDGET: [calculated] minutes per window
+
+BUDGET THRESHOLDS:
+  - 50% consumed: Team notification, increased vigilance
+  - 75% consumed: Feature freeze for this service, reliability focus
+  - 100% consumed: Full feature freeze, reliability sprint mandatory
+  - SLA threshold crossed: Executive escalation, customer communication
+
+REVIEW CADENCE: Monthly budget review on [day], quarterly SLO adjustment
+
+EXCEPTIONS: Planned maintenance windows excluded if communicated 72+ hours in advance
+            and within agreed maintenance allowance.
+
+APPROVED BY: [Engineering Lead] / [Product Lead] / [Date]
+```
+
+---
+
+## 3. SLA Breach Handling
+
+### Detection Methods
+
+**Automated detection (primary):**
+- Real-time monitoring dashboards with SLA burn-rate alerts
+- Automated SLA compliance calculations running every 5 minutes
+- Threshold-based alerts when cumulative downtime approaches SLA limits
+- Synthetic monitoring (external probes) for customer-perspective validation
+
+**Manual review (secondary):**
+- Monthly SLA compliance reports generated on the 1st of each month
+- Customer-reported incidents cross-referenced with internal metrics
+- Quarterly audits comparing measured SLIs against contracted SLAs
+- Discrepancy review between internal metrics and customer-perceived availability
+
+### Breach Classification
+
+**Minor Breach:**
+- SLA missed by less than 0.05 percentage points (e.g., 99.85% vs 99.9% SLA)
+- Fewer than 3 discrete incidents contributed
+- No single incident exceeded 30 minutes
+- Customer impact was limited or partial degradation only
+- Financial credit: typically 5-10% of monthly service fee
+
+**Major Breach:**
+- SLA missed by 0.05 to 0.5 percentage points
+- Extended outage of 1-4 hours in a single incident, or multiple significant incidents
+- Clear customer impact with support tickets generated
+- Financial credit: typically 10-25% of monthly service fee
+
+**Critical Breach:**
+- SLA missed by more than 0.5 percentage points
+- Total outage exceeding 4 hours, or repeated major incidents in same window
+- Data loss, security incident, or compliance violation involved
+- Financial credit: typically 25-100% of monthly service fee
+- May trigger contract termination clauses
+
+### Response Protocol
+
+**For Minor Breach (within 3 business days):**
+1. Generate SLA compliance report with exact metrics
+2. Document contributing incidents with root causes
+3. Send proactive notification to customer success manager
+4. Issue service credits if contractually required (do not wait for customer to ask)
+5. File internal improvement ticket with 30-day remediation target
+
+**For Major Breach (within 24 hours):**
+1. Incident commander confirms SLA impact calculation
+2. Draft customer communication (see template below)
+3. Executive sponsor reviews and approves communication
+4. Issue service credits with detailed breakdown
+5. Schedule root cause review with customer within 5 business days
+6. Produce remediation plan with committed timelines
+
+**For Critical Breach (immediate):**
+1. Activate executive escalation chain
+2. Legal team reviews contractual exposure
+3. Finance team calculates credit obligations
+4. Customer communication from VP or C-level within 4 hours
+5. Dedicated remediation task force assigned
+6. Weekly status updates to customer until remediation complete
+7. Formal postmortem document shared with customer within 10 business days
+
+### Customer Communication Template
+
+```
+Subject: Service Level Update - [Service Name] - [Month Year]
+
+Dear [Customer Name],
+
+We are writing to inform you that [Service Name] did not meet the committed
+service level of [SLA target]% availability during [time period].
+
+MEASURED PERFORMANCE: [actual]% availability
+COMMITTED SLA: [SLA target]% availability
+SHORTFALL: [delta] percentage points
+
+CONTRIBUTING FACTORS:
+- [Date/Time]: [Brief description of incident] ([duration] impact)
+- [Date/Time]: [Brief description of incident] ([duration] impact)
+
+SERVICE CREDIT: In accordance with our agreement, a credit of [amount/percentage]
+will be applied to your next invoice.
+
+REMEDIATION ACTIONS:
+1. [Specific technical fix with completion date]
+2. [Process improvement with implementation date]
+3. [Monitoring enhancement with deployment date]
+
+We take our service commitments seriously. [Name], [Title] is personally
+overseeing the remediation and is available to discuss further at your convenience.
+
+Sincerely,
+[Name, Title]
+```
+
+### Legal and Compliance Considerations
+
+- Maintain auditable records of all SLA measurements for the full contract term plus 2 years
+- SLA calculations must use the measurement methodology defined in the contract, not internal approximations
+- Force majeure clauses typically exclude natural disasters, but verify per contract
+- Planned maintenance exclusions must match the exact notification procedures in the contract
+- Multi-region SLAs may have separate calculations per region; verify aggregation method
+
+---
+
+## 4. Incident-to-SLA Mapping
+
+### Downtime Calculation Methodologies
+
+**Full outage:** Service completely unavailable. Every minute counts as a full minute of downtime.
+
+```
+Downtime = End Time - Start Time (in minutes)
+```
+
+**Partial degradation:** Service available but impaired. Apply a degradation factor:
+
+```
+Effective Downtime = Actual Duration x Degradation Factor
+```
+
+| Degradation Level | Factor | Description |
+|---|---|---|
+| Complete outage | 1.0 | Service fully unavailable |
+| Severe degradation | 0.75 | >50% of requests failing or >10x latency |
+| Moderate degradation | 0.5 | 10-50% of requests affected or 3-10x latency |
+| Minor degradation | 0.25 | <10% of requests affected or <3x latency increase |
+| Cosmetic / non-functional | 0.0 | No impact on core SLI metrics |
+
+**Note:** The exact degradation factors must be agreed upon in the SLA contract. The above are industry-standard starting points.
+
+### Planned vs Unplanned Downtime
+
+Most SLAs exclude pre-announced maintenance windows from availability calculations, subject to conditions:
+
+- Notification provided N hours/days in advance (commonly 72 hours)
+- Maintenance occurs within an agreed window (e.g., Sunday 02:00-06:00 UTC)
+- Total planned downtime does not exceed the monthly maintenance allowance (e.g., 4 hours/month)
+- Any overrun beyond the planned window counts as unplanned downtime
+
+```
+SLA Availability = (Total Minutes - Excluded Maintenance - Unplanned Downtime) / (Total Minutes - Excluded Maintenance) x 100
+```
+
+### Multi-Service SLA Composition
+
+When a customer-facing product depends on multiple services, composite SLA is calculated as:
+
+**Serial dependency (all must be up):**
+```
+Composite SLA = SLA_A x SLA_B x SLA_C
+Example: 99.9% x 99.95% x 99.99% = 99.84%
+```
+
+**Parallel / redundant (any one must be up):**
+```
+Composite Availability = 1 - ((1 - SLA_A) x (1 - SLA_B))
+Example: 1 - ((1 - 0.999) x (1 - 0.999)) = 1 - 0.000001 = 99.9999%
+```
+
+This is critical during incidents: an outage in a shared dependency may breach SLAs for multiple customer-facing products simultaneously.
+
+### Worked Examples
+
+**Example 1: Simple outage**
+- Service: Core API (SLA: 99.9%)
+- Month: 30 days = 43,200 minutes
+- Incident: Full outage from 14:23 to 14:38 UTC on the 12th (15 minutes)
+- No other incidents this month
+
+```
+Availability = (43,200 - 15) / 43,200 x 100 = 99.965%
+SLA Status: PASS (99.965% > 99.9%)
+Error Budget Consumed: 15 / 43.2 = 34.7%
+```
+
+**Example 2: Partial degradation**
+- Service: Payment Processing (SLA: 99.95%)
+- Month: 30 days = 43,200 minutes
+- Incident: 50% of transactions failing for 4 hours (240 minutes)
+- Degradation factor: 0.5 (moderate - 50% of requests affected)
+
+```
+Effective Downtime = 240 x 0.5 = 120 minutes
+Availability = (43,200 - 120) / 43,200 x 100 = 99.722%
+SLA Status: FAIL (99.722% < 99.95%)
+Shortfall: 0.228 percentage points → Major Breach
+```
+
+**Example 3: Multiple incidents**
+- Service: Dashboard (SLA: 99.5%)
+- Month: 31 days = 44,640 minutes
+- Incident A: 45-minute full outage on the 5th
+- Incident B: 2-hour severe degradation (factor 0.75) on the 18th
+- Incident C: 30-minute full outage on the 25th
+
+```
+Total Effective Downtime = 45 + (120 x 0.75) + 30 = 45 + 90 + 30 = 165 minutes
+Availability = (44,640 - 165) / 44,640 x 100 = 99.630%
+SLA Status: PASS (99.630% > 99.5%)
+Error Budget Consumed: 165 / 223.2 = 73.9% → Yellow threshold, feature freeze recommended
+```
+
+---
+
+## 5. SLO Best Practices
+
+### Start with User Journeys
+
+Do not set SLOs based on infrastructure metrics. Start from what users experience:
+
+1. Identify critical user journeys (e.g., "User completes checkout")
+2. Map each journey to the services and dependencies involved
+3. Define what "good" looks like for each journey (fast, error-free, complete)
+4. Select the SLIs that most directly measure that user experience
+5. Set SLO targets that reflect the minimum acceptable user experience
+
+A database with 99.99% uptime is meaningless if the API in front of it has a bug causing 5% error rates.
+
+### The Four Golden Signals as SLI Sources
+
+From Google SRE, the four golden signals provide comprehensive service health:
+
+| Signal | SLI Example | Typical SLO |
+|---|---|---|
+| Latency | p99 request duration < 500ms | 99% of requests under threshold |
+| Traffic | Requests per second | N/A (capacity planning, not SLO) |
+| Errors | 5xx rate as % of total requests | < 0.1% error rate over rolling window |
+| Saturation | CPU/memory/queue depth | < 80% utilization (capacity SLI) |
+
+For most services, latency and error rate are the two most important SLIs to back with SLOs.
+
+### Setting SLO Targets
+
+1. Collect 90 days of historical SLI data
+2. Calculate the 5th percentile performance (worst 5% of days)
+3. Set SLO slightly above that baseline (this ensures the SLO is achievable without heroics)
+4. Validate: would a breach at this level actually impact users negatively?
+5. Adjust upward only if user impact analysis demands it
+
+**Never set SLOs by aspiration.** A 99.99% SLO on a service that has historically achieved 99.93% is a guaranteed source of perpetual firefighting with no reliability improvement.
+
+### Review Cadence
+
+- **Weekly:** Review current error budget burn rate, flag services approaching thresholds
+- **Monthly:** Full SLO compliance review, adjust alert thresholds if needed
+- **Quarterly:** Reassess SLO targets based on 90-day data, review SLA contract alignment
+- **Annually:** Strategic SLO review tied to product roadmap and infrastructure investments
+
+### Anti-Patterns
+
+| Anti-Pattern | Problem | Fix |
+|---|---|---|
+| Vanity SLOs | Setting 99.99% to impress, then ignoring breaches | Set achievable targets, enforce budget policy |
+| SLO Inflation | Ratcheting SLOs up whenever performance is good | Only increase SLOs when users demonstrably need it |
+| Unmeasured SLAs | Committing contractual SLAs without actual SLI measurement | Instrument SLIs before signing SLA contracts |
+| Copy-Paste SLOs | Same SLO for every service regardless of criticality | Tier services by business impact, set SLOs accordingly |
+| Ignoring Dependencies | Setting aggressive SLOs without accounting for dependency reliability | Calculate composite SLA; your SLO cannot exceed dependency chain |
+| Alert-Free SLOs | Having SLOs but no automated alerting on budget consumption | Every SLO must have corresponding burn rate alerts |
+
+---
+
+## 6. Monitoring & Alerting for SLAs
+
+### Multi-Window Burn Rate Alerting
+
+The Google SRE approach uses multiple time windows to balance speed of detection against alert noise. Each alert condition requires both a short window (for speed) and a long window (for confirmation):
+
+**Alert configuration matrix:**
+
+| Severity | Short Window | Short Threshold | Long Window | Long Threshold | Action |
+|---|---|---|---|---|---|
+| Critical (Page) | 1 hour | > 14.4x burn rate | 5 minutes | > 14.4x burn rate | Wake someone up |
+| High (Page) | 6 hours | > 6x burn rate | 30 minutes | > 6x burn rate | Page on-call within 30 min |
+| Medium (Ticket) | 3 days | > 1x burn rate | 6 hours | > 1x burn rate | Create ticket, next business day |
+
+**Why these specific numbers:**
+
+- 14.4x burn rate over 1 hour consumes 2% of monthly budget in that hour. At this rate, the entire 30-day budget is gone in ~50 hours. This demands immediate human attention.
+- 6x burn rate over 6 hours consumes 5% of monthly budget. The budget will be exhausted in 5 days. Urgent but not wake-up-at-3am urgent.
+- 1x burn rate over 3 days means you are on pace to exactly exhaust the budget. This needs investigation but is not an emergency.
+
+### Burn Rate Alert Formulas
+
+For a given time window, calculate the burn rate:
+
+```
+burn_rate = (error_count_in_window / request_count_in_window) / (1 - SLO_target)
+```
+
+Example for a 99.9% SLO, observing 50 errors out of 10,000 requests in a 1-hour window:
+
+```
+observed_error_rate = 50 / 10,000 = 0.005 (0.5%)
+allowed_error_rate = 1 - 0.999 = 0.001 (0.1%)
+burn_rate = 0.005 / 0.001 = 5.0
+```
+
+A burn rate of 5.0 means the error budget is being consumed 5 times faster than the sustainable rate.
+
+### Alert Severity to SLA Risk Mapping
+
+| Burn Rate | Budget Impact | SLA Risk | Response |
+|---|---|---|---|
+| < 1x | Under budget pace | None | Routine monitoring |
+| 1x - 3x | On pace or slightly over | Low | Investigate next business day |
+| 3x - 6x | Budget will exhaust in 5-10 days | Moderate | Investigate within 4 hours |
+| 6x - 14.4x | Budget will exhaust in 2-5 days | High | Page on-call, respond in 30 min |
+| > 14.4x | Budget will exhaust in < 2 days | Critical | Immediate page, incident declared |
+| > 100x | Active major outage | SLA breach imminent | All-hands incident response |
+
+### Dashboard Design for SLA Tracking
+
+Every SLA-tracked service should have a dashboard with these panels:
+
+**Row 1 - Current Status:**
+- Current availability (real-time, rolling 5-minute window)
+- Current error rate (real-time)
+- Current p99 latency (real-time)
+
+**Row 2 - Budget Status:**
+- Error budget remaining (% of monthly budget, gauge visualization)
+- Budget consumption timeline (line chart, actual vs expected burn)
+- Budget burn rate (current 1h, 6h, and 3d burn rates)
+
+**Row 3 - Historical Context:**
+- 30-day availability trend (daily granularity)
+- SLA compliance status for current and previous 3 months
+- Incident markers overlaid on availability timeline
+
+**Row 4 - Dependencies:**
+- Upstream dependency availability (services this service depends on)
+- Downstream impact scope (services that depend on this service)
+- Composite SLA calculation for customer-facing products
+
+### Alert Fatigue Prevention
+
+Alert fatigue is the primary reason SLA monitoring fails in practice. Mitigation strategies:
+
+1. **Require dual-window confirmation.** Never page on a single short window. Always require both the short window (for speed) and long window (for persistence) to fire simultaneously.
+
+2. **Separate page-worthy from ticket-worthy.** Only two conditions should wake someone up: >14.4x burn rate sustained, or >6x burn rate sustained. Everything else is a ticket.
+
+3. **Deduplicate aggressively.** If the same service triggers both a latency and error rate alert for the same underlying issue, group them into a single notification.
+
+4. **Auto-resolve.** Alerts must auto-resolve when the burn rate drops below threshold. Never leave stale alerts open.
+
+5. **Review alert quality monthly.** Track the ratio of actionable alerts to total alerts. Target >80% actionable rate. If an alert fires and no human action is needed, tune or remove it.
+
+6. **Escalation, not repetition.** If an alert is not acknowledged within the response window, escalate to the next tier. Do not re-send the same alert every 5 minutes.
+
+### Practical Monitoring Stack
+
+| Layer | Tool Category | Purpose |
+|---|---|---|
+| Collection | Prometheus, OpenTelemetry, StatsD | Gather SLI metrics from services |
+| Storage | Prometheus TSDB, Thanos, Mimir | Retain metrics for SLO window + 90 days |
+| Calculation | Prometheus recording rules, Sloth | Pre-compute burn rates and budget consumption |
+| Alerting | Alertmanager, PagerDuty, OpsGenie | Route alerts by severity and schedule |
+| Visualization | Grafana, Datadog | Dashboards for real-time and historical SLA views |
+| Reporting | Custom scripts, SLO generators | Monthly SLA compliance reports for customers |
+
+**Retention requirement:** SLI data must be retained for at least the SLA reporting period (typically monthly or quarterly) plus a 90-day dispute window. Annual SLA reviews require 12 months of data at daily granularity minimum.
+
+---
+
+*Last updated: February 2026*
+*For use with: incident-commander skill*
+*Maintainer: Engineering Team*