codymaster 4.1.3 → 4.3.0

package/skills/release-it/references/deployment-strategies.md

@@ -0,0 +1,331 @@
+# Deployment Strategies
+
+The distinction between deployment (placing code on infrastructure) and release (exposing code to users) is fundamental. Conflating the two means every deployment is a high-risk event. Separating them gives you the ability to deploy with confidence, test in production, and release when ready.
+
+Most production outages are caused by changes. Deployment strategies exist to make changes safe.
+
+## Zero-Downtime Deployment
+
+Zero-downtime deployment is not optional for any system with users. Users should never see an error page because you are deploying code.
+
+### Prerequisites for Zero-Downtime Deployment
+
+| Requirement | Why |
+|-------------|-----|
+| **Backward-compatible changes** | Old code and new code run simultaneously during deployment |
+| **Graceful shutdown** | In-flight requests must complete before an instance is terminated |
+| **Health checks** | Load balancer must know when a new instance is ready to receive traffic |
+| **Session independence** | Requests from the same user can be routed to any instance |
+| **Database compatibility** | Schema changes must work with both old and new application code |
+
+---
+
+## Rolling Deployment
+
+The simplest zero-downtime strategy. Replace instances one at a time (or in small batches), verifying health after each replacement.
+
+### How It Works
+
+```
+Cluster: [v1] [v1] [v1] [v1] [v1]
+
+Step 1:  [v2] [v1] [v1] [v1] [v1]   ← Deploy to instance 1, verify health
+Step 2:  [v2] [v2] [v1] [v1] [v1]   ← Deploy to instance 2, verify health
+Step 3:  [v2] [v2] [v2] [v1] [v1]   ← Deploy to instance 3, verify health
+Step 4:  [v2] [v2] [v2] [v2] [v1]   ← Deploy to instance 4, verify health
+Step 5:  [v2] [v2] [v2] [v2] [v2]   ← Deploy to instance 5, verify health
+```
+
+### Configuration Parameters
+
+| Parameter | Description | Typical Value |
+|-----------|-------------|---------------|
+| **Max unavailable** | Maximum instances being replaced simultaneously | 1 or 25% |
+| **Max surge** | Extra instances during deployment | 1 or 25% |
+| **Readiness probe** | Health check before receiving traffic | HTTP 200 on `/ready` |
+| **Min ready seconds** | How long instance must be healthy before proceeding | 30-60 seconds |
+
+### Advantages and Limitations
+
+| Advantage | Limitation |
+|-----------|-----------|
+| Simple to implement | Both versions run simultaneously (must be compatible) |
+| Gradual rollout | Rollback requires re-deploying old version |
+| No extra infrastructure | Reduced capacity during deployment |
+| Built into Kubernetes | Slow for large clusters |
+
+---
+
+## Blue-Green Deployment
+
+Maintain two identical production environments. One (blue) serves live traffic. Deploy to the other (green), verify, and switch the router.
+
+### How It Works
+
+```
+                    ┌──────────────┐
+Users → Router ────→│  Blue (v1)   │  ← Currently live
+                    └──────────────┘
+                    ┌──────────────┐
+                    │  Green (v2)  │  ← Deploy here, test
+                    └──────────────┘
+
+After verification:
+
+                    ┌──────────────┐
+                    │  Blue (v1)   │  ← Standby (instant rollback)
+                    └──────────────┘
+                    ┌──────────────┐
+Users → Router ────→│  Green (v2)  │  ← Now live
+                    └──────────────┘
+```
+
+### Implementation Steps
+
+1. Deploy new version to green environment
+2. Run automated smoke tests against green
+3. Optionally, route internal/test traffic to green for manual verification
+4. Switch the router (load balancer, DNS, or service mesh) to green
+5. Monitor for errors -- if problems arise, switch back to blue immediately
+6. Keep blue running as the rollback target until the next deployment
+
+### Advantages and Limitations
+
+| Advantage | Limitation |
+|-----------|-----------|
+| Instant rollback (switch router back) | Requires double the infrastructure |
+| Full testing in production-like environment | Database changes must be compatible with both versions |
+| Zero capacity reduction during deployment | Stateful applications need careful session handling |
+| Clear separation of deploy and release | Cost of maintaining two full environments |
+
+---
+
+## Canary Releases
+
+Route a small percentage of production traffic to the new version. Monitor for errors. Gradually increase the percentage if healthy.
+
+### How It Works
+
+```
+Phase 1: 5% → canary (v2), 95% → stable (v1)
+Phase 2: 25% → canary (v2), 75% → stable (v1)
+Phase 3: 50% → canary (v2), 50% → stable (v1)
+Phase 4: 100% → canary (v2), 0% → stable (v1)
+```
+
+### Canary Evaluation Criteria
+
+At each phase, evaluate before proceeding:
+
+| Metric | Threshold | Action if Exceeded |
+|--------|-----------|-------------------|
+| **Error rate** | > 1% above baseline | Automatic rollback |
+| **p99 latency** | > 2x baseline | Pause and investigate |
+| **CPU usage** | > 80% sustained | Pause and investigate |
+| **Memory usage** | Growing trend (potential leak) | Rollback |
+| **Business metrics** | Conversion rate drops > 5% | Rollback |
+
+### Automated Canary Analysis
+
+Progressive delivery tools can automate canary evaluation:
+
+1. Deploy canary with 5% traffic
+2. Collect metrics for evaluation window (10-30 minutes)
+3. Compare canary metrics against baseline (stable version) using statistical analysis
+4. If pass: increase traffic percentage and repeat
+5. If fail: automatic rollback and alert the team
+
+### Canary vs. Blue-Green
+
+| Aspect | Canary | Blue-Green |
+|--------|--------|-----------|
+| **Risk exposure** | Minimal (small % of traffic) | All-or-nothing switch |
+| **Rollback speed** | Instant (route away from canary) | Instant (switch router) |
+| **Infrastructure cost** | Minimal extra (small canary fleet) | Double infrastructure |
+| **Verification depth** | Real user traffic at scale | Synthetic tests + internal traffic |
+| **Complexity** | Higher (traffic splitting, metric comparison) | Lower (router switch) |
+
+---
+
+## Feature Flags
+
+Feature flags (also called feature toggles) decouple deployment from release. Code is deployed but not activated until the flag is enabled.
+
+### Types of Feature Flags
+
+| Type | Lifetime | Purpose | Example |
+|------|----------|---------|---------|
+| **Release flag** | Days to weeks | Gate incomplete or untested features | `new_checkout_flow` |
+| **Experiment flag** | Weeks to months | A/B testing and gradual rollout | `show_recommendations_v2` |
+| **Ops flag** | Permanent | Runtime control over system behavior | `enable_expensive_query_cache` |
+| **Kill switch** | Permanent | Disable features during incidents | `disable_search_suggestions` |
+
+### Feature Flag Best Practices
+
+**Do:**
+- Use a centralized flag management system (not config files or environment variables)
+- Set a default value for every flag (what happens if the flag service is down?)
+- Clean up release flags after full rollout (flag debt is real technical debt)
+- Log flag evaluations for debugging ("User X saw feature Y because flag Z was true")
+- Test both flag states in your test suite
+
+**Do not:**
+- Use feature flags for long-lived branching (creates combinatorial testing nightmare)
+- Nest feature flags (flag A enables feature which checks flag B -- unmaintainable)
+- Deploy code that only works with the flag enabled (always support both states)
+- Forget to remove old flags (accumulation makes the code unreadable)
+
+### Feature Flag and Deployment Pipeline
+
+```
+1. Developer merges code with feature flag (default: off)
+2. Code deploys to production (flag off -- no user impact)
+3. QA enables flag for internal users and tests
+4. Product enables flag for 5% of users (canary)
+5. Monitor metrics for 24-48 hours
+6. Ramp to 25%, 50%, 100% with monitoring at each stage
+7. When 100% and stable, remove the flag and the old code path
+```
+
+---
+
+## Database Migrations Without Downtime
+
+Database schema changes are the most dangerous part of deployment because they are difficult to roll back and must be compatible with both old and new application code during the deployment window.
+
+### The Expand-Contract Pattern
+
+Never make a breaking schema change in a single step. Instead, expand (add), migrate data, then contract (remove).
+
+### Example: Renaming a Column
+
+**Wrong (causes downtime):**
+```sql
+ALTER TABLE users RENAME COLUMN name TO full_name;
+-- Old code looking for "name" column fails immediately
+```
+
+**Right (zero-downtime expand-contract):**
+
+| Step | Migration | Application Code |
+|------|-----------|-----------------|
+| 1. Expand | `ALTER TABLE users ADD COLUMN full_name VARCHAR(255);` | Writes to both `name` and `full_name` |
+| 2. Backfill | `UPDATE users SET full_name = name WHERE full_name IS NULL;` | Reads from `full_name`, falls back to `name` |
+| 3. Switch | No schema change | Reads and writes only `full_name` |
+| 4. Contract | `ALTER TABLE users DROP COLUMN name;` | Only uses `full_name` |
+
+Each step is a separate deployment. Each step is individually rollback-safe.
+
+### Example: Adding a NOT NULL Column
+
+**Wrong (locks table, breaks old code):**
+```sql
+ALTER TABLE orders ADD COLUMN status VARCHAR(20) NOT NULL DEFAULT 'pending';
+-- On large tables, this locks the table for minutes/hours
+```
+
+**Right (phased approach):**
+
+| Step | Action |
+|------|--------|
+| 1 | Add column as nullable: `ALTER TABLE orders ADD COLUMN status VARCHAR(20);` |
+| 2 | Deploy code that writes `status` on all new rows |
+| 3 | Backfill existing rows: `UPDATE orders SET status = 'pending' WHERE status IS NULL;` (in batches) |
+| 4 | Add NOT NULL constraint: `ALTER TABLE orders ALTER COLUMN status SET NOT NULL;` |
+
+### Migration Safety Checklist
+
+- [ ] Can the old code work with the new schema?
+- [ ] Can the new code work with the old schema (for rollback)?
+- [ ] Are large data migrations batched (not one giant UPDATE)?
+- [ ] Is the migration tested against production-volume data?
+- [ ] Is the migration reversible?
+- [ ] Are table locks avoided (no ALTER TABLE on large tables without online DDL)?
+
+---
+
+## Immutable Infrastructure
+
+Never patch, update, or modify a running server. Instead, build a new image with the changes, deploy it, and destroy the old one.
+
+### Why Immutable
+
+| Mutable Infrastructure | Immutable Infrastructure |
+|----------------------|-------------------------|
+| SSH into servers, apply patches | Build new image with patches baked in |
+| Configuration drift over time | Every instance is identical |
+| "Snowflake" servers that are irreplaceable | Instances are disposable and replaceable |
+| "It works on that server" debugging | Consistent behavior across all instances |
+| Manual changes accumulate and are undocumented | All changes are in version control |
+
+### Immutable Infrastructure Pipeline
+
+```
+1. Code change committed to version control
+2. CI builds application artifact (binary, JAR, bundle)
+3. CI builds infrastructure image (Docker image, AMI, VM image)
+4. Image is tagged with version and stored in registry
+5. Deployment tool replaces old instances with new image
+6. Old instances are terminated (not modified, not reused)
+```
+
+### Implementation Patterns
+
+| Pattern | Technology | Use Case |
+|---------|-----------|----------|
+| **Container images** | Docker, OCI | Microservices, cloud-native applications |
+| **Machine images** | AMI, GCE image | VM-based workloads |
+| **Serverless packages** | Lambda ZIP, Cloud Function | Event-driven workloads |
+| **Helm charts** | Kubernetes + Helm | Kubernetes-native applications |
+
+---
+
+## Infrastructure as Code
+
+All infrastructure -- servers, networks, databases, load balancers, DNS entries -- is defined in version-controlled code, not manually configured through web consoles or SSH sessions.
+
+### Principles
+
+| Principle | Practice |
+|-----------|----------|
+| **Everything in code** | No manual changes; all config in Terraform, CloudFormation, Pulumi, or similar |
+| **Version controlled** | Infrastructure code lives in git alongside application code |
+| **Reviewed** | Infrastructure changes go through code review like application changes |
+| **Tested** | Infrastructure changes are validated in staging before production |
+| **Reproducible** | Any environment can be recreated from code in minutes |
+| **Idempotent** | Applying the same code twice produces the same result |
+
+### Anti-Patterns
+
+| Anti-Pattern | Problem | Fix |
+|-------------|---------|-----|
+| **ClickOps** | Changes made through web console are undocumented and unreproducible | Define all infrastructure in code |
+| **SSH and modify** | Manual changes create drift between instances | Use immutable infrastructure |
+| **Shared credentials** | No audit trail of who changed what | Individual credentials with role-based access |
+| **No staging** | Infrastructure changes tested directly in production | Maintain a staging environment that mirrors production topology |
+| **Monolithic templates** | One giant infrastructure file that is impossible to review | Modularize infrastructure into composable components |
+
+---
+
+## Rollback Strategy
+
+Rollback must be faster and simpler than rolling forward. If rolling back takes 30 minutes of manual steps, teams will hesitate to deploy -- and when they do deploy, they will hesitate to roll back when things go wrong.
+
+### Rollback Approaches
+
+| Approach | Speed | Complexity | Limitation |
+|----------|-------|-----------|-----------|
+| **Traffic switch** (blue-green) | Seconds | Low | Requires both versions running |
+| **Revert canary** | Seconds | Low | Only for canary percentage |
+| **Redeploy previous version** | Minutes | Medium | Requires previous artifact available |
+| **Feature flag disable** | Seconds | Low | Only for flag-gated features |
+| **Database rollback** | Minutes to hours | High | May require data migration reversal |
+
+### Rollback Checklist
+
+- [ ] Is the previous version's artifact still available in the registry?
+- [ ] Can the previous version work with the current database schema?
+- [ ] Are feature flags in a state that supports rollback?
+- [ ] Has the rollback procedure been tested (not just documented)?
+- [ ] Can rollback be executed by the on-call engineer without escalation?
+- [ ] Is the rollback time under 5 minutes?

package/skills/release-it/references/observability.md

@@ -0,0 +1,301 @@
+# Observability
+
+You cannot operate what you cannot observe. Observability is the ability to understand the internal state of a system by examining its external outputs. It is not an afterthought or a nice-to-have -- it is a first-class design concern that must be built into every service from day one.
+
+A well-observed system lets you answer questions you did not anticipate at design time. A poorly observed system forces you to deploy new instrumentation during an outage -- exactly when you can least afford the risk.
+
+## The Three Pillars of Observability
+
+### 1. Structured Logs
+
+Logs answer the question: "What happened?"
+
+**Structured logging** means emitting logs as key-value pairs (JSON), not free-form text. Structured logs are searchable, filterable, and aggregatable. Free-form text logs require regex parsing and break when the format changes.
+
+**Essential log fields:**
+
+| Field | Purpose | Example |
+|-------|---------|---------|
+| `timestamp` | When the event occurred | `2024-01-15T14:23:45.123Z` |
+| `level` | Severity (DEBUG, INFO, WARN, ERROR) | `ERROR` |
+| `service` | Which service emitted the log | `payment-service` |
+| `trace_id` | Correlation ID across services | `abc123def456` |
+| `span_id` | Specific operation within the trace | `span789` |
+| `user_id` | Which user was affected (if applicable) | `user_42` |
+| `message` | Human-readable description | `Payment processing failed` |
+| `error` | Error type and stack trace | `TimeoutException: read timed out` |
+| `duration_ms` | How long the operation took | `5230` |
+| `request_id` | Unique identifier for the request | `req_abc123` |
+
+**Logging best practices:**
+- Log at service boundaries (incoming request, outgoing call, response)
+- Include enough context to understand the event without reading code
+- Use consistent field names across all services
+- Do not log sensitive data (passwords, tokens, PII) -- use redaction
+- Log errors with full context, not just the exception message
+- Use sampling for high-volume debug logs in production
+- Emit logs asynchronously to avoid blocking application threads
+
+### 2. Metrics
+
+Metrics answer the question: "How much?"
+
+Metrics are numeric measurements collected over time. They are cheap to store, fast to query, and essential for dashboards and alerts.
+
+**Metric types:**
+
+| Type | What It Measures | Example |
+|------|-----------------|---------|
+| **Counter** | Cumulative count of events | `http_requests_total`, `errors_total` |
+| **Gauge** | Current value that can go up or down | `active_connections`, `queue_depth`, `cpu_usage` |
+| **Histogram** | Distribution of values | `request_duration_seconds` (with buckets for p50, p95, p99) |
+| **Summary** | Pre-calculated percentiles | `request_duration_seconds{quantile="0.99"}` |
+
+**Metric naming conventions:**
+- Use snake_case: `http_request_duration_seconds`
+- Include the unit in the name: `_seconds`, `_bytes`, `_total`
+- Use labels for dimensions: `http_requests_total{method="GET", status="200", endpoint="/api/users"}`
+- Do not use high-cardinality labels (user IDs, request IDs) -- these explode metric storage
+
+### 3. Distributed Traces
+
+Traces answer the question: "Where did the time go?"
+
+A distributed trace follows a single request across multiple services, showing the sequence of operations, their durations, and their relationships.
+
+**Trace anatomy:**
+
+```
+Trace: user request to checkout
+├── Span: API Gateway (2ms)
+│   ├── Span: Auth Service - validate token (15ms)
+│   ├── Span: Cart Service - get cart (45ms)
+│   │   └── Span: Database - SELECT cart items (12ms)
+│   ├── Span: Inventory Service - check stock (120ms)  ← Bottleneck
+│   │   └── Span: Database - SELECT inventory (95ms)
+│   └── Span: Payment Service - charge card (230ms)
+│       └── Span: Stripe API - create charge (180ms)
+Total: 412ms
+```
+
+**Trace implementation:**
+- Inject trace context (trace ID, span ID) into all outgoing requests (HTTP headers, message metadata)
+- Extract trace context from all incoming requests
+- Create a new span for each significant operation (service call, database query, cache lookup)
+- Annotate spans with relevant metadata (query, parameters, result count)
+- Use sampling in production (1-10% of requests) to manage cost and volume
+- Always trace errors at 100% (do not sample error traces)
+
+---
+
+## Health Check Patterns
+
+Health checks tell load balancers, orchestrators, and monitoring systems whether an instance is able to serve traffic.
+
+### Shallow Health Check
+
+A shallow health check verifies that the process is running and can respond to HTTP requests. It does not verify dependencies.
+
+```
+GET /health
+200 OK {"status": "up"}
+```
+
+**Use for:** Liveness probes (is the process alive?). If this fails, the orchestrator should restart the process.
+
+### Deep Health Check
+
+A deep health check verifies that the instance can actually serve requests by checking connectivity to all critical dependencies.
+
+```
+GET /health/ready
+200 OK {
+  "status": "ready",
+  "checks": {
+    "database": {"status": "up", "latency_ms": 3},
+    "cache": {"status": "up", "latency_ms": 1},
+    "queue": {"status": "up", "latency_ms": 5},
+    "disk": {"status": "up", "free_gb": 42}
+  }
+}
+
+503 Service Unavailable {
+  "status": "not_ready",
+  "checks": {
+    "database": {"status": "down", "error": "connection refused"},
+    "cache": {"status": "up", "latency_ms": 1},
+    "queue": {"status": "up", "latency_ms": 5},
+    "disk": {"status": "up", "free_gb": 42}
+  }
+}
+```
+
+**Use for:** Readiness probes (can this instance serve traffic?). If this fails, the load balancer should stop routing traffic to this instance -- but should not restart it (the problem might be a downstream dependency, not this process).
+
+### Health Check Design Rules
+
+| Rule | Rationale |
+|------|-----------|
+| **Shallow checks should be fast (<100ms)** | Frequent liveness checks should not consume significant resources |
+| **Deep checks should have their own timeout** | A hanging dependency check should not make the health endpoint hang |
+| **Do not cache health check results** | Health checks must reflect current state, not cached state |
+| **Separate liveness from readiness** | A process can be alive but not ready (warming up, dependency down) |
+| **Include version information** | Helps verify deployment status: `"version": "2.3.1", "commit": "abc123"` |
+| **Rate-limit deep checks** | Running deep checks every second can stress dependencies |
+
+---
+
+## The RED Method
+
+The RED method is a monitoring framework for request-driven services (APIs, web applications).
+
+### RED Metrics
+
+| Metric | What It Measures | Why It Matters |
+|--------|-----------------|---------------|
+| **Rate** | Requests per second | Is traffic normal? Dropping? Spiking? |
+| **Errors** | Error rate (errors / total requests) | Are users experiencing failures? |
+| **Duration** | Latency distribution (p50, p95, p99) | Are users experiencing slowness? |
+
+### Implementation
+
+For every service endpoint, instrument:
+
+```
+# Rate
+http_requests_total{service, method, endpoint, status}
+
+# Errors
+http_errors_total{service, method, endpoint, error_type}
+# Error rate = http_errors_total / http_requests_total
+
+# Duration
+http_request_duration_seconds{service, method, endpoint}
+# Report as histogram with p50, p95, p99 percentiles
+```
+
+### RED Dashboard
+
+A RED dashboard for each service should answer three questions at a glance:
+1. **Is traffic arriving?** (Rate graph -- sudden drops indicate upstream problems or DNS issues)
+2. **Are requests succeeding?** (Error rate graph -- spikes indicate bugs or dependency failures)
+3. **Are requests fast?** (Duration graph -- p99 latency increasing indicates saturation)
+
+---
+
+## The USE Method
+
+The USE method is a monitoring framework for infrastructure resources (CPU, memory, disk, network).
+
+### USE Metrics
+
+| Metric | What It Measures | Why It Matters |
+|--------|-----------------|---------------|
+| **Utilization** | Percentage of resource currently in use | High utilization means approaching capacity |
+| **Saturation** | Amount of work waiting (queue depth) | Saturation means demand exceeds capacity |
+| **Errors** | Count of error events for this resource | Hardware errors, packet drops, OOM kills |
+
+### USE by Resource
+
+| Resource | Utilization | Saturation | Errors |
+|----------|------------|------------|--------|
+| **CPU** | % time busy | Run queue length | Machine check exceptions |
+| **Memory** | % used | Swap usage, OOM events | ECC errors, OOM kills |
+| **Disk** | % capacity used, IOPS utilization | I/O queue depth | Read/write errors |
+| **Network** | Bandwidth utilization | TCP retransmit queue, dropped packets | Interface errors, CRC errors |
+| **Thread pool** | Active threads / max threads | Queued tasks | Rejected tasks |
+| **Connection pool** | Active connections / max connections | Wait count | Timeout errors |
+
+---
+
+## SLIs, SLOs, and SLAs
+
+### Definitions
+
+| Term | Definition | Example |
+|------|-----------|---------|
+| **SLI** (Service Level Indicator) | A quantitative measure of a specific aspect of service quality | 99.2% of requests complete in < 200ms |
+| **SLO** (Service Level Objective) | A target value for an SLI | 99.5% of requests should complete in < 200ms |
+| **SLA** (Service Level Agreement) | A contractual commitment with consequences for violation | 99.9% availability; credit issued if breached |
+
+### Choosing SLIs
+
+Good SLIs measure what users actually experience:
+
+| SLI Type | What It Measures | Measurement Point |
+|----------|-----------------|-------------------|
+| **Availability** | Proportion of successful requests | Load balancer or edge proxy |
+| **Latency** | Proportion of requests faster than threshold | Application instrumentation |
+| **Correctness** | Proportion of requests returning correct results | End-to-end tests or data validation |
+| **Freshness** | Proportion of data updated within threshold | Data pipeline monitoring |
+
+### Error Budget
+
+The error budget is the allowed amount of unreliability: `error_budget = 1 - SLO`.
+
+For a 99.9% availability SLO:
+- Error budget = 0.1% = 43.8 minutes/month of downtime
+- If you have consumed 30 minutes of budget, you have 13.8 minutes remaining
+- If the budget is exhausted, freeze deployments and focus on reliability
+
+**Error budget policy:**
+- Budget remaining > 50%: deploy freely, run experiments
+- Budget remaining 20-50%: deploy with caution, increase monitoring
+- Budget remaining < 20%: freeze non-critical deploys, prioritize reliability work
+- Budget exhausted: halt all feature deploys until budget replenishes
+
+---
+
+## Alerting Strategy
+
+### Alert on Symptoms, Not Causes
+
+| Cause-Based Alert (Avoid) | Symptom-Based Alert (Prefer) |
+|--------------------------|------------------------------|
+| CPU > 80% | p99 latency > 500ms |
+| Memory > 90% | Error rate > 1% |
+| Disk > 85% | Availability < 99.9% |
+| Queue depth > 1000 | User-facing errors increasing |
+
+Cause-based alerts generate noise. CPU can be at 90% and users are fine. CPU can be at 50% and users are seeing errors because of a deadlock.
+
+### Alert Severity Levels
+
+| Level | Criteria | Response | Example |
+|-------|----------|----------|---------|
+| **Critical** | Users actively impacted; error budget burning fast | Page on-call immediately | Error rate > 5% for 5 minutes |
+| **Warning** | Approaching threshold; action needed soon | Notify team during business hours | Error budget burn rate 2x normal |
+| **Info** | Notable but not actionable | Log and dashboard only | Deployment completed; circuit breaker tripped and recovered |
+
+### Alerting Anti-Patterns
+
+| Anti-Pattern | Problem | Fix |
+|-------------|---------|-----|
+| **Alert on every metric** | Alert fatigue; team ignores pages | Only alert on user-facing symptoms |
+| **No alert grouping** | 50 alerts fire for one incident | Group related alerts; alert on the root symptom |
+| **No runbook** | On-call does not know what to do | Every alert links to a runbook with diagnostic steps |
+| **Stale alerts** | Alerts for services that no longer exist | Review and prune alerts quarterly |
+| **Missing alerts** | Critical failures go unnoticed | Regularly audit: "If X fails, would we know?" |
+
+---
+
+## Dashboards That Matter
+
+A dashboard should answer "Is the system healthy right now?" within 5 seconds of looking at it.
+
+### Dashboard Hierarchy
+
+| Level | Audience | Content |
+|-------|----------|---------|
+| **Executive** | Leadership | SLO status (green/red), error budget remaining, incident count |
+| **Service overview** | On-call engineer | RED metrics per service, dependency status |
+| **Service deep-dive** | Service owner | Detailed metrics, resource utilization, deployment markers |
+| **Debug** | Investigating engineer | Traces, log queries, specific metric breakdowns |
+
+### Dashboard Design Rules
+
+- **USE traffic lights:** Green (healthy), yellow (degraded), red (critical) at the top of every dashboard
+- **Show trends, not just current values:** A metric at 70% is fine if it has been at 70% for a week; it is alarming if it was at 30% an hour ago
+- **Mark deployments:** Overlay deployment events on metric graphs to correlate changes with behavior
+- **Time range matters:** Default to 6 hours for operational dashboards; allow quick switching to 1h, 24h, 7d
+- **Less is more:** A dashboard with 50 graphs is useless; a dashboard with 4 graphs that answer the right questions is invaluable