agentic-team-templates 0.3.0

package/templates/devops-sre/.cursorrules/toil-reduction.md

@@ -0,0 +1,567 @@
+# Toil Reduction
+
+Comprehensive guidelines for identifying, measuring, and eliminating operational toil.
+
+## Core Principles
+
+1. **Identify Systematically** - Measure toil before trying to reduce it
+2. **Automate Ruthlessly** - If you do it more than twice, automate it
+3. **Eliminate, Don't Optimize** - Sometimes the best automation is not doing the task
+4. **Invest Continuously** - Dedicate time specifically for toil reduction
+
+## Understanding Toil
+
+### Definition of Toil
+
+```yaml
+toil_characteristics:
+  manual:
+    description: "Requires human intervention"
+    example: "SSH to server to check status"
+    
+  repetitive:
+    description: "Done over and over"
+    example: "Weekly certificate rotation"
+    
+  automatable:
+    description: "Could be done by a machine"
+    example: "Restarting a service on alert"
+    
+  tactical:
+    description: "Reactive, not strategic"
+    example: "Responding to disk space alerts"
+    
+  no_enduring_value:
+    description: "Doesn't improve the system"
+    example: "Manual deploys that don't improve process"
+    
+  scales_linearly:
+    description: "Grows with service growth"
+    example: "Provisioning new user accounts"
+
+not_toil:
+  - "Coding new automation"
+  - "Designing systems"
+  - "Writing documentation"
+  - "On-call work (though it can become toil)"
+  - "Postmortem analysis"
+  - "Architecture reviews"
+```
+
+### Toil vs Engineering Work
+
+```yaml
+comparison:
+  toil:
+    definition: "Running the service"
+    value: "Keeps lights on, temporary"
+    scaling: "Grows with service size"
+    outcome: "Maintains status quo"
+    examples:
+      - "Manual deployments"
+      - "Ticket queue processing"
+      - "Capacity reports"
+      - "Alert response"
+      
+  engineering:
+    definition: "Improving the service"
+    value: "Permanent improvements"
+    scaling: "Reduces future work"
+    outcome: "Makes service better"
+    examples:
+      - "Automating deployments"
+      - "Building self-service tools"
+      - "Improving monitoring"
+      - "Writing runbooks"
+
+target_balance:
+  toil: "< 50% of SRE time"
+  engineering: "> 50% of SRE time"
+  rationale: "Too much toil prevents improvement"
+```
+
+## Measuring Toil
+
+### Toil Tracking
+
+```yaml
+tracking_methods:
+  time_tracking:
+    description: "Log time spent on tasks"
+    tools:
+      - "Jira work logs"
+      - "Toggl/Clockify"
+      - "Custom spreadsheet"
+    categories:
+      - "Incident response"
+      - "Deployments"
+      - "Access requests"
+      - "Capacity management"
+      - "Other operational tasks"
+      
+  ticket_analysis:
+    description: "Analyze support ticket patterns"
+    metrics:
+      - "Tickets per week by type"
+      - "Time per ticket"
+      - "Recurring ticket themes"
+      
+  survey:
+    description: "Ask team about toil burden"
+    frequency: "Monthly or quarterly"
+    questions:
+      - "Top 3 most tedious tasks"
+      - "Tasks you wish were automated"
+      - "Estimated toil percentage"
+
+toil_inventory:
+  template: |
+    | Task | Frequency | Duration | Automatable? | Priority |
+    |------|-----------|----------|--------------|----------|
+    | Manual deploys | 10x/week | 30 min | Yes | High |
+    | User provisioning | 5x/week | 15 min | Yes | Medium |
+    | Alert investigation | 20x/week | 15 min | Partial | High |
+    | Capacity report | 1x/week | 2 hours | Yes | Medium |
+```
+
+### Toil Metrics
+
+```yaml
+key_metrics:
+  toil_percentage:
+    formula: "toil_hours / total_work_hours * 100"
+    target: "< 50%"
+    measurement: "Weekly or monthly"
+    
+  toil_per_incident:
+    formula: "total_toil_hours / incident_count"
+    target: "Decreasing trend"
+    indicates: "Incident response efficiency"
+    
+  automation_rate:
+    formula: "automated_tasks / total_tasks"
+    target: "Increasing trend"
+    indicates: "Progress on automation"
+    
+  time_to_automate:
+    formula: "time_since_task_identified / is_automated"
+    target: "< 30 days for high-priority"
+    indicates: "Automation velocity"
+
+tracking_dashboard:
+  visualizations:
+    - "Toil trend over time"
+    - "Toil by category"
+    - "Toil per team member"
+    - "Automation backlog size"
+```
+
+## Identifying Toil
+
+### Common Sources of Toil
+
+```yaml
+deployment_toil:
+  symptoms:
+    - "Manual deployment steps"
+    - "Deployment-related pages"
+    - "Post-deploy verification"
+  examples:
+    - "SSH to servers to deploy"
+    - "Manually running database migrations"
+    - "Verifying each deployment manually"
+  solutions:
+    - "CI/CD pipelines"
+    - "Automated rollbacks"
+    - "Deployment verification tests"
+
+access_management_toil:
+  symptoms:
+    - "Manual account creation"
+    - "Permission requests via tickets"
+    - "Access audits"
+  examples:
+    - "Creating cloud IAM users manually"
+    - "Adding SSH keys to servers"
+    - "Granting database access"
+  solutions:
+    - "Self-service access portal"
+    - "RBAC with group management"
+    - "Automated access reviews"
+
+capacity_toil:
+  symptoms:
+    - "Manual scaling"
+    - "Capacity planning spreadsheets"
+    - "Alert-driven scaling"
+  examples:
+    - "Resizing instances manually"
+    - "Adding nodes before expected load"
+    - "Responding to disk space alerts"
+  solutions:
+    - "Autoscaling"
+    - "Automated capacity reporting"
+    - "Predictive scaling"
+
+incident_response_toil:
+  symptoms:
+    - "Manual remediation steps"
+    - "Repeated troubleshooting"
+    - "Alert fatigue"
+  examples:
+    - "Restarting services manually"
+    - "Same investigation for same alerts"
+    - "Too many non-actionable alerts"
+  solutions:
+    - "Auto-remediation"
+    - "Better alerts"
+    - "Self-healing systems"
+
+maintenance_toil:
+  symptoms:
+    - "Manual updates"
+    - "Certificate management"
+    - "Secret rotation"
+  examples:
+    - "Patching servers one by one"
+    - "Renewing certificates manually"
+    - "Rotating passwords"
+  solutions:
+    - "Automated patching"
+    - "Auto-renewing certificates"
+    - "Secret management systems"
+```
+
+### Toil Discovery Methods
+
+```yaml
+discovery_techniques:
+  retrospective:
+    frequency: "Weekly"
+    questions:
+      - "What took the most time this week?"
+      - "What felt repetitive?"
+      - "What would you automate?"
+      
+  shadowing:
+    description: "Observe team members working"
+    purpose: "Find unrecognized toil"
+    outcome: "List of automation candidates"
+    
+  ticket_mining:
+    process:
+      - "Export last 3 months of tickets"
+      - "Categorize by type"
+      - "Identify patterns"
+      - "Calculate time per category"
+      
+  on_call_analysis:
+    data:
+      - "Alert frequency by type"
+      - "Actions taken per alert"
+      - "Time to resolve"
+    patterns:
+      - "Same alerts recurring"
+      - "Same remediation steps"
+      - "Alerts during off-hours"
+```
+
+## Eliminating Toil
+
+### Elimination Strategies
+
+```yaml
+eliminate:
+  description: "Remove the need entirely"
+  examples:
+    - "Remove unused service instead of maintaining it"
+    - "Simplify architecture to remove failure modes"
+    - "Delegate to managed service"
+  when_to_use: "When task has questionable value"
+
+automate:
+  description: "Build software to do the task"
+  examples:
+    - "CI/CD for deployments"
+    - "Auto-scaling for capacity"
+    - "Auto-remediation for common issues"
+  when_to_use: "When task is valuable but repetitive"
+
+self_service:
+  description: "Enable users to do it themselves"
+  examples:
+    - "Self-service access requests"
+    - "Self-service environment provisioning"
+    - "Documentation for common questions"
+  when_to_use: "When SRE shouldn't be in the critical path"
+
+optimize:
+  description: "Make the task faster"
+  examples:
+    - "Better tooling"
+    - "Streamlined processes"
+    - "Parallel execution"
+  when_to_use: "When elimination/automation not feasible"
+```
+
+### Prioritization Framework
+
+```yaml
+prioritization_matrix:
+  high_priority:
+    criteria:
+      - "High frequency (daily+)"
+      - "High time cost (> 1 hour)"
+      - "Clearly automatable"
+      - "Affects multiple people"
+    examples:
+      - "Manual deployments"
+      - "Recurring alerts needing same fix"
+      
+  medium_priority:
+    criteria:
+      - "Medium frequency (weekly)"
+      - "Medium time cost (30-60 min)"
+      - "Partially automatable"
+    examples:
+      - "Access requests"
+      - "Capacity adjustments"
+      
+  low_priority:
+    criteria:
+      - "Low frequency (monthly+)"
+      - "Low time cost (< 30 min)"
+      - "Complex to automate"
+    examples:
+      - "Quarterly audits"
+      - "One-off requests"
+
+roi_calculation:
+  formula: |
+    ROI = (time_saved_per_occurrence * frequency * duration) / automation_effort
+    
+  example:
+    task: "Manual deployment"
+    time_per_occurrence: "30 minutes"
+    frequency: "10 per week"
+    duration: "52 weeks"
+    total_toil: "260 hours/year"
+    automation_effort: "40 hours"
+    roi: "6.5x in first year"
+```
+
+### Automation Best Practices
+
+```yaml
+automation_principles:
+  start_simple:
+    description: "Automate the 80% case first"
+    example: "Automate happy path, handle edge cases manually initially"
+    
+  iterate:
+    description: "Improve automation over time"
+    approach: "Ship basic automation, add features based on real usage"
+    
+  document:
+    description: "Explain what automation does"
+    purpose: "Others can debug, maintain, improve"
+    
+  monitor:
+    description: "Track automation success/failure"
+    metrics:
+      - "Success rate"
+      - "Time saved"
+      - "Failure modes"
+      
+  maintain:
+    description: "Keep automation working"
+    consideration: "Automation has maintenance cost too"
+
+self_healing_patterns:
+  auto_restart:
+    trigger: "Process crashed"
+    action: "Restart process"
+    tools: "Kubernetes, systemd"
+    
+  auto_scale:
+    trigger: "High load"
+    action: "Add capacity"
+    tools: "HPA, cloud autoscaling"
+    
+  auto_failover:
+    trigger: "Primary failure"
+    action: "Promote secondary"
+    tools: "Patroni, RDS Multi-AZ"
+    
+  auto_remediate:
+    trigger: "Known error condition"
+    action: "Apply known fix"
+    tools: "Custom scripts, Rundeck"
+```
+
+## Self-Service
+
+### Building Self-Service
+
+```yaml
+self_service_principles:
+  reduce_friction:
+    goal: "Make it easier to self-serve than to ask"
+    methods:
+      - "Good documentation"
+      - "Intuitive interfaces"
+      - "Sensible defaults"
+      
+  guardrails:
+    goal: "Prevent mistakes without blocking"
+    methods:
+      - "Validation"
+      - "Quotas and limits"
+      - "Approval for high-risk actions"
+      
+  transparency:
+    goal: "Users understand what's happening"
+    methods:
+      - "Clear status updates"
+      - "Audit logs"
+      - "Error messages that help"
+
+self_service_examples:
+  environment_provisioning:
+    before: "File ticket, wait 2 days"
+    after: "Click button, get environment in 10 minutes"
+    implementation:
+      - "Terraform modules"
+      - "GitOps workflow"
+      - "Web portal"
+      
+  access_management:
+    before: "File ticket, get manual approval"
+    after: "Request in portal, automatic approval for standard access"
+    implementation:
+      - "Access catalog"
+      - "Automated approval rules"
+      - "Time-bound access"
+      
+  database_requests:
+    before: "DBA runs queries on request"
+    after: "Self-service query tool with safety limits"
+    implementation:
+      - "Read replica access"
+      - "Query timeout limits"
+      - "Audit logging"
+```
+
+### Internal Developer Platform
+
+```yaml
+idp_components:
+  service_catalog:
+    purpose: "Discover available services"
+    features:
+      - "Service documentation"
+      - "Ownership information"
+      - "Dependency mapping"
+    tools: "Backstage, Port"
+    
+  self_service_portal:
+    purpose: "Request resources"
+    features:
+      - "Environment provisioning"
+      - "Access requests"
+      - "Resource creation"
+    tools: "Backstage scaffolder, custom portals"
+    
+  golden_paths:
+    purpose: "Recommended ways to do things"
+    features:
+      - "Project templates"
+      - "Best practice defaults"
+      - "Integrated tooling"
+    benefit: "Reduce decisions, improve consistency"
+```
+
+## Sustaining Toil Reduction
+
+### Ongoing Process
+
+```yaml
+continuous_improvement:
+  weekly:
+    - "Review toil from past week"
+    - "Identify new automation opportunities"
+    - "Progress on existing automation"
+    
+  monthly:
+    - "Toil metrics review"
+    - "Prioritize automation backlog"
+    - "Celebrate wins"
+    
+  quarterly:
+    - "Comprehensive toil assessment"
+    - "Set reduction targets"
+    - "Allocate engineering time"
+
+dedicated_time:
+  approach: "Reserve time specifically for toil reduction"
+  options:
+    rotation: "One person focused on automation each sprint"
+    percentage: "20% of sprint capacity for automation"
+    hack_week: "Quarterly week focused on toil reduction"
+    
+  protection: "Don't let interrupt work consume automation time"
+```
+
+### Cultural Aspects
+
+```yaml
+cultural_practices:
+  celebrate_automation:
+    - "Announce when automation ships"
+    - "Track time saved"
+    - "Recognize contributors"
+    
+  make_toil_visible:
+    - "Dashboard showing toil metrics"
+    - "Regular reports to leadership"
+    - "Include in team reviews"
+    
+  empower_team:
+    - "Anyone can propose automation"
+    - "Time allocated for experimentation"
+    - "Safe to try and fail"
+    
+  learn_from_others:
+    - "Share automation across teams"
+    - "Internal automation showcase"
+    - "Reuse common patterns"
+```
+
+## Common Pitfalls
+
+```yaml
+pitfall_over_engineering:
+  problem: "Spend more time automating than task would take"
+  example: "Week automating task that happens once a year"
+  solution: "Calculate ROI before starting"
+
+pitfall_automation_debt:
+  problem: "Automation breaks and no one fixes it"
+  example: "CI pipeline that's always red"
+  solution: "Maintain automation like production code"
+
+pitfall_false_automation:
+  problem: "Automation that still needs manual steps"
+  example: "Script that requires SSH and manual verification"
+  solution: "Full automation or don't call it automated"
+
+pitfall_local_optimization:
+  problem: "Automate symptoms instead of fixing causes"
+  example: "Auto-restart instead of fixing memory leak"
+  solution: "Ask why the toil exists"
+
+pitfall_resistance:
+  problem: "Team resistant to changing processes"
+  example: "'We've always done it this way'"
+  solution: "Demonstrate value, involve team in solutions"
+```