claude-autopm 2.8.2 → 2.8.3

package/packages/plugin-devops/rules/devops-troubleshooting-playbook.md

@@ -0,0 +1,450 @@
+# DevOps Troubleshooting Playbook
+
+This document contains standard procedures for resolving common issues in our CI/CD and Kubernetes environments. All DevOps agents MUST follow these guidelines.
+
+---
+
+## 🚀 Kubernetes CI/CD Error Debugging
+
+### Error: `PodInitializing` or `ImagePullBackOff`
+
+**Diagnosis:** This error means the Pod cannot start. The most common cause is an issue with the init container or missing resources.
+
+**Debugging Procedure:**
+
+1. **Add debug step to workflow:** In the workflow file (`.github/workflows/*.yml`), after the failed step, add a conditional step (`if: failure()`).
+
+2. **Collect diagnostic information:** In the debug step, use the following commands to gather logs before the pod is deleted:
+
+```yaml
+- name: 🐞 Debug Pod on Failure
+  if: failure()
+  run: |
+    POD_NAME=$(kubectl get pods --selector=job-name=YOUR_JOB_NAME -o jsonpath='{.items[0].metadata.name}')
+    echo "--- Pod Description ($POD_NAME) ---"
+    kubectl describe pod $POD_NAME
+    echo "--- Init Container Logs ---"
+    kubectl logs $POD_NAME -c INIT_CONTAINER_NAME
+    echo "--- Main Container Logs (if available) ---"
+    kubectl logs $POD_NAME -c main || echo "Main container not yet started"
+    echo "--- Pod Events ---"
+    kubectl get events --field-selector involvedObject.name=$POD_NAME
+```
+
+### Error: `field is immutable`
+
+**Diagnosis:** This error indicates an attempt to modify an existing, immutable Kubernetes resource (e.g., Job).
+
+**Solution (MANDATORY):** Always delete the old resource before creating a new one. This ensures each CI/CD run is clean.
+
+**Implementation Pattern in Workflow:**
+
+```yaml
+- name: 🏗️ Deploy Job with Cleanup
+  run: |
+    echo "🧹 Cleaning up previous job (if any)..."
+    kubectl delete job YOUR_JOB_NAME --ignore-not-found=true
+    
+    # Wait for cleanup to complete
+    kubectl wait --for=delete job/YOUR_JOB_NAME --timeout=30s 2>/dev/null || true
+    
+    echo "🚀 Creating new job..."
+    cat <<EOF | kubectl apply -f -
+    apiVersion: batch/v1
+    kind: Job
+    metadata:
+      name: YOUR_JOB_NAME
+    spec:
+      # ... your job definition ...
+    EOF
+```
+
+### Error: `CreateContainerConfigError`
+
+**Diagnosis:** Configuration issue preventing container creation, often related to secrets or configmaps.
+
+**Debugging Steps:**
+
+1. Check if referenced secrets/configmaps exist:
+
+```bash
+kubectl get secret YOUR_SECRET_NAME
+kubectl get configmap YOUR_CONFIG_NAME
+```
+
+2. Verify secret keys match what's expected:
+
+```bash
+kubectl describe secret YOUR_SECRET_NAME
+```
+
+3. Check pod events for detailed error:
+
+```bash
+kubectl describe pod POD_NAME | grep -A 10 Events
+```
+
+### Error: ConfigMap "..." is invalid: Too long
+
+**Diagnosis:** This error means you're trying to store data in a `ConfigMap` that exceeds the 1MB size limit.
+
+**Root Cause (Anti-pattern):** Packaging entire source code (e.g., as `.tar.gz` archive) into a `ConfigMap` to use as build context for Kaniko. ConfigMaps are designed for small configuration files, not for storing entire applications.
+
+**Solution (MANDATORY):** Deliver build context to the pod using a volume. Best practice is to use an `initContainer` that clones the repository into a shared `emptyDir` volume.
+
+**Implementation Pattern in Workflow:**
+
+```yaml
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: kaniko-build-job
+spec:
+  template:
+    spec:
+      restartPolicy: Never
+      initContainers:
+      - name: prepare-build-context
+        image: alpine/git:latest
+        command: ['sh', '-c']
+        args:
+          - |
+            git clone --depth 1 https://github.com/YOUR/REPO.git /workspace
+            cd /workspace && git checkout $COMMIT_SHA
+        volumeMounts:
+        - name: workspace
+          mountPath: /workspace
+      containers:
+      - name: kaniko
+        image: gcr.io/kaniko-project/executor:latest
+        args:
+        - --dockerfile=/workspace/Dockerfile
+        - --context=dir:///workspace
+        - --destination=YOUR_REGISTRY/IMAGE:TAG
+        - --cache=true
+        - --cache-ttl=24h
+        volumeMounts:
+        - name: workspace
+          mountPath: /workspace
+      volumes:
+      - name: workspace
+        emptyDir: {}
+```
+
+**Alternative Solutions:**
+
+1. **Git Context (Recommended for public repos):**
+
+```yaml
+args:
+  - --context=git://github.com/YOUR/REPO.git#refs/heads/main
+  - --dockerfile=Dockerfile
+```
+
+2. **S3/GCS Bucket for build context:**
+
+```yaml
+args:
+  - --context=s3://your-bucket/build-context.tar.gz
+  - --dockerfile=Dockerfile
+```
+
+3. **Registry-based builds (for layer caching):**
+
+```yaml
+args:
+  - --cache-repo=YOUR_REGISTRY/cache
+  - --cache=true
+```
+
+---
+
+## 📊 Database Management Strategy
+
+Database management strategy MUST align with environment purpose:
+
+| Environment | Tool | Purpose | Persistence | Backup Strategy |
+|------------|------|---------|-------------|-----------------|
+| **Local (Dev)** | docker compose | Convenience | Yes (Named Volume) | Optional |
+| **Testing (CI/CD)** | kubectl | Isolation | No (Ephemeral) | Not Required |
+| **Staging** | Managed DB | Pre-prod Testing | Yes (PVC) | Daily |
+| **Production** | Managed Service | Reliability | Yes (HA + Backups) | Continuous |
+
+### Development Database Setup
+
+```yaml
+# docker-compose.yml
+services:
+  postgres:
+    image: postgres:15-alpine
+    volumes:
+      - postgres_data:/var/lib/postgresql/data  # Named volume for persistence
+    environment:
+      POSTGRES_DB: ${DB_NAME:-devdb}
+      POSTGRES_USER: ${DB_USER:-developer}
+      POSTGRES_PASSWORD: ${DB_PASSWORD:-localpass}
+    ports:
+      - "5432:5432"
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U ${DB_USER:-developer}"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+
+volumes:
+  postgres_data:  # Persistent across container restarts
+```
+
+### CI/CD Database Setup
+
+```yaml
+# Ephemeral database for testing
+apiVersion: v1
+kind: Service
+metadata:
+  name: test-postgres
+spec:
+  selector:
+    app: test-postgres
+  ports:
+    - port: 5432
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: test-postgres
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: test-postgres
+  template:
+    metadata:
+      labels:
+        app: test-postgres
+    spec:
+      containers:
+      - name: postgres
+        image: postgres:15-alpine
+        env:
+        - name: POSTGRES_DB
+          value: testdb
+        - name: POSTGRES_USER
+          value: testuser
+        - name: POSTGRES_PASSWORD
+          value: testpass
+        - name: POSTGRES_HOST_AUTH_METHOD
+          value: trust  # For CI/CD only!
+        # No volume mount - data is ephemeral
+```
+
+---
+
+## 🔄 CI/CD Pipeline Best Practices
+
+### GitHub Actions Kubernetes Debugging
+
+**ALWAYS include these debug steps in Kubernetes workflows:**
+
+```yaml
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - name: 🎯 Deploy Application
+        id: deploy
+        run: |
+          # Your deployment commands
+          kubectl apply -f manifests/
+      
+      - name: 🔍 Debug on Failure
+        if: failure()
+        run: |
+          echo "=== Pod Status ==="
+          kubectl get pods -o wide
+          
+          echo "=== Recent Events ==="
+          kubectl get events --sort-by='.lastTimestamp' | tail -20
+          
+          echo "=== Failed Pods Details ==="
+          kubectl get pods --field-selector=status.phase!=Running,status.phase!=Succeeded -o json | \
+            jq -r '.items[] | "\(.metadata.name): \(.status.containerStatuses[0].state)"'
+          
+          echo "=== Resource Usage ==="
+          kubectl top nodes || echo "Metrics not available"
+          kubectl top pods || echo "Metrics not available"
+```
+
+### Container Build Optimization
+
+**Cache layers effectively in CI/CD:**
+
+```dockerfile
+# Good: Dependencies first (cached often)
+FROM node:18-alpine AS deps
+WORKDIR /app
+COPY package*.json ./
+RUN npm ci --only=production
+
+# Bad: Everything at once (cache busted on any change)
+FROM node:18-alpine
+COPY . .
+RUN npm install
+```
+
+---
+
+## 🛡️ Security Best Practices
+
+### Secret Management
+
+**NEVER hardcode secrets. Use this hierarchy:**
+
+1. **Local Development:** `.env` files (gitignored)
+2. **CI/CD:** GitHub Secrets / Environment Variables
+3. **Kubernetes:** Secrets or External Secrets Operator
+4. **Production:** Vault, AWS Secrets Manager, or Azure Key Vault
+
+**Example Secret Injection Pattern:**
+
+```yaml
+# GitHub Actions
+- name: 🔐 Inject Secrets
+  run: |
+    kubectl create secret generic app-secrets \
+      --from-literal=db-password=${{ secrets.DB_PASSWORD }} \
+      --from-literal=api-key=${{ secrets.API_KEY }} \
+      --dry-run=client -o yaml | kubectl apply -f -
+```
+
+---
+
+## 🔧 Common Debugging Commands
+
+### Quick Diagnostics Checklist
+
+```bash
+# 1. Check pod status
+kubectl get pods -A | grep -v Running
+
+# 2. Get pod logs
+kubectl logs POD_NAME --previous  # If pod restarted
+kubectl logs POD_NAME -f          # Follow logs
+
+# 3. Shell into pod
+kubectl exec -it POD_NAME -- /bin/sh
+
+# 4. Check resource consumption
+kubectl top pods
+kubectl describe node
+
+# 5. Check recent errors
+kubectl get events --sort-by='.lastTimestamp' | grep Warning
+
+# 6. Verify service endpoints
+kubectl get endpoints
+
+# 7. Test service connectivity
+kubectl run debug --image=busybox:1.28 --rm -it --restart=Never -- wget -O- SERVICE_NAME:PORT
+```
+
+### Emergency Rollback Procedure
+
+```bash
+# 1. Quick rollback to previous deployment
+kubectl rollout undo deployment/APP_NAME
+
+# 2. Check rollback status
+kubectl rollout status deployment/APP_NAME
+
+# 3. Verify pods are healthy
+kubectl get pods -l app=APP_NAME
+
+# 4. If still broken, scale to zero and debug
+kubectl scale deployment/APP_NAME --replicas=0
+# Fix issues...
+kubectl scale deployment/APP_NAME --replicas=3
+```
+
+---
+
+## 📝 Logging and Monitoring
+
+### Structured Logging Requirements
+
+All applications MUST:
+
+1. Log in JSON format for parsing
+2. Include correlation IDs for request tracing
+3. Use appropriate log levels (DEBUG, INFO, WARN, ERROR)
+4. Never log sensitive data (passwords, tokens, PII)
+
+**Example Structured Log:**
+
+```json
+{
+  "timestamp": "2024-01-15T10:30:00Z",
+  "level": "ERROR",
+  "correlation_id": "abc-123-def",
+  "service": "api-gateway",
+  "message": "Database connection failed",
+  "error": "connection timeout after 30s",
+  "retry_count": 3
+}
+```
+
+---
+
+## 🚨 Incident Response Template
+
+When production issues occur, follow this template:
+
+```markdown
+## Incident Report
+
+**Date:** YYYY-MM-DD HH:MM UTC
+**Severity:** Critical | High | Medium | Low
+**Duration:** XX minutes
+
+### Impact
+- What broke and who was affected
+
+### Root Cause
+- Technical reason for the failure
+
+### Resolution
+- Steps taken to fix
+
+### Prevention
+- Changes to prevent recurrence
+
+### Timeline
+- HH:MM - Issue detected
+- HH:MM - Team notified
+- HH:MM - Root cause identified
+- HH:MM - Fix deployed
+- HH:MM - Service restored
+```
+
+---
+
+## 🎯 Golden Rules
+
+1. **Always clean before create** - Delete old resources before creating new ones
+2. **Debug immediately on failure** - Add conditional debug steps to all CI/CD workflows
+3. **Never assume, always verify** - Check that resources exist before using them
+4. **Fail fast, recover faster** - Quick detection and rollback procedures
+5. **Document everything** - If it's not documented, it didn't happen
+
+---
+
+## 📚 References
+
+- [Kubernetes Troubleshooting Guide](https://kubernetes.io/docs/tasks/debug/)
+- [GitHub Actions Best Practices](https://docs.github.com/en/actions/guides)
+- [12 Factor App Methodology](https://12factor.net/)
+- [SRE Principles](https://sre.google/sre-book/table-of-contents/)
+
+---
+
+**Remember:** This playbook is a living document. Update it with new solutions as we discover them.