claude-autopm 2.8.1 → 2.8.2

package/autopm/.claude/rules/devops-troubleshooting-playbook.md

@@ -1,450 +0,0 @@
-# DevOps Troubleshooting Playbook
-
-This document contains standard procedures for resolving common issues in our CI/CD and Kubernetes environments. All DevOps agents MUST follow these guidelines.
-
----
-
-## 🚀 Kubernetes CI/CD Error Debugging
-
-### Error: `PodInitializing` or `ImagePullBackOff`
-
-**Diagnosis:** This error means the Pod cannot start. The most common cause is an issue with the init container or missing resources.
-
-**Debugging Procedure:**
-
-1. **Add debug step to workflow:** In the workflow file (`.github/workflows/*.yml`), after the failed step, add a conditional step (`if: failure()`).
-
-2. **Collect diagnostic information:** In the debug step, use the following commands to gather logs before the pod is deleted:
-
-```yaml
-- name: 🐞 Debug Pod on Failure
-  if: failure()
-  run: |
-    POD_NAME=$(kubectl get pods --selector=job-name=YOUR_JOB_NAME -o jsonpath='{.items[0].metadata.name}')
-    echo "--- Pod Description ($POD_NAME) ---"
-    kubectl describe pod $POD_NAME
-    echo "--- Init Container Logs ---"
-    kubectl logs $POD_NAME -c INIT_CONTAINER_NAME
-    echo "--- Main Container Logs (if available) ---"
-    kubectl logs $POD_NAME -c main || echo "Main container not yet started"
-    echo "--- Pod Events ---"
-    kubectl get events --field-selector involvedObject.name=$POD_NAME
-```
-
-### Error: `field is immutable`
-
-**Diagnosis:** This error indicates an attempt to modify an existing, immutable Kubernetes resource (e.g., Job).
-
-**Solution (MANDATORY):** Always delete the old resource before creating a new one. This ensures each CI/CD run is clean.
-
-**Implementation Pattern in Workflow:**
-
-```yaml
-- name: 🏗️ Deploy Job with Cleanup
-  run: |
-    echo "🧹 Cleaning up previous job (if any)..."
-    kubectl delete job YOUR_JOB_NAME --ignore-not-found=true
-    
-    # Wait for cleanup to complete
-    kubectl wait --for=delete job/YOUR_JOB_NAME --timeout=30s 2>/dev/null || true
-    
-    echo "🚀 Creating new job..."
-    cat <<EOF | kubectl apply -f -
-    apiVersion: batch/v1
-    kind: Job
-    metadata:
-      name: YOUR_JOB_NAME
-    spec:
-      # ... your job definition ...
-    EOF
-```
-
-### Error: `CreateContainerConfigError`
-
-**Diagnosis:** Configuration issue preventing container creation, often related to secrets or configmaps.
-
-**Debugging Steps:**
-
-1. Check if referenced secrets/configmaps exist:
-
-```bash
-kubectl get secret YOUR_SECRET_NAME
-kubectl get configmap YOUR_CONFIG_NAME
-```
-
-2. Verify secret keys match what's expected:
-
-```bash
-kubectl describe secret YOUR_SECRET_NAME
-```
-
-3. Check pod events for detailed error:
-
-```bash
-kubectl describe pod POD_NAME | grep -A 10 Events
-```
-
-### Error: ConfigMap "..." is invalid: Too long
-
-**Diagnosis:** This error means you're trying to store data in a `ConfigMap` that exceeds the 1MB size limit.
-
-**Root Cause (Anti-pattern):** Packaging entire source code (e.g., as `.tar.gz` archive) into a `ConfigMap` to use as build context for Kaniko. ConfigMaps are designed for small configuration files, not for storing entire applications.
-
-**Solution (MANDATORY):** Deliver build context to the pod using a volume. Best practice is to use an `initContainer` that clones the repository into a shared `emptyDir` volume.
-
-**Implementation Pattern in Workflow:**
-
-```yaml
-apiVersion: batch/v1
-kind: Job
-metadata:
-  name: kaniko-build-job
-spec:
-  template:
-    spec:
-      restartPolicy: Never
-      initContainers:
-      - name: prepare-build-context
-        image: alpine/git:latest
-        command: ['sh', '-c']
-        args:
-          - |
-            git clone --depth 1 https://github.com/YOUR/REPO.git /workspace
-            cd /workspace && git checkout $COMMIT_SHA
-        volumeMounts:
-        - name: workspace
-          mountPath: /workspace
-      containers:
-      - name: kaniko
-        image: gcr.io/kaniko-project/executor:latest
-        args:
-        - --dockerfile=/workspace/Dockerfile
-        - --context=dir:///workspace
-        - --destination=YOUR_REGISTRY/IMAGE:TAG
-        - --cache=true
-        - --cache-ttl=24h
-        volumeMounts:
-        - name: workspace
-          mountPath: /workspace
-      volumes:
-      - name: workspace
-        emptyDir: {}
-```
-
-**Alternative Solutions:**
-
-1. **Git Context (Recommended for public repos):**
-
-```yaml
-args:
-  - --context=git://github.com/YOUR/REPO.git#refs/heads/main
-  - --dockerfile=Dockerfile
-```
-
-2. **S3/GCS Bucket for build context:**
-
-```yaml
-args:
-  - --context=s3://your-bucket/build-context.tar.gz
-  - --dockerfile=Dockerfile
-```
-
-3. **Registry-based builds (for layer caching):**
-
-```yaml
-args:
-  - --cache-repo=YOUR_REGISTRY/cache
-  - --cache=true
-```
-
----
-
-## 📊 Database Management Strategy
-
-Database management strategy MUST align with environment purpose:
-
-| Environment | Tool | Purpose | Persistence | Backup Strategy |
-|------------|------|---------|-------------|-----------------|
-| **Local (Dev)** | docker compose | Convenience | Yes (Named Volume) | Optional |
-| **Testing (CI/CD)** | kubectl | Isolation | No (Ephemeral) | Not Required |
-| **Staging** | Managed DB | Pre-prod Testing | Yes (PVC) | Daily |
-| **Production** | Managed Service | Reliability | Yes (HA + Backups) | Continuous |
-
-### Development Database Setup
-
-```yaml
-# docker-compose.yml
-services:
-  postgres:
-    image: postgres:15-alpine
-    volumes:
-      - postgres_data:/var/lib/postgresql/data  # Named volume for persistence
-    environment:
-      POSTGRES_DB: ${DB_NAME:-devdb}
-      POSTGRES_USER: ${DB_USER:-developer}
-      POSTGRES_PASSWORD: ${DB_PASSWORD:-localpass}
-    ports:
-      - "5432:5432"
-    healthcheck:
-      test: ["CMD-SHELL", "pg_isready -U ${DB_USER:-developer}"]
-      interval: 10s
-      timeout: 5s
-      retries: 5
-
-volumes:
-  postgres_data:  # Persistent across container restarts
-```
-
-### CI/CD Database Setup
-
-```yaml
-# Ephemeral database for testing
-apiVersion: v1
-kind: Service
-metadata:
-  name: test-postgres
-spec:
-  selector:
-    app: test-postgres
-  ports:
-    - port: 5432
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: test-postgres
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app: test-postgres
-  template:
-    metadata:
-      labels:
-        app: test-postgres
-    spec:
-      containers:
-      - name: postgres
-        image: postgres:15-alpine
-        env:
-        - name: POSTGRES_DB
-          value: testdb
-        - name: POSTGRES_USER
-          value: testuser
-        - name: POSTGRES_PASSWORD
-          value: testpass
-        - name: POSTGRES_HOST_AUTH_METHOD
-          value: trust  # For CI/CD only!
-        # No volume mount - data is ephemeral
-```
-
----
-
-## 🔄 CI/CD Pipeline Best Practices
-
-### GitHub Actions Kubernetes Debugging
-
-**ALWAYS include these debug steps in Kubernetes workflows:**
-
-```yaml
-jobs:
-  deploy:
-    runs-on: ubuntu-latest
-    steps:
-      - name: 🎯 Deploy Application
-        id: deploy
-        run: |
-          # Your deployment commands
-          kubectl apply -f manifests/
-      
-      - name: 🔍 Debug on Failure
-        if: failure()
-        run: |
-          echo "=== Pod Status ==="
-          kubectl get pods -o wide
-          
-          echo "=== Recent Events ==="
-          kubectl get events --sort-by='.lastTimestamp' | tail -20
-          
-          echo "=== Failed Pods Details ==="
-          kubectl get pods --field-selector=status.phase!=Running,status.phase!=Succeeded -o json | \
-            jq -r '.items[] | "\(.metadata.name): \(.status.containerStatuses[0].state)"'
-          
-          echo "=== Resource Usage ==="
-          kubectl top nodes || echo "Metrics not available"
-          kubectl top pods || echo "Metrics not available"
-```
-
-### Container Build Optimization
-
-**Cache layers effectively in CI/CD:**
-
-```dockerfile
-# Good: Dependencies first (cached often)
-FROM node:18-alpine AS deps
-WORKDIR /app
-COPY package*.json ./
-RUN npm ci --only=production
-
-# Bad: Everything at once (cache busted on any change)
-FROM node:18-alpine
-COPY . .
-RUN npm install
-```
-
----
-
-## 🛡️ Security Best Practices
-
-### Secret Management
-
-**NEVER hardcode secrets. Use this hierarchy:**
-
-1. **Local Development:** `.env` files (gitignored)
-2. **CI/CD:** GitHub Secrets / Environment Variables
-3. **Kubernetes:** Secrets or External Secrets Operator
-4. **Production:** Vault, AWS Secrets Manager, or Azure Key Vault
-
-**Example Secret Injection Pattern:**
-
-```yaml
-# GitHub Actions
-- name: 🔐 Inject Secrets
-  run: |
-    kubectl create secret generic app-secrets \
-      --from-literal=db-password=${{ secrets.DB_PASSWORD }} \
-      --from-literal=api-key=${{ secrets.API_KEY }} \
-      --dry-run=client -o yaml | kubectl apply -f -
-```
-
----
-
-## 🔧 Common Debugging Commands
-
-### Quick Diagnostics Checklist
-
-```bash
-# 1. Check pod status
-kubectl get pods -A | grep -v Running
-
-# 2. Get pod logs
-kubectl logs POD_NAME --previous  # If pod restarted
-kubectl logs POD_NAME -f          # Follow logs
-
-# 3. Shell into pod
-kubectl exec -it POD_NAME -- /bin/sh
-
-# 4. Check resource consumption
-kubectl top pods
-kubectl describe node
-
-# 5. Check recent errors
-kubectl get events --sort-by='.lastTimestamp' | grep Warning
-
-# 6. Verify service endpoints
-kubectl get endpoints
-
-# 7. Test service connectivity
-kubectl run debug --image=busybox:1.28 --rm -it --restart=Never -- wget -O- SERVICE_NAME:PORT
-```
-
-### Emergency Rollback Procedure
-
-```bash
-# 1. Quick rollback to previous deployment
-kubectl rollout undo deployment/APP_NAME
-
-# 2. Check rollback status
-kubectl rollout status deployment/APP_NAME
-
-# 3. Verify pods are healthy
-kubectl get pods -l app=APP_NAME
-
-# 4. If still broken, scale to zero and debug
-kubectl scale deployment/APP_NAME --replicas=0
-# Fix issues...
-kubectl scale deployment/APP_NAME --replicas=3
-```
-
----
-
-## 📝 Logging and Monitoring
-
-### Structured Logging Requirements
-
-All applications MUST:
-
-1. Log in JSON format for parsing
-2. Include correlation IDs for request tracing
-3. Use appropriate log levels (DEBUG, INFO, WARN, ERROR)
-4. Never log sensitive data (passwords, tokens, PII)
-
-**Example Structured Log:**
-
-```json
-{
-  "timestamp": "2024-01-15T10:30:00Z",
-  "level": "ERROR",
-  "correlation_id": "abc-123-def",
-  "service": "api-gateway",
-  "message": "Database connection failed",
-  "error": "connection timeout after 30s",
-  "retry_count": 3
-}
-```
-
----
-
-## 🚨 Incident Response Template
-
-When production issues occur, follow this template:
-
-```markdown
-## Incident Report
-
-**Date:** YYYY-MM-DD HH:MM UTC
-**Severity:** Critical | High | Medium | Low
-**Duration:** XX minutes
-
-### Impact
-- What broke and who was affected
-
-### Root Cause
-- Technical reason for the failure
-
-### Resolution
-- Steps taken to fix
-
-### Prevention
-- Changes to prevent recurrence
-
-### Timeline
-- HH:MM - Issue detected
-- HH:MM - Team notified
-- HH:MM - Root cause identified
-- HH:MM - Fix deployed
-- HH:MM - Service restored
-```
-
----
-
-## 🎯 Golden Rules
-
-1. **Always clean before create** - Delete old resources before creating new ones
-2. **Debug immediately on failure** - Add conditional debug steps to all CI/CD workflows
-3. **Never assume, always verify** - Check that resources exist before using them
-4. **Fail fast, recover faster** - Quick detection and rollback procedures
-5. **Document everything** - If it's not documented, it didn't happen
-
----
-
-## 📚 References
-
-- [Kubernetes Troubleshooting Guide](https://kubernetes.io/docs/tasks/debug/)
-- [GitHub Actions Best Practices](https://docs.github.com/en/actions/guides)
-- [12 Factor App Methodology](https://12factor.net/)
-- [SRE Principles](https://sre.google/sre-book/table-of-contents/)
-
----
-
-**Remember:** This playbook is a living document. Update it with new solutions as we discover them.