ai-playbook 1.0.0

package/roles/devops/ci-cd/checklists.md

@@ -0,0 +1,289 @@
+# CI/CD Pipeline Checklists
+
+Detailed checklists for CI/CD pipeline design and implementation.
+
+---
+
+## CI/CD Pipeline Design Checklist
+
+### Requirements & Goals
+- [ ] Business requirements for pipeline speed understood
+- [ ] Deployment frequency target defined (daily, weekly, on-demand)
+- [ ] Approval workflows and gates defined
+- [ ] Rollout strategy chosen (rolling, blue-green, canary)
+- [ ] Target deployment platforms identified (Kubernetes, VMs, serverless)
+- [ ] Success metrics defined (deployment speed, failure rate, rollback rate)
+
+### Pipeline Architecture
+- [ ] Source repository structure documented
+- [ ] Branching strategy defined (main, develop, feature branches)
+- [ ] Pipeline trigger events identified (push, PR, schedule)
+- [ ] Parallel execution opportunities identified
+- [ ] Pipeline dependencies mapped
+- [ ] Build artifact strategy defined
+- [ ] Secrets and credentials strategy defined
+
+### Build Stage Design
+- [ ] Build tool/platform selected
+- [ ] Dependency management approach defined
+- [ ] Caching strategy for dependencies defined
+- [ ] Build versioning scheme defined
+- [ ] Build artifact storage configured
+- [ ] Build timeout appropriate for typical build time
+
+### Test Stage Design
+- [ ] Unit test execution configured
+- [ ] Test coverage thresholds defined
+- [ ] Integration test strategy defined
+- [ ] Test result reporting configured
+- [ ] Test flakiness tracking implemented
+- [ ] Long-running tests parallelized (if applicable)
+
+### Security Scanning Design
+- [ ] Dependency scanning tool selected
+- [ ] Dependency scanning frequency defined
+- [ ] SAST scanning tool selected and configured
+- [ ] Container scanning strategy defined
+- [ ] Secrets scanning enabled
+- [ ] Security scan result review process defined
+- [ ] Security scan failure policy (block vs. warn) defined
+
+### Artifact Management
+- [ ] Artifact registry/repository selected
+- [ ] Artifact versioning scheme defined
+- [ ] Container image naming convention defined
+- [ ] Artifact retention policy defined
+- [ ] Artifact signature/integrity verification configured
+- [ ] Artifact pull and authentication configured
+
+### Deployment Design
+- [ ] Deployment strategy selected (rolling/blue-green/canary)
+- [ ] Staging environment available for testing
+- [ ] Production environment infrastructure ready
+- [ ] Deployment automation tool selected
+- [ ] Blue-green/canary traffic shifting strategy defined
+- [ ] Health checks for deployment validation defined
+- [ ] Rollback procedure documented
+
+### Approval & Release
+- [ ] Approval process defined for production deployments
+- [ ] Approvers identified and notified
+- [ ] SLA for approval defined
+- [ ] Release notes generation automated
+- [ ] Change tracking/ticketing integration configured
+- [ ] Deployment notifications configured
+
+---
+
+## GitHub Actions Implementation Checklist
+
+### Workflow Setup
+- [ ] `.github/workflows/` directory created
+- [ ] Workflow file naming convention established
+- [ ] Workflow triggers defined (push, PR, schedule)
+- [ ] Environment variables defined
+- [ ] Secrets configured in GitHub settings
+- [ ] Workflow permissions configured (GITHUB_TOKEN scope)
+
+### Build Job
+- [ ] Runner OS selected (ubuntu-latest, macos-latest, windows-latest)
+- [ ] Dependency caching configured (actions/cache)
+- [ ] Build step runs successfully locally first
+- [ ] Build step outputs artifacts
+- [ ] Build step set correct exit codes on failure
+
+### Test Job
+- [ ] Test runner configured
+- [ ] Test discovery automated
+- [ ] Test results reported (JUnit XML, Cobertura)
+- [ ] Coverage thresholds enforced
+- [ ] Failed tests cause workflow failure
+
+### Security Scanning Job
+- [ ] Dependency scanning configured (Dependabot)
+- [ ] Container scanning configured (if applicable)
+- [ ] Secrets scanning enabled
+- [ ] SAST scanning integrated (CodeQL, SonarQube)
+
+### Deployment Job
+- [ ] Deployment credentials stored as secrets
+- [ ] Deployment tool configured (kubectl, Helm, Terraform)
+- [ ] Staging deployment works consistently
+- [ ] Health checks validate deployment
+- [ ] Rollback procedure tested
+
+### Production Deployment
+- [ ] Environment protection rules configured
+- [ ] Approval requirements set for production
+- [ ] Deployment notifications configured
+- [ ] Monitoring alerts configured post-deployment
+
+### Error Handling & Notifications
+- [ ] Failed workflow notifications sent
+- [ ] Status badges configured for readme
+- [ ] Slack integration configured (if used)
+- [ ] PagerDuty integration configured (if used)
+
+---
+
+## GitLab CI Implementation Checklist
+
+### Pipeline Configuration
+- [ ] `.gitlab-ci.yml` created in repository root
+- [ ] Stages defined (build, test, security, deploy)
+- [ ] Pipeline triggers configured
+- [ ] Variables defined (CI_COMMIT_SHA, custom vars)
+- [ ] Protected variables configured for secrets
+- [ ] Pipeline concurrency limits defined
+
+### Docker & Containers
+- [ ] Docker image specified for runners
+- [ ] Docker-in-Docker enabled (if needed)
+- [ ] Container registry URL configured
+- [ ] Container registry credentials configured
+- [ ] Artifact caching configured
+
+### Runners
+- [ ] Runner type selected (Docker, machine, shell)
+- [ ] Runner tags defined and applied
+- [ ] Runner resource limits appropriate
+- [ ] Runner concurrency configured
+
+### Artifacts & Caching
+- [ ] Artifacts configured with retention policy
+- [ ] Cache directories configured
+- [ ] Cache keys optimized to avoid conflicts
+- [ ] Artifact cleanup policy set
+
+### Testing & Code Quality
+- [ ] Test jobs parallelized
+- [ ] Test coverage reported
+- [ ] Code quality gates configured
+- [ ] Test report merge into MR
+
+### Deployment
+- [ ] Manual deployment jobs created (prod gates)
+- [ ] Auto DevOps configured (if used)
+- [ ] Environment-specific configuration
+- [ ] Deployment history visible
+
+---
+
+## Jenkins Pipeline Checklist
+
+### Jenkins Setup
+- [ ] Jenkins instance provisioned and configured
+- [ ] Pipeline plugin installed
+- [ ] Credentials plugin configured
+- [ ] Docker plugin installed (if using Docker agents)
+- [ ] Git plugin configured
+
+### Jenkinsfile
+- [ ] `Jenkinsfile` created in repository
+- [ ] Declarative or Scripted pipeline syntax chosen
+- [ ] Stages defined
+- [ ] Post-build actions configured
+- [ ] Notifications configured
+
+### Pipeline Stages
+- [ ] Checkout stage checks out code
+- [ ] Build stage compiles/packages code
+- [ ] Test stage runs tests
+- [ ] Security scanning stage configured
+- [ ] Deploy stage automates deployment
+- [ ] Validation stage confirms deployment success
+
+### Agents & Resources
+- [ ] Agent labels defined
+- [ ] Resource requirements specified
+- [ ] Docker agents configured (if applicable)
+- [ ] Agent cleanup configured
+
+### Credentials Management
+- [ ] Credentials stored in Jenkins credentials store
+- [ ] Secret masking in logs enabled
+- [ ] Credential scoping (global, folder, etc.) configured
+
+### Webhooks & Triggers
+- [ ] GitHub/GitLab webhooks configured
+- [ ] Poll SCM configured (if webhooks unavailable)
+- [ ] Pipeline triggers (other pipelines) configured
+
+### Notifications
+- [ ] Email notifications configured
+- [ ] Slack notifications configured
+- [ ] Build status badges configured
+
+---
+
+## Deployment Strategy Testing Checklist
+
+### Rolling Deployment
+- [ ] Old instances gradually terminated as new ones start
+- [ ] Health checks configured to prevent bad deployments
+- [ ] Rollback tested and works correctly
+- [ ] Connection draining configured (if load balancer)
+- [ ] Database schema changes backward compatible
+
+### Blue-Green Deployment
+- [ ] Two identical production environments prepared
+- [ ] Traffic switch mechanism configured (DNS, load balancer)
+- [ ] Switch tested (blue→green and green→blue)
+- [ ] Rollback can happen instantly
+- [ ] Parallel testing possible in both environments
+- [ ] Resource doubling cost accepted
+
+### Canary Deployment
+- [ ] Traffic splitting configured (5-10% to canary)
+- [ ] Metrics monitored during rollout
+- [ ] Automated rollback on error rate/latency threshold
+- [ ] Gradual rollout schedule defined
+- [ ] Health checks validate canary
+- [ ] Canary version tagged in monitoring
+
+---
+
+## Post-Deployment Validation Checklist
+
+### Immediate Validation
+- [ ] Application health checks passing
+- [ ] Error rate normal or lower
+- [ ] Response latency normal
+- [ ] Database queries performing normally
+- [ ] External integrations working
+
+### Monitoring & Alerting
+- [ ] Dashboards updated and accessible
+- [ ] Alerts configured for new services/features
+- [ ] Alert thresholds tuned
+- [ ] Log aggregation working
+
+### Testing Post-Deployment
+- [ ] Smoke tests automated and passing
+- [ ] Integration tests automated and passing
+- [ ] E2E tests covering critical paths
+- [ ] Performance tests passing
+
+### Documentation
+- [ ] Release notes published
+- [ ] Runbook updated
+- [ ] Known issues documented
+- [ ] Rollback procedures documented
+
+---
+
+## Pipeline Troubleshooting Checklist
+
+When pipeline fails:
+
+- [ ] Check build logs for error messages
+- [ ] Verify dependencies are accessible
+- [ ] Check secrets and credentials are configured
+- [ ] Verify deployment target is accessible
+- [ ] Check for resource exhaustion (disk, memory)
+- [ ] Review recent code changes
+- [ ] Test locally to reproduce issue
+- [ ] Check for timeouts (network, build)
+- [ ] Verify environment configuration
+- [ ] Review system logs on build agents
+

package/roles/devops/ci-cd/examples.md

@@ -0,0 +1,429 @@
+# CI/CD Pipeline Examples
+
+Concrete examples of CI/CD pipeline configurations.
+
+---
+
+## GitHub Actions Example
+
+See `roles/devops/examples.md` for a complete GitHub Actions workflow example covering:
+- Build and push Docker image
+- Unit tests with coverage
+- Security scanning (Trivy)
+- Staging deployment
+- Production deployment with approval
+- Smoke tests
+
+---
+
+## GitLab CI Configuration Example
+
+```yaml
+# .gitlab-ci.yml
+stages:
+  - build
+  - test
+  - security
+  - deploy
+
+variables:
+  REGISTRY: registry.gitlab.com
+  IMAGE_NAME: $CI_PROJECT_PATH
+  IMAGE_TAG: $CI_COMMIT_SHA
+
+default:
+  image: docker:24
+  services:
+    - docker:24-dind
+
+build:
+  stage: build
+  script:
+    - docker build -t $REGISTRY/$IMAGE_NAME:$IMAGE_TAG .
+    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
+    - docker push $REGISTRY/$IMAGE_NAME:$IMAGE_TAG
+  only:
+    - main
+    - develop
+
+test:
+  stage: test
+  image: golang:1.21
+  before_script:
+    - go mod download
+  script:
+    - go test -v -race -coverprofile=coverage.out ./...
+    - go tool cover -html=coverage.out -o coverage.html
+  artifacts:
+    paths:
+      - coverage.html
+    reports:
+      coverage_report:
+        coverage_format: cobertura
+        path: coverage.out
+  coverage: '/total coverage: \d+\.\d+/'
+
+security:scanning:
+  stage: security
+  image: aquasec/trivy:latest
+  script:
+    - trivy image --exit-code 0 --severity HIGH,CRITICAL --format template --template '@/contrib/html.tpl' -o report.html $REGISTRY/$IMAGE_NAME:$IMAGE_TAG
+  artifacts:
+    paths:
+      - report.html
+  only:
+    - main
+    - develop
+
+deploy:staging:
+  stage: deploy
+  image: bitnami/kubectl:latest
+  script:
+    - kubectl set image deployment/myapp myapp=$REGISTRY/$IMAGE_NAME:$IMAGE_TAG -n staging
+    - kubectl rollout status deployment/myapp -n staging
+  environment:
+    name: staging
+    url: https://staging.example.com
+  only:
+    - develop
+
+deploy:production:
+  stage: deploy
+  image: bitnami/kubectl:latest
+  script:
+    - kubectl set image deployment/myapp myapp=$REGISTRY/$IMAGE_NAME:$IMAGE_TAG -n production
+    - kubectl rollout status deployment/myapp -n production
+  environment:
+    name: production
+    url: https://example.com
+  only:
+    - main
+  when: manual
+```
+
+---
+
+## Jenkins Declarative Pipeline Example
+
+```groovy
+// Jenkinsfile
+pipeline {
+  agent any
+  
+  options {
+    timeout(time: 1, unit: 'HOURS')
+    timestamps()
+    buildDiscarder(logRotator(numToKeepStr: '10'))
+  }
+  
+  environment {
+    REGISTRY = 'docker.io'
+    IMAGE_NAME = 'myapp'
+    IMAGE_TAG = "${BUILD_NUMBER}"
+    DOCKER_CREDENTIALS = credentials('docker-registry')
+  }
+  
+  stages {
+    stage('Checkout') {
+      steps {
+        checkout scm
+      }
+    }
+    
+    stage('Build') {
+      steps {
+        script {
+          sh '''
+            docker build -t ${REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG} .
+            docker tag ${REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG} ${REGISTRY}/${IMAGE_NAME}:latest
+          '''
+        }
+      }
+    }
+    
+    stage('Test') {
+      steps {
+        script {
+          sh '''
+            docker run --rm ${REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG} \
+              go test -v -race -coverprofile=coverage.out ./...
+          '''
+        }
+      }
+    }
+    
+    stage('Security Scan') {
+      steps {
+        script {
+          sh '''
+            trivy image --exit-code 0 --severity HIGH,CRITICAL \
+              ${REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG}
+          '''
+        }
+      }
+    }
+    
+    stage('Push Image') {
+      when {
+        branch 'main'
+      }
+      steps {
+        script {
+          sh '''
+            echo $DOCKER_CREDENTIALS_PSW | docker login -u $DOCKER_CREDENTIALS_USR --password-stdin
+            docker push ${REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG}
+            docker push ${REGISTRY}/${IMAGE_NAME}:latest
+          '''
+        }
+      }
+    }
+    
+    stage('Deploy to Staging') {
+      when {
+        branch 'develop'
+      }
+      steps {
+        script {
+          sh '''
+            kubectl set image deployment/myapp \
+              myapp=${REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG} \
+              -n staging --record
+            kubectl rollout status deployment/myapp -n staging
+          '''
+        }
+      }
+    }
+    
+    stage('Deploy to Production') {
+      when {
+        branch 'main'
+      }
+      input {
+        message "Deploy to production?"
+        ok "Deploy"
+      }
+      steps {
+        script {
+          sh '''
+            kubectl set image deployment/myapp \
+              myapp=${REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG} \
+              -n production --record
+            kubectl rollout status deployment/myapp -n production
+          '''
+        }
+      }
+    }
+    
+    stage('Smoke Tests') {
+      when {
+        branch 'main'
+      }
+      steps {
+        script {
+          sh '''
+            ./scripts/smoke-tests.sh https://example.com
+          '''
+        }
+      }
+    }
+  }
+  
+  post {
+    always {
+      cleanWs()
+    }
+    
+    failure {
+      script {
+        sh '''
+          curl -X POST -H 'Content-type: application/json' \
+            --data '{"text":"Build ${BUILD_NUMBER} failed"}' \
+            $SLACK_WEBHOOK_URL
+        '''
+      }
+    }
+    
+    success {
+      script {
+        sh '''
+          curl -X POST -H 'Content-type: application/json' \
+            --data '{"text":"Build ${BUILD_NUMBER} succeeded"}' \
+            $SLACK_WEBHOOK_URL
+        '''
+      }
+    }
+  }
+}
+```
+
+---
+
+## Canary Deployment Example
+
+```yaml
+# .github/workflows/canary-deploy.yml
+name: Canary Deployment
+
+on:
+  workflow_dispatch:
+    inputs:
+      percentage:
+        description: 'Traffic percentage for canary'
+        required: true
+        default: '5'
+
+jobs:
+  deploy-canary:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    
+    - name: Configure kubectl
+      run: |
+        mkdir -p $HOME/.kube
+        echo "${{ secrets.KUBE_CONFIG }}" | base64 -d > $HOME/.kube/config
+    
+    - name: Deploy canary version
+      run: |
+        kubectl set image deployment/myapp-canary \
+          myapp=${{ env.IMAGE_NAME }}:${{ github.sha }} \
+          -n production
+    
+    - name: Configure traffic split
+      run: |
+        cat <<EOF | kubectl apply -f -
+        apiVersion: networking.istio.io/v1beta1
+        kind: VirtualService
+        metadata:
+          name: myapp
+          namespace: production
+        spec:
+          hosts:
+          - myapp
+          http:
+          - match:
+            - uri:
+                prefix: /
+            route:
+            - destination:
+                host: myapp
+                subset: stable
+              weight: $((100 - ${{ github.event.inputs.percentage }}))
+            - destination:
+                host: myapp
+                subset: canary
+              weight: ${{ github.event.inputs.percentage }}
+        EOF
+    
+    - name: Monitor canary metrics
+      run: |
+        ./scripts/monitor-canary.sh "${{ github.event.inputs.percentage }}"
+    
+    - name: Automatic rollback on failure
+      if: failure()
+      run: |
+        # Reset traffic to 100% stable
+        kubectl patch vs myapp -n production --patch '{"spec":{"http":[{"route":[{"destination":{"host":"myapp","subset":"stable"},"weight":100}]}]}}'
+        # Revert canary deployment
+        kubectl rollout undo deployment/myapp-canary -n production
+```
+
+---
+
+## Multi-Environment Deployment Example
+
+```yaml
+# .github/workflows/multi-env-deploy.yml
+name: Multi-Environment Deploy
+
+on:
+  push:
+    branches:
+      - main
+      - develop
+      - 'release/**'
+
+env:
+  REGISTRY: ghcr.io
+  IMAGE_NAME: ${{ github.repository }}
+
+jobs:
+  build-and-test:
+    runs-on: ubuntu-latest
+    outputs:
+      image-tag: ${{ steps.meta.outputs.tags }}
+    steps:
+    - uses: actions/checkout@v4
+    
+    - name: Build and test
+      run: |
+        docker build -t test-image .
+        docker run --rm test-image go test -v ./...
+    
+    - name: Build production image
+      run: |
+        docker build -t ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} .
+        docker push ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
+
+  deploy-dev:
+    runs-on: ubuntu-latest
+    needs: build-and-test
+    if: github.ref == 'refs/heads/develop'
+    environment: development
+    steps:
+    - uses: actions/checkout@v4
+    - name: Deploy to development
+      run: |
+        kubectl set image deployment/myapp \
+          myapp=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} \
+          -n dev
+
+  deploy-staging:
+    runs-on: ubuntu-latest
+    needs: build-and-test
+    if: startsWith(github.ref, 'refs/heads/release/')
+    environment: staging
+    steps:
+    - uses: actions/checkout@v4
+    - name: Deploy to staging
+      run: |
+        kubectl set image deployment/myapp \
+          myapp=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} \
+          -n staging
+
+  deploy-production:
+    runs-on: ubuntu-latest
+    needs: build-and-test
+    if: github.ref == 'refs/heads/main'
+    environment:
+      name: production
+      url: https://example.com
+    steps:
+    - uses: actions/checkout@v4
+    - name: Deploy to production
+      run: |
+        kubectl set image deployment/myapp \
+          myapp=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} \
+          -n production
+    - name: Run smoke tests
+      run: ./scripts/smoke-tests.sh https://example.com
+```
+
+---
+
+## Notes on Examples
+
+These examples demonstrate:
+
+1. **Multi-stage pipelines** with proper stage ordering
+2. **Security scanning** integrated into pipeline
+3. **Artifact versioning** with commit SHA
+4. **Environment-specific deployments** (dev, staging, prod)
+5. **Approval gates** for sensitive environments
+6. **Health check validation** after deployment
+7. **Monitoring and alerting** post-deployment
+8. **Rollback capability** for failed deployments
+9. **Notification** to team on success/failure
+10. **Log retention** for troubleshooting
+
+Use these as templates and adapt them to your specific requirements and platform.