@1aboveio/skills 0.10.0

package/skills/cicd-pipeline/cloud-debug/LICENSE

@@ -0,0 +1,3 @@
+MIT License
+
+Copyright (c) 2026 1AboveIO

package/skills/cicd-pipeline/cloud-debug/SKILL.md

@@ -0,0 +1,316 @@
+---
+name: cloud-debug
+description: Debug Cloud Build and Cloud Deploy failures. Use when builds fail, deployments hang, health checks fail, or Cloud Deploy pipelines error. Accepts service name, build ID, region as parameters.
+---
+
+# Cloud Build & Cloud Deploy Debug Skill
+
+**Stack:** Cloud Build | Cloud Deploy | Cloud Run | Cloud Run Jobs | Secret Manager
+
+## Required Parameters
+
+Gather these from user or infer from context:
+
+| Parameter | Description | Example |
+|-----------|-------------|---------|
+| `REGION` | GCP region | us-central1 |
+| `SERVICE` | Cloud Run service name | my-api |
+| `BUILD_ID` | Cloud Build ID | abc123-def456 |
+| `PIPELINE` | Cloud Deploy pipeline name | my-pipeline-prod |
+| `RELEASE` | Release name | my-release-v1 |
+| `ROLLOUT` | Rollout name | my-release-v1-to-prod-0001 |
+| `JOB` | Cloud Run job name | my-job |
+| `COMMIT_SHA` | Git commit hash (full or short) | abc123def or abc123def456... |
+
+---
+
+## Diagnostic Workflow
+
+```
+User reports failure                    Track commit
+        |                                     |
+        v                                     v
++-------------------+                 Get commit SHA
+| What type of      |                         |
+| failure?          |                         v
++-------------------+                 Find build by SHA
+        |                                     |
+   +----+----+------------+                   v
+   v         v            v           Monitor build
+Build    Deployment    Pipeline               |
+   |         |            |                   v
+   v         v            v           Find release
+Get       Get Cloud    Get release/           |
+build ID  Run service  pipeline name          v
+   |         |            |           Monitor render
+   v         v            v                   |
+Pull      Pull         Pull render/           v
+build     service      rollout logs   Monitor rollout
+logs      logs
+```
+
+---
+
+## Quick Diagnostics
+
+### 1. Build Failures
+
+```bash
+# List recent failed builds
+gcloud builds list --region=$REGION --filter="status=FAILURE" --limit=5
+
+# Stream build logs
+gcloud builds log $BUILD_ID --region=$REGION --stream
+
+# Get detailed build info
+gcloud builds describe $BUILD_ID --region=$REGION --format="yaml(steps,status,statusDetail)"
+
+# Check build service account
+gcloud builds describe $BUILD_ID --region=$REGION --format="value(serviceAccount)"
+
+# List builds by trigger
+gcloud builds list --region=$REGION --filter="substitutions.TRIGGER_NAME=$TRIGGER"
+```
+
+Load [references/build-failures.md](references/build-failures.md) when diagnosing a build step failure — covers Dockerfile errors, dependency issues, and timeout patterns.
+
+### 2. Deployment Failures
+
+```bash
+# Check service status
+gcloud run services describe $SERVICE --region=$REGION
+
+# Get service details including errors
+gcloud run services describe $SERVICE --region=$REGION --format="yaml(status)"
+
+# Tail service logs (real-time)
+gcloud run services logs tail $SERVICE --region=$REGION
+
+# Read recent service logs
+gcloud run services logs read $SERVICE --region=$REGION --limit=50
+
+# List revisions
+gcloud run revisions list --service=$SERVICE --region=$REGION
+
+# Get revision details
+gcloud run revisions describe $REVISION --region=$REGION
+```
+
+Load [references/deploy-failures.md](references/deploy-failures.md) when diagnosing deploy or rollout failures — covers health checks, startup probes, and revision scaling.
+
+### 3. Job Failures
+
+```bash
+# List job executions
+gcloud run jobs executions list --job=$JOB --region=$REGION
+
+# Get execution logs
+gcloud run jobs executions logs $EXECUTION --job=$JOB --region=$REGION
+
+# Get execution details
+gcloud run jobs executions describe $EXECUTION --job=$JOB --region=$REGION
+
+# Execute job manually (for testing)
+gcloud run jobs execute $JOB --region=$REGION --wait
+```
+
+### 4. Pipeline Failures
+
+```bash
+# List pipelines
+gcloud deploy delivery-pipelines list --region=$REGION
+
+# List recent releases
+gcloud deploy releases list --delivery-pipeline=$PIPELINE --region=$REGION --limit=5
+
+# Check release render status
+gcloud deploy releases describe $RELEASE --delivery-pipeline=$PIPELINE --region=$REGION --format="yaml(renderState,targetRenders)"
+
+# List rollouts for a release
+gcloud deploy rollouts list --release=$RELEASE --delivery-pipeline=$PIPELINE --region=$REGION
+
+# Get rollout phase details
+gcloud deploy rollouts describe $ROLLOUT --release=$RELEASE --delivery-pipeline=$PIPELINE --region=$REGION --format="yaml(phases)"
+
+# List job runs (custom actions, verify, predeploy, postdeploy)
+gcloud deploy job-runs list --rollout=$ROLLOUT --release=$RELEASE --delivery-pipeline=$PIPELINE --region=$REGION
+
+# Get job run details
+gcloud deploy job-runs describe $JOB_RUN --rollout=$ROLLOUT --release=$RELEASE --delivery-pipeline=$PIPELINE --region=$REGION
+```
+
+Load [references/pipeline-failures.md](references/pipeline-failures.md) when diagnosing Cloud Deploy pipeline issues — covers promotion failures, verification errors, and target configuration.
+
+### 5. Track Commit Through Pipeline
+
+Monitor the full pipeline (build → release → rollout) for a specific commit hash.
+
+**Step 1: Find and Monitor Build**
+```bash
+# Find the latest build for a commit
+gcloud builds list --region=$REGION \
+  --filter="substitutions.COMMIT_SHA=$COMMIT_SHA OR substitutions.SHORT_SHA=$COMMIT_SHA" \
+  --limit=1 \
+  --format="value(id,status)"
+
+# Monitor build until completion (poll every 15 seconds)
+BUILD_ID=$(gcloud builds list --region=$REGION \
+  --filter="substitutions.COMMIT_SHA=$COMMIT_SHA" \
+  --limit=1 --format="value(id)")
+
+while true; do
+  STATUS=$(gcloud builds describe $BUILD_ID --region=$REGION --format="value(status)")
+  echo "[$(date +%H:%M:%S)] Build $BUILD_ID: $STATUS"
+  case $STATUS in
+    SUCCESS) echo "✓ Build succeeded"; break ;;
+    FAILURE|TIMEOUT|CANCELLED|EXPIRED) echo "✗ Build failed: $STATUS"; exit 1 ;;
+    QUEUED|WORKING|PENDING) sleep 15 ;;
+  esac
+done
+```
+
+**Step 2: Find Cloud Deploy Release**
+```bash
+# Get the release name from build substitutions or find by timing
+gcloud deploy releases list \
+  --delivery-pipeline=$PIPELINE \
+  --region=$REGION \
+  --filter="createTime>$(date -u -d '5 minutes ago' +%Y-%m-%dT%H:%M:%SZ)" \
+  --limit=1 \
+  --format="value(name)"
+
+# Or list recent releases to find the one created by build
+gcloud deploy releases list \
+  --delivery-pipeline=$PIPELINE \
+  --region=$REGION \
+  --limit=5 \
+  --format="table(name,renderState,createTime)"
+```
+
+**Step 3: Monitor Release Render**
+```bash
+# Check release render status
+while true; do
+  RENDER_STATE=$(gcloud deploy releases describe $RELEASE \
+    --delivery-pipeline=$PIPELINE \
+    --region=$REGION \
+    --format="value(renderState)")
+  echo "[$(date +%H:%M:%S)] Release render: $RENDER_STATE"
+  case $RENDER_STATE in
+    SUCCEEDED) echo "✓ Render succeeded"; break ;;
+    FAILED) echo "✗ Render failed"; exit 1 ;;
+    *) sleep 10 ;;
+  esac
+done
+```
+
+**Step 4: Monitor Rollout to Target**
+```bash
+# Find the rollout for the release
+ROLLOUT=$(gcloud deploy rollouts list \
+  --release=$RELEASE \
+  --delivery-pipeline=$PIPELINE \
+  --region=$REGION \
+  --limit=1 \
+  --format="value(name)")
+
+# Monitor rollout status
+while true; do
+  ROLLOUT_STATE=$(gcloud deploy rollouts describe $ROLLOUT \
+    --release=$RELEASE \
+    --delivery-pipeline=$PIPELINE \
+    --region=$REGION \
+    --format="value(state)")
+  echo "[$(date +%H:%M:%S)] Rollout $ROLLOUT: $ROLLOUT_STATE"
+  case $ROLLOUT_STATE in
+    SUCCEEDED) echo "✓ Deployment succeeded!"; break ;;
+    FAILED|CANCELLED) echo "✗ Deployment failed: $ROLLOUT_STATE"; exit 1 ;;
+    PENDING_APPROVAL) echo "⏸ Awaiting approval"; sleep 30 ;;
+    *) sleep 15 ;;
+  esac
+done
+```
+
+**Quick Alternative: Stream Build Logs**
+```bash
+# Stream build logs (blocks until complete, shows real-time output)
+gcloud builds log $BUILD_ID --region=$REGION --stream
+```
+
+---
+
+## Recovery Commands
+
+```bash
+# Retry failed job in rollout
+gcloud deploy rollouts retry-job $ROLLOUT \
+  --release=$RELEASE \
+  --delivery-pipeline=$PIPELINE \
+  --region=$REGION \
+  --job-id=$JOB_ID \
+  --phase-id=$PHASE_ID
+
+# Approve rollout (for approval-gated targets)
+gcloud deploy rollouts approve $ROLLOUT \
+  --release=$RELEASE \
+  --delivery-pipeline=$PIPELINE \
+  --region=$REGION
+
+# Cancel a rollout
+gcloud deploy rollouts cancel $ROLLOUT \
+  --release=$RELEASE \
+  --delivery-pipeline=$PIPELINE \
+  --region=$REGION
+```
+
+---
+
+## Secret Manager Diagnostics
+
+```bash
+# List secrets
+gcloud secrets list
+
+# Verify secret exists and has versions
+gcloud secrets versions list $SECRET_NAME
+
+# Access secret value (debugging only)
+gcloud secrets versions access latest --secret=$SECRET_NAME
+
+# Check IAM permissions on secret
+gcloud secrets get-iam-policy $SECRET_NAME
+```
+
+---
+
+## Closed-Loop Execution (Incident Handling)
+
+- After any monitor/describe/log tail, always state `Next step:` and what you will do next.
+- DONE criteria: root cause identified + fix applied + redeploy/verification completed + short clean-log window. If not achievable, escalate with captured evidence.
+- Timebox streaming monitors to 5–10 minutes, then describe/pinpoint the failing step.
+- On `FAILURE`, immediately capture: failing step, key error lines, and link/log pointers. Then loop: debug → fix → rerun → monitor.
+- Escalation boundary: ask before risky ops (prod migrations/seeds, rollbacks, IAM/VPC/secrets changes, deletions).
+
+Checklist template:
+```
+[ ] Issue + scope
+[ ] Last good / first bad
+[ ] Monitor/describe run
+[ ] Next step:
+[ ] Root cause
+[ ] Fix + redeploy/verify
+[ ] Clean logs (short window)
+[ ] Escalate w/ evidence (if needed)
+```
+
+---
+
+## Usage Pattern
+
+1. **Ask what failed** - Build? Deployment? Pipeline? Or track a commit?
+2. **Gather identifiers** - Region, service/build/pipeline name, or commit SHA
+3. **Run diagnostic commands** - Pull logs, describe resources
+4. **Analyze output** - Match against known failure patterns
+5. **Propose resolution** - Suggest fix based on pattern match
+
+**Track commit mode**: Monitor full pipeline (build → release → rollout) for a commit hash until success or failure. Use when user pushes code and wants to track deployment progress.

package/skills/cicd-pipeline/cloud-debug/references/build-failures.md

@@ -0,0 +1,238 @@
+# Cloud Build Failure Patterns
+
+## Common Failures
+
+| Symptom | Likely Cause | Resolution |
+|---------|--------------|------------|
+| `docker build` fails with missing package | Missing system dependency | Add to `apt-get install` in Dockerfile |
+| Tests pass locally but fail in CI | Missing env vars or secrets | Check `secretEnv` and `env` in cloudbuild.yaml |
+| `permission denied` pushing to Artifact Registry | IAM misconfiguration | Grant `roles/artifactregistry.writer` to Cloud Build SA |
+| Build timeout | Step takes too long | Increase `timeout` in cloudbuild.yaml or optimize step |
+| Secret access denied | Missing IAM binding | Grant `roles/secretmanager.secretAccessor` to Cloud Build SA |
+| `pnpm install` fails | Lockfile mismatch | Run `pnpm install` locally, commit lockfile |
+| `npm ci` fails | package-lock.json out of sync | Run `npm install` locally, commit lockfile |
+| `pip install` fails | Missing system libs | Add build dependencies to Dockerfile (libpq-dev, etc.) |
+
+---
+
+## Docker Build Failures
+
+### Missing System Dependencies
+
+**Symptom:**
+```
+E: Unable to locate package libldap2-dev
+```
+
+**Fix:** Add missing packages to your Dockerfile or test step:
+```dockerfile
+RUN apt-get update && apt-get install -y \
+    libldap2-dev \
+    libsasl2-dev \
+    libpq-dev
+```
+
+### Multi-stage Build Issues
+
+**Symptom:**
+```
+COPY failed: file not found in build context
+```
+
+**Fix:** Ensure files are in the correct stage and context:
+```dockerfile
+# Copy from builder stage
+COPY --from=builder /app/dist ./dist
+```
+
+### Build Context Issues (Monorepo)
+
+**Symptom:**
+```
+COPY packages/lib ./packages/lib
+COPY failed: file not found
+```
+
+**Fix:** Set build context to repository root in cloudbuild.yaml:
+```yaml
+- name: 'gcr.io/cloud-builders/docker'
+  args: ['build', '-t', '$_IMAGE', '-f', 'frontend/portal/Dockerfile', '.']
+  # Context is repo root (.), Dockerfile path is explicit
+```
+
+---
+
+## Test Failures
+
+### Missing Environment Variables
+
+**Symptom:** Tests pass locally, fail in CI with undefined config errors.
+
+**Fix:** Add required env vars to cloudbuild.yaml:
+```yaml
+steps:
+  - id: 'test'
+    name: 'node:20'
+    env:
+      - 'NODE_ENV=test'
+      - 'DATABASE_URL=postgresql://localhost:5432/test'
+```
+
+### Missing Secrets
+
+**Symptom:**
+```
+Error: Secret not found or access denied
+```
+
+**Fix:** Add `secretEnv` and `availableSecrets`:
+```yaml
+steps:
+  - id: 'test'
+    name: 'node:20'
+    secretEnv: ['API_KEY']
+
+availableSecrets:
+  secretManager:
+    - versionName: projects/$PROJECT_ID/secrets/api-key/versions/latest
+      env: 'API_KEY'
+```
+
+### Database Connection in Tests
+
+**Symptom:** Integration tests fail with connection refused.
+
+**Fix:** Use Cloud Build's Docker network or mock the database:
+```yaml
+steps:
+  - id: 'start-postgres'
+    name: 'gcr.io/cloud-builders/docker'
+    args: ['run', '-d', '--name=postgres', '--network=cloudbuild',
+           '-e', 'POSTGRES_PASSWORD=test', 'postgres:15']
+
+  - id: 'wait-for-postgres'
+    name: 'gcr.io/cloud-builders/docker'
+    args: ['run', '--network=cloudbuild', 'postgres:15',
+           'sh', '-c', 'until pg_isready -h postgres; do sleep 1; done']
+```
+
+---
+
+## Permission Issues
+
+### Artifact Registry Push Failed
+
+**Symptom:**
+```
+denied: Permission "artifactregistry.repositories.uploadArtifacts" denied
+```
+
+**Fix:** Grant permissions to Cloud Build service account:
+```bash
+# Get the Cloud Build SA
+PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format='value(projectNumber)')
+SA="${PROJECT_NUMBER}@cloudbuild.gserviceaccount.com"
+
+# Grant Artifact Registry Writer
+gcloud artifacts repositories add-iam-policy-binding $REPO \
+  --location=$REGION \
+  --member="serviceAccount:$SA" \
+  --role="roles/artifactregistry.writer"
+```
+
+### Secret Access Denied
+
+**Symptom:**
+```
+Error: Permission denied on secret
+```
+
+**Fix:** Grant Secret Manager access:
+```bash
+gcloud secrets add-iam-policy-binding $SECRET_NAME \
+  --member="serviceAccount:$SA" \
+  --role="roles/secretmanager.secretAccessor"
+```
+
+---
+
+## Timeout Issues
+
+### Build Timeout vs Step Timeout
+
+**Build timeout** - Total time for entire build (default: 10 minutes)
+```yaml
+timeout: '1800s'  # 30 minutes for entire build
+```
+
+**Step timeout** - Time for individual step (default: unlimited within build timeout)
+```yaml
+steps:
+  - id: 'long-step'
+    timeout: '600s'  # 10 minutes for this step
+```
+
+### Common Timeout Causes
+
+1. **Slow dependency install** - Use cache or pre-built images
+2. **Large Docker image** - Use multi-stage builds, smaller base images
+3. **Slow tests** - Parallelize or skip slow tests in CI
+4. **Network issues** - Check VPC/firewall settings
+
+---
+
+## Substitution Issues
+
+### Variable Not Substituted
+
+**Symptom:** `${SHORT_SHA}` appears literally in output.
+
+**Fix:** Use correct syntax based on context:
+```yaml
+# In step args - use $VAR
+args: ['tag', 'image:$SHORT_SHA']
+
+# In substitutions block - use ${VAR}
+substitutions:
+  _TAG: '${SHORT_SHA}'
+
+# In bash script - escape or use single quotes
+args:
+  - '-c'
+  - 'echo "Building ${SHORT_SHA}"'  # Works
+  - 'echo "Building $SHORT_SHA"'    # Also works
+```
+
+### Custom Substitution Not Available
+
+**Symptom:**
+```
+substitution _MY_VAR is not defined
+```
+
+**Fix:** Define in substitutions block or pass via trigger:
+```yaml
+substitutions:
+  _MY_VAR: 'default-value'
+```
+
+---
+
+## Diagnostic Commands
+
+```bash
+# Get detailed build info with step status
+gcloud builds describe $BUILD_ID --region=$REGION \
+  --format="yaml(steps[].id,steps[].status,steps[].timing)"
+
+# Check which step failed
+gcloud builds describe $BUILD_ID --region=$REGION \
+  --format="table(steps[].id,steps[].status)"
+
+# Get build logs for specific step
+gcloud builds log $BUILD_ID --region=$REGION --stream 2>&1 | grep -A 50 "Step #X"
+
+# List recent builds with trigger info
+gcloud builds list --region=$REGION --limit=10 \
+  --format="table(id,status,substitutions.TRIGGER_NAME,createTime)"
+```