@1aboveio/skills 0.10.0

package/skills/cicd-pipeline/cloud-debug/references/deploy-failures.md

@@ -0,0 +1,376 @@
+# Cloud Run Deployment Failure Patterns
+
+## Common Failures
+
+| Symptom | Likely Cause | Resolution |
+|---------|--------------|------------|
+| Container fails to start | Missing required env var | Check `--set-env-vars` in deploy command |
+| Health check timeout | Slow startup or wrong port | Increase `initialDelaySeconds` or fix port |
+| Health check 503 (calls backend) | Missing invoker IAM on target service | Grant `roles/run.invoker` to caller's SA |
+| `Connection refused` to database | Cloud SQL proxy not configured | Add `--set-cloudsql-instances` |
+| Secret not found at runtime | Secret not mounted | Check `--set-secrets` configuration |
+| VPC connector error | Connector doesn't exist or wrong region | Verify VPC connector in same region |
+| Revision stuck in "Deploying" | Resource quota exceeded | Check quota with `gcloud compute project-info describe` |
+| `Container called exit(1)` | Application crash on startup | Check service logs for stack trace |
+| Memory limit exceeded | OOMKilled | Increase `--memory` limit |
+
+---
+
+## Container Startup Failures
+
+### Missing Environment Variables
+
+**Symptom:**
+```
+Container called exit(1).
+Error: DATABASE_URL is not defined
+```
+
+**Fix:** Add environment variables to deploy command:
+```bash
+gcloud run deploy $SERVICE \
+  --set-env-vars="DATABASE_URL=postgresql://..." \
+  --set-env-vars="NODE_ENV=production"
+```
+
+Or use secrets:
+```bash
+gcloud run deploy $SERVICE \
+  --set-secrets="DATABASE_URL=my-db-url:latest"
+```
+
+### Wrong Entry Point
+
+**Symptom:**
+```
+Container failed to start. Failed to start and then listen on the port defined by PORT
+```
+
+**Fix:** Ensure your app listens on `$PORT` (provided by Cloud Run):
+```javascript
+// Node.js
+const port = process.env.PORT || 8080;
+app.listen(port);
+```
+
+```python
+# Python/FastAPI
+import os
+port = int(os.getenv("PORT", 8080))
+uvicorn.run(app, host="0.0.0.0", port=port)
+```
+
+### Application Crash
+
+**Symptom:** Container exits immediately with non-zero code.
+
+**Diagnose:**
+```bash
+# Check logs for stack trace
+gcloud run services logs read $SERVICE --region=$REGION --limit=100
+
+# Look for specific error patterns
+gcloud run services logs read $SERVICE --region=$REGION --limit=100 | grep -i "error\|exception\|failed"
+```
+
+---
+
+## Health Check Failures
+
+### Startup Probe Timeout
+
+**Symptom:**
+```
+Revision failed to become ready: Health check failed
+```
+
+**Causes:**
+1. App takes too long to start (model loading, warm-up)
+2. Health endpoint not implemented
+3. Wrong port configuration
+
+**Fix - Increase startup time:**
+```yaml
+# In service.yaml or deploy command
+spec:
+  template:
+    spec:
+      containers:
+        - startupProbe:
+            httpGet:
+              path: /health
+              port: 8080
+            initialDelaySeconds: 30
+            periodSeconds: 10
+            failureThreshold: 10
+```
+
+**Fix - Via gcloud:**
+```bash
+gcloud run deploy $SERVICE \
+  --startup-cpu-boost \
+  --cpu=2 \
+  --memory=2Gi
+```
+
+### Health Endpoint Not Responding
+
+**Symptom:** Health check returns 404 or times out.
+
+**Fix:** Implement health endpoint:
+```python
+# FastAPI
+@app.get("/health")
+def health():
+    return {"status": "healthy"}
+```
+
+```javascript
+// Express
+app.get('/health', (req, res) => res.json({ status: 'healthy' }));
+```
+
+```javascript
+// Next.js API route (pages/api/health.ts or app/api/health/route.ts)
+export function GET() {
+  return Response.json({ status: 'healthy' });
+}
+```
+
+### Health Check Calls Dependent Service (403/404)
+
+**Symptom:** Health check returns 503 with degraded status, logs show:
+```json
+{"status":"degraded","checks":{"backend":{"status":"unhealthy","error":"HTTP 403"}}}
+```
+or
+```json
+{"status":"degraded","checks":{"backend":{"status":"unhealthy","error":"HTTP 404"}}}
+```
+
+**Causes:**
+1. Health endpoint calls another Cloud Run service (e.g., portal → engine)
+2. Calling service's service account lacks `roles/run.invoker` on the target service
+3. Target service URL is incorrect
+
+**Diagnose:**
+```bash
+# Get the calling service's service account
+gcloud run services describe $SERVICE --region=$REGION \
+  --format="value(spec.template.spec.serviceAccountName)"
+
+# Check IAM policy on target service
+gcloud run services get-iam-policy $TARGET_SERVICE --region=$REGION
+```
+
+**Fix - Grant invoker permission:**
+```bash
+# Get caller's service account
+CALLER_SA=$(gcloud run services describe $CALLER_SERVICE --region=$REGION \
+  --format="value(spec.template.spec.serviceAccountName)")
+
+# Grant invoker role on target service
+gcloud run services add-iam-policy-binding $TARGET_SERVICE \
+  --region=$REGION \
+  --member="serviceAccount:$CALLER_SA" \
+  --role="roles/run.invoker"
+```
+
+**Note:** If 404, also verify the target service URL is correct and the endpoint path exists.
+
+---
+
+## Database Connection Failures
+
+### Cloud SQL Connection Refused
+
+**Symptom:**
+```
+Connection refused to /cloudsql/project:region:instance
+```
+
+**Causes:**
+1. Cloud SQL instance not specified
+2. VPC connector not configured
+3. IAM permissions missing
+
+**Fix - Add Cloud SQL instance:**
+```bash
+gcloud run deploy $SERVICE \
+  --set-cloudsql-instances=PROJECT:REGION:INSTANCE
+```
+
+**Fix - Grant IAM permissions:**
+```bash
+# Get service account
+SA=$(gcloud run services describe $SERVICE --region=$REGION --format='value(spec.template.spec.serviceAccountName)')
+
+# Grant Cloud SQL Client role
+gcloud projects add-iam-policy-binding $PROJECT_ID \
+  --member="serviceAccount:$SA" \
+  --role="roles/cloudsql.client"
+```
+
+### Connection String Format
+
+**PostgreSQL via Unix socket:**
+```
+postgresql://user:password@/dbname?host=/cloudsql/project:region:instance
+```
+
+**PostgreSQL via TCP (with VPC):**
+```
+postgresql://user:password@PRIVATE_IP:5432/dbname
+```
+
+---
+
+## Secret Injection Failures
+
+### Secret Not Found
+
+**Symptom:**
+```
+Secret version projects/xxx/secrets/yyy/versions/latest not found
+```
+
+**Fix:** Verify secret exists and has a version:
+```bash
+gcloud secrets versions list $SECRET_NAME
+```
+
+### Secret Format Issues
+
+**Symptom:** Secret value is base64 encoded or has unexpected format.
+
+**Check:**
+```bash
+# View secret value
+gcloud secrets versions access latest --secret=$SECRET_NAME
+
+# Ensure it's plain text, not base64
+```
+
+### Secret IAM Permissions
+
+**Fix:**
+```bash
+# Get Cloud Run service account
+SA=$(gcloud run services describe $SERVICE --region=$REGION --format='value(spec.template.spec.serviceAccountName)')
+
+# Grant access
+gcloud secrets add-iam-policy-binding $SECRET_NAME \
+  --member="serviceAccount:$SA" \
+  --role="roles/secretmanager.secretAccessor"
+```
+
+---
+
+## VPC/Networking Issues
+
+### VPC Connector Not Found
+
+**Symptom:**
+```
+VPC connector projects/xxx/locations/yyy/connectors/zzz not found
+```
+
+**Fix:** Verify connector exists in the same region:
+```bash
+gcloud compute networks vpc-access connectors list --region=$REGION
+```
+
+### Egress Configuration
+
+**Internal services only (default):**
+```bash
+gcloud run deploy $SERVICE --vpc-egress=private-ranges-only
+```
+
+**All traffic through VPC:**
+```bash
+gcloud run deploy $SERVICE --vpc-egress=all-traffic
+```
+
+---
+
+## Resource Issues
+
+### Memory Limit Exceeded (OOMKilled)
+
+**Symptom:**
+```
+Container was killed due to memory limit exceeded
+```
+
+**Fix:** Increase memory:
+```bash
+gcloud run deploy $SERVICE --memory=2Gi
+```
+
+### CPU Throttling
+
+**Symptom:** Slow performance, timeouts under load.
+
+**Fix:** Increase CPU or enable CPU boost:
+```bash
+gcloud run deploy $SERVICE --cpu=2 --startup-cpu-boost
+```
+
+---
+
+## Migration Job Failures
+
+### Prisma Migration Fails
+
+**Symptom:**
+```
+Error: P3009 migrate failed to apply cleanly
+```
+
+**Diagnose:**
+```bash
+# Check job execution logs
+gcloud run jobs executions list --job=$JOB --region=$REGION
+gcloud run jobs executions logs $EXECUTION --job=$JOB --region=$REGION
+```
+
+**Common fixes:**
+1. Reset dev database: `prisma migrate reset`
+2. Check for manual schema changes
+3. Verify DATABASE_URL is correct
+
+### Tortoise/Aerich Migration Fails
+
+**Symptom:**
+```
+aerich.exceptions.DowngradeError
+```
+
+**Fix:** Check migration history and schema state:
+```bash
+# In container or locally
+aerich history
+aerich heads
+```
+
+---
+
+## Diagnostic Commands
+
+```bash
+# Full service status with conditions
+gcloud run services describe $SERVICE --region=$REGION --format="yaml(status.conditions)"
+
+# Get the latest revision name
+gcloud run services describe $SERVICE --region=$REGION --format="value(status.latestCreatedRevisionName)"
+
+# Describe specific revision
+gcloud run revisions describe $REVISION --region=$REGION
+
+# Check revision scaling
+gcloud run revisions describe $REVISION --region=$REGION --format="yaml(spec.containerConcurrency,status.desiredReplicas)"
+
+# Get container configuration
+gcloud run services describe $SERVICE --region=$REGION --format="yaml(spec.template.spec.containers)"
+```