@diottodev/runic 1.0.0 → 1.1.0

package/engineering-team/skills/gcp-cloud-architect/references/best_practices.md

@@ -0,0 +1,467 @@
+# GCP Best Practices
+
+Production-ready practices for naming, labels, IAM, networking, monitoring, and disaster recovery.
+
+---
+
+## Table of Contents
+
+- [Naming Conventions](#naming-conventions)
+- [Labels and Organization](#labels-and-organization)
+- [IAM and Security](#iam-and-security)
+- [Networking](#networking)
+- [Monitoring and Logging](#monitoring-and-logging)
+- [Cost Optimization](#cost-optimization)
+- [Disaster Recovery](#disaster-recovery)
+- [Common Pitfalls](#common-pitfalls)
+
+---
+
+## Naming Conventions
+
+### Resource Naming Pattern
+
+```
+{environment}-{project}-{resource-type}-{purpose}
+
+Examples:
+  prod-myapp-gke-cluster
+  dev-myapp-sql-primary
+  staging-myapp-run-api
+  prod-myapp-gcs-uploads
+```
+
+### Project Naming
+
+```
+{org}-{team}-{environment}
+
+Examples:
+  acme-platform-prod
+  acme-platform-dev
+  acme-data-prod
+```
+
+### Naming Rules
+
+| Resource | Format | Max Length | Example |
+|----------|--------|-----------|---------|
+| Project ID | lowercase, hyphens | 30 chars | acme-platform-prod |
+| GKE Cluster | lowercase, hyphens | 40 chars | prod-api-cluster |
+| Cloud Run | lowercase, hyphens | 49 chars | prod-myapp-api |
+| Cloud SQL | lowercase, hyphens | 84 chars | prod-myapp-sql-primary |
+| GCS Bucket | lowercase, hyphens, dots | 63 chars | acme-prod-myapp-uploads |
+| Service Account | lowercase, hyphens | 30 chars | myapp-run-sa |
+
+---
+
+## Labels and Organization
+
+### Required Labels
+
+Apply these labels to all resources:
+
+```
+labels:
+  environment: "prod"          # dev, staging, prod
+  team: "platform"             # team owning the resource
+  app: "myapp"                 # application name
+  cost-center: "eng-001"       # billing allocation
+  managed-by: "terraform"      # terraform, gcloud, console
+```
+
+### Label-Based Cost Reporting
+
+```bash
+# Export billing data to BigQuery with labels
+# Then query by label:
+SELECT
+  labels.value AS environment,
+  SUM(cost) AS total_cost
+FROM `billing_export.gcp_billing_export_v1_*`
+CROSS JOIN UNNEST(labels) AS labels
+WHERE labels.key = 'environment'
+GROUP BY environment
+ORDER BY total_cost DESC
+```
+
+### Organization Hierarchy
+
+```
+Organization
+├── Folder: Production
+│   ├── Project: platform-prod
+│   ├── Project: data-prod
+│   └── Project: ml-prod
+├── Folder: Non-Production
+│   ├── Project: platform-dev
+│   ├── Project: platform-staging
+│   └── Project: data-dev
+└── Folder: Shared Services
+    ├── Project: shared-networking
+    ├── Project: shared-security
+    └── Project: shared-monitoring
+```
+
+---
+
+## IAM and Security
+
+### Principle of Least Privilege
+
+```bash
+# BAD: Basic roles are too broad
+gcloud projects add-iam-policy-binding my-project \
+  --member="user:dev@example.com" \
+  --role="roles/editor"
+
+# GOOD: Use predefined roles
+gcloud projects add-iam-policy-binding my-project \
+  --member="user:dev@example.com" \
+  --role="roles/run.developer"
+```
+
+### Service Account Best Practices
+
+```bash
+# 1. Create dedicated SA per workload
+gcloud iam service-accounts create myapp-api-sa \
+  --display-name="MyApp API Service Account"
+
+# 2. Grant only required roles
+gcloud projects add-iam-policy-binding my-project \
+  --member="serviceAccount:myapp-api-sa@my-project.iam.gserviceaccount.com" \
+  --role="roles/datastore.user"
+
+# 3. Use Workload Identity for GKE (no key files)
+gcloud iam service-accounts add-iam-policy-binding \
+  myapp-api-sa@my-project.iam.gserviceaccount.com \
+  --role="roles/iam.workloadIdentityUser" \
+  --member="serviceAccount:my-project.svc.id.goog[default/myapp-api-ksa]"
+
+# 4. NEVER download SA key files in production
+# Instead, use attached service accounts or impersonation
+```
+
+### VPC Service Controls
+
+```bash
+# Create a service perimeter to restrict data exfiltration
+gcloud access-context-manager perimeters create my-perimeter \
+  --title="Production Data Perimeter" \
+  --resources="projects/123456" \
+  --restricted-services="bigquery.googleapis.com,storage.googleapis.com" \
+  --policy=$POLICY_ID
+```
+
+### Organization Policies
+
+```bash
+# Restrict external IPs on VMs
+gcloud resource-manager org-policies set-policy \
+  --project=my-project policy.yaml
+
+# policy.yaml
+constraint: compute.vmExternalIpAccess
+listPolicy:
+  allValues: DENY
+
+# Restrict public Cloud Storage
+constraint: storage.publicAccessPrevention
+booleanPolicy:
+  enforced: true
+```
+
+### Encryption
+
+| Layer | Service | Default |
+|-------|---------|---------|
+| At rest | Google-managed keys | Always enabled |
+| At rest | CMEK (Cloud KMS) | Optional, recommended |
+| In transit | TLS 1.3 | Always enabled |
+| Application | Cloud KMS | Encrypt sensitive fields |
+
+```bash
+# Create CMEK key for Cloud SQL
+gcloud kms keys create myapp-sql-key \
+  --keyring=myapp-keyring \
+  --location=us-central1 \
+  --purpose=encryption
+
+# Use CMEK with Cloud SQL
+gcloud sql instances create myapp-db \
+  --disk-encryption-key=projects/my-project/locations/us-central1/keyRings/myapp-keyring/cryptoKeys/myapp-sql-key
+```
+
+---
+
+## Networking
+
+### VPC Design
+
+```bash
+# Create custom VPC (avoid default network)
+gcloud compute networks create myapp-vpc \
+  --subnet-mode=custom
+
+# Create subnets with secondary ranges for GKE
+gcloud compute networks subnets create myapp-subnet \
+  --network=myapp-vpc \
+  --region=us-central1 \
+  --range=10.0.0.0/20 \
+  --secondary-range pods=10.4.0.0/14,services=10.8.0.0/20 \
+  --enable-private-google-access
+```
+
+### Shared VPC
+
+Use Shared VPC for multi-project environments:
+
+```
+Host Project (shared-networking)
+├── VPC: shared-vpc
+│   ├── Subnet: prod-us-central1 → Service Project: platform-prod
+│   ├── Subnet: prod-europe-west1 → Service Project: platform-prod
+│   └── Subnet: dev-us-central1 → Service Project: platform-dev
+```
+
+### Firewall Rules
+
+```bash
+# Allow internal traffic
+gcloud compute firewall-rules create allow-internal \
+  --network=myapp-vpc \
+  --allow=tcp,udp,icmp \
+  --source-ranges=10.0.0.0/8
+
+# Allow health checks from Google load balancers
+gcloud compute firewall-rules create allow-health-checks \
+  --network=myapp-vpc \
+  --allow=tcp:8080 \
+  --source-ranges=35.191.0.0/16,130.211.0.0/22 \
+  --target-tags=allow-health-check
+
+# Deny all other ingress (implicit, but be explicit)
+gcloud compute firewall-rules create deny-all-ingress \
+  --network=myapp-vpc \
+  --action=DENY \
+  --rules=all \
+  --direction=INGRESS \
+  --priority=65534
+```
+
+### Private Google Access
+
+Always enable Private Google Access to reach GCP APIs without public IPs:
+
+```bash
+gcloud compute networks subnets update myapp-subnet \
+  --region=us-central1 \
+  --enable-private-google-access
+```
+
+---
+
+## Monitoring and Logging
+
+### Cloud Monitoring Setup
+
+```bash
+# Create uptime check
+gcloud monitoring uptime create \
+  --display-name="API Health Check" \
+  --resource-type=cloud-run-revision \
+  --resource-labels="service_name=myapp-api,location=us-central1" \
+  --check-request-path="/health" \
+  --period=60s
+
+# Create alerting policy
+gcloud alpha monitoring policies create \
+  --display-name="High Error Rate" \
+  --condition-display-name="Cloud Run 5xx > 1%" \
+  --condition-filter='resource.type="cloud_run_revision" AND metric.type="run.googleapis.com/request_count" AND metric.labels.response_code_class="5xx"' \
+  --condition-threshold-value=1 \
+  --notification-channels="projects/my-project/notificationChannels/12345"
+```
+
+### Key Metrics to Monitor
+
+| Service | Metric | Alert Threshold |
+|---------|--------|-----------------|
+| Cloud Run | request_latencies (p99) | >2s |
+| Cloud Run | request_count (5xx) | >1% of total |
+| Cloud SQL | cpu/utilization | >80% |
+| Cloud SQL | disk/utilization | >85% |
+| GKE | container/cpu/utilization | >80% |
+| GKE | node/cpu/allocatable_utilization | >85% |
+| Pub/Sub | subscription/oldest_unacked_message_age | >300s |
+| BigQuery | query/execution_time | >60s |
+
+### Log-Based Metrics
+
+```bash
+# Create a metric for application errors
+gcloud logging metrics create app-errors \
+  --description="Application error count" \
+  --log-filter='resource.type="cloud_run_revision" AND severity>=ERROR'
+
+# Create log sink to BigQuery for analysis
+gcloud logging sinks create audit-logs-bq \
+  bigquery.googleapis.com/projects/my-project/datasets/audit_logs \
+  --log-filter='logName="projects/my-project/logs/cloudaudit.googleapis.com%2Factivity"'
+```
+
+### Log Exclusion (Cost Reduction)
+
+```bash
+# Exclude verbose debug logs to save on Cloud Logging costs
+gcloud logging sinks create _Default \
+  --log-filter='NOT (severity="DEBUG" OR severity="DEFAULT")' \
+  --description="Exclude debug-level logs"
+
+# Or create exclusion filters
+gcloud logging exclusions create exclude-debug \
+  --log-filter='severity="DEBUG"' \
+  --description="Exclude debug logs to reduce costs"
+```
+
+---
+
+## Cost Optimization
+
+### Committed Use Discounts
+
+| Term | Compute Discount | Memory Discount |
+|------|-----------------|-----------------|
+| 1 year | 37% | 37% |
+| 3 years | 55% | 55% |
+
+```bash
+# Check recommendations
+gcloud recommender recommendations list \
+  --project=my-project \
+  --location=us-central1 \
+  --recommender=google.compute.commitment.UsageCommitmentRecommender
+```
+
+### Sustained Use Discounts
+
+Automatic discounts for resources running >25% of the month:
+
+| Usage | Discount |
+|-------|----------|
+| 25-50% | 20% |
+| 50-75% | 40% |
+| 75-100% | 60% |
+
+### BigQuery Cost Control
+
+```sql
+-- Use partitioning to limit data scanned
+CREATE TABLE my_dataset.events
+PARTITION BY DATE(timestamp)
+CLUSTER BY event_type
+AS SELECT * FROM raw_events;
+
+-- Estimate query cost before running
+-- Use --dry_run flag
+bq query --dry_run --use_legacy_sql=false \
+  'SELECT * FROM my_dataset.events WHERE DATE(timestamp) = "2026-01-01"'
+```
+
+### Cloud Storage Optimization
+
+```bash
+# Enable Autoclass for automatic class management
+gsutil mb -l us-central1 --autoclass gs://my-bucket/
+
+# Set lifecycle policy
+gsutil lifecycle set lifecycle.json gs://my-bucket/
+```
+
+---
+
+## Disaster Recovery
+
+### RPO/RTO Targets
+
+| Tier | RPO | RTO | Strategy |
+|------|-----|-----|----------|
+| Tier 1 (Critical) | 0 | <1 hour | Multi-region active-active |
+| Tier 2 (Important) | <1 hour | <4 hours | Regional HA + cross-region backup |
+| Tier 3 (Standard) | <24 hours | <24 hours | Automated backups + restore |
+
+### Backup Strategy
+
+```bash
+# Cloud SQL automated backups
+gcloud sql instances patch myapp-db \
+  --backup-start-time=02:00 \
+  --enable-point-in-time-recovery
+
+# Firestore scheduled exports
+gcloud firestore export gs://myapp-backups/firestore/$(date +%Y%m%d)
+
+# GKE cluster backup with Backup for GKE
+gcloud beta container backup-restore backup-plans create myapp-plan \
+  --project=my-project \
+  --location=us-central1 \
+  --cluster=projects/my-project/locations/us-central1/clusters/myapp-cluster \
+  --all-namespaces \
+  --cron-schedule="0 2 * * *"
+```
+
+### Multi-Region Failover
+
+```bash
+# Cloud SQL cross-region replica for DR
+gcloud sql instances create myapp-db-replica \
+  --master-instance-name=myapp-db \
+  --region=us-east1
+
+# Promote replica during failover
+gcloud sql instances promote-replica myapp-db-replica
+```
+
+---
+
+## Common Pitfalls
+
+### Technical Debt
+
+| Pitfall | Solution |
+|---------|----------|
+| Using default VPC | Always create custom VPCs |
+| Not enabling audit logs | Enable Cloud Audit Logs from day one |
+| Single-region deployment | Plan for multi-zone at minimum |
+| No IaC | Use Terraform from the start |
+
+### Security Mistakes
+
+| Mistake | Prevention |
+|---------|------------|
+| SA key files in code | Use Workload Identity, attached SAs |
+| Public GCS buckets | Enable org policy for public access prevention |
+| Basic roles (Owner/Editor) | Use predefined or custom roles |
+| No encryption key management | Use CMEK for sensitive data |
+| Default service account | Create dedicated SAs per workload |
+
+### Performance Issues
+
+| Issue | Solution |
+|-------|----------|
+| Cold starts on Cloud Run | Set min-instances=1 for latency-critical services |
+| Slow BigQuery queries | Partition tables, use clustering, avoid SELECT * |
+| GKE pod scheduling delays | Use PodDisruptionBudget, pre-provision with Autopilot |
+| Firestore hotspots | Distribute writes across document IDs evenly |
+
+### Cost Surprises
+
+| Surprise | Prevention |
+|----------|------------|
+| Undeleted resources | Label everything, review weekly |
+| Egress costs | Keep traffic in same region, use Private Google Access |
+| Cloud NAT charges | Use Private Google Access for GCP service traffic |
+| Log ingestion costs | Set exclusion filters for debug/verbose logs |
+| BigQuery full scans | Always use partitioning and clustering |
+| Idle GKE clusters | Delete dev clusters nightly, use Autopilot |