@1aboveio/skills 0.10.0

package/skills/cicd-pipeline/google-cloud/references/gcs.md

@@ -0,0 +1,469 @@
+# Google Cloud Storage (GCS)
+
+Common operations for GCS buckets and objects using `gcloud storage`.
+
+## Project Configuration
+
+| Field | Value |
+|-------|-------|
+| Project ID | `$PROJECT_ID` |
+| Project Number | `$PROJECT_NUMBER` |
+
+## Quick Setup
+
+```bash
+# Set default project
+gcloud config set project $PROJECT_ID
+
+# Enable required APIs
+gcloud services enable storage.googleapis.com --project=$PROJECT_ID
+```
+
+---
+
+## Bucket Operations
+
+### List Buckets
+
+```bash
+# All buckets in project
+gcloud storage buckets list
+
+# With details
+gcloud storage buckets list --format="table(name,location,storageClass)"
+
+# Filter by label
+gcloud storage buckets list --filter="labels.env:dev"
+```
+
+### Create Bucket
+
+```bash
+# Standard storage class
+gcloud storage buckets create gs://$BUCKET_NAME \
+  --project=$PROJECT_ID \
+  --location=us-central1 \
+  --storage-class=STANDARD
+
+# Nearline (cheaper, accessed <1x/month)
+gcloud storage buckets create gs://$BUCKET_NAME \
+  --project=$PROJECT_ID \
+  --location=us-central1 \
+  --storage-class=NEARLINE
+
+# Multi-region
+gcloud storage buckets create gs://$BUCKET_NAME \
+  --project=$PROJECT_ID \
+  --location=US \
+  --storage-class=STANDARD
+```
+
+### Describe Bucket
+
+```bash
+gcloud storage buckets describe gs://$BUCKET_NAME
+```
+
+### Delete Bucket
+
+```bash
+# Delete empty bucket
+gcloud storage buckets delete gs://$BUCKET_NAME
+
+# Delete with objects (force)
+gcloud storage buckets delete gs://$BUCKET_NAME --quiet
+```
+
+---
+
+## Object Operations
+
+### List Objects
+
+```bash
+# In bucket root
+gcloud storage ls gs://$BUCKET_NAME
+
+# Recursive
+gcloud storage ls gs://$BUCKET_NAME/**
+
+# With details
+gcloud storage ls -l gs://$BUCKET_NAME/**
+```
+
+### Upload Files
+
+```bash
+# Single file
+gcloud storage cp file.txt gs://$BUCKET_NAME/
+
+# With specific storage class
+gcloud storage cp large-file.dat gs://$BUCKET_NAME/ \
+  --storage-class=NEARLINE
+
+# Recursive directory
+gcloud storage cp --recursive ./folder/ gs://$BUCKET_NAME/folder/
+
+# Set metadata
+gcloud storage cp file.txt gs://$BUCKET_NAME/ \
+  --content-type=text/plain \
+  --cache-control="max-age=3600"
+```
+
+### Download Objects
+
+```bash
+# Single file
+gcloud storage cp gs://$BUCKET_NAME/file.txt ./
+
+# Recursive
+gcloud storage cp --recursive gs://$BUCKET_NAME/folder/ ./
+
+# Specific object
+gcloud storage cp gs://$BUCKET_NAME/path/to/object ./local-path
+```
+
+### Copy Objects
+
+```bash
+# Within same bucket
+gcloud storage cp gs://$BUCKET_NAME/source.txt gs://$BUCKET_NAME/backup/
+
+# Between buckets
+gcloud storage cp gs://$BUCKET_A/file.txt gs://$BUCKET_B/
+
+# With metadata update
+gcloud storage cp gs://$BUCKET_A/file.txt gs://$BUCKET_B/ \
+  --storage-class=NEARLINE
+```
+
+### Delete Objects
+
+```bash
+# Single object
+gcloud storage rm gs://$BUCKET_NAME/file.txt
+
+# With pattern
+gcloud storage rm gs://$BUCKET_NAME/old/*.tmp
+
+# Recursive with filter
+gcloud storage rm gs://$BUCKET_NAME/** --filter="age:30"
+```
+
+---
+
+## IAM Access
+
+### Grant Access to Bucket
+
+```bash
+# Read access to user
+gcloud storage buckets add-iam-policy-binding gs://$BUCKET_NAME \
+  --member=user:$EMAIL \
+  --role=roles/storage.objectViewer
+
+# Write access to service account
+gcloud storage buckets add-iam-policy-binding gs://$BUCKET_NAME \
+  --member=serviceAccount:$SA_EMAIL \
+  --role=roles/storage.objectAdmin
+
+# Admin access
+gcloud storage buckets add-iam-policy-binding gs://$BUCKET_NAME \
+  --member=serviceAccount:$SA_EMAIL \
+  --role=roles/storage.admin
+```
+
+### Remove Access
+
+```bash
+gcloud storage buckets remove-iam-policy-binding gs://$BUCKET_NAME \
+  --member=user:$EMAIL \
+  --role=roles/storage.objectViewer
+```
+
+### Check IAM Policy
+
+```bash
+gcloud storage buckets get-iam-policy gs://$BUCKET_NAME
+```
+
+---
+
+## Signed URLs
+
+### Generate Signed Download URL
+
+```bash
+# Using service account key
+gcloud storage sign-url gs://$BUCKET_NAME/file.txt \
+  --method=GET \
+  --expires-in=3600 \
+  --service-account=$SA_EMAIL \
+  --private-key-file=/path/to/key.json
+
+# Using ADC (Application Default Credentials)
+gcloud storage sign-url gs://$BUCKET_NAME/file.txt \
+  --method=GET \
+  --expires-in=3600
+```
+
+### Generate Signed Upload URL
+
+```bash
+gcloud storage sign-url gs://$BUCKET_NAME/upload-file.txt \
+  --method=PUT \
+  --expires-in=3600 \
+  --headers="Content-Type:text/plain"
+```
+
+---
+
+## Versioning
+
+### Enable Versioning
+
+```bash
+gcloud storage buckets update gs://$BUCKET_NAME \
+  --versioning
+```
+
+### List Versions
+
+```bash
+gcloud storage ls -a gs://$BUCKET_NAME/**
+```
+
+### Restore Previous Version
+
+```bash
+gcloud storage cp gs://$BUCKET_NAME/file.txt#${VERSION_ID} gs://$BUCKET_NAME/file.txt
+```
+
+---
+
+## Lifecycle Policies
+
+### Set Lifecycle Rule (delete objects older than 30 days)
+
+```bash
+cat > /tmp/lifecycle.json <<EOF
+{
+  "lifecycle": {
+    "rule": [
+      {
+        "action": {"type": "Delete"},
+        "condition": {"age": 30}
+      }
+    ]
+  }
+}
+EOF
+
+gcloud storage buckets update gs://$BUCKET_NAME \
+  --lifecycle-file=/tmp/lifecycle.json
+```
+
+### Set Lifecycle Rule (archive after 90 days)
+
+```bash
+cat > /tmp/lifecycle.json <<EOF
+{
+  "lifecycle": {
+    "rule": [
+      {
+        "action": {
+          "type": "SetStorageClass",
+          "storageClass": "NEARLINE"
+        },
+        "condition": {"age": 90}
+      }
+    ]
+  }
+}
+EOF
+
+gcloud storage buckets update gs://$BUCKET_NAME \
+  --lifecycle-file=/tmp/lifecycle.json
+```
+
+### Clear Lifecycle Policy
+
+```bash
+gcloud storage buckets update gs://$BUCKET_NAME \
+  --clear-lifecycle
+```
+
+---
+
+## Retention Policy
+
+### Set Retention Policy
+
+```bash
+# 30-day retention (objects cannot be deleted/overwritten)
+gcloud storage buckets update gs://$BUCKET_NAME \
+  --retention-period=30d
+
+# 1-year retention
+gcloud storage buckets update gs://$BUCKET_NAME \
+  --retention-period=365d
+```
+
+### Check Retention Policy
+
+```bash
+gcloud storage buckets describe gs://$BUCKET_NAME \
+  --format="table(retentionPeriod,retentionPolicyLocked)"
+```
+
+---
+
+## Common Patterns
+
+### Upload with Public Access
+
+```bash
+# Make object publicly readable
+gcloud storage cp file.txt gs://$BUCKET_NAME/public/
+gcloud storage objects update gs://$BUCKET_NAME/public/file.txt \
+  --acl=public-read
+
+# Or use uniform bucket-level access with IAM
+gcloud storage buckets update gs://$BUCKET_NAME \
+  --public-access-prevention
+```
+
+### Sync Directory to Bucket
+
+```bash
+# Mirror sync (delete extras in destination)
+gcloud storage sync ./local-dir/ gs://$BUCKET_NAME/remote-dir/ \
+  --delete-unmatched-destination
+
+# Copy only (no delete)
+gcloud storage sync ./local-dir/ gs://$BUCKET_NAME/remote-dir/
+```
+
+### Upload to Folder with Date
+
+```bash
+DATE=$(date +%Y/%m/%d)
+gcloud storage cp report.csv gs://$BUCKET_NAME/backups/${DATE}/
+```
+
+### Download Latest Object
+
+```bash
+# Get most recent file matching pattern
+LATEST=$(gcloud storage ls gs://$BUCKET_NAME/data/*.csv \
+  --sort-by=updated | tail -1)
+
+gcloud storage cp "$LATEST" ./
+```
+
+---
+
+## Website Configuration
+
+### Configure Bucket as Website
+
+```bash
+# Set main page
+gcloud storage buckets update gs://$BUCKET_NAME \
+  --web-main-page=index.html
+
+# Set error page
+gcloud storage buckets update gs://$BUCKET_NAME \
+  --web-error-page=404.html
+```
+
+### Make Objects Public
+
+```bash
+# All objects in bucket (careful!)
+gcloud storage buckets update gs://$BUCKET_NAME \
+  --uniform-bucket-level-access
+
+gcloud storage buckets add-iam-policy-binding gs://$BUCKET_NAME \
+  --member=allUsers \
+  --role=roles/storage.objectViewer
+```
+
+---
+
+## Local Development
+
+### Mount Bucket (using gcsfuse)
+
+```bash
+# Install gcsfuse
+wget https://github.com/GoogleCloudPlatform/gcsfuse/releases/latest/download/gcsfuse_llinux_amd64.tar.gz
+tar -xzf gcsfuse_linux_amd64.tar.gz
+sudo mv gcsfuse /usr/local/bin/
+
+# Mount bucket
+mkdir -p ~/mnt/$BUCKET_NAME
+gcsfuse $BUCKET_NAME ~/mnt/$BUCKET_NAME
+
+# Unmount
+fusermount -u ~/mnt/$BUCKET_NAME
+```
+
+---
+
+## Debugging
+
+### Check Object Exists
+
+```bash
+gcloud storage ls gs://$BUCKET_NAME/path/to/file
+```
+
+### Check Bucket Permissions
+
+```bash
+gcloud storage buckets get-iam-policy gs://$BUCKET_NAME | grep $SA_EMAIL
+```
+
+### View Object Metadata
+
+```bash
+gcloud storage objects describe gs://$BUCKET_NAME/path/to/file \
+  --format="table(name,size,updated,metadata)"
+```
+
+### Test Write Access
+
+```bash
+echo "test" | gcloud storage cp - gs://$BUCKET_NAME/test-write-access.txt
+```
+
+---
+
+## Useful Queries
+
+### Count Objects
+
+```bash
+gcloud storage ls gs://$BUCKET_NAME/** | wc -l
+```
+
+### Find Large Objects
+
+```bash
+gcloud storage ls -s gs://$BUCKET_NAME/** | sort -rh | head -10
+```
+
+### Find Old Objects
+
+```bash
+# Objects older than 365 days
+gcloud storage ls -a gs://$BUCKET_NAME/** \
+  --filter="updated < $(date -d '365 days ago' +%Y-%m-%d)"
+```
+
+### Total Bucket Size
+
+```bash
+gcloud storage du gs://$BUCKET_NAME --summarize
+```