blockmine 1.24.0 → 1.27.0

package/.claude/skills/platform-engineering/resources/cost-optimization.md

@@ -1,634 +0,0 @@
-# Cost Optimization & FinOps
-
-FinOps practices, cloud cost optimization, right-sizing resources, spot instances, reserved capacity, and cost allocation for platform engineering.
-
-## Table of Contents
-
-- [FinOps Principles](#finops-principles)
-- [Cost Visibility](#cost-visibility)
-- [Right-Sizing](#right-sizing)
-- [Spot Instances](#spot-instances)
-- [Reserved Capacity](#reserved-capacity)
-- [Storage Optimization](#storage-optimization)
-- [Network Cost Optimization](#network-cost-optimization)
-- [Best Practices](#best-practices)
-
-## FinOps Principles
-
-### Three Phases
-
-```
-┌──────────────┐     ┌──────────────┐     ┌──────────────┐
-│   Inform     │ →   │   Optimize   │ →   │   Operate    │
-│              │     │              │     │              │
-│ Visibility   │     │ Right-size   │     │ Continuous   │
-│ Allocation   │     │ Commit       │     │ Improvement  │
-│ Reporting    │     │ Efficiency   │     │ Governance   │
-└──────────────┘     └──────────────┘     └──────────────┘
-```
-
-### Cost Categories
-
-```
-Infrastructure Costs:
-├── Compute (40-50%)
-├── Storage (20-30%)
-├── Network (10-20%)
-├── Databases (10-15%)
-└── Other Services (5-10%)
-```
-
-## Cost Visibility
-
-### Tagging Strategy
-
-**Required Tags:**
-```yaml
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: production
-  labels:
-    # Cost allocation
-    cost-center: "12345"
-    department: "engineering"
-    team: "platform"
-    environment: "production"
-
-    # Business context
-    project: "customer-api"
-    owner: "team-platform@company.com"
-
-    # Technical context
-    managed-by: "terraform"
-    service-tier: "critical"
-```
-
-**Apply to All Resources:**
-```hcl
-# Terraform default tags
-provider "aws" {
-  region = "us-east-1"
-
-  default_tags {
-    tags = {
-      CostCenter  = "12345"
-      Department  = "engineering"
-      Team        = "platform"
-      Environment = "production"
-      ManagedBy   = "terraform"
-    }
-  }
-}
-
-resource "aws_instance" "web" {
-  # ... instance config
-
-  tags = {
-    Name        = "web-server"
-    Application = "customer-api"
-    ServiceTier = "critical"
-  }
-}
-```
-
-### Cost Dashboards
-
-**Grafana Dashboard (Kubecost Integration):**
-```yaml
-# prometheus-queries.yaml
-
-# Total cluster cost
-sum(
-  avg_over_time(node_cpu_hourly_cost[1h])
-  * on(node) group_left()
-  avg_over_time(node_total_hourly_cost[1h])
-)
-
-# Cost by namespace
-sum(
-  container_memory_allocation_bytes
-  * on(namespace) group_left(label_cost_center)
-  kube_namespace_labels
-) by (label_cost_center, namespace)
-
-# Cost by team
-sum(
-  avg_over_time(kubecost_cluster_costs[1d])
-) by (team)
-
-# Idle resource cost
-sum(
-  (
-    kube_pod_container_resource_limits{resource="cpu"}
-    -
-    rate(container_cpu_usage_seconds_total[5m])
-  )
-  * on(node) group_left()
-  avg_over_time(node_cpu_hourly_cost[1h])
-)
-```
-
-### AWS Cost Explorer Tags
-
-```bash
-# AWS CLI cost query
-aws ce get-cost-and-usage \
-  --time-period Start=2024-01-01,End=2024-01-31 \
-  --granularity MONTHLY \
-  --metrics UnblendedCost \
-  --group-by Type=TAG,Key=Team \
-  --group-by Type=TAG,Key=Environment
-```
-
-## Right-Sizing
-
-### Analysis Tools
-
-**Vertical Pod Autoscaler Recommendations:**
-```bash
-# Get VPA recommendations
-kubectl get vpa api-service-vpa -o jsonpath='{.status.recommendation}'
-
-# Example output:
-{
-  "containerRecommendations": [{
-    "containerName": "api",
-    "lowerBound": {
-      "cpu": "100m",
-      "memory": "128Mi"
-    },
-    "target": {
-      "cpu": "250m",
-      "memory": "256Mi"
-    },
-    "uncappedTarget": {
-      "cpu": "300m",
-      "memory": "300Mi"
-    },
-    "upperBound": {
-      "cpu": "500m",
-      "memory": "512Mi"
-    }
-  }]
-}
-```
-
-**Resource Waste Detection:**
-```promql
-# Over-provisioned CPU (using < 50% of request)
-(
-  sum by (pod, namespace) (
-    rate(container_cpu_usage_seconds_total[5m])
-  )
-  /
-  sum by (pod, namespace) (
-    kube_pod_container_resource_requests{resource="cpu"}
-  )
-) < 0.5
-
-# Over-provisioned memory (using < 50% of request)
-(
-  sum by (pod, namespace) (
-    container_memory_working_set_bytes
-  )
-  /
-  sum by (pod, namespace) (
-    kube_pod_container_resource_requests{resource="memory"}
-  )
-) < 0.5
-```
-
-### Right-Sizing Actions
-
-```yaml
-# Before (over-provisioned)
-resources:
-  requests:
-    memory: "2Gi"
-    cpu: "2000m"
-  limits:
-    memory: "4Gi"
-    cpu: "4000m"
-
-# After (right-sized based on actual usage)
-resources:
-  requests:
-    memory: "512Mi"
-    cpu: "500m"
-  limits:
-    memory: "1Gi"
-    cpu: "1000m"
-
-# Potential savings: ~75% reduction
-```
-
-## Spot Instances
-
-### Kubernetes with Spot Nodes
-
-**AWS EKS Node Group:**
-```hcl
-resource "aws_eks_node_group" "spot" {
-  cluster_name    = aws_eks_cluster.main.name
-  node_group_name = "spot-workers"
-  node_role_arn   = aws_iam_role.node.arn
-  subnet_ids      = var.private_subnet_ids
-
-  capacity_type = "SPOT"
-
-  scaling_config {
-    desired_size = 5
-    min_size     = 3
-    max_size     = 20
-  }
-
-  instance_types = [
-    "t3.large",
-    "t3a.large",
-    "t2.large"
-  ]
-
-  labels = {
-    workload-type = "spot"
-  }
-
-  taint {
-    key    = "workload-type"
-    value  = "spot"
-    effect = "NO_SCHEDULE"
-  }
-}
-```
-
-**Workload on Spot:**
-```yaml
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: batch-processor
-spec:
-  replicas: 10
-  template:
-    spec:
-      nodeSelector:
-        workload-type: spot
-
-      tolerations:
-      - key: workload-type
-        operator: Equal
-        value: spot
-        effect: NoSchedule
-
-      # Graceful shutdown for spot interruptions
-      terminationGracePeriodSeconds: 120
-
-      containers:
-      - name: processor
-        image: batch-processor:1.0
-        # ... config
-```
-
-**AWS Node Termination Handler:**
-```yaml
-# Handles spot interruption notices
-apiVersion: apps/v1
-kind: DaemonSet
-metadata:
-  name: aws-node-termination-handler
-  namespace: kube-system
-spec:
-  selector:
-    matchLabels:
-      app: aws-node-termination-handler
-  template:
-    metadata:
-      labels:
-        app: aws-node-termination-handler
-    spec:
-      serviceAccountName: aws-node-termination-handler
-      containers:
-      - name: aws-node-termination-handler
-        image: public.ecr.aws/aws-ec2/aws-node-termination-handler:v1.19.0
-        env:
-        - name: NODE_NAME
-          valueFrom:
-            fieldRef:
-              fieldPath: spec.nodeName
-        - name: POD_NAME
-          valueFrom:
-            fieldRef:
-              fieldPath: metadata.name
-        - name: NAMESPACE
-          valueFrom:
-            fieldRef:
-              fieldPath: metadata.namespace
-        - name: ENABLE_SPOT_INTERRUPTION_DRAINING
-          value: "true"
-        - name: ENABLE_SCHEDULED_EVENT_DRAINING
-          value: "true"
-```
-
-### Spot Savings
-
-```
-On-Demand Cost:  $0.096/hour (t3.large)
-Spot Cost:       $0.029/hour (70% savings)
-Monthly Savings: $48 per instance
-100 instances:   $4,800/month savings
-```
-
-## Reserved Capacity
-
-### AWS Savings Plans
-
-**Compute Savings Plan (Most Flexible):**
-```
-Commitment:      $500/month for 1 year
-Discount:        up to 66% vs on-demand
-Applies to:      EC2, Fargate, Lambda
-Instance flex:   Any instance type/size/region
-```
-
-**EC2 Instance Savings Plan:**
-```
-Commitment:      $1,000/month for 3 years
-Discount:        up to 72% vs on-demand
-Applies to:      EC2 only
-Instance flex:   Same family, any size
-```
-
-**Reserved Instances (Legacy):**
-```
-Less flexible than Savings Plans
-Consider Savings Plans instead
-```
-
-### Right Commitment Level
-
-**Analysis Script:**
-```python
-import boto3
-from datetime import datetime, timedelta
-
-ce = boto3.client('ce')
-
-# Get last 30 days usage
-end = datetime.now()
-start = end - timedelta(days=30)
-
-response = ce.get_cost_and_usage(
-    TimePeriod={
-        'Start': start.strftime('%Y-%m-%d'),
-        'End': end.strftime('%Y-%m-%d')
-    },
-    Granularity='DAILY',
-    Metrics=['UnblendedCost'],
-    Filter={
-        'Dimensions': {
-            'Key': 'SERVICE',
-            'Values': ['Amazon Elastic Compute Cloud - Compute']
-        }
-    }
-)
-
-# Calculate baseline usage (minimum daily cost)
-daily_costs = [float(day['Total']['UnblendedCost']['Amount'])
-               for day in response['ResultsByTime']]
-baseline = min(daily_costs)
-monthly_commitment = baseline * 30
-
-print(f"Recommended monthly commitment: ${monthly_commitment:.2f}")
-print(f"Estimated annual savings: ${monthly_commitment * 12 * 0.3:.2f}")
-```
-
-## Storage Optimization
-
-### S3 Lifecycle Policies
-
-```hcl
-resource "aws_s3_bucket_lifecycle_configuration" "data" {
-  bucket = aws_s3_bucket.data.id
-
-  rule {
-    id     = "archive-old-data"
-    status = "Enabled"
-
-    # Transition to cheaper storage classes
-    transition {
-      days          = 30
-      storage_class = "STANDARD_IA"  # Infrequent Access
-    }
-
-    transition {
-      days          = 90
-      storage_class = "GLACIER_IR"  # Instant Retrieval
-    }
-
-    transition {
-      days          = 180
-      storage_class = "DEEP_ARCHIVE"  # Cheapest
-    }
-
-    # Delete after retention period
-    expiration {
-      days = 365
-    }
-  }
-
-  rule {
-    id     = "delete-incomplete-uploads"
-    status = "Enabled"
-
-    abort_incomplete_multipart_upload {
-      days_after_initiation = 7
-    }
-  }
-}
-```
-
-**Savings:**
-```
-Standard:       $0.023/GB/month
-Standard-IA:    $0.0125/GB/month (46% cheaper)
-Glacier IR:     $0.004/GB/month (83% cheaper)
-Deep Archive:   $0.00099/GB/month (96% cheaper)
-```
-
-### EBS Optimization
-
-**Delete Unattached Volumes:**
-```bash
-#!/bin/bash
-# find-unused-ebs.sh
-
-aws ec2 describe-volumes \
-  --filters Name=status,Values=available \
-  --query 'Volumes[*].[VolumeId,Size,VolumeType,CreateTime]' \
-  --output table
-
-# Delete after verification
-aws ec2 delete-volume --volume-id vol-xxxxx
-```
-
-**Snapshot Cleanup:**
-```bash
-# Delete snapshots older than 30 days
-aws ec2 describe-snapshots --owner-ids self \
-  --query 'Snapshots[?StartTime<=`'$(date -d '30 days ago' -Iseconds)'`].[SnapshotId]' \
-  --output text | \
-  xargs -I {} aws ec2 delete-snapshot --snapshot-id {}
-```
-
-### Persistent Volume Cleanup
-
-```bash
-# Find PVs not bound to PVC
-kubectl get pv | grep Released
-
-# Delete released PVs
-kubectl delete pv <pv-name>
-```
-
-## Network Cost Optimization
-
-### Data Transfer Costs
-
-**AWS Inter-Region Costs:**
-```
-Same AZ:              Free
-Same Region, diff AZ: $0.01/GB
-Cross-Region:         $0.02/GB
-Internet Egress:      $0.09/GB (first 10TB)
-```
-
-**Optimization Strategies:**
-```yaml
-# 1. Use VPC endpoints (avoid internet gateway)
-resource "aws_vpc_endpoint" "s3" {
-  vpc_id       = aws_vpc.main.id
-  service_name = "com.amazonaws.us-east-1.s3"
-}
-
-# 2. Use CloudFront for static assets
-resource "aws_cloudfront_distribution" "cdn" {
-  # CloudFront egress: $0.085/GB (cheaper than direct S3)
-}
-
-# 3. Enable S3 Transfer Acceleration (for uploads)
-resource "aws_s3_bucket_accelerate_configuration" "data" {
-  bucket = aws_s3_bucket.data.id
-  status = "Enabled"
-}
-```
-
-### NAT Gateway Costs
-
-```
-NAT Gateway:     $0.045/hour + $0.045/GB processed
-NAT Instance:    EC2 cost + $0.00/GB processed
-VPC Endpoints:   $0.01/hour + $0.01/GB (S3/DynamoDB free)
-```
-
-**Cost Comparison (100GB/month):**
-```
-NAT Gateway:     $32.40 + $4.50 = $36.90/month
-NAT Instance:    ~$8 + $0 = $8/month (75% cheaper)
-VPC Endpoint:    $7.20 + $1.00 = $8.20/month (77% cheaper)
-```
-
-## Best Practices
-
-### 1. Implement Cost Awareness
-
-Make cost data visible to teams during development.
-
-### 2. Set Budgets and Alerts
-
-```yaml
-# AWS Budget
-resource "aws_budgets_budget" "team_platform" {
-  name              = "team-platform-monthly"
-  budget_type       = "COST"
-  limit_amount      = "10000"
-  limit_unit        = "USD"
-  time_period_start = "2024-01-01_00:00"
-  time_unit         = "MONTHLY"
-
-  notification {
-    comparison_operator        = "GREATER_THAN"
-    threshold                  = 80
-    threshold_type            = "PERCENTAGE"
-    notification_type         = "ACTUAL"
-    subscriber_email_addresses = ["platform-team@company.com"]
-  }
-
-  cost_filters = {
-    TagKeyValue = "user:Team$platform"
-  }
-}
-```
-
-### 3. Regular Right-Sizing Reviews
-
-Schedule monthly reviews of resource utilization.
-
-### 4. Automate Cleanup
-
-```yaml
-# CronJob to delete old resources
-apiVersion: batch/v1
-kind: CronJob
-metadata:
-  name: cleanup-old-pvcs
-spec:
-  schedule: "0 2 * * 0"  # Weekly Sunday 2 AM
-  jobTemplate:
-    spec:
-      template:
-        spec:
-          containers:
-          - name: cleanup
-            image: bitnami/kubectl:latest
-            command:
-            - /bin/bash
-            - -c
-            - |
-              # Delete PVCs older than 30 days with label cleanup=enabled
-              kubectl get pvc -A \
-                -l cleanup=enabled \
-                -o json | \
-                jq -r '.items[] |
-                  select(
-                    (.metadata.creationTimestamp | fromdateiso8601) <
-                    (now - 30*24*60*60)
-                  ) |
-                  "\(.metadata.namespace) \(.metadata.name)"' | \
-                while read ns name; do
-                  kubectl delete pvc -n "$ns" "$name"
-                done
-          restartPolicy: OnFailure
-```
-
-### 5. Use Cost Allocation Tags
-
-Mandatory tags for all resources.
-
-### 6. Optimize Development Environments
-
-```bash
-# Stop dev/staging clusters overnight
-0 19 * * * kubectl scale deployment --all --replicas=0 -n development
-0 7 * * * kubectl scale deployment --all --replicas=1 -n development
-```
-
-### 7. Monitor Idle Resources
-
-Alert on resources with <10% utilization for 7+ days.
-
----
-
-**Related Resources:**
-- [resource-management.md](resource-management.md) - Resource optimization
-- [multi-tenancy.md](multi-tenancy.md) - Cost allocation
-- [infrastructure-standards.md](infrastructure-standards.md) - Tagging standards