pangea 0.0.45 → 0.0.46

data/lib/pangea/components/capabilities.rb

@@ -1,93 +0,0 @@
-# Copyright 2025 The Pangea Authors
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-module Pangea
-  module Components
-    module Capabilities
-      module HighAvailability
-        def self.included(base)
-          base.class_eval do
-            option :high_availability, default: -> { false }
-            option :availability_zones, default: -> { [] }
-            
-            validation :high_availability_zones do
-              if high_availability && availability_zones.size < 2
-                raise ValidationError, "High availability requires at least 2 availability zones"
-              end
-            end
-          end
-        end
-        
-        def apply_high_availability(resource_config)
-          return resource_config unless high_availability
-          
-          case resource_config[:type]
-          when :aws_rds_instance
-            resource_config.merge(multi_az: true)
-          when :aws_elasticache_replication_group
-            resource_config.merge(automatic_failover_enabled: true)
-          else
-            resource_config
-          end
-        end
-      end
-      
-      module Monitoring
-        def self.included(base)
-          base.class_eval do
-            option :enable_monitoring, default: -> { true }
-            option :monitoring_level, default: -> { :standard }
-          end
-        end
-        
-        def monitoring_config
-          return {} unless enable_monitoring
-          
-          case monitoring_level
-          when :basic
-            { detailed_monitoring: false, log_retention: 7 }
-          when :standard
-            { detailed_monitoring: true, log_retention: 30 }
-          when :enhanced
-            { detailed_monitoring: true, log_retention: 90, custom_metrics: true }
-          end
-        end
-      end
-      
-      module Security
-        def self.included(base)
-          base.class_eval do
-            option :enable_encryption, default: -> { true }
-            option :enable_logging, default: -> { true }
-          end
-        end
-        
-        def apply_security(resource_config)
-          config = resource_config.dup
-          
-          if enable_encryption
-            config[:encrypted] = true if config[:type] == :aws_ebs_volume
-            config[:storage_encrypted] = true if config[:type] == :aws_db_instance
-          end
-          
-          if enable_logging
-            config[:logging_enabled] = true
-          end
-          
-          config
-        end
-      end
-    end
-  end
-end

data/lib/pangea/components/carbon_aware_compute/CLAUDE.md

@@ -1,295 +0,0 @@
-# Carbon Aware Compute - Architecture Documentation
-
-## Component Purpose
-
-The Carbon Aware Compute component represents a paradigm shift in cloud computing sustainability. It transforms traditional "always-on" or "schedule-only" workload execution into an intelligent system that considers the environmental impact of computational work. By integrating real-time carbon intensity data with workload scheduling, this component enables organizations to reduce their carbon footprint without sacrificing functionality.
-
-## Architecture Decisions
-
-### 1. Serverless-First Architecture
-
-**Decision**: Use Lambda functions for all compute operations rather than persistent EC2 instances or containers.
-
-**Rationale**: 
-- Zero idle emissions - functions only consume energy when executing
-- Automatic scaling eliminates over-provisioning
-- Reduced operational overhead allows focus on optimization logic
-- Native integration with EventBridge for scheduling
-
-**Trade-offs**:
-- 15-minute execution limit requires workload chunking
-- Cold starts may impact time-sensitive workloads
-- Limited to Lambda-compatible runtimes
-
-### 2. DynamoDB for State Management
-
-**Decision**: Use DynamoDB for workload queue and carbon data caching.
-
-**Rationale**:
-- Serverless, eliminating always-on database instances
-- Pay-per-request pricing aligns with sustainability goals
-- Built-in TTL for automatic data lifecycle management
-- Global secondary indexes enable efficient queries
-
-**Trade-offs**:
-- Eventually consistent reads may cause slight delays
-- Limited query flexibility compared to SQL databases
-- Requires careful partition key design
-
-### 3. Multi-Strategy Optimization
-
-**Decision**: Support time-shifting, location-shifting, and combined strategies.
-
-**Rationale**:
-- Different workloads have different flexibility constraints
-- Maximizes carbon reduction opportunities
-- Allows gradual adoption (start with time-shifting only)
-- Future-proof as more regions become renewable
-
-**Implementation**:
-- Strategy pattern for optimization algorithms
-- Pluggable scoring functions for different approaches
-- Configurable weights for carbon vs. cost optimization
-
-### 4. Real-Time Carbon Data Integration
-
-**Decision**: Cache external carbon intensity data with 15-minute refresh.
-
-**Rationale**:
-- Grid carbon intensity changes throughout the day
-- External APIs have rate limits and latency
-- 15-minute cache balances freshness with API costs
-- Fallback to historical patterns if API fails
-
-**Data Sources**:
-- Primary: Electricity Maps API (requires subscription)
-- Secondary: WattTime API (marginal emissions)
-- Fallback: Historical regional patterns
-
-### 5. Graviton and Spot Instance Defaults
-
-**Decision**: Enable ARM processors and spot instances by default.
-
-**Rationale**:
-- Graviton provides 20% better energy efficiency
-- Spot instances use excess capacity (already provisioned)
-- Both reduce cost and carbon footprint
-- Can be disabled for compatibility
-
-**Implementation**:
-- Architecture-aware Lambda deployment
-- Spot instance integration for long-running workloads
-- Automatic fallback to on-demand if needed
-
-## Security Architecture
-
-### 1. Least Privilege IAM
-
-Each Lambda function has minimal required permissions:
-- Scheduler: Read/write DynamoDB, invoke executor
-- Executor: Read/write specific DynamoDB items, emit metrics
-- Monitor: Read carbon APIs, write cache, emit metrics
-
-### 2. VPC Integration
-
-- Functions deploy into customer VPC for network isolation
-- Security groups restrict outbound to required services
-- VPC endpoints eliminate internet gateway requirements
-
-### 3. Encryption
-
-- All data encrypted at rest (DynamoDB, CloudWatch Logs)
-- TLS for all API communications
-- KMS integration for sensitive workload data
-
-## Scalability Patterns
-
-### 1. Horizontal Scaling
-
-- Lambda concurrency scales automatically
-- DynamoDB on-demand handles any workload
-- Regional distribution reduces single-region load
-
-### 2. Workload Batching
-
-- Small workloads grouped for efficiency
-- Reduces Lambda invocation overhead
-- Improves carbon calculations accuracy
-
-### 3. Multi-Region Architecture
-
-- Each region operates independently
-- Cross-region replication for global workloads
-- Regional carbon data caching reduces API calls
-
-## Cost Optimization
-
-### 1. Intelligent Resource Allocation
-
-- Right-sized Lambda memory based on workload type
-- Automatic adjustment based on execution patterns
-- Spot instance bidding for cost reduction
-
-### 2. Efficient Scheduling
-
-- Batch similar workloads together
-- Minimize Lambda cold starts
-- Use EventBridge for cost-effective scheduling
-
-### 3. Data Lifecycle
-
-- Automatic TTL on temporary data
-- Compressed storage for historical metrics
-- Tiered storage for long-term analytics
-
-## Monitoring Strategy
-
-### 1. Carbon Metrics
-
-Primary metrics tracked:
-- **Carbon Intensity**: Real-time grid carbon per region
-- **Carbon Emissions**: Actual gCO2eq per workload
-- **Carbon Saved**: Compared to baseline execution
-- **Renewable Percentage**: Clean energy usage
-
-### 2. Operational Metrics
-
-- **Workload Queue Depth**: Pending executions
-- **Execution Latency**: Time from schedule to completion
-- **Success Rate**: Successful vs. failed executions
-- **Cost Savings**: From optimization strategies
-
-### 3. Dashboards
-
-Automated CloudWatch dashboard includes:
-- Regional carbon intensity heatmap
-- Workload execution timeline
-- Carbon savings accumulator
-- Cost optimization tracker
-
-## Integration Patterns
-
-### 1. Event-Driven Integration
-
-```ruby
-# Trigger carbon-aware execution from S3 upload
-s3_trigger = aws_s3_bucket_notification(bucket, {
-  lambda_function_arn: carbon_compute.executor_function.arn,
-  events: ["s3:ObjectCreated:*"],
-  filter_prefix: "input/"
-})
-```
-
-### 2. API Gateway Integration
-
-```ruby
-# REST API for workload submission
-api = aws_api_gateway_rest_api(:carbon_api, {
-  name: "carbon-aware-workload-api"
-})
-
-aws_api_gateway_integration(api, {
-  integration_type: "AWS_PROXY",
-  uri: carbon_compute.scheduler_function.invoke_arn
-})
-```
-
-### 3. Step Functions Integration
-
-```ruby
-# Complex workflows with carbon awareness
-workflow = aws_stepfunctions_state_machine(:carbon_workflow, {
-  definition: {
-    StartAt: "CheckCarbon",
-    States: {
-      CheckCarbon: {
-        Type: "Task",
-        Resource: carbon_compute.monitor_function.arn,
-        Next: "DecideExecution"
-      },
-      DecideExecution: {
-        Type: "Choice",
-        Choices: [{
-          Variable: "$.carbonIntensity",
-          NumericLessThan: 150,
-          Next: "ExecuteNow"
-        }],
-        Default: "ScheduleLater"
-      }
-    }
-  }
-})
-```
-
-## Future Enhancements
-
-### 1. Machine Learning Integration
-
-- Predict future carbon intensity patterns
-- Optimize scheduling based on historical data
-- Anomaly detection for unusual grid conditions
-
-### 2. Multicloud Support
-
-- Extend to Azure and GCP regions
-- Unified carbon tracking across clouds
-- Workload migration between cloud providers
-
-### 3. Carbon Credit Integration
-
-- Automatic carbon offset purchasing
-- Renewable energy certificate tracking
-- Sustainability reporting automation
-
-### 4. Enhanced Data Sources
-
-- Direct utility API integration
-- Real-time renewable energy data
-- Weather-based predictions
-
-## Performance Characteristics
-
-### 1. Latency
-
-- Scheduling decision: < 100ms
-- Carbon data fetch: < 500ms (cached)
-- Workload startup: 1-5 seconds (cold start)
-
-### 2. Throughput
-
-- Scheduler: 1000+ workloads/second
-- Executor: Limited by Lambda concurrency
-- Monitor: Updates all regions in < 30 seconds
-
-### 3. Reliability
-
-- 99.9% uptime (Lambda SLA)
-- Automatic retry with exponential backoff
-- Dead letter queues for failed workloads
-
-## Deployment Considerations
-
-### 1. Regional Deployment
-
-Deploy in regions with:
-- Low baseline carbon intensity
-- Good renewable energy availability
-- Low latency to workload data
-
-### 2. Compliance
-
-Consider:
-- Data residency requirements
-- Workload sensitivity to delays
-- Audit trail requirements
-
-### 3. Migration Strategy
-
-1. Start with non-critical batch workloads
-2. Monitor carbon savings and performance
-3. Gradually expand to more workload types
-4. Implement organization-wide standards
-
-## Conclusion
-
-The Carbon Aware Compute component demonstrates that sustainability and performance are not mutually exclusive. By intelligently scheduling workloads based on grid carbon intensity, organizations can significantly reduce their environmental impact while often reducing costs. The serverless architecture ensures the solution itself maintains minimal carbon footprint, creating a truly sustainable approach to cloud computing.

data/lib/pangea/components/carbon_aware_compute/README.md

@@ -1,194 +0,0 @@
-# Carbon Aware Compute Component
-
-A sophisticated infrastructure component that enables carbon-aware workload scheduling and execution, automatically optimizing for both carbon emissions and cost efficiency by shifting workloads across time and regions based on grid carbon intensity.
-
-## Overview
-
-The Carbon Aware Compute component implements intelligent workload scheduling that considers real-time carbon intensity data to minimize the environmental impact of computational workloads. It supports multiple optimization strategies including time-shifting (delaying workloads to cleaner times) and location-shifting (moving workloads to greener regions).
-
-## Key Features
-
-- **Real-time Carbon Tracking**: Monitors grid carbon intensity across multiple AWS regions
-- **Intelligent Scheduling**: Automatically schedules workloads during low-carbon periods
-- **Multi-Region Optimization**: Shifts workloads to regions with cleaner energy grids
-- **Flexible Strategies**: Supports time-shifting, location-shifting, or combined approaches
-- **Comprehensive Monitoring**: Built-in CloudWatch dashboard for carbon and efficiency metrics
-- **Cost Optimization**: Reduces costs while minimizing carbon footprint
-- **Graviton Support**: Leverages energy-efficient ARM processors
-- **Spot Instance Integration**: Uses excess capacity for additional sustainability
-
-## Usage
-
-```ruby
-carbon_compute = Pangea::Components::CarbonAwareCompute.build(
-  name: "sustainable-batch-processor",
-  workload_type: "batch",
-  vpc_id: "vpc-12345",
-  subnet_ids: ["subnet-1a", "subnet-1b"],
-  
-  # Carbon optimization settings
-  optimization_strategy: "combined",
-  carbon_intensity_threshold: 150,  # gCO2/kWh
-  preferred_regions: ["us-west-2", "eu-north-1", "ca-central-1"],
-  
-  # Execution windows
-  min_execution_window_hours: 2,
-  max_execution_window_hours: 24,
-  deadline_hours: 48,
-  
-  # Performance settings
-  cpu_units: 512,
-  memory_mb: 1024,
-  use_graviton: true,
-  use_spot_instances: true,
-  
-  tags: {
-    "Environment" => "production",
-    "Sustainability" => "enabled"
-  }
-)
-
-# Access outputs
-puts carbon_compute.scheduler_function_arn
-puts carbon_compute.dashboard_url
-```
-
-## Optimization Strategies
-
-### Time Shifting
-Delays workload execution to times when the electricity grid is cleaner (e.g., nighttime when renewable sources are more prevalent).
-
-### Location Shifting  
-Moves workloads to AWS regions powered by cleaner energy sources (e.g., Oregon with hydroelectric power).
-
-### Combined Strategy
-Uses both time and location shifting to find the optimal execution plan with the lowest carbon footprint.
-
-### Efficiency First
-Prioritizes computational efficiency to reduce overall energy consumption regardless of grid carbon intensity.
-
-## Carbon Intensity Thresholds
-
-The component uses the following carbon intensity classifications:
-
-- **Very Low**: < 50 gCO2/kWh (e.g., Quebec, Norway)
-- **Low**: 50-150 gCO2/kWh (e.g., Oregon, Sweden)
-- **Medium**: 150-300 gCO2/kWh (e.g., California, Ireland)
-- **High**: 300-500 gCO2/kWh (e.g., Virginia, Germany)
-- **Very High**: > 500 gCO2/kWh (e.g., Singapore, Poland)
-
-## Preferred Green Regions
-
-Default regions with high renewable energy usage:
-
-1. **us-west-2** (Oregon): ~80% hydroelectric power
-2. **eu-north-1** (Stockholm): ~95% renewable energy
-3. **eu-west-1** (Ireland): Carbon neutral data centers
-4. **ca-central-1** (Montreal): ~99% hydroelectric power
-
-## Monitoring and Metrics
-
-The component automatically creates a CloudWatch dashboard showing:
-
-- Real-time carbon emissions (gCO2eq)
-- Regional carbon intensity (gCO2/kWh)
-- Workload scheduling metrics
-- Computational efficiency scores
-- Cost savings from optimization
-
-## Architecture Components
-
-- **Lambda Functions**: Scheduler, executor, and monitor functions
-- **DynamoDB Tables**: Workload queue and carbon data cache
-- **EventBridge Rules**: Automated scheduling and monitoring
-- **CloudWatch**: Metrics, alarms, and dashboards
-- **IAM Roles**: Least-privilege access policies
-
-## Best Practices
-
-1. **Set Realistic Deadlines**: Allow sufficient time windows for optimal scheduling
-2. **Monitor Carbon Savings**: Review dashboard regularly to track environmental impact
-3. **Use Graviton**: Enable ARM-based instances for 20% better energy efficiency
-4. **Implement Batching**: Group small workloads for more efficient execution
-5. **Consider Time Zones**: Account for regional time differences in scheduling
-
-## Example: ML Training Pipeline
-
-```ruby
-ml_training = Pangea::Components::CarbonAwareCompute.build(
-  name: "ml-model-training",
-  workload_type: "ml_training",
-  vpc_id: vpc_id,
-  subnet_ids: private_subnet_ids,
-  
-  # ML workloads can be flexible with timing
-  optimization_strategy: "time_shifting",
-  carbon_intensity_threshold: 100,
-  min_execution_window_hours: 1,
-  max_execution_window_hours: 72,  # 3 days flexibility
-  
-  # Higher resources for ML
-  cpu_units: 4096,
-  memory_mb: 16384,
-  ephemeral_storage_gb: 100,
-  
-  # Cost optimization for long-running training
-  use_spot_instances: true,
-  
-  tags: {
-    "Project" => "sustainability-ml",
-    "CostCenter" => "research"
-  }
-)
-```
-
-## Integration with CI/CD
-
-```ruby
-# In your CI/CD pipeline
-ci_builds = Pangea::Components::CarbonAwareCompute.build(
-  name: "ci-build-jobs",
-  workload_type: "batch",
-  vpc_id: vpc_id,
-  subnet_ids: subnet_ids,
-  
-  # CI can usually wait a bit
-  optimization_strategy: "time_shifting",
-  min_execution_window_hours: 0,
-  max_execution_window_hours: 4,
-  deadline_hours: 6,
-  
-  # Smaller resources for builds
-  cpu_units: 1024,
-  memory_mb: 2048
-)
-```
-
-## Carbon Data Sources
-
-The component supports multiple carbon intensity data sources:
-
-- **Electricity Maps**: Real-time grid carbon intensity
-- **WattTime**: Marginal emissions data
-- **Custom Sources**: Integrate your own data providers
-
-## Troubleshooting
-
-### High Carbon Alerts
-If receiving frequent high carbon alerts:
-1. Increase the carbon intensity threshold
-2. Add more preferred regions
-3. Extend execution time windows
-
-### Workloads Not Executing
-Check:
-1. DynamoDB table permissions
-2. Lambda function logs
-3. EventBridge rule status
-4. Network connectivity in VPC
-
-### Metrics Not Appearing
-Ensure:
-1. CloudWatch permissions are correct
-2. Lambda functions are executing successfully
-3. Carbon reporting is enabled