cdk-factory 0.16.15__py3-none-any.whl → 0.20.0__py3-none-any.whl

cdk_factory/stack_library/lambda_edge/EDGE_LOG_RETENTION_TODO.md

@@ -0,0 +1,226 @@
+# Lambda@Edge Log Retention - Implementation Plan
+
+## 🚨 Current Status: DISABLED
+
+Lambda@Edge log retention configuration has been **disabled** because edge log groups are created on-demand when the function is invoked at edge locations, not during CloudFormation deployment.
+
+## 🔍 Problem Analysis
+
+### Why Deployment-Time Configuration Fails
+1. **On-Demand Creation**: Lambda@Edge log groups are created only when the function is actually invoked at edge locations
+2. **Timing Issue**: CloudFormation deployment happens before any edge invocations occur
+3. **Error**: `The specified log group does not exist` when trying to set retention policies
+
+### Log Group Naming Pattern
+```
+Pattern: /aws/lambda/{edge-region}.{function-name}
+Example: /aws/lambda/eu-central-1.trav-talks-blue-green-edge-function
+Location: All edge log groups are created in us-east-1
+```
+
+## 💡 Proposed Solutions
+
+### Solution 1: EventBridge + Lambda (Recommended)
+```yaml
+# EventBridge rule to detect log group creation
+EventPattern:
+  source: ["aws.logs"]
+  detail-type: ["AWS API Call via CloudTrail"]
+  detail:
+    eventSource: ["logs.amazonaws.com"]
+    eventName: ["CreateLogGroup"]
+    requestParameters:
+      logGroupName: ["/aws/lambda/*.edge-function"]
+```
+
+**Implementation:**
+1. Create EventBridge rule that triggers on log group creation
+2. Lambda function receives event and sets retention policy
+3. Automatic handling of new edge log groups
+
+**Pros:**
+- Automatic and real-time
+- No manual intervention required
+- Handles all edge regions
+
+**Cons:**
+- Additional Lambda function to maintain
+- Requires CloudTrail enabled for CloudWatch Logs
+
+### Solution 2: Periodic Lambda Function
+```python
+def lambda_handler(event, context):
+    # Scan for edge log groups
+    log_groups = logs.describe_log_groups(
+        logGroupNamePrefix='/aws/lambda/eu-central-1.trav-talks-blue-green-edge-function'
+    )
+    
+    # Apply retention policy
+    for log_group in log_groups['logGroups']:
+        logs.put_retention_policy(
+            logGroupName=log_group['logGroupName'],
+            retentionInDays=7
+        )
+```
+
+**Implementation:**
+1. Create Lambda function on schedule (e.g., every hour)
+2. Scan for edge log groups with function name pattern
+3. Apply retention policy if not already set
+
+**Pros:**
+- Simple to implement
+- No CloudTrail dependency
+- Can handle existing log groups
+
+**Cons:**
+- Not real-time (delayed retention)
+- Runs periodically even when not needed
+
+### Solution 3: Post-Deployment Script
+```bash
+#!/bin/bash
+# Wait for edge log groups to appear
+function_name="trav-talks-blue-green-edge-function"
+edge_regions=("eu-central-1" "eu-west-1" "ap-southeast-1")
+
+for region in "${edge_regions[@]}"; do
+    log_group="/aws/lambda/${region}.${function_name}"
+    
+    # Wait for log group to exist
+    until aws logs describe-log-groups --log-group-name-prefix "$log_group" --region us-east-1; do
+        echo "Waiting for log group: $log_group"
+        sleep 30
+    done
+    
+    # Set retention policy
+    aws logs put-retention-policy --log-group-name "$log_group" --retention-in-days 7 --region us-east-1
+done
+```
+
+**Implementation:**
+1. Script runs after Lambda@Edge deployment
+2. Waits for edge log groups to be created
+3. Sets retention policies when they appear
+
+**Pros:**
+- Direct control over timing
+- No additional AWS resources needed
+
+**Cons:**
+- Manual process
+- Hard to determine when log groups will appear
+- Not automated
+
+### Solution 4: CloudWatch Logs Subscription
+```python
+# Lambda triggered by log group creation via subscription filter
+def lambda_handler(event, context):
+    for record in event['Records']:
+        log_group = record['logGroup']
+        if 'edge-function' in log_group:
+            # Set retention policy
+            logs.put_retention_policy(
+                logGroupName=log_group,
+                retentionInDays=7
+            )
+```
+
+**Implementation:**
+1. Create subscription filter on log group pattern
+2. Lambda function triggered by log events
+3. Set retention policy on first log event
+
+**Pros:**
+- Event-driven
+- No CloudTrail needed
+
+**Cons:**
+- Requires log group to exist first
+- Complex subscription filter setup
+
+## 🎯 Recommended Implementation
+
+### Phase 1: Quick Win (Solution 2)
+Implement periodic Lambda function as temporary solution:
+- Easy to implement quickly
+- Solves immediate problem
+- Can be replaced later with better solution
+
+### Phase 2: Production Solution (Solution 1)
+Implement EventBridge + Lambda for production:
+- Real-time response
+- Automatic handling
+- Best long-term solution
+
+## 📋 Implementation Steps for Solution 1
+
+### 1. Create EventBridge Rule
+```python
+event_rule = events.Rule(
+    self, "EdgeLogGroupRule",
+    event_pattern=events.EventPattern(
+        source=["aws.logs"],
+        detail_type=["AWS API Call via CloudTrail"],
+        detail={
+            "eventSource": ["logs.amazonaws.com"],
+            "eventName": ["CreateLogGroup"],
+            "requestParameters": {
+                "logGroupName": [{"prefix": "/aws/lambda/"}]
+            }
+        }
+    )
+)
+```
+
+### 2. Create Lambda Function
+```python
+retention_handler = _lambda.Function(
+    self, "EdgeLogRetentionHandler",
+    runtime=_lambda.Runtime.PYTHON_3_9,
+    handler="handler.lambda_handler",
+    code=_lambda.Code.from_asset("lambda/edge_log_retention"),
+    environment={
+        "RETENTION_DAYS": "7",
+        "FUNCTION_NAME_PATTERN": "*edge-function"
+    }
+)
+```
+
+### 3. Add Permissions
+```python
+retention_handler.add_to_role_policy(
+    iam.PolicyStatement(
+        actions=["logs:PutRetentionPolicy", "logs:DescribeLogGroups"],
+        resources=["*"]
+    )
+)
+```
+
+### 4. Connect EventBridge to Lambda
+```python
+event_rule.add_target(targets.LambdaFunction(retention_handler))
+```
+
+## 🔧 Current Configuration
+
+The edge log retention configuration is currently **disabled** in the Lambda Edge stack:
+
+```python
+def _configure_edge_log_retention(self, function_name: str) -> None:
+    # DISABLED: See implementation plan above
+    logger.warning("Edge log retention disabled - see TODO for implementation")
+    return
+```
+
+## 📊 Configuration Impact
+
+| Setting | Current Behavior | Target Behavior |
+|---------|------------------|-----------------|
+| `edge_log_retention_days` | Warning logged, no action applied | Retention policy set on all edge log groups |
+| Edge log groups | Created with default retention (never expire) | Created with specified retention (e.g., 7 days) |
+| Cost impact | Potential high log storage costs | Controlled log storage costs |
+
+---
+
+**Status**: Ready for implementation when edge log retention is required.

cdk_factory/stack_library/lambda_edge/LAMBDA_EDGE_LOG_RETENTION_BLOG.md

@@ -0,0 +1,215 @@
+# The Lambda@Edge Log Retention Nightmare: A Tale of CloudFormation Frustration
+
+## 🎭 The Setup
+
+You've just built an amazing Lambda@Edge function. It's deployed globally, handling requests at the edge with lightning speed. You're feeling proud - until you get the AWS bill and realize those edge logs are costing you a fortune because they never expire.
+
+"No problem," you think. "I'll just set a log retention policy in my CloudFormation template."
+
+Famous last words.
+
+## 🚨 The Problem: "The specified log group does not exist"
+
+Your deployment starts, and then you see it:
+
+```
+EdgeLogRetentioneucentral180637808 - FAILED
+Received response status [FAILED] from custom resource. 
+Message returned: The specified log group does not exist.
+```
+
+Wait, what? How can the log group not exist? I'm deploying a Lambda function!
+
+## 🔍 The Investigation: Understanding Lambda@Edge Logging
+
+After hours of digging through AWS documentation and Stack Overflow posts, you discover the truth:
+
+**Lambda@Edge log groups are created on-demand.**
+
+Not during deployment. Not when you create the function. Only when the function is actually invoked at an edge location.
+
+This means:
+- Your CloudFormation template runs
+- Lambda@Edge function gets deployed to edge locations
+- Custom resource tries to set log retention
+- **FAILS** because log groups don't exist yet
+- Log groups get created later when someone actually uses your function
+
+## 🤯 The Mind-Bending Architecture
+
+Here's where it gets even more confusing:
+
+```bash
+# Log group naming pattern
+/aws/lambda/{edge-region}.{function-name}
+
+# Examples
+/aws/lambda/eu-central-1.my-edge-function
+/aws/lambda/ap-southeast-1.my-edge-function
+/aws/lambda/us-west-2.my-edge-function
+
+# BUT THEY'RE ALL IN us-east-1!
+arn:aws:logs:us-east-1:123456789:log-group:/aws/lambda/eu-central-1.my-edge-function
+```
+
+Yes, you read that right. A log group named `/aws/lambda/eu-central-1.my-function` is physically located in `us-east-1`. This makes perfect sense to someone at AWS, apparently.
+
+## 💡 The Failed Attempts
+
+### Attempt 1: Custom Resource with SDK Calls
+```python
+cr.AwsCustomResource(
+    self, "EdgeLogRetention",
+    on_update={
+        "service": "CloudWatchLogs",
+        "action": "putRetentionPolicy",
+        "parameters": {
+            "logGroupName": "/aws/lambda/eu-central-1.my-function",
+            "retentionInDays": 7
+        }
+    }
+)
+```
+**Result**: "The specified log group does not exist"
+
+### Attempt 2: Fix the Service Name
+```
+Package @aws-sdk/client-logs does not exist
+```
+Oh, it's `CloudWatchLogs` not `Logs`. Easy fix!
+
+**Result**: Still "The specified log group does not exist"
+
+### Attempt 3: Fix IAM Permissions
+```python
+policy=cr.AwsCustomResourcePolicy.from_statements([
+    iam.PolicyStatement(
+        actions=["logs:PutRetentionPolicy", "logs:DeleteRetentionPolicy"],
+        resources=["arn:aws:logs:us-east-1:*:log-group:/aws/lambda/eu-central-1.my-function*"]
+    )
+])
+```
+**Result**: "User is not authorized to perform: logs:PutRetentionPolicy"
+
+### Attempt 4: Fix Region and Permissions
+```python
+# All edge log groups are in us-east-1, not the edge region!
+resources=["arn:aws:logs:us-east-1:*:log-group:/aws/lambda/eu-central-1.my-function*"]
+```
+**Result**: "The specified log group does not exist"
+
+## 🎭 The Realization
+
+You finally understand: **You're trying to set retention on something that doesn't exist yet.**
+
+It's like trying to put a roof on a house before the foundation is poured. Lambda@Edge log groups are created when the function is first invoked at that edge location, which could be minutes, hours, or days after deployment.
+
+## 🤔 The Solutions We Considered
+
+### Solution 1: EventBridge + Lambda (The "Proper" Way)
+```yaml
+EventPattern:
+  source: ["aws.logs"]
+  detail-type: ["AWS API Call via CloudTrail"]
+  detail:
+    eventSource: ["logs.amazonaws.com"]
+    eventName: ["CreateLogGroup"]
+```
+
+**Pros**: Real-time, automatic, handles all edge regions
+**Cons**: Requires CloudTrail, another Lambda to maintain, complex setup
+
+### Solution 2: Periodic Lambda Function (The "Good Enough" Way)
+```python
+def lambda_handler(event, context):
+    # Scan for edge log groups every hour
+    # Apply retention if needed
+```
+
+**Pros**: Simple, no CloudTrail dependency
+**Cons**: Not real-time, runs even when not needed
+
+### Solution 3: Post-Deployment Script (The "Manual" Way)
+```bash
+# Wait for log groups to appear
+until aws logs describe-log-groups --log-group-name-prefix "/aws/lambda/eu-central-1.my-function"; do
+    sleep 30
+done
+```
+
+**Pros**: Direct control, no extra resources
+**Cons**: Manual, timing issues, not scalable
+
+### Solution 4: CloudWatch Subscription (The "Overkill" Way)
+**Pros**: Event-driven
+**Cons**: Complex, requires log group to exist first
+
+## 🏳️ The Decision: We're Tabling This
+
+After weighing all options, we decided to **disable edge log retention for now**:
+
+```python
+def _configure_edge_log_retention(self, function_name: str) -> None:
+    # DISABLED: Edge log groups don't exist during deployment
+    logger.warning("Edge log retention disabled - log groups are created on-demand")
+    return
+```
+
+Why?
+
+1. **Complexity vs. Benefit**: The solutions are complex for what should be a simple configuration
+2. **Cost Impact**: We'll monitor costs and implement something if it becomes a problem
+3. **AWS Responsibility**: This feels like something AWS should handle better
+4. **Time Constraints**: We have features to ship, not logging architecture to perfect
+
+## 🤷‍♂️ The Frustration
+
+Here's what frustrates us most:
+
+1. **No Clear Documentation**: AWS doesn't clearly document this limitation
+2. **Misleading Error Messages**: "Log group does not exist" doesn't explain WHY
+3. **Counterintuitive Architecture**: Why are edge logs in us-east-1?
+4. **No Built-in Solution**: AWS should provide a way to set default retention
+
+## 📣 The Call for Help
+
+**We're throwing this out to the community:**
+
+Has anyone solved this elegantly? Are we missing something obvious? Is there a magical CloudFormation feature we don't know about?
+
+### What We Tried:
+- ✅ Custom resources with proper IAM permissions
+- ✅ Correct service names (`CloudWatchLogs` not `Logs`)
+- ✅ Right region targeting (`us-east-1` for all edge logs)
+- ✅ Multiple deployment approaches
+
+### What We Need:
+- 🤔 A way to set retention policies on edge log groups during deployment
+- 🤔 Or a clean post-deployment solution that doesn't require manual intervention
+- 🤔 Or confirmation that this is impossible and we should stop trying
+
+## 🎯 The Current State
+
+For now, our Lambda@Edge functions deploy successfully with a warning:
+
+```
+Edge log retention configuration disabled - log groups are created on-demand.
+Desired retention: 7 days. See TODO for implementation approach.
+```
+
+We'll monitor the costs and implement one of the solutions if it becomes a real problem. But for now, we're choosing **shipping features over perfect logging architecture**.
+
+## 🔮 The Future
+
+Hopefully, AWS will:
+1. **Document this limitation** clearly
+2. **Provide a built-in solution** for edge log retention
+3. **Fix the confusing architecture** (or at least explain it better)
+
+Until then, we'll keep an eye on our logging costs and hope someone in the community has a better solution.
+
+---
+
+**Have you solved this? Drop your solutions in the comments! Let's save the next developer from this nightmare.**
+
+#AWS #Lambda #CloudFormation #EdgeComputing #Frustration #LoggingNightmare