agentic-threat-hunting-framework 0.2.2__py3-none-any.whl → 0.2.4__py3-none-any.whl

athf/data/hunts/H-0003.md

@@ -0,0 +1,546 @@
+---
+hunt_id: H-0003
+title: AWS Lambda Persistence Detection
+status: completed
+date: 2025-11-28
+hunter: Sydney Marrone
+platform: [AWS Cloud]
+tactics: [persistence, privilege-escalation]
+techniques: [T1546.004, T1098, T1078.004]
+data_sources: [AWS CloudTrail, AWS Lambda Logs, IAM Access Analyzer]
+related_hunts: []
+findings_count: 2
+true_positives: 2
+false_positives: 0
+customer_deliverables: []
+tags: [aws, lambda, serverless, iam, persistence, api-gateway, eventbridge, cloud]
+---
+
+# H-0003: AWS Lambda Persistence Detection
+
+**Hunt Metadata**
+
+- **Date:** 2025-11-28
+- **Hunter:** [Your Name]
+- **Status:** Completed
+- **MITRE ATT&CK:** T1546.004 - Event Triggered Execution, T1098 - Account Manipulation
+
+---
+
+## LEARN: Prepare the Hunt
+
+### Hypothesis Statement
+
+Adversaries with initial access to AWS environments will establish persistence by creating Lambda functions with overly permissive IAM roles, scheduled triggers (EventBridge), or public-facing API Gateway endpoints to maintain backdoor access. These functions can execute on-demand or automatically, enabling long-term control even after initial credentials are revoked.
+
+### Threat Context
+
+AWS Lambda provides serverless compute that executes code in response to events. Adversaries abuse this by:
+- Creating Lambda functions with administrative IAM roles for privilege escalation
+- Configuring API Gateway to expose functions as public HTTP endpoints
+- Using EventBridge (CloudWatch Events) for scheduled execution
+- Deploying backdoor code for command execution, data exfiltration, or reconnaissance
+
+**Key attack patterns:**
+- **Function naming:** Masquerade as legitimate services (`sys-health-check`, `cloudwatch-logs-processor`, `backup-automation`)
+- **IAM roles:** Attach `AdministratorAccess`, `PowerUserAccess`, or overly permissive custom policies
+- **Triggers:** API Gateway (public), EventBridge schedules (periodic), or event source mappings
+- **Code characteristics:** Base64-encoded payloads, reverse shells, environment variable exfiltration, AWS API enumeration
+
+This technique maps to:
+- **T1546.004 - Event Triggered Execution:** Lambda functions triggered by events
+- **T1098 - Account Manipulation:** IAM role manipulation for persistence
+- **T1078.004 - Cloud Accounts:** Compromised cloud credentials enable initial Lambda creation
+
+Recent threat intel indicates APT groups and initial access brokers increasingly target cloud environments for persistence mechanisms that survive credential rotation.
+
+### ABLE Scoping
+
+Define your hunt scope using the ABLE framework:
+
+| **Field**   | **Your Input** |
+|-------------|----------------|
+| **Actor** *(Optional)* | N/A - Focus on behavioral patterns consistent with persistence establishment |
+| **Behavior** | Malicious Lambda function creation with overly permissive IAM roles and public/scheduled triggers (T1546.004, T1098) |
+| **Location** | AWS production accounts (all regions where Lambda is enabled) |
+| **Evidence** | **Source:** AWS CloudTrail<br>**Key Fields:** eventName (CreateFunction, UpdateFunctionConfiguration, AttachRolePolicy, CreateApi, CreateEventSourceMapping), userIdentity, requestParameters, sourceIPAddress, userAgent<br>**Example:** CreateFunction event with requestParameters.role containing AdministratorAccess, followed by CreateApi for API Gateway endpoint |
+
+### Threat Intel & Research
+
+- **MITRE ATT&CK Techniques:**
+  - `T1546.004 - Event Triggered Execution` (Lambda functions respond to events)
+  - `T1098 - Account Manipulation` (IAM role escalation)
+  - `T1078.004 - Cloud Accounts` (compromised credentials for initial access)
+  - `T1098.001 - Additional Cloud Credentials` (IAM role creation)
+- **CTI Sources & References:**
+  - [MITRE ATT&CK - T1546.004](https://attack.mitre.org/techniques/T1546/004/)
+  - AWS Lambda persistence documented in MITRE Cloud Matrix
+  - Multiple APT groups observed using serverless for persistence (public reporting from threat intel vendors)
+  - Service account compromise remains primary initial access vector for cloud environments
+- **Historical Context:**
+  - Recent trend: Attackers prefer Lambda over EC2 for persistence (harder to detect, lower cost, scales automatically)
+  - Known false positives: DevOps automation, CI/CD pipelines, infrastructure-as-code deployments
+  - Legitimate use cases: Scheduled Lambda for log processing, API Gateway for microservices, EventBridge for automation
+
+### Related Tickets
+
+| **Team** | **Ticket/Details** |
+|----------|-------------------|
+| **SOC/IR** | INC-2341 - Compromised IAM service account `developer-jenkins` |
+| **Threat Intel** | TI-0092 - Increased cloud persistence TTPs in recent campaigns |
+| **Detection Engineering** | DET-0058 - Implement Lambda function code analysis for IOCs |
+| **Cloud Security** | CLOUD-0789 - Review and restrict IAM permissions for Lambda roles |
+
+---
+
+## OBSERVE: Expected Behaviors
+
+### What Normal Looks Like
+
+Legitimate Lambda activity that should not trigger alerts:
+
+- **DevOps automation:** Infrastructure-as-code (Terraform, CloudFormation) creating Lambda functions with documented IAM roles
+- **CI/CD pipelines:** Automated deployments from known service accounts (e.g., `jenkins-prod`, `github-actions`)
+- **Monitoring/logging:** Lambda functions processing CloudWatch Logs, S3 bucket events, or DynamoDB streams
+- **API backends:** Lambda functions behind API Gateway for microservices with appropriate IAM roles (not admin)
+- **Scheduled maintenance:** EventBridge rules triggering Lambda for backups, cleanups, or data processing
+- **Known automation accounts:** Service accounts in whitelist performing legitimate infrastructure operations
+
+### What Suspicious Looks Like
+
+Adversaries will create Lambda functions to establish persistence. We expect to see:
+
+1. **Suspicious IAM policies:**
+   - `AdministratorAccess` or `PowerUserAccess` attached to Lambda execution roles
+   - Wildcard permissions (e.g., `iam:*`, `s3:*`, `lambda:*`)
+   - Privilege escalation paths (e.g., `iam:PassRole`, `iam:AttachRolePolicy`)
+
+2. **Unusual triggers:**
+   - API Gateway endpoints created for newly deployed functions (public-facing backdoor)
+   - EventBridge scheduled rules with unusual frequencies (every minute, odd times)
+   - Event source mappings to unexpected services
+
+3. **Suspicious timing:**
+   - Lambda function created, IAM role attached, and API Gateway configured within minutes
+   - Activity outside business hours (e.g., 02:00-05:00 UTC)
+   - Rapid succession of Lambda-related API calls
+
+4. **Suspicious source:**
+   - Residential IPs or VPN exit nodes (not corporate network)
+   - User agents from AWS CLI in unexpected locations
+   - Service accounts operating from anomalous geolocations
+
+5. **Suspicious function characteristics:**
+   - Function names masquerading as legitimate services (`sys-health-check`, `cloudwatch-processor`)
+   - Runtime mismatches (e.g., Python 3.11 when org standard is Python 3.9)
+   - Execution from non-standard regions
+
+### Expected Observables
+
+- **Processes:** N/A (Lambda is serverless)
+- **Network:** API Gateway endpoints created, HTTP requests to newly created endpoints
+- **Files:** Lambda function code packages (ZIP files), deployment packages in S3
+- **Registry:** N/A (cloud-based hunt)
+- **Authentication:** IAM API calls (CreateFunction, AttachRolePolicy, CreateApi), CloudTrail logs
+
+---
+
+## CHECK: Execute & Analyze
+
+### Data Source Information
+
+- **Index/Data Source:** AWS CloudTrail (all regions)
+- **Time Range:** Last 7 days (2025-11-21 00:00:00 to 2025-11-28 23:59:59)
+- **Events Analyzed:** ~450,000 CloudTrail events (Lambda, IAM, API Gateway, EventBridge)
+- **Data Quality:** Excellent - CloudTrail enabled in all regions, logging to centralized S3 bucket with Splunk ingestion
+
+### Hunting Queries
+
+#### Initial Query: Monitor All Lambda Function Creations
+
+```spl
+index=aws_cloudtrail eventName="CreateFunction"
+| stats count by userIdentity.principalId, requestParameters.functionName, userIdentity.arn, sourceIPAddress
+| sort -count
+```
+
+**Query Notes:**
+
+- Returned 89 CreateFunction events over 7-day period
+- 70% from known CI/CD automation accounts (jenkins-prod, github-actions)
+- 20% from developer accounts during business hours (expected)
+- 10% from unexpected sources requiring investigation
+- Most activity during business hours (09:00-17:00 UTC)
+
+**Initial Findings:**
+- Overwhelming legitimate activity from DevOps automation
+- Need to filter known automation accounts and focus on IAM policy analysis
+
+#### Refined Query: Focus on Suspicious IAM Policies
+
+```spl
+index=aws_cloudtrail (eventName="CreateFunction" OR eventName="UpdateFunctionConfiguration")
+| spath input=requestParameters.role output=lambda_role
+| join type=left lambda_role [
+    search index=aws_cloudtrail (eventName="PutRolePolicy" OR eventName="AttachRolePolicy")
+    | spath input=requestParameters.policyArn output=policy_arn
+    | spath input=requestParameters.policyDocument output=policy_doc
+    | eval policy=coalesce(policy_arn, policy_doc)
+]
+| where policy="*" OR policy LIKE "%AdministratorAccess%" OR policy LIKE "%PowerUserAccess%" OR policy LIKE "%iam:*%"
+| stats values(eventName) as events,
+        values(requestParameters.functionName) as functions,
+        values(sourceIPAddress) as source_ips,
+        values(policy) as policies
+        by userIdentity.principalId, userIdentity.arn, lambda_role
+| sort -count
+```
+
+**Refinement Rationale:**
+
+- Shifted focus from all Lambda creations to those with overly permissive IAM roles
+- Joined CreateFunction events with IAM policy attachment events
+- Filtered for administrative policies (AdministratorAccess, PowerUserAccess, wildcard permissions)
+- This identifies potential privilege escalation via Lambda execution roles
+
+**Query Results:**
+
+- 8 Lambda functions with suspicious IAM policies
+- 5 legitimate admin functions for automation (documented in runbooks)
+- 3 require investigation (unknown accounts, unexpected timing)
+
+**Problem:** Still includes legitimate admin Lambda functions for infrastructure automation
+
+#### Final Query: Anomaly + Permissions + Trigger Analysis
+
+```spl
+index=aws_cloudtrail (eventName="CreateFunction" OR eventName="UpdateFunctionConfiguration" OR eventName="CreateEventSourceMapping")
+| spath input=requestParameters.role output=lambda_role
+| eval function_name=coalesce('requestParameters.functionName', 'requestParameters.FunctionName')
+| join type=left lambda_role [
+    search index=aws_cloudtrail eventName="AttachRolePolicy"
+    | spath input=requestParameters.policyArn output=policy_arn
+    | where policy_arn LIKE "%AdministratorAccess%" OR policy_arn LIKE "%PowerUserAccess%"
+    | stats values(policy_arn) as attached_policies by requestParameters.roleName
+    | rename requestParameters.roleName as lambda_role
+]
+| join type=left function_name [
+    search index=aws_cloudtrail (eventName="CreateEventSourceMapping" OR eventName="CreateApi" OR eventName="CreateRestApi")
+    | stats values(eventName) as trigger_events by requestParameters.functionName
+    | rename requestParameters.functionName as function_name
+]
+| where NOT [| inputlookup known_automation_accounts.csv | fields userIdentity.principalId]
+| eval time_since_creation=now()-_time
+| where time_since_creation < 3600
+| stats earliest(_time) as first_event,
+        latest(_time) as last_event,
+        values(eventName) as events,
+        values(function_name) as functions,
+        values(lambda_role) as roles,
+        values(attached_policies) as policies,
+        values(trigger_events) as triggers,
+        values(sourceIPAddress) as source_ips,
+        values(userAgent) as user_agents
+        by userIdentity.principalId, userIdentity.arn
+| eval duration_minutes=round((last_event-first_event)/60, 2)
+| where isnotnull(attached_policies) AND isnotnull(triggers)
+| sort -duration_minutes
+```
+
+**Refinement Rationale:**
+
+- Multi-dimensional analysis: Lambda creation + IAM policy + trigger mechanism
+- Temporal analysis: Events occurring within 1 hour (indicates rapid setup)
+- Whitelist known automation accounts to reduce false positives
+- Focus on functions with both administrative IAM roles AND public/scheduled triggers
+- This identifies complete persistence setup (function + permissions + accessibility)
+
+**Final Results:**
+
+- **2 true positives:** Malicious Lambda functions with admin roles and public API Gateway endpoints
+- **0 false positives:** Legitimate automation filtered out via whitelist
+- **Success!** Query identified full attack chain from creation to weaponization
+
+### Visualization & Analytics
+
+- **Timeline:** Lambda creation events show clustering during maintenance windows (expected) and isolated events during off-hours (suspicious)
+- **Heatmap:** User vs. IAM role shows `developer-jenkins` creating Lambda with `AdministratorAccess` (anomalous)
+- **Network diagram:** API Gateway endpoints linked to suspicious Lambda functions
+- **Geolocation:** Source IPs for suspicious Lambda creations map to residential ISP (103.253.145.22)
+
+### Query Performance
+
+**What Worked Well:**
+
+- CloudTrail comprehensive logging captured full attack lifecycle
+- Join operations between Lambda, IAM, and API Gateway events revealed complete picture
+- Temporal analysis (< 1 hour window) identified rapid setup indicative of attacker behavior
+- Known automation whitelist reduced noise by 87%
+- Multi-event correlation (CreateFunction + AttachRolePolicy + CreateApi) high signal
+
+**What Didn't Work:**
+
+- Initial query too broad - captured all Lambda activity including legitimate DevOps
+- IAM policy analysis challenging due to varied policy document formats (inline vs. managed)
+- Query performance slow on 7-day window (450k events) - consider reducing to 24-48h for real-time detection
+- No baseline of "normal" Lambda function names to compare against
+- User agent strings inconsistently formatted across AWS CLI versions
+
+**Iterations Made:**
+
+- Iteration 1: Added IAM policy join to focus on overly permissive roles (reduced results by 70%)
+- Iteration 2: Added trigger analysis (API Gateway, EventBridge) to identify accessibility
+- Iteration 3: Implemented whitelist for known automation accounts
+- Iteration 4: Added temporal analysis (< 1 hour window) to identify rapid setup
+
+### Manual Validation Steps
+
+```bash
+# Retrieve Lambda function configuration
+aws lambda get-function --function-name sys-health-check-v2
+
+# Review Lambda function code (download and inspect)
+aws lambda get-function --function-name sys-health-check-v2 --query 'Code.Location'
+
+# Check IAM role permissions
+aws iam get-role --role-name lambda-admin-role-temp
+aws iam list-attached-role-policies --role-name lambda-admin-role-temp
+
+# List API Gateway endpoints
+aws apigateway get-rest-apis
+
+# Check EventBridge rules
+aws events list-rules | grep cloudwatch-logs-processor
+
+# Review CloudTrail for additional context
+aws cloudtrail lookup-events --lookup-attributes AttributeKey=Username,AttributeValue=developer-jenkins
+```
+
+**Validation Results:**
+
+- Confirmed `sys-health-check-v2` Lambda function contains base64-encoded reverse shell
+- IAM role `lambda-admin-role-temp` has `AdministratorAccess` (full account control)
+- API Gateway endpoint publicly accessible without authentication
+- Source IP 103.253.145.22 maps to residential ISP (suspicious for service account)
+- `developer-jenkins` service account compromised (credentials leaked or stolen)
+
+---
+
+## KEEP: Findings & Response
+
+### Executive Summary
+
+This hunt investigated Lambda-based persistence mechanisms (T1546.004, T1098) in AWS production accounts over a 7-day period. Analysis of 450,000 CloudTrail events identified 89 Lambda function creations, with 2 confirmed malicious functions used for persistence. The hypothesis was fully confirmed: an adversary with compromised service account credentials (`developer-jenkins`) established persistence via Lambda functions with administrative IAM roles and public API Gateway triggers. Both functions were detected within 3 minutes of creation, contained within 20 minutes, and fully remediated within 2 hours. No data exfiltration occurred. Overall, 95% of Lambda activity was legitimate DevOps automation, with 5% requiring investigation.
+
+### Findings
+
+| **Finding** | **Ticket** | **Description** |
+|-------------|-----------|-----------------|
+| **True Positive** | INC-2341 | Lambda function `sys-health-check-v2` with `AdministratorAccess` and public API Gateway - confirmed malicious backdoor |
+| **True Positive** | INC-2341 | Lambda function `cloudwatch-logs-processor` with recon capabilities on EventBridge schedule - confirmed malicious |
+| **Suspicious** | CLOUD-0789 | 3 additional Lambda functions with overly permissive IAM roles - legitimate but violate least privilege principle |
+| **False Positive** | N/A | 0 false positives after whitelisting known automation accounts |
+
+**True Positives:** 2 confirmed malicious Lambda persistence mechanisms
+**False Positives:** 0
+**Suspicious Events:** 3 legitimate functions with excessive IAM permissions (policy violations)
+
+### Detection Results
+
+**Finding #1: Persistence Lambda (`sys-health-check-v2`)**
+
+- **Function Name:** `sys-health-check-v2` (masquerading as legitimate health check)
+- **Runtime:** Python 3.11
+- **IAM Role:** `lambda-admin-role-temp` with `AdministratorAccess` policy
+- **Trigger:** API Gateway endpoint `https://abc123.execute-api.us-east-1.amazonaws.com/prod/health`
+- **Creator:** `arn:aws:iam::123456789012:user/developer-jenkins` (compromised service account)
+- **Source IP:** `103.253.145.22` (residential ISP - suspicious)
+- **User Agent:** `aws-cli/2.13.5 Python/3.11.4`
+- **Created:** 2025-11-28 03:15:42 UTC
+- **Code Analysis:** Base64-encoded reverse shell, environment variable exfiltration, AWS API enumeration
+- **Intent:** On-demand command execution via HTTP requests to API Gateway endpoint
+- **Action Taken:** Lambda function deleted, IAM role removed, API Gateway endpoint deleted
+
+**Finding #2: Scheduled Recon Lambda (`cloudwatch-logs-processor`)**
+
+- **Function Name:** `cloudwatch-logs-processor` (masquerading as log processing)
+- **Runtime:** Python 3.9
+- **IAM Role:** `lambda-recon-role` with `ReadOnlyAccess` + `iam:ListUsers`, `iam:ListRoles`
+- **Trigger:** EventBridge scheduled rule (every 6 hours)
+- **Creator:** Same compromised `developer-jenkins` account
+- **Source IP:** `103.253.145.22` (same attacker)
+- **Created:** 2025-11-28 03:16:55 UTC
+- **Code Analysis:** AWS API calls to enumerate IAM principals, S3 buckets, EC2 instances
+- **Intent:** Automated reconnaissance for privilege escalation opportunities
+- **Action Taken:** Lambda function deleted, IAM role removed, EventBridge rule deleted
+
+### Timeline
+
+```
+2025-11-28 03:15:42 - CreateFunction event for sys-health-check-v2 detected
+2025-11-28 03:16:18 - AttachRolePolicy event: AdministratorAccess attached to lambda-admin-role-temp
+2025-11-28 03:16:55 - CreateFunction event for cloudwatch-logs-processor detected
+2025-11-28 03:17:05 - CreateApi event: API Gateway endpoint created for sys-health-check-v2
+2025-11-28 03:17:42 - CreateRule event: EventBridge rule created for cloudwatch-logs-processor
+2025-11-28 03:18:33 - DETECTION ALERT TRIGGERED (automated CloudWatch Logs Insights query)
+2025-11-28 03:22:15 - Incident response analyst begins investigation (INC-2341 created)
+2025-11-28 03:28:45 - Lambda function code retrieved and analyzed (confirmed malicious)
+2025-11-28 03:35:47 - Containment: Lambda functions deleted, IAM roles removed, API Gateway deleted
+2025-11-28 03:45:12 - EventBridge rule deleted
+2025-11-28 04:12:22 - Compromised service account credentials rotated
+2025-11-28 05:30:00 - SCPs implemented to restrict Lambda IAM role permissions
+2025-11-28 08:00:00 - Incident response complete, forensics finalized
+```
+
+### Impact Metrics
+
+| Metric | Value |
+|--------|-------|
+| **Time to Detection** | 3 minutes from Lambda creation |
+| **Time to Investigation Start** | 6 minutes from detection |
+| **Time to Containment** | 20 minutes from detection |
+| **Total Incident Duration** | ~5 hours (creation to full remediation) |
+| **Lambda Functions Created** | 2 malicious functions |
+| **IAM Roles Created** | 2 roles with excessive permissions |
+| **API Gateway Endpoints** | 1 publicly accessible backdoor |
+| **EventBridge Rules** | 1 scheduled recon task |
+| **Compromised Accounts** | 1 service account (`developer-jenkins`) |
+| **Data Exfiltration** | 0 (contained before attacker executed functions) |
+| **Lambda Execution Cost** | $0.47 (attacker testing, minimal invocations) |
+| **Blast Radius Prevented** | Full AWS account compromise (AdministratorAccess available) |
+| **False Positives** | 0 (after automation whitelist) |
+
+### Detection Logic
+
+**Automation Opportunity:**
+
+This hunt can be fully automated with the following approach:
+
+- Real-time CloudWatch Logs Insights queries on CloudTrail logs
+- Alert on Lambda creation with administrative IAM policies AND public/scheduled triggers
+- Baseline known automation accounts and IAM role naming conventions
+- Correlate Lambda activity with IAM policy attachment and API Gateway creation
+- Lambda trigger for automated initial containment (disable function, notify SOC)
+
+**Proposed Detection:**
+
+```spl
+# Automated Lambda Persistence Detection Rule
+# Run every 5 minutes, alert on suspicious Lambda + IAM + Trigger combinations
+index=aws_cloudtrail earliest=-5m
+(eventName="CreateFunction" OR eventName="UpdateFunctionConfiguration" OR eventName="AttachRolePolicy" OR eventName="CreateApi" OR eventName="CreateRule")
+| spath input=requestParameters.role output=lambda_role
+| spath input=requestParameters.policyArn output=policy_arn
+| eval function_name=coalesce('requestParameters.functionName', 'requestParameters.FunctionName')
+| eval is_admin_policy=if(match(policy_arn, "AdministratorAccess|PowerUserAccess"), "true", "false")
+| stats earliest(_time) as first_event,
+        latest(_time) as last_event,
+        values(eventName) as events,
+        dc(eventName) as unique_events,
+        values(function_name) as functions,
+        values(lambda_role) as roles,
+        values(policy_arn) as policies,
+        values(sourceIPAddress) as source_ips,
+        max(is_admin_policy) as has_admin_policy
+        by userIdentity.principalId, userIdentity.arn
+| where unique_events >= 2 AND has_admin_policy="true"
+| lookup known_automation_accounts.csv userIdentity.principalId OUTPUT is_known
+| where isnull(is_known) OR is_known="false"
+| eval duration_seconds=last_event-first_event
+| where duration_seconds < 1800
+| eval severity=case(
+    has_admin_policy="true" AND match(events, "CreateApi|CreateRule"), "critical",
+    has_admin_policy="true", "high",
+    1==1, "medium"
+)
+| eval description="Suspicious Lambda persistence detected: ".userIdentity.arn." created function(s) ".functions." with admin IAM role and trigger"
+| table first_event, severity, userIdentity.arn, functions, roles, policies, events, source_ips, description
+```
+
+**Detection Deployment:**
+
+- Scheduled as CloudWatch Logs Insights query with SNS notification
+- Integrated with Lambda function for automated initial containment:
+  - Disable suspicious Lambda function
+  - Remove IAM role policies
+  - Create incident ticket in JIRA
+  - Notify SOC via Slack
+  - Snapshot function code for forensics
+
+### Lessons Learned
+
+**What Worked Well:**
+
+- CloudTrail comprehensive logging provided full attack visibility
+- Multi-event correlation (Lambda + IAM + triggers) identified complete attack chain
+- Temporal analysis (rapid succession within 1 hour) strong indicator of malicious activity
+- Known automation whitelist eliminated 87% of false positives
+- Real-time alerting enabled detection within 3 minutes of Lambda creation
+- Automated incident response reduced containment time from hours to minutes
+
+**What Could Be Improved:**
+
+- Baseline of "normal" Lambda function naming conventions needed
+- Lambda function code analysis not automated (requires manual download and inspection)
+- Service account credential security needs strengthening (MFA, short-lived credentials)
+- IAM policy least privilege principle not enforced (too many admin roles for Lambda)
+- Threat intel integration for malicious IPs/user agents would improve context
+
+**Telemetry Gaps Identified:**
+
+- No automated Lambda function code scanning for IOCs (base64, reverse shells, exfiltration)
+- Lambda execution logs not centralized (only creation events monitored)
+- API Gateway access logs not enabled (no visibility into function invocations)
+- EventBridge rule invocations not logged (can't confirm if recon Lambda executed)
+- IAM Access Analyzer alerts not integrated with SIEM (delayed visibility)
+
+### Remediation Actions
+
+1. **Immediate Containment (Completed):**
+   - [x] Deleted malicious Lambda functions (`sys-health-check-v2`, `cloudwatch-logs-processor`)
+   - [x] Removed IAM roles (`lambda-admin-role-temp`, `lambda-recon-role`)
+   - [x] Deleted API Gateway endpoint (public backdoor)
+   - [x] Removed EventBridge scheduled rule
+   - [x] Rotated compromised service account credentials (`developer-jenkins`)
+   - [x] Reviewed all Lambda functions created by compromised account (no additional malicious functions)
+
+2. **Preventive Controls (In Progress):**
+   - [x] Implemented Service Control Policies (SCPs) to restrict Lambda IAM role permissions (max `PowerUserAccess`)
+   - [x] Enabled IAM Access Analyzer for continuous policy analysis
+   - [ ] Enforce MFA for all service accounts (target: 2025-12-05) - CLOUD-0790
+   - [ ] Implement short-lived credentials for CI/CD pipelines (target: 2025-12-15) - CLOUD-0791
+   - [ ] Deploy Lambda function code scanning for IOCs (target: 2026-01-10) - DET-0058
+
+3. **Detective Controls (In Progress):**
+   - [x] Deployed automated Lambda persistence detection rule (5-minute frequency)
+   - [x] Enabled API Gateway access logging for all endpoints
+   - [ ] Centralize Lambda execution logs in SIEM (target: 2025-12-10) - CLOUD-0792
+   - [ ] Integrate IAM Access Analyzer findings with SOAR (target: 2025-12-20) - DET-0059
+   - [ ] Create dashboard for Lambda creation activity and IAM role analysis
+
+4. **Playbook Development:**
+   - [x] Documented Lambda persistence detection and response playbook for SOC
+   - [x] Created runbook for Lambda function code analysis
+   - [x] Updated incident response procedures for cloud persistence
+
+### Follow-up Actions
+
+- [x] Escalate INC-2341 to incident response (completed, forensics finalized)
+- [x] Create baseline inventory of legitimate Lambda functions and automation accounts (target: 2025-12-01)
+- [ ] Conduct security review of all existing Lambda functions for excessive IAM permissions (target: 2025-12-15) - CLOUD-0793
+- [ ] Implement Lambda function code signing to prevent unauthorized modifications (target: 2026-01-30) - CLOUD-0794
+- [ ] Deploy AWS GuardDuty for additional serverless threat detection (target: 2026-02-15) - CLOUD-0795
+- [ ] Schedule recurring hunt execution (monthly) for Lambda persistence patterns
+- [ ] Threat intel sharing: Submit IOCs to MISP (IP, function names, code hashes)
+
+### Follow-up Hunts
+
+- H-0012: AWS Step Functions Persistence Detection (T1546.004)
+- H-0013: EventBridge Rule Abuse for Persistence
+- H-0014: IAM Role Privilege Escalation Patterns
+- H-0015: S3 Bucket Event-Driven Lambda Backdoors
+- H-0016: Cross-Account Lambda Invocation Analysis
+
+---
+
+**Hunt Completed:** 2025-11-28