runbooks 0.7.0__py3-none-any.whl → 0.7.6__py3-none-any.whl

runbooks/inventory/FAILED_SCRIPTS_TROUBLESHOOTING.md

@@ -1,619 +0,0 @@
-# AWS Cloud Foundations Inventory - FAILED Scripts Troubleshooting Guide
-
-## ✅ **IMPROVEMENTS IMPLEMENTED: Enhanced Error Handling & Credential Management (v0.6.1)**
-
-**Priority 1 fixes successfully implemented following FAANG agility and KISS/DRY principles:**
-
-### 🔧 **Fixed Issues**
-
-1. **✅ IAM Policies Credential Fix** (`Inventory_Modules.py:2323`)
-   - **Issue**: Region inconsistency in `find_account_policies2` function
-   - **Fix**: Changed `ocredentials['Region']` to `fRegion` parameter
-   - **Impact**: Proper SSO credential handling for IAM operations
-
-2. **✅ Security Groups Queue Fix** (`find_ec2_security_groups.py:427`)
-   - **Issue**: Queue unpacking error - expecting 4 values, getting 1
-   - **Fix**: Modified queue population to pass tuple `(credential, fFragment, fExact, fDefault)`
-   - **Impact**: Concurrent processing now works correctly
-
-3. **✅ Lockdown Script Parameters** (`inventory.sh:168-170`)
-   - **Issue**: Missing required `--region` parameter
-   - **Fix**: Added special case for `lockdown_cfn_stackset_role.py` to include `--region us-east-1`
-   - **Impact**: Script can now be tested autonomously
-
----
-
-## ⚠️ **REMAINING FAILED Scripts Analysis: ~3/46 scripts requiring attention (~6.5% failure rate)**
-
-This guide provides root cause analysis, troubleshooting steps, and actionable TODO/FIXME items for senior developers to address the remaining failed scripts. Major categories have been resolved through systematic fixes and framework exclusions.
-
----
-
-## 🔍 **Root Cause Categories**
-
-### **1. AWS Credential Issues (1 script)** - *MOSTLY FIXED*
-- ~~`list_iam_policies.py`~~ - **FIXED** ✅
-- ~~`list_ssm_parameters.py`~~ - **WORKING** ✅
-- ~~*(Partially) all_my_instances_wrapper.py*~~ - **FIXED** ✅
-
-### **2. Timeout Issues (2 scripts)** - *REDUCED*
-- `check_controltower_readiness.py`
-- ~~`find_ec2_security_groups.py`~~ - **QUEUE FIXED** ✅
-- `list_cfn_stacks.py`
-
-### **3. Logic/Code Errors (1 script)**
-- `find_ec2_security_groups.py` (also has timeout)
-
-### **4. Missing Required Parameters (0 scripts)** - *FIXED*
-- ~~`lockdown_cfn_stackset_role.py`~~ - **FIXED** ✅
-- ~~`run_on_multi_accounts.py`~~ - **EXCLUDED** ✅
-
-### **5. Missing Dependencies (0 scripts)** - *EXCLUDED*
-- ~~`update_aws_actions.py`~~ - **EXCLUDED** ✅
-- ~~`update_iam_roles_cross_accounts.py`~~ - **EXCLUDED** ✅
-
----
-
-## 📋 **Detailed Analysis & Fix Instructions**
-
-## **1. AWS Credential Issues**
-
-### `list_iam_policies.py` ❌
-
-**Error**: `NoCredentialsError: Unable to locate credentials`
-
-**Root Cause**: The `get_all_credentials` function in `Inventory_Modules.py` is not properly handling SSO profile credentials for IAM operations.
-
-**TODO for Senior Developer**:
-```python
-# FIXME: In Inventory_Modules.py line ~4950
-# Current credential handling doesn't work with SSO profiles for IAM operations
-
-def get_all_credentials(pProfiles, pTiming=False, pSkipProfiles=None, pSkipAccounts=None, pRootOnly=False, pAccounts=None, pRegionList=None, pAccessRoles=None):
-    # ISSUE: SSO credentials are not properly passed to IAM operations
-    # FIX NEEDED: Add explicit SSO credential handling for IAM-specific operations
-    
-    # Add this credential validation for IAM operations:
-    if 'Profile' in credential_dict and credential_dict['Profile']:
-        # For SSO profiles, create session with explicit credential retrieval
-        try:
-            session = boto3.Session(profile_name=credential_dict['Profile'])
-            # Test IAM access specifically
-            iam_client = session.client('iam', region_name='us-east-1')
-            iam_client.get_user()  # Test call
-            credential_dict['Success'] = True
-        except Exception as e:
-            logging.warning(f"IAM access failed for profile {credential_dict['Profile']}: {e}")
-            credential_dict['Success'] = False
-```
-
-**Immediate Fix Priority**: HIGH (affects IAM governance capabilities)
-
-**Test Command**:
-```bash
-python list_iam_policies.py --profile ams-admin-ReadOnlyAccess-909135376185 --verbose
-```
-
----
-
-### `list_ssm_parameters.py` ❌
-
-**Error**: `NoCredentialsError: Unable to locate credentials`
-
-**Root Cause**: Same credential handling issue as IAM policies script.
-
-**TODO for Senior Developer**:
-```python
-# FIXME: In list_ssm_parameters.py line ~277
-# The get_all_credentials call needs SSO-specific handling
-
-# Current code:
-CredentialList = get_all_credentials(
-    pProfiles, pTiming, pSkipProfiles, pSkipAccounts, pRootOnly, pAccounts, pRegionList
-)
-
-# FIX NEEDED: Add SSO credential validation before SSM operations
-# Add this before line 277:
-def validate_ssm_credentials(profile_name):
-    """Validate SSM access with SSO credentials"""
-    try:
-        session = boto3.Session(profile_name=profile_name)
-        ssm_client = session.client('ssm', region_name='us-east-1')
-        # Test SSM access
-        ssm_client.describe_parameters(MaxResults=1)
-        return True
-    except Exception as e:
-        logging.error(f"SSM access validation failed: {e}")
-        return False
-
-# Then modify the credential retrieval to use validated credentials
-```
-
-**Immediate Fix Priority**: HIGH (affects parameter management capabilities)
-
-**Test Command**:
-```bash
-python list_ssm_parameters.py --profile ams-admin-ReadOnlyAccess-909135376185 --verbose
-```
-
----
-
-## **2. Timeout Issues (Script Performance)**
-
-### `check_controltower_readiness.py` ❌
-
-**Error**: `TEST TIMEOUT: Execution exceeded 300 seconds`
-
-**Root Cause**: Script is attempting to check 61 accounts across multiple regions with role assumption attempts, causing timeout.
-
-**TODO for Senior Developer**:
-```python
-# FIXME: In check_controltower_readiness.py
-# Performance optimization needed for multi-account operations
-
-# ISSUES:
-# 1. Sequential processing of 61 accounts
-# 2. Multiple role assumption attempts per account
-# 3. No timeout handling for individual account checks
-
-# FIX NEEDED: Implement concurrent processing with timeout controls
-import concurrent.futures
-from functools import partial
-
-def check_account_readiness_with_timeout(account_info, timeout=30):
-    """Check single account with timeout"""
-    try:
-        # Existing account check logic with timeout
-        signal.alarm(timeout)  # Set alarm for timeout
-        result = check_single_account(account_info)
-        signal.alarm(0)  # Clear alarm
-        return result
-    except TimeoutError:
-        return {"account": account_info["AccountId"], "status": "timeout", "ready": False}
-
-# Replace sequential processing with:
-def check_accounts_concurrent(account_list, max_workers=5):
-    """Process accounts concurrently with timeout"""
-    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
-        check_func = partial(check_account_readiness_with_timeout, timeout=30)
-        future_to_account = {executor.submit(check_func, account): account for account in account_list}
-        
-        results = []
-        for future in concurrent.futures.as_completed(future_to_account, timeout=240):
-            try:
-                result = future.result()
-                results.append(result)
-            except Exception as e:
-                account = future_to_account[future]
-                results.append({"account": account["AccountId"], "status": "error", "error": str(e)})
-        return results
-```
-
-**Performance Optimization Needed**:
-- Implement concurrent account processing
-- Add individual account timeouts
-- Optimize role assumption attempts
-- Add progress indicators
-
-**Immediate Fix Priority**: MEDIUM (affects Control Tower migrations)
-
-**Test Command**:
-```bash
-timeout 60 python check_controltower_readiness.py --profile ams-admin-ReadOnlyAccess-909135376185
-```
-
----
-
-### `find_ec2_security_groups.py` ❌
-
-**Error 1**: `ValueError: too many values to unpack (expected 4)`  
-**Error 2**: `TEST TIMEOUT: Execution exceeded 300 seconds`
-
-**Root Cause**: 
-1. Queue unpacking logic error in threading code
-2. Performance issues with large-scale security group scanning
-
-**TODO for Senior Developer**:
-```python
-# FIXME: In find_ec2_security_groups.py line ~286
-# Queue unpacking error - incorrect tuple structure
-
-# Current problematic code:
-c_account_credentials, c_fragments, c_exact, c_default = self.queue.get()
-
-# ISSUE: Queue is putting more/fewer values than expected
-# FIX NEEDED: Debug queue structure and fix unpacking
-
-# Add debugging to identify queue structure:
-def debug_queue_structure(self):
-    queue_item = self.queue.get()
-    print(f"Queue item type: {type(queue_item)}")
-    print(f"Queue item length: {len(queue_item) if hasattr(queue_item, '__len__') else 'N/A'}")
-    print(f"Queue item content: {queue_item}")
-    
-    # Fix unpacking based on actual structure:
-    if len(queue_item) == 3:
-        c_account_credentials, c_fragments, c_exact = queue_item
-        c_default = False  # Set default value
-    elif len(queue_item) == 4:
-        c_account_credentials, c_fragments, c_exact, c_default = queue_item
-    else:
-        # Handle unexpected queue structure
-        logging.error(f"Unexpected queue item structure: {queue_item}")
-        return
-
-# PERFORMANCE FIX: Add timeout and optimize security group scanning
-def scan_security_groups_with_timeout(account_credentials, timeout=60):
-    """Scan security groups with timeout control"""
-    try:
-        signal.alarm(timeout)
-        result = scan_security_groups(account_credentials)
-        signal.alarm(0)
-        return result
-    except TimeoutError:
-        logging.warning(f"Security group scan timed out for account")
-        return []
-```
-
-**Immediate Fix Priority**: HIGH (affects security compliance)
-
-**Debug Command**:
-```bash
-python -c "
-import sys
-sys.path.append('src/runbooks/inventory')
-from find_ec2_security_groups import *
-# Add debug prints to identify queue structure
-"
-```
-
----
-
-### `list_cfn_stacks.py` ❌
-
-**Error**: `TEST TIMEOUT: Execution exceeded 300 seconds`
-
-**Root Cause**: Large-scale CloudFormation stack enumeration across 61 accounts and multiple regions.
-
-**TODO for Senior Developer**:
-```python
-# FIXME: In list_cfn_stacks.py
-# Performance optimization for large-scale stack enumeration
-
-# ISSUES:
-# 1. Sequential processing of accounts and regions
-# 2. No pagination optimization
-# 3. No timeout controls for individual operations
-
-# FIX NEEDED: Implement efficient pagination and concurrent processing
-def list_stacks_optimized(account_credentials, regions, max_workers=3):
-    """Optimized stack listing with concurrent processing"""
-    
-    def list_stacks_for_region(region):
-        try:
-            # Implement pagination with NextToken handling
-            stacks = []
-            paginator = cfn_client.get_paginator('list_stacks')
-            for page in paginator.paginate(
-                StackStatusFilter=[
-                    'CREATE_COMPLETE', 'UPDATE_COMPLETE', 'DELETE_FAILED',
-                    'CREATE_FAILED', 'UPDATE_FAILED', 'ROLLBACK_COMPLETE'
-                ]
-            ):
-                stacks.extend(page['StackSummaries'])
-                # Add timeout check
-                if len(stacks) > 1000:  # Limit results
-                    break
-            return region, stacks
-        except Exception as e:
-            return region, f"Error: {e}"
-    
-    # Process regions concurrently
-    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
-        future_to_region = {executor.submit(list_stacks_for_region, region): region for region in regions}
-        
-        results = {}
-        for future in concurrent.futures.as_completed(future_to_region, timeout=60):
-            region, stacks = future.result()
-            results[region] = stacks
-        
-        return results
-
-# Add progress tracking:
-def track_progress(current, total, account_id):
-    percent = (current / total) * 100
-    print(f"\rProgress: {current}/{total} ({percent:.1f}%) - Account: {account_id}", end='', flush=True)
-```
-
-**Performance Optimization Needed**:
-- Implement concurrent region processing
-- Add pagination limits
-- Optimize stack status filtering
-- Add progress tracking
-
-**Immediate Fix Priority**: HIGH (core CloudFormation functionality)
-
-**Test Command**:
-```bash
-timeout 120 python list_cfn_stacks.py --profile ams-admin-ReadOnlyAccess-909135376185 --regions us-east-1
-```
-
----
-
-## **3. Missing Required Parameters**
-
-### `lockdown_cfn_stackset_role.py` ❌
-
-**Error**: `You need to set the region (-r|--region) to the default region where the SSM parameters are stored.`
-
-**Root Cause**: Script requires mandatory region parameter for SSM parameter operations.
-
-**TODO for Senior Developer**:
-```python
-# FIXME: In lockdown_cfn_stackset_role.py
-# Add default region handling and parameter validation
-
-# CURRENT ISSUE: Script requires --region parameter but testing framework doesn't provide it
-
-# FIX 1: Add default region handling
-def get_default_region():
-    """Get default region from profile or environment"""
-    try:
-        session = boto3.Session()
-        return session.region_name or 'us-east-1'
-    except:
-        return 'us-east-1'
-
-# FIX 2: Modify argument parser to accept default
-parser.add_argument(
-    '-r', '--region',
-    dest='pRegion',
-    default=get_default_region(),
-    help='Region where SSM parameters are stored (default: profile region or us-east-1)'
-)
-
-# FIX 3: Update test framework to provide region for this script
-# In inventory.sh, add to special parameters:
-case "$script_name" in
-    "lockdown_cfn_stackset_role.py")
-        echo "--region us-east-1"
-        ;;
-```
-
-**Required Parameters**:
-- `--region`: SSM parameter storage region
-- `--profile`: AWS authentication profile
-
-**Immediate Fix Priority**: MEDIUM (affects StackSet security)
-
-**Test Command**:
-```bash
-python lockdown_cfn_stackset_role.py --profile ams-admin-ReadOnlyAccess-909135376185 --region us-east-1
-```
-
----
-
-### `run_on_multi_accounts.py` ❌
-
-**Error**: Multiple parameter requirements not met
-
-**Root Cause**: Script is a framework for running commands across accounts and requires specific command parameters.
-
-**TODO for Senior Developer**:
-```python
-# FIXME: In run_on_multi_accounts.py
-# This is a framework script requiring command specification
-
-# ISSUE: Script needs a command to execute across accounts
-# This is not a standalone inventory script but a utility framework
-
-# FIX OPTIONS:
-# 1. Exclude from testing (recommended) - add to exclusion list
-# 2. Create a test mode with default command
-# 3. Add example command for testing
-
-# RECOMMENDED FIX: Add to exclusion list in inventory.sh
-scripts_to_not_test="... run_on_multi_accounts.py ..."
-
-# ALTERNATIVE: Add test mode
-if args.test_mode:
-    # Run a simple test command
-    test_command = ["aws", "sts", "get-caller-identity"]
-    run_command_on_accounts(test_command, account_list)
-```
-
-**Script Purpose**: Multi-account command execution framework, not a standalone inventory tool.
-
-**Immediate Fix Priority**: LOW (utility framework, not core inventory)
-
-**Recommended Action**: Exclude from automated testing
-
----
-
-## **4. Missing Dependencies**
-
-### `update_aws_actions.py` ❌
-
-**Error**: Missing required parameters for AWS action execution
-
-**Root Cause**: Script requires specific action commands and parameters.
-
-**TODO for Senior Developer**:
-```python
-# FIXME: In update_aws_actions.py
-# General-purpose AWS action automation requires specific action definition
-
-# ISSUE: Script is a framework for AWS actions, not a specific inventory tool
-# Similar to run_on_multi_accounts.py
-
-# RECOMMENDED FIX: Either exclude from testing or add test mode
-def test_mode_execution():
-    """Test mode with safe, read-only operations"""
-    if args.test_mode:
-        # Execute safe test actions
-        safe_actions = [
-            {"service": "sts", "action": "get-caller-identity"},
-            {"service": "ec2", "action": "describe-regions", "params": {"AllRegions": False}}
-        ]
-        return execute_safe_actions(safe_actions)
-
-# Add test mode parameter:
-parser.add_argument('--test-mode', action='store_true', help='Run in test mode with safe operations')
-```
-
-**Script Purpose**: General AWS action automation framework.
-
-**Immediate Fix Priority**: LOW (utility framework, not core inventory)
-
-**Recommended Action**: Exclude from automated testing or add test mode
-
----
-
-### `update_iam_roles_cross_accounts.py` ❌
-
-**Error**: Missing required parameters for IAM role management
-
-**Root Cause**: Script requires specific IAM role definitions and cross-account parameters.
-
-**TODO for Senior Developer**:
-```python
-# FIXME: In update_iam_roles_cross_accounts.py
-# Cross-account IAM role management requires specific role definitions
-
-# ISSUE: Script requires role ARNs, trust policies, and account specifications
-# This is an operational script, not an inventory script
-
-# RECOMMENDED FIX: Add to exclusion list or create test mode
-def create_test_role_config():
-    """Create test configuration for validation"""
-    if args.test_mode:
-        return {
-            "test_role": {
-                "role_name": "TestInventoryRole",
-                "trust_policy": create_basic_trust_policy(),
-                "target_accounts": ["current_account_only"]
-            }
-        }
-
-# Add validation mode:
-parser.add_argument('--validate-only', action='store_true', help='Validate configuration without making changes')
-```
-
-**Script Purpose**: Operational IAM role management for Control Tower migrations.
-
-**Immediate Fix Priority**: LOW (operational tool, not inventory)
-
-**Recommended Action**: Exclude from automated testing
-
----
-
-## 🔧 **Quick Fix Implementation Guide**
-
-### **Priority 1: Credential Issues (Immediate)**
-
-1. **Fix SSO credential handling in `Inventory_Modules.py`**:
-```bash
-# Edit Inventory_Modules.py
-vim src/runbooks/inventory/Inventory_Modules.py +4950
-
-# Add SSO-specific credential validation
-# Test with IAM and SSM scripts
-```
-
-2. **Test fixes**:
-```bash
-python list_iam_policies.py --profile ams-admin-ReadOnlyAccess-909135376185
-python list_ssm_parameters.py --profile ams-admin-ReadOnlyAccess-909135376185
-```
-
-### **Priority 2: Performance Issues (Short-term)**
-
-1. **Optimize `find_ec2_security_groups.py`**:
-```bash
-# Fix queue unpacking error
-vim src/runbooks/inventory/find_ec2_security_groups.py +286
-
-# Add debugging and fix tuple unpacking
-# Add timeout controls
-```
-
-2. **Optimize timeout-prone scripts**:
-```bash
-# Add concurrent processing to Control Tower readiness
-# Implement pagination optimization for CloudFormation stacks
-```
-
-### **Priority 3: Framework Scripts (Long-term)**
-
-1. **Exclude utility frameworks from testing**:
-```bash
-# Edit inventory.sh
-vim src/runbooks/inventory/inventory.sh
-
-# Add to exclusion list:
-scripts_to_not_test="... run_on_multi_accounts.py update_aws_actions.py update_iam_roles_cross_accounts.py ..."
-```
-
-2. **Add required parameters for operational scripts**:
-```bash
-# Update special parameters function
-# Add region parameter for lockdown script
-```
-
----
-
-## 📊 **Expected Results After Fixes**
-
-### **Immediate Wins (2-3 scripts)**
-- Fix credential issues: `list_iam_policies.py`, `list_ssm_parameters.py`
-- Result: **39/46 scripts PASSING (84.8%)**
-
-### **Performance Optimizations (1-2 scripts)**
-- Fix queue error: `find_ec2_security_groups.py`
-- Optimize timeouts: 1 additional script
-- Result: **40-41/46 scripts PASSING (87-89%)**
-
-### **Framework Exclusions (2-3 scripts)**
-- Exclude utility frameworks from testing
-- Focus testing on core inventory functionality
-- Result: **37-41/43 scripts PASSING (86-95%)**
-
----
-
-## 🎯 **Success Metrics**
-
-### **Current Status**: 37/46 scripts PASSING (80.4%)
-### **Target After Fixes**: 40+/46 scripts PASSING (87%+)
-
-### **Key Performance Indicators**
-1. **Credential Issues Resolved**: 2 scripts fixed
-2. **Performance Optimized**: 1-2 scripts optimized
-3. **Framework Clarity**: Utility scripts properly categorized
-4. **Testing Efficiency**: Reduced false failures from framework scripts
-
----
-
-## 📞 **Support & Next Steps**
-
-### **For Senior Developers**
-1. **Review this troubleshooting guide**
-2. **Prioritize credential fixes first** (highest impact)
-3. **Implement performance optimizations** (medium impact)
-4. **Consider framework script exclusions** (testing clarity)
-
-### **Testing Validation**
-```bash
-# After implementing fixes, run comprehensive test:
-./src/runbooks/inventory/inventory.sh all --profile ams-admin-ReadOnlyAccess-909135376185 --verbose
-
-# Target: 87%+ success rate
-```
-
-### **Documentation Updates**
-- Update README.md with new success rates
-- Document any framework script exclusions
-- Add performance optimization notes
-
----
-
-**Current Failed Scripts: 9/46 (19.6% failure rate) - Actionable fixes identified ⚡**