aws-cis-controls-assessment 1.0.8__py3-none-any.whl → 1.0.10__py3-none-any.whl

docs/scoring-comparison-aws-config.md

@@ -0,0 +1,379 @@
+# Scoring Comparison: Our Approach vs AWS Config Conformance Packs
+
+## Overview
+
+This document compares our weighted scoring methodology with AWS Config's Conformance Pack approach.
+
+## AWS Config Conformance Pack Approach
+
+### Formula
+```
+Compliance Score = Compliant Rule-Resources / Total Rule-Resources
+```
+
+### Characteristics
+- **Simple percentage** - No weighting applied
+- **Flat structure** - All rules treated equally
+- **Resource-centric** - Counts individual rule-resource combinations
+- **No prioritization** - Critical and minor rules have equal impact
+
+### Example Calculation
+```
+Rule 1: 90/100 resources compliant
+Rule 2: 50/50 resources compliant
+Rule 3: 10/50 resources compliant
+
+Total: (90 + 50 + 10) / (100 + 50 + 50)
+     = 150 / 200
+     = 75% compliance
+```
+
+## Our Weighted Approach
+
+### Formula
+```
+Overall Score = Σ(IG Score × IG Weight) / Σ(IG Weights)
+  where IG Score = Σ(Control Score × Control Weight) / Σ(Control Weights)
+  where Control Score = Compliant Resources / Total Resources
+```
+
+### Characteristics
+- **Weighted average** - Critical controls have more impact
+- **Hierarchical structure** - Controls → IGs → Overall
+- **Security-centric** - Prioritizes critical security controls
+- **Maturity-aware** - Advanced IGs (IG2/IG3) weighted higher
+
+### Example Calculation
+```
+Control 1 (weight 1.0): 90/100 = 90%
+Control 2 (weight 1.5): 50/50 = 100%
+Control 3 (weight 1.0): 10/50 = 20%
+
+Weighted: (90×1.0 + 100×1.5 + 20×1.0) / (1.0 + 1.5 + 1.0)
+        = (90 + 150 + 20) / 3.5
+        = 260 / 3.5
+        = 74.3% compliance
+```
+
+## Side-by-Side Comparison
+
+| Aspect | AWS Config Conformance Pack | Our Weighted Approach |
+|--------|----------------------------|----------------------|
+| **Formula** | Simple average | Weighted average |
+| **Structure** | Flat (all rules equal) | Hierarchical (Controls → IGs → Overall) |
+| **Weighting** | None | Control weights + IG weights |
+| **Prioritization** | No | Yes (critical controls weighted higher) |
+| **Maturity Levels** | Not considered | IG1/IG2/IG3 weighted differently |
+| **Complexity** | Low | Medium |
+| **Customization** | Limited | Highly customizable |
+| **Focus** | Resource compliance | Security posture |
+
+## Real-World Impact Comparison
+
+### Scenario 1: Critical Control Failure
+
+**Setup:**
+- 3 controls assessed
+- Control 1 (Asset Inventory, weight 1.0): 90/100 = 90%
+- Control 2 (Encryption at Rest, weight 1.4): 10/100 = 10% ⚠️ CRITICAL
+- Control 3 (Logging, weight 1.2): 80/100 = 80%
+
+**AWS Config Approach:**
+```
+Score = (90 + 10 + 80) / (100 + 100 + 100)
+      = 180 / 300
+      = 60% compliance
+```
+
+**Our Weighted Approach:**
+```
+Score = (90×1.0 + 10×1.4 + 80×1.2) / (1.0 + 1.4 + 1.2)
+      = (90 + 14 + 96) / 3.6
+      = 200 / 3.6
+      = 55.6% compliance
+```
+
+**Analysis:**
+- Our approach scores **4.4% lower** because encryption (critical) is weighted higher
+- This better reflects the **security risk** of poor encryption compliance
+- AWS Config treats encryption failure same as asset inventory issues
+
+### Scenario 2: Minor Control Failure
+
+**Setup:**
+- 3 controls assessed
+- Control 1 (Asset Inventory, weight 1.0): 10/100 = 10% ⚠️ MINOR
+- Control 2 (Encryption at Rest, weight 1.4): 90/100 = 90%
+- Control 3 (Logging, weight 1.2): 80/100 = 80%
+
+**AWS Config Approach:**
+```
+Score = (10 + 90 + 80) / (100 + 100 + 100)
+      = 180 / 300
+      = 60% compliance
+```
+
+**Our Weighted Approach:**
+```
+Score = (10×1.0 + 90×1.4 + 80×1.2) / (1.0 + 1.4 + 1.2)
+      = (10 + 126 + 96) / 3.6
+      = 232 / 3.6
+      = 64.4% compliance
+```
+
+**Analysis:**
+- Our approach scores **4.4% higher** because critical controls (encryption) are compliant
+- This better reflects the **actual security posture** despite asset inventory issues
+- AWS Config penalizes equally regardless of control importance
+
+### Scenario 3: Multiple Implementation Groups
+
+**Setup:**
+- IG1: 85% compliance (74 controls)
+- IG2: 75% compliance (58 additional controls)
+- IG3: 60% compliance (13 additional controls)
+
+**AWS Config Approach:**
+```
+All rules treated equally:
+Score = (85 + 75 + 60) / 3
+      = 73.3% compliance
+```
+
+**Our Weighted Approach:**
+```
+Score = (85×1.0 + 75×1.5 + 60×2.0) / (1.0 + 1.5 + 2.0)
+      = (85 + 112.5 + 120) / 4.5
+      = 317.5 / 4.5
+      = 70.6% compliance
+```
+
+**Analysis:**
+- Our approach scores **2.7% lower** because IG3 (advanced security) is weighted higher
+- This reflects that **advanced security failures** are more concerning
+- AWS Config doesn't distinguish between basic and advanced security
+
+## Key Differences Explained
+
+### 1. Security Prioritization
+
+**AWS Config:**
+- Treats all rules equally
+- 100 non-compliant S3 buckets = 100 non-compliant IAM users
+- No distinction between critical and minor issues
+
+**Our Approach:**
+- Critical controls (encryption, access control) weighted higher
+- 100 non-encrypted databases > 100 untagged EC2 instances
+- Reflects actual security risk
+
+### 2. Maturity Recognition
+
+**AWS Config:**
+- No concept of security maturity levels
+- Basic and advanced controls treated the same
+
+**Our Approach:**
+- IG1 (Essential) = baseline weight
+- IG2 (Enhanced) = 1.5x weight
+- IG3 (Advanced) = 2x weight
+- Encourages progression to higher security maturity
+
+### 3. Resource Distribution Impact
+
+**AWS Config:**
+- Heavily influenced by resource count
+- 1 rule with 1000 resources dominates score
+- Can mask issues in rules with fewer resources
+
+**Our Approach:**
+- Each control scored independently first
+- Then weighted and averaged
+- Prevents resource count from dominating
+- Better reflects control-level compliance
+
+### 4. Actionable Insights
+
+**AWS Config:**
+- Simple percentage
+- Doesn't indicate which areas need focus
+- All non-compliance treated equally
+
+**Our Approach:**
+- Identifies high-priority remediation areas
+- Weights guide where to focus effort
+- Risk areas highlighted based on criticality
+
+## Practical Examples
+
+### Example 1: Encryption Compliance
+
+**Scenario:** Organization has poor encryption but good asset management
+
+| Control | Resources | Compliant | AWS Config Impact | Our Impact |
+|---------|-----------|-----------|-------------------|------------|
+| Asset Inventory (1.0) | 1000 | 950 (95%) | 950/1000 | 95% × 1.0 |
+| Encryption at Rest (1.4) | 100 | 20 (20%) | 20/100 | 20% × 1.4 |
+
+**AWS Config Score:**
+```
+(950 + 20) / (1000 + 100) = 970/1100 = 88.2%
+```
+
+**Our Score:**
+```
+(95×1.0 + 20×1.4) / (1.0 + 1.4) = (95 + 28) / 2.4 = 51.3%
+```
+
+**Difference:** -36.9%
+
+**Why?** Our approach correctly identifies this as a **critical security issue** despite high resource compliance in less critical areas.
+
+### Example 2: Balanced Compliance
+
+**Scenario:** Organization has consistent compliance across all controls
+
+| Control | Resources | Compliant | Compliance % |
+|---------|-----------|-----------|--------------|
+| Control 1 (1.0) | 100 | 80 | 80% |
+| Control 2 (1.5) | 100 | 80 | 80% |
+| Control 3 (1.2) | 100 | 80 | 80% |
+
+**AWS Config Score:**
+```
+(80 + 80 + 80) / (100 + 100 + 100) = 240/300 = 80%
+```
+
+**Our Score:**
+```
+(80×1.0 + 80×1.5 + 80×1.2) / (1.0 + 1.5 + 1.2) = (80 + 120 + 96) / 3.7 = 80%
+```
+
+**Difference:** 0%
+
+**Why?** When compliance is **consistent across controls**, both approaches yield the same result.
+
+### Example 3: Resource Count Skew
+
+**Scenario:** One rule has many resources, others have few
+
+| Control | Resources | Compliant | Compliance % |
+|---------|-----------|-----------|--------------|
+| Control 1 (1.0) | 1000 | 900 | 90% |
+| Control 2 (1.5) | 10 | 2 | 20% |
+| Control 3 (1.2) | 10 | 2 | 20% |
+
+**AWS Config Score:**
+```
+(900 + 2 + 2) / (1000 + 10 + 10) = 904/1020 = 88.6%
+```
+
+**Our Score:**
+```
+(90×1.0 + 20×1.5 + 20×1.2) / (1.0 + 1.5 + 1.2) = (90 + 30 + 24) / 3.7 = 38.9%
+```
+
+**Difference:** -49.7%
+
+**Why?** AWS Config is **dominated by the high resource count** in Control 1. Our approach treats each control equally, revealing the **poor compliance in critical areas**.
+
+## When Each Approach is Better
+
+### AWS Config Approach is Better When:
+
+1. **Simplicity is paramount** - Easy to understand and explain
+2. **All rules are equally important** - No need for prioritization
+3. **Resource-level tracking** - Focus on individual resource compliance
+4. **Regulatory compliance** - Simple pass/fail requirements
+5. **Audit purposes** - Straightforward percentage for auditors
+
+### Our Weighted Approach is Better When:
+
+1. **Security prioritization matters** - Critical controls should have more impact
+2. **Risk-based decision making** - Focus on highest-risk areas
+3. **Maturity progression** - Encouraging advancement through IG levels
+4. **Executive reporting** - Reflects actual security posture
+5. **Remediation planning** - Guides where to focus effort
+6. **Resource optimization** - Prevents resource count from dominating
+
+## Conversion Between Approaches
+
+### Converting Our Score to AWS Config Style
+
+To get an "unweighted" score similar to AWS Config:
+
+```python
+# Sum all compliant resources across all controls
+total_compliant = sum(control.compliant_resources for control in controls)
+
+# Sum all total resources across all controls
+total_resources = sum(control.total_resources for control in controls)
+
+# Calculate simple percentage
+aws_config_style_score = (total_compliant / total_resources) * 100
+```
+
+### Converting AWS Config to Our Style
+
+To add weighting to AWS Config scores:
+
+```python
+# Apply control weights to each rule's compliance
+weighted_scores = []
+for rule in rules:
+    rule_compliance = rule.compliant / rule.total
+    weight = get_control_weight(rule.control_id)
+    weighted_scores.append(rule_compliance * weight)
+
+# Calculate weighted average
+our_style_score = sum(weighted_scores) / sum(weights)
+```
+
+## Recommendations
+
+### Use AWS Config Approach If:
+- You need simple, auditable compliance reporting
+- All controls have equal business importance
+- You're reporting to non-technical stakeholders
+- Regulatory requirements specify simple percentage
+
+### Use Our Weighted Approach If:
+- You need risk-based security prioritization
+- Critical controls should influence score more
+- You're managing security maturity progression
+- You need actionable remediation guidance
+- You want to prevent resource count skew
+
+### Use Both Approaches:
+- Report **AWS Config style** for auditors and compliance
+- Use **weighted approach** for security decision-making
+- Track both metrics over time for comprehensive view
+
+## Summary Table
+
+| Metric | AWS Config | Our Approach | Difference |
+|--------|-----------|--------------|------------|
+| **Complexity** | Low | Medium | More complex but more insightful |
+| **Accuracy** | Resource-level | Security-level | Better reflects security posture |
+| **Actionability** | Limited | High | Clear prioritization guidance |
+| **Customization** | None | High | Adaptable to organization needs |
+| **Audit-friendly** | Very | Moderate | May need explanation |
+| **Risk-awareness** | No | Yes | Reflects actual security risk |
+
+## Conclusion
+
+**AWS Config's approach** is simpler and more straightforward - it counts compliant resources and divides by total resources. This works well for basic compliance tracking but doesn't reflect security priorities.
+
+**Our weighted approach** adds complexity but provides **better security insights** by:
+1. Prioritizing critical controls (encryption, access control)
+2. Recognizing security maturity levels (IG1/IG2/IG3)
+3. Preventing resource count from dominating scores
+4. Providing actionable remediation guidance
+
+**Best Practice:** Use both approaches:
+- **AWS Config style** for compliance reporting and audits
+- **Weighted approach** for security decision-making and prioritization
+
+---
+
+**Recommendation:** Consider adding an "unweighted score" output option to provide both perspectives to users.