e11y 0.2.0 → 1.1.0

data/docs/use_cases/UC-013-high-cardinality-protection.md

@@ -13,12 +13,10 @@
 
 **The $68,000/month mistake:**
 ```ruby
-# ❌ CATASTROPHIC: Using user_id as metric label
-E11y.configure do |config|
-  config.metrics do
-    counter_for pattern: 'user.action',
-                name: 'user_actions_total',
-                tags: [:user_id, :action_type]  # ← 💸💸💸
+# ❌ CATASTROPHIC: Using user_id as metric label (event-level example - avoid!)
+class Events::UserAction < E11y::Event::Base
+  metrics do
+    counter :user_actions_total, tags: [:user_id, :action_type]  # ← 💸💸💸 DON'T
   end
 end
 
@@ -35,42 +33,24 @@ end
 - **Query timeouts** (PromQL queries take 30+ seconds)
 - **Incident during Black Friday** (metrics system collapsed)
 
-### E11y Solution
+### E11y Solution (Event-Level)
 
-**4-Layer Defense System + 99% Cost Reduction:**
+**Use low-cardinality tags in event-level metrics:**
 ```ruby
-# ✅ SAFE: Aggregate user_id → user_segment
-E11y.configure do |config|
-  config.metrics do
-    # Layer 1: Denylist (hard block)
-    forbidden_labels :user_id, :order_id, :session_id, :trace_id
-    
-    # Layer 2: Safe aggregation
-    counter_for pattern: 'user.action',
-                name: 'user_actions_total',
-                tags: [:user_segment, :action_type],  # ← 3 segments × 10 actions = 30 series
-                tag_extractors: {
-                  user_segment: ->(event) {
-                    user = User.find(event.payload[:user_id])
-                    user.segment  # 'free', 'paid', 'enterprise'
-                  }
-                }
-    
-    # Layer 3: Per-metric limits
-    cardinality_limit_for 'user_actions_total', max: 100
-    
-    # Layer 4: Dynamic monitoring
-    cardinality_monitoring do
-      warn_threshold 0.7   # Alert at 70%
-      auto_aggregate true  # Auto-fix if exceeded
-    end
+# ✅ SAFE: Use user_segment, not user_id
+class Events::UserAction < E11y::Event::Base
+  schema do
+    required(:user_id).filled(:string)
+    required(:action_type).filled(:string)
+    required(:user_segment).filled(:string)  # pre-aggregated: 'free', 'paid', 'enterprise'
+  end
+
+  metrics do
+    counter :user_actions_total, tags: [:user_segment, :action_type]  # 3 × 10 = 30 series
   end
 end
 
-# Result:
-# - 200 services × 10 segments × 5 dimensions = 10,000 series
-# - Datadog cost: $680/month
-# - Savings: $67,320/month (99% reduction) ✅
+# Result: low cardinality, manageable cost
 ```
 
 ---
@@ -283,7 +263,7 @@ end
 
 ### Layer Processing Flow
 
-> **Implementation:** See [ADR-002 Section 4.1: Four-Layer Defense](../ADR-002-metrics-yabeda.md#41-four-layer-defense) for detailed architecture.
+> **Implementation:** See [ADR-002 Section 4.1: Four-Layer Defense](../architecture/ADR-002-metrics-yabeda.md#41-four-layer-defense) for detailed architecture.
 
 **🔑 Critical: Layers execute SEQUENTIALLY (not simultaneously).**
 
@@ -399,14 +379,14 @@ Events::OrderPlaced.track(
 ### Layer 1: Denylist (Hard Block)
 
 > **⚠️ CRITICAL: Adapter-Specific Filtering**  
-> **Implementation:** See [ADR-002 Section 4.2: Layer 1 - Universal Denylist](../ADR-002-metrics-yabeda.md#42-layer-1-universal-denylist) for detailed architecture.
+> **Implementation:** See [ADR-002 Section 4.2: Layer 1 - Universal Denylist](../architecture/ADR-002-metrics-yabeda.md#42-layer-1-universal-denylist) for detailed architecture.
 >
 > **Cardinality protection (denylist/allowlist) applies ONLY to metrics adapters (Yabeda/Prometheus), NOT to other adapters:**
 >
 > | Adapter Type | Denylist Applied? | Why? |
 > |---|---|---|
 > | **Metrics (Yabeda/Prometheus)** | ✅ YES | High-cardinality labels cause memory explosion in time-series databases (1M labels = 1GB RAM). |
-> | **Logs (Loki)** | ❌ NO | Loki is designed for high-cardinality labels and uses different indexing strategy. Full payload preserved. |
+> | **Logs (Loki)** | Optional | Loki labels = event_name + severity (low cardinality). Payload (user_id, etc.) in log line. Optional `enable_cardinality_protection` for labels. |
 > | **Errors (Sentry)** | ❌ NO | Sentry needs full context for debugging. High cardinality is acceptable for error tracking. |
 > | **Audit (File/PostgreSQL)** | ❌ NO | Audit trails require complete, unfiltered data for compliance. |
 >
@@ -428,85 +408,11 @@ Events::OrderPlaced.track(
 > - ✅ **Compliance stays intact:** Audit logs remain complete and unfiltered
 > - ✅ **Best of both worlds:** Safety for metrics + completeness for logs/errors
 
-**Universal denylist - NEVER use these as labels (for metrics adapters):**
-
-```ruby
-E11y.configure do |config|
-  config.metrics do
-    # === UNBOUNDED IDENTIFIERS (FORBIDDEN) ===
-    forbidden_labels :user_id, :customer_id, :account_id,
-                     :order_id, :transaction_id, :invoice_id,
-                     :session_id, :request_id, :trace_id, :span_id
-    
-    # === INFRASTRUCTURE (FORBIDDEN) ===
-    forbidden_labels :pod_uid, :container_id, :instance_id,
-                     :node_name  # If dynamic
-    
-    # === NETWORK/HTTP (FORBIDDEN) ===
-    forbidden_labels :url,          # With query strings
-                     :ip_address,
-                     :user_agent,
-                     :hostname      # If ephemeral
-    
-    # === TIME-BASED (FORBIDDEN) ===
-    forbidden_labels :timestamp, :created_at,
-                     :version      # Patch-level: 2.5.7234
-    
-    # === ENFORCEMENT ===
-    enforcement :strict  # ERROR on forbidden label usage
-    # OR
-    enforcement :warn    # Log warning but allow
-    # OR
-    enforcement :aggregate  # Auto-aggregate to "_other"
-  end
-end
-
-# Usage:
-counter_for pattern: 'user.action',
-            tags: [:user_id]  # ← ERROR: "user_id is forbidden!"
-
-# Development warning:
-# [E11y ERROR] Metric 'user.action_total' uses forbidden label 'user_id'
-# Cardinality explosion risk! Use 'user_segment' instead.
-```
+**Avoid these as metric tags:** user_id, customer_id, order_id, session_id, trace_id, url, ip_address, timestamp.
 
 ---
 
-### Layer 2: Allowlist (Strict Mode)
-
-**Only allow explicitly safe labels:**
-
-```ruby
-E11y.configure do |config|
-  config.metrics do
-    # Strict mode: ONLY these labels allowed
-    allowed_labels_only true
-    
-    # === BUSINESS DIMENSIONS (< 50 values) ===
-    allowed_labels :status,          # pending, paid, failed (4-10 values)
-                   :payment_method,  # card, paypal (5-20 values)
-                   :plan_tier        # free, pro, enterprise (3-5 values)
-    
-    # === INFRASTRUCTURE (< 20 values) ===
-    allowed_labels :env,             # production, staging, dev (3 values)
-                   :region,          # us-east, eu-west (5-20 values)
-                   :cluster,         # main, backup (2-5 values)
-                   :availability_zone
-    
-    # === HTTP/SERVICE (< 100 values) ===
-    allowed_labels :http_method,     # GET, POST, PUT, DELETE (10 values)
-                   :http_status_code, # 200, 404, 500 (50 values)
-                   :controller_action # UsersController#show (20-100 values)
-  end
-end
-
-# Usage:
-counter_for pattern: 'order.paid',
-            tags: [:currency]  # ← ERROR: "currency not in allowlist!"
-
-# Must explicitly allow:
-allowed_labels :currency  # USD, EUR, GBP (3-20 values)
-```
+### Layer 2: Safe Labels
 
 **Rule of thumb:**
 - ✅ **< 10 values** - Always safe
@@ -517,55 +423,11 @@ allowed_labels :currency  # USD, EUR, GBP (3-20 values)
 
 ### Layer 3: Per-Metric Limits
 
-**Set cardinality limits per metric:**
-
-```ruby
-E11y.configure do |config|
-  config.metrics do
-    # === GLOBAL DEFAULT ===
-    default_cardinality_limit 1_000
-    
-    # === PER-METRIC LIMITS ===
-    cardinality_limit_for 'http.requests' do
-      max_cardinality 2_000           # Higher limit for this metric
-      overflow_strategy :drop         # → Drop overflow events
-      overflow_sample_rate 0.1        # Sample 10% of overflow events
-    end
-    
-    cardinality_limit_for 'user.actions' do
-      max_cardinality 500             # Lower limit
-      overflow_strategy :drop         # Drop overflow events
-      overflow_alert true             # Alert on overflow
-    end
-    
-    cardinality_limit_for 'orders.paid' do
-      max_cardinality 100
-      overflow_strategy :alert        # Alert ops team + drop
-    end
-  end
-end
-
-# How it works:
-# 1. Track unique label combinations per metric
-# 2. If exceeds limit:
-#    - :drop → Discard overflow events (increment drop counter)
-#    - :alert → Alert ops team + drop
-# 
-# NOTE: For aggregation/relabeling (e.g., user_id → user_segment),
-#       use tag_extractors (see "Aggregation" section below), 
-#       NOT overflow_strategy.
-```
-
-**Overflow strategies:**
-
-| Strategy | Behavior | Use Case |
-|----------|----------|----------|
-| `:drop` | Discard overflow events | Default, simplest |
-| `:alert` | Alert ops team + drop | Critical metrics |
+**Yabeda adapter supports cardinality limits** via its config. Use low-cardinality tags in event-level metrics.
 
 #### Thread Safety
 
-> **Implementation:** See [ADR-002 Section 4.4: Layer 3 - Per-Metric Cardinality Limits](../ADR-002-metrics-yabeda.md#44-layer-3-per-metric-cardinality-limits) for detailed architecture.
+> **Implementation:** See [ADR-002 Section 4.4: Layer 3 - Per-Metric Cardinality Limits](../architecture/ADR-002-metrics-yabeda.md#44-layer-3-per-metric-cardinality-limits) for detailed architecture.
 > 
 > **Sources:**
 > - [Ruby Hash thread safety - Stack Overflow](https://stackoverflow.com/questions/22674498/thread-safety-for-hashes-in-ruby)
@@ -716,7 +578,7 @@ end
 
 #### Action Selection Guide
 
-> **Implementation:** See [ADR-002 Section 4.5: Layer 4 - Dynamic Actions](../ADR-002-metrics-yabeda.md#45-layer-4-dynamic-actions) for detailed architecture.
+> **Implementation:** See [ADR-002 Section 4.5: Layer 4 - Dynamic Actions](../architecture/ADR-002-metrics-yabeda.md#45-layer-4-dynamic-actions) for detailed architecture.
 
 **🎯 When cardinality limit is exceeded, which action should you choose?**
 
@@ -862,7 +724,7 @@ rate(e11y_cardinality_actions_total{action="alert"}[5m])
 
 ### 1. Aggregation (Best ROI - 99% Reduction)
 
-> **Note:** This section describes **relabeling/normalization** (e.g., `user_id` → `user_segment`) via `tag_extractors`, which is different from `overflow_strategy`. Aggregation reduces cardinality **before** metrics are created, while overflow handling (`drop`/`alert`) deals with exceeding limits **after** creation. See [ADR-002 Section 4.5](../ADR-002-metrics-yabeda.md#45-cardinality-protection) for implementation details.
+> **Note:** This section describes **relabeling/normalization** (e.g., `user_id` → `user_segment`) via `tag_extractors`, which is different from `overflow_strategy`. Aggregation reduces cardinality **before** metrics are created, while overflow handling (`drop`/`alert`) deals with exceeding limits **after** creation. See [ADR-002 Section 4.5](../architecture/ADR-002-metrics-yabeda.md#45-cardinality-protection) for implementation details.
 
 **Problem:** 1M users = 1M metric series
 
@@ -1039,7 +901,7 @@ end
 ### 6. Universal Cardinality Protection (C04 Resolution) ⚠️ CRITICAL
 
 > **⚠️ CRITICAL: C04 Conflict Resolution - Cardinality Protection for ALL Backends**  
-> **See:** [ADR-009 Section 8](../ADR-009-cost-optimization.md#8-cardinality-protection-c04-resolution--critical) for detailed architecture and cost impact analysis.  
+> **See:** [ADR-009 Section 8](../architecture/ADR-009-cost-optimization.md#8-cardinality-protection-c04-resolution--critical) for detailed architecture and cost impact analysis.  
 > **Problem:** Original UC-013 cardinality protection applied ONLY to Yabeda/Prometheus metrics, but NOT to OpenTelemetry span attributes or Loki log labels. High-cardinality values (`user_id`, `order_id`) bypassed protection and caused cost explosions in OTLP backends (Datadog, Honeycomb).  
 > **Solution:** Universal `CardinalityFilter` middleware applies protection to **ALL backends** (Yabeda, OpenTelemetry, Loki) with optional per-backend overrides.
 
@@ -1623,7 +1485,7 @@ after = calculate_cardinality_cost(
 
 ## ❓ Frequently Asked Questions
 
-> **Technical Details:** See [ADR-002 Section 11: FAQ & Critical Clarifications](../ADR-002-metrics-yabeda.md#11-faq--critical-clarifications) for architectural rationale.
+> **Technical Details:** See [ADR-002 Section 11: FAQ & Critical Clarifications](../architecture/ADR-002-metrics-yabeda.md#11-faq--critical-clarifications) for architectural rationale.
 
 ### Q1: Does cardinality protection apply to all my logs and metrics?
 
@@ -2097,7 +1959,7 @@ end
 
 ## 📚 Related Use Cases
 
-- **[UC-003: Pattern-Based Metrics](./UC-003-pattern-based-metrics.md)** - Auto-generate metrics
+- **[UC-003: Event Metrics](./UC-003-event-metrics.md)** - Metrics in event classes
 - **[UC-008: OpenTelemetry Integration](./UC-008-opentelemetry-integration.md)** - OTLP cardinality protection (C04)
 - **[UC-015: Cost Optimization](./UC-015-cost-optimization.md)** - Reduce observability costs
 

data/docs/use_cases/UC-014-adaptive-sampling.md

@@ -1026,7 +1026,7 @@ end
 
 ### Strategy 8: Stratified Sampling for Accurate SLO (C11 Resolution) ⚠️
 
-> **Reference:** See [ADR-009 §3.7: Stratified Sampling for SLO Accuracy](../ADR-009-cost-optimization.md#37-stratified-sampling-for-slo-accuracy-c11-resolution) for full architecture and [UC-004: SLO Tracking with Sampling Correction](./UC-004-zero-config-slo-tracking.md#sampling-correction-for-accurate-slo-c11-resolution) for SLO calculation details.
+> **Reference:** See [ADR-009 §3.7: Stratified Sampling for SLO Accuracy](../architecture/ADR-009-cost-optimization.md#37-stratified-sampling-for-slo-accuracy-c11-resolution) for full architecture and [UC-004: SLO Tracking with Sampling Correction](./UC-004-zero-config-slo-tracking.md#sampling-correction-for-accurate-slo-c11-resolution) for SLO calculation details.
 
 **Problem with Random Sampling:** Breaks SLO metrics! Errors are rare (5%) → random 10% sampling drops 90% of errors → SLO appears better than reality.
 
@@ -1046,7 +1046,7 @@ E11y.configure do |config|
       stratified_rates do
         error 1.0    # 100% - Keep ALL errors (critical for SLO!)
         warn  0.5    # 50%  - Medium priority
-        info  0.1    # 10%  - Low priority (успешные запросы)
+        info  0.1    # 10%  - Low priority (successful requests)
         debug 0.05   # 5%   - Very low priority
       end
     end

data/docs/use_cases/UC-015-cost-optimization.md

@@ -56,12 +56,7 @@ E11y.configure do |config|
                       drop_empty_strings: true,
                       truncate_strings: 1000  # chars
     
-    # 5. Tiered storage (60% cheaper)
-    retention_tiers do
-      hot 7.days, storage: :loki       # Fast queries
-      warm 30.days, storage: :s3        # Slower, cheaper
-      cold 1.year, storage: :s3_glacier # Archive
-    end
+    # 5. Routing by retention_until — config.routing_rules (see Strategy 4)
     
     # 6. Smart routing (send only what's needed)
     routing do
@@ -79,13 +74,13 @@ end
 # Result:
 # - 100k events/sec → 10k events/sec (adaptive sampling)
 # - 2KB/event → 0.6KB/event (compression + minimization)
-# - 30 days hot storage → 7 days hot + 23 days warm (tiered)
+# - Short retention → stdout, long → Loki (routing by retention_until)
 # - Datadog: Only errors (3k/sec instead of 100k/sec)
 #
 # New monthly cost:
 # - Datadog: $3,000 → $500 (only errors)
 # - Loki: $10,368 → $1,200 (10% volume, 70% smaller, 7 days hot)
-# - S3: $200 (warm storage)
+# - Archival job (separate): exports by retention_until to cold storage
 # - Total: $1,900/month = $22,800/year
 #
 # SAVINGS: $160,416 - $22,800 = $137,616/year (86% reduction!)
@@ -95,7 +90,7 @@ end
 
 ## 🎯 Cost Optimization Strategies
 
-> **Note:** This UC focuses on proven, low-overhead optimizations. **Deduplication is intentionally NOT included** as a strategy. While it may seem like an obvious cost optimization, [ADR-009 Section 9.2.D](../ADR-009-cost-optimization.md#alternatives-considered) explains why it was rejected: high computational overhead (hash + Redis lookup per event), large memory cost (3.6GB for 1000 events/sec), false positives on legitimate retries, and debug confusion. Better alternatives (sampling + compression) achieve the same cost goals without these drawbacks.
+> **Note:** This UC focuses on proven, low-overhead optimizations. **Deduplication is intentionally NOT included** as a strategy. While it may seem like an obvious cost optimization, [ADR-009 Section 9.2.D](../architecture/ADR-009-cost-optimization.md#alternatives-considered) explains why it was rejected: high computational overhead (hash + Redis lookup per event), large memory cost (3.6GB for 1000 events/sec), false positives on legitimate retries, and debug confusion. Better alternatives (sampling + compression) achieve the same cost goals without these drawbacks.
 
 ### Strategy 1: Intelligent Sampling by Value
 
@@ -254,57 +249,34 @@ end
 
 ---
 
-### Strategy 4: Tiered Storage
+### Strategy 4: Routing by retention_until
 
-**Hot/warm/cold storage based on age:**
+**Route events to adapters based on retention (at collection):**
 ```ruby
+# Events declare retention_period; retention_until is auto-calculated in payload.
+# Routing rules use it to choose adapter — short retention → cheap storage.
 E11y.configure do |config|
-  config.cost_optimization do
-    tiered_storage do
-      # HOT: Fast queries, expensive ($0.20/GB/month)
-      hot_tier do
-        duration 7.days
-        storage :loki  # OR :elasticsearch
-        query_performance :fast
-      end
-      
-      # WARM: Slower queries, cheaper ($0.05/GB/month)
-      warm_tier do
-        duration 30.days
-        storage :s3
-        query_performance :medium
-        compression :zstd  # Compress when moving to warm
-      end
-      
-      # COLD: Archive, very cheap ($0.004/GB/month)
-      cold_tier do
-        duration 1.year
-        storage :s3_glacier
-        query_performance :slow  # Minutes to hours
-        compression :zstd
-      end
-      
-      # Auto-archival
-      auto_archive enabled: true,
-                   schedule: '0 2 * * *'  # 2 AM daily
-    end
-  end
+  config.routing_rules = [
+    ->(event) { :audit_encrypted if event[:audit_event] },
+    ->(event) {
+      return :loki unless event[:retention_until]
+      days = (Time.parse(event[:retention_until]) - Time.now) / 86400
+      days <= 7 ? :stdout : :loki  # Short → free, long → Loki
+    }
+  ]
+  config.fallback_adapters = [:loki]
 end
 
-# Cost comparison (per 1TB):
-# Hot (Loki): $0.20/GB × 1000 = $200/month
-# Warm (S3): $0.05/GB × 1000 = $50/month
-# Cold (Glacier): $0.004/GB × 1000 = $4/month
-#
-# Strategy:
-# - 7 days hot (for active debugging)
-# - 30 days warm (for recent lookups)
-# - 1 year cold (for compliance)
-#
-# Cost for 30 days of data:
-# Before: 30 days × $200 = $6,000/month
-# After: (7 × $200) + (23 × $50) + (0 × $4) = $1,400 + $1,150 = $2,550/month
-# Savings: $3,450/month (58% reduction!)
+# Event classes declare retention:
+#   class DebugEvent < E11y::Event::Base
+#     retention_period 7.days   # → stdout
+#   end
+#   class AuditEvent < E11y::Event::Base
+#     retention_period 7.years  # → Loki (archival job exports later)
+#   end
+
+# Cost: Short retention events never hit Loki. Archival job (separate) filters by
+# retention_until for cold storage. Savings: ~58% vs all-events-to-Loki.
 ```
 
 ---
@@ -324,11 +296,11 @@ E11y.configure do |config|
       # High-value transactions → All (audit + analytics)
       route event_patterns: ['payment.*', 'order.*'],
             when: ->(e) { e.payload[:amount].to_i > 1000 },
-            to: [:datadog, :loki, :s3_archive]
+            to: [:datadog, :loki]
       
       # Security events → Specific SIEM
       route event_patterns: ['security.*', 'audit.*'],
-            to: [:splunk, :s3_archive]
+            to: [:splunk]
       
       # Debug events → Only Loki (no expensive Datadog)
       route severities: [:debug],
@@ -356,50 +328,25 @@ end
 
 ---
 
-### Strategy 6: Retention-Aware Tagging
+### Strategy 6: retention_period DSL
 
-**Tag events with retention requirements:**
+**Declare retention per event; routing uses retention_until:**
 ```ruby
-E11y.configure do |config|
-  config.cost_optimization do
-    retention_aware_tagging do
-      # Auto-tag events with retention hints
-      tag_with_retention do
-        # Compliance events: Long retention
-        when_pattern 'audit.*', 'gdpr.*', retention: 7.years
-        
-        # Financial: Long retention
-        when_pattern 'payment.*', 'transaction.*', retention: 7.years
-        
-        # Errors: Medium retention
-        when_severity :error, :fatal, retention: 90.days
-        
-        # Debug: Short retention
-        when_severity :debug, retention: 7.days
-        
-        # Default
-        default_retention 30.days
-      end
-      
-      # Backend respects retention tags
-      backends do
-        loki retention_based: true,
-             max_retention: 30.days
-        
-        s3_archive retention_based: true,
-                   max_retention: 7.years
-      end
-    end
-  end
+# Event-level retention (used by routing_rules + archival job)
+class DebugEvent < E11y::Event::Base
+  retention_period 7.days
 end
 
-# Result:
-# - Debug events: 7 days in Loki (cheap)
-# - Errors: 90 days in Loki
-# - Compliance: 7 years in S3 Glacier (very cheap)
-# - Default: 30 days in Loki
-#
-# Cost optimization: Store data only as long as needed!
+class PaymentEvent < E11y::Event::Base
+  retention_period 7.years
+end
+
+class OrderEvent < E11y::Event::Base
+  # Uses config.default_retention_period (30 days)
+end
+
+# retention_until is auto-calculated in payload. Routing (Strategy 4) and
+# archival job both use it. No separate tagging — one field, two consumers.
 ```
 
 ---
@@ -641,7 +588,7 @@ end
 config.cost_optimization do
   intelligent_sampling { ... }  # 90% reduction
   compression { ... }           # 70% smaller payloads
-  tiered_storage { ... }        # 60% cheaper storage
+  routing_rules (by retention_until)  # Short → stdout, long → Loki
   smart_routing { ... }         # 50% fewer expensive destinations
 end
 # Combined: ~95% cost reduction!
@@ -653,7 +600,7 @@ end
 # Dashboard: "Cost Optimization Savings"
 # - Monthly savings: $X
 # - YTD savings: $Y
-# - Optimization breakdown (sampling, compression, tiered storage)
+# - Optimization breakdown (sampling, compression, routing)
 ```
 
 **3. Test in staging first**