e11y 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (157) hide show
  1. checksums.yaml +7 -0
  2. data/.rspec +4 -0
  3. data/.rubocop.yml +69 -0
  4. data/CHANGELOG.md +26 -0
  5. data/CODE_OF_CONDUCT.md +64 -0
  6. data/LICENSE.txt +21 -0
  7. data/README.md +179 -0
  8. data/Rakefile +37 -0
  9. data/benchmarks/run_all.rb +33 -0
  10. data/config/README.md +83 -0
  11. data/config/loki-local-config.yaml +35 -0
  12. data/config/prometheus.yml +15 -0
  13. data/docker-compose.yml +78 -0
  14. data/docs/00-ICP-AND-TIMELINE.md +483 -0
  15. data/docs/01-SCALE-REQUIREMENTS.md +858 -0
  16. data/docs/ADR-001-architecture.md +2617 -0
  17. data/docs/ADR-002-metrics-yabeda.md +1395 -0
  18. data/docs/ADR-003-slo-observability.md +3337 -0
  19. data/docs/ADR-004-adapter-architecture.md +2385 -0
  20. data/docs/ADR-005-tracing-context.md +1372 -0
  21. data/docs/ADR-006-security-compliance.md +4143 -0
  22. data/docs/ADR-007-opentelemetry-integration.md +1385 -0
  23. data/docs/ADR-008-rails-integration.md +1911 -0
  24. data/docs/ADR-009-cost-optimization.md +2993 -0
  25. data/docs/ADR-010-developer-experience.md +2166 -0
  26. data/docs/ADR-011-testing-strategy.md +1836 -0
  27. data/docs/ADR-012-event-evolution.md +958 -0
  28. data/docs/ADR-013-reliability-error-handling.md +2750 -0
  29. data/docs/ADR-014-event-driven-slo.md +1533 -0
  30. data/docs/ADR-015-middleware-order.md +1061 -0
  31. data/docs/ADR-016-self-monitoring-slo.md +1234 -0
  32. data/docs/API-REFERENCE-L28.md +914 -0
  33. data/docs/COMPREHENSIVE-CONFIGURATION.md +2366 -0
  34. data/docs/IMPLEMENTATION_NOTES.md +2804 -0
  35. data/docs/IMPLEMENTATION_PLAN.md +1971 -0
  36. data/docs/IMPLEMENTATION_PLAN_ARCHITECTURE.md +586 -0
  37. data/docs/PLAN.md +148 -0
  38. data/docs/QUICK-START.md +934 -0
  39. data/docs/README.md +296 -0
  40. data/docs/design/00-memory-optimization.md +593 -0
  41. data/docs/guides/MIGRATION-L27-L28.md +692 -0
  42. data/docs/guides/PERFORMANCE-BENCHMARKS.md +434 -0
  43. data/docs/guides/README.md +44 -0
  44. data/docs/prd/01-overview-vision.md +440 -0
  45. data/docs/use_cases/README.md +119 -0
  46. data/docs/use_cases/UC-001-request-scoped-debug-buffering.md +813 -0
  47. data/docs/use_cases/UC-002-business-event-tracking.md +1953 -0
  48. data/docs/use_cases/UC-003-pattern-based-metrics.md +1627 -0
  49. data/docs/use_cases/UC-004-zero-config-slo-tracking.md +728 -0
  50. data/docs/use_cases/UC-005-sentry-integration.md +759 -0
  51. data/docs/use_cases/UC-006-trace-context-management.md +905 -0
  52. data/docs/use_cases/UC-007-pii-filtering.md +2648 -0
  53. data/docs/use_cases/UC-008-opentelemetry-integration.md +1153 -0
  54. data/docs/use_cases/UC-009-multi-service-tracing.md +1043 -0
  55. data/docs/use_cases/UC-010-background-job-tracking.md +1018 -0
  56. data/docs/use_cases/UC-011-rate-limiting.md +1906 -0
  57. data/docs/use_cases/UC-012-audit-trail.md +2301 -0
  58. data/docs/use_cases/UC-013-high-cardinality-protection.md +2127 -0
  59. data/docs/use_cases/UC-014-adaptive-sampling.md +1940 -0
  60. data/docs/use_cases/UC-015-cost-optimization.md +735 -0
  61. data/docs/use_cases/UC-016-rails-logger-migration.md +785 -0
  62. data/docs/use_cases/UC-017-local-development.md +867 -0
  63. data/docs/use_cases/UC-018-testing-events.md +1081 -0
  64. data/docs/use_cases/UC-019-tiered-storage-migration.md +562 -0
  65. data/docs/use_cases/UC-020-event-versioning.md +708 -0
  66. data/docs/use_cases/UC-021-error-handling-retry-dlq.md +956 -0
  67. data/docs/use_cases/UC-022-event-registry.md +648 -0
  68. data/docs/use_cases/backlog.md +226 -0
  69. data/e11y.gemspec +76 -0
  70. data/lib/e11y/adapters/adaptive_batcher.rb +207 -0
  71. data/lib/e11y/adapters/audit_encrypted.rb +239 -0
  72. data/lib/e11y/adapters/base.rb +580 -0
  73. data/lib/e11y/adapters/file.rb +224 -0
  74. data/lib/e11y/adapters/in_memory.rb +216 -0
  75. data/lib/e11y/adapters/loki.rb +333 -0
  76. data/lib/e11y/adapters/otel_logs.rb +203 -0
  77. data/lib/e11y/adapters/registry.rb +141 -0
  78. data/lib/e11y/adapters/sentry.rb +230 -0
  79. data/lib/e11y/adapters/stdout.rb +108 -0
  80. data/lib/e11y/adapters/yabeda.rb +370 -0
  81. data/lib/e11y/buffers/adaptive_buffer.rb +339 -0
  82. data/lib/e11y/buffers/base_buffer.rb +40 -0
  83. data/lib/e11y/buffers/request_scoped_buffer.rb +246 -0
  84. data/lib/e11y/buffers/ring_buffer.rb +267 -0
  85. data/lib/e11y/buffers.rb +14 -0
  86. data/lib/e11y/console.rb +122 -0
  87. data/lib/e11y/current.rb +48 -0
  88. data/lib/e11y/event/base.rb +894 -0
  89. data/lib/e11y/event/value_sampling_config.rb +84 -0
  90. data/lib/e11y/events/base_audit_event.rb +43 -0
  91. data/lib/e11y/events/base_payment_event.rb +33 -0
  92. data/lib/e11y/events/rails/cache/delete.rb +21 -0
  93. data/lib/e11y/events/rails/cache/read.rb +23 -0
  94. data/lib/e11y/events/rails/cache/write.rb +22 -0
  95. data/lib/e11y/events/rails/database/query.rb +45 -0
  96. data/lib/e11y/events/rails/http/redirect.rb +21 -0
  97. data/lib/e11y/events/rails/http/request.rb +26 -0
  98. data/lib/e11y/events/rails/http/send_file.rb +21 -0
  99. data/lib/e11y/events/rails/http/start_processing.rb +26 -0
  100. data/lib/e11y/events/rails/job/completed.rb +22 -0
  101. data/lib/e11y/events/rails/job/enqueued.rb +22 -0
  102. data/lib/e11y/events/rails/job/failed.rb +22 -0
  103. data/lib/e11y/events/rails/job/scheduled.rb +23 -0
  104. data/lib/e11y/events/rails/job/started.rb +22 -0
  105. data/lib/e11y/events/rails/log.rb +56 -0
  106. data/lib/e11y/events/rails/view/render.rb +23 -0
  107. data/lib/e11y/events.rb +18 -0
  108. data/lib/e11y/instruments/active_job.rb +201 -0
  109. data/lib/e11y/instruments/rails_instrumentation.rb +141 -0
  110. data/lib/e11y/instruments/sidekiq.rb +175 -0
  111. data/lib/e11y/logger/bridge.rb +205 -0
  112. data/lib/e11y/metrics/cardinality_protection.rb +172 -0
  113. data/lib/e11y/metrics/cardinality_tracker.rb +134 -0
  114. data/lib/e11y/metrics/registry.rb +234 -0
  115. data/lib/e11y/metrics/relabeling.rb +226 -0
  116. data/lib/e11y/metrics.rb +102 -0
  117. data/lib/e11y/middleware/audit_signing.rb +174 -0
  118. data/lib/e11y/middleware/base.rb +140 -0
  119. data/lib/e11y/middleware/event_slo.rb +167 -0
  120. data/lib/e11y/middleware/pii_filter.rb +266 -0
  121. data/lib/e11y/middleware/pii_filtering.rb +280 -0
  122. data/lib/e11y/middleware/rate_limiting.rb +214 -0
  123. data/lib/e11y/middleware/request.rb +163 -0
  124. data/lib/e11y/middleware/routing.rb +157 -0
  125. data/lib/e11y/middleware/sampling.rb +254 -0
  126. data/lib/e11y/middleware/slo.rb +168 -0
  127. data/lib/e11y/middleware/trace_context.rb +131 -0
  128. data/lib/e11y/middleware/validation.rb +118 -0
  129. data/lib/e11y/middleware/versioning.rb +132 -0
  130. data/lib/e11y/middleware.rb +12 -0
  131. data/lib/e11y/pii/patterns.rb +90 -0
  132. data/lib/e11y/pii.rb +13 -0
  133. data/lib/e11y/pipeline/builder.rb +155 -0
  134. data/lib/e11y/pipeline/zone_validator.rb +110 -0
  135. data/lib/e11y/pipeline.rb +12 -0
  136. data/lib/e11y/presets/audit_event.rb +65 -0
  137. data/lib/e11y/presets/debug_event.rb +34 -0
  138. data/lib/e11y/presets/high_value_event.rb +51 -0
  139. data/lib/e11y/presets.rb +19 -0
  140. data/lib/e11y/railtie.rb +138 -0
  141. data/lib/e11y/reliability/circuit_breaker.rb +216 -0
  142. data/lib/e11y/reliability/dlq/file_storage.rb +277 -0
  143. data/lib/e11y/reliability/dlq/filter.rb +117 -0
  144. data/lib/e11y/reliability/retry_handler.rb +207 -0
  145. data/lib/e11y/reliability/retry_rate_limiter.rb +117 -0
  146. data/lib/e11y/sampling/error_spike_detector.rb +225 -0
  147. data/lib/e11y/sampling/load_monitor.rb +161 -0
  148. data/lib/e11y/sampling/stratified_tracker.rb +92 -0
  149. data/lib/e11y/sampling/value_extractor.rb +82 -0
  150. data/lib/e11y/self_monitoring/buffer_monitor.rb +79 -0
  151. data/lib/e11y/self_monitoring/performance_monitor.rb +97 -0
  152. data/lib/e11y/self_monitoring/reliability_monitor.rb +146 -0
  153. data/lib/e11y/slo/event_driven.rb +150 -0
  154. data/lib/e11y/slo/tracker.rb +119 -0
  155. data/lib/e11y/version.rb +9 -0
  156. data/lib/e11y.rb +283 -0
  157. metadata +452 -0
@@ -0,0 +1,4143 @@
1
+ # ADR-006: Security & Compliance
2
+
3
+ **Status:** Draft
4
+ **Date:** January 12, 2026
5
+ **Covers:** UC-007 (PII Filtering), UC-011 (Rate Limiting), UC-012 (Audit Trail)
6
+ **Depends On:** ADR-001 (Core Architecture), ADR-004 (Adapters)
7
+
8
+ ---
9
+
10
+ ## 📋 Table of Contents
11
+
12
+ 1. [Context & Problem](#1-context--problem)
13
+ 2. [Architecture Overview](#2-architecture-overview)
14
+ 3. [PII Filtering](#3-pii-filtering)
15
+ - 3.1. [Rails Integration](#31-rails-integration)
16
+ - 3.2. [Pattern-Based Filtering](#32-pattern-based-filtering)
17
+ - 3.3. [Deep Scanning](#33-deep-scanning)
18
+ - 3.4. [Per-Adapter Rules](#34-per-adapter-rules)
19
+ - 3.5. [Sampling for Debug](#35-sampling-for-debug)
20
+ 4. [Rate Limiting](#4-rate-limiting)
21
+ - 4.1. [Global Rate Limiting](#41-global-rate-limiting)
22
+ - 4.2. [Per-Event Rate Limiting](#42-per-event-rate-limiting)
23
+ - 4.3. [Per-Context Rate Limiting](#43-per-context-rate-limiting)
24
+ - 4.4. [Redis Integration](#44-redis-integration)
25
+ 5. [Audit Trail](#5-audit-trail)
26
+ - 5.1. [Immutable Events](#51-immutable-events)
27
+ - 5.2. [Cryptographic Signing](#52-cryptographic-signing)
28
+ - 5.3. [Compliance Features](#53-compliance-features)
29
+ - 5.4. [Tamper Detection](#54-tamper-detection)
30
+ - 5.5. [OpenTelemetry Baggage PII Protection (C08 Resolution)](#55-opentelemetry-baggage-pii-protection-c08-resolution) ⚠️ CRITICAL
31
+ - 5.5.1. The Problem: PII Leaking via OpenTelemetry Baggage
32
+ - 5.5.2. Decision: Block PII from Baggage Entirely
33
+ - 5.5.3. BaggageProtection Middleware Implementation
34
+ - 5.5.4. Configuration
35
+ - 5.5.5. Usage Examples
36
+ - 5.5.6. Strict Mode (Development/Staging)
37
+ - 5.5.7. Trade-offs & GDPR Compliance (C08)
38
+ - 5.6. [PII Handling for Event Replay from DLQ (C07 Resolution)](#56-pii-handling-for-event-replay-from-dlq-c07-resolution) ⚠️ HIGH
39
+ - 5.6.1. The Problem: Double-Hashing PII on Replay
40
+ - 5.6.2. Decision: Skip PII Filtering for Replayed Events
41
+ - 5.6.3. PiiFilter Middleware with Replay Detection
42
+ - 5.6.4. DLQ Replay Service with Metadata Flags
43
+ - 5.6.5. Configuration
44
+ - 5.6.6. Usage Examples
45
+ - 5.6.7. Idempotency Verification (Testing)
46
+ - 5.6.8. Trade-offs & Audit Trail Integrity (C07)
47
+ 6. [GDPR Compliance](#6-gdpr-compliance)
48
+ 7. [Configuration](#7-configuration)
49
+ 8. [Testing](#8-testing)
50
+ 9. [Trade-offs](#9-trade-offs)
51
+
52
+ ---
53
+
54
+ ## 1. Context & Problem
55
+
56
+ ### 1.1. Problem Statement
57
+
58
+ **Current Pain Points:**
59
+
60
+ 1. **PII Exposure:**
61
+ ```ruby
62
+ # ❌ Sensitive data in logs
63
+ Events::UserLogin.track(
64
+ email: 'user@example.com', # PII!
65
+ password: 'secret123', # Credentials!
66
+ ip_address: '192.168.1.100' # PII!
67
+ )
68
+ ```
69
+
70
+ 2. **No Rate Limiting:**
71
+ ```ruby
72
+ # ❌ Event flood can overwhelm system
73
+ 1_000_000.times do
74
+ Events::Debug.track(message: 'spam')
75
+ end
76
+ ```
77
+
78
+ 3. **No Audit Trail:**
79
+ ```ruby
80
+ # ❌ Can't prove what happened
81
+ Events::GdprDeletion.track(user_id: '123')
82
+ # Was this really tracked? Can it be tampered?
83
+ ```
84
+
85
+ ### 1.2. Goals
86
+
87
+ **Primary Goals:**
88
+ - ✅ Rails-compatible PII filtering
89
+ - ✅ Per-adapter filtering rules
90
+ - ✅ Global + per-event + per-context rate limiting
91
+ - ✅ Cryptographically-signed audit trail
92
+ - ✅ GDPR compliance ready
93
+
94
+ **Non-Goals:**
95
+ - ❌ Perfect PII detection (false positives OK)
96
+ - ❌ Zero-impact performance (small overhead acceptable)
97
+ - ❌ Full GDPR implementation (provide tools, not guarantee)
98
+
99
+ ### 1.3. Success Metrics
100
+
101
+ | Metric | Target | Critical? |
102
+ |--------|--------|-----------|
103
+ | **PII filtering overhead** | <0.2ms per event | ✅ Yes |
104
+ | **Rate limit accuracy** | >99% | ✅ Yes |
105
+ | **Audit signature time** | <1ms | ✅ Yes |
106
+ | **False positive PII** | <5% | ⚠️ Important |
107
+
108
+ ---
109
+
110
+ ## 2. Architecture Overview
111
+
112
+ ### 2.1. System Context
113
+
114
+ ```mermaid
115
+ C4Context
116
+ title Security & Compliance System Context
117
+
118
+ Person(dev, "Developer", "Tracks events")
119
+ Person(auditor, "Auditor", "Reviews audit logs")
120
+ Person(dpo, "DPO", "GDPR compliance")
121
+
122
+ System(e11y, "E11y Gem", "Event tracking with security")
123
+
124
+ System_Ext(redis, "Redis", "Rate limiting state")
125
+ System_Ext(kms, "KMS", "Signing keys")
126
+ System_Ext(audit_store, "Audit Store", "Immutable audit logs")
127
+
128
+ Rel(dev, e11y, "Tracks events", "Events::OrderPaid.track(...)")
129
+ Rel(e11y, redis, "Check rate limits", "Redis commands")
130
+ Rel(e11y, kms, "Sign audit events", "HMAC-SHA256")
131
+ Rel(e11y, audit_store, "Store signed events", "Append-only")
132
+
133
+ Rel(auditor, audit_store, "Review logs", "Read-only")
134
+ Rel(dpo, e11y, "Configure PII rules", "Config")
135
+ ```
136
+
137
+ ### 2.2. Security Pipeline
138
+
139
+ ```mermaid
140
+ graph TB
141
+ Event[Event.track] --> Pipeline[Middleware Chain]
142
+
143
+ Pipeline --> PII[PII Filter Middleware]
144
+ Pipeline --> Rate[Rate Limit Middleware]
145
+ Pipeline --> Audit[Audit Middleware]
146
+
147
+ subgraph "PII Filtering"
148
+ PII --> Rails[Rails Filters]
149
+ PII --> Pattern[Pattern Matching]
150
+ PII --> Deep[Deep Scan]
151
+ PII --> PerAdapter[Per-Adapter Rules]
152
+ end
153
+
154
+ subgraph "Rate Limiting"
155
+ Rate --> Global[Global Limiter]
156
+ Rate --> PerEvent[Per-Event Limiter]
157
+ Rate --> PerContext[Per-Context Limiter]
158
+
159
+ Global --> Redis1[Redis]
160
+ PerEvent --> Redis2[Redis]
161
+ PerContext --> Redis3[Redis]
162
+ end
163
+
164
+ subgraph "Audit Trail"
165
+ Audit --> Sign[Cryptographic Signing]
166
+ Audit --> Chain[Chain Verification]
167
+ Audit --> Store[Immutable Storage]
168
+ end
169
+
170
+ PII --> Adapters[Adapters]
171
+ Rate --> Adapters
172
+ Audit --> AuditAdapter[Audit Adapter Only]
173
+
174
+ style PII fill:#f8d7da
175
+ style Rate fill:#fff3cd
176
+ style Audit fill:#d1ecf1
177
+ ```
178
+
179
+ ### 2.3. Data Flow with Security
180
+
181
+ ```mermaid
182
+ sequenceDiagram
183
+ participant App as Application
184
+ participant Event as Events::OrderPaid
185
+ participant PII as PII Filter
186
+ participant Rate as Rate Limiter
187
+ participant Audit as Audit Trail
188
+ participant Adapters as Adapters
189
+
190
+ App->>Event: .track(email: 'user@x.com', order_id: '123')
191
+ Event->>PII: filter(event_data)
192
+
193
+ PII->>PII: Check Rails filters
194
+ PII->>PII: Check pattern matchers
195
+ PII->>PII: Deep scan nested hashes
196
+
197
+ Note over PII: email: 'user@x.com' → '[FILTERED]'
198
+
199
+ PII->>Rate: process(filtered_data)
200
+
201
+ Rate->>Rate: Check global limit (10k/sec)
202
+ Rate->>Rate: Check event limit (100/sec for order.paid)
203
+ Rate->>Rate: Check context limit (10/sec per user)
204
+
205
+ alt Rate limit exceeded
206
+ Rate-->>Event: Drop event (return false)
207
+ else Within limits
208
+ Rate->>Audit: process(event_data)
209
+
210
+ alt Is audit event
211
+ Audit->>Audit: Sign with HMAC-SHA256
212
+ Audit->>Audit: Chain to previous
213
+ Audit->>Adapters: route(signed_event)
214
+ else Regular event
215
+ Audit->>Adapters: route(event_data)
216
+ end
217
+ end
218
+ ```
219
+
220
+ ---
221
+
222
+ ## 3. PII Filtering
223
+
224
+ ### 3.0. PII Filtering Strategy
225
+
226
+ **Critical Design Decision:** Explicit opt-in, per-adapter filtering to minimize performance overhead.
227
+
228
+ #### 3.0.1. The Performance Problem
229
+
230
+ **Problem:** Filtering ALL events by default = massive overhead!
231
+
232
+ ```ruby
233
+ # ❌ BAD: Filter every event (expensive!)
234
+ 1,000,000 events/day × 0.2ms filtering = 200 seconds CPU/day = waste!
235
+
236
+ # ✅ GOOD: Filter only PII events
237
+ 100,000 PII events/day × 0.2ms = 20 seconds CPU/day = acceptable
238
+ ```
239
+
240
+ #### 3.0.2. Three-Tier Filtering Strategy
241
+
242
+ E11y uses a **3-tier approach** to balance security and performance:
243
+
244
+ | Tier | Strategy | Cost | Use Case | Events/sec |
245
+ |------|----------|------|----------|------------|
246
+ | **Tier 1** | Skip filtering | 0ms | Health checks, metrics, internal events | 500 |
247
+ | **Tier 2** | Rails filters only | ~0.05ms | Standard events (known PII keys) | 400 |
248
+ | **Tier 3** | Deep filtering | ~0.2ms | User data, payments, complex nested | 100 |
249
+
250
+ **Performance Budget:**
251
+ ```
252
+ 500 events/sec × 0ms = 0ms CPU/sec (Tier 1)
253
+ 400 events/sec × 0.05ms = 20ms CPU/sec (Tier 2)
254
+ 100 events/sec × 0.2ms = 20ms CPU/sec (Tier 3)
255
+ ---
256
+ Total: 40ms CPU/sec = 4% CPU on single core ✅
257
+ ```
258
+
259
+ #### 3.0.3. Explicit PII Declaration
260
+
261
+ **Rule:** Event class MUST declare PII handling explicitly.
262
+
263
+ **Option A: No PII (Skip Filtering)**
264
+ ```ruby
265
+ class Events::HealthCheck < E11y::Event::Base
266
+ schema do
267
+ required(:status).filled(:string)
268
+ end
269
+
270
+ # ✅ Explicit: This event contains NO PII
271
+ contains_pii false # Skip all PII filtering (Tier 1)
272
+ end
273
+ ```
274
+
275
+ **Option B: Default (Rails Filters Only)**
276
+ ```ruby
277
+ class Events::OrderCreated < E11y::Event::Base
278
+ schema do
279
+ required(:order_id).filled(:string)
280
+ required(:amount).filled(:float)
281
+ end
282
+
283
+ # No declaration → Rails filters applied (Tier 2, default)
284
+ # Uses: Rails.application.config.filter_parameters
285
+ end
286
+ ```
287
+
288
+ **Option C: Explicit PII (Deep Filtering)**
289
+ ```ruby
290
+ class Events::UserRegistered < E11y::Event::Base
291
+ schema do
292
+ required(:email).filled(:string)
293
+ required(:password).filled(:string)
294
+ required(:address).filled(:hash)
295
+ required(:user_id).filled(:string)
296
+ end
297
+
298
+ # ✅ Explicit: This event contains PII
299
+ contains_pii true # Tier 3: Deep filtering
300
+
301
+ pii_filtering do
302
+ # ✅ MANDATORY: Explicit declaration for EACH schema field
303
+ # Linter will FAIL if any field is missing!
304
+
305
+ field :email do
306
+ strategy :hash # Pseudonymize for searchability
307
+ exclude_adapters [:file_audit] # Audit needs original (GDPR Art. 6(1)(c))
308
+ end
309
+
310
+ field :password do
311
+ strategy :mask # Full mask everywhere
312
+ # No exclude_adapters → applies to ALL adapters
313
+ end
314
+
315
+ field :address do
316
+ strategy :redact # Remove completely
317
+ exclude_adapters [:file_audit]
318
+ end
319
+
320
+ field :user_id do
321
+ strategy :skip # Not PII, no filtering
322
+ end
323
+
324
+ # ❌ If you forget a field → Linter FAILS at boot:
325
+ # "Missing PII declaration for: [:forgotten_field]"
326
+ end
327
+ end
328
+ ```
329
+
330
+ **Option D: Audit Events (Simplified)**
331
+ ```ruby
332
+ class Events::GdprDeletion < E11y::AuditEvent
333
+ schema do
334
+ required(:user_id).filled(:string)
335
+ required(:email).filled(:string)
336
+ required(:reason).filled(:string)
337
+ required(:deleted_at).filled(:time)
338
+ end
339
+
340
+ contains_pii true
341
+
342
+ pii_filtering do
343
+ # ✅ AuditEvent automatically excludes :file_audit for all fields
344
+
345
+ field :user_id do
346
+ strategy :skip # Not PII
347
+ end
348
+
349
+ field :email do
350
+ strategy :hash # For non-audit adapters (Elasticsearch, Loki)
351
+ # exclude_adapters [:file_audit] ← AUTOMATIC for AuditEvent
352
+ end
353
+
354
+ field :reason do
355
+ strategy :skip # Not PII
356
+ end
357
+
358
+ field :deleted_at do
359
+ strategy :skip # Not PII
360
+ end
361
+ end
362
+ end
363
+
364
+ # Result:
365
+ # - file_audit: { email: 'user@example.com' } # Original (GDPR Art. 6(1)(c))
366
+ # - elasticsearch: { email: 'a1b2c3d4...' } # Hashed
367
+ # - loki: { email: 'a1b2c3d4...' } # Hashed
368
+ # - sentry: { email: 'a1b2c3d4...' } # Hashed
369
+ ```
370
+
371
+ #### 3.0.4. Filtering Strategies
372
+
373
+ | Strategy | Example | Use Case | Reversible? | Cost |
374
+ |----------|---------|----------|-------------|------|
375
+ | **:skip** | `user@example.com` | Audit trail (compliance) | N/A | 0ms |
376
+ | **:mask** | `[FILTERED]` | External services (Sentry) | ❌ No | 0.01ms |
377
+ | **:hash** | `a1b2c3d4e5f6...` | Pseudonymization (searchable) | ❌ No | 0.05ms |
378
+ | **:truncate** | `192.168.x.x` | Partial masking (logs) | ❌ No | 0.02ms |
379
+ | **:redact** | *(removed)* | Complete removal | ❌ No | 0.01ms |
380
+ | **:encrypt** | `encrypted_blob` | Searchable encryption | ✅ Yes (with key) | 0.15ms |
381
+
382
+ #### 3.0.5. PII Declaration Linter
383
+
384
+ **Critical:** Linter ENFORCES explicit declaration for every field when `contains_pii true`.
385
+
386
+ ```ruby
387
+ # lib/e11y/linters/pii_declaration_linter.rb
388
+ module E11y
389
+ module Linters
390
+ class PiiDeclarationLinter
391
+ def self.validate_all!
392
+ E11y::Registry.all_events.each do |event_class|
393
+ validate!(event_class)
394
+ end
395
+ end
396
+
397
+ def self.validate!(event_class)
398
+ return unless event_class.contains_pii?
399
+
400
+ # Get declared fields from pii_filtering block
401
+ declared_fields = event_class.pii_filtering_config&.fields&.keys || []
402
+
403
+ # Get all fields from schema
404
+ schema_fields = event_class.schema.keys
405
+
406
+ # Check for missing declarations
407
+ missing = schema_fields - declared_fields
408
+
409
+ if missing.any?
410
+ raise PiiDeclarationError, build_error_message(event_class, missing)
411
+ end
412
+
413
+ # Validate each field has valid strategy
414
+ declared_fields.each do |field|
415
+ config = event_class.pii_filtering_config.fields[field]
416
+ validate_field_config!(event_class, field, config)
417
+ end
418
+ end
419
+
420
+ private
421
+
422
+ def self.build_error_message(event_class, missing_fields)
423
+ <<~ERROR
424
+ ❌ PII Declaration Error: #{event_class.name}
425
+
426
+ Event declared `contains_pii true` but missing field declarations:
427
+
428
+ Missing fields: #{missing_fields.map(&:inspect).join(', ')}
429
+
430
+ Fix: Add explicit PII strategy for each field in pii_filtering block:
431
+
432
+ class #{event_class.name} < E11y::Event::Base
433
+ contains_pii true
434
+
435
+ pii_filtering do
436
+ #{missing_fields.map { |f| "field :#{f} do\n strategy :mask # or :hash, :skip, :redact\n end" }.join("\n ")}
437
+ end
438
+ end
439
+
440
+ Available strategies: :skip, :mask, :hash, :redact, :truncate, :encrypt
441
+ ERROR
442
+ end
443
+
444
+ def self.validate_field_config!(event_class, field, config)
445
+ valid_strategies = [:skip, :mask, :hash, :redact, :truncate, :encrypt]
446
+
447
+ unless valid_strategies.include?(config[:strategy])
448
+ raise PiiDeclarationError, <<~ERROR
449
+ ❌ Invalid PII strategy for #{event_class.name}##{field}
450
+
451
+ Strategy: #{config[:strategy].inspect}
452
+ Valid strategies: #{valid_strategies.map(&:inspect).join(', ')}
453
+ ERROR
454
+ end
455
+
456
+ # Validate exclude_adapters if present
457
+ if config[:exclude_adapters]
458
+ unless config[:exclude_adapters].is_a?(Array)
459
+ raise PiiDeclarationError, "exclude_adapters must be an array for #{event_class.name}##{field}"
460
+ end
461
+ end
462
+ end
463
+ end
464
+
465
+ class PiiDeclarationError < StandardError; end
466
+ end
467
+ end
468
+ ```
469
+
470
+ **Rake Task:**
471
+ ```ruby
472
+ # lib/tasks/e11y_pii.rake
473
+ namespace :e11y do
474
+ namespace :lint do
475
+ desc 'Validate PII declarations for all events'
476
+ task pii: :environment do
477
+ puts "Checking PII declarations..."
478
+ puts "=" * 80
479
+
480
+ errors = []
481
+ warnings = []
482
+
483
+ E11y::Registry.all_events.each do |event_class|
484
+ begin
485
+ E11y::Linters::PiiDeclarationLinter.validate!(event_class)
486
+
487
+ if event_class.contains_pii?
488
+ puts "✅ #{event_class.name} - All #{event_class.schema.keys.size} fields declared"
489
+ else
490
+ puts "⚪ #{event_class.name} - No PII (skipped)"
491
+ end
492
+ rescue E11y::Linters::PiiDeclarationError => error
493
+ errors << error.message
494
+ puts "❌ #{event_class.name}"
495
+ end
496
+ end
497
+
498
+ puts "=" * 80
499
+
500
+ if errors.any?
501
+ puts "\n❌ ERRORS:\n\n"
502
+ errors.each { |e| puts e; puts "\n" }
503
+ exit 1
504
+ else
505
+ puts "\n✅ All PII declarations are valid"
506
+ exit 0
507
+ end
508
+ end
509
+ end
510
+ end
511
+ ```
512
+
513
+ **Usage:**
514
+ ```bash
515
+ # Validate all events
516
+ $ bundle exec rake e11y:lint:pii
517
+
518
+ Checking PII declarations...
519
+ ================================================================================
520
+ ✅ Events::OrderCreated - All 3 fields declared
521
+ ✅ Events::PaymentProcessed - All 5 fields declared
522
+ ❌ Events::UserRegistered
523
+ ⚪ Events::HealthCheck - No PII (skipped)
524
+ ================================================================================
525
+
526
+ ❌ ERRORS:
527
+
528
+ ❌ PII Declaration Error: Events::UserRegistered
529
+
530
+ Event declared `contains_pii true` but missing field declarations:
531
+
532
+ Missing fields: :ip_address, :user_agent
533
+
534
+ Fix: Add explicit PII strategy for each field in pii_filtering block:
535
+
536
+ class Events::UserRegistered < E11y::Event::Base
537
+ contains_pii true
538
+
539
+ pii_filtering do
540
+ field :ip_address do
541
+ strategy :mask # or :hash, :skip, :redact
542
+ end
543
+ field :user_agent do
544
+ strategy :mask # or :hash, :skip, :redact
545
+ end
546
+ end
547
+ end
548
+
549
+ Available strategies: :skip, :mask, :hash, :redact, :truncate, :encrypt
550
+ ```
551
+
552
+ **Boot-time Validation:**
553
+ ```ruby
554
+ # config/initializers/e11y.rb
555
+ E11y.configure do |config|
556
+ # ... other config ...
557
+
558
+ # Validate PII declarations at boot (in development/test)
559
+ if Rails.env.development? || Rails.env.test?
560
+ config.after_initialize do
561
+ E11y::Linters::PiiDeclarationLinter.validate_all!
562
+ end
563
+ end
564
+ end
565
+ ```
566
+
567
+ #### 3.0.6. Default Behavior
568
+
569
+ **If `contains_pii` not specified:**
570
+ - ✅ Apply Rails filters only (Tier 2)
571
+ - ✅ Fast (0.05ms overhead)
572
+ - ✅ Covers 90% of use cases (password, token, secret, api_key)
573
+ - ⚠️ No linter validation (assumes you know what you're doing)
574
+
575
+ **Recommended approach:**
576
+ ```ruby
577
+ # For most events: don't specify (default is good)
578
+ class Events::OrderPaid < E11y::Event::Base
579
+ # No contains_pii declaration
580
+ # → Rails filters applied automatically
581
+ # → No linter validation
582
+ end
583
+
584
+ # Only for internal/metrics: opt-out
585
+ class Events::HealthCheck < E11y::Event::Base
586
+ contains_pii false # Explicit opt-out
587
+ # → No filtering, no validation
588
+ end
589
+
590
+ # For sensitive data: opt-in + explicit declarations
591
+ class Events::UserLogin < E11y::Event::Base
592
+ contains_pii true # ✅ Triggers linter validation
593
+
594
+ pii_filtering do
595
+ # ✅ MUST declare ALL schema fields
596
+ field :email do
597
+ strategy :hash
598
+ exclude_adapters [:file_audit]
599
+ end
600
+
601
+ field :password do
602
+ strategy :mask
603
+ end
604
+
605
+ field :ip_address do
606
+ strategy :mask
607
+ end
608
+
609
+ field :session_id do
610
+ strategy :skip # Not PII
611
+ end
612
+ end
613
+ end
614
+ ```
615
+
616
+ #### 3.0.6. Critical Design Decisions from Conflict Analysis
617
+
618
+ **From CONFLICT-ANALYSIS.md:**
619
+
620
+ 1. **Conflict #4: Audit Trail + PII Filtering**
621
+ - ✅ Decision: PII filtering happens **per-adapter** (not globally)
622
+ - Rationale: Audit needs original data (GDPR Art. 6(1)(c)), Sentry needs masked
623
+ - Implementation: Each adapter applies its own PII rules
624
+
625
+ 2. **Conflict #3: PII Filtering + OpenTelemetry**
626
+ - ✅ Decision: Per-adapter PII rules
627
+ - OTel: pseudonymize (one-way hash)
628
+ - Audit: skip filtering (compliance)
629
+ - Sentry: strict masking (external service)
630
+
631
+ 3. **Pipeline Order (UPDATED):**
632
+ ```
633
+ 1. Schema Validation
634
+ 2. Context Enrichment
635
+ 3. Rate Limiting (moved before PII for efficiency)
636
+ 4. Adaptive Sampling
637
+ 5. Audit Signing (BEFORE filtering, preserves original signature)
638
+ 6. Adapter Routing
639
+ └─→ PII Filtering (per-adapter, different rules per destination)
640
+ ```
641
+
642
+ **Key Insight:** PII filtering is **NOT a global middleware** — it's applied **inside each adapter** with different rules!
643
+
644
+ ### 3.1. Rails Integration
645
+
646
+ **Design Decision:** Use `Rails.application.config.filter_parameters` as base.
647
+
648
+ ```ruby
649
+ module E11y
650
+ module Security
651
+ class PiiFilter
652
+ def initialize(config)
653
+ @rails_filters = extract_rails_filters
654
+ @custom_patterns = config.patterns || []
655
+ @custom_filters = config.custom_filters || []
656
+ @allowlist = config.allowlist || []
657
+ @deep_scan = config.deep_scan || true
658
+ end
659
+
660
+ def filter(event_data)
661
+ filtered_data = event_data.dup
662
+
663
+ # Apply Rails filters
664
+ filtered_data[:payload] = apply_rails_filters(filtered_data[:payload])
665
+
666
+ # Apply custom patterns
667
+ filtered_data[:payload] = apply_pattern_filters(filtered_data[:payload])
668
+
669
+ # Apply custom filter functions
670
+ filtered_data[:payload] = apply_custom_filters(filtered_data[:payload])
671
+
672
+ # Deep scan if enabled
673
+ if @deep_scan
674
+ filtered_data[:payload] = deep_scan(filtered_data[:payload])
675
+ end
676
+
677
+ filtered_data
678
+ end
679
+
680
+ private
681
+
682
+ def extract_rails_filters
683
+ return [] unless defined?(Rails)
684
+
685
+ Rails.application.config.filter_parameters || []
686
+ end
687
+
688
+ def apply_rails_filters(payload)
689
+ return payload if @rails_filters.empty?
690
+
691
+ # Use Rails' parameter filter
692
+ filter = ActiveSupport::ParameterFilter.new(@rails_filters)
693
+ filter.filter(payload)
694
+ end
695
+ end
696
+ end
697
+ end
698
+ ```
699
+
700
+ **Example:**
701
+
702
+ ```ruby
703
+ # config/initializers/filter_parameters.rb
704
+ Rails.application.config.filter_parameters += [
705
+ :password,
706
+ :password_confirmation,
707
+ :secret,
708
+ :token,
709
+ :api_key,
710
+ :credit_card_number,
711
+ :ssn
712
+ ]
713
+
714
+ # E11y automatically uses these filters!
715
+ Events::UserLogin.track(
716
+ email: 'user@example.com',
717
+ password: 'secret123' # ← Automatically filtered by Rails
718
+ )
719
+
720
+ # Result in logs:
721
+ # { email: 'user@example.com', password: '[FILTERED]' }
722
+ ```
723
+
724
+ ---
725
+
726
+ ### 3.2. Pattern-Based Filtering
727
+
728
+ **Design Decision:** Regex patterns for content scanning.
729
+
730
+ ```ruby
731
+ module E11y
732
+ module Security
733
+ class PatternMatcher
734
+ BUILT_IN_PATTERNS = {
735
+ email: /\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b/,
736
+ credit_card: /\b(?:\d{4}[-\s]?){3}\d{4}\b/,
737
+ ssn: /\b\d{3}-\d{2}-\d{4}\b/,
738
+ phone: /\b\d{3}[-.]?\d{3}[-.]?\d{4}\b/,
739
+ ip_address: /\b(?:\d{1,3}\.){3}\d{1,3}\b/,
740
+ jwt: /\beyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\b/,
741
+ api_key: /\b[A-Za-z0-9]{32,}\b/
742
+ }.freeze
743
+
744
+ def initialize(enabled_patterns: BUILT_IN_PATTERNS.keys, custom_patterns: {})
745
+ @patterns = BUILT_IN_PATTERNS.slice(*enabled_patterns)
746
+ @patterns.merge!(custom_patterns)
747
+ end
748
+
749
+ def filter(value)
750
+ return value unless value.is_a?(String)
751
+
752
+ filtered_value = value.dup
753
+
754
+ @patterns.each do |name, pattern|
755
+ filtered_value.gsub!(pattern, "[FILTERED:#{name.upcase}]")
756
+ end
757
+
758
+ filtered_value
759
+ end
760
+
761
+ def contains_pii?(value)
762
+ return false unless value.is_a?(String)
763
+
764
+ @patterns.any? { |_, pattern| value.match?(pattern) }
765
+ end
766
+ end
767
+ end
768
+ end
769
+ ```
770
+
771
+ **Example:**
772
+
773
+ ```ruby
774
+ E11y.configure do |config|
775
+ config.security.pii_filtering do
776
+ # Enable built-in patterns
777
+ patterns [:email, :credit_card, :ssn, :phone, :ip_address]
778
+
779
+ # Add custom patterns
780
+ custom_pattern :internal_id, /\bINT-\d{6,}\b/
781
+ custom_pattern :license_plate, /\b[A-Z]{3}-\d{4}\b/
782
+ end
783
+ end
784
+
785
+ # Usage:
786
+ Events::CustomerSupport.track(
787
+ message: "Customer email is user@example.com, IP: 192.168.1.1"
788
+ )
789
+
790
+ # Result:
791
+ # {
792
+ # message: "Customer email is [FILTERED:EMAIL], IP: [FILTERED:IP_ADDRESS]"
793
+ # }
794
+ ```
795
+
796
+ ---
797
+
798
+ ### 3.3. Deep Scanning
799
+
800
+ **Design Decision:** Recursively scan nested hashes and arrays.
801
+
802
+ ```ruby
803
+ module E11y
804
+ module Security
805
+ class DeepScanner
806
+ def initialize(pattern_matcher, allowlist: [])
807
+ @pattern_matcher = pattern_matcher
808
+ @allowlist = allowlist
809
+ end
810
+
811
+ def scan(value, key_path: [])
812
+ case value
813
+ when Hash
814
+ scan_hash(value, key_path)
815
+ when Array
816
+ scan_array(value, key_path)
817
+ when String
818
+ scan_string(value, key_path)
819
+ else
820
+ value
821
+ end
822
+ end
823
+
824
+ private
825
+
826
+ def scan_hash(hash, key_path)
827
+ hash.transform_values do |v|
828
+ new_key_path = key_path + [hash.keys.find { |k| hash[k] == v }]
829
+
830
+ # Skip if in allowlist
831
+ next v if allowlisted?(new_key_path)
832
+
833
+ scan(v, key_path: new_key_path)
834
+ end
835
+ end
836
+
837
+ def scan_array(array, key_path)
838
+ array.map.with_index do |item, index|
839
+ scan(item, key_path: key_path + [index])
840
+ end
841
+ end
842
+
843
+ def scan_string(string, key_path)
844
+ return string if allowlisted?(key_path)
845
+
846
+ @pattern_matcher.filter(string)
847
+ end
848
+
849
+ def allowlisted?(key_path)
850
+ key_path_string = key_path.join('.')
851
+
852
+ @allowlist.any? do |pattern|
853
+ case pattern
854
+ when String
855
+ key_path_string == pattern
856
+ when Regexp
857
+ key_path_string.match?(pattern)
858
+ else
859
+ false
860
+ end
861
+ end
862
+ end
863
+ end
864
+ end
865
+ end
866
+ ```
867
+
868
+ **Example:**
869
+
870
+ ```ruby
871
+ E11y.configure do |config|
872
+ config.security.pii_filtering do
873
+ deep_scan true
874
+
875
+ # Allowlist certain paths
876
+ allowlist [
877
+ 'order.id', # Exact match
878
+ /^metadata\.internal_/, # Regex match
879
+ 'debug.raw_request' # Development debugging
880
+ ]
881
+ end
882
+ end
883
+
884
+ # Deep nested structure:
885
+ Events::OrderCreated.track(
886
+ order: {
887
+ id: 'ORD-123',
888
+ customer: {
889
+ email: 'user@example.com', # ← Will be filtered
890
+ phone: '555-1234' # ← Will be filtered
891
+ },
892
+ metadata: {
893
+ internal_ref: 'INT-999999', # ← Allowlisted, kept
894
+ note: 'Contact at user@example.com' # ← Will be filtered
895
+ }
896
+ }
897
+ )
898
+
899
+ # Result:
900
+ # {
901
+ # order: {
902
+ # id: 'ORD-123',
903
+ # customer: {
904
+ # email: '[FILTERED:EMAIL]',
905
+ # phone: '[FILTERED:PHONE]'
906
+ # },
907
+ # metadata: {
908
+ # internal_ref: 'INT-999999', # Kept
909
+ # note: 'Contact at [FILTERED:EMAIL]'
910
+ # }
911
+ # }
912
+ # }
913
+ ```
914
+
915
+ ---
916
+
917
+ ### 3.4. Per-Adapter Rules (Architecture)
918
+
919
+ **Critical Design Decision:** PII filtering happens **inside each adapter**, not globally.
920
+
921
+ #### 3.4.1. Why Per-Adapter Filtering?
922
+
923
+ **Problem with Global Filtering:**
924
+ ```ruby
925
+ # ❌ BAD: Filter once globally
926
+ event_data = { email: 'user@example.com', ip: '192.168.1.1' }
927
+ filtered_event = global_pii_filter(event_data)
928
+ # → { email: '[FILTERED]', ip: '[FILTERED]' }
929
+
930
+ # Now ALL adapters get filtered data
931
+ send_to_audit_file(filtered_event) # ❌ Audit needs ORIGINAL email!
932
+ send_to_sentry(filtered_event) # ✅ Sentry gets filtered (correct)
933
+ send_to_loki(filtered_event) # ❌ Loki could have kept hashed email
934
+ ```
935
+
936
+ **Solution: Filter Per-Adapter:**
937
+ ```ruby
938
+ # ✅ GOOD: Each adapter applies its own rules
939
+ adapters.each do |adapter|
940
+ # Adapter decides filtering strategy
941
+ adapter_filtered_event = adapter.apply_pii_rules(event_data)
942
+ adapter.write(adapter_filtered_event)
943
+ end
944
+
945
+ # Result:
946
+ # - Audit File: { email: 'user@example.com', ip: '192.168.1.1' } (original)
947
+ # - Sentry: { email: '[FILTERED]', ip: '[FILTERED]' } (masked)
948
+ # - Loki: { email: 'a1b2c3d4...', ip: '192.168.x.x' } (hashed/truncated)
949
+ # - Elasticsearch: { email: 'a1b2c3d4...', ip: '192.168.1.1' } (email hashed only)
950
+ ```
951
+
952
+ **Architecture Diagram:**
953
+
954
+ ```mermaid
955
+ graph TB
956
+ Event[Event Data<br/>email: user@x.com<br/>ip: 192.168.1.1] --> Router[Adapter Router]
957
+
958
+ Router --> Audit[Audit Adapter]
959
+ Router --> Sentry[Sentry Adapter]
960
+ Router --> Loki[Loki Adapter]
961
+ Router --> ES[Elasticsearch Adapter]
962
+
963
+ subgraph "Per-Adapter PII Filtering"
964
+ Audit --> AuditFilter[PII Filter: SKIP<br/>mode: :skip]
965
+ Sentry --> SentryFilter[PII Filter: MASK<br/>mode: :mask]
966
+ Loki --> LokiFilter[PII Filter: CUSTOM<br/>mode: :custom]
967
+ ES --> ESFilter[PII Filter: HASH<br/>mode: :hash]
968
+ end
969
+
970
+ AuditFilter --> AuditStore[Audit File<br/>email: user@x.com<br/>ip: 192.168.1.1]
971
+ SentryFilter --> SentryAPI[Sentry<br/>email: [FILTERED]<br/>ip: [FILTERED]]
972
+ LokiFilter --> LokiAPI[Loki<br/>email: [FILTERED]<br/>ip: 192.168.x.x]
973
+ ESFilter --> ESAPI[Elasticsearch<br/>email: a1b2c3d4...<br/>ip: 192.168.1.1]
974
+
975
+ style AuditFilter fill:#d4edda
976
+ style SentryFilter fill:#f8d7da
977
+ style LokiFilter fill:#fff3cd
978
+ style ESFilter fill:#d1ecf1
979
+ ```
980
+
981
+ #### 3.4.2. Implementation
982
+
983
+ ```ruby
984
+ module E11y
985
+ module Adapters
986
+ class Base
987
+ def write(event_data)
988
+ # Apply adapter-specific PII filtering
989
+ filtered_data = apply_pii_rules(event_data)
990
+
991
+ # Write filtered data
992
+ write_impl(filtered_data)
993
+ end
994
+
995
+ private
996
+
997
+ def apply_pii_rules(event_data)
998
+ # Get event class
999
+ event_class = E11y::Registry.get_class(event_data[:event_name])
1000
+
1001
+ # Check if event has per-adapter rules
1002
+ return event_data unless event_class.respond_to?(:pii_filtering_for_adapter)
1003
+
1004
+ # Get rules for THIS adapter
1005
+ rules = event_class.pii_filtering_for_adapter(adapter_name)
1006
+
1007
+ # Apply filtering based on rules
1008
+ case rules[:strategy]
1009
+ when :skip
1010
+ event_data # No filtering
1011
+ when :mask
1012
+ mask_pii_fields(event_data, rules[:fields])
1013
+ when :hash
1014
+ hash_pii_fields(event_data, rules[:fields])
1015
+ when :custom
1016
+ rules[:filter].call(event_data)
1017
+ else
1018
+ # Fallback to default Rails filters
1019
+ apply_rails_filters(event_data)
1020
+ end
1021
+ end
1022
+
1023
+ def mask_pii_fields(event_data, fields)
1024
+ filtered = event_data.dup
1025
+ fields.each do |field|
1026
+ deep_mask(filtered[:payload], field)
1027
+ end
1028
+ filtered
1029
+ end
1030
+
1031
+ def hash_pii_fields(event_data, fields)
1032
+ filtered = event_data.dup
1033
+ fields.each do |field|
1034
+ deep_hash(filtered[:payload], field)
1035
+ end
1036
+ filtered
1037
+ end
1038
+ end
1039
+ end
1040
+ end
1041
+ ```
1042
+
1043
+ #### 3.4.3. Event Class Declaration (Simplified)
1044
+
1045
+ ```ruby
1046
+ class Events::UserLogin < E11y::Event::Base
1047
+ schema do
1048
+ required(:email).filled(:string)
1049
+ required(:ip_address).filled(:string)
1050
+ required(:session_id).filled(:string)
1051
+ end
1052
+
1053
+ contains_pii true
1054
+
1055
+ pii_filtering do
1056
+ # ✅ Simple per-field declaration
1057
+ field :email do
1058
+ strategy :hash # Apply to all adapters
1059
+ exclude_adapters [:file_audit] # Except audit (needs original)
1060
+ hash_algorithm 'SHA256' # Optional: specify algorithm
1061
+ end
1062
+
1063
+ field :ip_address do
1064
+ strategy :mask # Mask for all adapters
1065
+ exclude_adapters [:file_audit]
1066
+ end
1067
+
1068
+ field :session_id do
1069
+ strategy :skip # Not PII, no filtering
1070
+ end
1071
+ end
1072
+ end
1073
+
1074
+ # Result across adapters:
1075
+ #
1076
+ # file_audit: { email: 'user@example.com', ip: '192.168.1.1', session_id: 's123' }
1077
+ # elasticsearch: { email: 'a1b2c3d4...', ip: '[FILTERED]', session_id: 's123' }
1078
+ # loki: { email: 'a1b2c3d4...', ip: '[FILTERED]', session_id: 's123' }
1079
+ # sentry: { email: 'a1b2c3d4...', ip: '[FILTERED]', session_id: 's123' }
1080
+ ```
1081
+
1082
+ **For complex per-adapter rules (advanced use case):**
1083
+ ```ruby
1084
+ class Events::UserLogin < E11y::Event::Base
1085
+ contains_pii true
1086
+
1087
+ pii_filtering do
1088
+ field :ip_address do
1089
+ # Different strategy per adapter (advanced)
1090
+ strategy :mask # Default for most adapters
1091
+
1092
+ # Custom strategy for specific adapter
1093
+ custom_for_adapter :loki do
1094
+ # Truncate IP instead of full mask
1095
+ ->(value) { value.split('.')[0..2].join('.') + '.x' }
1096
+ end
1097
+
1098
+ exclude_adapters [:file_audit]
1099
+ end
1100
+ end
1101
+ end
1102
+
1103
+ # Result:
1104
+ # loki: { ip: '192.168.1.x' } # Custom truncate
1105
+ # sentry: { ip: '[FILTERED]' } # Default mask
1106
+ # elasticsearch: { ip: '[FILTERED]' } # Default mask
1107
+ ```
1108
+
1109
+ #### 3.4.4. Configuration API (Rails-Style DSL)
1110
+
1111
+ ```ruby
1112
+ module E11y
1113
+ module Event
1114
+ class Base
1115
+ class << self
1116
+ # DSL for PII filtering (Rails-style)
1117
+ def pii_filtering(&block)
1118
+ @pii_filtering_config = PiiFilteringConfig.new(self)
1119
+ @pii_filtering_config.instance_eval(&block)
1120
+ end
1121
+
1122
+ def pii_filtering_config
1123
+ @pii_filtering_config
1124
+ end
1125
+
1126
+ # Get PII rules for specific field and adapter
1127
+ def pii_rule_for_field(field_name, adapter_name)
1128
+ return nil unless @pii_filtering_config
1129
+
1130
+ field_config = @pii_filtering_config.fields[field_name]
1131
+ return nil unless field_config
1132
+
1133
+ # Check if this adapter is excluded
1134
+ if field_config[:exclude_adapters]&.include?(adapter_name)
1135
+ return { strategy: :skip }
1136
+ end
1137
+
1138
+ # Check if only specific adapters allowed
1139
+ if field_config[:only_adapters] && !field_config[:only_adapters].include?(adapter_name)
1140
+ return { strategy: :skip }
1141
+ end
1142
+
1143
+ # Check for custom adapter-specific rule
1144
+ if field_config[:custom_adapters] && field_config[:custom_adapters][adapter_name]
1145
+ return {
1146
+ strategy: :custom,
1147
+ filter: field_config[:custom_adapters][adapter_name]
1148
+ }
1149
+ end
1150
+
1151
+ # Return default strategy for this field
1152
+ {
1153
+ strategy: field_config[:strategy],
1154
+ hash_algorithm: field_config[:hash_algorithm],
1155
+ truncate_to: field_config[:truncate_to],
1156
+ custom_filter: field_config[:custom_filter]
1157
+ }.compact
1158
+ end
1159
+ end
1160
+ end
1161
+
1162
+ class PiiFilteringConfig
1163
+ attr_reader :fields
1164
+
1165
+ def initialize(event_class)
1166
+ @event_class = event_class
1167
+ @fields = {}
1168
+ end
1169
+
1170
+ # ===================================================================
1171
+ # RAILS-STYLE DSL: Single-line shortcuts
1172
+ # ===================================================================
1173
+
1174
+ # Shortcut: masks(*fields)
1175
+ # Like validates :name, :email, presence: true
1176
+ def masks(*field_names, **options)
1177
+ field_names.each do |field_name|
1178
+ field(field_name) do
1179
+ strategy :mask
1180
+ apply_options(options)
1181
+ end
1182
+ end
1183
+ end
1184
+
1185
+ # Shortcut: hashes(*fields)
1186
+ def hashes(*field_names, **options)
1187
+ field_names.each do |field_name|
1188
+ field(field_name) do
1189
+ strategy :hash
1190
+ apply_options(options)
1191
+ end
1192
+ end
1193
+ end
1194
+
1195
+ # Shortcut: redacts(*fields)
1196
+ def redacts(*field_names, **options)
1197
+ field_names.each do |field_name|
1198
+ field(field_name) do
1199
+ strategy :redact
1200
+ apply_options(options)
1201
+ end
1202
+ end
1203
+ end
1204
+
1205
+ # Shortcut: skips(*fields) - not PII
1206
+ def skips(*field_names, **options)
1207
+ field_names.each do |field_name|
1208
+ field(field_name) do
1209
+ strategy :skip
1210
+ apply_options(options)
1211
+ end
1212
+ end
1213
+ end
1214
+
1215
+ # Shortcut: truncates(*fields)
1216
+ def truncates(*field_names, to:, **options)
1217
+ field_names.each do |field_name|
1218
+ field(field_name) do
1219
+ strategy :truncate
1220
+ truncate_to to
1221
+ apply_options(options)
1222
+ end
1223
+ end
1224
+ end
1225
+
1226
+ # ===================================================================
1227
+ # CONDITIONAL FILTERING (Rails-style)
1228
+ # ===================================================================
1229
+
1230
+ # Like validates :admin, presence: true, if: :admin?
1231
+ def masks_if(condition, *field_names, **options)
1232
+ field_names.each do |field_name|
1233
+ field(field_name) do
1234
+ strategy :mask
1235
+ only_if condition
1236
+ apply_options(options)
1237
+ end
1238
+ end
1239
+ end
1240
+
1241
+ def hashes_if(condition, *field_names, **options)
1242
+ field_names.each do |field_name|
1243
+ field(field_name) do
1244
+ strategy :hash
1245
+ only_if condition
1246
+ apply_options(options)
1247
+ end
1248
+ end
1249
+ end
1250
+
1251
+ # ===================================================================
1252
+ # GROUPING (Rails-style)
1253
+ # ===================================================================
1254
+
1255
+ # Like with_options model: Post do ... end
1256
+ def with_strategy(strategy, &block)
1257
+ @current_strategy = strategy
1258
+ instance_eval(&block)
1259
+ @current_strategy = nil
1260
+ end
1261
+
1262
+ def with_options(**options, &block)
1263
+ @current_options = options
1264
+ instance_eval(&block)
1265
+ @current_options = nil
1266
+ end
1267
+
1268
+ # ===================================================================
1269
+ # FIELD DECLARATION
1270
+ # ===================================================================
1271
+
1272
+ def field(name, strategy: nil, **options, &block)
1273
+ config = FieldConfig.new(name, @event_class)
1274
+
1275
+ # Apply current context (from with_strategy/with_options)
1276
+ config.strategy(@current_strategy) if @current_strategy
1277
+ config.apply_options(@current_options) if @current_options
1278
+
1279
+ # Apply inline options
1280
+ config.strategy(strategy) if strategy
1281
+ config.apply_options(options)
1282
+
1283
+ # Apply block if provided
1284
+ config.instance_eval(&block) if block_given?
1285
+
1286
+ @fields[name] = config.to_h
1287
+ end
1288
+
1289
+ # ===================================================================
1290
+ # BULK OPERATIONS (Rails-style)
1291
+ # ===================================================================
1292
+
1293
+ # Like validates_presence_of :name, :email
1294
+ def masks_all_except(*exceptions)
1295
+ schema_fields = @event_class.schema.keys
1296
+ (schema_fields - exceptions).each do |field_name|
1297
+ masks field_name
1298
+ end
1299
+ end
1300
+
1301
+ def hashes_all_except(*exceptions, **options)
1302
+ schema_fields = @event_class.schema.keys
1303
+ (schema_fields - exceptions).each do |field_name|
1304
+ hashes field_name, **options
1305
+ end
1306
+ end
1307
+ end
1308
+
1309
+ class FieldConfig
1310
+ attr_reader :field_name
1311
+
1312
+ def initialize(field_name, event_class)
1313
+ @field_name = field_name
1314
+ @event_class = event_class
1315
+ @config = {}
1316
+ end
1317
+
1318
+ # ===================================================================
1319
+ # CORE METHODS
1320
+ # ===================================================================
1321
+
1322
+ def strategy(value)
1323
+ @config[:strategy] = value
1324
+ end
1325
+
1326
+ def exclude_adapters(adapters)
1327
+ @config[:exclude_adapters] = Array(adapters)
1328
+ end
1329
+
1330
+ def only_adapters(adapters)
1331
+ @config[:only_adapters] = Array(adapters)
1332
+ end
1333
+
1334
+ def hash_algorithm(algo)
1335
+ @config[:hash_algorithm] = algo
1336
+ end
1337
+
1338
+ def truncate_to(length)
1339
+ @config[:truncate_to] = length
1340
+ end
1341
+
1342
+ def custom_for_adapter(adapter_name, &block)
1343
+ @config[:custom_adapters] ||= {}
1344
+ @config[:custom_adapters][adapter_name] = block
1345
+ end
1346
+
1347
+ def custom_filter(&block)
1348
+ @config[:custom_filter] = block
1349
+ end
1350
+
1351
+ # ===================================================================
1352
+ # CONDITIONAL FILTERING (Rails-style)
1353
+ # ===================================================================
1354
+
1355
+ def only_if(condition)
1356
+ @config[:only_if] = condition
1357
+ end
1358
+
1359
+ def unless(condition)
1360
+ @config[:unless] = condition
1361
+ end
1362
+
1363
+ # ===================================================================
1364
+ # SHORTCUTS (Rails-style chainable)
1365
+ # ===================================================================
1366
+
1367
+ def except(*adapters)
1368
+ exclude_adapters(adapters)
1369
+ end
1370
+
1371
+ def only(*adapters)
1372
+ only_adapters(adapters)
1373
+ end
1374
+
1375
+ def with_algorithm(algo)
1376
+ hash_algorithm(algo)
1377
+ end
1378
+
1379
+ # ===================================================================
1380
+ # OPTIONS HELPERS
1381
+ # ===================================================================
1382
+
1383
+ def apply_options(options)
1384
+ return unless options
1385
+
1386
+ exclude_adapters(options[:except]) if options[:except]
1387
+ only_adapters(options[:only]) if options[:only]
1388
+ hash_algorithm(options[:algorithm]) if options[:algorithm]
1389
+ truncate_to(options[:to]) if options[:to]
1390
+ only_if(options[:if]) if options[:if]
1391
+ self.unless(options[:unless]) if options[:unless]
1392
+ end
1393
+
1394
+ def to_h
1395
+ @config
1396
+ end
1397
+ end
1398
+ end
1399
+ end
1400
+
1401
+ # AuditEvent: Automatically exclude :file_audit for all fields
1402
+ module E11y
1403
+ class AuditEvent < Event::Base
1404
+ class << self
1405
+ def pii_rule_for_field(field_name, adapter_name)
1406
+ rule = super
1407
+
1408
+ # Override: AuditEvent always skips filtering for :file_audit
1409
+ if adapter_name == :file_audit
1410
+ return { strategy: :skip }
1411
+ end
1412
+
1413
+ rule
1414
+ end
1415
+ end
1416
+ end
1417
+ end
1418
+ ```
1419
+
1420
+ **Configuration:**
1421
+
1422
+ ```ruby
1423
+ E11y.configure do |config|
1424
+ config.security.pii_filtering do
1425
+ # Global rules (apply to all adapters by default)
1426
+ patterns [:email, :phone, :ssn]
1427
+
1428
+ # Per-adapter overrides
1429
+ adapter_rules do
1430
+ # Audit logs: keep original data (internal only)
1431
+ adapter :file_audit, mode: :skip
1432
+
1433
+ # Sentry: extra strict filtering
1434
+ adapter :sentry, mode: :strict, additional_patterns: {
1435
+ user_agent: /Mozilla.*\(.*/,
1436
+ url_params: /\?.*$/
1437
+ }
1438
+
1439
+ # Loki: custom filter
1440
+ adapter :loki, mode: :custom, filter: ->(event_data) {
1441
+ # Keep emails for internal Loki, but mask credit cards
1442
+ filtered = event_data.dup
1443
+ filtered[:payload] = mask_credit_cards(event_data[:payload])
1444
+ filtered
1445
+ }
1446
+ end
1447
+ end
1448
+ end
1449
+ ```
1450
+
1451
+ **Use Case Diagram:**
1452
+
1453
+ ```mermaid
1454
+ graph TB
1455
+ Event[Event Data] --> Router[Adapter Router]
1456
+
1457
+ Router --> Audit[Audit Adapter]
1458
+ Router --> Sentry[Sentry Adapter]
1459
+ Router --> Loki[Loki Adapter]
1460
+
1461
+ subgraph "PII Filtering"
1462
+ Audit --> Skip[Skip Filtering<br/>mode: :skip]
1463
+ Sentry --> Strict[Strict Filtering<br/>mode: :strict]
1464
+ Loki --> Custom[Custom Filter<br/>mode: :custom]
1465
+ end
1466
+
1467
+ Skip --> AuditStore[Full Data<br/>email: user@x.com]
1468
+ Strict --> SentryAPI[Heavily Filtered<br/>email: [FILTERED]<br/>user_agent: [FILTERED]]
1469
+ Custom --> LokiAPI[Custom Filtered<br/>email: user@x.com<br/>cc: [FILTERED]]
1470
+
1471
+ style Skip fill:#d4edda
1472
+ style Strict fill:#f8d7da
1473
+ style Custom fill:#fff3cd
1474
+ ```
1475
+
1476
+ ---
1477
+
1478
+ ### 3.5. Sampling for Debug
1479
+
1480
+ **Design Decision:** Sample filtered values for debugging false positives.
1481
+
1482
+ ```ruby
1483
+ module E11y
1484
+ module Security
1485
+ class FilterSampler
1486
+ def initialize(sample_rate: 0.01, max_samples: 100)
1487
+ @sample_rate = sample_rate
1488
+ @max_samples = max_samples
1489
+ @samples = []
1490
+ @mutex = Mutex.new
1491
+ end
1492
+
1493
+ def record_filter(key_path, original_value, filtered_value)
1494
+ return unless should_sample?
1495
+
1496
+ @mutex.synchronize do
1497
+ return if @samples.size >= @max_samples
1498
+
1499
+ @samples << {
1500
+ key_path: key_path.join('.'),
1501
+ original: truncate(original_value),
1502
+ filtered: filtered_value,
1503
+ timestamp: Time.now
1504
+ }
1505
+ end
1506
+ end
1507
+
1508
+ def samples
1509
+ @mutex.synchronize { @samples.dup }
1510
+ end
1511
+
1512
+ def clear!
1513
+ @mutex.synchronize { @samples.clear }
1514
+ end
1515
+
1516
+ private
1517
+
1518
+ def should_sample?
1519
+ rand < @sample_rate
1520
+ end
1521
+
1522
+ def truncate(value, max_length: 100)
1523
+ str = value.to_s
1524
+ str.length > max_length ? "#{str[0...max_length]}..." : str
1525
+ end
1526
+ end
1527
+ end
1528
+ end
1529
+ ```
1530
+
1531
+ **Usage:**
1532
+
1533
+ ```ruby
1534
+ # Check sampled filters in console
1535
+ E11y.security.filter_sampler.samples
1536
+ # [
1537
+ # {
1538
+ # key_path: 'customer.email',
1539
+ # original: 'user@example.com',
1540
+ # filtered: '[FILTERED:EMAIL]',
1541
+ # timestamp: 2026-01-12 10:30:00 UTC
1542
+ # },
1543
+ # ...
1544
+ # ]
1545
+
1546
+ # Clear samples
1547
+ E11y.security.filter_sampler.clear!
1548
+ ```
1549
+
1550
+ #### 3.4.5. Rails-Style DSL Examples
1551
+
1552
+ **Pattern 1: Single-line shortcuts (like Rails validations)**
1553
+
1554
+ ```ruby
1555
+ class Events::UserRegistered < E11y::Event::Base
1556
+ schema do
1557
+ required(:email).filled(:string)
1558
+ required(:password).filled(:string)
1559
+ required(:ip_address).filled(:string)
1560
+ required(:user_id).filled(:string)
1561
+ end
1562
+
1563
+ contains_pii true
1564
+
1565
+ pii_filtering do
1566
+ # ✅ Rails-style: masks :password, :ip_address
1567
+ masks :password, :ip_address
1568
+
1569
+ # ✅ Rails-style: hashes :email, except: [:file_audit]
1570
+ hashes :email, except: [:file_audit]
1571
+
1572
+ # ✅ Not PII
1573
+ skips :user_id
1574
+ end
1575
+ end
1576
+ ```
1577
+
1578
+ **Pattern 2: with_options (grouping)**
1579
+
1580
+ ```ruby
1581
+ class Events::PaymentProcessed < E11y::Event::Base
1582
+ contains_pii true
1583
+
1584
+ pii_filtering do
1585
+ # ✅ Like Rails: with_options validatable: true do
1586
+ with_options except: [:file_audit] do
1587
+ hashes :email, :user_id
1588
+ masks :ip_address
1589
+ end
1590
+
1591
+ # Always mask (no exceptions)
1592
+ masks :credit_card_last4, :cvv
1593
+
1594
+ skips :payment_id, :amount
1595
+ end
1596
+ end
1597
+ ```
1598
+
1599
+ **Pattern 3: with_strategy (bulk)**
1600
+
1601
+ ```ruby
1602
+ class Events::GdprExport < E11y::Event::Base
1603
+ contains_pii true
1604
+
1605
+ pii_filtering do
1606
+ # ✅ All hashed except audit
1607
+ with_strategy :hash do
1608
+ with_options except: [:file_audit] do
1609
+ field :email
1610
+ field :phone_number
1611
+ field :address
1612
+ end
1613
+ end
1614
+
1615
+ # Sensitive data → always mask
1616
+ with_strategy :mask do
1617
+ field :ssn
1618
+ field :credit_card
1619
+ end
1620
+
1621
+ skips :user_id, :export_id
1622
+ end
1623
+ end
1624
+ ```
1625
+
1626
+ **Pattern 4: Conditional filtering (if/unless)**
1627
+
1628
+ ```ruby
1629
+ class Events::AdminAction < E11y::Event::Base
1630
+ contains_pii true
1631
+
1632
+ pii_filtering do
1633
+ # ✅ Like Rails: validates :admin_notes, presence: true, if: :admin?
1634
+ masks_if ->(payload) { payload[:level] == 'admin' }, :admin_notes
1635
+
1636
+ # Hash user data unless it's a system user
1637
+ hashes_if ->(payload) { !payload[:is_system] }, :email, :ip_address
1638
+
1639
+ skips :action_type, :timestamp
1640
+ end
1641
+ end
1642
+ ```
1643
+
1644
+ **Pattern 5: Custom per-adapter (advanced)**
1645
+
1646
+ ```ruby
1647
+ class Events::UserLogin < E11y::Event::Base
1648
+ contains_pii true
1649
+
1650
+ pii_filtering do
1651
+ # Default: hash
1652
+ hashes :email, except: [:file_audit]
1653
+
1654
+ # Custom for Loki: truncate IP
1655
+ field :ip_address do
1656
+ strategy :mask # Default
1657
+ except :file_audit
1658
+
1659
+ # ✅ Custom filter for specific adapter
1660
+ custom_for_adapter :loki do |value|
1661
+ value.split('.')[0..2].join('.') + '.x'
1662
+ end
1663
+ end
1664
+
1665
+ masks :password
1666
+ skips :session_id
1667
+ end
1668
+ end
1669
+ ```
1670
+
1671
+ **Pattern 6: Bulk operations**
1672
+
1673
+ ```ruby
1674
+ class Events::UserProfileUpdated < E11y::Event::Base
1675
+ schema do
1676
+ required(:user_id).filled(:string)
1677
+ required(:email).filled(:string)
1678
+ required(:phone).filled(:string)
1679
+ required(:address).filled(:hash)
1680
+ required(:bio).filled(:string)
1681
+ required(:avatar_url).filled(:string)
1682
+ end
1683
+
1684
+ contains_pii true
1685
+
1686
+ pii_filtering do
1687
+ # ✅ Like Rails: validates_presence_of :all, except: [:id]
1688
+ # Hash everything except user_id and avatar_url
1689
+ hashes_all_except :user_id, :avatar_url, except: [:file_audit]
1690
+ end
1691
+ end
1692
+ ```
1693
+
1694
+ **Pattern 7: Chainable methods**
1695
+
1696
+ ```ruby
1697
+ class Events::PaymentAttempt < E11y::Event::Base
1698
+ contains_pii true
1699
+
1700
+ pii_filtering do
1701
+ # ✅ Chainable (Rails-style)
1702
+ field :email do
1703
+ strategy :hash
1704
+ except :file_audit
1705
+ with_algorithm 'SHA256'
1706
+ end
1707
+
1708
+ field :ip_address do
1709
+ strategy :truncate
1710
+ truncate_to 10 # "192.168.1.xxx" → "192.168.1."
1711
+ except :file_audit
1712
+ end
1713
+
1714
+ masks :card_number
1715
+ skips :payment_id
1716
+ end
1717
+ end
1718
+ ```
1719
+
1720
+ **Pattern 8: Smart defaults with overrides**
1721
+
1722
+ ```ruby
1723
+ class Events::SensitiveDataAccess < E11y::Event::Base
1724
+ contains_pii true
1725
+
1726
+ pii_filtering do
1727
+ # Default strategy for most fields
1728
+ with_strategy :hash do
1729
+ with_options algorithm: 'SHA256', except: [:file_audit] do
1730
+ field :accessed_by_email
1731
+ field :accessed_record_id
1732
+ field :ip_address
1733
+ end
1734
+ end
1735
+
1736
+ # Override for ultra-sensitive data
1737
+ field :ssn do
1738
+ strategy :redact # Complete removal
1739
+ only :sentry # Only for external service
1740
+ end
1741
+
1742
+ skips :access_timestamp, :action_type
1743
+ end
1744
+ end
1745
+ ```
1746
+
1747
+ **Pattern 9: Truncation shortcuts**
1748
+
1749
+ ```ruby
1750
+ class Events::ApiRequest < E11y::Event::Base
1751
+ contains_pii true
1752
+
1753
+ pii_filtering do
1754
+ # ✅ Truncate with inline option
1755
+ truncates :api_key, to: 8 # Show first 8 chars: "sk_live_..." → "sk_live_"
1756
+
1757
+ hashes :user_email, except: [:file_audit]
1758
+
1759
+ skips :request_id, :method, :path
1760
+ end
1761
+ end
1762
+ ```
1763
+
1764
+ **Pattern 10: Multiple adapters (only/except)**
1765
+
1766
+ ```ruby
1767
+ class Events::SecurityIncident < E11y::Event::Base
1768
+ contains_pii true
1769
+
1770
+ pii_filtering do
1771
+ # Send hashed to internal tools only
1772
+ field :suspect_email do
1773
+ strategy :hash
1774
+ only [:elasticsearch, :loki] # Only these adapters
1775
+ end
1776
+
1777
+ # Mask for external services
1778
+ field :victim_email do
1779
+ strategy :mask
1780
+ only [:sentry, :pagerduty]
1781
+ end
1782
+
1783
+ # Audit gets everything (AuditEvent auto-skip)
1784
+ skips :incident_id, :severity
1785
+ end
1786
+ end
1787
+ ```
1788
+
1789
+ ---
1790
+
1791
+ #### 3.4.6. Rails-Style Validators & Helpers
1792
+
1793
+ **Custom validators (like Rails):**
1794
+
1795
+ ```ruby
1796
+ # lib/e11y/pii/validators/email_validator.rb
1797
+ module E11y
1798
+ module Pii
1799
+ module Validators
1800
+ class EmailValidator
1801
+ def self.validate(value)
1802
+ return true if value =~ URI::MailTo::EMAIL_REGEXP
1803
+
1804
+ raise E11y::Pii::ValidationError, "Invalid email format: #{value}"
1805
+ end
1806
+ end
1807
+ end
1808
+ end
1809
+ end
1810
+
1811
+ # Usage in event:
1812
+ class Events::UserRegistered < E11y::Event::Base
1813
+ pii_filtering do
1814
+ field :email do
1815
+ strategy :hash
1816
+ validate_with E11y::Pii::Validators::EmailValidator
1817
+ end
1818
+ end
1819
+ end
1820
+ ```
1821
+
1822
+ **Macros (like Rails concerns):**
1823
+
1824
+ ```ruby
1825
+ # lib/e11y/pii/macros/standard_user_fields.rb
1826
+ module E11y
1827
+ module Pii
1828
+ module Macros
1829
+ module StandardUserFields
1830
+ extend ActiveSupport::Concern
1831
+
1832
+ class_methods do
1833
+ def filter_standard_user_pii
1834
+ pii_filtering do
1835
+ hashes :email, except: [:file_audit]
1836
+ masks :password
1837
+ skips :user_id
1838
+ end
1839
+ end
1840
+ end
1841
+ end
1842
+ end
1843
+ end
1844
+ end
1845
+
1846
+ # Usage:
1847
+ class Events::UserLogin < E11y::Event::Base
1848
+ include E11y::Pii::Macros::StandardUserFields
1849
+
1850
+ contains_pii true
1851
+ filter_standard_user_pii # ✅ Macro!
1852
+
1853
+ pii_filtering do
1854
+ # Additional fields
1855
+ masks :ip_address
1856
+ end
1857
+ end
1858
+ ```
1859
+
1860
+ ---
1861
+
1862
+ #### 3.4.7. Testing PII Filters (RSpec Examples)
1863
+
1864
+ **Critical:** PII filtering MUST be tested to ensure no leaks.
1865
+
1866
+ ```ruby
1867
+ # spec/lib/e11y/security/pii_filtering_spec.rb
1868
+ RSpec.describe 'PII Filtering', type: :integration do
1869
+ describe 'Per-Adapter Filtering' do
1870
+ let(:event_class) { Events::UserLogin }
1871
+ let(:payload) do
1872
+ {
1873
+ email: 'user@example.com',
1874
+ ip_address: '192.168.1.100',
1875
+ user_id: 'u123'
1876
+ }
1877
+ end
1878
+
1879
+ before do
1880
+ # Track events sent to adapters
1881
+ E11y::Adapters.each do |adapter|
1882
+ allow(adapter).to receive(:write).and_call_original
1883
+ end
1884
+ end
1885
+
1886
+ it 'applies different PII rules for each adapter' do
1887
+ event_class.track(payload)
1888
+
1889
+ # Check Audit Adapter: NO filtering
1890
+ audit_event = E11y::Adapters[:file_audit].written_events.last
1891
+ expect(audit_event[:payload][:email]).to eq('user@example.com')
1892
+ expect(audit_event[:payload][:ip_address]).to eq('192.168.1.100')
1893
+
1894
+ # Check Sentry Adapter: FULL masking
1895
+ sentry_event = E11y::Adapters[:sentry].written_events.last
1896
+ expect(sentry_event[:payload][:email]).to eq('[FILTERED]')
1897
+ expect(sentry_event[:payload][:ip_address]).to eq('[FILTERED]')
1898
+
1899
+ # Check Loki Adapter: CUSTOM (email masked, IP truncated)
1900
+ loki_event = E11y::Adapters[:loki].written_events.last
1901
+ expect(loki_event[:payload][:email]).to eq('[FILTERED]')
1902
+ expect(loki_event[:payload][:ip_address]).to eq('192.168.1.x')
1903
+
1904
+ # Check Elasticsearch Adapter: HASHED email only
1905
+ es_event = E11y::Adapters[:elasticsearch].written_events.last
1906
+ expect(es_event[:payload][:email]).to match(/^[a-f0-9]{64}$/) # SHA256 hash
1907
+ expect(es_event[:payload][:ip_address]).to eq('192.168.1.100') # Not filtered
1908
+ end
1909
+ end
1910
+
1911
+ describe 'Tier 1: No PII (Skip Filtering)' do
1912
+ let(:event_class) { Events::HealthCheck }
1913
+ let(:payload) { { status: 'ok' } }
1914
+
1915
+ it 'skips PII filtering entirely' do
1916
+ start = Time.now
1917
+
1918
+ 1000.times { event_class.track(payload) }
1919
+
1920
+ duration = Time.now - start
1921
+
1922
+ # Should be VERY fast (no filtering overhead)
1923
+ expect(duration).to be < 0.1 # <100ms for 1000 events
1924
+ end
1925
+ end
1926
+
1927
+ describe 'Tier 2: Rails Filters Only' do
1928
+ let(:event_class) { Events::OrderCreated }
1929
+ let(:payload) do
1930
+ {
1931
+ order_id: 'o123',
1932
+ amount: 99.99,
1933
+ password: 'secret123', # Rails filter should catch this
1934
+ api_key: 'sk_live_xxx' # Rails filter should catch this
1935
+ }
1936
+ end
1937
+
1938
+ it 'applies Rails.application.config.filter_parameters' do
1939
+ event_class.track(payload)
1940
+
1941
+ event = E11y::Adapters[:loki].written_events.last
1942
+
1943
+ # Known Rails filter keys
1944
+ expect(event[:payload][:password]).to eq('[FILTERED]')
1945
+ expect(event[:payload][:api_key]).to eq('[FILTERED]')
1946
+
1947
+ # Not filtered
1948
+ expect(event[:payload][:order_id]).to eq('o123')
1949
+ expect(event[:payload][:amount]).to eq(99.99)
1950
+ end
1951
+ end
1952
+
1953
+ describe 'Tier 3: Deep Filtering' do
1954
+ let(:event_class) { Events::UserRegistered }
1955
+ let(:payload) do
1956
+ {
1957
+ email: 'user@example.com',
1958
+ address: {
1959
+ street: '123 Main St',
1960
+ city: 'New York',
1961
+ zip: '10001'
1962
+ },
1963
+ metadata: {
1964
+ referrer: 'https://example.com?email=leaked@example.com'
1965
+ }
1966
+ }
1967
+ end
1968
+
1969
+ before do
1970
+ event_class.class_eval do
1971
+ contains_pii true
1972
+
1973
+ pii_filtering do
1974
+ deep_scan true
1975
+ patterns [:email] # Scan for emails in all strings
1976
+ end
1977
+ end
1978
+ end
1979
+
1980
+ it 'filters PII in nested structures' do
1981
+ event_class.track(payload)
1982
+
1983
+ event = E11y::Adapters[:sentry].written_events.last
1984
+
1985
+ # Top-level email filtered
1986
+ expect(event[:payload][:email]).to eq('[FILTERED]')
1987
+
1988
+ # Nested email in string also filtered
1989
+ expect(event[:payload][:metadata][:referrer]).not_to include('leaked@example.com')
1990
+ expect(event[:payload][:metadata][:referrer]).to include('[FILTERED]')
1991
+ end
1992
+ end
1993
+
1994
+ describe 'Performance Regression Test' do
1995
+ it 'meets performance budget for 1000 events/sec' do
1996
+ events = [
1997
+ { class: Events::HealthCheck, payload: { status: 'ok' }, count: 500 }, # Tier 1
1998
+ { class: Events::OrderCreated, payload: { id: 'o1' }, count: 400 }, # Tier 2
1999
+ { class: Events::UserLogin, payload: { email: 'u@x.com' }, count: 100 } # Tier 3
2000
+ ]
2001
+
2002
+ start = Time.now
2003
+
2004
+ events.each do |batch|
2005
+ batch[:count].times { batch[:class].track(batch[:payload]) }
2006
+ end
2007
+
2008
+ duration = Time.now - start
2009
+
2010
+ # Total: 1000 events
2011
+ # Budget: 40ms CPU (4% of 1 second)
2012
+ # Allow 2x buffer for test overhead
2013
+ expect(duration).to be < 0.08 # 80ms
2014
+ end
2015
+ end
2016
+
2017
+ describe 'GDPR Compliance Audit' do
2018
+ it 'logs PII filtering decisions for audit trail' do
2019
+ event_class = Events::UserLogin
2020
+
2021
+ expect(E11y.logger).to receive(:info).with(
2022
+ hash_including(
2023
+ event: 'pii_filtering_applied',
2024
+ adapter: :sentry,
2025
+ strategy: :mask,
2026
+ fields: [:email, :ip_address]
2027
+ )
2028
+ )
2029
+
2030
+ event_class.track(email: 'user@example.com', ip_address: '192.168.1.1')
2031
+ end
2032
+
2033
+ it 'logs justification for skipping PII filtering (audit)' do
2034
+ event_class = Events::GdprDeletion
2035
+
2036
+ expect(E11y.logger).to receive(:info).with(
2037
+ hash_including(
2038
+ event: 'pii_filtering_skipped',
2039
+ adapter: :file_audit,
2040
+ justification: /GDPR Art. 6\(1\)\(c\)/
2041
+ )
2042
+ )
2043
+
2044
+ event_class.track(user_id: 'u123', email: 'user@example.com')
2045
+ end
2046
+ end
2047
+
2048
+ describe 'Edge Cases' do
2049
+ it 'handles missing PII fields gracefully' do
2050
+ event_class = Events::UserLogin
2051
+
2052
+ # Missing 'email' field (required by PII config)
2053
+ expect {
2054
+ event_class.track(ip_address: '192.168.1.1')
2055
+ }.not_to raise_error
2056
+ end
2057
+
2058
+ it 'handles nil values in PII fields' do
2059
+ event_class = Events::UserLogin
2060
+
2061
+ event_class.track(email: nil, ip_address: '192.168.1.1')
2062
+
2063
+ event = E11y::Adapters[:sentry].written_events.last
2064
+ expect(event[:payload][:email]).to be_nil # Not "[FILTERED]"
2065
+ end
2066
+
2067
+ it 'handles very long PII values efficiently' do
2068
+ long_email = "#{'a' * 10_000}@example.com"
2069
+
2070
+ event_class = Events::UserLogin
2071
+
2072
+ start = Time.now
2073
+ event_class.track(email: long_email, ip_address: '192.168.1.1')
2074
+ duration = Time.now - start
2075
+
2076
+ # Should not hang or be extremely slow
2077
+ expect(duration).to be < 0.5 # <500ms
2078
+ end
2079
+ end
2080
+ end
2081
+
2082
+ # spec/support/pii_filter_matcher.rb
2083
+ RSpec::Matchers.define :be_pii_filtered do
2084
+ match do |value|
2085
+ ['[FILTERED]', '[REDACTED]', '[MASKED]'].include?(value) ||
2086
+ value =~ /^[a-f0-9]{32,}$/ # Hash
2087
+ end
2088
+
2089
+ failure_message do |value|
2090
+ "expected #{value.inspect} to be PII-filtered (e.g., '[FILTERED]' or hash)"
2091
+ end
2092
+ end
2093
+
2094
+ RSpec::Matchers.define :be_pii_free do |original|
2095
+ match do |value|
2096
+ value == original # Not filtered
2097
+ end
2098
+
2099
+ failure_message do |value|
2100
+ "expected #{value.inspect} to be PII-free (original: #{original.inspect})"
2101
+ end
2102
+ end
2103
+
2104
+ # Usage:
2105
+ # expect(event[:payload][:email]).to be_pii_filtered
2106
+ # expect(event[:payload][:order_id]).to be_pii_free('o123')
2107
+ ```
2108
+
2109
+ **Running PII Tests:**
2110
+ ```bash
2111
+ # Test all PII filtering
2112
+ bundle exec rspec spec/lib/e11y/security/pii_filtering_spec.rb
2113
+
2114
+ # Test specific tier
2115
+ bundle exec rspec spec/lib/e11y/security/pii_filtering_spec.rb -e "Tier 3"
2116
+
2117
+ # Performance test
2118
+ bundle exec rspec spec/lib/e11y/security/pii_filtering_spec.rb -e "Performance"
2119
+
2120
+ # GDPR compliance audit
2121
+ bundle exec rspec spec/lib/e11y/security/pii_filtering_spec.rb -e "GDPR"
2122
+ ```
2123
+
2124
+ ---
2125
+
2126
+ ## 4. Rate Limiting
2127
+
2128
+ ### 4.0. Rate Limiting + Retry Policy Resolution (Conflict #14)
2129
+
2130
+ **Critical Decision from CONFLICT-ANALYSIS.md:**
2131
+
2132
+ **Problem:** Should retry attempts count toward rate limits?
2133
+
2134
+ **Resolution:**
2135
+ ```ruby
2136
+ # ✅ Retries DO count toward rate limit
2137
+ # Reason: Prevent retry amplification attack
2138
+
2139
+ config.error_handling do
2140
+ retry_policy do
2141
+ respect_rate_limits true # ← Retries checked
2142
+ on_retry_rate_limited :send_to_dlq # Safety net
2143
+ end
2144
+ end
2145
+
2146
+ # Optional: Separate limit for retries
2147
+ config.rate_limiting do
2148
+ global_limit 100.per_second
2149
+
2150
+ retry_limit do
2151
+ enabled true
2152
+ limit 20.per_second # Additional headroom
2153
+ end
2154
+
2155
+ # Critical events bypass
2156
+ bypass_for_severities [:error, :fatal]
2157
+ bypass_for_patterns ['audit.*']
2158
+ end
2159
+ ```
2160
+
2161
+ **Pipeline Flow:**
2162
+ ```
2163
+ Original event
2164
+ → Rate Limiting (check)
2165
+ → Adapter Write (fail)
2166
+ → Retry #1
2167
+ → Rate Limiting (check again) ← Retry counts!
2168
+ → Adapter Write (fail)
2169
+ → Retry #2 (rate limited)
2170
+ → DLQ (rate-limited retry)
2171
+ ```
2172
+
2173
+ ---
2174
+
2175
+ ### 4.1. Global Rate Limiting
2176
+
2177
+ **Design Decision:** System-wide rate limit for all events.
2178
+
2179
+ ```ruby
2180
+ module E11y
2181
+ module RateLimiting
2182
+ class GlobalLimiter
2183
+ def initialize(config)
2184
+ @limit = config.limit # events per second
2185
+ @window = config.window || 1.second
2186
+ @strategy = config.strategy || :sliding_window
2187
+ @redis = config.redis
2188
+
2189
+ @counter = initialize_counter
2190
+ end
2191
+
2192
+ def allow?(event_data = nil)
2193
+ @counter.increment('global')
2194
+
2195
+ current_rate = @counter.rate('global', window: @window)
2196
+
2197
+ current_rate <= @limit
2198
+ end
2199
+
2200
+ def current_rate
2201
+ @counter.rate('global', window: @window)
2202
+ end
2203
+
2204
+ private
2205
+
2206
+ def initialize_counter
2207
+ if @redis
2208
+ RedisCounter.new(@redis, strategy: @strategy)
2209
+ else
2210
+ InMemoryCounter.new(strategy: @strategy)
2211
+ end
2212
+ end
2213
+ end
2214
+ end
2215
+ end
2216
+ ```
2217
+
2218
+ **Configuration:**
2219
+
2220
+ ```ruby
2221
+ E11y.configure do |config|
2222
+ config.rate_limiting do
2223
+ global do
2224
+ enabled true
2225
+ limit 10_000 # 10k events/sec
2226
+ window 1.second
2227
+ strategy :sliding_window
2228
+
2229
+ # Overflow behavior
2230
+ on_overflow :drop # :drop, :sample, or :queue
2231
+
2232
+ # Monitoring
2233
+ on_limit_exceeded do |current_rate, limit|
2234
+ Rails.logger.warn "[E11y] Global rate limit exceeded: #{current_rate}/#{limit}"
2235
+ end
2236
+ end
2237
+ end
2238
+ end
2239
+ ```
2240
+
2241
+ ---
2242
+
2243
+ ### 4.2. Per-Event Rate Limiting
2244
+
2245
+ **Design Decision:** Different limits for different event types.
2246
+
2247
+ ```ruby
2248
+ module E11y
2249
+ module RateLimiting
2250
+ class PerEventLimiter
2251
+ def initialize(config)
2252
+ @limits = config.limits || {}
2253
+ @default_limit = config.default_limit || Float::INFINITY
2254
+ @window = config.window || 1.second
2255
+ @redis = config.redis
2256
+
2257
+ @counter = initialize_counter
2258
+ end
2259
+
2260
+ def allow?(event_data)
2261
+ event_name = event_data[:event_name]
2262
+ limit = @limits[event_name] || @default_limit
2263
+
2264
+ return true if limit == Float::INFINITY
2265
+
2266
+ @counter.increment(event_name)
2267
+
2268
+ current_rate = @counter.rate(event_name, window: @window)
2269
+
2270
+ current_rate <= limit
2271
+ end
2272
+
2273
+ def current_rate(event_name)
2274
+ @counter.rate(event_name, window: @window)
2275
+ end
2276
+ end
2277
+ end
2278
+ end
2279
+ ```
2280
+
2281
+ **Configuration:**
2282
+
2283
+ ```ruby
2284
+ E11y.configure do |config|
2285
+ config.rate_limiting do
2286
+ per_event do
2287
+ enabled true
2288
+
2289
+ # Specific limits per event
2290
+ limits do
2291
+ event 'order.paid', limit: 1000 # 1k/sec
2292
+ event 'user.login', limit: 500 # 500/sec
2293
+ event 'debug.*', limit: 100 # 100/sec for all debug events
2294
+ end
2295
+
2296
+ # Default for unlisted events
2297
+ default_limit 1000
2298
+
2299
+ window 1.second
2300
+ end
2301
+ end
2302
+ end
2303
+ ```
2304
+
2305
+ ---
2306
+
2307
+ ### 4.3. Per-Context Rate Limiting
2308
+
2309
+ **Design Decision:** Limit per user/tenant/IP address.
2310
+
2311
+ ```ruby
2312
+ module E11y
2313
+ module RateLimiting
2314
+ class PerContextLimiter
2315
+ def initialize(config)
2316
+ @limit = config.limit
2317
+ @window = config.window || 1.minute
2318
+ @context_keys = config.context_keys || [:user_id]
2319
+ @redis = config.redis
2320
+
2321
+ @counter = initialize_counter
2322
+ end
2323
+
2324
+ def allow?(event_data)
2325
+ context_value = extract_context(event_data)
2326
+
2327
+ return true unless context_value
2328
+
2329
+ key = "context:#{context_value}"
2330
+ @counter.increment(key)
2331
+
2332
+ current_rate = @counter.rate(key, window: @window)
2333
+
2334
+ current_rate <= @limit
2335
+ end
2336
+
2337
+ private
2338
+
2339
+ def extract_context(event_data)
2340
+ @context_keys.each do |key|
2341
+ value = event_data.dig(:payload, key) || event_data[key]
2342
+ return value if value
2343
+ end
2344
+
2345
+ nil
2346
+ end
2347
+ end
2348
+ end
2349
+ end
2350
+ ```
2351
+
2352
+ **Configuration:**
2353
+
2354
+ ```ruby
2355
+ E11y.configure do |config|
2356
+ config.rate_limiting do
2357
+ per_context do
2358
+ enabled true
2359
+
2360
+ # Limit per user
2361
+ context_key :user_id
2362
+ limit 100 # 100 events per minute per user
2363
+ window 1.minute
2364
+
2365
+ # Multiple context keys (fallback)
2366
+ context_keys [:user_id, :session_id, :ip_address]
2367
+
2368
+ # Allowlist (no limits)
2369
+ allowlist do
2370
+ user_ids ['admin_user', 'system']
2371
+ end
2372
+ end
2373
+ end
2374
+ end
2375
+ ```
2376
+
2377
+ ---
2378
+
2379
+ ### 4.4. Redis Integration
2380
+
2381
+ **Design Decision:** Distributed rate limiting with Redis.
2382
+
2383
+ ```ruby
2384
+ module E11y
2385
+ module RateLimiting
2386
+ class RedisCounter
2387
+ def initialize(redis, strategy: :sliding_window)
2388
+ @redis = redis
2389
+ @strategy = strategy
2390
+ end
2391
+
2392
+ def increment(key)
2393
+ case @strategy
2394
+ when :sliding_window
2395
+ increment_sliding_window(key)
2396
+ when :fixed_window
2397
+ increment_fixed_window(key)
2398
+ when :token_bucket
2399
+ increment_token_bucket(key)
2400
+ end
2401
+ end
2402
+
2403
+ def rate(key, window:)
2404
+ case @strategy
2405
+ when :sliding_window
2406
+ count_sliding_window(key, window)
2407
+ when :fixed_window
2408
+ count_fixed_window(key, window)
2409
+ when :token_bucket
2410
+ tokens_remaining(key)
2411
+ end
2412
+ end
2413
+
2414
+ private
2415
+
2416
+ # Sliding Window (most accurate)
2417
+ def increment_sliding_window(key)
2418
+ now = Time.now.to_f
2419
+ redis_key = "e11y:rate:#{key}"
2420
+
2421
+ @redis.multi do |multi|
2422
+ multi.zadd(redis_key, now, "#{now}:#{SecureRandom.uuid}")
2423
+ multi.expire(redis_key, 60)
2424
+ end
2425
+ end
2426
+
2427
+ def count_sliding_window(key, window)
2428
+ now = Time.now.to_f
2429
+ cutoff = now - window.to_f
2430
+ redis_key = "e11y:rate:#{key}"
2431
+
2432
+ # Remove old entries
2433
+ @redis.zremrangebyscore(redis_key, '-inf', cutoff)
2434
+
2435
+ # Count remaining
2436
+ @redis.zcard(redis_key)
2437
+ end
2438
+
2439
+ # Fixed Window (simpler, less accurate)
2440
+ def increment_fixed_window(key)
2441
+ window_key = current_window_key(key)
2442
+
2443
+ @redis.multi do |multi|
2444
+ multi.incr(window_key)
2445
+ multi.expire(window_key, 60)
2446
+ end
2447
+ end
2448
+
2449
+ def count_fixed_window(key, window)
2450
+ window_key = current_window_key(key)
2451
+ @redis.get(window_key).to_i
2452
+ end
2453
+
2454
+ def current_window_key(key)
2455
+ window_start = (Time.now.to_i / 1) * 1 # 1-second windows
2456
+ "e11y:rate:fixed:#{key}:#{window_start}"
2457
+ end
2458
+
2459
+ # Token Bucket (burst-friendly)
2460
+ def increment_token_bucket(key)
2461
+ redis_key = "e11y:rate:bucket:#{key}"
2462
+
2463
+ # Lua script for atomic token consumption
2464
+ lua_script = <<~LUA
2465
+ local key = KEYS[1]
2466
+ local capacity = tonumber(ARGV[1])
2467
+ local refill_rate = tonumber(ARGV[2])
2468
+ local now = tonumber(ARGV[3])
2469
+
2470
+ local tokens = tonumber(redis.call('GET', key) or capacity)
2471
+ local last_refill = tonumber(redis.call('GET', key .. ':last') or now)
2472
+
2473
+ -- Refill tokens
2474
+ local time_passed = now - last_refill
2475
+ local tokens_to_add = time_passed * refill_rate
2476
+ tokens = math.min(capacity, tokens + tokens_to_add)
2477
+
2478
+ -- Consume token
2479
+ if tokens >= 1 then
2480
+ tokens = tokens - 1
2481
+ redis.call('SET', key, tokens)
2482
+ redis.call('SET', key .. ':last', now)
2483
+ return 1 -- Success
2484
+ else
2485
+ return 0 -- No tokens
2486
+ end
2487
+ LUA
2488
+
2489
+ @redis.eval(lua_script, keys: [redis_key], argv: [100, 10, Time.now.to_f])
2490
+ end
2491
+
2492
+ def tokens_remaining(key)
2493
+ redis_key = "e11y:rate:bucket:#{key}"
2494
+ @redis.get(redis_key).to_i
2495
+ end
2496
+ end
2497
+ end
2498
+ end
2499
+ ```
2500
+
2501
+ **Rate Limiting Strategies Comparison:**
2502
+
2503
+ ```mermaid
2504
+ graph TB
2505
+ subgraph "Sliding Window (Most Accurate)"
2506
+ SW1[Request at t=0] --> SW2[Request at t=0.5]
2507
+ SW2 --> SW3[Request at t=0.9]
2508
+ SW3 --> SW4[Window: 0.0-1.0s<br/>Count: 3]
2509
+ end
2510
+
2511
+ subgraph "Fixed Window (Simpler)"
2512
+ FW1[Request at t=0.9] --> FW2[Request at t=1.1]
2513
+ FW2 --> FW3[Window 1: 1 request<br/>Window 2: 1 request]
2514
+ Note right of FW3: Burst at boundary!
2515
+ end
2516
+
2517
+ subgraph "Token Bucket (Burst-Friendly)"
2518
+ TB1[Capacity: 100 tokens] --> TB2[Refill: 10/sec]
2519
+ TB2 --> TB3[Burst: OK if tokens available]
2520
+ end
2521
+
2522
+ style SW4 fill:#d4edda
2523
+ style FW3 fill:#fff3cd
2524
+ style TB3 fill:#d1ecf1
2525
+ ```
2526
+
2527
+ ---
2528
+
2529
+ ## 5. Audit Trail
2530
+
2531
+ ### 5.0. Audit Trail + PII Filtering Resolution (Conflict #4)
2532
+
2533
+ **Critical Decision from CONFLICT-ANALYSIS.md:**
2534
+
2535
+ **Problem:** Audit events need original data for compliance, but PII filtering requires masking for GDPR.
2536
+
2537
+ **Resolution:**
2538
+ ```ruby
2539
+ # Option A: Audit events skip PII filtering (RECOMMENDED)
2540
+ config.audit_trail do
2541
+ skip_pii_filtering true # Legal obligation > privacy
2542
+ end
2543
+
2544
+ # Compensating controls:
2545
+ config.audit_trail do
2546
+ encryption_at_rest true
2547
+ access_control do
2548
+ read_access_role :auditor
2549
+ read_access_requires_reason true
2550
+ read_access_logged true # Meta-audit
2551
+ end
2552
+ end
2553
+
2554
+ # Option B: Per-adapter PII rules (more flexible)
2555
+ class UserPermissionChanged < E11y::AuditEvent
2556
+ adapters [:audit_file, :elasticsearch, :sentry]
2557
+
2558
+ pii_rules do
2559
+ # Audit file: keep all PII (compliance)
2560
+ adapter :audit_file do
2561
+ skip_filtering true
2562
+ end
2563
+
2564
+ # Elasticsearch: pseudonymize
2565
+ adapter :elasticsearch do
2566
+ pseudonymize_fields :email, :ip_address
2567
+ end
2568
+
2569
+ # Sentry: mask all
2570
+ adapter :sentry do
2571
+ mask_fields :email, :ip_address, :user_id
2572
+ end
2573
+ end
2574
+ end
2575
+ ```
2576
+
2577
+ **GDPR Justification:**
2578
+ - Art. 6(1)(c): "Legal obligation" is valid processing basis
2579
+ - Audit logs = necessary for accountability (SOX, HIPAA, GDPR Art. 30)
2580
+ - Mitigation: encryption + access control + retention limits
2581
+
2582
+ ---
2583
+
2584
+ ### 5.1. Immutable Events
2585
+
2586
+ **Design Decision:** Audit events are append-only, never modified.
2587
+
2588
+ ```ruby
2589
+ module E11y
2590
+ class AuditEvent < Event::Base
2591
+ # Audit events CANNOT be modified after creation
2592
+ def self.inherited(subclass)
2593
+ super
2594
+
2595
+ subclass.class_eval do
2596
+ # Freeze payload after creation
2597
+ after_track :freeze_payload!
2598
+
2599
+ private
2600
+
2601
+ def freeze_payload!
2602
+ @payload.deep_freeze
2603
+ end
2604
+ end
2605
+ end
2606
+
2607
+ # Audit-specific metadata
2608
+ def self.audit_retention(days)
2609
+ @audit_retention_days = days
2610
+ end
2611
+
2612
+ def self.audit_reason(reason)
2613
+ @audit_reason = reason
2614
+ end
2615
+
2616
+ def self.signing_enabled(enabled = true)
2617
+ @signing_enabled = enabled
2618
+ end
2619
+ end
2620
+ end
2621
+ ```
2622
+
2623
+ **Example:**
2624
+
2625
+ ```ruby
2626
+ module Events
2627
+ class GdprDeletion < E11y::AuditEvent
2628
+ adapters [:file_audit, :elasticsearch_audit]
2629
+
2630
+ audit_retention 2555 # 7 years
2631
+ audit_reason 'GDPR Article 17 - Right to erasure'
2632
+ signing_enabled true
2633
+
2634
+ schema do
2635
+ required(:user_id).filled(:string)
2636
+ required(:deletion_reason).filled(:string)
2637
+ required(:requested_by).filled(:string)
2638
+ required(:approved_by).filled(:string)
2639
+ end
2640
+ end
2641
+
2642
+ class FinancialTransaction < E11y::AuditEvent
2643
+ audit_retention 2555 # 7 years (SOX compliance)
2644
+ audit_reason 'SOX Section 802'
2645
+ signing_enabled true
2646
+
2647
+ schema do
2648
+ required(:transaction_id).filled(:string)
2649
+ required(:amount).filled(:float)
2650
+ required(:currency).filled(:string)
2651
+ required(:from_account).filled(:string)
2652
+ required(:to_account).filled(:string)
2653
+ end
2654
+ end
2655
+ end
2656
+ ```
2657
+
2658
+ ---
2659
+
2660
+ ### 5.2. Cryptographic Signing
2661
+
2662
+ **Design Decision:** HMAC-SHA256 signatures for tamper detection.
2663
+
2664
+ ```ruby
2665
+ module E11y
2666
+ module Security
2667
+ class EventSigner
2668
+ def initialize(config)
2669
+ @secret_key = config.secret_key || raise("Secret key required for signing")
2670
+ @algorithm = config.algorithm || 'SHA256'
2671
+ @include_chain = config.include_chain || true
2672
+ @previous_hash = nil
2673
+ end
2674
+
2675
+ def sign(event_data)
2676
+ # Create canonical representation
2677
+ canonical = canonicalize(event_data)
2678
+
2679
+ # Add chain if enabled
2680
+ if @include_chain && @previous_hash
2681
+ canonical[:previous_hash] = @previous_hash
2682
+ end
2683
+
2684
+ # Generate signature
2685
+ signature = generate_signature(canonical)
2686
+
2687
+ # Store hash for next event
2688
+ @previous_hash = signature
2689
+
2690
+ # Add signature to event
2691
+ event_data.merge(
2692
+ signature: signature,
2693
+ signed_at: Time.now.utc.iso8601,
2694
+ signature_algorithm: "HMAC-#{@algorithm}"
2695
+ )
2696
+ end
2697
+
2698
+ def verify(event_data)
2699
+ stored_signature = event_data[:signature]
2700
+ return false unless stored_signature
2701
+
2702
+ # Remove signature for verification
2703
+ data_without_sig = event_data.except(:signature, :signed_at, :signature_algorithm)
2704
+
2705
+ # Canonicalize
2706
+ canonical = canonicalize(data_without_sig)
2707
+
2708
+ # Generate expected signature
2709
+ expected_signature = generate_signature(canonical)
2710
+
2711
+ # Constant-time comparison
2712
+ secure_compare(expected_signature, stored_signature)
2713
+ end
2714
+
2715
+ def verify_chain(events)
2716
+ events.each_cons(2).all? do |prev_event, current_event|
2717
+ # Check if current event references previous
2718
+ current_event[:previous_hash] == prev_event[:signature]
2719
+ end
2720
+ end
2721
+
2722
+ private
2723
+
2724
+ def canonicalize(data)
2725
+ # Sort keys recursively for deterministic output
2726
+ case data
2727
+ when Hash
2728
+ data.sort.to_h.transform_values { |v| canonicalize(v) }
2729
+ when Array
2730
+ data.map { |item| canonicalize(item) }
2731
+ else
2732
+ data
2733
+ end
2734
+ end
2735
+
2736
+ def generate_signature(canonical_data)
2737
+ json = JSON.generate(canonical_data)
2738
+ OpenSSL::HMAC.hexdigest(@algorithm, @secret_key, json)
2739
+ end
2740
+
2741
+ def secure_compare(a, b)
2742
+ return false if a.bytesize != b.bytesize
2743
+
2744
+ l = a.unpack("C*")
2745
+ r = b.unpack("C*")
2746
+
2747
+ result = 0
2748
+ l.zip(r) { |x, y| result |= x ^ y }
2749
+ result == 0
2750
+ end
2751
+ end
2752
+ end
2753
+ end
2754
+ ```
2755
+
2756
+ **Per-Event Signing Configuration:**
2757
+
2758
+ For fine-grained control, individual audit events can override signing behavior:
2759
+
2760
+ ```ruby
2761
+ # Disable signing for low-severity audit events
2762
+ class Events::AuditLogViewed < E11y::Event::Base
2763
+ audit_event true
2764
+ signing enabled: false # ⚠️ No cryptographic signing
2765
+
2766
+ schema do
2767
+ required(:log_id).filled(:integer)
2768
+ required(:viewed_by).filled(:integer)
2769
+ end
2770
+ end
2771
+
2772
+ # Explicitly enable signing (default behavior)
2773
+ class Events::UserDeleted < E11y::Event::Base
2774
+ audit_event true
2775
+ signing enabled: true # ✅ Cryptographic signing (default)
2776
+
2777
+ schema do
2778
+ required(:user_id).filled(:integer)
2779
+ required(:deleted_by).filled(:integer)
2780
+ end
2781
+ end
2782
+ ```
2783
+
2784
+ **DSL Consistency:**
2785
+
2786
+ The `signing enabled:` DSL matches the global configuration pattern:
2787
+
2788
+ ```ruby
2789
+ # Global config (affects ALL audit events)
2790
+ E11y.configure do |config|
2791
+ config.audit_trail do
2792
+ signing enabled: true, # ← Same DSL pattern
2793
+ algorithm: 'HMAC-SHA256'
2794
+ end
2795
+ end
2796
+
2797
+ # Per-event override (affects ONE event class)
2798
+ class Events::SomeEvent < E11y::Event::Base
2799
+ signing enabled: false # ← Same DSL pattern
2800
+ end
2801
+ ```
2802
+
2803
+ **When to disable signing:**
2804
+ - Low-severity audit events (e.g., log views)
2805
+ - High-volume events with performance constraints
2806
+ - Internal monitoring (non-compliance use cases)
2807
+
2808
+ **When signing is REQUIRED:**
2809
+ - Financial transactions (SOX)
2810
+ - User data deletion (GDPR Art. 17)
2811
+ - Permission changes (access control)
2812
+ - Any event requiring non-repudiation
2813
+
2814
+ **Default:** All audit events are signed (`signing enabled: true`) unless explicitly disabled.
2815
+
2816
+ **Signature Chain Visualization:**
2817
+
2818
+ ```mermaid
2819
+ graph LR
2820
+ Event1[Event 1<br/>signature: abc123] --> Event2[Event 2<br/>previous_hash: abc123<br/>signature: def456]
2821
+ Event2 --> Event3[Event 3<br/>previous_hash: def456<br/>signature: ghi789]
2822
+ Event3 --> Event4[Event 4<br/>previous_hash: ghi789<br/>signature: jkl012]
2823
+
2824
+ Event2 -.->|Verify| Check1{previous_hash<br/>matches?}
2825
+ Check1 -->|Yes ✅| Valid1[Valid Chain]
2826
+ Check1 -->|No ❌| Tampered1[Tampered!]
2827
+
2828
+ style Event1 fill:#d4edda
2829
+ style Event2 fill:#d4edda
2830
+ style Event3 fill:#d4edda
2831
+ style Event4 fill:#d4edda
2832
+ style Valid1 fill:#d4edda
2833
+ style Tampered1 fill:#f8d7da
2834
+ ```
2835
+
2836
+ ---
2837
+
2838
+ ### 5.3. Compliance Features
2839
+
2840
+ **GDPR, HIPAA, SOX, PCI DSS Requirements:**
2841
+
2842
+ ```ruby
2843
+ module E11y
2844
+ module Compliance
2845
+ class AuditConfig
2846
+ # GDPR Article 30 - Records of processing activities
2847
+ def gdpr_compliant!
2848
+ audit_retention 2555 # 7 years
2849
+ audit_reason 'GDPR Article 30'
2850
+ signing_enabled true
2851
+
2852
+ # Log who accessed what
2853
+ log_access true
2854
+ access_control do
2855
+ require_justification true
2856
+ require_approval true
2857
+ end
2858
+
2859
+ # Right to be forgotten
2860
+ support_deletion_requests true
2861
+ deletion_proof_required true
2862
+ end
2863
+
2864
+ # HIPAA - Health Insurance Portability and Accountability Act
2865
+ def hipaa_compliant!
2866
+ audit_retention 2190 # 6 years
2867
+ audit_reason 'HIPAA 45 CFR 164.312'
2868
+ signing_enabled true
2869
+
2870
+ # PHI access logging
2871
+ log_phi_access true
2872
+ encrypt_at_rest true
2873
+
2874
+ # Breach notification
2875
+ breach_detection true
2876
+ notification_within 60.days
2877
+ end
2878
+
2879
+ # SOX - Sarbanes-Oxley Act
2880
+ def sox_compliant!
2881
+ audit_retention 2555 # 7 years
2882
+ audit_reason 'SOX Section 802'
2883
+ signing_enabled true
2884
+
2885
+ # Financial records
2886
+ tamper_proof true
2887
+ dual_approval_required true
2888
+
2889
+ # Retention enforcement
2890
+ prevent_deletion true
2891
+ archive_after 2555.days
2892
+ end
2893
+
2894
+ # PCI DSS - Payment Card Industry Data Security Standard
2895
+ def pci_dss_compliant!
2896
+ audit_retention 365 # 1 year minimum
2897
+ audit_reason 'PCI DSS Requirement 10'
2898
+ signing_enabled true
2899
+
2900
+ # Card data events
2901
+ log_all_access true
2902
+ log_failed_attempts true
2903
+
2904
+ # Time synchronization
2905
+ ntp_sync_required true
2906
+
2907
+ # Review
2908
+ daily_log_review true
2909
+ end
2910
+ end
2911
+ end
2912
+ end
2913
+ ```
2914
+
2915
+ **Configuration:**
2916
+
2917
+ ```ruby
2918
+ E11y.configure do |config|
2919
+ config.compliance do
2920
+ # Enable GDPR compliance
2921
+ gdpr do
2922
+ enabled true
2923
+ data_controller 'Company Inc.'
2924
+ dpo_email 'dpo@company.com'
2925
+
2926
+ # Consent tracking
2927
+ track_consent true
2928
+ consent_required_for [:analytics, :marketing]
2929
+ end
2930
+
2931
+ # Enable SOX compliance
2932
+ sox do
2933
+ enabled true
2934
+ dual_approval_required true
2935
+
2936
+ # Audit settings
2937
+ audit_settings do
2938
+ retention_period 7.years
2939
+ tamper_proof true
2940
+ end
2941
+ end
2942
+ end
2943
+ end
2944
+ ```
2945
+
2946
+ ---
2947
+
2948
+ ### 5.4. Tamper Detection
2949
+
2950
+ **Design Decision:** Automated verification and alerting.
2951
+
2952
+ ```ruby
2953
+ module E11y
2954
+ module Security
2955
+ class TamperDetector
2956
+ def initialize(signer, alert_handler)
2957
+ @signer = signer
2958
+ @alert_handler = alert_handler
2959
+ end
2960
+
2961
+ def verify_event(event_data)
2962
+ valid = @signer.verify(event_data)
2963
+
2964
+ unless valid
2965
+ alert_tampering!(event_data)
2966
+ end
2967
+
2968
+ valid
2969
+ end
2970
+
2971
+ def verify_batch(events)
2972
+ # Verify each signature
2973
+ signature_results = events.map do |event|
2974
+ { event: event, valid: @signer.verify(event) }
2975
+ end
2976
+
2977
+ # Verify chain
2978
+ chain_valid = @signer.verify_chain(events)
2979
+
2980
+ # Check for tampering
2981
+ tampered_events = signature_results.reject { |r| r[:valid] }
2982
+
2983
+ if tampered_events.any? || !chain_valid
2984
+ alert_batch_tampering!(tampered_events, chain_valid)
2985
+ end
2986
+
2987
+ {
2988
+ total: events.size,
2989
+ valid: signature_results.count { |r| r[:valid] },
2990
+ invalid: tampered_events.size,
2991
+ chain_valid: chain_valid
2992
+ }
2993
+ end
2994
+
2995
+ private
2996
+
2997
+ def alert_tampering!(event_data)
2998
+ @alert_handler.call(
2999
+ type: :single_event_tampered,
3000
+ event_id: event_data[:event_id],
3001
+ event_name: event_data[:event_name],
3002
+ timestamp: event_data[:timestamp],
3003
+ severity: :critical
3004
+ )
3005
+ end
3006
+
3007
+ def alert_batch_tampering!(tampered_events, chain_valid)
3008
+ @alert_handler.call(
3009
+ type: :batch_tampered,
3010
+ tampered_count: tampered_events.size,
3011
+ chain_valid: chain_valid,
3012
+ event_ids: tampered_events.map { |e| e[:event][:event_id] },
3013
+ severity: :critical
3014
+ )
3015
+ end
3016
+ end
3017
+ end
3018
+ end
3019
+ ```
3020
+
3021
+ **Monitoring:**
3022
+
3023
+ ```ruby
3024
+ E11y.configure do |config|
3025
+ config.security.tamper_detection do
3026
+ enabled true
3027
+
3028
+ # Verify on read
3029
+ verify_on_read true
3030
+
3031
+ # Periodic verification
3032
+ periodic_verification do
3033
+ enabled true
3034
+ interval 1.hour
3035
+ sample_size 1000
3036
+ end
3037
+
3038
+ # Alerting
3039
+ on_tampering_detected do |alert|
3040
+ # Critical alert
3041
+ PagerDuty.trigger(
3042
+ service: 'e11y-audit',
3043
+ incident_key: "tamper-#{alert[:type]}",
3044
+ description: "Audit log tampering detected: #{alert[:tampered_count]} events",
3045
+ severity: 'critical'
3046
+ )
3047
+
3048
+ # Log
3049
+ Rails.logger.error "[E11y] TAMPERING DETECTED: #{alert.inspect}"
3050
+
3051
+ # Disable writes (safety)
3052
+ E11y.emergency_readonly_mode!
3053
+ end
3054
+ end
3055
+ end
3056
+ ```
3057
+
3058
+ ---
3059
+
3060
+ ## 5.5. OpenTelemetry Baggage PII Protection (C08 Resolution)
3061
+
3062
+ > **⚠️ CRITICAL: C08 Conflict Resolution - PII × OpenTelemetry Baggage**
3063
+ > **See:** [CONFLICT-ANALYSIS.md C08](researches/CONFLICT-ANALYSIS.md#c08-pii-filtering--opentelemetry-baggage) for detailed analysis
3064
+ > **Problem:** OpenTelemetry Baggage automatically propagates via HTTP headers and can leak PII to downstream services
3065
+ > **Solution:** Block PII from baggage entirely with safe key allowlist
3066
+
3067
+ ### 5.5.1. The Problem: PII Leaking via OpenTelemetry Baggage
3068
+
3069
+ **Scenario - GDPR Violation:**
3070
+
3071
+ ```ruby
3072
+ # Service A: User registration
3073
+ Events::UserRegistered.track(
3074
+ user_id: '123',
3075
+ email: 'user@example.com', # ← PII, filtered in event payload ✅
3076
+ name: 'John Doe' # ← PII, filtered in event payload ✅
3077
+ )
3078
+
3079
+ # Developer accidentally sets PII in baggage:
3080
+ OpenTelemetry::Baggage.set_value('user_email', 'user@example.com') # ❌ PII in baggage!
3081
+ OpenTelemetry::Baggage.set_value('user_name', 'John Doe') # ❌ PII in baggage!
3082
+
3083
+ # Service A → Service B (HTTP call)
3084
+ # HTTP headers AUTOMATICALLY include baggage:
3085
+ # baggage: user_email=user@example.com,user_name=John Doe
3086
+
3087
+ # Service B receives baggage with PII!
3088
+ # Service B logs baggage → PII LEAKED to downstream service!
3089
+
3090
+ # Result: PII BYPASS!
3091
+ # - Event payload: PII filtered ✅
3092
+ # - Baggage: PII NOT filtered ❌ ← GDPR violation!
3093
+ ```
3094
+
3095
+ **Why This Is Critical:**
3096
+ - ✅ OpenTelemetry Baggage is **W3C standard** for trace context propagation
3097
+ - ❌ Baggage is **automatically transmitted** via HTTP headers (`baggage: key=value`)
3098
+ - ❌ Downstream services **receive PII without filtering**
3099
+ - ❌ Logs, metrics, traces in downstream services **contain PII**
3100
+ - ❌ **GDPR Article 5(1)(c) violation:** PII transmitted beyond necessary scope
3101
+
3102
+ ### 5.5.2. Decision: Block PII from Baggage Entirely
3103
+
3104
+ **Strategy:** Allowlist-only mode for safe keys, block all others.
3105
+
3106
+ **Safe Keys (Allowed):**
3107
+ - `trace_id` - OpenTelemetry trace ID (not PII)
3108
+ - `span_id` - OpenTelemetry span ID (not PII)
3109
+ - `environment` - Environment name (production, staging, etc.)
3110
+ - `version` - Application version (deployment ID)
3111
+ - `service_name` - Service identifier
3112
+ - `deployment_id` - Deployment identifier
3113
+ - `request_id` - Internal request ID (UUID, not PII)
3114
+
3115
+ **Blocked Keys (PII Risk):**
3116
+ - `user_id`, `user_email`, `user_name` - User identifiers
3117
+ - `ip_address`, `client_ip` - IP addresses (GDPR Article 4(1))
3118
+ - `session_id`, `session_token` - Session identifiers (can be PII)
3119
+ - `api_key`, `token`, `password` - Credentials
3120
+ - Any custom keys not in allowlist
3121
+
3122
+ ### 5.5.3. BaggageProtection Middleware Implementation
3123
+
3124
+ ```ruby
3125
+ module E11y
3126
+ module Middleware
3127
+ class BaggageProtection
3128
+ # Safe keys that can be set in baggage
3129
+ ALLOWED_KEYS = %w[
3130
+ trace_id
3131
+ span_id
3132
+ environment
3133
+ version
3134
+ service_name
3135
+ deployment_id
3136
+ request_id
3137
+ ].freeze
3138
+
3139
+ def initialize(config)
3140
+ @enabled = config.baggage_protection_enabled || true
3141
+ @allowed_keys = config.baggage_allowed_keys || ALLOWED_KEYS
3142
+ @block_mode = config.baggage_block_mode || :silent # :silent, :warn, :raise
3143
+ @logger = E11y.logger
3144
+ end
3145
+
3146
+ def call(event_data)
3147
+ return event_data unless @enabled
3148
+
3149
+ # Intercept OpenTelemetry::Baggage.set_value calls
3150
+ protect_baggage!
3151
+
3152
+ event_data
3153
+ end
3154
+
3155
+ private
3156
+
3157
+ def protect_baggage!
3158
+ # Monkey-patch OpenTelemetry::Baggage (runtime protection)
3159
+ OpenTelemetry::Baggage.singleton_class.prepend(BaggageInterceptor.new(
3160
+ allowed_keys: @allowed_keys,
3161
+ block_mode: @block_mode,
3162
+ logger: @logger
3163
+ ))
3164
+ end
3165
+ end
3166
+
3167
+ # Interceptor for OpenTelemetry::Baggage.set_value
3168
+ module BaggageInterceptor
3169
+ def initialize(allowed_keys:, block_mode:, logger:)
3170
+ @allowed_keys = allowed_keys
3171
+ @block_mode = block_mode
3172
+ @logger = logger
3173
+ super()
3174
+ end
3175
+
3176
+ def set_value(key, value, context = nil)
3177
+ # Check if key is allowed
3178
+ unless @allowed_keys.include?(key.to_s)
3179
+ handle_blocked_key(key, value)
3180
+ return context || OpenTelemetry::Context.current
3181
+ end
3182
+
3183
+ # Key is safe, proceed
3184
+ super(key, value, context)
3185
+ end
3186
+
3187
+ private
3188
+
3189
+ def handle_blocked_key(key, value)
3190
+ message = "[E11y] Blocked PII from OpenTelemetry baggage: key=#{key.inspect}"
3191
+
3192
+ case @block_mode
3193
+ when :silent
3194
+ # Block silently (default)
3195
+ @logger.debug(message)
3196
+ when :warn
3197
+ # Block with warning
3198
+ @logger.warn(message)
3199
+ when :raise
3200
+ # Block with exception (strict mode)
3201
+ raise BaggagePiiError, "#{message}. Only allowed keys: #{@allowed_keys.join(', ')}"
3202
+ end
3203
+ end
3204
+ end
3205
+
3206
+ class BaggagePiiError < StandardError; end
3207
+ end
3208
+ end
3209
+ ```
3210
+
3211
+ ### 5.5.4. Configuration
3212
+
3213
+ ```ruby
3214
+ # config/initializers/e11y.rb
3215
+ E11y.configure do |config|
3216
+ config.security.baggage_protection do
3217
+ enabled true # ✅ CRITICAL: Always enable in production
3218
+
3219
+ # Allowlist: Only these keys allowed in baggage
3220
+ allowed_keys [
3221
+ 'trace_id',
3222
+ 'span_id',
3223
+ 'environment',
3224
+ 'version',
3225
+ 'service_name',
3226
+ 'deployment_id',
3227
+ 'request_id',
3228
+ # Custom safe keys (optional):
3229
+ 'feature_flag_id', # Feature flag identifier (not PII)
3230
+ 'ab_test_variant' # A/B test variant (not PII)
3231
+ ]
3232
+
3233
+ # Block mode (how to handle violations)
3234
+ block_mode :silent # Options: :silent, :warn, :raise
3235
+
3236
+ # Monitoring
3237
+ on_blocked_key do |key, value, caller_location|
3238
+ # Track violations for security audit
3239
+ Yabeda.e11y_baggage_pii_blocked.increment(
3240
+ key: key,
3241
+ service: ENV['SERVICE_NAME']
3242
+ )
3243
+
3244
+ # Alert on critical violations
3245
+ if key.match?(/email|password|ssn|credit_card/)
3246
+ Sentry.capture_message(
3247
+ "Critical PII blocked from baggage: #{key}",
3248
+ level: :warning,
3249
+ extra: { caller: caller_location }
3250
+ )
3251
+ end
3252
+ end
3253
+ end
3254
+ end
3255
+ ```
3256
+
3257
+ ### 5.5.5. Usage Examples
3258
+
3259
+ **❌ BAD: Blocked PII Keys**
3260
+
3261
+ ```ruby
3262
+ # Service A:
3263
+ OpenTelemetry::Baggage.set_value('user_email', 'user@example.com')
3264
+ # → BLOCKED (not in allowlist)
3265
+ # → Logged: "[E11y] Blocked PII from OpenTelemetry baggage: key='user_email'"
3266
+
3267
+ OpenTelemetry::Baggage.set_value('ip_address', '192.168.1.100')
3268
+ # → BLOCKED (not in allowlist)
3269
+
3270
+ OpenTelemetry::Baggage.set_value('session_id', 'abc123')
3271
+ # → BLOCKED (not in allowlist)
3272
+
3273
+ # HTTP call to Service B:
3274
+ # baggage: (empty - all blocked)
3275
+ ```
3276
+
3277
+ **✅ GOOD: Allowed Safe Keys**
3278
+
3279
+ ```ruby
3280
+ # Service A:
3281
+ OpenTelemetry::Baggage.set_value('trace_id', 'abc123def456')
3282
+ # → ALLOWED ✅
3283
+
3284
+ OpenTelemetry::Baggage.set_value('environment', 'production')
3285
+ # → ALLOWED ✅
3286
+
3287
+ OpenTelemetry::Baggage.set_value('version', 'v2.1.0')
3288
+ # → ALLOWED ✅
3289
+
3290
+ OpenTelemetry::Baggage.set_value('feature_flag_id', 'new_checkout_v2')
3291
+ # → ALLOWED ✅ (custom safe key)
3292
+
3293
+ # HTTP call to Service B:
3294
+ # baggage: trace_id=abc123def456,environment=production,version=v2.1.0,feature_flag_id=new_checkout_v2
3295
+ # → All safe keys propagated ✅
3296
+ ```
3297
+
3298
+ **✅ ALTERNATIVE: Use Non-PII Identifiers**
3299
+
3300
+ ```ruby
3301
+ # Instead of user_email in baggage:
3302
+ OpenTelemetry::Baggage.set_value('user_id_hash', Digest::SHA256.hexdigest(user.email))
3303
+ # → Pseudonymized, no PII ✅
3304
+
3305
+ # Instead of session_id:
3306
+ OpenTelemetry::Baggage.set_value('request_id', SecureRandom.uuid)
3307
+ # → Non-PII identifier ✅
3308
+ ```
3309
+
3310
+ ### 5.5.6. Strict Mode (Development/Staging)
3311
+
3312
+ **Recommended for non-production environments:**
3313
+
3314
+ ```ruby
3315
+ # config/environments/development.rb
3316
+ E11y.configure do |config|
3317
+ config.security.baggage_protection do
3318
+ enabled true
3319
+
3320
+ # RAISE exception on blocked keys (fail fast)
3321
+ block_mode :raise # ← Developer sees error immediately
3322
+
3323
+ allowed_keys E11y::Middleware::BaggageProtection::ALLOWED_KEYS
3324
+ end
3325
+ end
3326
+
3327
+ # Developer tries to set PII in baggage:
3328
+ OpenTelemetry::Baggage.set_value('user_email', 'test@example.com')
3329
+ # → RAISES BaggagePiiError:
3330
+ # "[E11y] Blocked PII from OpenTelemetry baggage: key='user_email'.
3331
+ # Only allowed keys: trace_id, span_id, environment, version, ..."
3332
+ ```
3333
+
3334
+ ### 5.5.7. Trade-offs & GDPR Compliance (C08)
3335
+
3336
+ **Trade-offs:**
3337
+
3338
+ | Decision | Pro | Con | Rationale |
3339
+ |----------|-----|-----|-----------|
3340
+ | **Allowlist-only mode** | No PII can leak | Less flexible | GDPR compliance > flexibility |
3341
+ | **Block at runtime** | No code changes needed | Performance overhead (~0.01ms) | Security > performance |
3342
+ | **Silent mode default** | No breaking changes | Harder to debug | Gradual rollout safer |
3343
+ | **Raise mode (dev)** | Fail fast | Breaks tests | Catch violations early |
3344
+
3345
+ **GDPR Compliance:**
3346
+
3347
+ ✅ **Article 5(1)(c) - Data minimisation:**
3348
+ Baggage protection ensures PII is not transmitted beyond necessary scope.
3349
+
3350
+ ✅ **Article 5(1)(f) - Integrity and confidentiality:**
3351
+ Blocking PII from automatic propagation protects data integrity.
3352
+
3353
+ ✅ **Article 32 - Security of processing:**
3354
+ Technical measure to prevent PII leakage via trace context.
3355
+
3356
+ **Monitoring Metrics:**
3357
+
3358
+ ```ruby
3359
+ # Track baggage protection effectiveness
3360
+ Yabeda.e11y_baggage_pii_blocked.increment(
3361
+ key: 'user_email',
3362
+ service: 'api-gateway'
3363
+ )
3364
+
3365
+ # Alert on repeated violations (indicates developer education needed)
3366
+ Yabeda.e11y_baggage_pii_violations_total.increment(
3367
+ caller_service: 'user-service',
3368
+ blocked_key_pattern: 'user_*'
3369
+ )
3370
+ ```
3371
+
3372
+ **Related Conflicts:**
3373
+ - **C07:** DLQ replay with PII filtering (see §5.6 below)
3374
+ - **C01:** Audit trail signing with PII (see ADR-015 §3.3)
3375
+
3376
+ ---
3377
+
3378
+ ## 5.6. PII Handling for Event Replay from DLQ (C07 Resolution)
3379
+
3380
+ > **⚠️ HIGH: C07 Conflict Resolution - PII Pseudonymization × DLQ Replay**
3381
+ > **See:** [CONFLICT-ANALYSIS.md C07](researches/CONFLICT-ANALYSIS.md#c07-pii-pseudonymization--dlq-replay) for detailed analysis
3382
+ > **Problem:** Replayed events from DLQ go through pipeline again and get double-hashed PII
3383
+ > **Solution:** Mark events as already filtered to prevent idempotency violations
3384
+
3385
+ ### 5.6.1. The Problem: Double-Hashing PII on Replay
3386
+
3387
+ **Scenario - Data Corruption:**
3388
+
3389
+ ```ruby
3390
+ # Original event (first processing):
3391
+ Events::UserLogin.track(
3392
+ user_id: '123',
3393
+ email: 'user@example.com', # ← Original PII
3394
+ ip_address: '192.168.1.1' # ← Original PII
3395
+ )
3396
+
3397
+ # Pipeline step 2: PII Filtering (ADR-015)
3398
+ # → email: 'user@example.com' → SHA256 hash → 'a1b2c3d4...'
3399
+ # → ip_address: '192.168.1.1' → SHA256 hash → 'e5f6g7h8...'
3400
+
3401
+ # Event sent to adapters, but Loki adapter fails
3402
+ # → Event goes to Dead Letter Queue (DLQ)
3403
+
3404
+ # UC-021: DLQ Replay
3405
+ # Event replayed from DLQ → goes through pipeline AGAIN
3406
+
3407
+ # Pipeline step 2: PII Filtering runs AGAIN!
3408
+ # → email: 'a1b2c3d4...' (already hashed!) → SHA256 hash → 'x9y8z7w6...'
3409
+ # ❌ DOUBLE-HASHED! Original: a1b2c3d4, Replay: x9y8z7w6
3410
+
3411
+ # Result: DATA CORRUPTION!
3412
+ # - Original event: { email: 'a1b2c3d4...', ip: 'e5f6g7h8...' }
3413
+ # - Replayed event: { email: 'x9y8z7w6...', ip: 'k9l8m7n6...' }
3414
+ # - Same user, DIFFERENT hashes!
3415
+ ```
3416
+
3417
+ **Why This Breaks:**
3418
+ - ❌ **Idempotency violated:** Replay produces different output than original
3419
+ - ❌ **Audit trail corrupted:** Can't correlate original event with replayed event
3420
+ - ❌ **Forensics impossible:** User tracking broken (different email hashes)
3421
+ - ❌ **GDPR violation:** Can't fulfill data deletion requests (can't find all user data)
3422
+
3423
+ ### 5.6.2. Decision: Skip PII Filtering for Replayed Events
3424
+
3425
+ **Strategy:** Metadata flag prevents double-processing.
3426
+
3427
+ **Metadata Flags:**
3428
+ - `:pii_filtered` - Event already went through PII filtering (set by PII filter middleware)
3429
+ - `:replayed` - Event is being replayed from DLQ (set by DLQ replay service)
3430
+
3431
+ **When to Skip PII Filtering:**
3432
+ 1. Event has `:pii_filtered => true` (already processed)
3433
+ 2. Event has `:replayed => true` (replay scenario)
3434
+ 3. Both flags present → skip PII filtering entirely
3435
+
3436
+ ### 5.6.3. PiiFilter Middleware with Replay Detection
3437
+
3438
+ ```ruby
3439
+ module E11y
3440
+ module Middleware
3441
+ class PiiFilter < Base
3442
+ def initialize(config)
3443
+ @config = config
3444
+ @rails_filter = Rails::ParameterFilter.new(Rails.application.config.filter_parameters)
3445
+ @pattern_matcher = PatternMatcher.new(config.patterns)
3446
+ end
3447
+
3448
+ def call(event_data)
3449
+ # ✅ CRITICAL: Check if already filtered (replay scenario)
3450
+ if already_filtered?(event_data)
3451
+ E11y.logger.debug "[E11y] Skipping PII filtering for replayed event: #{event_data[:event_id]}"
3452
+ return event_data # Skip filtering!
3453
+ end
3454
+
3455
+ # Get event class
3456
+ event_class = E11y::Registry.get_class(event_data[:event_name])
3457
+
3458
+ # Check if event contains PII
3459
+ unless event_class.contains_pii?
3460
+ # No PII declared, skip filtering (Tier 1)
3461
+ return event_data
3462
+ end
3463
+
3464
+ # Apply PII filtering
3465
+ filtered_data = apply_pii_rules(event_class, event_data)
3466
+
3467
+ # ✅ Mark as filtered (prevents double-processing)
3468
+ filtered_data[:metadata] ||= {}
3469
+ filtered_data[:metadata][:pii_filtered] = true
3470
+
3471
+ filtered_data
3472
+ end
3473
+
3474
+ private
3475
+
3476
+ def already_filtered?(event_data)
3477
+ metadata = event_data[:metadata] || {}
3478
+
3479
+ # Check for replay flag
3480
+ return true if metadata[:replayed]
3481
+
3482
+ # Check for already-filtered flag
3483
+ return true if metadata[:pii_filtered]
3484
+
3485
+ false
3486
+ end
3487
+
3488
+ def apply_pii_rules(event_class, event_data)
3489
+ # Per-field PII filtering logic
3490
+ # (implementation details in §3.4)
3491
+ #...
3492
+ end
3493
+ end
3494
+ end
3495
+ end
3496
+ ```
3497
+
3498
+ ### 5.6.4. DLQ Replay Service with Metadata Flags
3499
+
3500
+ ```ruby
3501
+ module E11y
3502
+ module DLQ
3503
+ class ReplayService
3504
+ def replay_event(dlq_event)
3505
+ # Extract original event data
3506
+ event_data = dlq_event[:event_data]
3507
+
3508
+ # ✅ CRITICAL: Mark as replayed (skip transformations)
3509
+ event_data[:metadata] ||= {}
3510
+ event_data[:metadata][:replayed] = true
3511
+ event_data[:metadata][:pii_filtered] = true # Already filtered!
3512
+ event_data[:metadata][:replayed_at] = Time.now.utc.iso8601
3513
+ event_data[:metadata][:original_event_id] = event_data[:event_id]
3514
+ event_data[:metadata][:replay_reason] = dlq_event[:failure_reason]
3515
+
3516
+ # Send through pipeline
3517
+ # PII filter middleware will skip (already_filtered? returns true)
3518
+ E11y::Pipeline.process(event_data)
3519
+
3520
+ # Log successful replay
3521
+ E11y.logger.info "[E11y] Replayed event from DLQ: #{event_data[:event_id]}"
3522
+ end
3523
+
3524
+ def replay_batch(dlq_events)
3525
+ dlq_events.each do |event|
3526
+ begin
3527
+ replay_event(event)
3528
+ rescue StandardError => e
3529
+ E11y.logger.error "[E11y] Failed to replay event: #{e.message}"
3530
+ # Re-queue to DLQ with updated metadata
3531
+ end
3532
+ end
3533
+ end
3534
+ end
3535
+ end
3536
+ end
3537
+ ```
3538
+
3539
+ ### 5.6.5. Configuration
3540
+
3541
+ ```ruby
3542
+ # config/initializers/e11y.rb
3543
+ E11y.configure do |config|
3544
+ config.security.pii_filtering do
3545
+ enabled true
3546
+
3547
+ # Replay handling
3548
+ replay_handling do
3549
+ # Skip PII filtering for replayed events (default: true)
3550
+ skip_on_replay true
3551
+
3552
+ # Validate metadata flags (safety check)
3553
+ validate_replay_flags true
3554
+
3555
+ # Warn if replayed event missing pii_filtered flag
3556
+ warn_on_missing_flag :warn # Options: :silent, :warn, :raise
3557
+ end
3558
+ end
3559
+
3560
+ config.dlq.replay do
3561
+ # Metadata flags for replayed events
3562
+ set_metadata_flags do
3563
+ replayed true
3564
+ pii_filtered true
3565
+ replay_timestamp true
3566
+ original_event_id true
3567
+ end
3568
+ end
3569
+ end
3570
+ ```
3571
+
3572
+ ### 5.6.6. Usage Examples
3573
+
3574
+ **✅ CORRECT: Replay from DLQ with Metadata**
3575
+
3576
+ ```ruby
3577
+ # UC-021: Event failed to write to adapter
3578
+ original_event = {
3579
+ event_id: 'evt_123',
3580
+ event_name: 'user.login',
3581
+ payload: {
3582
+ user_id: '123',
3583
+ email: 'a1b2c3d4...', # Already hashed by first pass
3584
+ ip_address: 'e5f6g7h8...' # Already hashed
3585
+ },
3586
+ metadata: {
3587
+ pii_filtered: true # ✅ Set during first pass
3588
+ }
3589
+ }
3590
+
3591
+ # Event sent to DLQ after adapter failure
3592
+ DLQ.enqueue(original_event)
3593
+
3594
+ # Later: Replay from DLQ
3595
+ replay_service = E11y::DLQ::ReplayService.new
3596
+ replay_service.replay_event(original_event)
3597
+
3598
+ # ✅ Result: PII filter skipped, email hash unchanged
3599
+ # Replayed event: { email: 'a1b2c3d4...' } (same as original!)
3600
+ ```
3601
+
3602
+ **❌ BAD: Replay without Metadata (Double-Hashing)**
3603
+
3604
+ ```ruby
3605
+ # Hypothetical scenario: Replayed event missing metadata
3606
+ replayed_event = {
3607
+ event_id: 'evt_123',
3608
+ event_name: 'user.login',
3609
+ payload: {
3610
+ email: 'a1b2c3d4...', # Already hashed!
3611
+ },
3612
+ metadata: {} # ❌ Missing pii_filtered flag!
3613
+ }
3614
+
3615
+ # Pipeline processes event
3616
+ # PII filter middleware does NOT skip (no flag)
3617
+ # → email: 'a1b2c3d4...' → SHA256 hash → 'x9y8z7w6...'
3618
+ # ❌ DOUBLE-HASHED!
3619
+
3620
+ # UC-021 MUST set metadata flags to prevent this!
3621
+ ```
3622
+
3623
+ **✅ ALTERNATIVE: Separate Replay Pipeline**
3624
+
3625
+ ```ruby
3626
+ # Optional: Dedicated pipeline for replays (skip all transformations)
3627
+ E11y.configure do |config|
3628
+ config.pipeline_order do
3629
+ # Standard pipeline (for new events)
3630
+ standard_pipeline do
3631
+ step :validation
3632
+ step :pii_filtering # ← Runs for NEW events
3633
+ step :rate_limiting
3634
+ step :sampling
3635
+ step :trace_context
3636
+ step :buffer
3637
+ step :adapters
3638
+ end
3639
+
3640
+ # Replay pipeline (skip transformations)
3641
+ replay_pipeline do
3642
+ step :validation # ← Still validate schema
3643
+ # NO pii_filtering # ← Skip! Already filtered
3644
+ # NO rate_limiting # ← Skip! Already passed
3645
+ # NO sampling # ← Skip! Already sampled
3646
+ step :trace_context # ← Restore trace context
3647
+ step :buffer
3648
+ step :adapters # ← Write to adapters
3649
+ end
3650
+ end
3651
+
3652
+ # Use replay pipeline for DLQ events
3653
+ config.dlq.replay do
3654
+ use_pipeline :replay_pipeline
3655
+ end
3656
+ end
3657
+ ```
3658
+
3659
+ ### 5.6.7. Idempotency Verification (Testing)
3660
+
3661
+ **Critical:** Verify replay produces identical output.
3662
+
3663
+ ```ruby
3664
+ # spec/lib/e11y/dlq/replay_service_spec.rb
3665
+ RSpec.describe E11y::DLQ::ReplayService do
3666
+ describe '#replay_event' do
3667
+ it 'produces identical output for replayed events (idempotency)' do
3668
+ # Original event with PII
3669
+ original_event = Events::UserLogin.track(
3670
+ user_id: '123',
3671
+ email: 'user@example.com',
3672
+ ip_address: '192.168.1.1'
3673
+ )
3674
+
3675
+ # Capture filtered output (first pass)
3676
+ first_pass_output = E11y::Adapters[:elasticsearch].written_events.last
3677
+
3678
+ # Simulate DLQ scenario
3679
+ dlq_event = {
3680
+ event_data: original_event.to_h,
3681
+ failure_reason: 'Adapter timeout'
3682
+ }
3683
+
3684
+ # Replay event
3685
+ replay_service = described_class.new
3686
+ replay_service.replay_event(dlq_event)
3687
+
3688
+ # Capture replayed output (second pass)
3689
+ second_pass_output = E11y::Adapters[:elasticsearch].written_events.last
3690
+
3691
+ # ✅ CRITICAL: Hashes must be IDENTICAL
3692
+ expect(second_pass_output[:payload][:email]).to eq(first_pass_output[:payload][:email])
3693
+ expect(second_pass_output[:payload][:ip_address]).to eq(first_pass_output[:payload][:ip_address])
3694
+
3695
+ # Verify no double-hashing
3696
+ expect(second_pass_output[:payload][:email]).not_to match(/x9y8z7w6/) # Wrong hash
3697
+ expect(second_pass_output[:payload][:email]).to match(/^[a-f0-9]{64}$/) # SHA256
3698
+ end
3699
+
3700
+ it 'skips PII filtering for replayed events' do
3701
+ # Event already processed
3702
+ processed_event = {
3703
+ event_id: 'evt_123',
3704
+ event_name: 'user.login',
3705
+ payload: {
3706
+ email: 'a1b2c3d4...', # Already hashed
3707
+ },
3708
+ metadata: {
3709
+ pii_filtered: true
3710
+ }
3711
+ }
3712
+
3713
+ # Spy on PII filter middleware
3714
+ pii_filter = E11y::Middleware::PiiFilter.instance
3715
+ allow(pii_filter).to receive(:apply_pii_rules).and_call_original
3716
+
3717
+ # Replay event
3718
+ replay_service = described_class.new
3719
+ replay_service.replay_event({ event_data: processed_event })
3720
+
3721
+ # ✅ PII filter should NOT be called (already filtered)
3722
+ expect(pii_filter).not_to have_received(:apply_pii_rules)
3723
+ end
3724
+ end
3725
+ end
3726
+ ```
3727
+
3728
+ ### 5.6.8. Trade-offs & Audit Trail Integrity (C07)
3729
+
3730
+ **Trade-offs:**
3731
+
3732
+ | Decision | Pro | Con | Rationale |
3733
+ |----------|-----|-----|-----------|
3734
+ | **Skip PII on replay** | Idempotent replay | Complexity (metadata flags) | Audit integrity > simplicity |
3735
+ | **Metadata flag check** | Prevents double-hashing | Runtime overhead (~0.01ms) | Data correctness > performance |
3736
+ | **Separate replay pipeline** | Clear separation | More configuration | Optional advanced feature |
3737
+ | **Validate flags** | Catches missing metadata | May raise errors | Safety > convenience |
3738
+
3739
+ **Audit Trail Integrity:**
3740
+
3741
+ ✅ **Idempotency guarantee:**
3742
+ Replay produces identical output → audit trail remains consistent.
3743
+
3744
+ ✅ **User tracking:**
3745
+ Email hashes remain stable across replays → GDPR data deletion requests work correctly.
3746
+
3747
+ ✅ **Forensic analysis:**
3748
+ Can correlate original event with replayed event using `original_event_id` metadata.
3749
+
3750
+ **Monitoring Metrics:**
3751
+
3752
+ ```ruby
3753
+ # Track replay operations
3754
+ Yabeda.e11y_dlq_replays_total.increment(
3755
+ event_type: 'user.login',
3756
+ pii_filtered_skipped: true
3757
+ )
3758
+
3759
+ # Alert on potential double-hashing
3760
+ Yabeda.e11y_pii_double_hash_prevented.increment(
3761
+ event_id: 'evt_123'
3762
+ )
3763
+ ```
3764
+
3765
+ **Related Conflicts:**
3766
+ - **C08:** OpenTelemetry Baggage PII protection (see §5.5 above)
3767
+ - **C01:** Audit trail signing (see ADR-015 §3.3)
3768
+ - **C15:** Event versioning × replay (see ADR-012 - user responsibility for schema migrations)
3769
+
3770
+ ---
3771
+
3772
+ ## 6. GDPR Compliance
3773
+
3774
+ ### 6.1. GDPR Features
3775
+
3776
+ ```ruby
3777
+ module E11y
3778
+ module Compliance
3779
+ class GdprSupport
3780
+ def initialize(config)
3781
+ @enabled = config.enabled
3782
+ @data_controller = config.data_controller
3783
+ @dpo_email = config.dpo_email
3784
+ end
3785
+
3786
+ # Article 15 - Right of access
3787
+ def export_user_data(user_id)
3788
+ events = E11y::Storage.find_by_user(user_id)
3789
+
3790
+ {
3791
+ data_controller: @data_controller,
3792
+ dpo_contact: @dpo_email,
3793
+ export_date: Time.now.utc,
3794
+ user_id: user_id,
3795
+ events: events.map { |e| sanitize_for_export(e) }
3796
+ }
3797
+ end
3798
+
3799
+ # Article 17 - Right to erasure
3800
+ def delete_user_data(user_id, reason:, requested_by:)
3801
+ # Log deletion (immutable audit)
3802
+ Events::GdprDeletion.track(
3803
+ user_id: user_id,
3804
+ deletion_reason: reason,
3805
+ requested_by: requested_by,
3806
+ approved_by: current_user_id
3807
+ )
3808
+
3809
+ # Mark for deletion (not immediate)
3810
+ E11y::Storage.mark_for_deletion(user_id)
3811
+
3812
+ # Return proof
3813
+ {
3814
+ deletion_requested_at: Time.now.utc,
3815
+ deletion_effective_date: 30.days.from_now,
3816
+ deletion_proof_id: SecureRandom.uuid
3817
+ }
3818
+ end
3819
+
3820
+ # Article 20 - Right to data portability
3821
+ def portable_format(user_id)
3822
+ data = export_user_data(user_id)
3823
+
3824
+ # Convert to portable JSON
3825
+ {
3826
+ format: 'JSON',
3827
+ version: '1.0',
3828
+ data: data
3829
+ }.to_json
3830
+ end
3831
+ end
3832
+ end
3833
+ end
3834
+ ```
3835
+
3836
+ ---
3837
+
3838
+ ## 7. Configuration
3839
+
3840
+ ### 7.1. Complete Configuration
3841
+
3842
+ ```ruby
3843
+ # config/initializers/e11y.rb
3844
+ E11y.configure do |config|
3845
+ # ============================================================================
3846
+ # PII Filtering
3847
+ # ============================================================================
3848
+ config.security.pii_filtering do
3849
+ enabled true
3850
+
3851
+ # Rails integration
3852
+ use_rails_filters true
3853
+
3854
+ # Pattern matching
3855
+ patterns [:email, :phone, :ssn, :credit_card, :ip_address]
3856
+
3857
+ # Custom patterns
3858
+ custom_pattern :internal_id, /\bINT-\d{6,}\b/
3859
+
3860
+ # Deep scanning
3861
+ deep_scan true
3862
+
3863
+ # Allowlist
3864
+ allowlist [
3865
+ 'order.id',
3866
+ /^metadata\.internal_/
3867
+ ]
3868
+
3869
+ # Per-adapter rules
3870
+ adapter_rules do
3871
+ adapter :file_audit, mode: :skip
3872
+ adapter :sentry, mode: :strict
3873
+ end
3874
+
3875
+ # Sampling for debug
3876
+ sampling do
3877
+ enabled true
3878
+ sample_rate 0.01
3879
+ max_samples 100
3880
+ end
3881
+ end
3882
+
3883
+ # ============================================================================
3884
+ # Rate Limiting
3885
+ # ============================================================================
3886
+ config.rate_limiting do
3887
+ # Global limit
3888
+ global do
3889
+ enabled true
3890
+ limit 10_000
3891
+ window 1.second
3892
+ strategy :sliding_window
3893
+ redis Redis.current
3894
+
3895
+ on_overflow :drop
3896
+ end
3897
+
3898
+ # Per-event limits
3899
+ per_event do
3900
+ enabled true
3901
+
3902
+ limits do
3903
+ event 'order.paid', limit: 1000
3904
+ event 'user.login', limit: 500
3905
+ event 'debug.*', limit: 100
3906
+ end
3907
+
3908
+ default_limit 1000
3909
+ end
3910
+
3911
+ # Per-context limits
3912
+ per_context do
3913
+ enabled true
3914
+ context_keys [:user_id, :session_id, :ip_address]
3915
+ limit 100
3916
+ window 1.minute
3917
+
3918
+ allowlist do
3919
+ user_ids ['admin', 'system']
3920
+ end
3921
+ end
3922
+ end
3923
+
3924
+ # ============================================================================
3925
+ # Audit Trail
3926
+ # ============================================================================
3927
+ config.security.audit do
3928
+ enabled true
3929
+
3930
+ # Signing
3931
+ signing do
3932
+ enabled true
3933
+ secret_key ENV['E11Y_AUDIT_SECRET']
3934
+ algorithm 'SHA256'
3935
+ include_chain true
3936
+ end
3937
+
3938
+ # Tamper detection
3939
+ tamper_detection do
3940
+ enabled true
3941
+ verify_on_read true
3942
+
3943
+ periodic_verification do
3944
+ enabled true
3945
+ interval 1.hour
3946
+ sample_size 1000
3947
+ end
3948
+
3949
+ on_tampering_detected do |alert|
3950
+ PagerDuty.trigger(alert)
3951
+ E11y.emergency_readonly_mode!
3952
+ end
3953
+ end
3954
+
3955
+ # Retention
3956
+ default_retention 2555.days # 7 years
3957
+ end
3958
+
3959
+ # ============================================================================
3960
+ # GDPR Compliance
3961
+ # ============================================================================
3962
+ config.compliance.gdpr do
3963
+ enabled true
3964
+ data_controller 'Company Inc.'
3965
+ dpo_email 'dpo@company.com'
3966
+
3967
+ # Consent
3968
+ track_consent true
3969
+ consent_required_for [:analytics, :marketing]
3970
+
3971
+ # Data subject rights
3972
+ support_access_requests true
3973
+ support_deletion_requests true
3974
+ support_portability true
3975
+
3976
+ # Deletion
3977
+ deletion_delay 30.days
3978
+ permanent_deletion_after 90.days
3979
+ end
3980
+ end
3981
+ ```
3982
+
3983
+ ---
3984
+
3985
+ ## 8. Testing
3986
+
3987
+ ### 8.1. RSpec Examples
3988
+
3989
+ ```ruby
3990
+ RSpec.describe E11y::Security::PiiFilter do
3991
+ describe 'PII filtering' do
3992
+ it 'filters email addresses' do
3993
+ event = Events::UserLogin.track(
3994
+ email: 'user@example.com',
3995
+ password: 'secret123'
3996
+ )
3997
+
3998
+ expect(event.payload[:email]).to eq('[FILTERED:EMAIL]')
3999
+ expect(event.payload[:password]).to eq('[FILTERED]')
4000
+ end
4001
+
4002
+ it 'deep scans nested data' do
4003
+ event = Events::OrderCreated.track(
4004
+ order: {
4005
+ customer: {
4006
+ email: 'user@example.com'
4007
+ }
4008
+ }
4009
+ )
4010
+
4011
+ expect(event.payload[:order][:customer][:email]).to eq('[FILTERED:EMAIL]')
4012
+ end
4013
+ end
4014
+ end
4015
+
4016
+ RSpec.describe E11y::RateLimiting do
4017
+ describe 'global rate limiting' do
4018
+ it 'enforces global limit' do
4019
+ # Track 100 events (within limit)
4020
+ 100.times do
4021
+ expect(Events::Test.track(message: 'test')).to be_truthy
4022
+ end
4023
+
4024
+ # Exceed limit
4025
+ E11y.config.rate_limiting.global.limit = 100
4026
+
4027
+ expect(Events::Test.track(message: 'over limit')).to be_falsey
4028
+ end
4029
+ end
4030
+ end
4031
+
4032
+ RSpec.describe E11y::Security::EventSigner do
4033
+ describe 'cryptographic signing' do
4034
+ it 'signs events' do
4035
+ event = Events::AuditEvent.track(action: 'delete_user')
4036
+
4037
+ expect(event.signature).to be_present
4038
+ expect(event.signed_at).to be_present
4039
+ end
4040
+
4041
+ it 'detects tampering' do
4042
+ event = Events::AuditEvent.track(action: 'delete_user')
4043
+
4044
+ # Tamper with event
4045
+ event.payload[:action] = 'create_user'
4046
+
4047
+ expect(E11y.security.signer.verify(event)).to be_falsey
4048
+ end
4049
+
4050
+ it 'verifies signature chain' do
4051
+ events = 3.times.map { Events::AuditEvent.track(action: 'test') }
4052
+
4053
+ expect(E11y.security.signer.verify_chain(events)).to be_truthy
4054
+ end
4055
+ end
4056
+ end
4057
+ ```
4058
+
4059
+ ---
4060
+
4061
+ ## 9. Trade-offs
4062
+
4063
+ ### 9.1. Key Decisions
4064
+
4065
+ | Decision | Pro | Con | Rationale |
4066
+ |----------|-----|-----|-----------|
4067
+ | **Rails filter integration** | Zero config | Rails dependency | Most teams use Rails |
4068
+ | **Pattern-based PII** | Flexible | False positives | Good enough for 95% |
4069
+ | **Per-adapter rules** | Fine-grained | Complexity | Critical for compliance |
4070
+ | **Redis rate limiting** | Distributed | External dep | Required for scale |
4071
+ | **HMAC signatures** | Fast | Not PKI | Speed > PKI for logs |
4072
+ | **Chain verification** | Detects gaps | Memory overhead | Critical for audit |
4073
+ | **Audit skip PII** | Compliance | Privacy risk | Legal obligation wins |
4074
+ | **Retry count toward limit** | System safety | Less delivery | Prevent amplification |
4075
+ | **Baggage allowlist (C08)** | No PII leaks | Less flexible | GDPR compliance > flexibility |
4076
+ | **Block baggage at runtime** | No code changes | 0.01ms overhead | Security > performance |
4077
+
4078
+ ### 9.2. Pipeline Order (From CONFLICT-ANALYSIS.md)
4079
+
4080
+ **Definitive processing order:**
4081
+
4082
+ ```ruby
4083
+ track(event)
4084
+
4085
+ 1. Schema Validation (fail fast)
4086
+
4087
+ 2. Context Enrichment (trace_id, user_id)
4088
+
4089
+ 3. PII Filtering (security first)
4090
+ ↓ [Per-adapter: different rules]
4091
+
4092
+ 4. Rate Limiting (system protection) ← Retries also checked here
4093
+ ↓ [Drop if exceeded → DLQ if retry]
4094
+
4095
+ 5. Adaptive Sampling (cost optimization)
4096
+ ↓ [Drop if not sampled]
4097
+
4098
+ 6. Buffer Routing (debug vs. main)
4099
+
4100
+ 7. Adapters (with circuit breakers)
4101
+ ```
4102
+
4103
+ **Key Insight:** Rate limiting BEFORE sampling (Conflict #2 resolution)
4104
+ - Rate limiting = system stability (higher priority)
4105
+ - Sampling = cost optimization (lower priority)
4106
+
4107
+ ### 9.3. Precedence Rules (From CONFLICT-ANALYSIS.md)
4108
+
4109
+ **When multiple configs apply:**
4110
+
4111
+ 1. **Event-level config** (highest priority)
4112
+ - `event.retention = 7.years`
4113
+ - `event.adapters = [:audit_file]`
4114
+ - `event.skip_pii_filtering = true`
4115
+
4116
+ 2. **Per-event-type config**
4117
+ - `per_event 'payment.*' { sample_rate: 1.0 }`
4118
+
4119
+ 3. **Per-severity config**
4120
+ - `per_severity :fatal { adapters: [:sentry, :pagerduty] }`
4121
+
4122
+ 4. **Global config** (lowest priority)
4123
+ - `config.default_adapters = [:loki]`
4124
+
4125
+ ### 9.4. Alternatives Considered
4126
+
4127
+ **A) No PII filtering**
4128
+ - ❌ Rejected: GDPR violations
4129
+
4130
+ **B) ML-based PII detection**
4131
+ - ❌ Rejected: Too slow, over-engineering for v1.0
4132
+
4133
+ **C) Token bucket only**
4134
+ - ❌ Rejected: Need sliding window for accuracy
4135
+
4136
+ **D) PKI signatures**
4137
+ - ❌ Rejected: Overkill, HMAC sufficient
4138
+
4139
+ ---
4140
+
4141
+ **Status:** ✅ Draft Complete
4142
+ **Next:** ADR-008 (Rails Integration) or ADR-011 (Testing Strategy)
4143
+ **Estimated Implementation:** 3 weeks