e11y 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.rspec +4 -0
- data/.rubocop.yml +69 -0
- data/CHANGELOG.md +26 -0
- data/CODE_OF_CONDUCT.md +64 -0
- data/LICENSE.txt +21 -0
- data/README.md +179 -0
- data/Rakefile +37 -0
- data/benchmarks/run_all.rb +33 -0
- data/config/README.md +83 -0
- data/config/loki-local-config.yaml +35 -0
- data/config/prometheus.yml +15 -0
- data/docker-compose.yml +78 -0
- data/docs/00-ICP-AND-TIMELINE.md +483 -0
- data/docs/01-SCALE-REQUIREMENTS.md +858 -0
- data/docs/ADR-001-architecture.md +2617 -0
- data/docs/ADR-002-metrics-yabeda.md +1395 -0
- data/docs/ADR-003-slo-observability.md +3337 -0
- data/docs/ADR-004-adapter-architecture.md +2385 -0
- data/docs/ADR-005-tracing-context.md +1372 -0
- data/docs/ADR-006-security-compliance.md +4143 -0
- data/docs/ADR-007-opentelemetry-integration.md +1385 -0
- data/docs/ADR-008-rails-integration.md +1911 -0
- data/docs/ADR-009-cost-optimization.md +2993 -0
- data/docs/ADR-010-developer-experience.md +2166 -0
- data/docs/ADR-011-testing-strategy.md +1836 -0
- data/docs/ADR-012-event-evolution.md +958 -0
- data/docs/ADR-013-reliability-error-handling.md +2750 -0
- data/docs/ADR-014-event-driven-slo.md +1533 -0
- data/docs/ADR-015-middleware-order.md +1061 -0
- data/docs/ADR-016-self-monitoring-slo.md +1234 -0
- data/docs/API-REFERENCE-L28.md +914 -0
- data/docs/COMPREHENSIVE-CONFIGURATION.md +2366 -0
- data/docs/IMPLEMENTATION_NOTES.md +2804 -0
- data/docs/IMPLEMENTATION_PLAN.md +1971 -0
- data/docs/IMPLEMENTATION_PLAN_ARCHITECTURE.md +586 -0
- data/docs/PLAN.md +148 -0
- data/docs/QUICK-START.md +934 -0
- data/docs/README.md +296 -0
- data/docs/design/00-memory-optimization.md +593 -0
- data/docs/guides/MIGRATION-L27-L28.md +692 -0
- data/docs/guides/PERFORMANCE-BENCHMARKS.md +434 -0
- data/docs/guides/README.md +44 -0
- data/docs/prd/01-overview-vision.md +440 -0
- data/docs/use_cases/README.md +119 -0
- data/docs/use_cases/UC-001-request-scoped-debug-buffering.md +813 -0
- data/docs/use_cases/UC-002-business-event-tracking.md +1953 -0
- data/docs/use_cases/UC-003-pattern-based-metrics.md +1627 -0
- data/docs/use_cases/UC-004-zero-config-slo-tracking.md +728 -0
- data/docs/use_cases/UC-005-sentry-integration.md +759 -0
- data/docs/use_cases/UC-006-trace-context-management.md +905 -0
- data/docs/use_cases/UC-007-pii-filtering.md +2648 -0
- data/docs/use_cases/UC-008-opentelemetry-integration.md +1153 -0
- data/docs/use_cases/UC-009-multi-service-tracing.md +1043 -0
- data/docs/use_cases/UC-010-background-job-tracking.md +1018 -0
- data/docs/use_cases/UC-011-rate-limiting.md +1906 -0
- data/docs/use_cases/UC-012-audit-trail.md +2301 -0
- data/docs/use_cases/UC-013-high-cardinality-protection.md +2127 -0
- data/docs/use_cases/UC-014-adaptive-sampling.md +1940 -0
- data/docs/use_cases/UC-015-cost-optimization.md +735 -0
- data/docs/use_cases/UC-016-rails-logger-migration.md +785 -0
- data/docs/use_cases/UC-017-local-development.md +867 -0
- data/docs/use_cases/UC-018-testing-events.md +1081 -0
- data/docs/use_cases/UC-019-tiered-storage-migration.md +562 -0
- data/docs/use_cases/UC-020-event-versioning.md +708 -0
- data/docs/use_cases/UC-021-error-handling-retry-dlq.md +956 -0
- data/docs/use_cases/UC-022-event-registry.md +648 -0
- data/docs/use_cases/backlog.md +226 -0
- data/e11y.gemspec +76 -0
- data/lib/e11y/adapters/adaptive_batcher.rb +207 -0
- data/lib/e11y/adapters/audit_encrypted.rb +239 -0
- data/lib/e11y/adapters/base.rb +580 -0
- data/lib/e11y/adapters/file.rb +224 -0
- data/lib/e11y/adapters/in_memory.rb +216 -0
- data/lib/e11y/adapters/loki.rb +333 -0
- data/lib/e11y/adapters/otel_logs.rb +203 -0
- data/lib/e11y/adapters/registry.rb +141 -0
- data/lib/e11y/adapters/sentry.rb +230 -0
- data/lib/e11y/adapters/stdout.rb +108 -0
- data/lib/e11y/adapters/yabeda.rb +370 -0
- data/lib/e11y/buffers/adaptive_buffer.rb +339 -0
- data/lib/e11y/buffers/base_buffer.rb +40 -0
- data/lib/e11y/buffers/request_scoped_buffer.rb +246 -0
- data/lib/e11y/buffers/ring_buffer.rb +267 -0
- data/lib/e11y/buffers.rb +14 -0
- data/lib/e11y/console.rb +122 -0
- data/lib/e11y/current.rb +48 -0
- data/lib/e11y/event/base.rb +894 -0
- data/lib/e11y/event/value_sampling_config.rb +84 -0
- data/lib/e11y/events/base_audit_event.rb +43 -0
- data/lib/e11y/events/base_payment_event.rb +33 -0
- data/lib/e11y/events/rails/cache/delete.rb +21 -0
- data/lib/e11y/events/rails/cache/read.rb +23 -0
- data/lib/e11y/events/rails/cache/write.rb +22 -0
- data/lib/e11y/events/rails/database/query.rb +45 -0
- data/lib/e11y/events/rails/http/redirect.rb +21 -0
- data/lib/e11y/events/rails/http/request.rb +26 -0
- data/lib/e11y/events/rails/http/send_file.rb +21 -0
- data/lib/e11y/events/rails/http/start_processing.rb +26 -0
- data/lib/e11y/events/rails/job/completed.rb +22 -0
- data/lib/e11y/events/rails/job/enqueued.rb +22 -0
- data/lib/e11y/events/rails/job/failed.rb +22 -0
- data/lib/e11y/events/rails/job/scheduled.rb +23 -0
- data/lib/e11y/events/rails/job/started.rb +22 -0
- data/lib/e11y/events/rails/log.rb +56 -0
- data/lib/e11y/events/rails/view/render.rb +23 -0
- data/lib/e11y/events.rb +18 -0
- data/lib/e11y/instruments/active_job.rb +201 -0
- data/lib/e11y/instruments/rails_instrumentation.rb +141 -0
- data/lib/e11y/instruments/sidekiq.rb +175 -0
- data/lib/e11y/logger/bridge.rb +205 -0
- data/lib/e11y/metrics/cardinality_protection.rb +172 -0
- data/lib/e11y/metrics/cardinality_tracker.rb +134 -0
- data/lib/e11y/metrics/registry.rb +234 -0
- data/lib/e11y/metrics/relabeling.rb +226 -0
- data/lib/e11y/metrics.rb +102 -0
- data/lib/e11y/middleware/audit_signing.rb +174 -0
- data/lib/e11y/middleware/base.rb +140 -0
- data/lib/e11y/middleware/event_slo.rb +167 -0
- data/lib/e11y/middleware/pii_filter.rb +266 -0
- data/lib/e11y/middleware/pii_filtering.rb +280 -0
- data/lib/e11y/middleware/rate_limiting.rb +214 -0
- data/lib/e11y/middleware/request.rb +163 -0
- data/lib/e11y/middleware/routing.rb +157 -0
- data/lib/e11y/middleware/sampling.rb +254 -0
- data/lib/e11y/middleware/slo.rb +168 -0
- data/lib/e11y/middleware/trace_context.rb +131 -0
- data/lib/e11y/middleware/validation.rb +118 -0
- data/lib/e11y/middleware/versioning.rb +132 -0
- data/lib/e11y/middleware.rb +12 -0
- data/lib/e11y/pii/patterns.rb +90 -0
- data/lib/e11y/pii.rb +13 -0
- data/lib/e11y/pipeline/builder.rb +155 -0
- data/lib/e11y/pipeline/zone_validator.rb +110 -0
- data/lib/e11y/pipeline.rb +12 -0
- data/lib/e11y/presets/audit_event.rb +65 -0
- data/lib/e11y/presets/debug_event.rb +34 -0
- data/lib/e11y/presets/high_value_event.rb +51 -0
- data/lib/e11y/presets.rb +19 -0
- data/lib/e11y/railtie.rb +138 -0
- data/lib/e11y/reliability/circuit_breaker.rb +216 -0
- data/lib/e11y/reliability/dlq/file_storage.rb +277 -0
- data/lib/e11y/reliability/dlq/filter.rb +117 -0
- data/lib/e11y/reliability/retry_handler.rb +207 -0
- data/lib/e11y/reliability/retry_rate_limiter.rb +117 -0
- data/lib/e11y/sampling/error_spike_detector.rb +225 -0
- data/lib/e11y/sampling/load_monitor.rb +161 -0
- data/lib/e11y/sampling/stratified_tracker.rb +92 -0
- data/lib/e11y/sampling/value_extractor.rb +82 -0
- data/lib/e11y/self_monitoring/buffer_monitor.rb +79 -0
- data/lib/e11y/self_monitoring/performance_monitor.rb +97 -0
- data/lib/e11y/self_monitoring/reliability_monitor.rb +146 -0
- data/lib/e11y/slo/event_driven.rb +150 -0
- data/lib/e11y/slo/tracker.rb +119 -0
- data/lib/e11y/version.rb +9 -0
- data/lib/e11y.rb +283 -0
- metadata +452 -0
|
@@ -0,0 +1,4143 @@
|
|
|
1
|
+
# ADR-006: Security & Compliance
|
|
2
|
+
|
|
3
|
+
**Status:** Draft
|
|
4
|
+
**Date:** January 12, 2026
|
|
5
|
+
**Covers:** UC-007 (PII Filtering), UC-011 (Rate Limiting), UC-012 (Audit Trail)
|
|
6
|
+
**Depends On:** ADR-001 (Core Architecture), ADR-004 (Adapters)
|
|
7
|
+
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## 📋 Table of Contents
|
|
11
|
+
|
|
12
|
+
1. [Context & Problem](#1-context--problem)
|
|
13
|
+
2. [Architecture Overview](#2-architecture-overview)
|
|
14
|
+
3. [PII Filtering](#3-pii-filtering)
|
|
15
|
+
- 3.1. [Rails Integration](#31-rails-integration)
|
|
16
|
+
- 3.2. [Pattern-Based Filtering](#32-pattern-based-filtering)
|
|
17
|
+
- 3.3. [Deep Scanning](#33-deep-scanning)
|
|
18
|
+
- 3.4. [Per-Adapter Rules](#34-per-adapter-rules)
|
|
19
|
+
- 3.5. [Sampling for Debug](#35-sampling-for-debug)
|
|
20
|
+
4. [Rate Limiting](#4-rate-limiting)
|
|
21
|
+
- 4.1. [Global Rate Limiting](#41-global-rate-limiting)
|
|
22
|
+
- 4.2. [Per-Event Rate Limiting](#42-per-event-rate-limiting)
|
|
23
|
+
- 4.3. [Per-Context Rate Limiting](#43-per-context-rate-limiting)
|
|
24
|
+
- 4.4. [Redis Integration](#44-redis-integration)
|
|
25
|
+
5. [Audit Trail](#5-audit-trail)
|
|
26
|
+
- 5.1. [Immutable Events](#51-immutable-events)
|
|
27
|
+
- 5.2. [Cryptographic Signing](#52-cryptographic-signing)
|
|
28
|
+
- 5.3. [Compliance Features](#53-compliance-features)
|
|
29
|
+
- 5.4. [Tamper Detection](#54-tamper-detection)
|
|
30
|
+
- 5.5. [OpenTelemetry Baggage PII Protection (C08 Resolution)](#55-opentelemetry-baggage-pii-protection-c08-resolution) ⚠️ CRITICAL
|
|
31
|
+
- 5.5.1. The Problem: PII Leaking via OpenTelemetry Baggage
|
|
32
|
+
- 5.5.2. Decision: Block PII from Baggage Entirely
|
|
33
|
+
- 5.5.3. BaggageProtection Middleware Implementation
|
|
34
|
+
- 5.5.4. Configuration
|
|
35
|
+
- 5.5.5. Usage Examples
|
|
36
|
+
- 5.5.6. Strict Mode (Development/Staging)
|
|
37
|
+
- 5.5.7. Trade-offs & GDPR Compliance (C08)
|
|
38
|
+
- 5.6. [PII Handling for Event Replay from DLQ (C07 Resolution)](#56-pii-handling-for-event-replay-from-dlq-c07-resolution) ⚠️ HIGH
|
|
39
|
+
- 5.6.1. The Problem: Double-Hashing PII on Replay
|
|
40
|
+
- 5.6.2. Decision: Skip PII Filtering for Replayed Events
|
|
41
|
+
- 5.6.3. PiiFilter Middleware with Replay Detection
|
|
42
|
+
- 5.6.4. DLQ Replay Service with Metadata Flags
|
|
43
|
+
- 5.6.5. Configuration
|
|
44
|
+
- 5.6.6. Usage Examples
|
|
45
|
+
- 5.6.7. Idempotency Verification (Testing)
|
|
46
|
+
- 5.6.8. Trade-offs & Audit Trail Integrity (C07)
|
|
47
|
+
6. [GDPR Compliance](#6-gdpr-compliance)
|
|
48
|
+
7. [Configuration](#7-configuration)
|
|
49
|
+
8. [Testing](#8-testing)
|
|
50
|
+
9. [Trade-offs](#9-trade-offs)
|
|
51
|
+
|
|
52
|
+
---
|
|
53
|
+
|
|
54
|
+
## 1. Context & Problem
|
|
55
|
+
|
|
56
|
+
### 1.1. Problem Statement
|
|
57
|
+
|
|
58
|
+
**Current Pain Points:**
|
|
59
|
+
|
|
60
|
+
1. **PII Exposure:**
|
|
61
|
+
```ruby
|
|
62
|
+
# ❌ Sensitive data in logs
|
|
63
|
+
Events::UserLogin.track(
|
|
64
|
+
email: 'user@example.com', # PII!
|
|
65
|
+
password: 'secret123', # Credentials!
|
|
66
|
+
ip_address: '192.168.1.100' # PII!
|
|
67
|
+
)
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
2. **No Rate Limiting:**
|
|
71
|
+
```ruby
|
|
72
|
+
# ❌ Event flood can overwhelm system
|
|
73
|
+
1_000_000.times do
|
|
74
|
+
Events::Debug.track(message: 'spam')
|
|
75
|
+
end
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
3. **No Audit Trail:**
|
|
79
|
+
```ruby
|
|
80
|
+
# ❌ Can't prove what happened
|
|
81
|
+
Events::GdprDeletion.track(user_id: '123')
|
|
82
|
+
# Was this really tracked? Can it be tampered?
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
### 1.2. Goals
|
|
86
|
+
|
|
87
|
+
**Primary Goals:**
|
|
88
|
+
- ✅ Rails-compatible PII filtering
|
|
89
|
+
- ✅ Per-adapter filtering rules
|
|
90
|
+
- ✅ Global + per-event + per-context rate limiting
|
|
91
|
+
- ✅ Cryptographically-signed audit trail
|
|
92
|
+
- ✅ GDPR compliance ready
|
|
93
|
+
|
|
94
|
+
**Non-Goals:**
|
|
95
|
+
- ❌ Perfect PII detection (false positives OK)
|
|
96
|
+
- ❌ Zero-impact performance (small overhead acceptable)
|
|
97
|
+
- ❌ Full GDPR implementation (provide tools, not guarantee)
|
|
98
|
+
|
|
99
|
+
### 1.3. Success Metrics
|
|
100
|
+
|
|
101
|
+
| Metric | Target | Critical? |
|
|
102
|
+
|--------|--------|-----------|
|
|
103
|
+
| **PII filtering overhead** | <0.2ms per event | ✅ Yes |
|
|
104
|
+
| **Rate limit accuracy** | >99% | ✅ Yes |
|
|
105
|
+
| **Audit signature time** | <1ms | ✅ Yes |
|
|
106
|
+
| **False positive PII** | <5% | ⚠️ Important |
|
|
107
|
+
|
|
108
|
+
---
|
|
109
|
+
|
|
110
|
+
## 2. Architecture Overview
|
|
111
|
+
|
|
112
|
+
### 2.1. System Context
|
|
113
|
+
|
|
114
|
+
```mermaid
|
|
115
|
+
C4Context
|
|
116
|
+
title Security & Compliance System Context
|
|
117
|
+
|
|
118
|
+
Person(dev, "Developer", "Tracks events")
|
|
119
|
+
Person(auditor, "Auditor", "Reviews audit logs")
|
|
120
|
+
Person(dpo, "DPO", "GDPR compliance")
|
|
121
|
+
|
|
122
|
+
System(e11y, "E11y Gem", "Event tracking with security")
|
|
123
|
+
|
|
124
|
+
System_Ext(redis, "Redis", "Rate limiting state")
|
|
125
|
+
System_Ext(kms, "KMS", "Signing keys")
|
|
126
|
+
System_Ext(audit_store, "Audit Store", "Immutable audit logs")
|
|
127
|
+
|
|
128
|
+
Rel(dev, e11y, "Tracks events", "Events::OrderPaid.track(...)")
|
|
129
|
+
Rel(e11y, redis, "Check rate limits", "Redis commands")
|
|
130
|
+
Rel(e11y, kms, "Sign audit events", "HMAC-SHA256")
|
|
131
|
+
Rel(e11y, audit_store, "Store signed events", "Append-only")
|
|
132
|
+
|
|
133
|
+
Rel(auditor, audit_store, "Review logs", "Read-only")
|
|
134
|
+
Rel(dpo, e11y, "Configure PII rules", "Config")
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
### 2.2. Security Pipeline
|
|
138
|
+
|
|
139
|
+
```mermaid
|
|
140
|
+
graph TB
|
|
141
|
+
Event[Event.track] --> Pipeline[Middleware Chain]
|
|
142
|
+
|
|
143
|
+
Pipeline --> PII[PII Filter Middleware]
|
|
144
|
+
Pipeline --> Rate[Rate Limit Middleware]
|
|
145
|
+
Pipeline --> Audit[Audit Middleware]
|
|
146
|
+
|
|
147
|
+
subgraph "PII Filtering"
|
|
148
|
+
PII --> Rails[Rails Filters]
|
|
149
|
+
PII --> Pattern[Pattern Matching]
|
|
150
|
+
PII --> Deep[Deep Scan]
|
|
151
|
+
PII --> PerAdapter[Per-Adapter Rules]
|
|
152
|
+
end
|
|
153
|
+
|
|
154
|
+
subgraph "Rate Limiting"
|
|
155
|
+
Rate --> Global[Global Limiter]
|
|
156
|
+
Rate --> PerEvent[Per-Event Limiter]
|
|
157
|
+
Rate --> PerContext[Per-Context Limiter]
|
|
158
|
+
|
|
159
|
+
Global --> Redis1[Redis]
|
|
160
|
+
PerEvent --> Redis2[Redis]
|
|
161
|
+
PerContext --> Redis3[Redis]
|
|
162
|
+
end
|
|
163
|
+
|
|
164
|
+
subgraph "Audit Trail"
|
|
165
|
+
Audit --> Sign[Cryptographic Signing]
|
|
166
|
+
Audit --> Chain[Chain Verification]
|
|
167
|
+
Audit --> Store[Immutable Storage]
|
|
168
|
+
end
|
|
169
|
+
|
|
170
|
+
PII --> Adapters[Adapters]
|
|
171
|
+
Rate --> Adapters
|
|
172
|
+
Audit --> AuditAdapter[Audit Adapter Only]
|
|
173
|
+
|
|
174
|
+
style PII fill:#f8d7da
|
|
175
|
+
style Rate fill:#fff3cd
|
|
176
|
+
style Audit fill:#d1ecf1
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
### 2.3. Data Flow with Security
|
|
180
|
+
|
|
181
|
+
```mermaid
|
|
182
|
+
sequenceDiagram
|
|
183
|
+
participant App as Application
|
|
184
|
+
participant Event as Events::OrderPaid
|
|
185
|
+
participant PII as PII Filter
|
|
186
|
+
participant Rate as Rate Limiter
|
|
187
|
+
participant Audit as Audit Trail
|
|
188
|
+
participant Adapters as Adapters
|
|
189
|
+
|
|
190
|
+
App->>Event: .track(email: 'user@x.com', order_id: '123')
|
|
191
|
+
Event->>PII: filter(event_data)
|
|
192
|
+
|
|
193
|
+
PII->>PII: Check Rails filters
|
|
194
|
+
PII->>PII: Check pattern matchers
|
|
195
|
+
PII->>PII: Deep scan nested hashes
|
|
196
|
+
|
|
197
|
+
Note over PII: email: 'user@x.com' → '[FILTERED]'
|
|
198
|
+
|
|
199
|
+
PII->>Rate: process(filtered_data)
|
|
200
|
+
|
|
201
|
+
Rate->>Rate: Check global limit (10k/sec)
|
|
202
|
+
Rate->>Rate: Check event limit (100/sec for order.paid)
|
|
203
|
+
Rate->>Rate: Check context limit (10/sec per user)
|
|
204
|
+
|
|
205
|
+
alt Rate limit exceeded
|
|
206
|
+
Rate-->>Event: Drop event (return false)
|
|
207
|
+
else Within limits
|
|
208
|
+
Rate->>Audit: process(event_data)
|
|
209
|
+
|
|
210
|
+
alt Is audit event
|
|
211
|
+
Audit->>Audit: Sign with HMAC-SHA256
|
|
212
|
+
Audit->>Audit: Chain to previous
|
|
213
|
+
Audit->>Adapters: route(signed_event)
|
|
214
|
+
else Regular event
|
|
215
|
+
Audit->>Adapters: route(event_data)
|
|
216
|
+
end
|
|
217
|
+
end
|
|
218
|
+
```
|
|
219
|
+
|
|
220
|
+
---
|
|
221
|
+
|
|
222
|
+
## 3. PII Filtering
|
|
223
|
+
|
|
224
|
+
### 3.0. PII Filtering Strategy
|
|
225
|
+
|
|
226
|
+
**Critical Design Decision:** Explicit opt-in, per-adapter filtering to minimize performance overhead.
|
|
227
|
+
|
|
228
|
+
#### 3.0.1. The Performance Problem
|
|
229
|
+
|
|
230
|
+
**Problem:** Filtering ALL events by default = massive overhead!
|
|
231
|
+
|
|
232
|
+
```ruby
|
|
233
|
+
# ❌ BAD: Filter every event (expensive!)
|
|
234
|
+
1,000,000 events/day × 0.2ms filtering = 200 seconds CPU/day = waste!
|
|
235
|
+
|
|
236
|
+
# ✅ GOOD: Filter only PII events
|
|
237
|
+
100,000 PII events/day × 0.2ms = 20 seconds CPU/day = acceptable
|
|
238
|
+
```
|
|
239
|
+
|
|
240
|
+
#### 3.0.2. Three-Tier Filtering Strategy
|
|
241
|
+
|
|
242
|
+
E11y uses a **3-tier approach** to balance security and performance:
|
|
243
|
+
|
|
244
|
+
| Tier | Strategy | Cost | Use Case | Events/sec |
|
|
245
|
+
|------|----------|------|----------|------------|
|
|
246
|
+
| **Tier 1** | Skip filtering | 0ms | Health checks, metrics, internal events | 500 |
|
|
247
|
+
| **Tier 2** | Rails filters only | ~0.05ms | Standard events (known PII keys) | 400 |
|
|
248
|
+
| **Tier 3** | Deep filtering | ~0.2ms | User data, payments, complex nested | 100 |
|
|
249
|
+
|
|
250
|
+
**Performance Budget:**
|
|
251
|
+
```
|
|
252
|
+
500 events/sec × 0ms = 0ms CPU/sec (Tier 1)
|
|
253
|
+
400 events/sec × 0.05ms = 20ms CPU/sec (Tier 2)
|
|
254
|
+
100 events/sec × 0.2ms = 20ms CPU/sec (Tier 3)
|
|
255
|
+
---
|
|
256
|
+
Total: 40ms CPU/sec = 4% CPU on single core ✅
|
|
257
|
+
```
|
|
258
|
+
|
|
259
|
+
#### 3.0.3. Explicit PII Declaration
|
|
260
|
+
|
|
261
|
+
**Rule:** Event class MUST declare PII handling explicitly.
|
|
262
|
+
|
|
263
|
+
**Option A: No PII (Skip Filtering)**
|
|
264
|
+
```ruby
|
|
265
|
+
class Events::HealthCheck < E11y::Event::Base
|
|
266
|
+
schema do
|
|
267
|
+
required(:status).filled(:string)
|
|
268
|
+
end
|
|
269
|
+
|
|
270
|
+
# ✅ Explicit: This event contains NO PII
|
|
271
|
+
contains_pii false # Skip all PII filtering (Tier 1)
|
|
272
|
+
end
|
|
273
|
+
```
|
|
274
|
+
|
|
275
|
+
**Option B: Default (Rails Filters Only)**
|
|
276
|
+
```ruby
|
|
277
|
+
class Events::OrderCreated < E11y::Event::Base
|
|
278
|
+
schema do
|
|
279
|
+
required(:order_id).filled(:string)
|
|
280
|
+
required(:amount).filled(:float)
|
|
281
|
+
end
|
|
282
|
+
|
|
283
|
+
# No declaration → Rails filters applied (Tier 2, default)
|
|
284
|
+
# Uses: Rails.application.config.filter_parameters
|
|
285
|
+
end
|
|
286
|
+
```
|
|
287
|
+
|
|
288
|
+
**Option C: Explicit PII (Deep Filtering)**
|
|
289
|
+
```ruby
|
|
290
|
+
class Events::UserRegistered < E11y::Event::Base
|
|
291
|
+
schema do
|
|
292
|
+
required(:email).filled(:string)
|
|
293
|
+
required(:password).filled(:string)
|
|
294
|
+
required(:address).filled(:hash)
|
|
295
|
+
required(:user_id).filled(:string)
|
|
296
|
+
end
|
|
297
|
+
|
|
298
|
+
# ✅ Explicit: This event contains PII
|
|
299
|
+
contains_pii true # Tier 3: Deep filtering
|
|
300
|
+
|
|
301
|
+
pii_filtering do
|
|
302
|
+
# ✅ MANDATORY: Explicit declaration for EACH schema field
|
|
303
|
+
# Linter will FAIL if any field is missing!
|
|
304
|
+
|
|
305
|
+
field :email do
|
|
306
|
+
strategy :hash # Pseudonymize for searchability
|
|
307
|
+
exclude_adapters [:file_audit] # Audit needs original (GDPR Art. 6(1)(c))
|
|
308
|
+
end
|
|
309
|
+
|
|
310
|
+
field :password do
|
|
311
|
+
strategy :mask # Full mask everywhere
|
|
312
|
+
# No exclude_adapters → applies to ALL adapters
|
|
313
|
+
end
|
|
314
|
+
|
|
315
|
+
field :address do
|
|
316
|
+
strategy :redact # Remove completely
|
|
317
|
+
exclude_adapters [:file_audit]
|
|
318
|
+
end
|
|
319
|
+
|
|
320
|
+
field :user_id do
|
|
321
|
+
strategy :skip # Not PII, no filtering
|
|
322
|
+
end
|
|
323
|
+
|
|
324
|
+
# ❌ If you forget a field → Linter FAILS at boot:
|
|
325
|
+
# "Missing PII declaration for: [:forgotten_field]"
|
|
326
|
+
end
|
|
327
|
+
end
|
|
328
|
+
```
|
|
329
|
+
|
|
330
|
+
**Option D: Audit Events (Simplified)**
|
|
331
|
+
```ruby
|
|
332
|
+
class Events::GdprDeletion < E11y::AuditEvent
|
|
333
|
+
schema do
|
|
334
|
+
required(:user_id).filled(:string)
|
|
335
|
+
required(:email).filled(:string)
|
|
336
|
+
required(:reason).filled(:string)
|
|
337
|
+
required(:deleted_at).filled(:time)
|
|
338
|
+
end
|
|
339
|
+
|
|
340
|
+
contains_pii true
|
|
341
|
+
|
|
342
|
+
pii_filtering do
|
|
343
|
+
# ✅ AuditEvent automatically excludes :file_audit for all fields
|
|
344
|
+
|
|
345
|
+
field :user_id do
|
|
346
|
+
strategy :skip # Not PII
|
|
347
|
+
end
|
|
348
|
+
|
|
349
|
+
field :email do
|
|
350
|
+
strategy :hash # For non-audit adapters (Elasticsearch, Loki)
|
|
351
|
+
# exclude_adapters [:file_audit] ← AUTOMATIC for AuditEvent
|
|
352
|
+
end
|
|
353
|
+
|
|
354
|
+
field :reason do
|
|
355
|
+
strategy :skip # Not PII
|
|
356
|
+
end
|
|
357
|
+
|
|
358
|
+
field :deleted_at do
|
|
359
|
+
strategy :skip # Not PII
|
|
360
|
+
end
|
|
361
|
+
end
|
|
362
|
+
end
|
|
363
|
+
|
|
364
|
+
# Result:
|
|
365
|
+
# - file_audit: { email: 'user@example.com' } # Original (GDPR Art. 6(1)(c))
|
|
366
|
+
# - elasticsearch: { email: 'a1b2c3d4...' } # Hashed
|
|
367
|
+
# - loki: { email: 'a1b2c3d4...' } # Hashed
|
|
368
|
+
# - sentry: { email: 'a1b2c3d4...' } # Hashed
|
|
369
|
+
```
|
|
370
|
+
|
|
371
|
+
#### 3.0.4. Filtering Strategies
|
|
372
|
+
|
|
373
|
+
| Strategy | Example | Use Case | Reversible? | Cost |
|
|
374
|
+
|----------|---------|----------|-------------|------|
|
|
375
|
+
| **:skip** | `user@example.com` | Audit trail (compliance) | N/A | 0ms |
|
|
376
|
+
| **:mask** | `[FILTERED]` | External services (Sentry) | ❌ No | 0.01ms |
|
|
377
|
+
| **:hash** | `a1b2c3d4e5f6...` | Pseudonymization (searchable) | ❌ No | 0.05ms |
|
|
378
|
+
| **:truncate** | `192.168.x.x` | Partial masking (logs) | ❌ No | 0.02ms |
|
|
379
|
+
| **:redact** | *(removed)* | Complete removal | ❌ No | 0.01ms |
|
|
380
|
+
| **:encrypt** | `encrypted_blob` | Searchable encryption | ✅ Yes (with key) | 0.15ms |
|
|
381
|
+
|
|
382
|
+
#### 3.0.5. PII Declaration Linter
|
|
383
|
+
|
|
384
|
+
**Critical:** Linter ENFORCES explicit declaration for every field when `contains_pii true`.
|
|
385
|
+
|
|
386
|
+
```ruby
|
|
387
|
+
# lib/e11y/linters/pii_declaration_linter.rb
|
|
388
|
+
module E11y
|
|
389
|
+
module Linters
|
|
390
|
+
class PiiDeclarationLinter
|
|
391
|
+
def self.validate_all!
|
|
392
|
+
E11y::Registry.all_events.each do |event_class|
|
|
393
|
+
validate!(event_class)
|
|
394
|
+
end
|
|
395
|
+
end
|
|
396
|
+
|
|
397
|
+
def self.validate!(event_class)
|
|
398
|
+
return unless event_class.contains_pii?
|
|
399
|
+
|
|
400
|
+
# Get declared fields from pii_filtering block
|
|
401
|
+
declared_fields = event_class.pii_filtering_config&.fields&.keys || []
|
|
402
|
+
|
|
403
|
+
# Get all fields from schema
|
|
404
|
+
schema_fields = event_class.schema.keys
|
|
405
|
+
|
|
406
|
+
# Check for missing declarations
|
|
407
|
+
missing = schema_fields - declared_fields
|
|
408
|
+
|
|
409
|
+
if missing.any?
|
|
410
|
+
raise PiiDeclarationError, build_error_message(event_class, missing)
|
|
411
|
+
end
|
|
412
|
+
|
|
413
|
+
# Validate each field has valid strategy
|
|
414
|
+
declared_fields.each do |field|
|
|
415
|
+
config = event_class.pii_filtering_config.fields[field]
|
|
416
|
+
validate_field_config!(event_class, field, config)
|
|
417
|
+
end
|
|
418
|
+
end
|
|
419
|
+
|
|
420
|
+
private
|
|
421
|
+
|
|
422
|
+
def self.build_error_message(event_class, missing_fields)
|
|
423
|
+
<<~ERROR
|
|
424
|
+
❌ PII Declaration Error: #{event_class.name}
|
|
425
|
+
|
|
426
|
+
Event declared `contains_pii true` but missing field declarations:
|
|
427
|
+
|
|
428
|
+
Missing fields: #{missing_fields.map(&:inspect).join(', ')}
|
|
429
|
+
|
|
430
|
+
Fix: Add explicit PII strategy for each field in pii_filtering block:
|
|
431
|
+
|
|
432
|
+
class #{event_class.name} < E11y::Event::Base
|
|
433
|
+
contains_pii true
|
|
434
|
+
|
|
435
|
+
pii_filtering do
|
|
436
|
+
#{missing_fields.map { |f| "field :#{f} do\n strategy :mask # or :hash, :skip, :redact\n end" }.join("\n ")}
|
|
437
|
+
end
|
|
438
|
+
end
|
|
439
|
+
|
|
440
|
+
Available strategies: :skip, :mask, :hash, :redact, :truncate, :encrypt
|
|
441
|
+
ERROR
|
|
442
|
+
end
|
|
443
|
+
|
|
444
|
+
def self.validate_field_config!(event_class, field, config)
|
|
445
|
+
valid_strategies = [:skip, :mask, :hash, :redact, :truncate, :encrypt]
|
|
446
|
+
|
|
447
|
+
unless valid_strategies.include?(config[:strategy])
|
|
448
|
+
raise PiiDeclarationError, <<~ERROR
|
|
449
|
+
❌ Invalid PII strategy for #{event_class.name}##{field}
|
|
450
|
+
|
|
451
|
+
Strategy: #{config[:strategy].inspect}
|
|
452
|
+
Valid strategies: #{valid_strategies.map(&:inspect).join(', ')}
|
|
453
|
+
ERROR
|
|
454
|
+
end
|
|
455
|
+
|
|
456
|
+
# Validate exclude_adapters if present
|
|
457
|
+
if config[:exclude_adapters]
|
|
458
|
+
unless config[:exclude_adapters].is_a?(Array)
|
|
459
|
+
raise PiiDeclarationError, "exclude_adapters must be an array for #{event_class.name}##{field}"
|
|
460
|
+
end
|
|
461
|
+
end
|
|
462
|
+
end
|
|
463
|
+
end
|
|
464
|
+
|
|
465
|
+
class PiiDeclarationError < StandardError; end
|
|
466
|
+
end
|
|
467
|
+
end
|
|
468
|
+
```
|
|
469
|
+
|
|
470
|
+
**Rake Task:**
|
|
471
|
+
```ruby
|
|
472
|
+
# lib/tasks/e11y_pii.rake
|
|
473
|
+
namespace :e11y do
|
|
474
|
+
namespace :lint do
|
|
475
|
+
desc 'Validate PII declarations for all events'
|
|
476
|
+
task pii: :environment do
|
|
477
|
+
puts "Checking PII declarations..."
|
|
478
|
+
puts "=" * 80
|
|
479
|
+
|
|
480
|
+
errors = []
|
|
481
|
+
warnings = []
|
|
482
|
+
|
|
483
|
+
E11y::Registry.all_events.each do |event_class|
|
|
484
|
+
begin
|
|
485
|
+
E11y::Linters::PiiDeclarationLinter.validate!(event_class)
|
|
486
|
+
|
|
487
|
+
if event_class.contains_pii?
|
|
488
|
+
puts "✅ #{event_class.name} - All #{event_class.schema.keys.size} fields declared"
|
|
489
|
+
else
|
|
490
|
+
puts "⚪ #{event_class.name} - No PII (skipped)"
|
|
491
|
+
end
|
|
492
|
+
rescue E11y::Linters::PiiDeclarationError => error
|
|
493
|
+
errors << error.message
|
|
494
|
+
puts "❌ #{event_class.name}"
|
|
495
|
+
end
|
|
496
|
+
end
|
|
497
|
+
|
|
498
|
+
puts "=" * 80
|
|
499
|
+
|
|
500
|
+
if errors.any?
|
|
501
|
+
puts "\n❌ ERRORS:\n\n"
|
|
502
|
+
errors.each { |e| puts e; puts "\n" }
|
|
503
|
+
exit 1
|
|
504
|
+
else
|
|
505
|
+
puts "\n✅ All PII declarations are valid"
|
|
506
|
+
exit 0
|
|
507
|
+
end
|
|
508
|
+
end
|
|
509
|
+
end
|
|
510
|
+
end
|
|
511
|
+
```
|
|
512
|
+
|
|
513
|
+
**Usage:**
|
|
514
|
+
```bash
|
|
515
|
+
# Validate all events
|
|
516
|
+
$ bundle exec rake e11y:lint:pii
|
|
517
|
+
|
|
518
|
+
Checking PII declarations...
|
|
519
|
+
================================================================================
|
|
520
|
+
✅ Events::OrderCreated - All 3 fields declared
|
|
521
|
+
✅ Events::PaymentProcessed - All 5 fields declared
|
|
522
|
+
❌ Events::UserRegistered
|
|
523
|
+
⚪ Events::HealthCheck - No PII (skipped)
|
|
524
|
+
================================================================================
|
|
525
|
+
|
|
526
|
+
❌ ERRORS:
|
|
527
|
+
|
|
528
|
+
❌ PII Declaration Error: Events::UserRegistered
|
|
529
|
+
|
|
530
|
+
Event declared `contains_pii true` but missing field declarations:
|
|
531
|
+
|
|
532
|
+
Missing fields: :ip_address, :user_agent
|
|
533
|
+
|
|
534
|
+
Fix: Add explicit PII strategy for each field in pii_filtering block:
|
|
535
|
+
|
|
536
|
+
class Events::UserRegistered < E11y::Event::Base
|
|
537
|
+
contains_pii true
|
|
538
|
+
|
|
539
|
+
pii_filtering do
|
|
540
|
+
field :ip_address do
|
|
541
|
+
strategy :mask # or :hash, :skip, :redact
|
|
542
|
+
end
|
|
543
|
+
field :user_agent do
|
|
544
|
+
strategy :mask # or :hash, :skip, :redact
|
|
545
|
+
end
|
|
546
|
+
end
|
|
547
|
+
end
|
|
548
|
+
|
|
549
|
+
Available strategies: :skip, :mask, :hash, :redact, :truncate, :encrypt
|
|
550
|
+
```
|
|
551
|
+
|
|
552
|
+
**Boot-time Validation:**
|
|
553
|
+
```ruby
|
|
554
|
+
# config/initializers/e11y.rb
|
|
555
|
+
E11y.configure do |config|
|
|
556
|
+
# ... other config ...
|
|
557
|
+
|
|
558
|
+
# Validate PII declarations at boot (in development/test)
|
|
559
|
+
if Rails.env.development? || Rails.env.test?
|
|
560
|
+
config.after_initialize do
|
|
561
|
+
E11y::Linters::PiiDeclarationLinter.validate_all!
|
|
562
|
+
end
|
|
563
|
+
end
|
|
564
|
+
end
|
|
565
|
+
```
|
|
566
|
+
|
|
567
|
+
#### 3.0.6. Default Behavior
|
|
568
|
+
|
|
569
|
+
**If `contains_pii` not specified:**
|
|
570
|
+
- ✅ Apply Rails filters only (Tier 2)
|
|
571
|
+
- ✅ Fast (0.05ms overhead)
|
|
572
|
+
- ✅ Covers 90% of use cases (password, token, secret, api_key)
|
|
573
|
+
- ⚠️ No linter validation (assumes you know what you're doing)
|
|
574
|
+
|
|
575
|
+
**Recommended approach:**
|
|
576
|
+
```ruby
|
|
577
|
+
# For most events: don't specify (default is good)
|
|
578
|
+
class Events::OrderPaid < E11y::Event::Base
|
|
579
|
+
# No contains_pii declaration
|
|
580
|
+
# → Rails filters applied automatically
|
|
581
|
+
# → No linter validation
|
|
582
|
+
end
|
|
583
|
+
|
|
584
|
+
# Only for internal/metrics: opt-out
|
|
585
|
+
class Events::HealthCheck < E11y::Event::Base
|
|
586
|
+
contains_pii false # Explicit opt-out
|
|
587
|
+
# → No filtering, no validation
|
|
588
|
+
end
|
|
589
|
+
|
|
590
|
+
# For sensitive data: opt-in + explicit declarations
|
|
591
|
+
class Events::UserLogin < E11y::Event::Base
|
|
592
|
+
contains_pii true # ✅ Triggers linter validation
|
|
593
|
+
|
|
594
|
+
pii_filtering do
|
|
595
|
+
# ✅ MUST declare ALL schema fields
|
|
596
|
+
field :email do
|
|
597
|
+
strategy :hash
|
|
598
|
+
exclude_adapters [:file_audit]
|
|
599
|
+
end
|
|
600
|
+
|
|
601
|
+
field :password do
|
|
602
|
+
strategy :mask
|
|
603
|
+
end
|
|
604
|
+
|
|
605
|
+
field :ip_address do
|
|
606
|
+
strategy :mask
|
|
607
|
+
end
|
|
608
|
+
|
|
609
|
+
field :session_id do
|
|
610
|
+
strategy :skip # Not PII
|
|
611
|
+
end
|
|
612
|
+
end
|
|
613
|
+
end
|
|
614
|
+
```
|
|
615
|
+
|
|
616
|
+
#### 3.0.6. Critical Design Decisions from Conflict Analysis
|
|
617
|
+
|
|
618
|
+
**From CONFLICT-ANALYSIS.md:**
|
|
619
|
+
|
|
620
|
+
1. **Conflict #4: Audit Trail + PII Filtering**
|
|
621
|
+
- ✅ Decision: PII filtering happens **per-adapter** (not globally)
|
|
622
|
+
- Rationale: Audit needs original data (GDPR Art. 6(1)(c)), Sentry needs masked
|
|
623
|
+
- Implementation: Each adapter applies its own PII rules
|
|
624
|
+
|
|
625
|
+
2. **Conflict #3: PII Filtering + OpenTelemetry**
|
|
626
|
+
- ✅ Decision: Per-adapter PII rules
|
|
627
|
+
- OTel: pseudonymize (one-way hash)
|
|
628
|
+
- Audit: skip filtering (compliance)
|
|
629
|
+
- Sentry: strict masking (external service)
|
|
630
|
+
|
|
631
|
+
3. **Pipeline Order (UPDATED):**
|
|
632
|
+
```
|
|
633
|
+
1. Schema Validation
|
|
634
|
+
2. Context Enrichment
|
|
635
|
+
3. Rate Limiting (moved before PII for efficiency)
|
|
636
|
+
4. Adaptive Sampling
|
|
637
|
+
5. Audit Signing (BEFORE filtering, preserves original signature)
|
|
638
|
+
6. Adapter Routing
|
|
639
|
+
└─→ PII Filtering (per-adapter, different rules per destination)
|
|
640
|
+
```
|
|
641
|
+
|
|
642
|
+
**Key Insight:** PII filtering is **NOT a global middleware** — it's applied **inside each adapter** with different rules!
|
|
643
|
+
|
|
644
|
+
### 3.1. Rails Integration
|
|
645
|
+
|
|
646
|
+
**Design Decision:** Use `Rails.application.config.filter_parameters` as base.
|
|
647
|
+
|
|
648
|
+
```ruby
|
|
649
|
+
module E11y
|
|
650
|
+
module Security
|
|
651
|
+
class PiiFilter
|
|
652
|
+
def initialize(config)
|
|
653
|
+
@rails_filters = extract_rails_filters
|
|
654
|
+
@custom_patterns = config.patterns || []
|
|
655
|
+
@custom_filters = config.custom_filters || []
|
|
656
|
+
@allowlist = config.allowlist || []
|
|
657
|
+
@deep_scan = config.deep_scan || true
|
|
658
|
+
end
|
|
659
|
+
|
|
660
|
+
def filter(event_data)
|
|
661
|
+
filtered_data = event_data.dup
|
|
662
|
+
|
|
663
|
+
# Apply Rails filters
|
|
664
|
+
filtered_data[:payload] = apply_rails_filters(filtered_data[:payload])
|
|
665
|
+
|
|
666
|
+
# Apply custom patterns
|
|
667
|
+
filtered_data[:payload] = apply_pattern_filters(filtered_data[:payload])
|
|
668
|
+
|
|
669
|
+
# Apply custom filter functions
|
|
670
|
+
filtered_data[:payload] = apply_custom_filters(filtered_data[:payload])
|
|
671
|
+
|
|
672
|
+
# Deep scan if enabled
|
|
673
|
+
if @deep_scan
|
|
674
|
+
filtered_data[:payload] = deep_scan(filtered_data[:payload])
|
|
675
|
+
end
|
|
676
|
+
|
|
677
|
+
filtered_data
|
|
678
|
+
end
|
|
679
|
+
|
|
680
|
+
private
|
|
681
|
+
|
|
682
|
+
def extract_rails_filters
|
|
683
|
+
return [] unless defined?(Rails)
|
|
684
|
+
|
|
685
|
+
Rails.application.config.filter_parameters || []
|
|
686
|
+
end
|
|
687
|
+
|
|
688
|
+
def apply_rails_filters(payload)
|
|
689
|
+
return payload if @rails_filters.empty?
|
|
690
|
+
|
|
691
|
+
# Use Rails' parameter filter
|
|
692
|
+
filter = ActiveSupport::ParameterFilter.new(@rails_filters)
|
|
693
|
+
filter.filter(payload)
|
|
694
|
+
end
|
|
695
|
+
end
|
|
696
|
+
end
|
|
697
|
+
end
|
|
698
|
+
```
|
|
699
|
+
|
|
700
|
+
**Example:**
|
|
701
|
+
|
|
702
|
+
```ruby
|
|
703
|
+
# config/initializers/filter_parameters.rb
|
|
704
|
+
Rails.application.config.filter_parameters += [
|
|
705
|
+
:password,
|
|
706
|
+
:password_confirmation,
|
|
707
|
+
:secret,
|
|
708
|
+
:token,
|
|
709
|
+
:api_key,
|
|
710
|
+
:credit_card_number,
|
|
711
|
+
:ssn
|
|
712
|
+
]
|
|
713
|
+
|
|
714
|
+
# E11y automatically uses these filters!
|
|
715
|
+
Events::UserLogin.track(
|
|
716
|
+
email: 'user@example.com',
|
|
717
|
+
password: 'secret123' # ← Automatically filtered by Rails
|
|
718
|
+
)
|
|
719
|
+
|
|
720
|
+
# Result in logs:
|
|
721
|
+
# { email: 'user@example.com', password: '[FILTERED]' }
|
|
722
|
+
```
|
|
723
|
+
|
|
724
|
+
---
|
|
725
|
+
|
|
726
|
+
### 3.2. Pattern-Based Filtering
|
|
727
|
+
|
|
728
|
+
**Design Decision:** Regex patterns for content scanning.
|
|
729
|
+
|
|
730
|
+
```ruby
|
|
731
|
+
module E11y
|
|
732
|
+
module Security
|
|
733
|
+
class PatternMatcher
|
|
734
|
+
BUILT_IN_PATTERNS = {
|
|
735
|
+
email: /\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b/,
|
|
736
|
+
credit_card: /\b(?:\d{4}[-\s]?){3}\d{4}\b/,
|
|
737
|
+
ssn: /\b\d{3}-\d{2}-\d{4}\b/,
|
|
738
|
+
phone: /\b\d{3}[-.]?\d{3}[-.]?\d{4}\b/,
|
|
739
|
+
ip_address: /\b(?:\d{1,3}\.){3}\d{1,3}\b/,
|
|
740
|
+
jwt: /\beyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\b/,
|
|
741
|
+
api_key: /\b[A-Za-z0-9]{32,}\b/
|
|
742
|
+
}.freeze
|
|
743
|
+
|
|
744
|
+
def initialize(enabled_patterns: BUILT_IN_PATTERNS.keys, custom_patterns: {})
|
|
745
|
+
@patterns = BUILT_IN_PATTERNS.slice(*enabled_patterns)
|
|
746
|
+
@patterns.merge!(custom_patterns)
|
|
747
|
+
end
|
|
748
|
+
|
|
749
|
+
def filter(value)
|
|
750
|
+
return value unless value.is_a?(String)
|
|
751
|
+
|
|
752
|
+
filtered_value = value.dup
|
|
753
|
+
|
|
754
|
+
@patterns.each do |name, pattern|
|
|
755
|
+
filtered_value.gsub!(pattern, "[FILTERED:#{name.upcase}]")
|
|
756
|
+
end
|
|
757
|
+
|
|
758
|
+
filtered_value
|
|
759
|
+
end
|
|
760
|
+
|
|
761
|
+
def contains_pii?(value)
|
|
762
|
+
return false unless value.is_a?(String)
|
|
763
|
+
|
|
764
|
+
@patterns.any? { |_, pattern| value.match?(pattern) }
|
|
765
|
+
end
|
|
766
|
+
end
|
|
767
|
+
end
|
|
768
|
+
end
|
|
769
|
+
```
|
|
770
|
+
|
|
771
|
+
**Example:**
|
|
772
|
+
|
|
773
|
+
```ruby
|
|
774
|
+
E11y.configure do |config|
|
|
775
|
+
config.security.pii_filtering do
|
|
776
|
+
# Enable built-in patterns
|
|
777
|
+
patterns [:email, :credit_card, :ssn, :phone, :ip_address]
|
|
778
|
+
|
|
779
|
+
# Add custom patterns
|
|
780
|
+
custom_pattern :internal_id, /\bINT-\d{6,}\b/
|
|
781
|
+
custom_pattern :license_plate, /\b[A-Z]{3}-\d{4}\b/
|
|
782
|
+
end
|
|
783
|
+
end
|
|
784
|
+
|
|
785
|
+
# Usage:
|
|
786
|
+
Events::CustomerSupport.track(
|
|
787
|
+
message: "Customer email is user@example.com, IP: 192.168.1.1"
|
|
788
|
+
)
|
|
789
|
+
|
|
790
|
+
# Result:
|
|
791
|
+
# {
|
|
792
|
+
# message: "Customer email is [FILTERED:EMAIL], IP: [FILTERED:IP_ADDRESS]"
|
|
793
|
+
# }
|
|
794
|
+
```
|
|
795
|
+
|
|
796
|
+
---
|
|
797
|
+
|
|
798
|
+
### 3.3. Deep Scanning
|
|
799
|
+
|
|
800
|
+
**Design Decision:** Recursively scan nested hashes and arrays.
|
|
801
|
+
|
|
802
|
+
```ruby
|
|
803
|
+
module E11y
|
|
804
|
+
module Security
|
|
805
|
+
class DeepScanner
|
|
806
|
+
def initialize(pattern_matcher, allowlist: [])
|
|
807
|
+
@pattern_matcher = pattern_matcher
|
|
808
|
+
@allowlist = allowlist
|
|
809
|
+
end
|
|
810
|
+
|
|
811
|
+
def scan(value, key_path: [])
|
|
812
|
+
case value
|
|
813
|
+
when Hash
|
|
814
|
+
scan_hash(value, key_path)
|
|
815
|
+
when Array
|
|
816
|
+
scan_array(value, key_path)
|
|
817
|
+
when String
|
|
818
|
+
scan_string(value, key_path)
|
|
819
|
+
else
|
|
820
|
+
value
|
|
821
|
+
end
|
|
822
|
+
end
|
|
823
|
+
|
|
824
|
+
private
|
|
825
|
+
|
|
826
|
+
def scan_hash(hash, key_path)
|
|
827
|
+
hash.transform_values do |v|
|
|
828
|
+
new_key_path = key_path + [hash.keys.find { |k| hash[k] == v }]
|
|
829
|
+
|
|
830
|
+
# Skip if in allowlist
|
|
831
|
+
next v if allowlisted?(new_key_path)
|
|
832
|
+
|
|
833
|
+
scan(v, key_path: new_key_path)
|
|
834
|
+
end
|
|
835
|
+
end
|
|
836
|
+
|
|
837
|
+
def scan_array(array, key_path)
|
|
838
|
+
array.map.with_index do |item, index|
|
|
839
|
+
scan(item, key_path: key_path + [index])
|
|
840
|
+
end
|
|
841
|
+
end
|
|
842
|
+
|
|
843
|
+
def scan_string(string, key_path)
|
|
844
|
+
return string if allowlisted?(key_path)
|
|
845
|
+
|
|
846
|
+
@pattern_matcher.filter(string)
|
|
847
|
+
end
|
|
848
|
+
|
|
849
|
+
def allowlisted?(key_path)
|
|
850
|
+
key_path_string = key_path.join('.')
|
|
851
|
+
|
|
852
|
+
@allowlist.any? do |pattern|
|
|
853
|
+
case pattern
|
|
854
|
+
when String
|
|
855
|
+
key_path_string == pattern
|
|
856
|
+
when Regexp
|
|
857
|
+
key_path_string.match?(pattern)
|
|
858
|
+
else
|
|
859
|
+
false
|
|
860
|
+
end
|
|
861
|
+
end
|
|
862
|
+
end
|
|
863
|
+
end
|
|
864
|
+
end
|
|
865
|
+
end
|
|
866
|
+
```
|
|
867
|
+
|
|
868
|
+
**Example:**
|
|
869
|
+
|
|
870
|
+
```ruby
|
|
871
|
+
E11y.configure do |config|
|
|
872
|
+
config.security.pii_filtering do
|
|
873
|
+
deep_scan true
|
|
874
|
+
|
|
875
|
+
# Allowlist certain paths
|
|
876
|
+
allowlist [
|
|
877
|
+
'order.id', # Exact match
|
|
878
|
+
/^metadata\.internal_/, # Regex match
|
|
879
|
+
'debug.raw_request' # Development debugging
|
|
880
|
+
]
|
|
881
|
+
end
|
|
882
|
+
end
|
|
883
|
+
|
|
884
|
+
# Deep nested structure:
|
|
885
|
+
Events::OrderCreated.track(
|
|
886
|
+
order: {
|
|
887
|
+
id: 'ORD-123',
|
|
888
|
+
customer: {
|
|
889
|
+
email: 'user@example.com', # ← Will be filtered
|
|
890
|
+
phone: '555-1234' # ← Will be filtered
|
|
891
|
+
},
|
|
892
|
+
metadata: {
|
|
893
|
+
internal_ref: 'INT-999999', # ← Allowlisted, kept
|
|
894
|
+
note: 'Contact at user@example.com' # ← Will be filtered
|
|
895
|
+
}
|
|
896
|
+
}
|
|
897
|
+
)
|
|
898
|
+
|
|
899
|
+
# Result:
|
|
900
|
+
# {
|
|
901
|
+
# order: {
|
|
902
|
+
# id: 'ORD-123',
|
|
903
|
+
# customer: {
|
|
904
|
+
# email: '[FILTERED:EMAIL]',
|
|
905
|
+
# phone: '[FILTERED:PHONE]'
|
|
906
|
+
# },
|
|
907
|
+
# metadata: {
|
|
908
|
+
# internal_ref: 'INT-999999', # Kept
|
|
909
|
+
# note: 'Contact at [FILTERED:EMAIL]'
|
|
910
|
+
# }
|
|
911
|
+
# }
|
|
912
|
+
# }
|
|
913
|
+
```
|
|
914
|
+
|
|
915
|
+
---
|
|
916
|
+
|
|
917
|
+
### 3.4. Per-Adapter Rules (Architecture)
|
|
918
|
+
|
|
919
|
+
**Critical Design Decision:** PII filtering happens **inside each adapter**, not globally.
|
|
920
|
+
|
|
921
|
+
#### 3.4.1. Why Per-Adapter Filtering?
|
|
922
|
+
|
|
923
|
+
**Problem with Global Filtering:**
|
|
924
|
+
```ruby
|
|
925
|
+
# ❌ BAD: Filter once globally
|
|
926
|
+
event_data = { email: 'user@example.com', ip: '192.168.1.1' }
|
|
927
|
+
filtered_event = global_pii_filter(event_data)
|
|
928
|
+
# → { email: '[FILTERED]', ip: '[FILTERED]' }
|
|
929
|
+
|
|
930
|
+
# Now ALL adapters get filtered data
|
|
931
|
+
send_to_audit_file(filtered_event) # ❌ Audit needs ORIGINAL email!
|
|
932
|
+
send_to_sentry(filtered_event) # ✅ Sentry gets filtered (correct)
|
|
933
|
+
send_to_loki(filtered_event) # ❌ Loki could have kept hashed email
|
|
934
|
+
```
|
|
935
|
+
|
|
936
|
+
**Solution: Filter Per-Adapter:**
|
|
937
|
+
```ruby
|
|
938
|
+
# ✅ GOOD: Each adapter applies its own rules
|
|
939
|
+
adapters.each do |adapter|
|
|
940
|
+
# Adapter decides filtering strategy
|
|
941
|
+
adapter_filtered_event = adapter.apply_pii_rules(event_data)
|
|
942
|
+
adapter.write(adapter_filtered_event)
|
|
943
|
+
end
|
|
944
|
+
|
|
945
|
+
# Result:
|
|
946
|
+
# - Audit File: { email: 'user@example.com', ip: '192.168.1.1' } (original)
|
|
947
|
+
# - Sentry: { email: '[FILTERED]', ip: '[FILTERED]' } (masked)
|
|
948
|
+
# - Loki: { email: 'a1b2c3d4...', ip: '192.168.x.x' } (hashed/truncated)
|
|
949
|
+
# - Elasticsearch: { email: 'a1b2c3d4...', ip: '192.168.1.1' } (email hashed only)
|
|
950
|
+
```
|
|
951
|
+
|
|
952
|
+
**Architecture Diagram:**
|
|
953
|
+
|
|
954
|
+
```mermaid
|
|
955
|
+
graph TB
|
|
956
|
+
Event[Event Data<br/>email: user@x.com<br/>ip: 192.168.1.1] --> Router[Adapter Router]
|
|
957
|
+
|
|
958
|
+
Router --> Audit[Audit Adapter]
|
|
959
|
+
Router --> Sentry[Sentry Adapter]
|
|
960
|
+
Router --> Loki[Loki Adapter]
|
|
961
|
+
Router --> ES[Elasticsearch Adapter]
|
|
962
|
+
|
|
963
|
+
subgraph "Per-Adapter PII Filtering"
|
|
964
|
+
Audit --> AuditFilter[PII Filter: SKIP<br/>mode: :skip]
|
|
965
|
+
Sentry --> SentryFilter[PII Filter: MASK<br/>mode: :mask]
|
|
966
|
+
Loki --> LokiFilter[PII Filter: CUSTOM<br/>mode: :custom]
|
|
967
|
+
ES --> ESFilter[PII Filter: HASH<br/>mode: :hash]
|
|
968
|
+
end
|
|
969
|
+
|
|
970
|
+
AuditFilter --> AuditStore[Audit File<br/>email: user@x.com<br/>ip: 192.168.1.1]
|
|
971
|
+
SentryFilter --> SentryAPI[Sentry<br/>email: [FILTERED]<br/>ip: [FILTERED]]
|
|
972
|
+
LokiFilter --> LokiAPI[Loki<br/>email: [FILTERED]<br/>ip: 192.168.x.x]
|
|
973
|
+
ESFilter --> ESAPI[Elasticsearch<br/>email: a1b2c3d4...<br/>ip: 192.168.1.1]
|
|
974
|
+
|
|
975
|
+
style AuditFilter fill:#d4edda
|
|
976
|
+
style SentryFilter fill:#f8d7da
|
|
977
|
+
style LokiFilter fill:#fff3cd
|
|
978
|
+
style ESFilter fill:#d1ecf1
|
|
979
|
+
```
|
|
980
|
+
|
|
981
|
+
#### 3.4.2. Implementation
|
|
982
|
+
|
|
983
|
+
```ruby
|
|
984
|
+
module E11y
|
|
985
|
+
module Adapters
|
|
986
|
+
class Base
|
|
987
|
+
def write(event_data)
|
|
988
|
+
# Apply adapter-specific PII filtering
|
|
989
|
+
filtered_data = apply_pii_rules(event_data)
|
|
990
|
+
|
|
991
|
+
# Write filtered data
|
|
992
|
+
write_impl(filtered_data)
|
|
993
|
+
end
|
|
994
|
+
|
|
995
|
+
private
|
|
996
|
+
|
|
997
|
+
def apply_pii_rules(event_data)
|
|
998
|
+
# Get event class
|
|
999
|
+
event_class = E11y::Registry.get_class(event_data[:event_name])
|
|
1000
|
+
|
|
1001
|
+
# Check if event has per-adapter rules
|
|
1002
|
+
return event_data unless event_class.respond_to?(:pii_filtering_for_adapter)
|
|
1003
|
+
|
|
1004
|
+
# Get rules for THIS adapter
|
|
1005
|
+
rules = event_class.pii_filtering_for_adapter(adapter_name)
|
|
1006
|
+
|
|
1007
|
+
# Apply filtering based on rules
|
|
1008
|
+
case rules[:strategy]
|
|
1009
|
+
when :skip
|
|
1010
|
+
event_data # No filtering
|
|
1011
|
+
when :mask
|
|
1012
|
+
mask_pii_fields(event_data, rules[:fields])
|
|
1013
|
+
when :hash
|
|
1014
|
+
hash_pii_fields(event_data, rules[:fields])
|
|
1015
|
+
when :custom
|
|
1016
|
+
rules[:filter].call(event_data)
|
|
1017
|
+
else
|
|
1018
|
+
# Fallback to default Rails filters
|
|
1019
|
+
apply_rails_filters(event_data)
|
|
1020
|
+
end
|
|
1021
|
+
end
|
|
1022
|
+
|
|
1023
|
+
def mask_pii_fields(event_data, fields)
|
|
1024
|
+
filtered = event_data.dup
|
|
1025
|
+
fields.each do |field|
|
|
1026
|
+
deep_mask(filtered[:payload], field)
|
|
1027
|
+
end
|
|
1028
|
+
filtered
|
|
1029
|
+
end
|
|
1030
|
+
|
|
1031
|
+
def hash_pii_fields(event_data, fields)
|
|
1032
|
+
filtered = event_data.dup
|
|
1033
|
+
fields.each do |field|
|
|
1034
|
+
deep_hash(filtered[:payload], field)
|
|
1035
|
+
end
|
|
1036
|
+
filtered
|
|
1037
|
+
end
|
|
1038
|
+
end
|
|
1039
|
+
end
|
|
1040
|
+
end
|
|
1041
|
+
```
|
|
1042
|
+
|
|
1043
|
+
#### 3.4.3. Event Class Declaration (Simplified)
|
|
1044
|
+
|
|
1045
|
+
```ruby
|
|
1046
|
+
class Events::UserLogin < E11y::Event::Base
|
|
1047
|
+
schema do
|
|
1048
|
+
required(:email).filled(:string)
|
|
1049
|
+
required(:ip_address).filled(:string)
|
|
1050
|
+
required(:session_id).filled(:string)
|
|
1051
|
+
end
|
|
1052
|
+
|
|
1053
|
+
contains_pii true
|
|
1054
|
+
|
|
1055
|
+
pii_filtering do
|
|
1056
|
+
# ✅ Simple per-field declaration
|
|
1057
|
+
field :email do
|
|
1058
|
+
strategy :hash # Apply to all adapters
|
|
1059
|
+
exclude_adapters [:file_audit] # Except audit (needs original)
|
|
1060
|
+
hash_algorithm 'SHA256' # Optional: specify algorithm
|
|
1061
|
+
end
|
|
1062
|
+
|
|
1063
|
+
field :ip_address do
|
|
1064
|
+
strategy :mask # Mask for all adapters
|
|
1065
|
+
exclude_adapters [:file_audit]
|
|
1066
|
+
end
|
|
1067
|
+
|
|
1068
|
+
field :session_id do
|
|
1069
|
+
strategy :skip # Not PII, no filtering
|
|
1070
|
+
end
|
|
1071
|
+
end
|
|
1072
|
+
end
|
|
1073
|
+
|
|
1074
|
+
# Result across adapters:
|
|
1075
|
+
#
|
|
1076
|
+
# file_audit: { email: 'user@example.com', ip: '192.168.1.1', session_id: 's123' }
|
|
1077
|
+
# elasticsearch: { email: 'a1b2c3d4...', ip: '[FILTERED]', session_id: 's123' }
|
|
1078
|
+
# loki: { email: 'a1b2c3d4...', ip: '[FILTERED]', session_id: 's123' }
|
|
1079
|
+
# sentry: { email: 'a1b2c3d4...', ip: '[FILTERED]', session_id: 's123' }
|
|
1080
|
+
```
|
|
1081
|
+
|
|
1082
|
+
**For complex per-adapter rules (advanced use case):**
|
|
1083
|
+
```ruby
|
|
1084
|
+
class Events::UserLogin < E11y::Event::Base
|
|
1085
|
+
contains_pii true
|
|
1086
|
+
|
|
1087
|
+
pii_filtering do
|
|
1088
|
+
field :ip_address do
|
|
1089
|
+
# Different strategy per adapter (advanced)
|
|
1090
|
+
strategy :mask # Default for most adapters
|
|
1091
|
+
|
|
1092
|
+
# Custom strategy for specific adapter
|
|
1093
|
+
custom_for_adapter :loki do
|
|
1094
|
+
# Truncate IP instead of full mask
|
|
1095
|
+
->(value) { value.split('.')[0..2].join('.') + '.x' }
|
|
1096
|
+
end
|
|
1097
|
+
|
|
1098
|
+
exclude_adapters [:file_audit]
|
|
1099
|
+
end
|
|
1100
|
+
end
|
|
1101
|
+
end
|
|
1102
|
+
|
|
1103
|
+
# Result:
|
|
1104
|
+
# loki: { ip: '192.168.1.x' } # Custom truncate
|
|
1105
|
+
# sentry: { ip: '[FILTERED]' } # Default mask
|
|
1106
|
+
# elasticsearch: { ip: '[FILTERED]' } # Default mask
|
|
1107
|
+
```
|
|
1108
|
+
|
|
1109
|
+
#### 3.4.4. Configuration API (Rails-Style DSL)
|
|
1110
|
+
|
|
1111
|
+
```ruby
|
|
1112
|
+
module E11y
|
|
1113
|
+
module Event
|
|
1114
|
+
class Base
|
|
1115
|
+
class << self
|
|
1116
|
+
# DSL for PII filtering (Rails-style)
|
|
1117
|
+
def pii_filtering(&block)
|
|
1118
|
+
@pii_filtering_config = PiiFilteringConfig.new(self)
|
|
1119
|
+
@pii_filtering_config.instance_eval(&block)
|
|
1120
|
+
end
|
|
1121
|
+
|
|
1122
|
+
def pii_filtering_config
|
|
1123
|
+
@pii_filtering_config
|
|
1124
|
+
end
|
|
1125
|
+
|
|
1126
|
+
# Get PII rules for specific field and adapter
|
|
1127
|
+
def pii_rule_for_field(field_name, adapter_name)
|
|
1128
|
+
return nil unless @pii_filtering_config
|
|
1129
|
+
|
|
1130
|
+
field_config = @pii_filtering_config.fields[field_name]
|
|
1131
|
+
return nil unless field_config
|
|
1132
|
+
|
|
1133
|
+
# Check if this adapter is excluded
|
|
1134
|
+
if field_config[:exclude_adapters]&.include?(adapter_name)
|
|
1135
|
+
return { strategy: :skip }
|
|
1136
|
+
end
|
|
1137
|
+
|
|
1138
|
+
# Check if only specific adapters allowed
|
|
1139
|
+
if field_config[:only_adapters] && !field_config[:only_adapters].include?(adapter_name)
|
|
1140
|
+
return { strategy: :skip }
|
|
1141
|
+
end
|
|
1142
|
+
|
|
1143
|
+
# Check for custom adapter-specific rule
|
|
1144
|
+
if field_config[:custom_adapters] && field_config[:custom_adapters][adapter_name]
|
|
1145
|
+
return {
|
|
1146
|
+
strategy: :custom,
|
|
1147
|
+
filter: field_config[:custom_adapters][adapter_name]
|
|
1148
|
+
}
|
|
1149
|
+
end
|
|
1150
|
+
|
|
1151
|
+
# Return default strategy for this field
|
|
1152
|
+
{
|
|
1153
|
+
strategy: field_config[:strategy],
|
|
1154
|
+
hash_algorithm: field_config[:hash_algorithm],
|
|
1155
|
+
truncate_to: field_config[:truncate_to],
|
|
1156
|
+
custom_filter: field_config[:custom_filter]
|
|
1157
|
+
}.compact
|
|
1158
|
+
end
|
|
1159
|
+
end
|
|
1160
|
+
end
|
|
1161
|
+
|
|
1162
|
+
class PiiFilteringConfig
|
|
1163
|
+
attr_reader :fields
|
|
1164
|
+
|
|
1165
|
+
def initialize(event_class)
|
|
1166
|
+
@event_class = event_class
|
|
1167
|
+
@fields = {}
|
|
1168
|
+
end
|
|
1169
|
+
|
|
1170
|
+
# ===================================================================
|
|
1171
|
+
# RAILS-STYLE DSL: Single-line shortcuts
|
|
1172
|
+
# ===================================================================
|
|
1173
|
+
|
|
1174
|
+
# Shortcut: masks(*fields)
|
|
1175
|
+
# Like validates :name, :email, presence: true
|
|
1176
|
+
def masks(*field_names, **options)
|
|
1177
|
+
field_names.each do |field_name|
|
|
1178
|
+
field(field_name) do
|
|
1179
|
+
strategy :mask
|
|
1180
|
+
apply_options(options)
|
|
1181
|
+
end
|
|
1182
|
+
end
|
|
1183
|
+
end
|
|
1184
|
+
|
|
1185
|
+
# Shortcut: hashes(*fields)
|
|
1186
|
+
def hashes(*field_names, **options)
|
|
1187
|
+
field_names.each do |field_name|
|
|
1188
|
+
field(field_name) do
|
|
1189
|
+
strategy :hash
|
|
1190
|
+
apply_options(options)
|
|
1191
|
+
end
|
|
1192
|
+
end
|
|
1193
|
+
end
|
|
1194
|
+
|
|
1195
|
+
# Shortcut: redacts(*fields)
|
|
1196
|
+
def redacts(*field_names, **options)
|
|
1197
|
+
field_names.each do |field_name|
|
|
1198
|
+
field(field_name) do
|
|
1199
|
+
strategy :redact
|
|
1200
|
+
apply_options(options)
|
|
1201
|
+
end
|
|
1202
|
+
end
|
|
1203
|
+
end
|
|
1204
|
+
|
|
1205
|
+
# Shortcut: skips(*fields) - not PII
|
|
1206
|
+
def skips(*field_names, **options)
|
|
1207
|
+
field_names.each do |field_name|
|
|
1208
|
+
field(field_name) do
|
|
1209
|
+
strategy :skip
|
|
1210
|
+
apply_options(options)
|
|
1211
|
+
end
|
|
1212
|
+
end
|
|
1213
|
+
end
|
|
1214
|
+
|
|
1215
|
+
# Shortcut: truncates(*fields)
|
|
1216
|
+
def truncates(*field_names, to:, **options)
|
|
1217
|
+
field_names.each do |field_name|
|
|
1218
|
+
field(field_name) do
|
|
1219
|
+
strategy :truncate
|
|
1220
|
+
truncate_to to
|
|
1221
|
+
apply_options(options)
|
|
1222
|
+
end
|
|
1223
|
+
end
|
|
1224
|
+
end
|
|
1225
|
+
|
|
1226
|
+
# ===================================================================
|
|
1227
|
+
# CONDITIONAL FILTERING (Rails-style)
|
|
1228
|
+
# ===================================================================
|
|
1229
|
+
|
|
1230
|
+
# Like validates :admin, presence: true, if: :admin?
|
|
1231
|
+
def masks_if(condition, *field_names, **options)
|
|
1232
|
+
field_names.each do |field_name|
|
|
1233
|
+
field(field_name) do
|
|
1234
|
+
strategy :mask
|
|
1235
|
+
only_if condition
|
|
1236
|
+
apply_options(options)
|
|
1237
|
+
end
|
|
1238
|
+
end
|
|
1239
|
+
end
|
|
1240
|
+
|
|
1241
|
+
def hashes_if(condition, *field_names, **options)
|
|
1242
|
+
field_names.each do |field_name|
|
|
1243
|
+
field(field_name) do
|
|
1244
|
+
strategy :hash
|
|
1245
|
+
only_if condition
|
|
1246
|
+
apply_options(options)
|
|
1247
|
+
end
|
|
1248
|
+
end
|
|
1249
|
+
end
|
|
1250
|
+
|
|
1251
|
+
# ===================================================================
|
|
1252
|
+
# GROUPING (Rails-style)
|
|
1253
|
+
# ===================================================================
|
|
1254
|
+
|
|
1255
|
+
# Like with_options model: Post do ... end
|
|
1256
|
+
def with_strategy(strategy, &block)
|
|
1257
|
+
@current_strategy = strategy
|
|
1258
|
+
instance_eval(&block)
|
|
1259
|
+
@current_strategy = nil
|
|
1260
|
+
end
|
|
1261
|
+
|
|
1262
|
+
def with_options(**options, &block)
|
|
1263
|
+
@current_options = options
|
|
1264
|
+
instance_eval(&block)
|
|
1265
|
+
@current_options = nil
|
|
1266
|
+
end
|
|
1267
|
+
|
|
1268
|
+
# ===================================================================
|
|
1269
|
+
# FIELD DECLARATION
|
|
1270
|
+
# ===================================================================
|
|
1271
|
+
|
|
1272
|
+
def field(name, strategy: nil, **options, &block)
|
|
1273
|
+
config = FieldConfig.new(name, @event_class)
|
|
1274
|
+
|
|
1275
|
+
# Apply current context (from with_strategy/with_options)
|
|
1276
|
+
config.strategy(@current_strategy) if @current_strategy
|
|
1277
|
+
config.apply_options(@current_options) if @current_options
|
|
1278
|
+
|
|
1279
|
+
# Apply inline options
|
|
1280
|
+
config.strategy(strategy) if strategy
|
|
1281
|
+
config.apply_options(options)
|
|
1282
|
+
|
|
1283
|
+
# Apply block if provided
|
|
1284
|
+
config.instance_eval(&block) if block_given?
|
|
1285
|
+
|
|
1286
|
+
@fields[name] = config.to_h
|
|
1287
|
+
end
|
|
1288
|
+
|
|
1289
|
+
# ===================================================================
|
|
1290
|
+
# BULK OPERATIONS (Rails-style)
|
|
1291
|
+
# ===================================================================
|
|
1292
|
+
|
|
1293
|
+
# Like validates_presence_of :name, :email
|
|
1294
|
+
def masks_all_except(*exceptions)
|
|
1295
|
+
schema_fields = @event_class.schema.keys
|
|
1296
|
+
(schema_fields - exceptions).each do |field_name|
|
|
1297
|
+
masks field_name
|
|
1298
|
+
end
|
|
1299
|
+
end
|
|
1300
|
+
|
|
1301
|
+
def hashes_all_except(*exceptions, **options)
|
|
1302
|
+
schema_fields = @event_class.schema.keys
|
|
1303
|
+
(schema_fields - exceptions).each do |field_name|
|
|
1304
|
+
hashes field_name, **options
|
|
1305
|
+
end
|
|
1306
|
+
end
|
|
1307
|
+
end
|
|
1308
|
+
|
|
1309
|
+
class FieldConfig
|
|
1310
|
+
attr_reader :field_name
|
|
1311
|
+
|
|
1312
|
+
def initialize(field_name, event_class)
|
|
1313
|
+
@field_name = field_name
|
|
1314
|
+
@event_class = event_class
|
|
1315
|
+
@config = {}
|
|
1316
|
+
end
|
|
1317
|
+
|
|
1318
|
+
# ===================================================================
|
|
1319
|
+
# CORE METHODS
|
|
1320
|
+
# ===================================================================
|
|
1321
|
+
|
|
1322
|
+
def strategy(value)
|
|
1323
|
+
@config[:strategy] = value
|
|
1324
|
+
end
|
|
1325
|
+
|
|
1326
|
+
def exclude_adapters(adapters)
|
|
1327
|
+
@config[:exclude_adapters] = Array(adapters)
|
|
1328
|
+
end
|
|
1329
|
+
|
|
1330
|
+
def only_adapters(adapters)
|
|
1331
|
+
@config[:only_adapters] = Array(adapters)
|
|
1332
|
+
end
|
|
1333
|
+
|
|
1334
|
+
def hash_algorithm(algo)
|
|
1335
|
+
@config[:hash_algorithm] = algo
|
|
1336
|
+
end
|
|
1337
|
+
|
|
1338
|
+
def truncate_to(length)
|
|
1339
|
+
@config[:truncate_to] = length
|
|
1340
|
+
end
|
|
1341
|
+
|
|
1342
|
+
def custom_for_adapter(adapter_name, &block)
|
|
1343
|
+
@config[:custom_adapters] ||= {}
|
|
1344
|
+
@config[:custom_adapters][adapter_name] = block
|
|
1345
|
+
end
|
|
1346
|
+
|
|
1347
|
+
def custom_filter(&block)
|
|
1348
|
+
@config[:custom_filter] = block
|
|
1349
|
+
end
|
|
1350
|
+
|
|
1351
|
+
# ===================================================================
|
|
1352
|
+
# CONDITIONAL FILTERING (Rails-style)
|
|
1353
|
+
# ===================================================================
|
|
1354
|
+
|
|
1355
|
+
def only_if(condition)
|
|
1356
|
+
@config[:only_if] = condition
|
|
1357
|
+
end
|
|
1358
|
+
|
|
1359
|
+
def unless(condition)
|
|
1360
|
+
@config[:unless] = condition
|
|
1361
|
+
end
|
|
1362
|
+
|
|
1363
|
+
# ===================================================================
|
|
1364
|
+
# SHORTCUTS (Rails-style chainable)
|
|
1365
|
+
# ===================================================================
|
|
1366
|
+
|
|
1367
|
+
def except(*adapters)
|
|
1368
|
+
exclude_adapters(adapters)
|
|
1369
|
+
end
|
|
1370
|
+
|
|
1371
|
+
def only(*adapters)
|
|
1372
|
+
only_adapters(adapters)
|
|
1373
|
+
end
|
|
1374
|
+
|
|
1375
|
+
def with_algorithm(algo)
|
|
1376
|
+
hash_algorithm(algo)
|
|
1377
|
+
end
|
|
1378
|
+
|
|
1379
|
+
# ===================================================================
|
|
1380
|
+
# OPTIONS HELPERS
|
|
1381
|
+
# ===================================================================
|
|
1382
|
+
|
|
1383
|
+
def apply_options(options)
|
|
1384
|
+
return unless options
|
|
1385
|
+
|
|
1386
|
+
exclude_adapters(options[:except]) if options[:except]
|
|
1387
|
+
only_adapters(options[:only]) if options[:only]
|
|
1388
|
+
hash_algorithm(options[:algorithm]) if options[:algorithm]
|
|
1389
|
+
truncate_to(options[:to]) if options[:to]
|
|
1390
|
+
only_if(options[:if]) if options[:if]
|
|
1391
|
+
self.unless(options[:unless]) if options[:unless]
|
|
1392
|
+
end
|
|
1393
|
+
|
|
1394
|
+
def to_h
|
|
1395
|
+
@config
|
|
1396
|
+
end
|
|
1397
|
+
end
|
|
1398
|
+
end
|
|
1399
|
+
end
|
|
1400
|
+
|
|
1401
|
+
# AuditEvent: Automatically exclude :file_audit for all fields
|
|
1402
|
+
module E11y
|
|
1403
|
+
class AuditEvent < Event::Base
|
|
1404
|
+
class << self
|
|
1405
|
+
def pii_rule_for_field(field_name, adapter_name)
|
|
1406
|
+
rule = super
|
|
1407
|
+
|
|
1408
|
+
# Override: AuditEvent always skips filtering for :file_audit
|
|
1409
|
+
if adapter_name == :file_audit
|
|
1410
|
+
return { strategy: :skip }
|
|
1411
|
+
end
|
|
1412
|
+
|
|
1413
|
+
rule
|
|
1414
|
+
end
|
|
1415
|
+
end
|
|
1416
|
+
end
|
|
1417
|
+
end
|
|
1418
|
+
```
|
|
1419
|
+
|
|
1420
|
+
**Configuration:**
|
|
1421
|
+
|
|
1422
|
+
```ruby
|
|
1423
|
+
E11y.configure do |config|
|
|
1424
|
+
config.security.pii_filtering do
|
|
1425
|
+
# Global rules (apply to all adapters by default)
|
|
1426
|
+
patterns [:email, :phone, :ssn]
|
|
1427
|
+
|
|
1428
|
+
# Per-adapter overrides
|
|
1429
|
+
adapter_rules do
|
|
1430
|
+
# Audit logs: keep original data (internal only)
|
|
1431
|
+
adapter :file_audit, mode: :skip
|
|
1432
|
+
|
|
1433
|
+
# Sentry: extra strict filtering
|
|
1434
|
+
adapter :sentry, mode: :strict, additional_patterns: {
|
|
1435
|
+
user_agent: /Mozilla.*\(.*/,
|
|
1436
|
+
url_params: /\?.*$/
|
|
1437
|
+
}
|
|
1438
|
+
|
|
1439
|
+
# Loki: custom filter
|
|
1440
|
+
adapter :loki, mode: :custom, filter: ->(event_data) {
|
|
1441
|
+
# Keep emails for internal Loki, but mask credit cards
|
|
1442
|
+
filtered = event_data.dup
|
|
1443
|
+
filtered[:payload] = mask_credit_cards(event_data[:payload])
|
|
1444
|
+
filtered
|
|
1445
|
+
}
|
|
1446
|
+
end
|
|
1447
|
+
end
|
|
1448
|
+
end
|
|
1449
|
+
```
|
|
1450
|
+
|
|
1451
|
+
**Use Case Diagram:**
|
|
1452
|
+
|
|
1453
|
+
```mermaid
|
|
1454
|
+
graph TB
|
|
1455
|
+
Event[Event Data] --> Router[Adapter Router]
|
|
1456
|
+
|
|
1457
|
+
Router --> Audit[Audit Adapter]
|
|
1458
|
+
Router --> Sentry[Sentry Adapter]
|
|
1459
|
+
Router --> Loki[Loki Adapter]
|
|
1460
|
+
|
|
1461
|
+
subgraph "PII Filtering"
|
|
1462
|
+
Audit --> Skip[Skip Filtering<br/>mode: :skip]
|
|
1463
|
+
Sentry --> Strict[Strict Filtering<br/>mode: :strict]
|
|
1464
|
+
Loki --> Custom[Custom Filter<br/>mode: :custom]
|
|
1465
|
+
end
|
|
1466
|
+
|
|
1467
|
+
Skip --> AuditStore[Full Data<br/>email: user@x.com]
|
|
1468
|
+
Strict --> SentryAPI[Heavily Filtered<br/>email: [FILTERED]<br/>user_agent: [FILTERED]]
|
|
1469
|
+
Custom --> LokiAPI[Custom Filtered<br/>email: user@x.com<br/>cc: [FILTERED]]
|
|
1470
|
+
|
|
1471
|
+
style Skip fill:#d4edda
|
|
1472
|
+
style Strict fill:#f8d7da
|
|
1473
|
+
style Custom fill:#fff3cd
|
|
1474
|
+
```
|
|
1475
|
+
|
|
1476
|
+
---
|
|
1477
|
+
|
|
1478
|
+
### 3.5. Sampling for Debug
|
|
1479
|
+
|
|
1480
|
+
**Design Decision:** Sample filtered values for debugging false positives.
|
|
1481
|
+
|
|
1482
|
+
```ruby
|
|
1483
|
+
module E11y
|
|
1484
|
+
module Security
|
|
1485
|
+
class FilterSampler
|
|
1486
|
+
def initialize(sample_rate: 0.01, max_samples: 100)
|
|
1487
|
+
@sample_rate = sample_rate
|
|
1488
|
+
@max_samples = max_samples
|
|
1489
|
+
@samples = []
|
|
1490
|
+
@mutex = Mutex.new
|
|
1491
|
+
end
|
|
1492
|
+
|
|
1493
|
+
def record_filter(key_path, original_value, filtered_value)
|
|
1494
|
+
return unless should_sample?
|
|
1495
|
+
|
|
1496
|
+
@mutex.synchronize do
|
|
1497
|
+
return if @samples.size >= @max_samples
|
|
1498
|
+
|
|
1499
|
+
@samples << {
|
|
1500
|
+
key_path: key_path.join('.'),
|
|
1501
|
+
original: truncate(original_value),
|
|
1502
|
+
filtered: filtered_value,
|
|
1503
|
+
timestamp: Time.now
|
|
1504
|
+
}
|
|
1505
|
+
end
|
|
1506
|
+
end
|
|
1507
|
+
|
|
1508
|
+
def samples
|
|
1509
|
+
@mutex.synchronize { @samples.dup }
|
|
1510
|
+
end
|
|
1511
|
+
|
|
1512
|
+
def clear!
|
|
1513
|
+
@mutex.synchronize { @samples.clear }
|
|
1514
|
+
end
|
|
1515
|
+
|
|
1516
|
+
private
|
|
1517
|
+
|
|
1518
|
+
def should_sample?
|
|
1519
|
+
rand < @sample_rate
|
|
1520
|
+
end
|
|
1521
|
+
|
|
1522
|
+
def truncate(value, max_length: 100)
|
|
1523
|
+
str = value.to_s
|
|
1524
|
+
str.length > max_length ? "#{str[0...max_length]}..." : str
|
|
1525
|
+
end
|
|
1526
|
+
end
|
|
1527
|
+
end
|
|
1528
|
+
end
|
|
1529
|
+
```
|
|
1530
|
+
|
|
1531
|
+
**Usage:**
|
|
1532
|
+
|
|
1533
|
+
```ruby
|
|
1534
|
+
# Check sampled filters in console
|
|
1535
|
+
E11y.security.filter_sampler.samples
|
|
1536
|
+
# [
|
|
1537
|
+
# {
|
|
1538
|
+
# key_path: 'customer.email',
|
|
1539
|
+
# original: 'user@example.com',
|
|
1540
|
+
# filtered: '[FILTERED:EMAIL]',
|
|
1541
|
+
# timestamp: 2026-01-12 10:30:00 UTC
|
|
1542
|
+
# },
|
|
1543
|
+
# ...
|
|
1544
|
+
# ]
|
|
1545
|
+
|
|
1546
|
+
# Clear samples
|
|
1547
|
+
E11y.security.filter_sampler.clear!
|
|
1548
|
+
```
|
|
1549
|
+
|
|
1550
|
+
#### 3.4.5. Rails-Style DSL Examples
|
|
1551
|
+
|
|
1552
|
+
**Pattern 1: Single-line shortcuts (like Rails validations)**
|
|
1553
|
+
|
|
1554
|
+
```ruby
|
|
1555
|
+
class Events::UserRegistered < E11y::Event::Base
|
|
1556
|
+
schema do
|
|
1557
|
+
required(:email).filled(:string)
|
|
1558
|
+
required(:password).filled(:string)
|
|
1559
|
+
required(:ip_address).filled(:string)
|
|
1560
|
+
required(:user_id).filled(:string)
|
|
1561
|
+
end
|
|
1562
|
+
|
|
1563
|
+
contains_pii true
|
|
1564
|
+
|
|
1565
|
+
pii_filtering do
|
|
1566
|
+
# ✅ Rails-style: masks :password, :ip_address
|
|
1567
|
+
masks :password, :ip_address
|
|
1568
|
+
|
|
1569
|
+
# ✅ Rails-style: hashes :email, except: [:file_audit]
|
|
1570
|
+
hashes :email, except: [:file_audit]
|
|
1571
|
+
|
|
1572
|
+
# ✅ Not PII
|
|
1573
|
+
skips :user_id
|
|
1574
|
+
end
|
|
1575
|
+
end
|
|
1576
|
+
```
|
|
1577
|
+
|
|
1578
|
+
**Pattern 2: with_options (grouping)**
|
|
1579
|
+
|
|
1580
|
+
```ruby
|
|
1581
|
+
class Events::PaymentProcessed < E11y::Event::Base
|
|
1582
|
+
contains_pii true
|
|
1583
|
+
|
|
1584
|
+
pii_filtering do
|
|
1585
|
+
# ✅ Like Rails: with_options validatable: true do
|
|
1586
|
+
with_options except: [:file_audit] do
|
|
1587
|
+
hashes :email, :user_id
|
|
1588
|
+
masks :ip_address
|
|
1589
|
+
end
|
|
1590
|
+
|
|
1591
|
+
# Always mask (no exceptions)
|
|
1592
|
+
masks :credit_card_last4, :cvv
|
|
1593
|
+
|
|
1594
|
+
skips :payment_id, :amount
|
|
1595
|
+
end
|
|
1596
|
+
end
|
|
1597
|
+
```
|
|
1598
|
+
|
|
1599
|
+
**Pattern 3: with_strategy (bulk)**
|
|
1600
|
+
|
|
1601
|
+
```ruby
|
|
1602
|
+
class Events::GdprExport < E11y::Event::Base
|
|
1603
|
+
contains_pii true
|
|
1604
|
+
|
|
1605
|
+
pii_filtering do
|
|
1606
|
+
# ✅ All hashed except audit
|
|
1607
|
+
with_strategy :hash do
|
|
1608
|
+
with_options except: [:file_audit] do
|
|
1609
|
+
field :email
|
|
1610
|
+
field :phone_number
|
|
1611
|
+
field :address
|
|
1612
|
+
end
|
|
1613
|
+
end
|
|
1614
|
+
|
|
1615
|
+
# Sensitive data → always mask
|
|
1616
|
+
with_strategy :mask do
|
|
1617
|
+
field :ssn
|
|
1618
|
+
field :credit_card
|
|
1619
|
+
end
|
|
1620
|
+
|
|
1621
|
+
skips :user_id, :export_id
|
|
1622
|
+
end
|
|
1623
|
+
end
|
|
1624
|
+
```
|
|
1625
|
+
|
|
1626
|
+
**Pattern 4: Conditional filtering (if/unless)**
|
|
1627
|
+
|
|
1628
|
+
```ruby
|
|
1629
|
+
class Events::AdminAction < E11y::Event::Base
|
|
1630
|
+
contains_pii true
|
|
1631
|
+
|
|
1632
|
+
pii_filtering do
|
|
1633
|
+
# ✅ Like Rails: validates :admin_notes, presence: true, if: :admin?
|
|
1634
|
+
masks_if ->(payload) { payload[:level] == 'admin' }, :admin_notes
|
|
1635
|
+
|
|
1636
|
+
# Hash user data unless it's a system user
|
|
1637
|
+
hashes_if ->(payload) { !payload[:is_system] }, :email, :ip_address
|
|
1638
|
+
|
|
1639
|
+
skips :action_type, :timestamp
|
|
1640
|
+
end
|
|
1641
|
+
end
|
|
1642
|
+
```
|
|
1643
|
+
|
|
1644
|
+
**Pattern 5: Custom per-adapter (advanced)**
|
|
1645
|
+
|
|
1646
|
+
```ruby
|
|
1647
|
+
class Events::UserLogin < E11y::Event::Base
|
|
1648
|
+
contains_pii true
|
|
1649
|
+
|
|
1650
|
+
pii_filtering do
|
|
1651
|
+
# Default: hash
|
|
1652
|
+
hashes :email, except: [:file_audit]
|
|
1653
|
+
|
|
1654
|
+
# Custom for Loki: truncate IP
|
|
1655
|
+
field :ip_address do
|
|
1656
|
+
strategy :mask # Default
|
|
1657
|
+
except :file_audit
|
|
1658
|
+
|
|
1659
|
+
# ✅ Custom filter for specific adapter
|
|
1660
|
+
custom_for_adapter :loki do |value|
|
|
1661
|
+
value.split('.')[0..2].join('.') + '.x'
|
|
1662
|
+
end
|
|
1663
|
+
end
|
|
1664
|
+
|
|
1665
|
+
masks :password
|
|
1666
|
+
skips :session_id
|
|
1667
|
+
end
|
|
1668
|
+
end
|
|
1669
|
+
```
|
|
1670
|
+
|
|
1671
|
+
**Pattern 6: Bulk operations**
|
|
1672
|
+
|
|
1673
|
+
```ruby
|
|
1674
|
+
class Events::UserProfileUpdated < E11y::Event::Base
|
|
1675
|
+
schema do
|
|
1676
|
+
required(:user_id).filled(:string)
|
|
1677
|
+
required(:email).filled(:string)
|
|
1678
|
+
required(:phone).filled(:string)
|
|
1679
|
+
required(:address).filled(:hash)
|
|
1680
|
+
required(:bio).filled(:string)
|
|
1681
|
+
required(:avatar_url).filled(:string)
|
|
1682
|
+
end
|
|
1683
|
+
|
|
1684
|
+
contains_pii true
|
|
1685
|
+
|
|
1686
|
+
pii_filtering do
|
|
1687
|
+
# ✅ Like Rails: validates_presence_of :all, except: [:id]
|
|
1688
|
+
# Hash everything except user_id and avatar_url
|
|
1689
|
+
hashes_all_except :user_id, :avatar_url, except: [:file_audit]
|
|
1690
|
+
end
|
|
1691
|
+
end
|
|
1692
|
+
```
|
|
1693
|
+
|
|
1694
|
+
**Pattern 7: Chainable methods**
|
|
1695
|
+
|
|
1696
|
+
```ruby
|
|
1697
|
+
class Events::PaymentAttempt < E11y::Event::Base
|
|
1698
|
+
contains_pii true
|
|
1699
|
+
|
|
1700
|
+
pii_filtering do
|
|
1701
|
+
# ✅ Chainable (Rails-style)
|
|
1702
|
+
field :email do
|
|
1703
|
+
strategy :hash
|
|
1704
|
+
except :file_audit
|
|
1705
|
+
with_algorithm 'SHA256'
|
|
1706
|
+
end
|
|
1707
|
+
|
|
1708
|
+
field :ip_address do
|
|
1709
|
+
strategy :truncate
|
|
1710
|
+
truncate_to 10 # "192.168.1.xxx" → "192.168.1."
|
|
1711
|
+
except :file_audit
|
|
1712
|
+
end
|
|
1713
|
+
|
|
1714
|
+
masks :card_number
|
|
1715
|
+
skips :payment_id
|
|
1716
|
+
end
|
|
1717
|
+
end
|
|
1718
|
+
```
|
|
1719
|
+
|
|
1720
|
+
**Pattern 8: Smart defaults with overrides**
|
|
1721
|
+
|
|
1722
|
+
```ruby
|
|
1723
|
+
class Events::SensitiveDataAccess < E11y::Event::Base
|
|
1724
|
+
contains_pii true
|
|
1725
|
+
|
|
1726
|
+
pii_filtering do
|
|
1727
|
+
# Default strategy for most fields
|
|
1728
|
+
with_strategy :hash do
|
|
1729
|
+
with_options algorithm: 'SHA256', except: [:file_audit] do
|
|
1730
|
+
field :accessed_by_email
|
|
1731
|
+
field :accessed_record_id
|
|
1732
|
+
field :ip_address
|
|
1733
|
+
end
|
|
1734
|
+
end
|
|
1735
|
+
|
|
1736
|
+
# Override for ultra-sensitive data
|
|
1737
|
+
field :ssn do
|
|
1738
|
+
strategy :redact # Complete removal
|
|
1739
|
+
only :sentry # Only for external service
|
|
1740
|
+
end
|
|
1741
|
+
|
|
1742
|
+
skips :access_timestamp, :action_type
|
|
1743
|
+
end
|
|
1744
|
+
end
|
|
1745
|
+
```
|
|
1746
|
+
|
|
1747
|
+
**Pattern 9: Truncation shortcuts**
|
|
1748
|
+
|
|
1749
|
+
```ruby
|
|
1750
|
+
class Events::ApiRequest < E11y::Event::Base
|
|
1751
|
+
contains_pii true
|
|
1752
|
+
|
|
1753
|
+
pii_filtering do
|
|
1754
|
+
# ✅ Truncate with inline option
|
|
1755
|
+
truncates :api_key, to: 8 # Show first 8 chars: "sk_live_..." → "sk_live_"
|
|
1756
|
+
|
|
1757
|
+
hashes :user_email, except: [:file_audit]
|
|
1758
|
+
|
|
1759
|
+
skips :request_id, :method, :path
|
|
1760
|
+
end
|
|
1761
|
+
end
|
|
1762
|
+
```
|
|
1763
|
+
|
|
1764
|
+
**Pattern 10: Multiple adapters (only/except)**
|
|
1765
|
+
|
|
1766
|
+
```ruby
|
|
1767
|
+
class Events::SecurityIncident < E11y::Event::Base
|
|
1768
|
+
contains_pii true
|
|
1769
|
+
|
|
1770
|
+
pii_filtering do
|
|
1771
|
+
# Send hashed to internal tools only
|
|
1772
|
+
field :suspect_email do
|
|
1773
|
+
strategy :hash
|
|
1774
|
+
only [:elasticsearch, :loki] # Only these adapters
|
|
1775
|
+
end
|
|
1776
|
+
|
|
1777
|
+
# Mask for external services
|
|
1778
|
+
field :victim_email do
|
|
1779
|
+
strategy :mask
|
|
1780
|
+
only [:sentry, :pagerduty]
|
|
1781
|
+
end
|
|
1782
|
+
|
|
1783
|
+
# Audit gets everything (AuditEvent auto-skip)
|
|
1784
|
+
skips :incident_id, :severity
|
|
1785
|
+
end
|
|
1786
|
+
end
|
|
1787
|
+
```
|
|
1788
|
+
|
|
1789
|
+
---
|
|
1790
|
+
|
|
1791
|
+
#### 3.4.6. Rails-Style Validators & Helpers
|
|
1792
|
+
|
|
1793
|
+
**Custom validators (like Rails):**
|
|
1794
|
+
|
|
1795
|
+
```ruby
|
|
1796
|
+
# lib/e11y/pii/validators/email_validator.rb
|
|
1797
|
+
module E11y
|
|
1798
|
+
module Pii
|
|
1799
|
+
module Validators
|
|
1800
|
+
class EmailValidator
|
|
1801
|
+
def self.validate(value)
|
|
1802
|
+
return true if value =~ URI::MailTo::EMAIL_REGEXP
|
|
1803
|
+
|
|
1804
|
+
raise E11y::Pii::ValidationError, "Invalid email format: #{value}"
|
|
1805
|
+
end
|
|
1806
|
+
end
|
|
1807
|
+
end
|
|
1808
|
+
end
|
|
1809
|
+
end
|
|
1810
|
+
|
|
1811
|
+
# Usage in event:
|
|
1812
|
+
class Events::UserRegistered < E11y::Event::Base
|
|
1813
|
+
pii_filtering do
|
|
1814
|
+
field :email do
|
|
1815
|
+
strategy :hash
|
|
1816
|
+
validate_with E11y::Pii::Validators::EmailValidator
|
|
1817
|
+
end
|
|
1818
|
+
end
|
|
1819
|
+
end
|
|
1820
|
+
```
|
|
1821
|
+
|
|
1822
|
+
**Macros (like Rails concerns):**
|
|
1823
|
+
|
|
1824
|
+
```ruby
|
|
1825
|
+
# lib/e11y/pii/macros/standard_user_fields.rb
|
|
1826
|
+
module E11y
|
|
1827
|
+
module Pii
|
|
1828
|
+
module Macros
|
|
1829
|
+
module StandardUserFields
|
|
1830
|
+
extend ActiveSupport::Concern
|
|
1831
|
+
|
|
1832
|
+
class_methods do
|
|
1833
|
+
def filter_standard_user_pii
|
|
1834
|
+
pii_filtering do
|
|
1835
|
+
hashes :email, except: [:file_audit]
|
|
1836
|
+
masks :password
|
|
1837
|
+
skips :user_id
|
|
1838
|
+
end
|
|
1839
|
+
end
|
|
1840
|
+
end
|
|
1841
|
+
end
|
|
1842
|
+
end
|
|
1843
|
+
end
|
|
1844
|
+
end
|
|
1845
|
+
|
|
1846
|
+
# Usage:
|
|
1847
|
+
class Events::UserLogin < E11y::Event::Base
|
|
1848
|
+
include E11y::Pii::Macros::StandardUserFields
|
|
1849
|
+
|
|
1850
|
+
contains_pii true
|
|
1851
|
+
filter_standard_user_pii # ✅ Macro!
|
|
1852
|
+
|
|
1853
|
+
pii_filtering do
|
|
1854
|
+
# Additional fields
|
|
1855
|
+
masks :ip_address
|
|
1856
|
+
end
|
|
1857
|
+
end
|
|
1858
|
+
```
|
|
1859
|
+
|
|
1860
|
+
---
|
|
1861
|
+
|
|
1862
|
+
#### 3.4.7. Testing PII Filters (RSpec Examples)
|
|
1863
|
+
|
|
1864
|
+
**Critical:** PII filtering MUST be tested to ensure no leaks.
|
|
1865
|
+
|
|
1866
|
+
```ruby
|
|
1867
|
+
# spec/lib/e11y/security/pii_filtering_spec.rb
|
|
1868
|
+
RSpec.describe 'PII Filtering', type: :integration do
|
|
1869
|
+
describe 'Per-Adapter Filtering' do
|
|
1870
|
+
let(:event_class) { Events::UserLogin }
|
|
1871
|
+
let(:payload) do
|
|
1872
|
+
{
|
|
1873
|
+
email: 'user@example.com',
|
|
1874
|
+
ip_address: '192.168.1.100',
|
|
1875
|
+
user_id: 'u123'
|
|
1876
|
+
}
|
|
1877
|
+
end
|
|
1878
|
+
|
|
1879
|
+
before do
|
|
1880
|
+
# Track events sent to adapters
|
|
1881
|
+
E11y::Adapters.each do |adapter|
|
|
1882
|
+
allow(adapter).to receive(:write).and_call_original
|
|
1883
|
+
end
|
|
1884
|
+
end
|
|
1885
|
+
|
|
1886
|
+
it 'applies different PII rules for each adapter' do
|
|
1887
|
+
event_class.track(payload)
|
|
1888
|
+
|
|
1889
|
+
# Check Audit Adapter: NO filtering
|
|
1890
|
+
audit_event = E11y::Adapters[:file_audit].written_events.last
|
|
1891
|
+
expect(audit_event[:payload][:email]).to eq('user@example.com')
|
|
1892
|
+
expect(audit_event[:payload][:ip_address]).to eq('192.168.1.100')
|
|
1893
|
+
|
|
1894
|
+
# Check Sentry Adapter: FULL masking
|
|
1895
|
+
sentry_event = E11y::Adapters[:sentry].written_events.last
|
|
1896
|
+
expect(sentry_event[:payload][:email]).to eq('[FILTERED]')
|
|
1897
|
+
expect(sentry_event[:payload][:ip_address]).to eq('[FILTERED]')
|
|
1898
|
+
|
|
1899
|
+
# Check Loki Adapter: CUSTOM (email masked, IP truncated)
|
|
1900
|
+
loki_event = E11y::Adapters[:loki].written_events.last
|
|
1901
|
+
expect(loki_event[:payload][:email]).to eq('[FILTERED]')
|
|
1902
|
+
expect(loki_event[:payload][:ip_address]).to eq('192.168.1.x')
|
|
1903
|
+
|
|
1904
|
+
# Check Elasticsearch Adapter: HASHED email only
|
|
1905
|
+
es_event = E11y::Adapters[:elasticsearch].written_events.last
|
|
1906
|
+
expect(es_event[:payload][:email]).to match(/^[a-f0-9]{64}$/) # SHA256 hash
|
|
1907
|
+
expect(es_event[:payload][:ip_address]).to eq('192.168.1.100') # Not filtered
|
|
1908
|
+
end
|
|
1909
|
+
end
|
|
1910
|
+
|
|
1911
|
+
describe 'Tier 1: No PII (Skip Filtering)' do
|
|
1912
|
+
let(:event_class) { Events::HealthCheck }
|
|
1913
|
+
let(:payload) { { status: 'ok' } }
|
|
1914
|
+
|
|
1915
|
+
it 'skips PII filtering entirely' do
|
|
1916
|
+
start = Time.now
|
|
1917
|
+
|
|
1918
|
+
1000.times { event_class.track(payload) }
|
|
1919
|
+
|
|
1920
|
+
duration = Time.now - start
|
|
1921
|
+
|
|
1922
|
+
# Should be VERY fast (no filtering overhead)
|
|
1923
|
+
expect(duration).to be < 0.1 # <100ms for 1000 events
|
|
1924
|
+
end
|
|
1925
|
+
end
|
|
1926
|
+
|
|
1927
|
+
describe 'Tier 2: Rails Filters Only' do
|
|
1928
|
+
let(:event_class) { Events::OrderCreated }
|
|
1929
|
+
let(:payload) do
|
|
1930
|
+
{
|
|
1931
|
+
order_id: 'o123',
|
|
1932
|
+
amount: 99.99,
|
|
1933
|
+
password: 'secret123', # Rails filter should catch this
|
|
1934
|
+
api_key: 'sk_live_xxx' # Rails filter should catch this
|
|
1935
|
+
}
|
|
1936
|
+
end
|
|
1937
|
+
|
|
1938
|
+
it 'applies Rails.application.config.filter_parameters' do
|
|
1939
|
+
event_class.track(payload)
|
|
1940
|
+
|
|
1941
|
+
event = E11y::Adapters[:loki].written_events.last
|
|
1942
|
+
|
|
1943
|
+
# Known Rails filter keys
|
|
1944
|
+
expect(event[:payload][:password]).to eq('[FILTERED]')
|
|
1945
|
+
expect(event[:payload][:api_key]).to eq('[FILTERED]')
|
|
1946
|
+
|
|
1947
|
+
# Not filtered
|
|
1948
|
+
expect(event[:payload][:order_id]).to eq('o123')
|
|
1949
|
+
expect(event[:payload][:amount]).to eq(99.99)
|
|
1950
|
+
end
|
|
1951
|
+
end
|
|
1952
|
+
|
|
1953
|
+
describe 'Tier 3: Deep Filtering' do
|
|
1954
|
+
let(:event_class) { Events::UserRegistered }
|
|
1955
|
+
let(:payload) do
|
|
1956
|
+
{
|
|
1957
|
+
email: 'user@example.com',
|
|
1958
|
+
address: {
|
|
1959
|
+
street: '123 Main St',
|
|
1960
|
+
city: 'New York',
|
|
1961
|
+
zip: '10001'
|
|
1962
|
+
},
|
|
1963
|
+
metadata: {
|
|
1964
|
+
referrer: 'https://example.com?email=leaked@example.com'
|
|
1965
|
+
}
|
|
1966
|
+
}
|
|
1967
|
+
end
|
|
1968
|
+
|
|
1969
|
+
before do
|
|
1970
|
+
event_class.class_eval do
|
|
1971
|
+
contains_pii true
|
|
1972
|
+
|
|
1973
|
+
pii_filtering do
|
|
1974
|
+
deep_scan true
|
|
1975
|
+
patterns [:email] # Scan for emails in all strings
|
|
1976
|
+
end
|
|
1977
|
+
end
|
|
1978
|
+
end
|
|
1979
|
+
|
|
1980
|
+
it 'filters PII in nested structures' do
|
|
1981
|
+
event_class.track(payload)
|
|
1982
|
+
|
|
1983
|
+
event = E11y::Adapters[:sentry].written_events.last
|
|
1984
|
+
|
|
1985
|
+
# Top-level email filtered
|
|
1986
|
+
expect(event[:payload][:email]).to eq('[FILTERED]')
|
|
1987
|
+
|
|
1988
|
+
# Nested email in string also filtered
|
|
1989
|
+
expect(event[:payload][:metadata][:referrer]).not_to include('leaked@example.com')
|
|
1990
|
+
expect(event[:payload][:metadata][:referrer]).to include('[FILTERED]')
|
|
1991
|
+
end
|
|
1992
|
+
end
|
|
1993
|
+
|
|
1994
|
+
describe 'Performance Regression Test' do
|
|
1995
|
+
it 'meets performance budget for 1000 events/sec' do
|
|
1996
|
+
events = [
|
|
1997
|
+
{ class: Events::HealthCheck, payload: { status: 'ok' }, count: 500 }, # Tier 1
|
|
1998
|
+
{ class: Events::OrderCreated, payload: { id: 'o1' }, count: 400 }, # Tier 2
|
|
1999
|
+
{ class: Events::UserLogin, payload: { email: 'u@x.com' }, count: 100 } # Tier 3
|
|
2000
|
+
]
|
|
2001
|
+
|
|
2002
|
+
start = Time.now
|
|
2003
|
+
|
|
2004
|
+
events.each do |batch|
|
|
2005
|
+
batch[:count].times { batch[:class].track(batch[:payload]) }
|
|
2006
|
+
end
|
|
2007
|
+
|
|
2008
|
+
duration = Time.now - start
|
|
2009
|
+
|
|
2010
|
+
# Total: 1000 events
|
|
2011
|
+
# Budget: 40ms CPU (4% of 1 second)
|
|
2012
|
+
# Allow 2x buffer for test overhead
|
|
2013
|
+
expect(duration).to be < 0.08 # 80ms
|
|
2014
|
+
end
|
|
2015
|
+
end
|
|
2016
|
+
|
|
2017
|
+
describe 'GDPR Compliance Audit' do
|
|
2018
|
+
it 'logs PII filtering decisions for audit trail' do
|
|
2019
|
+
event_class = Events::UserLogin
|
|
2020
|
+
|
|
2021
|
+
expect(E11y.logger).to receive(:info).with(
|
|
2022
|
+
hash_including(
|
|
2023
|
+
event: 'pii_filtering_applied',
|
|
2024
|
+
adapter: :sentry,
|
|
2025
|
+
strategy: :mask,
|
|
2026
|
+
fields: [:email, :ip_address]
|
|
2027
|
+
)
|
|
2028
|
+
)
|
|
2029
|
+
|
|
2030
|
+
event_class.track(email: 'user@example.com', ip_address: '192.168.1.1')
|
|
2031
|
+
end
|
|
2032
|
+
|
|
2033
|
+
it 'logs justification for skipping PII filtering (audit)' do
|
|
2034
|
+
event_class = Events::GdprDeletion
|
|
2035
|
+
|
|
2036
|
+
expect(E11y.logger).to receive(:info).with(
|
|
2037
|
+
hash_including(
|
|
2038
|
+
event: 'pii_filtering_skipped',
|
|
2039
|
+
adapter: :file_audit,
|
|
2040
|
+
justification: /GDPR Art. 6\(1\)\(c\)/
|
|
2041
|
+
)
|
|
2042
|
+
)
|
|
2043
|
+
|
|
2044
|
+
event_class.track(user_id: 'u123', email: 'user@example.com')
|
|
2045
|
+
end
|
|
2046
|
+
end
|
|
2047
|
+
|
|
2048
|
+
describe 'Edge Cases' do
|
|
2049
|
+
it 'handles missing PII fields gracefully' do
|
|
2050
|
+
event_class = Events::UserLogin
|
|
2051
|
+
|
|
2052
|
+
# Missing 'email' field (required by PII config)
|
|
2053
|
+
expect {
|
|
2054
|
+
event_class.track(ip_address: '192.168.1.1')
|
|
2055
|
+
}.not_to raise_error
|
|
2056
|
+
end
|
|
2057
|
+
|
|
2058
|
+
it 'handles nil values in PII fields' do
|
|
2059
|
+
event_class = Events::UserLogin
|
|
2060
|
+
|
|
2061
|
+
event_class.track(email: nil, ip_address: '192.168.1.1')
|
|
2062
|
+
|
|
2063
|
+
event = E11y::Adapters[:sentry].written_events.last
|
|
2064
|
+
expect(event[:payload][:email]).to be_nil # Not "[FILTERED]"
|
|
2065
|
+
end
|
|
2066
|
+
|
|
2067
|
+
it 'handles very long PII values efficiently' do
|
|
2068
|
+
long_email = "#{'a' * 10_000}@example.com"
|
|
2069
|
+
|
|
2070
|
+
event_class = Events::UserLogin
|
|
2071
|
+
|
|
2072
|
+
start = Time.now
|
|
2073
|
+
event_class.track(email: long_email, ip_address: '192.168.1.1')
|
|
2074
|
+
duration = Time.now - start
|
|
2075
|
+
|
|
2076
|
+
# Should not hang or be extremely slow
|
|
2077
|
+
expect(duration).to be < 0.5 # <500ms
|
|
2078
|
+
end
|
|
2079
|
+
end
|
|
2080
|
+
end
|
|
2081
|
+
|
|
2082
|
+
# spec/support/pii_filter_matcher.rb
|
|
2083
|
+
RSpec::Matchers.define :be_pii_filtered do
|
|
2084
|
+
match do |value|
|
|
2085
|
+
['[FILTERED]', '[REDACTED]', '[MASKED]'].include?(value) ||
|
|
2086
|
+
value =~ /^[a-f0-9]{32,}$/ # Hash
|
|
2087
|
+
end
|
|
2088
|
+
|
|
2089
|
+
failure_message do |value|
|
|
2090
|
+
"expected #{value.inspect} to be PII-filtered (e.g., '[FILTERED]' or hash)"
|
|
2091
|
+
end
|
|
2092
|
+
end
|
|
2093
|
+
|
|
2094
|
+
RSpec::Matchers.define :be_pii_free do |original|
|
|
2095
|
+
match do |value|
|
|
2096
|
+
value == original # Not filtered
|
|
2097
|
+
end
|
|
2098
|
+
|
|
2099
|
+
failure_message do |value|
|
|
2100
|
+
"expected #{value.inspect} to be PII-free (original: #{original.inspect})"
|
|
2101
|
+
end
|
|
2102
|
+
end
|
|
2103
|
+
|
|
2104
|
+
# Usage:
|
|
2105
|
+
# expect(event[:payload][:email]).to be_pii_filtered
|
|
2106
|
+
# expect(event[:payload][:order_id]).to be_pii_free('o123')
|
|
2107
|
+
```
|
|
2108
|
+
|
|
2109
|
+
**Running PII Tests:**
|
|
2110
|
+
```bash
|
|
2111
|
+
# Test all PII filtering
|
|
2112
|
+
bundle exec rspec spec/lib/e11y/security/pii_filtering_spec.rb
|
|
2113
|
+
|
|
2114
|
+
# Test specific tier
|
|
2115
|
+
bundle exec rspec spec/lib/e11y/security/pii_filtering_spec.rb -e "Tier 3"
|
|
2116
|
+
|
|
2117
|
+
# Performance test
|
|
2118
|
+
bundle exec rspec spec/lib/e11y/security/pii_filtering_spec.rb -e "Performance"
|
|
2119
|
+
|
|
2120
|
+
# GDPR compliance audit
|
|
2121
|
+
bundle exec rspec spec/lib/e11y/security/pii_filtering_spec.rb -e "GDPR"
|
|
2122
|
+
```
|
|
2123
|
+
|
|
2124
|
+
---
|
|
2125
|
+
|
|
2126
|
+
## 4. Rate Limiting
|
|
2127
|
+
|
|
2128
|
+
### 4.0. Rate Limiting + Retry Policy Resolution (Conflict #14)
|
|
2129
|
+
|
|
2130
|
+
**Critical Decision from CONFLICT-ANALYSIS.md:**
|
|
2131
|
+
|
|
2132
|
+
**Problem:** Should retry attempts count toward rate limits?
|
|
2133
|
+
|
|
2134
|
+
**Resolution:**
|
|
2135
|
+
```ruby
|
|
2136
|
+
# ✅ Retries DO count toward rate limit
|
|
2137
|
+
# Reason: Prevent retry amplification attack
|
|
2138
|
+
|
|
2139
|
+
config.error_handling do
|
|
2140
|
+
retry_policy do
|
|
2141
|
+
respect_rate_limits true # ← Retries checked
|
|
2142
|
+
on_retry_rate_limited :send_to_dlq # Safety net
|
|
2143
|
+
end
|
|
2144
|
+
end
|
|
2145
|
+
|
|
2146
|
+
# Optional: Separate limit for retries
|
|
2147
|
+
config.rate_limiting do
|
|
2148
|
+
global_limit 100.per_second
|
|
2149
|
+
|
|
2150
|
+
retry_limit do
|
|
2151
|
+
enabled true
|
|
2152
|
+
limit 20.per_second # Additional headroom
|
|
2153
|
+
end
|
|
2154
|
+
|
|
2155
|
+
# Critical events bypass
|
|
2156
|
+
bypass_for_severities [:error, :fatal]
|
|
2157
|
+
bypass_for_patterns ['audit.*']
|
|
2158
|
+
end
|
|
2159
|
+
```
|
|
2160
|
+
|
|
2161
|
+
**Pipeline Flow:**
|
|
2162
|
+
```
|
|
2163
|
+
Original event
|
|
2164
|
+
→ Rate Limiting (check)
|
|
2165
|
+
→ Adapter Write (fail)
|
|
2166
|
+
→ Retry #1
|
|
2167
|
+
→ Rate Limiting (check again) ← Retry counts!
|
|
2168
|
+
→ Adapter Write (fail)
|
|
2169
|
+
→ Retry #2 (rate limited)
|
|
2170
|
+
→ DLQ (rate-limited retry)
|
|
2171
|
+
```
|
|
2172
|
+
|
|
2173
|
+
---
|
|
2174
|
+
|
|
2175
|
+
### 4.1. Global Rate Limiting
|
|
2176
|
+
|
|
2177
|
+
**Design Decision:** System-wide rate limit for all events.
|
|
2178
|
+
|
|
2179
|
+
```ruby
|
|
2180
|
+
module E11y
|
|
2181
|
+
module RateLimiting
|
|
2182
|
+
class GlobalLimiter
|
|
2183
|
+
def initialize(config)
|
|
2184
|
+
@limit = config.limit # events per second
|
|
2185
|
+
@window = config.window || 1.second
|
|
2186
|
+
@strategy = config.strategy || :sliding_window
|
|
2187
|
+
@redis = config.redis
|
|
2188
|
+
|
|
2189
|
+
@counter = initialize_counter
|
|
2190
|
+
end
|
|
2191
|
+
|
|
2192
|
+
def allow?(event_data = nil)
|
|
2193
|
+
@counter.increment('global')
|
|
2194
|
+
|
|
2195
|
+
current_rate = @counter.rate('global', window: @window)
|
|
2196
|
+
|
|
2197
|
+
current_rate <= @limit
|
|
2198
|
+
end
|
|
2199
|
+
|
|
2200
|
+
def current_rate
|
|
2201
|
+
@counter.rate('global', window: @window)
|
|
2202
|
+
end
|
|
2203
|
+
|
|
2204
|
+
private
|
|
2205
|
+
|
|
2206
|
+
def initialize_counter
|
|
2207
|
+
if @redis
|
|
2208
|
+
RedisCounter.new(@redis, strategy: @strategy)
|
|
2209
|
+
else
|
|
2210
|
+
InMemoryCounter.new(strategy: @strategy)
|
|
2211
|
+
end
|
|
2212
|
+
end
|
|
2213
|
+
end
|
|
2214
|
+
end
|
|
2215
|
+
end
|
|
2216
|
+
```
|
|
2217
|
+
|
|
2218
|
+
**Configuration:**
|
|
2219
|
+
|
|
2220
|
+
```ruby
|
|
2221
|
+
E11y.configure do |config|
|
|
2222
|
+
config.rate_limiting do
|
|
2223
|
+
global do
|
|
2224
|
+
enabled true
|
|
2225
|
+
limit 10_000 # 10k events/sec
|
|
2226
|
+
window 1.second
|
|
2227
|
+
strategy :sliding_window
|
|
2228
|
+
|
|
2229
|
+
# Overflow behavior
|
|
2230
|
+
on_overflow :drop # :drop, :sample, or :queue
|
|
2231
|
+
|
|
2232
|
+
# Monitoring
|
|
2233
|
+
on_limit_exceeded do |current_rate, limit|
|
|
2234
|
+
Rails.logger.warn "[E11y] Global rate limit exceeded: #{current_rate}/#{limit}"
|
|
2235
|
+
end
|
|
2236
|
+
end
|
|
2237
|
+
end
|
|
2238
|
+
end
|
|
2239
|
+
```
|
|
2240
|
+
|
|
2241
|
+
---
|
|
2242
|
+
|
|
2243
|
+
### 4.2. Per-Event Rate Limiting
|
|
2244
|
+
|
|
2245
|
+
**Design Decision:** Different limits for different event types.
|
|
2246
|
+
|
|
2247
|
+
```ruby
|
|
2248
|
+
module E11y
|
|
2249
|
+
module RateLimiting
|
|
2250
|
+
class PerEventLimiter
|
|
2251
|
+
def initialize(config)
|
|
2252
|
+
@limits = config.limits || {}
|
|
2253
|
+
@default_limit = config.default_limit || Float::INFINITY
|
|
2254
|
+
@window = config.window || 1.second
|
|
2255
|
+
@redis = config.redis
|
|
2256
|
+
|
|
2257
|
+
@counter = initialize_counter
|
|
2258
|
+
end
|
|
2259
|
+
|
|
2260
|
+
def allow?(event_data)
|
|
2261
|
+
event_name = event_data[:event_name]
|
|
2262
|
+
limit = @limits[event_name] || @default_limit
|
|
2263
|
+
|
|
2264
|
+
return true if limit == Float::INFINITY
|
|
2265
|
+
|
|
2266
|
+
@counter.increment(event_name)
|
|
2267
|
+
|
|
2268
|
+
current_rate = @counter.rate(event_name, window: @window)
|
|
2269
|
+
|
|
2270
|
+
current_rate <= limit
|
|
2271
|
+
end
|
|
2272
|
+
|
|
2273
|
+
def current_rate(event_name)
|
|
2274
|
+
@counter.rate(event_name, window: @window)
|
|
2275
|
+
end
|
|
2276
|
+
end
|
|
2277
|
+
end
|
|
2278
|
+
end
|
|
2279
|
+
```
|
|
2280
|
+
|
|
2281
|
+
**Configuration:**
|
|
2282
|
+
|
|
2283
|
+
```ruby
|
|
2284
|
+
E11y.configure do |config|
|
|
2285
|
+
config.rate_limiting do
|
|
2286
|
+
per_event do
|
|
2287
|
+
enabled true
|
|
2288
|
+
|
|
2289
|
+
# Specific limits per event
|
|
2290
|
+
limits do
|
|
2291
|
+
event 'order.paid', limit: 1000 # 1k/sec
|
|
2292
|
+
event 'user.login', limit: 500 # 500/sec
|
|
2293
|
+
event 'debug.*', limit: 100 # 100/sec for all debug events
|
|
2294
|
+
end
|
|
2295
|
+
|
|
2296
|
+
# Default for unlisted events
|
|
2297
|
+
default_limit 1000
|
|
2298
|
+
|
|
2299
|
+
window 1.second
|
|
2300
|
+
end
|
|
2301
|
+
end
|
|
2302
|
+
end
|
|
2303
|
+
```
|
|
2304
|
+
|
|
2305
|
+
---
|
|
2306
|
+
|
|
2307
|
+
### 4.3. Per-Context Rate Limiting
|
|
2308
|
+
|
|
2309
|
+
**Design Decision:** Limit per user/tenant/IP address.
|
|
2310
|
+
|
|
2311
|
+
```ruby
|
|
2312
|
+
module E11y
|
|
2313
|
+
module RateLimiting
|
|
2314
|
+
class PerContextLimiter
|
|
2315
|
+
def initialize(config)
|
|
2316
|
+
@limit = config.limit
|
|
2317
|
+
@window = config.window || 1.minute
|
|
2318
|
+
@context_keys = config.context_keys || [:user_id]
|
|
2319
|
+
@redis = config.redis
|
|
2320
|
+
|
|
2321
|
+
@counter = initialize_counter
|
|
2322
|
+
end
|
|
2323
|
+
|
|
2324
|
+
def allow?(event_data)
|
|
2325
|
+
context_value = extract_context(event_data)
|
|
2326
|
+
|
|
2327
|
+
return true unless context_value
|
|
2328
|
+
|
|
2329
|
+
key = "context:#{context_value}"
|
|
2330
|
+
@counter.increment(key)
|
|
2331
|
+
|
|
2332
|
+
current_rate = @counter.rate(key, window: @window)
|
|
2333
|
+
|
|
2334
|
+
current_rate <= @limit
|
|
2335
|
+
end
|
|
2336
|
+
|
|
2337
|
+
private
|
|
2338
|
+
|
|
2339
|
+
def extract_context(event_data)
|
|
2340
|
+
@context_keys.each do |key|
|
|
2341
|
+
value = event_data.dig(:payload, key) || event_data[key]
|
|
2342
|
+
return value if value
|
|
2343
|
+
end
|
|
2344
|
+
|
|
2345
|
+
nil
|
|
2346
|
+
end
|
|
2347
|
+
end
|
|
2348
|
+
end
|
|
2349
|
+
end
|
|
2350
|
+
```
|
|
2351
|
+
|
|
2352
|
+
**Configuration:**
|
|
2353
|
+
|
|
2354
|
+
```ruby
|
|
2355
|
+
E11y.configure do |config|
|
|
2356
|
+
config.rate_limiting do
|
|
2357
|
+
per_context do
|
|
2358
|
+
enabled true
|
|
2359
|
+
|
|
2360
|
+
# Limit per user
|
|
2361
|
+
context_key :user_id
|
|
2362
|
+
limit 100 # 100 events per minute per user
|
|
2363
|
+
window 1.minute
|
|
2364
|
+
|
|
2365
|
+
# Multiple context keys (fallback)
|
|
2366
|
+
context_keys [:user_id, :session_id, :ip_address]
|
|
2367
|
+
|
|
2368
|
+
# Allowlist (no limits)
|
|
2369
|
+
allowlist do
|
|
2370
|
+
user_ids ['admin_user', 'system']
|
|
2371
|
+
end
|
|
2372
|
+
end
|
|
2373
|
+
end
|
|
2374
|
+
end
|
|
2375
|
+
```
|
|
2376
|
+
|
|
2377
|
+
---
|
|
2378
|
+
|
|
2379
|
+
### 4.4. Redis Integration
|
|
2380
|
+
|
|
2381
|
+
**Design Decision:** Distributed rate limiting with Redis.
|
|
2382
|
+
|
|
2383
|
+
```ruby
|
|
2384
|
+
module E11y
|
|
2385
|
+
module RateLimiting
|
|
2386
|
+
class RedisCounter
|
|
2387
|
+
def initialize(redis, strategy: :sliding_window)
|
|
2388
|
+
@redis = redis
|
|
2389
|
+
@strategy = strategy
|
|
2390
|
+
end
|
|
2391
|
+
|
|
2392
|
+
def increment(key)
|
|
2393
|
+
case @strategy
|
|
2394
|
+
when :sliding_window
|
|
2395
|
+
increment_sliding_window(key)
|
|
2396
|
+
when :fixed_window
|
|
2397
|
+
increment_fixed_window(key)
|
|
2398
|
+
when :token_bucket
|
|
2399
|
+
increment_token_bucket(key)
|
|
2400
|
+
end
|
|
2401
|
+
end
|
|
2402
|
+
|
|
2403
|
+
def rate(key, window:)
|
|
2404
|
+
case @strategy
|
|
2405
|
+
when :sliding_window
|
|
2406
|
+
count_sliding_window(key, window)
|
|
2407
|
+
when :fixed_window
|
|
2408
|
+
count_fixed_window(key, window)
|
|
2409
|
+
when :token_bucket
|
|
2410
|
+
tokens_remaining(key)
|
|
2411
|
+
end
|
|
2412
|
+
end
|
|
2413
|
+
|
|
2414
|
+
private
|
|
2415
|
+
|
|
2416
|
+
# Sliding Window (most accurate)
|
|
2417
|
+
def increment_sliding_window(key)
|
|
2418
|
+
now = Time.now.to_f
|
|
2419
|
+
redis_key = "e11y:rate:#{key}"
|
|
2420
|
+
|
|
2421
|
+
@redis.multi do |multi|
|
|
2422
|
+
multi.zadd(redis_key, now, "#{now}:#{SecureRandom.uuid}")
|
|
2423
|
+
multi.expire(redis_key, 60)
|
|
2424
|
+
end
|
|
2425
|
+
end
|
|
2426
|
+
|
|
2427
|
+
def count_sliding_window(key, window)
|
|
2428
|
+
now = Time.now.to_f
|
|
2429
|
+
cutoff = now - window.to_f
|
|
2430
|
+
redis_key = "e11y:rate:#{key}"
|
|
2431
|
+
|
|
2432
|
+
# Remove old entries
|
|
2433
|
+
@redis.zremrangebyscore(redis_key, '-inf', cutoff)
|
|
2434
|
+
|
|
2435
|
+
# Count remaining
|
|
2436
|
+
@redis.zcard(redis_key)
|
|
2437
|
+
end
|
|
2438
|
+
|
|
2439
|
+
# Fixed Window (simpler, less accurate)
|
|
2440
|
+
def increment_fixed_window(key)
|
|
2441
|
+
window_key = current_window_key(key)
|
|
2442
|
+
|
|
2443
|
+
@redis.multi do |multi|
|
|
2444
|
+
multi.incr(window_key)
|
|
2445
|
+
multi.expire(window_key, 60)
|
|
2446
|
+
end
|
|
2447
|
+
end
|
|
2448
|
+
|
|
2449
|
+
def count_fixed_window(key, window)
|
|
2450
|
+
window_key = current_window_key(key)
|
|
2451
|
+
@redis.get(window_key).to_i
|
|
2452
|
+
end
|
|
2453
|
+
|
|
2454
|
+
def current_window_key(key)
|
|
2455
|
+
window_start = (Time.now.to_i / 1) * 1 # 1-second windows
|
|
2456
|
+
"e11y:rate:fixed:#{key}:#{window_start}"
|
|
2457
|
+
end
|
|
2458
|
+
|
|
2459
|
+
# Token Bucket (burst-friendly)
|
|
2460
|
+
def increment_token_bucket(key)
|
|
2461
|
+
redis_key = "e11y:rate:bucket:#{key}"
|
|
2462
|
+
|
|
2463
|
+
# Lua script for atomic token consumption
|
|
2464
|
+
lua_script = <<~LUA
|
|
2465
|
+
local key = KEYS[1]
|
|
2466
|
+
local capacity = tonumber(ARGV[1])
|
|
2467
|
+
local refill_rate = tonumber(ARGV[2])
|
|
2468
|
+
local now = tonumber(ARGV[3])
|
|
2469
|
+
|
|
2470
|
+
local tokens = tonumber(redis.call('GET', key) or capacity)
|
|
2471
|
+
local last_refill = tonumber(redis.call('GET', key .. ':last') or now)
|
|
2472
|
+
|
|
2473
|
+
-- Refill tokens
|
|
2474
|
+
local time_passed = now - last_refill
|
|
2475
|
+
local tokens_to_add = time_passed * refill_rate
|
|
2476
|
+
tokens = math.min(capacity, tokens + tokens_to_add)
|
|
2477
|
+
|
|
2478
|
+
-- Consume token
|
|
2479
|
+
if tokens >= 1 then
|
|
2480
|
+
tokens = tokens - 1
|
|
2481
|
+
redis.call('SET', key, tokens)
|
|
2482
|
+
redis.call('SET', key .. ':last', now)
|
|
2483
|
+
return 1 -- Success
|
|
2484
|
+
else
|
|
2485
|
+
return 0 -- No tokens
|
|
2486
|
+
end
|
|
2487
|
+
LUA
|
|
2488
|
+
|
|
2489
|
+
@redis.eval(lua_script, keys: [redis_key], argv: [100, 10, Time.now.to_f])
|
|
2490
|
+
end
|
|
2491
|
+
|
|
2492
|
+
def tokens_remaining(key)
|
|
2493
|
+
redis_key = "e11y:rate:bucket:#{key}"
|
|
2494
|
+
@redis.get(redis_key).to_i
|
|
2495
|
+
end
|
|
2496
|
+
end
|
|
2497
|
+
end
|
|
2498
|
+
end
|
|
2499
|
+
```
|
|
2500
|
+
|
|
2501
|
+
**Rate Limiting Strategies Comparison:**
|
|
2502
|
+
|
|
2503
|
+
```mermaid
|
|
2504
|
+
graph TB
|
|
2505
|
+
subgraph "Sliding Window (Most Accurate)"
|
|
2506
|
+
SW1[Request at t=0] --> SW2[Request at t=0.5]
|
|
2507
|
+
SW2 --> SW3[Request at t=0.9]
|
|
2508
|
+
SW3 --> SW4[Window: 0.0-1.0s<br/>Count: 3]
|
|
2509
|
+
end
|
|
2510
|
+
|
|
2511
|
+
subgraph "Fixed Window (Simpler)"
|
|
2512
|
+
FW1[Request at t=0.9] --> FW2[Request at t=1.1]
|
|
2513
|
+
FW2 --> FW3[Window 1: 1 request<br/>Window 2: 1 request]
|
|
2514
|
+
Note right of FW3: Burst at boundary!
|
|
2515
|
+
end
|
|
2516
|
+
|
|
2517
|
+
subgraph "Token Bucket (Burst-Friendly)"
|
|
2518
|
+
TB1[Capacity: 100 tokens] --> TB2[Refill: 10/sec]
|
|
2519
|
+
TB2 --> TB3[Burst: OK if tokens available]
|
|
2520
|
+
end
|
|
2521
|
+
|
|
2522
|
+
style SW4 fill:#d4edda
|
|
2523
|
+
style FW3 fill:#fff3cd
|
|
2524
|
+
style TB3 fill:#d1ecf1
|
|
2525
|
+
```
|
|
2526
|
+
|
|
2527
|
+
---
|
|
2528
|
+
|
|
2529
|
+
## 5. Audit Trail
|
|
2530
|
+
|
|
2531
|
+
### 5.0. Audit Trail + PII Filtering Resolution (Conflict #4)
|
|
2532
|
+
|
|
2533
|
+
**Critical Decision from CONFLICT-ANALYSIS.md:**
|
|
2534
|
+
|
|
2535
|
+
**Problem:** Audit events need original data for compliance, but PII filtering requires masking for GDPR.
|
|
2536
|
+
|
|
2537
|
+
**Resolution:**
|
|
2538
|
+
```ruby
|
|
2539
|
+
# Option A: Audit events skip PII filtering (RECOMMENDED)
|
|
2540
|
+
config.audit_trail do
|
|
2541
|
+
skip_pii_filtering true # Legal obligation > privacy
|
|
2542
|
+
end
|
|
2543
|
+
|
|
2544
|
+
# Compensating controls:
|
|
2545
|
+
config.audit_trail do
|
|
2546
|
+
encryption_at_rest true
|
|
2547
|
+
access_control do
|
|
2548
|
+
read_access_role :auditor
|
|
2549
|
+
read_access_requires_reason true
|
|
2550
|
+
read_access_logged true # Meta-audit
|
|
2551
|
+
end
|
|
2552
|
+
end
|
|
2553
|
+
|
|
2554
|
+
# Option B: Per-adapter PII rules (more flexible)
|
|
2555
|
+
class UserPermissionChanged < E11y::AuditEvent
|
|
2556
|
+
adapters [:audit_file, :elasticsearch, :sentry]
|
|
2557
|
+
|
|
2558
|
+
pii_rules do
|
|
2559
|
+
# Audit file: keep all PII (compliance)
|
|
2560
|
+
adapter :audit_file do
|
|
2561
|
+
skip_filtering true
|
|
2562
|
+
end
|
|
2563
|
+
|
|
2564
|
+
# Elasticsearch: pseudonymize
|
|
2565
|
+
adapter :elasticsearch do
|
|
2566
|
+
pseudonymize_fields :email, :ip_address
|
|
2567
|
+
end
|
|
2568
|
+
|
|
2569
|
+
# Sentry: mask all
|
|
2570
|
+
adapter :sentry do
|
|
2571
|
+
mask_fields :email, :ip_address, :user_id
|
|
2572
|
+
end
|
|
2573
|
+
end
|
|
2574
|
+
end
|
|
2575
|
+
```
|
|
2576
|
+
|
|
2577
|
+
**GDPR Justification:**
|
|
2578
|
+
- Art. 6(1)(c): "Legal obligation" is valid processing basis
|
|
2579
|
+
- Audit logs = necessary for accountability (SOX, HIPAA, GDPR Art. 30)
|
|
2580
|
+
- Mitigation: encryption + access control + retention limits
|
|
2581
|
+
|
|
2582
|
+
---
|
|
2583
|
+
|
|
2584
|
+
### 5.1. Immutable Events
|
|
2585
|
+
|
|
2586
|
+
**Design Decision:** Audit events are append-only, never modified.
|
|
2587
|
+
|
|
2588
|
+
```ruby
|
|
2589
|
+
module E11y
|
|
2590
|
+
class AuditEvent < Event::Base
|
|
2591
|
+
# Audit events CANNOT be modified after creation
|
|
2592
|
+
def self.inherited(subclass)
|
|
2593
|
+
super
|
|
2594
|
+
|
|
2595
|
+
subclass.class_eval do
|
|
2596
|
+
# Freeze payload after creation
|
|
2597
|
+
after_track :freeze_payload!
|
|
2598
|
+
|
|
2599
|
+
private
|
|
2600
|
+
|
|
2601
|
+
def freeze_payload!
|
|
2602
|
+
@payload.deep_freeze
|
|
2603
|
+
end
|
|
2604
|
+
end
|
|
2605
|
+
end
|
|
2606
|
+
|
|
2607
|
+
# Audit-specific metadata
|
|
2608
|
+
def self.audit_retention(days)
|
|
2609
|
+
@audit_retention_days = days
|
|
2610
|
+
end
|
|
2611
|
+
|
|
2612
|
+
def self.audit_reason(reason)
|
|
2613
|
+
@audit_reason = reason
|
|
2614
|
+
end
|
|
2615
|
+
|
|
2616
|
+
def self.signing_enabled(enabled = true)
|
|
2617
|
+
@signing_enabled = enabled
|
|
2618
|
+
end
|
|
2619
|
+
end
|
|
2620
|
+
end
|
|
2621
|
+
```
|
|
2622
|
+
|
|
2623
|
+
**Example:**
|
|
2624
|
+
|
|
2625
|
+
```ruby
|
|
2626
|
+
module Events
|
|
2627
|
+
class GdprDeletion < E11y::AuditEvent
|
|
2628
|
+
adapters [:file_audit, :elasticsearch_audit]
|
|
2629
|
+
|
|
2630
|
+
audit_retention 2555 # 7 years
|
|
2631
|
+
audit_reason 'GDPR Article 17 - Right to erasure'
|
|
2632
|
+
signing_enabled true
|
|
2633
|
+
|
|
2634
|
+
schema do
|
|
2635
|
+
required(:user_id).filled(:string)
|
|
2636
|
+
required(:deletion_reason).filled(:string)
|
|
2637
|
+
required(:requested_by).filled(:string)
|
|
2638
|
+
required(:approved_by).filled(:string)
|
|
2639
|
+
end
|
|
2640
|
+
end
|
|
2641
|
+
|
|
2642
|
+
class FinancialTransaction < E11y::AuditEvent
|
|
2643
|
+
audit_retention 2555 # 7 years (SOX compliance)
|
|
2644
|
+
audit_reason 'SOX Section 802'
|
|
2645
|
+
signing_enabled true
|
|
2646
|
+
|
|
2647
|
+
schema do
|
|
2648
|
+
required(:transaction_id).filled(:string)
|
|
2649
|
+
required(:amount).filled(:float)
|
|
2650
|
+
required(:currency).filled(:string)
|
|
2651
|
+
required(:from_account).filled(:string)
|
|
2652
|
+
required(:to_account).filled(:string)
|
|
2653
|
+
end
|
|
2654
|
+
end
|
|
2655
|
+
end
|
|
2656
|
+
```
|
|
2657
|
+
|
|
2658
|
+
---
|
|
2659
|
+
|
|
2660
|
+
### 5.2. Cryptographic Signing
|
|
2661
|
+
|
|
2662
|
+
**Design Decision:** HMAC-SHA256 signatures for tamper detection.
|
|
2663
|
+
|
|
2664
|
+
```ruby
|
|
2665
|
+
module E11y
|
|
2666
|
+
module Security
|
|
2667
|
+
class EventSigner
|
|
2668
|
+
def initialize(config)
|
|
2669
|
+
@secret_key = config.secret_key || raise("Secret key required for signing")
|
|
2670
|
+
@algorithm = config.algorithm || 'SHA256'
|
|
2671
|
+
@include_chain = config.include_chain || true
|
|
2672
|
+
@previous_hash = nil
|
|
2673
|
+
end
|
|
2674
|
+
|
|
2675
|
+
def sign(event_data)
|
|
2676
|
+
# Create canonical representation
|
|
2677
|
+
canonical = canonicalize(event_data)
|
|
2678
|
+
|
|
2679
|
+
# Add chain if enabled
|
|
2680
|
+
if @include_chain && @previous_hash
|
|
2681
|
+
canonical[:previous_hash] = @previous_hash
|
|
2682
|
+
end
|
|
2683
|
+
|
|
2684
|
+
# Generate signature
|
|
2685
|
+
signature = generate_signature(canonical)
|
|
2686
|
+
|
|
2687
|
+
# Store hash for next event
|
|
2688
|
+
@previous_hash = signature
|
|
2689
|
+
|
|
2690
|
+
# Add signature to event
|
|
2691
|
+
event_data.merge(
|
|
2692
|
+
signature: signature,
|
|
2693
|
+
signed_at: Time.now.utc.iso8601,
|
|
2694
|
+
signature_algorithm: "HMAC-#{@algorithm}"
|
|
2695
|
+
)
|
|
2696
|
+
end
|
|
2697
|
+
|
|
2698
|
+
def verify(event_data)
|
|
2699
|
+
stored_signature = event_data[:signature]
|
|
2700
|
+
return false unless stored_signature
|
|
2701
|
+
|
|
2702
|
+
# Remove signature for verification
|
|
2703
|
+
data_without_sig = event_data.except(:signature, :signed_at, :signature_algorithm)
|
|
2704
|
+
|
|
2705
|
+
# Canonicalize
|
|
2706
|
+
canonical = canonicalize(data_without_sig)
|
|
2707
|
+
|
|
2708
|
+
# Generate expected signature
|
|
2709
|
+
expected_signature = generate_signature(canonical)
|
|
2710
|
+
|
|
2711
|
+
# Constant-time comparison
|
|
2712
|
+
secure_compare(expected_signature, stored_signature)
|
|
2713
|
+
end
|
|
2714
|
+
|
|
2715
|
+
def verify_chain(events)
|
|
2716
|
+
events.each_cons(2).all? do |prev_event, current_event|
|
|
2717
|
+
# Check if current event references previous
|
|
2718
|
+
current_event[:previous_hash] == prev_event[:signature]
|
|
2719
|
+
end
|
|
2720
|
+
end
|
|
2721
|
+
|
|
2722
|
+
private
|
|
2723
|
+
|
|
2724
|
+
def canonicalize(data)
|
|
2725
|
+
# Sort keys recursively for deterministic output
|
|
2726
|
+
case data
|
|
2727
|
+
when Hash
|
|
2728
|
+
data.sort.to_h.transform_values { |v| canonicalize(v) }
|
|
2729
|
+
when Array
|
|
2730
|
+
data.map { |item| canonicalize(item) }
|
|
2731
|
+
else
|
|
2732
|
+
data
|
|
2733
|
+
end
|
|
2734
|
+
end
|
|
2735
|
+
|
|
2736
|
+
def generate_signature(canonical_data)
|
|
2737
|
+
json = JSON.generate(canonical_data)
|
|
2738
|
+
OpenSSL::HMAC.hexdigest(@algorithm, @secret_key, json)
|
|
2739
|
+
end
|
|
2740
|
+
|
|
2741
|
+
def secure_compare(a, b)
|
|
2742
|
+
return false if a.bytesize != b.bytesize
|
|
2743
|
+
|
|
2744
|
+
l = a.unpack("C*")
|
|
2745
|
+
r = b.unpack("C*")
|
|
2746
|
+
|
|
2747
|
+
result = 0
|
|
2748
|
+
l.zip(r) { |x, y| result |= x ^ y }
|
|
2749
|
+
result == 0
|
|
2750
|
+
end
|
|
2751
|
+
end
|
|
2752
|
+
end
|
|
2753
|
+
end
|
|
2754
|
+
```
|
|
2755
|
+
|
|
2756
|
+
**Per-Event Signing Configuration:**
|
|
2757
|
+
|
|
2758
|
+
For fine-grained control, individual audit events can override signing behavior:
|
|
2759
|
+
|
|
2760
|
+
```ruby
|
|
2761
|
+
# Disable signing for low-severity audit events
|
|
2762
|
+
class Events::AuditLogViewed < E11y::Event::Base
|
|
2763
|
+
audit_event true
|
|
2764
|
+
signing enabled: false # ⚠️ No cryptographic signing
|
|
2765
|
+
|
|
2766
|
+
schema do
|
|
2767
|
+
required(:log_id).filled(:integer)
|
|
2768
|
+
required(:viewed_by).filled(:integer)
|
|
2769
|
+
end
|
|
2770
|
+
end
|
|
2771
|
+
|
|
2772
|
+
# Explicitly enable signing (default behavior)
|
|
2773
|
+
class Events::UserDeleted < E11y::Event::Base
|
|
2774
|
+
audit_event true
|
|
2775
|
+
signing enabled: true # ✅ Cryptographic signing (default)
|
|
2776
|
+
|
|
2777
|
+
schema do
|
|
2778
|
+
required(:user_id).filled(:integer)
|
|
2779
|
+
required(:deleted_by).filled(:integer)
|
|
2780
|
+
end
|
|
2781
|
+
end
|
|
2782
|
+
```
|
|
2783
|
+
|
|
2784
|
+
**DSL Consistency:**
|
|
2785
|
+
|
|
2786
|
+
The `signing enabled:` DSL matches the global configuration pattern:
|
|
2787
|
+
|
|
2788
|
+
```ruby
|
|
2789
|
+
# Global config (affects ALL audit events)
|
|
2790
|
+
E11y.configure do |config|
|
|
2791
|
+
config.audit_trail do
|
|
2792
|
+
signing enabled: true, # ← Same DSL pattern
|
|
2793
|
+
algorithm: 'HMAC-SHA256'
|
|
2794
|
+
end
|
|
2795
|
+
end
|
|
2796
|
+
|
|
2797
|
+
# Per-event override (affects ONE event class)
|
|
2798
|
+
class Events::SomeEvent < E11y::Event::Base
|
|
2799
|
+
signing enabled: false # ← Same DSL pattern
|
|
2800
|
+
end
|
|
2801
|
+
```
|
|
2802
|
+
|
|
2803
|
+
**When to disable signing:**
|
|
2804
|
+
- Low-severity audit events (e.g., log views)
|
|
2805
|
+
- High-volume events with performance constraints
|
|
2806
|
+
- Internal monitoring (non-compliance use cases)
|
|
2807
|
+
|
|
2808
|
+
**When signing is REQUIRED:**
|
|
2809
|
+
- Financial transactions (SOX)
|
|
2810
|
+
- User data deletion (GDPR Art. 17)
|
|
2811
|
+
- Permission changes (access control)
|
|
2812
|
+
- Any event requiring non-repudiation
|
|
2813
|
+
|
|
2814
|
+
**Default:** All audit events are signed (`signing enabled: true`) unless explicitly disabled.
|
|
2815
|
+
|
|
2816
|
+
**Signature Chain Visualization:**
|
|
2817
|
+
|
|
2818
|
+
```mermaid
|
|
2819
|
+
graph LR
|
|
2820
|
+
Event1[Event 1<br/>signature: abc123] --> Event2[Event 2<br/>previous_hash: abc123<br/>signature: def456]
|
|
2821
|
+
Event2 --> Event3[Event 3<br/>previous_hash: def456<br/>signature: ghi789]
|
|
2822
|
+
Event3 --> Event4[Event 4<br/>previous_hash: ghi789<br/>signature: jkl012]
|
|
2823
|
+
|
|
2824
|
+
Event2 -.->|Verify| Check1{previous_hash<br/>matches?}
|
|
2825
|
+
Check1 -->|Yes ✅| Valid1[Valid Chain]
|
|
2826
|
+
Check1 -->|No ❌| Tampered1[Tampered!]
|
|
2827
|
+
|
|
2828
|
+
style Event1 fill:#d4edda
|
|
2829
|
+
style Event2 fill:#d4edda
|
|
2830
|
+
style Event3 fill:#d4edda
|
|
2831
|
+
style Event4 fill:#d4edda
|
|
2832
|
+
style Valid1 fill:#d4edda
|
|
2833
|
+
style Tampered1 fill:#f8d7da
|
|
2834
|
+
```
|
|
2835
|
+
|
|
2836
|
+
---
|
|
2837
|
+
|
|
2838
|
+
### 5.3. Compliance Features
|
|
2839
|
+
|
|
2840
|
+
**GDPR, HIPAA, SOX, PCI DSS Requirements:**
|
|
2841
|
+
|
|
2842
|
+
```ruby
|
|
2843
|
+
module E11y
|
|
2844
|
+
module Compliance
|
|
2845
|
+
class AuditConfig
|
|
2846
|
+
# GDPR Article 30 - Records of processing activities
|
|
2847
|
+
def gdpr_compliant!
|
|
2848
|
+
audit_retention 2555 # 7 years
|
|
2849
|
+
audit_reason 'GDPR Article 30'
|
|
2850
|
+
signing_enabled true
|
|
2851
|
+
|
|
2852
|
+
# Log who accessed what
|
|
2853
|
+
log_access true
|
|
2854
|
+
access_control do
|
|
2855
|
+
require_justification true
|
|
2856
|
+
require_approval true
|
|
2857
|
+
end
|
|
2858
|
+
|
|
2859
|
+
# Right to be forgotten
|
|
2860
|
+
support_deletion_requests true
|
|
2861
|
+
deletion_proof_required true
|
|
2862
|
+
end
|
|
2863
|
+
|
|
2864
|
+
# HIPAA - Health Insurance Portability and Accountability Act
|
|
2865
|
+
def hipaa_compliant!
|
|
2866
|
+
audit_retention 2190 # 6 years
|
|
2867
|
+
audit_reason 'HIPAA 45 CFR 164.312'
|
|
2868
|
+
signing_enabled true
|
|
2869
|
+
|
|
2870
|
+
# PHI access logging
|
|
2871
|
+
log_phi_access true
|
|
2872
|
+
encrypt_at_rest true
|
|
2873
|
+
|
|
2874
|
+
# Breach notification
|
|
2875
|
+
breach_detection true
|
|
2876
|
+
notification_within 60.days
|
|
2877
|
+
end
|
|
2878
|
+
|
|
2879
|
+
# SOX - Sarbanes-Oxley Act
|
|
2880
|
+
def sox_compliant!
|
|
2881
|
+
audit_retention 2555 # 7 years
|
|
2882
|
+
audit_reason 'SOX Section 802'
|
|
2883
|
+
signing_enabled true
|
|
2884
|
+
|
|
2885
|
+
# Financial records
|
|
2886
|
+
tamper_proof true
|
|
2887
|
+
dual_approval_required true
|
|
2888
|
+
|
|
2889
|
+
# Retention enforcement
|
|
2890
|
+
prevent_deletion true
|
|
2891
|
+
archive_after 2555.days
|
|
2892
|
+
end
|
|
2893
|
+
|
|
2894
|
+
# PCI DSS - Payment Card Industry Data Security Standard
|
|
2895
|
+
def pci_dss_compliant!
|
|
2896
|
+
audit_retention 365 # 1 year minimum
|
|
2897
|
+
audit_reason 'PCI DSS Requirement 10'
|
|
2898
|
+
signing_enabled true
|
|
2899
|
+
|
|
2900
|
+
# Card data events
|
|
2901
|
+
log_all_access true
|
|
2902
|
+
log_failed_attempts true
|
|
2903
|
+
|
|
2904
|
+
# Time synchronization
|
|
2905
|
+
ntp_sync_required true
|
|
2906
|
+
|
|
2907
|
+
# Review
|
|
2908
|
+
daily_log_review true
|
|
2909
|
+
end
|
|
2910
|
+
end
|
|
2911
|
+
end
|
|
2912
|
+
end
|
|
2913
|
+
```
|
|
2914
|
+
|
|
2915
|
+
**Configuration:**
|
|
2916
|
+
|
|
2917
|
+
```ruby
|
|
2918
|
+
E11y.configure do |config|
|
|
2919
|
+
config.compliance do
|
|
2920
|
+
# Enable GDPR compliance
|
|
2921
|
+
gdpr do
|
|
2922
|
+
enabled true
|
|
2923
|
+
data_controller 'Company Inc.'
|
|
2924
|
+
dpo_email 'dpo@company.com'
|
|
2925
|
+
|
|
2926
|
+
# Consent tracking
|
|
2927
|
+
track_consent true
|
|
2928
|
+
consent_required_for [:analytics, :marketing]
|
|
2929
|
+
end
|
|
2930
|
+
|
|
2931
|
+
# Enable SOX compliance
|
|
2932
|
+
sox do
|
|
2933
|
+
enabled true
|
|
2934
|
+
dual_approval_required true
|
|
2935
|
+
|
|
2936
|
+
# Audit settings
|
|
2937
|
+
audit_settings do
|
|
2938
|
+
retention_period 7.years
|
|
2939
|
+
tamper_proof true
|
|
2940
|
+
end
|
|
2941
|
+
end
|
|
2942
|
+
end
|
|
2943
|
+
end
|
|
2944
|
+
```
|
|
2945
|
+
|
|
2946
|
+
---
|
|
2947
|
+
|
|
2948
|
+
### 5.4. Tamper Detection
|
|
2949
|
+
|
|
2950
|
+
**Design Decision:** Automated verification and alerting.
|
|
2951
|
+
|
|
2952
|
+
```ruby
|
|
2953
|
+
module E11y
|
|
2954
|
+
module Security
|
|
2955
|
+
class TamperDetector
|
|
2956
|
+
def initialize(signer, alert_handler)
|
|
2957
|
+
@signer = signer
|
|
2958
|
+
@alert_handler = alert_handler
|
|
2959
|
+
end
|
|
2960
|
+
|
|
2961
|
+
def verify_event(event_data)
|
|
2962
|
+
valid = @signer.verify(event_data)
|
|
2963
|
+
|
|
2964
|
+
unless valid
|
|
2965
|
+
alert_tampering!(event_data)
|
|
2966
|
+
end
|
|
2967
|
+
|
|
2968
|
+
valid
|
|
2969
|
+
end
|
|
2970
|
+
|
|
2971
|
+
def verify_batch(events)
|
|
2972
|
+
# Verify each signature
|
|
2973
|
+
signature_results = events.map do |event|
|
|
2974
|
+
{ event: event, valid: @signer.verify(event) }
|
|
2975
|
+
end
|
|
2976
|
+
|
|
2977
|
+
# Verify chain
|
|
2978
|
+
chain_valid = @signer.verify_chain(events)
|
|
2979
|
+
|
|
2980
|
+
# Check for tampering
|
|
2981
|
+
tampered_events = signature_results.reject { |r| r[:valid] }
|
|
2982
|
+
|
|
2983
|
+
if tampered_events.any? || !chain_valid
|
|
2984
|
+
alert_batch_tampering!(tampered_events, chain_valid)
|
|
2985
|
+
end
|
|
2986
|
+
|
|
2987
|
+
{
|
|
2988
|
+
total: events.size,
|
|
2989
|
+
valid: signature_results.count { |r| r[:valid] },
|
|
2990
|
+
invalid: tampered_events.size,
|
|
2991
|
+
chain_valid: chain_valid
|
|
2992
|
+
}
|
|
2993
|
+
end
|
|
2994
|
+
|
|
2995
|
+
private
|
|
2996
|
+
|
|
2997
|
+
def alert_tampering!(event_data)
|
|
2998
|
+
@alert_handler.call(
|
|
2999
|
+
type: :single_event_tampered,
|
|
3000
|
+
event_id: event_data[:event_id],
|
|
3001
|
+
event_name: event_data[:event_name],
|
|
3002
|
+
timestamp: event_data[:timestamp],
|
|
3003
|
+
severity: :critical
|
|
3004
|
+
)
|
|
3005
|
+
end
|
|
3006
|
+
|
|
3007
|
+
def alert_batch_tampering!(tampered_events, chain_valid)
|
|
3008
|
+
@alert_handler.call(
|
|
3009
|
+
type: :batch_tampered,
|
|
3010
|
+
tampered_count: tampered_events.size,
|
|
3011
|
+
chain_valid: chain_valid,
|
|
3012
|
+
event_ids: tampered_events.map { |e| e[:event][:event_id] },
|
|
3013
|
+
severity: :critical
|
|
3014
|
+
)
|
|
3015
|
+
end
|
|
3016
|
+
end
|
|
3017
|
+
end
|
|
3018
|
+
end
|
|
3019
|
+
```
|
|
3020
|
+
|
|
3021
|
+
**Monitoring:**
|
|
3022
|
+
|
|
3023
|
+
```ruby
|
|
3024
|
+
E11y.configure do |config|
|
|
3025
|
+
config.security.tamper_detection do
|
|
3026
|
+
enabled true
|
|
3027
|
+
|
|
3028
|
+
# Verify on read
|
|
3029
|
+
verify_on_read true
|
|
3030
|
+
|
|
3031
|
+
# Periodic verification
|
|
3032
|
+
periodic_verification do
|
|
3033
|
+
enabled true
|
|
3034
|
+
interval 1.hour
|
|
3035
|
+
sample_size 1000
|
|
3036
|
+
end
|
|
3037
|
+
|
|
3038
|
+
# Alerting
|
|
3039
|
+
on_tampering_detected do |alert|
|
|
3040
|
+
# Critical alert
|
|
3041
|
+
PagerDuty.trigger(
|
|
3042
|
+
service: 'e11y-audit',
|
|
3043
|
+
incident_key: "tamper-#{alert[:type]}",
|
|
3044
|
+
description: "Audit log tampering detected: #{alert[:tampered_count]} events",
|
|
3045
|
+
severity: 'critical'
|
|
3046
|
+
)
|
|
3047
|
+
|
|
3048
|
+
# Log
|
|
3049
|
+
Rails.logger.error "[E11y] TAMPERING DETECTED: #{alert.inspect}"
|
|
3050
|
+
|
|
3051
|
+
# Disable writes (safety)
|
|
3052
|
+
E11y.emergency_readonly_mode!
|
|
3053
|
+
end
|
|
3054
|
+
end
|
|
3055
|
+
end
|
|
3056
|
+
```
|
|
3057
|
+
|
|
3058
|
+
---
|
|
3059
|
+
|
|
3060
|
+
## 5.5. OpenTelemetry Baggage PII Protection (C08 Resolution)
|
|
3061
|
+
|
|
3062
|
+
> **⚠️ CRITICAL: C08 Conflict Resolution - PII × OpenTelemetry Baggage**
|
|
3063
|
+
> **See:** [CONFLICT-ANALYSIS.md C08](researches/CONFLICT-ANALYSIS.md#c08-pii-filtering--opentelemetry-baggage) for detailed analysis
|
|
3064
|
+
> **Problem:** OpenTelemetry Baggage automatically propagates via HTTP headers and can leak PII to downstream services
|
|
3065
|
+
> **Solution:** Block PII from baggage entirely with safe key allowlist
|
|
3066
|
+
|
|
3067
|
+
### 5.5.1. The Problem: PII Leaking via OpenTelemetry Baggage
|
|
3068
|
+
|
|
3069
|
+
**Scenario - GDPR Violation:**
|
|
3070
|
+
|
|
3071
|
+
```ruby
|
|
3072
|
+
# Service A: User registration
|
|
3073
|
+
Events::UserRegistered.track(
|
|
3074
|
+
user_id: '123',
|
|
3075
|
+
email: 'user@example.com', # ← PII, filtered in event payload ✅
|
|
3076
|
+
name: 'John Doe' # ← PII, filtered in event payload ✅
|
|
3077
|
+
)
|
|
3078
|
+
|
|
3079
|
+
# Developer accidentally sets PII in baggage:
|
|
3080
|
+
OpenTelemetry::Baggage.set_value('user_email', 'user@example.com') # ❌ PII in baggage!
|
|
3081
|
+
OpenTelemetry::Baggage.set_value('user_name', 'John Doe') # ❌ PII in baggage!
|
|
3082
|
+
|
|
3083
|
+
# Service A → Service B (HTTP call)
|
|
3084
|
+
# HTTP headers AUTOMATICALLY include baggage:
|
|
3085
|
+
# baggage: user_email=user@example.com,user_name=John Doe
|
|
3086
|
+
|
|
3087
|
+
# Service B receives baggage with PII!
|
|
3088
|
+
# Service B logs baggage → PII LEAKED to downstream service!
|
|
3089
|
+
|
|
3090
|
+
# Result: PII BYPASS!
|
|
3091
|
+
# - Event payload: PII filtered ✅
|
|
3092
|
+
# - Baggage: PII NOT filtered ❌ ← GDPR violation!
|
|
3093
|
+
```
|
|
3094
|
+
|
|
3095
|
+
**Why This Is Critical:**
|
|
3096
|
+
- ✅ OpenTelemetry Baggage is **W3C standard** for trace context propagation
|
|
3097
|
+
- ❌ Baggage is **automatically transmitted** via HTTP headers (`baggage: key=value`)
|
|
3098
|
+
- ❌ Downstream services **receive PII without filtering**
|
|
3099
|
+
- ❌ Logs, metrics, traces in downstream services **contain PII**
|
|
3100
|
+
- ❌ **GDPR Article 5(1)(c) violation:** PII transmitted beyond necessary scope
|
|
3101
|
+
|
|
3102
|
+
### 5.5.2. Decision: Block PII from Baggage Entirely
|
|
3103
|
+
|
|
3104
|
+
**Strategy:** Allowlist-only mode for safe keys, block all others.
|
|
3105
|
+
|
|
3106
|
+
**Safe Keys (Allowed):**
|
|
3107
|
+
- `trace_id` - OpenTelemetry trace ID (not PII)
|
|
3108
|
+
- `span_id` - OpenTelemetry span ID (not PII)
|
|
3109
|
+
- `environment` - Environment name (production, staging, etc.)
|
|
3110
|
+
- `version` - Application version (deployment ID)
|
|
3111
|
+
- `service_name` - Service identifier
|
|
3112
|
+
- `deployment_id` - Deployment identifier
|
|
3113
|
+
- `request_id` - Internal request ID (UUID, not PII)
|
|
3114
|
+
|
|
3115
|
+
**Blocked Keys (PII Risk):**
|
|
3116
|
+
- `user_id`, `user_email`, `user_name` - User identifiers
|
|
3117
|
+
- `ip_address`, `client_ip` - IP addresses (GDPR Article 4(1))
|
|
3118
|
+
- `session_id`, `session_token` - Session identifiers (can be PII)
|
|
3119
|
+
- `api_key`, `token`, `password` - Credentials
|
|
3120
|
+
- Any custom keys not in allowlist
|
|
3121
|
+
|
|
3122
|
+
### 5.5.3. BaggageProtection Middleware Implementation
|
|
3123
|
+
|
|
3124
|
+
```ruby
|
|
3125
|
+
module E11y
|
|
3126
|
+
module Middleware
|
|
3127
|
+
class BaggageProtection
|
|
3128
|
+
# Safe keys that can be set in baggage
|
|
3129
|
+
ALLOWED_KEYS = %w[
|
|
3130
|
+
trace_id
|
|
3131
|
+
span_id
|
|
3132
|
+
environment
|
|
3133
|
+
version
|
|
3134
|
+
service_name
|
|
3135
|
+
deployment_id
|
|
3136
|
+
request_id
|
|
3137
|
+
].freeze
|
|
3138
|
+
|
|
3139
|
+
def initialize(config)
|
|
3140
|
+
@enabled = config.baggage_protection_enabled || true
|
|
3141
|
+
@allowed_keys = config.baggage_allowed_keys || ALLOWED_KEYS
|
|
3142
|
+
@block_mode = config.baggage_block_mode || :silent # :silent, :warn, :raise
|
|
3143
|
+
@logger = E11y.logger
|
|
3144
|
+
end
|
|
3145
|
+
|
|
3146
|
+
def call(event_data)
|
|
3147
|
+
return event_data unless @enabled
|
|
3148
|
+
|
|
3149
|
+
# Intercept OpenTelemetry::Baggage.set_value calls
|
|
3150
|
+
protect_baggage!
|
|
3151
|
+
|
|
3152
|
+
event_data
|
|
3153
|
+
end
|
|
3154
|
+
|
|
3155
|
+
private
|
|
3156
|
+
|
|
3157
|
+
def protect_baggage!
|
|
3158
|
+
# Monkey-patch OpenTelemetry::Baggage (runtime protection)
|
|
3159
|
+
OpenTelemetry::Baggage.singleton_class.prepend(BaggageInterceptor.new(
|
|
3160
|
+
allowed_keys: @allowed_keys,
|
|
3161
|
+
block_mode: @block_mode,
|
|
3162
|
+
logger: @logger
|
|
3163
|
+
))
|
|
3164
|
+
end
|
|
3165
|
+
end
|
|
3166
|
+
|
|
3167
|
+
# Interceptor for OpenTelemetry::Baggage.set_value
|
|
3168
|
+
module BaggageInterceptor
|
|
3169
|
+
def initialize(allowed_keys:, block_mode:, logger:)
|
|
3170
|
+
@allowed_keys = allowed_keys
|
|
3171
|
+
@block_mode = block_mode
|
|
3172
|
+
@logger = logger
|
|
3173
|
+
super()
|
|
3174
|
+
end
|
|
3175
|
+
|
|
3176
|
+
def set_value(key, value, context = nil)
|
|
3177
|
+
# Check if key is allowed
|
|
3178
|
+
unless @allowed_keys.include?(key.to_s)
|
|
3179
|
+
handle_blocked_key(key, value)
|
|
3180
|
+
return context || OpenTelemetry::Context.current
|
|
3181
|
+
end
|
|
3182
|
+
|
|
3183
|
+
# Key is safe, proceed
|
|
3184
|
+
super(key, value, context)
|
|
3185
|
+
end
|
|
3186
|
+
|
|
3187
|
+
private
|
|
3188
|
+
|
|
3189
|
+
def handle_blocked_key(key, value)
|
|
3190
|
+
message = "[E11y] Blocked PII from OpenTelemetry baggage: key=#{key.inspect}"
|
|
3191
|
+
|
|
3192
|
+
case @block_mode
|
|
3193
|
+
when :silent
|
|
3194
|
+
# Block silently (default)
|
|
3195
|
+
@logger.debug(message)
|
|
3196
|
+
when :warn
|
|
3197
|
+
# Block with warning
|
|
3198
|
+
@logger.warn(message)
|
|
3199
|
+
when :raise
|
|
3200
|
+
# Block with exception (strict mode)
|
|
3201
|
+
raise BaggagePiiError, "#{message}. Only allowed keys: #{@allowed_keys.join(', ')}"
|
|
3202
|
+
end
|
|
3203
|
+
end
|
|
3204
|
+
end
|
|
3205
|
+
|
|
3206
|
+
class BaggagePiiError < StandardError; end
|
|
3207
|
+
end
|
|
3208
|
+
end
|
|
3209
|
+
```
|
|
3210
|
+
|
|
3211
|
+
### 5.5.4. Configuration
|
|
3212
|
+
|
|
3213
|
+
```ruby
|
|
3214
|
+
# config/initializers/e11y.rb
|
|
3215
|
+
E11y.configure do |config|
|
|
3216
|
+
config.security.baggage_protection do
|
|
3217
|
+
enabled true # ✅ CRITICAL: Always enable in production
|
|
3218
|
+
|
|
3219
|
+
# Allowlist: Only these keys allowed in baggage
|
|
3220
|
+
allowed_keys [
|
|
3221
|
+
'trace_id',
|
|
3222
|
+
'span_id',
|
|
3223
|
+
'environment',
|
|
3224
|
+
'version',
|
|
3225
|
+
'service_name',
|
|
3226
|
+
'deployment_id',
|
|
3227
|
+
'request_id',
|
|
3228
|
+
# Custom safe keys (optional):
|
|
3229
|
+
'feature_flag_id', # Feature flag identifier (not PII)
|
|
3230
|
+
'ab_test_variant' # A/B test variant (not PII)
|
|
3231
|
+
]
|
|
3232
|
+
|
|
3233
|
+
# Block mode (how to handle violations)
|
|
3234
|
+
block_mode :silent # Options: :silent, :warn, :raise
|
|
3235
|
+
|
|
3236
|
+
# Monitoring
|
|
3237
|
+
on_blocked_key do |key, value, caller_location|
|
|
3238
|
+
# Track violations for security audit
|
|
3239
|
+
Yabeda.e11y_baggage_pii_blocked.increment(
|
|
3240
|
+
key: key,
|
|
3241
|
+
service: ENV['SERVICE_NAME']
|
|
3242
|
+
)
|
|
3243
|
+
|
|
3244
|
+
# Alert on critical violations
|
|
3245
|
+
if key.match?(/email|password|ssn|credit_card/)
|
|
3246
|
+
Sentry.capture_message(
|
|
3247
|
+
"Critical PII blocked from baggage: #{key}",
|
|
3248
|
+
level: :warning,
|
|
3249
|
+
extra: { caller: caller_location }
|
|
3250
|
+
)
|
|
3251
|
+
end
|
|
3252
|
+
end
|
|
3253
|
+
end
|
|
3254
|
+
end
|
|
3255
|
+
```
|
|
3256
|
+
|
|
3257
|
+
### 5.5.5. Usage Examples
|
|
3258
|
+
|
|
3259
|
+
**❌ BAD: Blocked PII Keys**
|
|
3260
|
+
|
|
3261
|
+
```ruby
|
|
3262
|
+
# Service A:
|
|
3263
|
+
OpenTelemetry::Baggage.set_value('user_email', 'user@example.com')
|
|
3264
|
+
# → BLOCKED (not in allowlist)
|
|
3265
|
+
# → Logged: "[E11y] Blocked PII from OpenTelemetry baggage: key='user_email'"
|
|
3266
|
+
|
|
3267
|
+
OpenTelemetry::Baggage.set_value('ip_address', '192.168.1.100')
|
|
3268
|
+
# → BLOCKED (not in allowlist)
|
|
3269
|
+
|
|
3270
|
+
OpenTelemetry::Baggage.set_value('session_id', 'abc123')
|
|
3271
|
+
# → BLOCKED (not in allowlist)
|
|
3272
|
+
|
|
3273
|
+
# HTTP call to Service B:
|
|
3274
|
+
# baggage: (empty - all blocked)
|
|
3275
|
+
```
|
|
3276
|
+
|
|
3277
|
+
**✅ GOOD: Allowed Safe Keys**
|
|
3278
|
+
|
|
3279
|
+
```ruby
|
|
3280
|
+
# Service A:
|
|
3281
|
+
OpenTelemetry::Baggage.set_value('trace_id', 'abc123def456')
|
|
3282
|
+
# → ALLOWED ✅
|
|
3283
|
+
|
|
3284
|
+
OpenTelemetry::Baggage.set_value('environment', 'production')
|
|
3285
|
+
# → ALLOWED ✅
|
|
3286
|
+
|
|
3287
|
+
OpenTelemetry::Baggage.set_value('version', 'v2.1.0')
|
|
3288
|
+
# → ALLOWED ✅
|
|
3289
|
+
|
|
3290
|
+
OpenTelemetry::Baggage.set_value('feature_flag_id', 'new_checkout_v2')
|
|
3291
|
+
# → ALLOWED ✅ (custom safe key)
|
|
3292
|
+
|
|
3293
|
+
# HTTP call to Service B:
|
|
3294
|
+
# baggage: trace_id=abc123def456,environment=production,version=v2.1.0,feature_flag_id=new_checkout_v2
|
|
3295
|
+
# → All safe keys propagated ✅
|
|
3296
|
+
```
|
|
3297
|
+
|
|
3298
|
+
**✅ ALTERNATIVE: Use Non-PII Identifiers**
|
|
3299
|
+
|
|
3300
|
+
```ruby
|
|
3301
|
+
# Instead of user_email in baggage:
|
|
3302
|
+
OpenTelemetry::Baggage.set_value('user_id_hash', Digest::SHA256.hexdigest(user.email))
|
|
3303
|
+
# → Pseudonymized, no PII ✅
|
|
3304
|
+
|
|
3305
|
+
# Instead of session_id:
|
|
3306
|
+
OpenTelemetry::Baggage.set_value('request_id', SecureRandom.uuid)
|
|
3307
|
+
# → Non-PII identifier ✅
|
|
3308
|
+
```
|
|
3309
|
+
|
|
3310
|
+
### 5.5.6. Strict Mode (Development/Staging)
|
|
3311
|
+
|
|
3312
|
+
**Recommended for non-production environments:**
|
|
3313
|
+
|
|
3314
|
+
```ruby
|
|
3315
|
+
# config/environments/development.rb
|
|
3316
|
+
E11y.configure do |config|
|
|
3317
|
+
config.security.baggage_protection do
|
|
3318
|
+
enabled true
|
|
3319
|
+
|
|
3320
|
+
# RAISE exception on blocked keys (fail fast)
|
|
3321
|
+
block_mode :raise # ← Developer sees error immediately
|
|
3322
|
+
|
|
3323
|
+
allowed_keys E11y::Middleware::BaggageProtection::ALLOWED_KEYS
|
|
3324
|
+
end
|
|
3325
|
+
end
|
|
3326
|
+
|
|
3327
|
+
# Developer tries to set PII in baggage:
|
|
3328
|
+
OpenTelemetry::Baggage.set_value('user_email', 'test@example.com')
|
|
3329
|
+
# → RAISES BaggagePiiError:
|
|
3330
|
+
# "[E11y] Blocked PII from OpenTelemetry baggage: key='user_email'.
|
|
3331
|
+
# Only allowed keys: trace_id, span_id, environment, version, ..."
|
|
3332
|
+
```
|
|
3333
|
+
|
|
3334
|
+
### 5.5.7. Trade-offs & GDPR Compliance (C08)
|
|
3335
|
+
|
|
3336
|
+
**Trade-offs:**
|
|
3337
|
+
|
|
3338
|
+
| Decision | Pro | Con | Rationale |
|
|
3339
|
+
|----------|-----|-----|-----------|
|
|
3340
|
+
| **Allowlist-only mode** | No PII can leak | Less flexible | GDPR compliance > flexibility |
|
|
3341
|
+
| **Block at runtime** | No code changes needed | Performance overhead (~0.01ms) | Security > performance |
|
|
3342
|
+
| **Silent mode default** | No breaking changes | Harder to debug | Gradual rollout safer |
|
|
3343
|
+
| **Raise mode (dev)** | Fail fast | Breaks tests | Catch violations early |
|
|
3344
|
+
|
|
3345
|
+
**GDPR Compliance:**
|
|
3346
|
+
|
|
3347
|
+
✅ **Article 5(1)(c) - Data minimisation:**
|
|
3348
|
+
Baggage protection ensures PII is not transmitted beyond necessary scope.
|
|
3349
|
+
|
|
3350
|
+
✅ **Article 5(1)(f) - Integrity and confidentiality:**
|
|
3351
|
+
Blocking PII from automatic propagation protects data integrity.
|
|
3352
|
+
|
|
3353
|
+
✅ **Article 32 - Security of processing:**
|
|
3354
|
+
Technical measure to prevent PII leakage via trace context.
|
|
3355
|
+
|
|
3356
|
+
**Monitoring Metrics:**
|
|
3357
|
+
|
|
3358
|
+
```ruby
|
|
3359
|
+
# Track baggage protection effectiveness
|
|
3360
|
+
Yabeda.e11y_baggage_pii_blocked.increment(
|
|
3361
|
+
key: 'user_email',
|
|
3362
|
+
service: 'api-gateway'
|
|
3363
|
+
)
|
|
3364
|
+
|
|
3365
|
+
# Alert on repeated violations (indicates developer education needed)
|
|
3366
|
+
Yabeda.e11y_baggage_pii_violations_total.increment(
|
|
3367
|
+
caller_service: 'user-service',
|
|
3368
|
+
blocked_key_pattern: 'user_*'
|
|
3369
|
+
)
|
|
3370
|
+
```
|
|
3371
|
+
|
|
3372
|
+
**Related Conflicts:**
|
|
3373
|
+
- **C07:** DLQ replay with PII filtering (see §5.6 below)
|
|
3374
|
+
- **C01:** Audit trail signing with PII (see ADR-015 §3.3)
|
|
3375
|
+
|
|
3376
|
+
---
|
|
3377
|
+
|
|
3378
|
+
## 5.6. PII Handling for Event Replay from DLQ (C07 Resolution)
|
|
3379
|
+
|
|
3380
|
+
> **⚠️ HIGH: C07 Conflict Resolution - PII Pseudonymization × DLQ Replay**
|
|
3381
|
+
> **See:** [CONFLICT-ANALYSIS.md C07](researches/CONFLICT-ANALYSIS.md#c07-pii-pseudonymization--dlq-replay) for detailed analysis
|
|
3382
|
+
> **Problem:** Replayed events from DLQ go through pipeline again and get double-hashed PII
|
|
3383
|
+
> **Solution:** Mark events as already filtered to prevent idempotency violations
|
|
3384
|
+
|
|
3385
|
+
### 5.6.1. The Problem: Double-Hashing PII on Replay
|
|
3386
|
+
|
|
3387
|
+
**Scenario - Data Corruption:**
|
|
3388
|
+
|
|
3389
|
+
```ruby
|
|
3390
|
+
# Original event (first processing):
|
|
3391
|
+
Events::UserLogin.track(
|
|
3392
|
+
user_id: '123',
|
|
3393
|
+
email: 'user@example.com', # ← Original PII
|
|
3394
|
+
ip_address: '192.168.1.1' # ← Original PII
|
|
3395
|
+
)
|
|
3396
|
+
|
|
3397
|
+
# Pipeline step 2: PII Filtering (ADR-015)
|
|
3398
|
+
# → email: 'user@example.com' → SHA256 hash → 'a1b2c3d4...'
|
|
3399
|
+
# → ip_address: '192.168.1.1' → SHA256 hash → 'e5f6g7h8...'
|
|
3400
|
+
|
|
3401
|
+
# Event sent to adapters, but Loki adapter fails
|
|
3402
|
+
# → Event goes to Dead Letter Queue (DLQ)
|
|
3403
|
+
|
|
3404
|
+
# UC-021: DLQ Replay
|
|
3405
|
+
# Event replayed from DLQ → goes through pipeline AGAIN
|
|
3406
|
+
|
|
3407
|
+
# Pipeline step 2: PII Filtering runs AGAIN!
|
|
3408
|
+
# → email: 'a1b2c3d4...' (already hashed!) → SHA256 hash → 'x9y8z7w6...'
|
|
3409
|
+
# ❌ DOUBLE-HASHED! Original: a1b2c3d4, Replay: x9y8z7w6
|
|
3410
|
+
|
|
3411
|
+
# Result: DATA CORRUPTION!
|
|
3412
|
+
# - Original event: { email: 'a1b2c3d4...', ip: 'e5f6g7h8...' }
|
|
3413
|
+
# - Replayed event: { email: 'x9y8z7w6...', ip: 'k9l8m7n6...' }
|
|
3414
|
+
# - Same user, DIFFERENT hashes!
|
|
3415
|
+
```
|
|
3416
|
+
|
|
3417
|
+
**Why This Breaks:**
|
|
3418
|
+
- ❌ **Idempotency violated:** Replay produces different output than original
|
|
3419
|
+
- ❌ **Audit trail corrupted:** Can't correlate original event with replayed event
|
|
3420
|
+
- ❌ **Forensics impossible:** User tracking broken (different email hashes)
|
|
3421
|
+
- ❌ **GDPR violation:** Can't fulfill data deletion requests (can't find all user data)
|
|
3422
|
+
|
|
3423
|
+
### 5.6.2. Decision: Skip PII Filtering for Replayed Events
|
|
3424
|
+
|
|
3425
|
+
**Strategy:** Metadata flag prevents double-processing.
|
|
3426
|
+
|
|
3427
|
+
**Metadata Flags:**
|
|
3428
|
+
- `:pii_filtered` - Event already went through PII filtering (set by PII filter middleware)
|
|
3429
|
+
- `:replayed` - Event is being replayed from DLQ (set by DLQ replay service)
|
|
3430
|
+
|
|
3431
|
+
**When to Skip PII Filtering:**
|
|
3432
|
+
1. Event has `:pii_filtered => true` (already processed)
|
|
3433
|
+
2. Event has `:replayed => true` (replay scenario)
|
|
3434
|
+
3. Both flags present → skip PII filtering entirely
|
|
3435
|
+
|
|
3436
|
+
### 5.6.3. PiiFilter Middleware with Replay Detection
|
|
3437
|
+
|
|
3438
|
+
```ruby
|
|
3439
|
+
module E11y
|
|
3440
|
+
module Middleware
|
|
3441
|
+
class PiiFilter < Base
|
|
3442
|
+
def initialize(config)
|
|
3443
|
+
@config = config
|
|
3444
|
+
@rails_filter = Rails::ParameterFilter.new(Rails.application.config.filter_parameters)
|
|
3445
|
+
@pattern_matcher = PatternMatcher.new(config.patterns)
|
|
3446
|
+
end
|
|
3447
|
+
|
|
3448
|
+
def call(event_data)
|
|
3449
|
+
# ✅ CRITICAL: Check if already filtered (replay scenario)
|
|
3450
|
+
if already_filtered?(event_data)
|
|
3451
|
+
E11y.logger.debug "[E11y] Skipping PII filtering for replayed event: #{event_data[:event_id]}"
|
|
3452
|
+
return event_data # Skip filtering!
|
|
3453
|
+
end
|
|
3454
|
+
|
|
3455
|
+
# Get event class
|
|
3456
|
+
event_class = E11y::Registry.get_class(event_data[:event_name])
|
|
3457
|
+
|
|
3458
|
+
# Check if event contains PII
|
|
3459
|
+
unless event_class.contains_pii?
|
|
3460
|
+
# No PII declared, skip filtering (Tier 1)
|
|
3461
|
+
return event_data
|
|
3462
|
+
end
|
|
3463
|
+
|
|
3464
|
+
# Apply PII filtering
|
|
3465
|
+
filtered_data = apply_pii_rules(event_class, event_data)
|
|
3466
|
+
|
|
3467
|
+
# ✅ Mark as filtered (prevents double-processing)
|
|
3468
|
+
filtered_data[:metadata] ||= {}
|
|
3469
|
+
filtered_data[:metadata][:pii_filtered] = true
|
|
3470
|
+
|
|
3471
|
+
filtered_data
|
|
3472
|
+
end
|
|
3473
|
+
|
|
3474
|
+
private
|
|
3475
|
+
|
|
3476
|
+
def already_filtered?(event_data)
|
|
3477
|
+
metadata = event_data[:metadata] || {}
|
|
3478
|
+
|
|
3479
|
+
# Check for replay flag
|
|
3480
|
+
return true if metadata[:replayed]
|
|
3481
|
+
|
|
3482
|
+
# Check for already-filtered flag
|
|
3483
|
+
return true if metadata[:pii_filtered]
|
|
3484
|
+
|
|
3485
|
+
false
|
|
3486
|
+
end
|
|
3487
|
+
|
|
3488
|
+
def apply_pii_rules(event_class, event_data)
|
|
3489
|
+
# Per-field PII filtering logic
|
|
3490
|
+
# (implementation details in §3.4)
|
|
3491
|
+
#...
|
|
3492
|
+
end
|
|
3493
|
+
end
|
|
3494
|
+
end
|
|
3495
|
+
end
|
|
3496
|
+
```
|
|
3497
|
+
|
|
3498
|
+
### 5.6.4. DLQ Replay Service with Metadata Flags
|
|
3499
|
+
|
|
3500
|
+
```ruby
|
|
3501
|
+
module E11y
|
|
3502
|
+
module DLQ
|
|
3503
|
+
class ReplayService
|
|
3504
|
+
def replay_event(dlq_event)
|
|
3505
|
+
# Extract original event data
|
|
3506
|
+
event_data = dlq_event[:event_data]
|
|
3507
|
+
|
|
3508
|
+
# ✅ CRITICAL: Mark as replayed (skip transformations)
|
|
3509
|
+
event_data[:metadata] ||= {}
|
|
3510
|
+
event_data[:metadata][:replayed] = true
|
|
3511
|
+
event_data[:metadata][:pii_filtered] = true # Already filtered!
|
|
3512
|
+
event_data[:metadata][:replayed_at] = Time.now.utc.iso8601
|
|
3513
|
+
event_data[:metadata][:original_event_id] = event_data[:event_id]
|
|
3514
|
+
event_data[:metadata][:replay_reason] = dlq_event[:failure_reason]
|
|
3515
|
+
|
|
3516
|
+
# Send through pipeline
|
|
3517
|
+
# PII filter middleware will skip (already_filtered? returns true)
|
|
3518
|
+
E11y::Pipeline.process(event_data)
|
|
3519
|
+
|
|
3520
|
+
# Log successful replay
|
|
3521
|
+
E11y.logger.info "[E11y] Replayed event from DLQ: #{event_data[:event_id]}"
|
|
3522
|
+
end
|
|
3523
|
+
|
|
3524
|
+
def replay_batch(dlq_events)
|
|
3525
|
+
dlq_events.each do |event|
|
|
3526
|
+
begin
|
|
3527
|
+
replay_event(event)
|
|
3528
|
+
rescue StandardError => e
|
|
3529
|
+
E11y.logger.error "[E11y] Failed to replay event: #{e.message}"
|
|
3530
|
+
# Re-queue to DLQ with updated metadata
|
|
3531
|
+
end
|
|
3532
|
+
end
|
|
3533
|
+
end
|
|
3534
|
+
end
|
|
3535
|
+
end
|
|
3536
|
+
end
|
|
3537
|
+
```
|
|
3538
|
+
|
|
3539
|
+
### 5.6.5. Configuration
|
|
3540
|
+
|
|
3541
|
+
```ruby
|
|
3542
|
+
# config/initializers/e11y.rb
|
|
3543
|
+
E11y.configure do |config|
|
|
3544
|
+
config.security.pii_filtering do
|
|
3545
|
+
enabled true
|
|
3546
|
+
|
|
3547
|
+
# Replay handling
|
|
3548
|
+
replay_handling do
|
|
3549
|
+
# Skip PII filtering for replayed events (default: true)
|
|
3550
|
+
skip_on_replay true
|
|
3551
|
+
|
|
3552
|
+
# Validate metadata flags (safety check)
|
|
3553
|
+
validate_replay_flags true
|
|
3554
|
+
|
|
3555
|
+
# Warn if replayed event missing pii_filtered flag
|
|
3556
|
+
warn_on_missing_flag :warn # Options: :silent, :warn, :raise
|
|
3557
|
+
end
|
|
3558
|
+
end
|
|
3559
|
+
|
|
3560
|
+
config.dlq.replay do
|
|
3561
|
+
# Metadata flags for replayed events
|
|
3562
|
+
set_metadata_flags do
|
|
3563
|
+
replayed true
|
|
3564
|
+
pii_filtered true
|
|
3565
|
+
replay_timestamp true
|
|
3566
|
+
original_event_id true
|
|
3567
|
+
end
|
|
3568
|
+
end
|
|
3569
|
+
end
|
|
3570
|
+
```
|
|
3571
|
+
|
|
3572
|
+
### 5.6.6. Usage Examples
|
|
3573
|
+
|
|
3574
|
+
**✅ CORRECT: Replay from DLQ with Metadata**
|
|
3575
|
+
|
|
3576
|
+
```ruby
|
|
3577
|
+
# UC-021: Event failed to write to adapter
|
|
3578
|
+
original_event = {
|
|
3579
|
+
event_id: 'evt_123',
|
|
3580
|
+
event_name: 'user.login',
|
|
3581
|
+
payload: {
|
|
3582
|
+
user_id: '123',
|
|
3583
|
+
email: 'a1b2c3d4...', # Already hashed by first pass
|
|
3584
|
+
ip_address: 'e5f6g7h8...' # Already hashed
|
|
3585
|
+
},
|
|
3586
|
+
metadata: {
|
|
3587
|
+
pii_filtered: true # ✅ Set during first pass
|
|
3588
|
+
}
|
|
3589
|
+
}
|
|
3590
|
+
|
|
3591
|
+
# Event sent to DLQ after adapter failure
|
|
3592
|
+
DLQ.enqueue(original_event)
|
|
3593
|
+
|
|
3594
|
+
# Later: Replay from DLQ
|
|
3595
|
+
replay_service = E11y::DLQ::ReplayService.new
|
|
3596
|
+
replay_service.replay_event(original_event)
|
|
3597
|
+
|
|
3598
|
+
# ✅ Result: PII filter skipped, email hash unchanged
|
|
3599
|
+
# Replayed event: { email: 'a1b2c3d4...' } (same as original!)
|
|
3600
|
+
```
|
|
3601
|
+
|
|
3602
|
+
**❌ BAD: Replay without Metadata (Double-Hashing)**
|
|
3603
|
+
|
|
3604
|
+
```ruby
|
|
3605
|
+
# Hypothetical scenario: Replayed event missing metadata
|
|
3606
|
+
replayed_event = {
|
|
3607
|
+
event_id: 'evt_123',
|
|
3608
|
+
event_name: 'user.login',
|
|
3609
|
+
payload: {
|
|
3610
|
+
email: 'a1b2c3d4...', # Already hashed!
|
|
3611
|
+
},
|
|
3612
|
+
metadata: {} # ❌ Missing pii_filtered flag!
|
|
3613
|
+
}
|
|
3614
|
+
|
|
3615
|
+
# Pipeline processes event
|
|
3616
|
+
# PII filter middleware does NOT skip (no flag)
|
|
3617
|
+
# → email: 'a1b2c3d4...' → SHA256 hash → 'x9y8z7w6...'
|
|
3618
|
+
# ❌ DOUBLE-HASHED!
|
|
3619
|
+
|
|
3620
|
+
# UC-021 MUST set metadata flags to prevent this!
|
|
3621
|
+
```
|
|
3622
|
+
|
|
3623
|
+
**✅ ALTERNATIVE: Separate Replay Pipeline**
|
|
3624
|
+
|
|
3625
|
+
```ruby
|
|
3626
|
+
# Optional: Dedicated pipeline for replays (skip all transformations)
|
|
3627
|
+
E11y.configure do |config|
|
|
3628
|
+
config.pipeline_order do
|
|
3629
|
+
# Standard pipeline (for new events)
|
|
3630
|
+
standard_pipeline do
|
|
3631
|
+
step :validation
|
|
3632
|
+
step :pii_filtering # ← Runs for NEW events
|
|
3633
|
+
step :rate_limiting
|
|
3634
|
+
step :sampling
|
|
3635
|
+
step :trace_context
|
|
3636
|
+
step :buffer
|
|
3637
|
+
step :adapters
|
|
3638
|
+
end
|
|
3639
|
+
|
|
3640
|
+
# Replay pipeline (skip transformations)
|
|
3641
|
+
replay_pipeline do
|
|
3642
|
+
step :validation # ← Still validate schema
|
|
3643
|
+
# NO pii_filtering # ← Skip! Already filtered
|
|
3644
|
+
# NO rate_limiting # ← Skip! Already passed
|
|
3645
|
+
# NO sampling # ← Skip! Already sampled
|
|
3646
|
+
step :trace_context # ← Restore trace context
|
|
3647
|
+
step :buffer
|
|
3648
|
+
step :adapters # ← Write to adapters
|
|
3649
|
+
end
|
|
3650
|
+
end
|
|
3651
|
+
|
|
3652
|
+
# Use replay pipeline for DLQ events
|
|
3653
|
+
config.dlq.replay do
|
|
3654
|
+
use_pipeline :replay_pipeline
|
|
3655
|
+
end
|
|
3656
|
+
end
|
|
3657
|
+
```
|
|
3658
|
+
|
|
3659
|
+
### 5.6.7. Idempotency Verification (Testing)
|
|
3660
|
+
|
|
3661
|
+
**Critical:** Verify replay produces identical output.
|
|
3662
|
+
|
|
3663
|
+
```ruby
|
|
3664
|
+
# spec/lib/e11y/dlq/replay_service_spec.rb
|
|
3665
|
+
RSpec.describe E11y::DLQ::ReplayService do
|
|
3666
|
+
describe '#replay_event' do
|
|
3667
|
+
it 'produces identical output for replayed events (idempotency)' do
|
|
3668
|
+
# Original event with PII
|
|
3669
|
+
original_event = Events::UserLogin.track(
|
|
3670
|
+
user_id: '123',
|
|
3671
|
+
email: 'user@example.com',
|
|
3672
|
+
ip_address: '192.168.1.1'
|
|
3673
|
+
)
|
|
3674
|
+
|
|
3675
|
+
# Capture filtered output (first pass)
|
|
3676
|
+
first_pass_output = E11y::Adapters[:elasticsearch].written_events.last
|
|
3677
|
+
|
|
3678
|
+
# Simulate DLQ scenario
|
|
3679
|
+
dlq_event = {
|
|
3680
|
+
event_data: original_event.to_h,
|
|
3681
|
+
failure_reason: 'Adapter timeout'
|
|
3682
|
+
}
|
|
3683
|
+
|
|
3684
|
+
# Replay event
|
|
3685
|
+
replay_service = described_class.new
|
|
3686
|
+
replay_service.replay_event(dlq_event)
|
|
3687
|
+
|
|
3688
|
+
# Capture replayed output (second pass)
|
|
3689
|
+
second_pass_output = E11y::Adapters[:elasticsearch].written_events.last
|
|
3690
|
+
|
|
3691
|
+
# ✅ CRITICAL: Hashes must be IDENTICAL
|
|
3692
|
+
expect(second_pass_output[:payload][:email]).to eq(first_pass_output[:payload][:email])
|
|
3693
|
+
expect(second_pass_output[:payload][:ip_address]).to eq(first_pass_output[:payload][:ip_address])
|
|
3694
|
+
|
|
3695
|
+
# Verify no double-hashing
|
|
3696
|
+
expect(second_pass_output[:payload][:email]).not_to match(/x9y8z7w6/) # Wrong hash
|
|
3697
|
+
expect(second_pass_output[:payload][:email]).to match(/^[a-f0-9]{64}$/) # SHA256
|
|
3698
|
+
end
|
|
3699
|
+
|
|
3700
|
+
it 'skips PII filtering for replayed events' do
|
|
3701
|
+
# Event already processed
|
|
3702
|
+
processed_event = {
|
|
3703
|
+
event_id: 'evt_123',
|
|
3704
|
+
event_name: 'user.login',
|
|
3705
|
+
payload: {
|
|
3706
|
+
email: 'a1b2c3d4...', # Already hashed
|
|
3707
|
+
},
|
|
3708
|
+
metadata: {
|
|
3709
|
+
pii_filtered: true
|
|
3710
|
+
}
|
|
3711
|
+
}
|
|
3712
|
+
|
|
3713
|
+
# Spy on PII filter middleware
|
|
3714
|
+
pii_filter = E11y::Middleware::PiiFilter.instance
|
|
3715
|
+
allow(pii_filter).to receive(:apply_pii_rules).and_call_original
|
|
3716
|
+
|
|
3717
|
+
# Replay event
|
|
3718
|
+
replay_service = described_class.new
|
|
3719
|
+
replay_service.replay_event({ event_data: processed_event })
|
|
3720
|
+
|
|
3721
|
+
# ✅ PII filter should NOT be called (already filtered)
|
|
3722
|
+
expect(pii_filter).not_to have_received(:apply_pii_rules)
|
|
3723
|
+
end
|
|
3724
|
+
end
|
|
3725
|
+
end
|
|
3726
|
+
```
|
|
3727
|
+
|
|
3728
|
+
### 5.6.8. Trade-offs & Audit Trail Integrity (C07)
|
|
3729
|
+
|
|
3730
|
+
**Trade-offs:**
|
|
3731
|
+
|
|
3732
|
+
| Decision | Pro | Con | Rationale |
|
|
3733
|
+
|----------|-----|-----|-----------|
|
|
3734
|
+
| **Skip PII on replay** | Idempotent replay | Complexity (metadata flags) | Audit integrity > simplicity |
|
|
3735
|
+
| **Metadata flag check** | Prevents double-hashing | Runtime overhead (~0.01ms) | Data correctness > performance |
|
|
3736
|
+
| **Separate replay pipeline** | Clear separation | More configuration | Optional advanced feature |
|
|
3737
|
+
| **Validate flags** | Catches missing metadata | May raise errors | Safety > convenience |
|
|
3738
|
+
|
|
3739
|
+
**Audit Trail Integrity:**
|
|
3740
|
+
|
|
3741
|
+
✅ **Idempotency guarantee:**
|
|
3742
|
+
Replay produces identical output → audit trail remains consistent.
|
|
3743
|
+
|
|
3744
|
+
✅ **User tracking:**
|
|
3745
|
+
Email hashes remain stable across replays → GDPR data deletion requests work correctly.
|
|
3746
|
+
|
|
3747
|
+
✅ **Forensic analysis:**
|
|
3748
|
+
Can correlate original event with replayed event using `original_event_id` metadata.
|
|
3749
|
+
|
|
3750
|
+
**Monitoring Metrics:**
|
|
3751
|
+
|
|
3752
|
+
```ruby
|
|
3753
|
+
# Track replay operations
|
|
3754
|
+
Yabeda.e11y_dlq_replays_total.increment(
|
|
3755
|
+
event_type: 'user.login',
|
|
3756
|
+
pii_filtered_skipped: true
|
|
3757
|
+
)
|
|
3758
|
+
|
|
3759
|
+
# Alert on potential double-hashing
|
|
3760
|
+
Yabeda.e11y_pii_double_hash_prevented.increment(
|
|
3761
|
+
event_id: 'evt_123'
|
|
3762
|
+
)
|
|
3763
|
+
```
|
|
3764
|
+
|
|
3765
|
+
**Related Conflicts:**
|
|
3766
|
+
- **C08:** OpenTelemetry Baggage PII protection (see §5.5 above)
|
|
3767
|
+
- **C01:** Audit trail signing (see ADR-015 §3.3)
|
|
3768
|
+
- **C15:** Event versioning × replay (see ADR-012 - user responsibility for schema migrations)
|
|
3769
|
+
|
|
3770
|
+
---
|
|
3771
|
+
|
|
3772
|
+
## 6. GDPR Compliance
|
|
3773
|
+
|
|
3774
|
+
### 6.1. GDPR Features
|
|
3775
|
+
|
|
3776
|
+
```ruby
|
|
3777
|
+
module E11y
|
|
3778
|
+
module Compliance
|
|
3779
|
+
class GdprSupport
|
|
3780
|
+
def initialize(config)
|
|
3781
|
+
@enabled = config.enabled
|
|
3782
|
+
@data_controller = config.data_controller
|
|
3783
|
+
@dpo_email = config.dpo_email
|
|
3784
|
+
end
|
|
3785
|
+
|
|
3786
|
+
# Article 15 - Right of access
|
|
3787
|
+
def export_user_data(user_id)
|
|
3788
|
+
events = E11y::Storage.find_by_user(user_id)
|
|
3789
|
+
|
|
3790
|
+
{
|
|
3791
|
+
data_controller: @data_controller,
|
|
3792
|
+
dpo_contact: @dpo_email,
|
|
3793
|
+
export_date: Time.now.utc,
|
|
3794
|
+
user_id: user_id,
|
|
3795
|
+
events: events.map { |e| sanitize_for_export(e) }
|
|
3796
|
+
}
|
|
3797
|
+
end
|
|
3798
|
+
|
|
3799
|
+
# Article 17 - Right to erasure
|
|
3800
|
+
def delete_user_data(user_id, reason:, requested_by:)
|
|
3801
|
+
# Log deletion (immutable audit)
|
|
3802
|
+
Events::GdprDeletion.track(
|
|
3803
|
+
user_id: user_id,
|
|
3804
|
+
deletion_reason: reason,
|
|
3805
|
+
requested_by: requested_by,
|
|
3806
|
+
approved_by: current_user_id
|
|
3807
|
+
)
|
|
3808
|
+
|
|
3809
|
+
# Mark for deletion (not immediate)
|
|
3810
|
+
E11y::Storage.mark_for_deletion(user_id)
|
|
3811
|
+
|
|
3812
|
+
# Return proof
|
|
3813
|
+
{
|
|
3814
|
+
deletion_requested_at: Time.now.utc,
|
|
3815
|
+
deletion_effective_date: 30.days.from_now,
|
|
3816
|
+
deletion_proof_id: SecureRandom.uuid
|
|
3817
|
+
}
|
|
3818
|
+
end
|
|
3819
|
+
|
|
3820
|
+
# Article 20 - Right to data portability
|
|
3821
|
+
def portable_format(user_id)
|
|
3822
|
+
data = export_user_data(user_id)
|
|
3823
|
+
|
|
3824
|
+
# Convert to portable JSON
|
|
3825
|
+
{
|
|
3826
|
+
format: 'JSON',
|
|
3827
|
+
version: '1.0',
|
|
3828
|
+
data: data
|
|
3829
|
+
}.to_json
|
|
3830
|
+
end
|
|
3831
|
+
end
|
|
3832
|
+
end
|
|
3833
|
+
end
|
|
3834
|
+
```
|
|
3835
|
+
|
|
3836
|
+
---
|
|
3837
|
+
|
|
3838
|
+
## 7. Configuration
|
|
3839
|
+
|
|
3840
|
+
### 7.1. Complete Configuration
|
|
3841
|
+
|
|
3842
|
+
```ruby
|
|
3843
|
+
# config/initializers/e11y.rb
|
|
3844
|
+
E11y.configure do |config|
|
|
3845
|
+
# ============================================================================
|
|
3846
|
+
# PII Filtering
|
|
3847
|
+
# ============================================================================
|
|
3848
|
+
config.security.pii_filtering do
|
|
3849
|
+
enabled true
|
|
3850
|
+
|
|
3851
|
+
# Rails integration
|
|
3852
|
+
use_rails_filters true
|
|
3853
|
+
|
|
3854
|
+
# Pattern matching
|
|
3855
|
+
patterns [:email, :phone, :ssn, :credit_card, :ip_address]
|
|
3856
|
+
|
|
3857
|
+
# Custom patterns
|
|
3858
|
+
custom_pattern :internal_id, /\bINT-\d{6,}\b/
|
|
3859
|
+
|
|
3860
|
+
# Deep scanning
|
|
3861
|
+
deep_scan true
|
|
3862
|
+
|
|
3863
|
+
# Allowlist
|
|
3864
|
+
allowlist [
|
|
3865
|
+
'order.id',
|
|
3866
|
+
/^metadata\.internal_/
|
|
3867
|
+
]
|
|
3868
|
+
|
|
3869
|
+
# Per-adapter rules
|
|
3870
|
+
adapter_rules do
|
|
3871
|
+
adapter :file_audit, mode: :skip
|
|
3872
|
+
adapter :sentry, mode: :strict
|
|
3873
|
+
end
|
|
3874
|
+
|
|
3875
|
+
# Sampling for debug
|
|
3876
|
+
sampling do
|
|
3877
|
+
enabled true
|
|
3878
|
+
sample_rate 0.01
|
|
3879
|
+
max_samples 100
|
|
3880
|
+
end
|
|
3881
|
+
end
|
|
3882
|
+
|
|
3883
|
+
# ============================================================================
|
|
3884
|
+
# Rate Limiting
|
|
3885
|
+
# ============================================================================
|
|
3886
|
+
config.rate_limiting do
|
|
3887
|
+
# Global limit
|
|
3888
|
+
global do
|
|
3889
|
+
enabled true
|
|
3890
|
+
limit 10_000
|
|
3891
|
+
window 1.second
|
|
3892
|
+
strategy :sliding_window
|
|
3893
|
+
redis Redis.current
|
|
3894
|
+
|
|
3895
|
+
on_overflow :drop
|
|
3896
|
+
end
|
|
3897
|
+
|
|
3898
|
+
# Per-event limits
|
|
3899
|
+
per_event do
|
|
3900
|
+
enabled true
|
|
3901
|
+
|
|
3902
|
+
limits do
|
|
3903
|
+
event 'order.paid', limit: 1000
|
|
3904
|
+
event 'user.login', limit: 500
|
|
3905
|
+
event 'debug.*', limit: 100
|
|
3906
|
+
end
|
|
3907
|
+
|
|
3908
|
+
default_limit 1000
|
|
3909
|
+
end
|
|
3910
|
+
|
|
3911
|
+
# Per-context limits
|
|
3912
|
+
per_context do
|
|
3913
|
+
enabled true
|
|
3914
|
+
context_keys [:user_id, :session_id, :ip_address]
|
|
3915
|
+
limit 100
|
|
3916
|
+
window 1.minute
|
|
3917
|
+
|
|
3918
|
+
allowlist do
|
|
3919
|
+
user_ids ['admin', 'system']
|
|
3920
|
+
end
|
|
3921
|
+
end
|
|
3922
|
+
end
|
|
3923
|
+
|
|
3924
|
+
# ============================================================================
|
|
3925
|
+
# Audit Trail
|
|
3926
|
+
# ============================================================================
|
|
3927
|
+
config.security.audit do
|
|
3928
|
+
enabled true
|
|
3929
|
+
|
|
3930
|
+
# Signing
|
|
3931
|
+
signing do
|
|
3932
|
+
enabled true
|
|
3933
|
+
secret_key ENV['E11Y_AUDIT_SECRET']
|
|
3934
|
+
algorithm 'SHA256'
|
|
3935
|
+
include_chain true
|
|
3936
|
+
end
|
|
3937
|
+
|
|
3938
|
+
# Tamper detection
|
|
3939
|
+
tamper_detection do
|
|
3940
|
+
enabled true
|
|
3941
|
+
verify_on_read true
|
|
3942
|
+
|
|
3943
|
+
periodic_verification do
|
|
3944
|
+
enabled true
|
|
3945
|
+
interval 1.hour
|
|
3946
|
+
sample_size 1000
|
|
3947
|
+
end
|
|
3948
|
+
|
|
3949
|
+
on_tampering_detected do |alert|
|
|
3950
|
+
PagerDuty.trigger(alert)
|
|
3951
|
+
E11y.emergency_readonly_mode!
|
|
3952
|
+
end
|
|
3953
|
+
end
|
|
3954
|
+
|
|
3955
|
+
# Retention
|
|
3956
|
+
default_retention 2555.days # 7 years
|
|
3957
|
+
end
|
|
3958
|
+
|
|
3959
|
+
# ============================================================================
|
|
3960
|
+
# GDPR Compliance
|
|
3961
|
+
# ============================================================================
|
|
3962
|
+
config.compliance.gdpr do
|
|
3963
|
+
enabled true
|
|
3964
|
+
data_controller 'Company Inc.'
|
|
3965
|
+
dpo_email 'dpo@company.com'
|
|
3966
|
+
|
|
3967
|
+
# Consent
|
|
3968
|
+
track_consent true
|
|
3969
|
+
consent_required_for [:analytics, :marketing]
|
|
3970
|
+
|
|
3971
|
+
# Data subject rights
|
|
3972
|
+
support_access_requests true
|
|
3973
|
+
support_deletion_requests true
|
|
3974
|
+
support_portability true
|
|
3975
|
+
|
|
3976
|
+
# Deletion
|
|
3977
|
+
deletion_delay 30.days
|
|
3978
|
+
permanent_deletion_after 90.days
|
|
3979
|
+
end
|
|
3980
|
+
end
|
|
3981
|
+
```
|
|
3982
|
+
|
|
3983
|
+
---
|
|
3984
|
+
|
|
3985
|
+
## 8. Testing
|
|
3986
|
+
|
|
3987
|
+
### 8.1. RSpec Examples
|
|
3988
|
+
|
|
3989
|
+
```ruby
|
|
3990
|
+
RSpec.describe E11y::Security::PiiFilter do
|
|
3991
|
+
describe 'PII filtering' do
|
|
3992
|
+
it 'filters email addresses' do
|
|
3993
|
+
event = Events::UserLogin.track(
|
|
3994
|
+
email: 'user@example.com',
|
|
3995
|
+
password: 'secret123'
|
|
3996
|
+
)
|
|
3997
|
+
|
|
3998
|
+
expect(event.payload[:email]).to eq('[FILTERED:EMAIL]')
|
|
3999
|
+
expect(event.payload[:password]).to eq('[FILTERED]')
|
|
4000
|
+
end
|
|
4001
|
+
|
|
4002
|
+
it 'deep scans nested data' do
|
|
4003
|
+
event = Events::OrderCreated.track(
|
|
4004
|
+
order: {
|
|
4005
|
+
customer: {
|
|
4006
|
+
email: 'user@example.com'
|
|
4007
|
+
}
|
|
4008
|
+
}
|
|
4009
|
+
)
|
|
4010
|
+
|
|
4011
|
+
expect(event.payload[:order][:customer][:email]).to eq('[FILTERED:EMAIL]')
|
|
4012
|
+
end
|
|
4013
|
+
end
|
|
4014
|
+
end
|
|
4015
|
+
|
|
4016
|
+
RSpec.describe E11y::RateLimiting do
|
|
4017
|
+
describe 'global rate limiting' do
|
|
4018
|
+
it 'enforces global limit' do
|
|
4019
|
+
# Track 100 events (within limit)
|
|
4020
|
+
100.times do
|
|
4021
|
+
expect(Events::Test.track(message: 'test')).to be_truthy
|
|
4022
|
+
end
|
|
4023
|
+
|
|
4024
|
+
# Exceed limit
|
|
4025
|
+
E11y.config.rate_limiting.global.limit = 100
|
|
4026
|
+
|
|
4027
|
+
expect(Events::Test.track(message: 'over limit')).to be_falsey
|
|
4028
|
+
end
|
|
4029
|
+
end
|
|
4030
|
+
end
|
|
4031
|
+
|
|
4032
|
+
RSpec.describe E11y::Security::EventSigner do
|
|
4033
|
+
describe 'cryptographic signing' do
|
|
4034
|
+
it 'signs events' do
|
|
4035
|
+
event = Events::AuditEvent.track(action: 'delete_user')
|
|
4036
|
+
|
|
4037
|
+
expect(event.signature).to be_present
|
|
4038
|
+
expect(event.signed_at).to be_present
|
|
4039
|
+
end
|
|
4040
|
+
|
|
4041
|
+
it 'detects tampering' do
|
|
4042
|
+
event = Events::AuditEvent.track(action: 'delete_user')
|
|
4043
|
+
|
|
4044
|
+
# Tamper with event
|
|
4045
|
+
event.payload[:action] = 'create_user'
|
|
4046
|
+
|
|
4047
|
+
expect(E11y.security.signer.verify(event)).to be_falsey
|
|
4048
|
+
end
|
|
4049
|
+
|
|
4050
|
+
it 'verifies signature chain' do
|
|
4051
|
+
events = 3.times.map { Events::AuditEvent.track(action: 'test') }
|
|
4052
|
+
|
|
4053
|
+
expect(E11y.security.signer.verify_chain(events)).to be_truthy
|
|
4054
|
+
end
|
|
4055
|
+
end
|
|
4056
|
+
end
|
|
4057
|
+
```
|
|
4058
|
+
|
|
4059
|
+
---
|
|
4060
|
+
|
|
4061
|
+
## 9. Trade-offs
|
|
4062
|
+
|
|
4063
|
+
### 9.1. Key Decisions
|
|
4064
|
+
|
|
4065
|
+
| Decision | Pro | Con | Rationale |
|
|
4066
|
+
|----------|-----|-----|-----------|
|
|
4067
|
+
| **Rails filter integration** | Zero config | Rails dependency | Most teams use Rails |
|
|
4068
|
+
| **Pattern-based PII** | Flexible | False positives | Good enough for 95% |
|
|
4069
|
+
| **Per-adapter rules** | Fine-grained | Complexity | Critical for compliance |
|
|
4070
|
+
| **Redis rate limiting** | Distributed | External dep | Required for scale |
|
|
4071
|
+
| **HMAC signatures** | Fast | Not PKI | Speed > PKI for logs |
|
|
4072
|
+
| **Chain verification** | Detects gaps | Memory overhead | Critical for audit |
|
|
4073
|
+
| **Audit skip PII** | Compliance | Privacy risk | Legal obligation wins |
|
|
4074
|
+
| **Retry count toward limit** | System safety | Less delivery | Prevent amplification |
|
|
4075
|
+
| **Baggage allowlist (C08)** | No PII leaks | Less flexible | GDPR compliance > flexibility |
|
|
4076
|
+
| **Block baggage at runtime** | No code changes | 0.01ms overhead | Security > performance |
|
|
4077
|
+
|
|
4078
|
+
### 9.2. Pipeline Order (From CONFLICT-ANALYSIS.md)
|
|
4079
|
+
|
|
4080
|
+
**Definitive processing order:**
|
|
4081
|
+
|
|
4082
|
+
```ruby
|
|
4083
|
+
track(event)
|
|
4084
|
+
↓
|
|
4085
|
+
1. Schema Validation (fail fast)
|
|
4086
|
+
↓
|
|
4087
|
+
2. Context Enrichment (trace_id, user_id)
|
|
4088
|
+
↓
|
|
4089
|
+
3. PII Filtering (security first)
|
|
4090
|
+
↓ [Per-adapter: different rules]
|
|
4091
|
+
↓
|
|
4092
|
+
4. Rate Limiting (system protection) ← Retries also checked here
|
|
4093
|
+
↓ [Drop if exceeded → DLQ if retry]
|
|
4094
|
+
↓
|
|
4095
|
+
5. Adaptive Sampling (cost optimization)
|
|
4096
|
+
↓ [Drop if not sampled]
|
|
4097
|
+
↓
|
|
4098
|
+
6. Buffer Routing (debug vs. main)
|
|
4099
|
+
↓
|
|
4100
|
+
7. Adapters (with circuit breakers)
|
|
4101
|
+
```
|
|
4102
|
+
|
|
4103
|
+
**Key Insight:** Rate limiting BEFORE sampling (Conflict #2 resolution)
|
|
4104
|
+
- Rate limiting = system stability (higher priority)
|
|
4105
|
+
- Sampling = cost optimization (lower priority)
|
|
4106
|
+
|
|
4107
|
+
### 9.3. Precedence Rules (From CONFLICT-ANALYSIS.md)
|
|
4108
|
+
|
|
4109
|
+
**When multiple configs apply:**
|
|
4110
|
+
|
|
4111
|
+
1. **Event-level config** (highest priority)
|
|
4112
|
+
- `event.retention = 7.years`
|
|
4113
|
+
- `event.adapters = [:audit_file]`
|
|
4114
|
+
- `event.skip_pii_filtering = true`
|
|
4115
|
+
|
|
4116
|
+
2. **Per-event-type config**
|
|
4117
|
+
- `per_event 'payment.*' { sample_rate: 1.0 }`
|
|
4118
|
+
|
|
4119
|
+
3. **Per-severity config**
|
|
4120
|
+
- `per_severity :fatal { adapters: [:sentry, :pagerduty] }`
|
|
4121
|
+
|
|
4122
|
+
4. **Global config** (lowest priority)
|
|
4123
|
+
- `config.default_adapters = [:loki]`
|
|
4124
|
+
|
|
4125
|
+
### 9.4. Alternatives Considered
|
|
4126
|
+
|
|
4127
|
+
**A) No PII filtering**
|
|
4128
|
+
- ❌ Rejected: GDPR violations
|
|
4129
|
+
|
|
4130
|
+
**B) ML-based PII detection**
|
|
4131
|
+
- ❌ Rejected: Too slow, over-engineering for v1.0
|
|
4132
|
+
|
|
4133
|
+
**C) Token bucket only**
|
|
4134
|
+
- ❌ Rejected: Need sliding window for accuracy
|
|
4135
|
+
|
|
4136
|
+
**D) PKI signatures**
|
|
4137
|
+
- ❌ Rejected: Overkill, HMAC sufficient
|
|
4138
|
+
|
|
4139
|
+
---
|
|
4140
|
+
|
|
4141
|
+
**Status:** ✅ Draft Complete
|
|
4142
|
+
**Next:** ADR-008 (Rails Integration) or ADR-011 (Testing Strategy)
|
|
4143
|
+
**Estimated Implementation:** 3 weeks
|