e11y 0.2.0 → 1.0.0

data/docs/{ADR-006-security-compliance.md → architecture/ADR-006-security-compliance.md}

@@ -21,7 +21,7 @@
    - 4.1. [Global Rate Limiting](#41-global-rate-limiting)
    - 4.2. [Per-Event Rate Limiting](#42-per-event-rate-limiting)
    - 4.3. [Per-Context Rate Limiting](#43-per-context-rate-limiting)
-   - 4.4. [Redis Integration](#44-redis-integration)
+   - 4.4. [In-Memory Token Bucket Implementation](#44-in-memory-token-bucket-implementation)
 5. [Audit Trail](#5-audit-trail)
    - 5.1. [Immutable Events](#51-immutable-events)
    - 5.2. [Cryptographic Signing](#52-cryptographic-signing)
@@ -121,12 +121,10 @@ C4Context
     
     System(e11y, "E11y Gem", "Event tracking with security")
     
-    System_Ext(redis, "Redis", "Rate limiting state")
     System_Ext(kms, "KMS", "Signing keys")
     System_Ext(audit_store, "Audit Store", "Immutable audit logs")
     
     Rel(dev, e11y, "Tracks events", "Events::OrderPaid.track(...)")
-    Rel(e11y, redis, "Check rate limits", "Redis commands")
     Rel(e11y, kms, "Sign audit events", "HMAC-SHA256")
     Rel(e11y, audit_store, "Store signed events", "Append-only")
     
@@ -151,14 +149,12 @@ graph TB
         PII --> PerAdapter[Per-Adapter Rules]
     end
     
-    subgraph "Rate Limiting"
-        Rate --> Global[Global Limiter]
-        Rate --> PerEvent[Per-Event Limiter]
-        Rate --> PerContext[Per-Context Limiter]
+    subgraph "Rate Limiting (In-Memory)"
+        Rate --> Global[Global Token Bucket]
+        Rate --> PerEvent[Per-Event Token Buckets]
+        Rate --> PerContext[Per-Context Token Buckets]
         
-        Global --> Redis1[Redis]
-        PerEvent --> Redis2[Redis]
-        PerContext --> Redis3[Redis]
+        Note1[In-Memory<br/>No Redis Dependency]
     end
     
     subgraph "Audit Trail"
@@ -239,19 +235,19 @@ sequenceDiagram
 
 #### 3.0.2. Three-Tier Filtering Strategy
 
-E11y uses a **3-tier approach** to balance security and performance:
+E11y uses a **3-tier approach** to balance security and performance. Code uses `pii_filtering_mode` returning `:no_pii`, `:rails_filters`, or `:explicit_pii`:
 
-| Tier | Strategy | Cost | Use Case | Events/sec |
+| Mode | Strategy | Cost | Use Case | Events/sec |
 |------|----------|------|----------|------------|
-| **Tier 1** | Skip filtering | 0ms | Health checks, metrics, internal events | 500 |
-| **Tier 2** | Rails filters only | ~0.05ms | Standard events (known PII keys) | 400 |
-| **Tier 3** | Deep filtering | ~0.2ms | User data, payments, complex nested | 100 |
+| **:no_pii** | Skip filtering | 0ms | Health checks, metrics, internal events | 500 |
+| **:rails_filters** | Rails filters only | ~0.05ms | Standard events (known PII keys) | 400 |
+| **:explicit_pii** | Deep filtering | ~0.2ms | User data, payments, complex nested | 100 |
 
 **Performance Budget:**
 ```
-500 events/sec × 0ms     = 0ms CPU/sec      (Tier 1)
-400 events/sec × 0.05ms  = 20ms CPU/sec     (Tier 2)
-100 events/sec × 0.2ms   = 20ms CPU/sec     (Tier 3)
+500 events/sec × 0ms     = 0ms CPU/sec      (:no_pii)
+400 events/sec × 0.05ms  = 20ms CPU/sec     (:rails_filters)
+100 events/sec × 0.2ms   = 20ms CPU/sec     (:explicit_pii)
 ---
 Total: 40ms CPU/sec = 4% CPU on single core ✅
 ```
@@ -268,7 +264,7 @@ class Events::HealthCheck < E11y::Event::Base
   end
   
   # ✅ Explicit: This event contains NO PII
-  contains_pii false  # Skip all PII filtering (Tier 1)
+  contains_pii false  # Skip all PII filtering (:no_pii)
 end
 ```
 
@@ -280,7 +276,7 @@ class Events::OrderCreated < E11y::Event::Base
     required(:amount).filled(:float)
   end
   
-  # No declaration → Rails filters applied (Tier 2, default)
+  # No declaration → Rails filters applied (:rails_filters, default)
   # Uses: Rails.application.config.filter_parameters
 end
 ```
@@ -296,7 +292,7 @@ class Events::UserRegistered < E11y::Event::Base
   end
   
   # ✅ Explicit: This event contains PII
-  contains_pii true  # Tier 3: Deep filtering
+  contains_pii true  # :explicit_pii: Deep filtering
   
   pii_filtering do
     # ✅ MANDATORY: Explicit declaration for EACH schema field
@@ -389,7 +385,7 @@ module E11y
   module Linters
     class PiiDeclarationLinter
       def self.validate_all!
-        E11y::Registry.all_events.each do |event_class|
+        E11y::Registry.event_classes.each do |event_class|
           validate!(event_class)
         end
       end
@@ -480,7 +476,7 @@ namespace :e11y do
       errors = []
       warnings = []
       
-      E11y::Registry.all_events.each do |event_class|
+      E11y::Registry.event_classes.each do |event_class|
         begin
           E11y::Linters::PiiDeclarationLinter.validate!(event_class)
           
@@ -567,7 +563,7 @@ end
 #### 3.0.6. Default Behavior
 
 **If `contains_pii` not specified:**
-- ✅ Apply Rails filters only (Tier 2)
+- ✅ Apply Rails filters only (:rails_filters)
 - ✅ Fast (0.05ms overhead)
 - ✅ Covers 90% of use cases (password, token, secret, api_key)
 - ⚠️ No linter validation (assumes you know what you're doing)
@@ -639,7 +635,10 @@ end
       └─→ PII Filtering (per-adapter, different rules per destination)
    ```
 
-**Key Insight:** PII filtering is **NOT a global middleware** — it's applied **inside each adapter** with different rules!
+**Key Insight (Hybrid Implementation):** PII filtering uses a **hybrid model**:
+- **:no_pii / :rails_filters:** Global middleware (PIIFilter) — skip or Rails filters only
+- **:explicit_pii without exclude_adapters:** Global middleware — single filtered payload
+- **:explicit_pii with exclude_adapters:** Per-adapter — PIIFilter produces `payload_rewrites`, Routing merges overrides per adapter (e.g. audit gets original for GDPR Art. 6(1)(c))
 
 ### 3.1. Rails Integration
 
@@ -1908,7 +1907,7 @@ RSpec.describe 'PII Filtering', type: :integration do
     end
   end
   
-  describe 'Tier 1: No PII (Skip Filtering)' do
+  describe ':no_pii (Skip Filtering)' do
     let(:event_class) { Events::HealthCheck }
     let(:payload) { { status: 'ok' } }
     
@@ -1924,7 +1923,7 @@ RSpec.describe 'PII Filtering', type: :integration do
     end
   end
   
-  describe 'Tier 2: Rails Filters Only' do
+  describe ':rails_filters (Rails Filters Only)' do
     let(:event_class) { Events::OrderCreated }
     let(:payload) do
       {
@@ -1950,7 +1949,7 @@ RSpec.describe 'PII Filtering', type: :integration do
     end
   end
   
-  describe 'Tier 3: Deep Filtering' do
+  describe ':explicit_pii (Deep Filtering)' do
     let(:event_class) { Events::UserRegistered }
     let(:payload) do
       {
@@ -1994,9 +1993,9 @@ RSpec.describe 'PII Filtering', type: :integration do
   describe 'Performance Regression Test' do
     it 'meets performance budget for 1000 events/sec' do
       events = [
-        { class: Events::HealthCheck, payload: { status: 'ok' }, count: 500 },         # Tier 1
-        { class: Events::OrderCreated, payload: { id: 'o1' }, count: 400 },            # Tier 2
-        { class: Events::UserLogin, payload: { email: 'u@x.com' }, count: 100 }        # Tier 3
+        { class: Events::HealthCheck, payload: { status: 'ok' }, count: 500 },         # :no_pii
+        { class: Events::OrderCreated, payload: { id: 'o1' }, count: 400 },            # :rails_filters
+        { class: Events::UserLogin, payload: { email: 'u@x.com' }, count: 100 }        # :explicit_pii
       ]
       
       start = Time.now
@@ -2112,7 +2111,7 @@ end
 bundle exec rspec spec/lib/e11y/security/pii_filtering_spec.rb
 
 # Test specific tier
-bundle exec rspec spec/lib/e11y/security/pii_filtering_spec.rb -e "Tier 3"
+bundle exec rspec spec/e11y/middleware/pii_filtering_spec.rb -e "explicit_pii"
 
 # Performance test
 bundle exec rspec spec/lib/e11y/security/pii_filtering_spec.rb -e "Performance"
@@ -2183,31 +2182,65 @@ module E11y
       def initialize(config)
         @limit = config.limit  # events per second
         @window = config.window || 1.second
-        @strategy = config.strategy || :sliding_window
-        @redis = config.redis
         
-        @counter = initialize_counter
+        # In-memory token bucket (no Redis dependency)
+        @bucket = TokenBucket.new(
+          capacity: @limit,
+          refill_rate: @limit,
+          window: @window.to_f
+        )
       end
       
       def allow?(event_data = nil)
-        @counter.increment('global')
-        
-        current_rate = @counter.rate('global', window: @window)
-        
-        current_rate <= @limit
+        @bucket.allow?
       end
       
       def current_rate
-        @counter.rate('global', window: @window)
+        # Calculate current rate based on tokens consumed
+        @limit - @bucket.tokens
       end
       
       private
       
-      def initialize_counter
-        if @redis
-          RedisCounter.new(@redis, strategy: @strategy)
-        else
-          InMemoryCounter.new(strategy: @strategy)
+      class TokenBucket
+        def initialize(capacity:, refill_rate:, window:)
+          @capacity = capacity
+          @refill_rate = refill_rate
+          @window = window
+          @tokens = capacity.to_f
+          @last_refill = Time.now
+          @mutex = Mutex.new
+        end
+        
+        def allow?
+          @mutex.synchronize do
+            refill_tokens
+            if @tokens >= 1.0
+              @tokens -= 1.0
+              true
+            else
+              false
+            end
+          end
+        end
+        
+        def tokens
+          @mutex.synchronize do
+            refill_tokens
+            @tokens
+          end
+        end
+        
+        private
+        
+        def refill_tokens
+          now = Time.now
+          elapsed = now - @last_refill
+          return if elapsed <= 0
+          
+          tokens_to_add = elapsed * @refill_rate
+          @tokens = [@tokens + tokens_to_add, @capacity].min
+          @last_refill = now
         end
       end
     end
@@ -2252,9 +2285,18 @@ module E11y
         @limits = config.limits || {}
         @default_limit = config.default_limit || Float::INFINITY
         @window = config.window || 1.second
-        @redis = config.redis
         
-        @counter = initialize_counter
+        # In-memory token buckets per event type (lazy initialization)
+        @buckets = Hash.new do |hash, event_name|
+          limit = @limits[event_name] || @default_limit
+          next nil if limit == Float::INFINITY
+          hash[event_name] = TokenBucket.new(
+            capacity: limit,
+            refill_rate: limit,
+            window: @window.to_f
+          )
+        end
+        @mutex = Mutex.new
       end
       
       def allow?(event_data)
@@ -2263,15 +2305,15 @@ module E11y
         
         return true if limit == Float::INFINITY
         
-        @counter.increment(event_name)
-        
-        current_rate = @counter.rate(event_name, window: @window)
-        
-        current_rate <= limit
+        bucket = @mutex.synchronize { @buckets[event_name] }
+        bucket&.allow? || false
       end
       
       def current_rate(event_name)
-        @counter.rate(event_name, window: @window)
+        bucket = @mutex.synchronize { @buckets[event_name] }
+        return Float::INFINITY unless bucket
+        limit = @limits[event_name] || @default_limit
+        limit - bucket.tokens
       end
     end
   end
@@ -2316,9 +2358,16 @@ module E11y
         @limit = config.limit
         @window = config.window || 1.minute
         @context_keys = config.context_keys || [:user_id]
-        @redis = config.redis
         
-        @counter = initialize_counter
+        # In-memory token buckets per context value (lazy initialization)
+        @buckets = Hash.new do |hash, context_value|
+          hash[context_value] = TokenBucket.new(
+            capacity: @limit,
+            refill_rate: @limit,
+            window: @window.to_f
+          )
+        end
+        @mutex = Mutex.new
       end
       
       def allow?(event_data)
@@ -2326,12 +2375,8 @@ module E11y
         
         return true unless context_value
         
-        key = "context:#{context_value}"
-        @counter.increment(key)
-        
-        current_rate = @counter.rate(key, window: @window)
-        
-        current_rate <= @limit
+        bucket = @mutex.synchronize { @buckets[context_value] }
+        bucket.allow?
       end
       
       private
@@ -2376,88 +2421,58 @@ end
 
 ---
 
-### 4.4. Redis Integration
+### 4.4. In-Memory Token Bucket Implementation
+
+**Design Decision:** In-memory token bucket algorithm (no Redis dependency).
 
-**Design Decision:** Distributed rate limiting with Redis.
+**Rationale:**
+- ✅ **Fast:** No network latency (O(1) operations, ~0.003ms per event)
+- ✅ **Simple:** No external dependencies (Redis not required)
+- ✅ **Thread-safe:** Mutex-protected token buckets
+- ✅ **Smooth rate limiting:** Token bucket avoids bursty behavior
+- ⚠️ **Trade-off:** Per-process limits (not shared across instances)
 
+**Current Implementation:**
 ```ruby
 module E11y
-  module RateLimiting
-    class RedisCounter
-      def initialize(redis, strategy: :sliding_window)
-        @redis = redis
-        @strategy = strategy
-      end
-      
-      def increment(key)
-        case @strategy
-        when :sliding_window
-          increment_sliding_window(key)
-        when :fixed_window
-          increment_fixed_window(key)
-        when :token_bucket
-          increment_token_bucket(key)
-        end
-      end
-      
-      def rate(key, window:)
-        case @strategy
-        when :sliding_window
-          count_sliding_window(key, window)
-        when :fixed_window
-          count_fixed_window(key, window)
-        when :token_bucket
-          tokens_remaining(key)
-        end
-      end
-      
-      private
-      
-      # Sliding Window (most accurate)
-      def increment_sliding_window(key)
-        now = Time.now.to_f
-        redis_key = "e11y:rate:#{key}"
-        
-        @redis.multi do |multi|
-          multi.zadd(redis_key, now, "#{now}:#{SecureRandom.uuid}")
-          multi.expire(redis_key, 60)
-        end
-      end
-      
-      def count_sliding_window(key, window)
-        now = Time.now.to_f
-        cutoff = now - window.to_f
-        redis_key = "e11y:rate:#{key}"
-        
-        # Remove old entries
-        @redis.zremrangebyscore(redis_key, '-inf', cutoff)
-        
-        # Count remaining
-        @redis.zcard(redis_key)
-      end
-      
-      # Fixed Window (simpler, less accurate)
-      def increment_fixed_window(key)
-        window_key = current_window_key(key)
-        
-        @redis.multi do |multi|
-          multi.incr(window_key)
-          multi.expire(window_key, 60)
+  module Middleware
+    class RateLimiting < Base
+      def initialize(app, global_limit: 10_000, per_event_limit: 1_000, window: 1.0)
+        super(app)
+        @global_limit = global_limit
+        @per_event_limit = per_event_limit
+        @window = window
+
+        # Token buckets for rate limiting (in-memory)
+        @global_bucket = TokenBucket.new(
+          capacity: @global_limit,
+          refill_rate: @global_limit,
+          window: @window
+        )
+        @per_event_buckets = Hash.new do |hash, event_name|
+          hash[event_name] = TokenBucket.new(
+            capacity: @per_event_limit,
+            refill_rate: @per_event_limit,
+            window: @window
+          )
         end
+
+        @mutex = Mutex.new
       end
-      
-      def count_fixed_window(key, window)
-        window_key = current_window_key(key)
-        @redis.get(window_key).to_i
-      end
-      
-      def current_window_key(key)
-        window_start = (Time.now.to_i / 1) * 1  # 1-second windows
-        "e11y:rate:fixed:#{key}:#{window_start}"
-      end
-      
-      # Token Bucket (burst-friendly)
-      def increment_token_bucket(key)
+    end
+  end
+end
+```
+
+**Why Not Redis?**
+- ❌ **Complexity:** Adds external dependency and infrastructure overhead
+- ❌ **Latency:** Network round-trip adds ~0.25ms per event (83x slower)
+- ❌ **Not needed:** Per-process rate limiting is sufficient for event tracking
+- ✅ **User feedback:** "outdated solution"
+
+**Note:** In-memory rate limiting is sufficient for most use cases. Each application process maintains its own rate limits, which is appropriate for event tracking workloads. If distributed rate limiting is needed in the future, it can be added as an optional feature.
+
+---
         redis_key = "e11y:rate:bucket:#{key}"
         
         # Lua script for atomic token consumption
@@ -3213,44 +3228,24 @@ end
 ```ruby
 # config/initializers/e11y.rb
 E11y.configure do |config|
-  config.security.baggage_protection do
-    enabled true  # ✅ CRITICAL: Always enable in production
-    
-    # Allowlist: Only these keys allowed in baggage
-    allowed_keys [
-      'trace_id',
-      'span_id',
-      'environment',
-      'version',
-      'service_name',
-      'deployment_id',
-      'request_id',
-      # Custom safe keys (optional):
-      'feature_flag_id',   # Feature flag identifier (not PII)
-      'ab_test_variant'    # A/B test variant (not PII)
-    ]
-    
-    # Block mode (how to handle violations)
-    block_mode :silent   # Options: :silent, :warn, :raise
-    
-    # Monitoring
-    on_blocked_key do |key, value, caller_location|
-      # Track violations for security audit
-      Yabeda.e11y_baggage_pii_blocked.increment(
-        key: key,
-        service: ENV['SERVICE_NAME']
-      )
-      
-      # Alert on critical violations
-      if key.match?(/email|password|ssn|credit_card/)
-        Sentry.capture_message(
-          "Critical PII blocked from baggage: #{key}",
-          level: :warning,
-          extra: { caller: caller_location }
-        )
-      end
-    end
-  end
+  config.security_baggage_protection_enabled = true  # ✅ CRITICAL: Always enable in production
+  
+  # Allowlist: Only these keys allowed in baggage
+  config.security_baggage_protection_allowed_keys = [
+    'trace_id',
+    'span_id',
+    'environment',
+    'version',
+    'service_name',
+    'deployment_id',
+    'request_id',
+    # Custom safe keys (optional):
+    'feature_flag_id',   # Feature flag identifier (not PII)
+    'ab_test_variant'    # A/B test variant (not PII)
+  ]
+  
+  # Block mode (how to handle violations)
+  config.security_baggage_protection_block_mode = :silent  # Options: :silent, :warn, :raise
 end
 ```
 
@@ -3314,14 +3309,12 @@ OpenTelemetry::Baggage.set_value('request_id', SecureRandom.uuid)
 ```ruby
 # config/environments/development.rb
 E11y.configure do |config|
-  config.security.baggage_protection do
-    enabled true
-    
-    # RAISE exception on blocked keys (fail fast)
-    block_mode :raise  # ← Developer sees error immediately
-    
-    allowed_keys E11y::Middleware::BaggageProtection::ALLOWED_KEYS
-  end
+  config.security_baggage_protection_enabled = true
+  
+  # RAISE exception on blocked keys (fail fast)
+  config.security_baggage_protection_block_mode = :raise  # ← Developer sees error immediately
+  
+  config.security_baggage_protection_allowed_keys = E11y::BAGGAGE_PROTECTION_DEFAULT_ALLOWED_KEYS
 end
 
 # Developer tries to set PII in baggage:
@@ -3426,11 +3419,11 @@ Events::UserLogin.track(
 
 **Metadata Flags:**
 - `:pii_filtered` - Event already went through PII filtering (set by PII filter middleware)
-- `:replayed` - Event is being replayed from DLQ (set by DLQ replay service)
+- `:dlq_replayed` - Event is being replayed from DLQ (set by DLQ replay service)
 
 **When to Skip PII Filtering:**
 1. Event has `:pii_filtered => true` (already processed)
-2. Event has `:replayed => true` (replay scenario)
+2. Event has `:dlq_replayed => true` (replay scenario)
 3. Both flags present → skip PII filtering entirely
 
 ### 5.6.3. PiiFilter Middleware with Replay Detection
@@ -3457,7 +3450,7 @@ module E11y
         
         # Check if event contains PII
         unless event_class.contains_pii?
-          # No PII declared, skip filtering (Tier 1)
+          # No PII declared, skip filtering (:no_pii)
           return event_data
         end
         
@@ -3477,7 +3470,7 @@ module E11y
         metadata = event_data[:metadata] || {}
         
         # Check for replay flag
-        return true if metadata[:replayed]
+        return true if metadata[:dlq_replayed]
         
         # Check for already-filtered flag
         return true if metadata[:pii_filtered]
@@ -3507,7 +3500,7 @@ module E11y
         
         # ✅ CRITICAL: Mark as replayed (skip transformations)
         event_data[:metadata] ||= {}
-        event_data[:metadata][:replayed] = true
+        event_data[:metadata][:dlq_replayed] = true
         event_data[:metadata][:pii_filtered] = true  # Already filtered!
         event_data[:metadata][:replayed_at] = Time.now.utc.iso8601
         event_data[:metadata][:original_event_id] = event_data[:event_id]
@@ -3560,7 +3553,7 @@ E11y.configure do |config|
   config.dlq.replay do
     # Metadata flags for replayed events
     set_metadata_flags do
-      replayed true
+      dlq_replayed true
       pii_filtered true
       replay_timestamp true
       original_event_id true
@@ -4022,7 +4015,7 @@ RSpec.describe E11y::RateLimiting do
       end
       
       # Exceed limit
-      E11y.config.rate_limiting.global.limit = 100
+      E11y.config.rate_limiting_global_limit = 100
       
       expect(Events::Test.track(message: 'over limit')).to be_falsey
     end

data/docs/{ADR-007-opentelemetry-integration.md → architecture/ADR-007-opentelemetry-integration.md}

@@ -127,25 +127,7 @@ C4Context
 
 **Configuration:**
 
-```ruby
-# config/initializers/e11y.rb
-E11y.configure do |config|
-  # Option 1: Yabeda only (DEFAULT, recommended for Rails)
-  config.metrics do
-    backend :yabeda  # Prometheus via Yabeda
-  end
-  
-  # Option 2: OpenTelemetry only (for OTLP backends)
-  # config.metrics do
-  #   backend :opentelemetry  # OTLP via OTel SDK
-  # end
-  
-  # Option 3: Both (for migration period ONLY)
-  # config.metrics do
-  #   backend [:yabeda, :opentelemetry]  # ⚠️ DOUBLE OVERHEAD!
-  # end
-end
-```
+Metrics are emitted via Yabeda adapter when events with `metrics do` are tracked. Register Yabeda adapter in `config.adapters`.
 
 **Metrics Adapter Pattern:**
 
@@ -756,7 +738,7 @@ module E11y
         return true if event[:severity].in?([:error, :fatal])
         
         # Check if event matches span creation patterns
-        E11y.config.opentelemetry.span_creation_patterns.any? do |pattern|
+        E11y.config.opentelemetry_span_creation_patterns.any? do |pattern|
           File.fnmatch(pattern, event[:event_name])
         end
       end
@@ -1068,7 +1050,7 @@ module E11y
         return true if event[:severity].in?([:error, :fatal])
         
         # Check configured patterns
-        patterns = E11y.config.opentelemetry.span_creation_patterns || []
+        patterns = E11y.config.opentelemetry_span_creation_patterns || []
         patterns.any? { |pattern| File.fnmatch(pattern, event[:event_name]) }
       end