e11y 0.2.0 → 1.1.0

data/docs/RAILS_INTEGRATION.md

@@ -0,0 +1,79 @@
+# Rails Integration
+
+> Back to [README](../README.md#documentation)
+
+E11y integrates with Rails via `E11y::Railtie` (`lib/e11y/railtie.rb`). After `bundle install`, requiring the gem in a Rails app loads the Railtie automatically.
+
+## Request middleware
+
+`E11y::Middleware::Request` is inserted into the Rack stack (before `Rails::Rack::Logger` when present). It sets trace and request context on `E11y::Current`, optionally starts the **ephemeral (request-scoped) buffer** for debug events, and adds `X-E11y-Trace-Id` / `X-E11y-Span-Id` response headers.
+
+## Rails instrumentation (`ActiveSupport::Notifications`)
+
+When **`config.rails_instrumentation_enabled = true`**, E11y subscribes to Rails instrumentation and maps notifications to typed events (see `lib/e11y/instruments/rails_instrumentation.rb`).
+
+| Area | Event classes (under `E11y::Events::Rails::`) |
+|------|-----------------------------------------------|
+| HTTP | `Http::Request`, `Http::StartProcessing`, `Http::Redirect`, `Http::SendFile` |
+| Database | `Database::Query` |
+| Active Job (notification names) | `Job::Enqueued`, `Job::Scheduled`, `Job::Started`, `Job::Completed`, `Job::Failed` |
+| Cache | `Cache::Read`, `Cache::Write`, `Cache::Delete` |
+| Views | `View::Render` |
+
+This is **independent** of the Sidekiq and Active Job toggles below: instrumentation listens to Rails; the job toggles add **extra** process integration (buffer lifecycle, middleware, callbacks).
+
+## Sidekiq
+
+Enable **only if** you use Sidekiq:
+
+```ruby
+E11y.configure do |config|
+  config.sidekiq_enabled = true
+end
+```
+
+The Railtie registers client and server middleware (`E11y::Instruments::Sidekiq`) so jobs participate in the same **ephemeral buffer** semantics as HTTP requests when `config.ephemeral_buffer_enabled` is true.
+
+On enqueue, **`E11y::Current.user_id`** (when set, e.g. from request middleware) is merged into **`e11y_baggage`** together with any allowed `Current.baggage` keys. The worker restores **`E11y::Current.baggage`** and **`E11y::Current.user_id`** from that hash. Key **`user_id`** is in the default baggage allowlist (`E11y::BAGGAGE_PROTECTION_DEFAULT_ALLOWED_KEYS`).
+
+## Active Job
+
+Enable when you want callbacks and buffer handling on **`ActiveJob::Base`** (and **`ApplicationJob`** when that constant is already defined at hook time):
+
+```ruby
+E11y.configure do |config|
+  config.active_job_enabled = true
+end
+```
+
+You can use **both** `rails_instrumentation_enabled` and `active_job_enabled`; they complement each other. If you only enqueue via Sidekiq without Active Job, you may rely on `sidekiq_enabled` alone.
+
+The **`before_enqueue`** callback applies the same **`e11y_baggage`** merge as Sidekiq (including **`user_id`** from `E11y::Current`).
+
+## Rails.logger bridge
+
+Optional wrapper that still delegates to the original logger but also emits **`E11y::Events::Rails::Log::*`** events (`lib/e11y/events/rails/log.rb`):
+
+```ruby
+E11y.configure do |config|
+  config.logger_bridge_enabled = true
+  # Optional: only these severities (Symbol or String); nil / empty = all
+  config.logger_bridge_track_severities = %i[warn error fatal]
+  # Optional: skip noisy lines (String substrings or Regexp)
+  config.logger_bridge_ignore_patterns = [%r{health}]
+end
+```
+
+Filtering uses **`logger_bridge_track_severities`** and **`logger_bridge_ignore_patterns`** only.
+
+## Configuration reference
+
+| Flag | Purpose |
+|------|---------|
+| `rails_instrumentation_enabled` | Map `ActiveSupport::Notifications` to E11y events |
+| `sidekiq_enabled` | Sidekiq client/server middleware |
+| `active_job_enabled` | `ActiveJob::Base` / `ApplicationJob` callbacks |
+| `logger_bridge_enabled` | Wrap `Rails.logger` with `E11y::Logger::Bridge` |
+| `ephemeral_buffer_enabled` | Request/job-scoped debug buffer (see README) |
+
+Further detail: [ADR-008: Rails integration](architecture/ADR-008-rails-integration.md), [Quick Start](QUICK-START.md).

data/docs/SCHEMA_VALIDATION.md

@@ -0,0 +1,63 @@
+# Schema Validation
+
+> Back to [README](../README.md#documentation)
+
+E11y validates event data using [dry-schema](https://dry-rb.org/gems/dry-schema/).
+
+## Basic Example
+
+```ruby
+class OrderCreatedEvent < E11y::Event::Base
+  schema do
+    required(:order_id).filled(:string)
+    required(:total).filled(:float, gt?: 0)
+    required(:currency).filled(:string, included_in?: %w[USD EUR GBP])
+    optional(:coupon_code).maybe(:string)
+  end
+end
+
+# Valid event
+OrderCreatedEvent.track(order_id: "123", total: 99.99, currency: "USD")
+
+# Invalid event raises E11y::ValidationError
+OrderCreatedEvent.track(order_id: nil, total: -10, currency: "INVALID")
+# => ValidationError: order_id is missing, total must be > 0
+```
+
+## Validation Modes
+
+For high-frequency events, you can configure validation behavior:
+
+```ruby
+class HighFrequencyEvent < E11y::Event::Base
+  # Always validate (default)
+  validation_mode :always
+
+  # Sampled validation (validate 1% of events)
+  validation_mode :sampled, sample_rate: 0.01
+
+  # Never validate (use with caution)
+  validation_mode :never
+end
+```
+
+Use `:always` for user input and critical events. Use `:sampled` for high-frequency internal events. Use `:never` only for trusted, typed input.
+
+## Validation Behavior
+
+By default, invalid events raise `E11y::ValidationError`:
+
+```ruby
+OrderEvent.track(order_id: nil)
+# => E11y::ValidationError
+```
+
+To handle validation errors gracefully:
+
+```ruby
+begin
+  OrderEvent.track(order_id: nil)
+rescue E11y::ValidationError => e
+  Rails.logger.warn "Invalid event: #{e.message}"
+end
+```

data/docs/SLO-PROMQL-ALERTS.md

@@ -0,0 +1,161 @@
+# SLO: PromQL Queries and Alert Rules
+
+> Back to [README](../README.md#documentation)
+
+E11y emits SLO metrics to Prometheus via Yabeda. Use these PromQL queries and alert rules in Grafana dashboards and Prometheus.
+
+**Metrics emitted:**
+- `slo_http_requests_total{controller, action, status}` — HTTP request count
+- `slo_http_request_duration_seconds` — HTTP latency histogram
+- `slo_background_jobs_total{job_class, status, queue}` — Job count
+- `slo_event_result_total{slo_name, slo_status}` — Event-driven SLO (EventSlo middleware)
+- `e11y_track_duration_seconds` — E11y pipeline latency (TrackLatency middleware)
+- `e11y_events_tracked_total{result, event_name}` — E11y delivery success/drop
+
+---
+
+## HTTP Availability SLO
+
+**Success rate (30d window):**
+```promql
+sum(rate(slo_http_requests_total{status=~"2..|3.."}[30d])) by (controller, action)
+/
+sum(rate(slo_http_requests_total[30d])) by (controller, action)
+```
+
+**Error rate (5m, for alerts):**
+```promql
+sum(rate(slo_http_requests_total{status=~"4..|5.."}[5m])) by (controller, action)
+/
+sum(rate(slo_http_requests_total[5m])) by (controller, action)
+```
+
+**Per-endpoint availability (99.9% target):**
+```promql
+# Replace OrdersController, create with your controller#action
+sum(rate(slo_http_requests_total{controller="OrdersController",action="create",status=~"2..|3.."}[30d]))
+/
+sum(rate(slo_http_requests_total{controller="OrdersController",action="create"}[30d]))
+```
+
+---
+
+## HTTP Latency SLO
+
+**p99 latency (30d):**
+```promql
+histogram_quantile(0.99,
+  sum(rate(slo_http_request_duration_seconds_bucket[30d])) by (le, controller, action)
+)
+```
+
+**p99 > 500ms alert:**
+```promql
+histogram_quantile(0.99,
+  sum(rate(slo_http_request_duration_seconds_bucket[5m])) by (le, controller, action)
+) > 0.5
+```
+
+---
+
+## Event-Driven SLO (EventSlo)
+
+**Success rate by slo_name (30d):**
+```promql
+sum(rate(slo_event_result_total{slo_status="success"}[30d])) by (slo_name)
+/
+sum(rate(slo_event_result_total[30d])) by (slo_name)
+```
+
+**Example — payment success rate:**
+```promql
+sum(rate(slo_event_result_total{slo_name="payment_success_rate",slo_status="success"}[30d]))
+/
+sum(rate(slo_event_result_total{slo_name="payment_success_rate"}[30d]))
+```
+
+---
+
+## E11y Self-Monitoring
+
+**E11y pipeline latency p99 (<1ms target):**
+```promql
+histogram_quantile(0.99,
+  sum(rate(e11y_track_duration_seconds_bucket[30d])) by (le)
+)
+```
+
+**E11y delivery success rate (99.9% target):**
+```promql
+sum(rate(e11y_events_tracked_total{result="success"}[30d]))
+/
+sum(rate(e11y_events_tracked_total[30d]))
+```
+
+---
+
+## Prometheus Alert Rules
+
+Save as `prometheus/alerts/e11y_slo.yml`:
+
+```yaml
+groups:
+  - name: e11y_slo_http
+    rules:
+      - alert: SLOHttpAvailabilityLow
+        expr: |
+          sum(rate(slo_http_requests_total{status=~"4..|5.."}[5m])) by (controller, action)
+          /
+          sum(rate(slo_http_requests_total[5m])) by (controller, action)
+          > 0.01
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "HTTP availability below 99%"
+          description: "Error rate > 1% for 5 minutes"
+
+      - alert: SLOHttpLatencyHigh
+        expr: |
+          histogram_quantile(0.99,
+            sum(rate(slo_http_request_duration_seconds_bucket[5m])) by (le, controller, action)
+          ) > 0.5
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "HTTP p99 latency > 500ms"
+
+  - name: e11y_self_monitoring
+    rules:
+      - alert: E11yTrackLatencyHigh
+        expr: |
+          histogram_quantile(0.99,
+            sum(rate(e11y_track_duration_seconds_bucket[5m])) by (le)
+          ) > 0.001
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "E11y track() p99 > 1ms"
+
+      - alert: E11yDeliveryRateLow
+        expr: |
+          sum(rate(e11y_events_tracked_total{result="success"}[1h]))
+          /
+          sum(rate(e11y_events_tracked_total[1h]))
+          < 0.999
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "E11y delivery rate below 99.9%"
+```
+
+---
+
+## Grafana Dashboard
+
+Use `rake e11y:slo:dashboard` to generate a dashboard from `slo.yml`, or add panels manually with the PromQL above.
+
+**Metric name prefix:** Yabeda exports with `yabeda_` prefix. If queries return no data, try `yabeda_e11y_slo_http_requests_total` or check your Prometheus scrape config.

data/docs/TESTING.md

@@ -0,0 +1,69 @@
+# Testing
+
+> Back to [README](../README.md#documentation)
+
+Use the **InMemoryTest** adapter for testing. It extends `InMemory` and overrides `last_event` to skip Rails auto-instrumentation events (`E11y::Events::Rails::*`), so your business events aren't obscured by request lifecycle events.
+
+## Setup
+
+```ruby
+# spec/rails_helper.rb or spec/spec_helper.rb
+RSpec.configure do |config|
+  config.before(:each) do
+    E11y.configure do |e11y_config|
+      e11y_config.adapters[:test] = E11y::Adapters::InMemoryTest.new
+    end
+  end
+
+  config.after(:each) do
+    E11y.configuration.adapters[:test]&.clear!
+  end
+end
+```
+
+## Test Events
+
+```ruby
+RSpec.describe OrdersController do
+  let(:test_adapter) { E11y.configuration.adapters[:test] }
+  
+  it "tracks order creation" do
+    post :create, params: { item: "Book", price: 29.99 }
+    
+    events = test_adapter.events
+    expect(events).to include(
+      a_hash_including(
+        event_name: "OrderCreatedEvent",
+        payload: hash_including(item: "Book", price: 29.99)
+      )
+    )
+  end
+  
+  it "does not track payment for free orders" do
+    post :create, params: { item: "Free Book", price: 0 }
+    
+    payment_events = test_adapter.events.select { |e| e[:event_name] == "PaymentProcessedEvent" }
+    expect(payment_events).to be_empty
+  end
+end
+```
+
+## InMemoryTest Adapter API
+
+```ruby
+test_adapter = E11y::Adapters::InMemoryTest.new
+
+# Get all events
+test_adapter.events  # => Array<Hash>
+
+# Count events
+test_adapter.event_count  # => Integer
+
+# Find last event (skips Rails instrumentation events)
+test_adapter.last_event  # => Hash
+
+# Clear all events
+test_adapter.clear!
+```
+
+> **Note:** Use `InMemoryTest` in test suites; use `InMemory` in production configs (e.g. benchmarks).

data/docs/{ADR-001-architecture.md → architecture/ADR-001-architecture.md}

@@ -66,12 +66,12 @@ Modern Rails applications need:
 - ✅ Zero-allocation event tracking (class methods only)
 - ✅ <1ms p99 latency @ 1000 events/sec
 - ✅ <100MB memory footprint
-- ✅ Rails 8.0+ exclusive
+- ✅ Rails 7.0+ (7.x, 8.x)
 - ✅ Open-source extensibility
 
 **Non-Goals:**
 - ❌ Plain Ruby support (Rails only)
-- ❌ Backwards compatibility with Rails 7.x
+- ❌ Rails 6.x and earlier
 - ❌ Hot configuration reload
 - ❌ Distributed tracing coordination (only propagation)
 
@@ -452,9 +452,9 @@ graph TB
     end
     
     subgraph "Thread-Local Storage"
-        TL1[Current.trace_id<br/>Current.request_buffer]
-        TL2[Current.trace_id<br/>Current.request_buffer]
-        TL3[Current.trace_id<br/>Current.request_buffer]
+        TL1[Current.trace_id<br/>EphemeralBuffer<br/>Thread.current]
+        TL2[Current.trace_id<br/>EphemeralBuffer<br/>Thread.current]
+        TL3[Current.trace_id<br/>EphemeralBuffer<br/>Thread.current]
     end
     
     subgraph "Shared Resources"
@@ -805,7 +805,7 @@ class RoutingMiddleware < E11y::Middleware
     
     if severity == :debug
       # Route to request-scoped buffer
-      RequestBuffer.add(event_data)
+      EphemeralBuffer.add_event(event_data)
     else
       # Route to main buffer
       MainBuffer.add(event_data)
@@ -1239,74 +1239,46 @@ end
 
 ### 3.4. Request-Scoped Buffer
 
-**Design Decision:** Thread-local storage using ActiveSupport::CurrentAttributes.
+**Design Decision:** Thread-local storage via `EphemeralBuffer` with `Thread.current[:e11y_ephemeral_buffer]`. Context (trace_id, request_id) in `E11y::Current` (ActiveSupport::CurrentAttributes); buffer is separate.
 
 ```ruby
 module E11y
   class Current < ActiveSupport::CurrentAttributes
-    # Thread-local attributes
+    # Thread-local context attributes (no buffer here)
     attribute :trace_id
     attribute :user_id
     attribute :request_id
-    attribute :request_buffer  # Debug events buffer
     attribute :sampled  # Sampling decision
-    
-    def request_buffer
-      attributes[:request_buffer] ||= []
-    end
-    
-    def add_debug_event(event_data)
-      request_buffer << event_data if request_buffer.size < Config.request_buffer_limit
-    end
-    
-    def flush_debug_events
-      events = request_buffer.dup
-      request_buffer.clear
-      events
-    end
   end
-  
-  # Request-scoped buffer manager
-  class RequestBuffer
-    class << self
-      def add(event_data)
-        Current.add_debug_event(event_data)
+
+  module Buffers
+    class EphemeralBuffer
+      THREAD_KEY_BUFFER = :e11y_ephemeral_buffer
+
+      def self.initialize!(request_id: nil, buffer_limit: nil)
+        Thread.current[THREAD_KEY_BUFFER] = []
       end
-      
-      def flush
-        Current.flush_debug_events
+
+      def self.add_event(event_data)
+        buf = Thread.current[THREAD_KEY_BUFFER]
+        return false unless buf
+        buf << event_data if buf.size < (buffer_limit || Config.buffer (job_buffer_limit))
       end
-      
-      def setup_rails_integration
-        # Hook into Rails request cycle
-        ActiveSupport::Notifications.subscribe('process_action.action_controller') do |*args|
-          event = ActiveSupport::Notifications::Event.new(*args)
-          
-          # Flush on error
-          if event.payload[:exception]
-            flush_to_adapters
-          else
-            # Discard on success
-            flush
-          end
-        end
+
+      def self.flush_on_error
+        # Flush buffered events to adapters
       end
-      
-      private
-      
-      def flush_to_adapters
-        events = flush
-        
-        # Send debug events to adapters
-        events.each do |event_data|
-          MainBuffer.add(event_data)
-        end
+
+      def self.discard
+        Thread.current[THREAD_KEY_BUFFER] = nil
       end
     end
   end
 end
 ```
 
+Rails integration: `Middleware::Request` and Sidekiq/ActiveJob instruments call `EphemeralBuffer.initialize!`, `flush_on_error`, `discard` at request/job boundaries.
+
 ---
 
 ### 3.5. Adapter Base Class
@@ -1608,7 +1580,7 @@ Pipeline.process(event_data)
    └─ next
   ↓
 7. RoutingMiddleware
-   ├─ severity == :debug? → RequestBuffer
+   ├─ severity == :debug? → EphemeralBuffer
    └─ severity == :info+? → MainBuffer
   ↓
 Buffer → Adapters (receive normalized event_name)
@@ -1636,6 +1608,9 @@ Buffer → Adapters (receive normalized event_name)
 
 ## 5. Memory Optimization Strategy
 
+> **📖 For full design, implementation details, and trade-offs, see:**
+> **[ADR-018: Memory Optimization](ADR-018-memory-optimization.md)**
+
 ### 5.1. Zero-Allocation Pattern
 
 **Key Principle:** No object instances, only hashes.
@@ -1754,8 +1729,8 @@ end
 **Components:**
 
 1. **Thread-local (no sync needed):**
-   - Request-scoped buffer (Current.request_buffer)
-   - Context (Current.trace_id, etc.)
+   - Request-scoped buffer (EphemeralBuffer + Thread.current[:e11y_ephemeral_buffer])
+   - Context (Current.trace_id, request_id, etc.)
 
 2. **Concurrent (thread-safe):**
    - Main ring buffer (Concurrent::AtomicFixnum)
@@ -2037,7 +2012,7 @@ Gem::Specification.new do |spec|
   spec.required_ruby_version = '>= 3.3.0'
   
   # Required
-  spec.add_dependency 'rails', '>= 8.0.0'
+  spec.add_dependency 'rails', '>= 7.0'
   spec.add_dependency 'dry-schema', '~> 1.13'
   spec.add_dependency 'dry-configurable', '~> 1.1'
   spec.add_dependency 'concurrent-ruby', '~> 1.2'
@@ -2376,7 +2351,7 @@ end
 **Middleware checks opt-out flag before processing:**
 
 ```ruby
-# E11y::Middleware::PIIFiltering
+# E11y::Middleware::PIIFilter
 def call(event_data)
   event_class = event_data[:event_class]
   
@@ -2602,10 +2577,6 @@ end
 - **[ADR-006: Security & Compliance](ADR-006-security-compliance.md)** - PII filtering, rate limiting
 - **[ADR-011: Testing Strategy](ADR-011-testing-strategy.md)** - Testing approach
 
-**Configuration:**
-- **[COMPREHENSIVE-CONFIGURATION.md](COMPREHENSIVE-CONFIGURATION.md)** - Complete configuration examples
-- **[CONFLICT-ANALYSIS.md](CONFLICT-ANALYSIS.md)** - Feature conflict resolutions
-
 **Use Cases:**
 - **[docs/use_cases/](use_cases/)** - All 22 use cases documented