e11y 0.2.0 → 1.1.0

data/docker-compose.yml

@@ -1,5 +1,3 @@
-version: '3.8'
-
 services:
   loki:
     image: grafana/loki:2.9.0
@@ -69,6 +67,24 @@ services:
     networks:
       - e11y_network
 
+  # OpenTelemetry Collector - receives OTLP, exports to debug + Loki
+  # HTTP: localhost:4318, gRPC: localhost:4317
+  # Usage: docker compose up -d loki otel-collector
+  otel-collector:
+    image: otel/opentelemetry-collector-contrib:0.104.0
+    container_name: e11y_otel_collector
+    ports:
+      - "4317:4317"   # OTLP gRPC
+      - "4318:4318"   # OTLP HTTP
+    volumes:
+      - ./config/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
+    command: ["--config=/etc/otelcol-contrib/config.yaml"]
+    depends_on:
+      loki:
+        condition: service_healthy
+    networks:
+      - e11y_network
+
 networks:
   e11y_network:
     driver: bridge

data/docs/ADAPTERS.md

@@ -0,0 +1,76 @@
+# Adapters
+
+> Back to [README](../README.md#documentation)
+
+E11y supports multiple adapters for different backends.
+
+| Adapter | Purpose | Batching | Use Case |
+|---------|---------|----------|----------|
+| **Loki** | Log aggregation (Grafana) | Yes | Production logs |
+| **Sentry** | Error tracking | Via SDK | Error monitoring |
+| **OpenTelemetry** | OTLP export (OTelLogs, OpenTelemetryCollector) | Varies | Distributed tracing, logs |
+| **Yabeda** | Prometheus metrics | N/A | Metrics export |
+| **File** | Local logs | Yes | Development, CI |
+| **Stdout** | Console output | No | Development |
+| **InMemory** | Test buffer | No | Testing |
+
+## Configuration
+
+```ruby
+# config/initializers/e11y.rb
+E11y.configure do |config|
+  # Configure adapters
+  config.adapters[:logs] = E11y::Adapters::Loki.new(
+    url: ENV["LOKI_URL"],
+    batch_size: 100,
+    batch_timeout: 5,
+    compress: true
+  )
+  
+  config.adapters[:errors_tracker] = E11y::Adapters::Sentry.new(
+    dsn: ENV["SENTRY_DSN"]
+  )
+  
+  config.adapters[:stdout] = E11y::Adapters::Stdout.new(
+    format: :pretty
+  )
+
+  # OpenTelemetry Collector (compress: true default, requires Faraday)
+  # config.adapters[:otel] = E11y::Adapters::OpenTelemetryCollector.new(
+  #   endpoint: ENV["OTEL_EXPORTER_OTLP_ENDPOINT"],
+  #   service_name: "my-app"
+  # )
+end
+```
+
+## Adapter Routing by Severity
+
+Events are routed to adapters based on severity. The default mapping:
+
+- `error`, `fatal` → `[:logs, :errors_tracker]`
+- Other severities → `[:logs]`
+
+Override routing explicitly:
+
+```ruby
+class CustomEvent < E11y::Event::Base
+  adapters :logs, :stdout  # Explicit routing
+end
+```
+
+## Custom Adapters
+
+Implement the `write` method:
+
+```ruby
+class MyBackendAdapter < E11y::Adapters::Base
+  def write(event_data)
+    # event_data contains event_name, payload, severity, timestamp, etc.
+    MyBackend.send_event(event_data)
+  end
+end
+
+E11y.configure do |config|
+  config.adapters[:my_backend] = MyBackendAdapter.new
+end
+```

data/docs/ADAPTIVE_SAMPLING.md

@@ -0,0 +1,59 @@
+# Adaptive Sampling
+
+> Back to [README](../README.md#documentation)
+
+E11y supports adaptive sampling to reduce event volume during high load.
+
+Sampling strategies:
+
+1. **Error-based** - Increase sampling during error spikes
+2. **Load-based** - Reduce sampling under high throughput
+3. **Value-based** - Always sample high-value events
+
+> **Note:** Rate limiting (`E11y::Middleware::RateLimiting`) is **not included in the default
+> pipeline**. To enable it, add it manually:
+>
+> ```ruby
+> config.pipeline.use E11y::Middleware::RateLimiting
+> ```
+> Enabling `config.rate_limiting_enabled = true` alone has no effect without this step.
+
+## Configuration
+
+```ruby
+E11y.configure do |config|
+  config.pipeline.use E11y::Middleware::Sampling,
+    default_sample_rate: 0.1,
+    
+    # Error-based sampling
+    error_based_adaptive: true,
+    error_spike_config: {
+      window: 60,
+      absolute_threshold: 100,
+      relative_threshold: 3.0,
+      spike_duration: 300
+    },
+    
+    # Load-based sampling
+    load_based_adaptive: true,
+    load_monitor_config: {
+      window: 60,
+      thresholds: {
+        normal: 1_000,
+        high: 10_000,
+        very_high: 50_000,
+        overload: 100_000
+      }
+    }
+end
+```
+
+## Value-Based Sampling
+
+Sample events based on payload values:
+
+```ruby
+class PaymentEvent < E11y::Event::Base
+  sample_by_value :amount, greater_than: 1000  # Always sample large payments
+end
+```

data/docs/COMPARISON.md

@@ -0,0 +1,104 @@
+# E11y vs. Alternatives — Detailed Comparison
+
+> For a quick overview, see the [comparison table in README](../README.md#what-makes-e11y-different).
+
+### Detailed Comparisons
+
+#### vs. SaaS APM (Datadog, New Relic, Dynatrace)
+
+**Datadog / New Relic:**
+- ✅ **Pros:** Full-stack visibility, mature dashboards, auto-instrumentation
+- ❌ **Cons:** $500-5k/month, vendor lock-in, no debug buffering, no schema validation
+- **E11y advantage:** 10x cheaper, request-scoped buffering (unique), type-safe events, own your data
+
+**When to use Datadog/New Relic instead:**
+- You need frontend RUM (Real User Monitoring)
+- You have polyglot microservices (not just Rails)
+- Budget is unlimited, prefer turnkey solution
+
+---
+
+#### vs. Open-Source Logging (Semantic Logger, Lograge)
+
+**Semantic Logger:**
+- ✅ **Pros:** Structured logs (JSON), async writes, Rails integration
+- ❌ **Cons:** No debug buffering, no schema validation, no auto-metrics, logs-only
+- **E11y advantage:** Request-scoped buffering (unique), schema validation, auto-metrics, unified events
+
+**Lograge:**
+- ✅ **Pros:** Reduces Rails log noise (single-line requests)
+- ❌ **Cons:** Filtering only, no buffering, no validation, no metrics
+- **E11y advantage:** Request-scoped buffering (selective, not filtering), schema validation, auto-metrics
+
+**When to use Semantic Logger instead:**
+- You only need structured JSON logs (no events/metrics)
+- You don't need debug buffering or schema validation
+
+---
+
+#### vs. OpenTelemetry
+
+**OpenTelemetry:**
+- ✅ **Pros:** Industry standard, polyglot, vendor-neutral, mature ecosystem
+- ❌ **Cons:** Complex setup (1-2 weeks), no debug buffering, no schema validation, overkill for Rails monolith
+- **E11y advantage:** Fast setup, Rails-first, request-scoped buffering, schema validation
+
+**When to use OpenTelemetry instead:**
+- You have microservices in multiple languages (Go, Java, Python, etc.)
+- You need distributed tracing across services
+- You have a platform team to manage complexity
+
+**Use both:** E11y events can be sent to OpenTelemetry via `E11y::Adapters::OtelLogs`
+
+---
+
+#### vs. Grafana + Loki + Prometheus
+
+**Grafana Stack:**
+- ✅ **Pros:** Open-source, powerful visualizations, mature, self-hosted
+- ❌ **Cons:** Complex setup (2-3 days), requires DevOps, no Rails integration, no schema validation
+- **E11y advantage:** Fast setup, Rails-native, schema validation, no DevOps required
+
+**When to use Grafana Stack instead:**
+- You already have Grafana/Loki infrastructure
+- You have a dedicated DevOps team
+- You need custom dashboards across multiple systems
+
+**Use both:** E11y can send events to Loki via `E11y::Adapters::Loki`
+
+---
+
+#### vs. Error Tracking (Sentry, Honeybadger, Rollbar)
+
+**Sentry:**
+- ✅ **Pros:** Excellent error tracking, stack traces, breadcrumbs, release tracking
+- ❌ **Cons:** Errors-only, no debug buffering, no schema validation, $26-80/mo
+- **E11y advantage:** Events + errors + metrics unified, request-scoped buffering, schema validation
+
+**When to use Sentry instead:**
+- You only need error tracking (not general observability)
+- You need frontend JavaScript error tracking
+
+**Use both:** E11y can send error events to Sentry via `E11y::Adapters::Sentry`
+
+---
+
+#### vs. Rails-First APM (AppSignal, Skylight)
+
+**AppSignal:**
+- ✅ **Pros:** Rails-native, beautiful UI, performance monitoring, $23/mo entry
+- ❌ **Cons:** SaaS lock-in, no debug buffering, no schema validation, limited to supported languages
+- **E11y advantage:** Request-scoped buffering (unique), schema validation, own your data
+
+**Skylight:**
+- ✅ **Pros:** Rails performance profiling, SQL query analysis
+- ❌ **Cons:** Performance-only (no logs/events), SaaS lock-in, $20+/mo
+- **E11y advantage:** Unified events/logs/metrics, request-scoped buffering, own your data
+
+**When to use AppSignal/Skylight instead:**
+- You want zero-config turnkey solution
+- You prefer paying for hosted service over self-hosting
+
+**Use both:** E11y for events/logs/metrics, AppSignal for performance profiling
+
+---

data/docs/CONFIGURATION.md

@@ -0,0 +1,52 @@
+# Configuration
+
+> Back to [README](../README.md#documentation)
+
+## Basic Configuration
+
+```ruby
+# config/initializers/e11y.rb
+E11y.configure do |config|
+  # Service identification
+  config.service_name = "myapp"
+  config.environment = Rails.env
+  
+  # Configure adapters
+  config.adapters[:logs] = E11y::Adapters::Loki.new(
+    url: ENV["LOKI_URL"],
+    batch_size: 100,
+    batch_timeout: 5,
+    compress: true
+  )
+  
+  config.adapters[:errors_tracker] = E11y::Adapters::Sentry.new(
+    dsn: ENV["SENTRY_DSN"]
+  )
+  
+  # Default retention period
+  config.default_retention_period = 30.days
+end
+```
+
+## Middleware Pipeline
+
+Configure middleware for sampling, PII filtering, and more. Add `TrackLatency` first for self-monitoring (Event.track() latency):
+
+```ruby
+E11y.configure do |config|
+  # Self-monitoring: track latency (must be first)
+  config.pipeline.use E11y::Middleware::TrackLatency
+
+  # Sampling middleware
+  config.pipeline.use E11y::Middleware::Sampling,
+    default_sample_rate: 0.1,
+    error_based_adaptive: true,
+    load_based_adaptive: true
+  
+  # PII filtering middleware
+  config.pipeline.use E11y::Middleware::PIIFilter
+  
+  # Trace context middleware
+  config.pipeline.use E11y::Middleware::TraceContext
+end
+```

data/docs/DISTRIBUTED_TRACING.md

@@ -0,0 +1,44 @@
+# Distributed Tracing
+
+> Back to [README](../README.md#documentation)
+
+E11y automatically attaches W3C Trace Context headers to incoming requests via the `TraceContext` middleware and propagates trace/span IDs through the event pipeline.
+
+## Incoming Trace Context
+
+Incoming `traceparent` / `tracestate` headers are extracted automatically:
+
+```ruby
+E11y.configure do |config|
+  config.pipeline.use E11y::Middleware::TraceContext
+end
+```
+
+Events tracked during a request will include `trace_id` and `span_id` from the incoming context.
+
+## Outgoing HTTP Trace Propagation (Manual — v1.0)
+
+> **Note:** Automatic outgoing trace context injection (Faraday / Net::HTTP middleware) is planned for v1.1.
+> Until then, use the helper below to propagate W3C Trace Context manually:
+
+```ruby
+# Helper: build W3C traceparent header from current context
+def traceparent_header
+  return {} unless E11y::Current.trace_id
+
+  span_id = E11y::Current.span_id || SecureRandom.hex(8)
+  { "traceparent" => "00-#{E11y::Current.trace_id}-#{span_id}-01" }
+end
+
+# Faraday — inject on each connection
+conn = Faraday.new(url: "https://api.example.com") do |f|
+  f.headers.merge!(traceparent_header)
+end
+
+# Net::HTTP — inject per request
+request = Net::HTTP::Post.new("/events")
+traceparent_header.each { |k, v| request[k] = v }
+http.request(request)
+```
+
+This ensures downstream services receive a valid `traceparent` header and can correlate logs/traces back to the originating request.

data/docs/LIMITATIONS.md

@@ -0,0 +1,13 @@
+# Limitations & Tradeoffs
+
+> Back to [README](../README.md#documentation)
+
+E11y trades generality for Rails-specific ergonomics. Know what you're getting:
+
+| Limitation | Detail |
+|------------|--------|
+| **Rails only** | No Sinatra, Hanami, or pure-Ruby support. Railtie is required. |
+| **Ruby 3.2+** | Older projects can't use it without upgrading Ruby. |
+| **Rails 7.0–8.0** | Rails 8.1 excluded (sqlite3 bug in test environment). |
+| **Memory overhead** | Debug buffer holds events in RAM per request. Under heavy load with large payloads, monitor heap usage. |
+| **No distributed tracing UI** | OTel adapter emits spans, but e11y has no built-in trace visualization. Use Grafana Tempo or Jaeger. |

data/docs/METRICS_DSL.md

@@ -0,0 +1,84 @@
+# Metrics DSL
+
+> Back to [README](../README.md#documentation)
+
+Define Prometheus metrics alongside events.
+
+## Basic Example
+
+```ruby
+class OrderPaidEvent < E11y::Event::Base
+  schema do
+    required(:order_id).filled(:string)
+    required(:amount).filled(:float)
+    required(:currency).filled(:string)
+  end
+  
+  metrics do
+    # Counter: Track number of paid orders
+    counter :orders_total, tags: [:currency]
+    
+    # Histogram: Track order amount distribution
+    histogram :order_amount,
+              value: :amount,
+              tags: [:currency],
+              buckets: [10, 50, 100, 500, 1000]
+    
+    # Gauge: Track active orders
+    gauge :active_orders, value: :active_count
+  end
+end
+
+# One track() call = event + metrics
+OrderPaidEvent.track(order_id: "123", amount: 99.99, currency: "USD")
+# => orders_total{currency="USD"} +1
+# => order_amount{currency="USD"} observe 99.99
+```
+
+## Metric Types
+
+**Counter** - Monotonically increasing value:
+
+```ruby
+metrics do
+  counter :orders_total, tags: [:currency, :status]
+end
+# => orders_total{currency="USD", status="paid"} 42
+```
+
+**Histogram** - Distribution of values:
+
+```ruby
+metrics do
+  histogram :order_amount,
+            value: :amount,
+            tags: [:currency],
+            buckets: [10, 50, 100, 500, 1000]
+end
+# => order_amount_bucket{currency="USD", le="100"} 15
+```
+
+**Gauge** - Arbitrary value that can go up or down:
+
+```ruby
+metrics do
+  gauge :queue_depth, value: :size, tags: [:queue_name]
+end
+# => queue_depth{queue_name="emails"} 37
+```
+
+## How It Works
+
+1. Define metrics in event class
+2. Metrics registered in `E11y::Metrics::Registry` at boot time
+3. When `track()` is called, metrics are automatically updated **if the Yabeda adapter is configured and routed to**
+4. Metrics exported via Yabeda adapter (Prometheus format)
+
+> **Note:** The `metrics do` DSL only registers metric definitions. Metrics are actually updated
+> when an event is written to the `E11y::Adapters::Yabeda` adapter. If you omit the Yabeda adapter
+> from your configuration, `track()` will send events to Loki/Sentry but metric counters will not
+> be incremented. Make sure to add:
+>
+> ```ruby
+> config.adapters[:metrics] = E11y::Adapters::Yabeda.new
+> ```

data/docs/PERFORMANCE.md

@@ -0,0 +1,60 @@
+# Performance
+
+> Back to [README](../README.md#documentation)
+
+## Design Principles
+
+E11y is designed for performance:
+
+- **Hash-based events** - Events are Hashes, not objects, minimizing allocations
+- **Configurable validation** - Choose validation mode based on performance needs
+- **Batching** - Loki and other adapters support batching to reduce network overhead
+- **Sampling** - Adaptive sampling reduces event volume under high load
+
+## Benchmarks (Ruby 3.3, measured via `rake spec:benchmark` and `rake spec:memory`)
+
+| Metric | Value | Notes |
+|--------|-------|-------|
+| Event tracking latency (p99, `:always`) | <70µs | Full dry-schema validation per event |
+| Event tracking latency (p99, `:sampled` 1%) | <10µs | Schema runs ~1% of events |
+| Event tracking latency (p99, `:never`) | <50µs | Pipeline overhead, no validation |
+| Memory allocations (`:always` mode) | ~47 objects/event | Baseline; threshold ≤72 |
+| Memory allocations (`:never` mode) | ~33 objects/event | Baseline; threshold ≤50 |
+| Memory retained after 10K events | 0 objects | No leaks detected |
+| Memory consumption (1K events) | <100 MB allocated | Small-scale benchmark target |
+
+## Validation Mode Trade-offs
+
+```ruby
+# Fastest — skip schema checks in production hot paths
+validation_mode :never    # ~33 allocs/event, <50µs p99
+
+# Balanced — validate 1% of traffic for regression detection
+validation_mode :sampled, sample_rate: 0.01  # <10µs p99
+
+# Safest — validate every event (default, recommended for dev/staging)
+validation_mode :always   # ~47 allocs/event, <70µs p99
+```
+
+## Running Benchmarks
+
+```bash
+rake spec:benchmark   # latency benchmarks (~44 examples)
+rake spec:memory      # allocation and leak checks
+```
+
+See `spec/e11y/event/base_benchmark_spec.rb` and `spec/e11y/memory_spec.rb` for the full test suite.
+
+## Cardinality Protection
+
+Optional cardinality protection prevents high-cardinality labels from overwhelming metrics systems:
+
+```ruby
+E11y::Adapters::Loki.new(
+  url: "http://loki:3100",
+  enable_cardinality_protection: true,  # default
+  max_label_cardinality: 1000          # ~1000 event types
+)
+```
+
+Labels = `event_name` + `severity` only. Payload (user_uuid, etc.) stays in log line — filter via LogQL: `| json | user_uuid="xxx"`.

data/docs/PII_FILTERING.md

@@ -0,0 +1,40 @@
+# PII Filtering
+
+> Back to [README](../README.md#documentation)
+
+E11y provides PII filtering capabilities for sensitive data.
+
+## Rails Integration
+
+E11y can respect `Rails.application.config.filter_parameters` when configured:
+
+```ruby
+# config/application.rb
+config.filter_parameters += [:password, :email, :ssn, :credit_card]
+
+# E11y will filter these fields when PII filtering middleware is enabled
+```
+
+## Explicit PII Strategies
+
+Configure PII filtering per event:
+
+```ruby
+class PaymentEvent < E11y::Event::Base
+  contains_pii true
+  
+  pii_filtering do
+    masks :card_number     # Replace with "[FILTERED]"
+    hashes :user_email     # SHA256 hash (searchable)
+    allows :amount         # No filtering
+  end
+end
+```
+
+Available strategies:
+
+- `masks` - Replace with "[FILTERED]"
+- `hashes` - SHA256 hash (preserves searchability)
+- `partials` - Show first/last characters
+- `redacts` - Remove completely
+- `allows` - No filtering

data/docs/PRESETS.md

@@ -0,0 +1,65 @@
+# Presets
+
+> Back to [README](../README.md#documentation)
+
+E11y provides presets for common event types.
+
+## HighValueEvent
+
+For financial transactions and critical business events:
+
+```ruby
+class PaymentProcessedEvent < E11y::Event::Base
+  include E11y::Presets::HighValueEvent
+  
+  schema do
+    required(:transaction_id).filled(:string)
+    required(:amount).filled(:decimal)
+  end
+end
+
+# Configured with:
+# - severity: :success
+# - sample_rate: 1.0 (always sampled)
+# - adapters: [:logs, :errors_tracker]
+# - rate_limit: unlimited
+```
+
+## AuditEvent
+
+For compliance and audit trails:
+
+```ruby
+class UserDeletedEvent < E11y::Event::Base
+  include E11y::Presets::AuditEvent
+  
+  schema do
+    required(:user_id).filled(:string)
+    required(:deleted_by).filled(:string)
+  end
+end
+
+# Configured with:
+# - sample_rate: 1.0 (never sampled)
+# - rate_limit: unlimited
+# Note: Set severity based on event criticality
+```
+
+## DebugEvent
+
+For development and troubleshooting:
+
+```ruby
+class SlowQueryEvent < E11y::Event::Base
+  include E11y::Presets::DebugEvent
+  
+  schema do
+    required(:query).filled(:string)
+    required(:duration_ms).filled(:integer)
+  end
+end
+
+# Configured with:
+# - severity: :debug
+# - adapters: [:logs]
+```