e11y 0.1.0

data/docs/use_cases/backlog.md

@@ -0,0 +1,226 @@
+# Backlog (Future Enhancements)
+
+**Status:** Draft  
+**Priority:** Low (v1.1+)  
+**Category:** Future Ideas
+
+---
+
+## Overview
+
+This document captures promising ideas for future versions of `e11y` that are not planned for v1.0 but may provide significant value in subsequent releases.
+
+---
+
+## 1. Quick Start Presets
+
+### Problem
+
+Configuration complexity can be overwhelming for new users. Setting up production-ready configuration requires understanding multiple subsystems (sampling, compression, retention, payload optimization).
+
+### Proposal
+
+Provide pre-configured profiles for common scenarios:
+
+```ruby
+E11y.configure do |config|
+  # Option 1: Use a preset
+  config.use_preset :production_high_traffic
+  
+  # Option 2: Use a preset with overrides
+  config.use_preset :production_high_traffic do |preset|
+    preset.sampling.max_events_per_sec = 5_000  # Override default
+  end
+end
+```
+
+### Available Presets
+
+| Preset | Description | Sample Rate | Compression | Retention |
+|--------|-------------|-------------|-------------|-----------|
+| `:development` | Local dev, no sampling | 100% | None | 1 day |
+| `:staging` | Pre-prod testing | 50% | Gzip | 7 days |
+| `:production_low_traffic` | < 1K events/sec | 80% | Gzip | 30 days |
+| `:production_high_traffic` | > 10K events/sec | 10% | Zstd | 7 days hot, 90 days warm |
+| `:production_cost_optimized` | Aggressive cost reduction | 5% | Zstd level 9 | 3 days hot, 30 days warm |
+
+### Implementation Example
+
+```ruby
+module E11y
+  module Presets
+    PRODUCTION_HIGH_TRAFFIC = {
+      adaptive_sampling: {
+        enabled: true,
+        load_based: { max_events_per_sec: 10_000 },
+        error_based: { enabled: true },
+        value_based: {
+          high_value_patterns: [/^payment\./, /^order\./, /^error\./],
+          low_value_patterns: [/^debug\./, /^health_check/]
+        }
+      },
+      compression: {
+        enabled: true,
+        algorithm: :zstd,
+        level: 3
+      },
+      retention_tagging: {
+        enabled: true,
+        default_retention: 7.days,
+        retention_by_pattern: {
+          'audit.*' => 7.years,
+          'payment.*' => 1.year,
+          'debug.*' => 1.day
+        }
+      },
+      payload_minimization: {
+        enabled: true,
+        truncate_strings_at: 1000,
+        truncate_arrays_at: 100,
+        remove_null_fields: true
+      }
+    }.freeze
+  end
+end
+```
+
+### Benefits
+
+- ✅ Faster onboarding (< 5 minutes to production-ready config)
+- ✅ Best practices baked in
+- ✅ Easy to customize (override specific settings)
+- ✅ Reduces configuration errors
+
+### Priority
+
+**Medium (v1.1)**
+
+---
+
+## 2. Sampling Budget
+
+### Problem
+
+Current sampling is reactive (based on load). Hard to predict costs. Organizations need predictable telemetry budgets.
+
+### Proposal
+
+Proactive budget-based sampling:
+
+```ruby
+E11y.configure do |config|
+  config.cost_optimization do
+    sampling_budget do
+      enabled true
+      
+      # Set a daily event budget
+      daily_budget 10_000_000  # 10M events/day
+      
+      # Budget allocation by event type
+      allocate_by_pattern do
+        pattern 'payment.*', percent: 20  # 2M events/day
+        pattern 'order.*',   percent: 15  # 1.5M events/day
+        pattern 'error.*',   percent: 10  # 1M events/day (always track)
+        pattern 'debug.*',   percent: 5   # 500K events/day
+        pattern '*',         percent: 50  # 5M events/day (everything else)
+      end
+      
+      # Dynamic adjustment
+      rebalance_interval 1.hour  # Recalculate sample rates every hour
+      
+      # Overflow strategy
+      on_budget_exceeded :reduce_low_value  # or :stop_sampling, :alert_only
+      
+      # Alert when budget is 80% consumed
+      alert_threshold 0.8
+      alert_to :slack  # or :pagerduty, :email
+    end
+  end
+end
+```
+
+### How It Works
+
+#### 1. Budget Calculation
+
+```ruby
+# At start of each hour:
+hourly_budget = daily_budget / 24  # 416,666 events/hour
+
+# Per-pattern budget:
+payment_budget = hourly_budget * 0.20  # 83,333 events/hour
+```
+
+#### 2. Dynamic Sample Rate Adjustment
+
+```ruby
+# If payment events are at 50% of budget after 30 min:
+current_rate = 41,666 / (83,333 / 2)  # = 1.0 (100% sampling)
+
+# If payment events are at 90% of budget after 30 min:
+current_rate = 74,999 / (83,333 / 2)  # = 1.8 (need to reduce)
+new_sample_rate = 1.0 / 1.8  # = 55% sampling for next 30 min
+```
+
+#### 3. Cost Predictability
+
+```ruby
+# Known daily cost:
+cost_per_event = $0.0001  # e.g., Datadog pricing
+daily_cost = 10_000_000 * $0.0001 = $1,000/day
+monthly_cost = $1,000 * 30 = $30,000/month
+```
+
+### Benefits
+
+- ✅ Predictable costs (no surprises)
+- ✅ Budget enforcement (hard cap on events)
+- ✅ Intelligent allocation (high-value events get more budget)
+- ✅ Real-time alerts (before overspending)
+
+### Trade-offs
+
+- ⚠️ Complexity (requires Redis for distributed state)
+- ⚠️ Potential data loss (if budget exceeded)
+- ⚠️ Tuning required (optimal allocation per app)
+
+### Priority
+
+**Low (v1.2+)**
+
+### Alternative Approach
+
+- Use cloud cost management tools (AWS Cost Anomaly Detection, Datadog Cost Management)
+- Set alerts on actual billing, not event counts
+- Simpler, but less proactive
+
+---
+
+## 3. Additional Ideas (Placeholder)
+
+Future ideas to be added:
+
+- **ML-Based Anomaly Detection:** Automatically detect unusual patterns in events
+- **Event Replay from Storage:** Replay events from cold storage for debugging
+- **Multi-Tenant Support:** Isolate events by tenant/customer
+- **Event Transformation Rules:** Transform events before sending to adapters
+- **Custom Retention Policies:** More granular control over data lifecycle
+
+---
+
+## Related Use Cases
+
+- [UC-009: Cost Optimization](UC-015-cost-optimization.md) - Current cost optimization strategies
+- [UC-014: Adaptive Sampling](UC-014-adaptive-sampling.md) - Current sampling approach
+
+---
+
+## Related ADRs
+
+- [ADR-009: Cost Optimization](../ADR-009-cost-optimization.md) - Current implementation
+
+---
+
+**Status:** ✅ Draft Complete  
+**Next Review:** After v1.0 release  
+**Estimated Value:** High (for enterprise customers)

data/e11y.gemspec

@@ -0,0 +1,76 @@
+# frozen_string_literal: true
+
+require_relative "lib/e11y/version"
+
+Gem::Specification.new do |spec|
+  spec.name = "e11y"
+  spec.version = E11y::VERSION
+  spec.authors = ["Artur Seletskiy"]
+
+  spec.summary = "E11y - Easy Telemetry: Production-grade observability for Rails with zero-config SLO tracking"
+  spec.description = <<~DESC
+    E11y (Easy Telemetry) - production-ready observability gem for Ruby on Rails applications.
+
+    KEY FEATURES:
+    • 📊 Zero-Config SLO Tracking - automatic Service Level Objectives for HTTP endpoints and background jobs
+    • 🎯 Request-Scoped Debug Buffering - buffer debug logs in memory, flush only on errors (reduce log noise by 90%)
+    • 📈 Pattern-Based Metrics - auto-generate Prometheus/Yabeda metrics from business events
+    • 🔒 GDPR/SOC2 Compliance - built-in PII filtering and audit trails
+    • 🔌 Pluggable Adapters - send events to Loki, Sentry, OpenTelemetry, Elasticsearch, or custom backends
+    • 🚀 High Performance - zero-allocation event tracking, lock-free ring buffers, adaptive memory limits
+    • 🧵 Thread-Safe - designed for multi-threaded Rails apps and Sidekiq workers
+    • 🎭 Multi-Tenant Ready - trace context propagation across services with OpenTelemetry integration
+    • 📝 Type-Safe Events - declarative event schemas with dry-schema validation
+    • ⚡ Rate Limiting & Sampling - protect production from metric storms and cost overruns
+
+    Perfect for SuperApp architectures, microservices, and high-scale Rails applications.
+    Battle-tested patterns from Devise, Sidekiq, Sentry, and Yabeda.
+  DESC
+  spec.homepage = "https://github.com/arturseletskiy/e11y"
+  spec.license = "MIT"
+  spec.required_ruby_version = ">= 3.2.0"
+
+  spec.metadata["homepage_uri"] = spec.homepage
+  spec.metadata["source_code_uri"] = "https://github.com/arturseletskiy/e11y"
+  spec.metadata["changelog_uri"] = "https://github.com/arturseletskiy/e11y/blob/main/CHANGELOG.md"
+  spec.metadata["documentation_uri"] = "https://github.com/arturseletskiy/blob/main/e11y/docs"
+  spec.metadata["bug_tracker_uri"] = "https://github.com/arturseletskiy/e11y/issues"
+  spec.metadata["rubygems_mfa_required"] = "true"
+
+  # Specify which files should be added to the gem when it is released.
+  # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
+  spec.files = Dir.chdir(__dir__) do
+    `git ls-files -z`.split("\x0").reject do |f|
+      (File.expand_path(f) == __FILE__) ||
+        f.start_with?(*%w[bin/ test/ spec/ features/ .git .github appveyor Gemfile docs/researches/])
+    end
+  end
+  spec.bindir = "exe"
+  spec.executables = spec.files.grep(%r{\Aexe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  # Runtime dependencies
+  spec.add_dependency "activesupport", ">= 7.0"
+  spec.add_dependency "concurrent-ruby", "~> 1.2" # Thread-safe data structures
+  spec.add_dependency "dry-schema", "~> 1.13" # Event schema validation
+  spec.add_dependency "dry-types", "~> 1.7"
+  spec.add_dependency "zeitwerk", "~> 2.6"
+
+  # Development dependencies
+  spec.add_development_dependency "rack", "~> 3.0" # For Rack middleware testing
+  spec.add_development_dependency "rake", "~> 13.0"
+  spec.add_development_dependency "rspec", "~> 3.12"
+  spec.add_development_dependency "rubocop", "~> 1.50"
+  spec.add_development_dependency "rubocop-rake", "~> 0.6"
+  spec.add_development_dependency "rubocop-rspec", "~> 2.22"
+  spec.add_development_dependency "simplecov", "~> 0.22"
+  spec.add_development_dependency "webmock", "~> 3.19" # For HTTP adapter testing
+  spec.add_development_dependency "yard", "~> 0.9"
+
+  # Optional adapter dependencies (install only if using specific adapters)
+  # LokiAdapter: gem install faraday faraday-retry
+  # SentryAdapter: gem install sentry-ruby
+  spec.add_development_dependency "faraday", "~> 2.7" # For LokiAdapter
+  spec.add_development_dependency "faraday-retry", "~> 2.2" # For LokiAdapter retry middleware
+  spec.add_development_dependency "sentry-ruby", "~> 5.15" # For SentryAdapter
+end

data/lib/e11y/adapters/adaptive_batcher.rb

@@ -0,0 +1,207 @@
+# frozen_string_literal: true
+
+module E11y
+  module Adapters
+    # Adaptive batching helper for adapters
+    #
+    # Provides efficient event batching with automatic flushing based on:
+    # - Batch size threshold (max_size)
+    # - Time threshold (timeout)
+    # - Minimum batch size (min_size) for latency optimization
+    #
+    # Thread-safe implementation with mutex-protected buffer.
+    #
+    # @example Use in adapter
+    #   class MyAdapter < E11y::Adapters::Base
+    #     def initialize(config = {})
+    #       super
+    #       @batcher = AdaptiveBatcher.new(
+    #         max_size: 500,
+    #         timeout: 5.0,
+    #         flush_callback: method(:send_batch)
+    #       )
+    #     end
+    #
+    #     def write(event_data)
+    #       @batcher.add(event_data)
+    #     end
+    #
+    #     def write_batch(events)
+    #       @batcher.flush!
+    #       super
+    #     end
+    #
+    #     def close
+    #       @batcher.close
+    #       super
+    #     end
+    #
+    #     private
+    #
+    #     def send_batch(events)
+    #       # Send events to external system
+    #       http_client.post(events)
+    #     end
+    #   end
+    #
+    # @see ADR-004 Section 8.1 (Adaptive Batching)
+    class AdaptiveBatcher
+      # Initialize adaptive batcher
+      #
+      # @param min_size [Integer] Minimum batch size before timeout flush (default: 10)
+      # @param max_size [Integer] Maximum batch size (triggers immediate flush, default: 500)
+      # @param timeout [Float] Timeout in seconds for automatic flush (default: 5.0)
+      # @param flush_callback [Proc, Method] Callback to invoke on flush with events array
+      def initialize(flush_callback:, min_size: 10, max_size: 500, timeout: 5.0)
+        @min_size = min_size
+        @max_size = max_size
+        @timeout = timeout
+        @flush_callback = flush_callback
+
+        @buffer = []
+        @mutex = Mutex.new
+        @last_flush = Time.now
+        @closed = false
+        @timer_thread = nil
+
+        start_timer_thread!
+      end
+
+      # Add event to buffer
+      #
+      # Automatically flushes if max_size reached.
+      # Thread-safe operation.
+      #
+      # @param event_data [Hash] Event to add to buffer
+      # @return [Boolean] true if added successfully
+      def add(event_data)
+        return false if @closed
+
+        @mutex.synchronize do
+          @buffer << event_data
+
+          flush_unlocked! if should_flush_immediately?
+        end
+
+        true
+      end
+
+      # Flush buffer immediately
+      #
+      # Sends all buffered events to flush_callback.
+      # Thread-safe operation.
+      #
+      # @return [Boolean] true if flushed, false if buffer empty
+      def flush!
+        @mutex.synchronize { flush_unlocked! }
+      end
+
+      # Get current buffer size
+      #
+      # @return [Integer] Number of events in buffer
+      def buffer_size
+        @mutex.synchronize { @buffer.size }
+      end
+
+      # Check if buffer is empty
+      #
+      # @return [Boolean] true if buffer is empty
+      def empty?
+        @mutex.synchronize { @buffer.empty? }
+      end
+
+      # Close batcher and flush remaining events
+      #
+      # Stops timer thread and flushes any remaining events.
+      # Safe to call multiple times.
+      #
+      # @return [void]
+      def close
+        return if @closed
+
+        @closed = true
+        @timer_thread&.kill
+        @timer_thread = nil
+
+        flush!
+      end
+
+      private
+
+      # Start background timer thread for automatic flushing
+      #
+      # Timer thread checks periodically if timeout has expired
+      # and flushes buffer if min_size threshold is met.
+      #
+      # Check interval is min(timeout/2, 1 second) for responsiveness.
+      #
+      # @api private
+      def start_timer_thread!
+        check_interval = [@timeout / 2.0, 1.0].min
+
+        @timer_thread = Thread.new do
+          loop do
+            sleep check_interval
+
+            break if @closed
+
+            @mutex.synchronize do
+              flush_unlocked! if should_flush_timeout?
+            rescue StandardError => e
+              warn "[E11y] AdaptiveBatcher timer error: #{e.message}"
+            end
+          end
+        end
+
+        @timer_thread.name = "e11y-adaptive-batcher-timer"
+      end
+
+      # Flush buffer (unlocked - must be called within mutex.synchronize)
+      #
+      # @return [Boolean] true if flushed, false if buffer empty
+      # @api private
+      def flush_unlocked!
+        return false if @buffer.empty?
+
+        events = @buffer.dup
+        @buffer.clear
+        @last_flush = Time.now
+
+        # Release mutex before I/O operation
+        @mutex.unlock
+        begin
+          @flush_callback.call(events)
+          true
+        ensure
+          @mutex.lock
+        end
+      end
+
+      # Check if should flush immediately (max_size reached)
+      #
+      # @return [Boolean]
+      # @api private
+      def should_flush_immediately?
+        @buffer.size >= @max_size
+      end
+
+      # Check if should flush on timeout
+      #
+      # @return [Boolean]
+      # @api private
+      def should_flush_timeout?
+        return false if @buffer.empty?
+
+        timeout_expired? && @buffer.size >= @min_size
+      end
+
+      # Check if timeout has expired since last flush
+      #
+      # @return [Boolean]
+      # @api private
+      def timeout_expired?
+        (Time.now - @last_flush) >= @timeout
+      end
+    end
+  end
+end