e11y 0.2.0 → 1.1.0

data/docs/use_cases/UC-008-opentelemetry-integration.md

@@ -65,7 +65,7 @@ end
 
 ## 🎯 Features
 
-> **Implementation:** See [ADR-007: OpenTelemetry Integration](../ADR-007-opentelemetry-integration.md) for complete architecture, including [Section 3: OTel Collector Adapter](../ADR-007-opentelemetry-integration.md#3-otel-collector-adapter), [Section 4: Semantic Conventions](../ADR-007-opentelemetry-integration.md#4-semantic-conventions), and [Section 5: Logs Signal Export](../ADR-007-opentelemetry-integration.md#5-logs-signal-export).
+> **Implementation:** See [ADR-007: OpenTelemetry Integration](../architecture/ADR-007-opentelemetry-integration.md) for complete architecture, including [Section 3: OTel Collector Adapter](../architecture/ADR-007-opentelemetry-integration.md#3-otel-collector-adapter), [Section 4: Semantic Conventions](../architecture/ADR-007-opentelemetry-integration.md#4-semantic-conventions), and [Section 5: Logs Signal Export](../architecture/ADR-007-opentelemetry-integration.md#5-logs-signal-export).
 
 ### 1. OpenTelemetry Collector Adapter
 
@@ -73,28 +73,11 @@ end
 ```ruby
 # config/initializers/e11y.rb
 E11y.configure do |config|
-  config.adapters << E11y::Adapters::OpenTelemetryCollectorAdapter.new(
+  config.adapters[:otel] = E11y::Adapters::OpenTelemetryCollector.new(
     endpoint: ENV['OTEL_EXPORTER_OTLP_ENDPOINT'] || 'http://localhost:4318',
-    protocol: :http,  # :http or :grpc
-    headers: {
-      'X-API-Key' => ENV['OTEL_API_KEY']
-    },
-    
-    # Signal types
-    export_logs: true,      # E11y events → OTel Logs Signal
-    export_traces: true,    # Spans from events → OTel Traces
-    export_metrics: false,  # Use Yabeda for metrics (better)
-    
-    # Batching
-    batch_size: 100,
-    flush_interval: 10.seconds,
-    
-    # Compression
-    compression: :gzip,
-    
-    # Retry
-    retry_enabled: true,
-    max_retries: 3
+    service_name: 'my-app',
+    headers: { 'X-API-Key' => ENV['OTEL_API_KEY'] },
+    compress: true  # default, gzip on HTTP body
   )
 end
 
@@ -103,14 +86,14 @@ end
 #                                                   ├─→ Jaeger (traces)
 #                                                   ├─→ Loki (logs)
 #                                                   ├─→ Prometheus (metrics)
-#                                                   └─→ S3 (archive)
+#                                                   └─→ Object storage (archive, via OTel exporter)
 ```
 
 ---
 
 ### 2. Semantic Conventions Mapping
 
-> **Implementation:** See [ADR-007 Section 4: Semantic Conventions](../ADR-007-opentelemetry-integration.md#4-semantic-conventions) for automatic field mapping across HTTP, DB, RPC, Messaging, and Exception patterns.
+> **Implementation:** See [ADR-007 Section 4: Semantic Conventions](../architecture/ADR-007-opentelemetry-integration.md#4-semantic-conventions) for automatic field mapping across HTTP, DB, RPC, Messaging, and Exception patterns.
 
 **Automatic field mapping to OTel standards:**
 ```ruby
@@ -171,7 +154,7 @@ Events::HttpRequest.track(
 
 ### 3. Automatic Span Creation
 
-> **Implementation:** See [ADR-007 Section 6: Traces Signal Export](../ADR-007-opentelemetry-integration.md#6-traces-signal-export) for automatic span creation rules and parent-child relationships.
+> **Implementation:** See [ADR-007 Section 6: Traces Signal Export](../architecture/ADR-007-opentelemetry-integration.md#6-traces-signal-export) for automatic span creation rules and parent-child relationships.
 
 **Create spans from E11y events:**
 ```ruby
@@ -207,7 +190,7 @@ Events::OrderProcessingStarted.track(
 
 ### 4. W3C Trace Context Integration
 
-> **Implementation:** See [ADR-007 Section 8: Trace Context Integration](../ADR-007-opentelemetry-integration.md#8-trace-context-integration) for OTel SDK as primary trace context source.
+> **Implementation:** See [ADR-007 Section 8: Trace Context Integration](../architecture/ADR-007-opentelemetry-integration.md#8-trace-context-integration) for OTel SDK as primary trace context source.
 
 **Automatic trace context from OpenTelemetry SDK:**
 ```ruby
@@ -240,7 +223,7 @@ Events::OrderCreated.track(order_id: '123')
 
 ### 5. OTel Logs Signal Export
 
-> **Implementation:** See [ADR-007 Section 5: Logs Signal Export](../ADR-007-opentelemetry-integration.md#5-logs-signal-export) for OTLP JSON format and trace correlation details.
+> **Implementation:** See [ADR-007 Section 5: Logs Signal Export](../architecture/ADR-007-opentelemetry-integration.md#5-logs-signal-export) for OTLP JSON format and trace correlation details.
 
 **Export E11y events as OpenTelemetry Logs:**
 ```ruby
@@ -297,7 +280,7 @@ Events::OrderCreated.track(order_id: '123')
 ### 6. Baggage PII Protection (C08 Resolution) ⚠️ CRITICAL
 
 > **⚠️ CRITICAL: C08 Conflict Resolution - PII Leaking via OpenTelemetry Baggage**  
-> **See:** [ADR-006 Section 5.5](../ADR-006-security-compliance.md#55-opentelemetry-baggage-pii-protection-c08-resolution--critical) for detailed architecture and GDPR compliance rationale.  
+> **See:** [ADR-006 Section 5.5](../architecture/ADR-006-security-compliance.md#55-opentelemetry-baggage-pii-protection-c08-resolution--critical) for detailed architecture and GDPR compliance rationale.  
 > **Problem:** OpenTelemetry Baggage propagates data via HTTP headers (`baggage: key1=value1,key2=value2`), bypassing E11y's PII filtering. If a developer accidentally adds PII to baggage, it leaks across all services.  
 > **Solution:** Block ALL baggage keys by default, allow ONLY safe keys via allowlist.
 
@@ -330,33 +313,23 @@ E11y blocks ALL baggage keys by default, allowing ONLY safe keys (no PII):
 ```ruby
 # config/initializers/e11y.rb
 E11y.configure do |config|
-  config.security.baggage_protection do
-    enabled true  # ✅ CRITICAL: Always enable in production
-    
-    # Allowlist: ONLY these keys are safe
-    allowed_keys [
-      'trace_id',       # ✅ Safe: Correlation ID
-      'span_id',        # ✅ Safe: Trace context
-      'environment',    # ✅ Safe: Deployment context
-      'version',        # ✅ Safe: Service version
-      'service_name',   # ✅ Safe: Service identifier
-      'request_id',     # ✅ Safe: Request identifier
-      # Custom safe keys (no PII!):
-      'feature_flag_id',  # ✅ Safe: Feature flag name
-      'ab_test_variant'   # ✅ Safe: A/B test group
-    ]
-    
-    # Block mode: What happens when PII detected?
-    block_mode :silent   # Options: :silent (log), :warn (log+warn), :raise (exception)
-    
-    # Monitoring: Track violations
-    on_blocked_key do |key, value, caller_location|
-      Yabeda.e11y_baggage_pii_blocked.increment(
-        key: key,
-        service: ENV['SERVICE_NAME']
-      )
-    end
-  end
+  config.security_baggage_protection_enabled = true  # ✅ CRITICAL: Always enable in production
+  
+  # Allowlist: ONLY these keys are safe
+  config.security_baggage_protection_allowed_keys = [
+    'trace_id',       # ✅ Safe: Correlation ID
+    'span_id',        # ✅ Safe: Trace context
+    'environment',    # ✅ Safe: Deployment context
+    'version',        # ✅ Safe: Service version
+    'service_name',   # ✅ Safe: Service identifier
+    'request_id',     # ✅ Safe: Request identifier
+    # Custom safe keys (no PII!):
+    'feature_flag_id',  # ✅ Safe: Feature flag name
+    'ab_test_variant'   # ✅ Safe: A/B test group
+  ]
+  
+  # Block mode: What happens when PII detected?
+  config.security_baggage_protection_block_mode = :silent  # Options: :silent (log), :warn (log+warn), :raise (exception)
 end
 ```
 
@@ -421,11 +394,9 @@ Fail fast in non-production environments:
 ```ruby
 # config/environments/development.rb
 E11y.configure do |config|
-  config.security.baggage_protection do
-    enabled true
-    block_mode :raise  # ← RAISE exception on blocked keys (fail fast)
-    allowed_keys E11y::Middleware::BaggageProtection::ALLOWED_KEYS
-  end
+  config.security_baggage_protection_enabled = true
+  config.security_baggage_protection_block_mode = :raise  # ← RAISE exception on blocked keys (fail fast)
+  config.security_baggage_protection_allowed_keys = E11y::BAGGAGE_PROTECTION_DEFAULT_ALLOWED_KEYS
 end
 
 # Developer tries to set PII:
@@ -531,18 +502,14 @@ exporters:
   prometheus:
     endpoint: 0.0.0.0:8889
   
-  # Archive → S3
-  s3:
-    region: us-east-1
-    bucket: telemetry-archive
-    prefix: logs/
+  # Archive: OTel Collector can export to object storage (add exporter config as needed)
 
 service:
   pipelines:
     logs:
       receivers: [otlp]
       processors: [batch, resource, filter]
-      exporters: [loki, s3]
+      exporters: [loki]
     
     traces:
       receivers: [otlp]
@@ -558,11 +525,9 @@ service:
 ```ruby
 # config/initializers/e11y.rb
 E11y.configure do |config|
-  config.adapters << E11y::Adapters::OpenTelemetryCollectorAdapter.new(
+  config.adapters[:otel] = E11y::Adapters::OpenTelemetryCollector.new(
     endpoint: 'http://otel-collector:4318',
-    protocol: :http,
-    export_logs: true,
-    export_traces: true
+    service_name: 'my-app'
   )
 end
 
@@ -742,14 +707,11 @@ end
 # config/initializers/e11y.rb
 E11y.configure do |config|
   # Single adapter: OTel Collector
-  config.adapters = [
-    E11y::Adapters::OpenTelemetryCollectorAdapter.new(
-      endpoint: 'http://otel-collector:4318',
-      export_logs: true,
-      export_traces: true
-    )
-  ]
-  
+  config.adapters[:otel] = E11y::Adapters::OpenTelemetryCollector.new(
+    endpoint: 'http://otel-collector:4318',
+    service_name: 'my-app'
+  )
+
   # OTel Collector handles routing to multiple backends!
   # No need for multiple E11y adapters
 end
@@ -757,7 +719,7 @@ end
 # OTel Collector routes to:
 # - Loki (logs, last 30 days)
 # - Jaeger (traces, last 7 days)
-# - S3 (archive, long-term storage)
+# - Object storage (archive, long-term; OTel exporter)
 # - Prometheus (metrics via remote write)
 
 # Benefits:
@@ -931,7 +893,7 @@ end
 ┌─────────────┐ │                          ├─→ Loki (logs)
 │  Sidekiq    │─┘                          ├─→ Jaeger (traces)
 └─────────────┘                            ├─→ Prometheus (metrics)
-                                           ├─→ S3 (archive)
+                                           ├─→ Object storage (archive)
                                            └─→ Datadog (optional)
 ```
 
@@ -1048,11 +1010,10 @@ end
 **1. Use OTel Collector in production**
 ```ruby
 # ✅ GOOD: Central pipeline
-config.adapters = [
-  E11y::Adapters::OpenTelemetryCollectorAdapter.new(
-    endpoint: 'http://otel-collector:4318'
-  )
-]
+config.adapters[:otel] = E11y::Adapters::OpenTelemetryCollector.new(
+  endpoint: 'http://otel-collector:4318',
+  service_name: 'my-app'
+)
 
 # OTel Collector handles:
 # - Sampling
@@ -1094,13 +1055,14 @@ end
 config.adapters = [
   E11y::Adapters::JaegerAdapter.new(...),
   E11y::Adapters::LokiAdapter.new(...),
-  E11y::Adapters::S3Adapter.new(...)
+  E11y::Adapters::FileAdapter.new(...)  # Direct to file (bypasses OTel routing)
 ]
 
 # ✅ GOOD: Through OTel Collector
-config.adapters = [
-  E11y::Adapters::OpenTelemetryCollectorAdapter.new(...)
-]
+config.adapters[:otel] = E11y::Adapters::OpenTelemetryCollector.new(
+  endpoint: ENV['OTEL_EXPORTER_OTLP_ENDPOINT'],
+  service_name: 'my-app'
+)
 ```
 
 **2. Don't use custom field names**

data/docs/use_cases/UC-009-multi-service-tracing.md

@@ -60,7 +60,7 @@ Events::OrderShipping.track(order_id: '789', tracking: 'TRACK123')
 
 ## 🎯 Features
 
-> **Implementation:** See [ADR-005: Tracing Context](../ADR-005-tracing-context.md) for complete architecture, including [Section 5: W3C Trace Context](../ADR-005-tracing-context.md#5-w3c-trace-context), [Section 6.1: HTTP Propagator](../ADR-005-tracing-context.md#61-http-propagator-outgoing-requests), and [Section 8: Context Inheritance](../ADR-005-tracing-context.md#8-context-inheritance-thread-fiber-support).
+> **Implementation:** See [ADR-005: Tracing Context](../architecture/ADR-005-tracing-context.md) for complete architecture, including [Section 5: W3C Trace Context](../architecture/ADR-005-tracing-context.md#5-w3c-trace-context), [Section 6.1: HTTP Propagator](../architecture/ADR-005-tracing-context.md#61-http-propagator-outgoing-requests), and [Section 8: Context Inheritance](../architecture/ADR-005-tracing-context.md#8-context-inheritance-thread-fiber-support).
 
 ### 1. Automatic W3C Trace Context Propagation
 
@@ -206,7 +206,7 @@ end
 
 ### 3. Background Job Trace Propagation (C17 Resolution) ⚠️
 
-> **Implementation:** See [ADR-005 Section 8.3: Background Job Tracing Strategy](../ADR-005-tracing-context.md#83-background-job-tracing-strategy-c17-resolution) for the hybrid tracing model.
+> **Implementation:** See [ADR-005 Section 8.3: Background Job Tracing Strategy](../architecture/ADR-005-tracing-context.md#83-background-job-tracing-strategy-c17-resolution) for the hybrid tracing model.
 
 **Hybrid Tracing Model (New Trace + Parent Link):**
 ```ruby
@@ -279,7 +279,7 @@ end
 3. **Trace clarity:** Separate timelines for sync (request) vs async (job) operations
 4. **Link preserved:** `parent_trace_id` allows reconstructing full flow
 
-See [ADR-005 §8.3](../ADR-005-tracing-context.md#83-background-job-tracing-strategy-c17-resolution) for detailed rationale.
+See [ADR-005 §8.3](../architecture/ADR-005-tracing-context.md#83-background-job-tracing-strategy-c17-resolution) for detailed rationale.
 
 **Visual Diagram: Request Trace → Job Trace (with parent link)**
 
@@ -356,47 +356,23 @@ ORDER BY timestamp ASC;
 
 ### 4. Cross-Service Latency Measurement
 
-**Automatic service-to-service timing:**
-```ruby
-# config/initializers/e11y.rb
-E11y.configure do |config|
-  config.distributed_tracing do
-    # Measure cross-service latency
-    measure_service_latency true
-    
-    # Track service hops
-    track_service_hops true
-  end
-end
+**Service-to-service timing** is not switched on by a `config.distributed_tracing` block (that DSL does not exist). Model it with **event payloads and your metrics stack** (timestamps, `trace_id`, PromQL / LogQL), or extend tracing per [ADR-005](../architecture/ADR-005-tracing-context.md).
 
+```ruby
+# Example only: emit your own correlation fields from each service
 # Service A: API Gateway
-Events::OrderCreated.track(
-  order_id: '789',
-  timestamp_sent: Time.current  # Auto-added!
-)
-
+Events::OrderCreated.track(order_id: "789", timestamp_sent: Time.current.iso8601)
 # Service B: Payment Service
-Events::PaymentReceived.track(
-  order_id: '789',
-  timestamp_received: Time.current  # Auto-added!
-)
+Events::PaymentReceived.track(order_id: "789", timestamp_received: Time.current.iso8601)
 
-# E11y automatically calculates:
-# - Network latency: timestamp_received - timestamp_sent
-# - Service hop count: 1 (API → Payment)
-# - Total trace duration: last event - first event
-
-# Metrics (automatic!):
-# e11y_service_to_service_latency_ms{from="api",to="payment"} = 50
-# e11y_service_hops{trace_id="abc-123"} = 3
-# e11y_trace_duration_ms{trace_id="abc-123"} = 5020
+# You derive latency / hop counts in Loki, Prometheus, or SQL — not from a magic config block.
 ```
 
 ---
 
 ### 5. Service Mesh Integration
 
-> **Implementation:** See [ADR-005 Section 5.3: HTTP Header Extraction](../ADR-005-tracing-context.md#53-http-header-extraction-w3c-legacy-headers) for W3C standard header support enabling service mesh compatibility.
+> **Implementation:** See [ADR-005 Section 5.3: HTTP Header Extraction](../architecture/ADR-005-tracing-context.md#53-http-header-extraction-w3c-legacy-headers) for W3C standard header support enabling service mesh compatibility.
 
 **Automatic integration with Istio/Linkerd:**
 ```ruby
@@ -707,7 +683,7 @@ class OrdersController < ApplicationController
       key: order.id,
       value: order.to_json,
       headers: {
-        'traceparent' => E11y::TraceContext.current.to_traceparent
+        "traceparent" => E11y::Tracing::Propagator.build_traceparent
         # ↑ W3C Trace Context header in Kafka message!
       }
     )
@@ -777,127 +753,24 @@ end
 
 ## 🔧 Configuration
 
-### Full Configuration
+Older revisions of this document showed nested `config.distributed_tracing do`, `background_jobs do`, `message_queues do`, and similar blocks—**that DSL is not implemented** in the current gem. Do not paste those snippets into an app.
+
+### What exists today
 
 ```ruby
 # config/initializers/e11y.rb
 E11y.configure do |config|
-  config.distributed_tracing do
-    # === TRACE PROPAGATION ===
-    propagation do
-      # W3C Trace Context (standard)
-      w3c_trace_context enabled: true
-      
-      # B3 (Zipkin)
-      b3 enabled: true, single_header: true
-      
-      # Jaeger
-      jaeger enabled: false
-      
-      # Datadog
-      datadog enabled: false
-    end
-    
-    # === HTTP CLIENTS ===
-    http_clients do
-      # Faraday
-      faraday do
-        enabled true
-        inject_headers true
-        extract_headers true
-      end
-      
-      # Net::HTTP
-      net_http do
-        enabled true
-        inject_headers true
-      end
-      
-      # HTTParty
-      httparty do
-        enabled true
-        inject_headers true
-      end
-      
-      # RestClient
-      rest_client do
-        enabled true
-        inject_headers true
-      end
-    end
-    
-    # === BACKGROUND JOBS ===
-    background_jobs do
-      # Sidekiq
-      sidekiq do
-        enabled true
-        propagate_trace_context true
-        store_in_job_metadata true
-      end
-      
-      # ActiveJob
-      active_job do
-        enabled true
-        propagate_trace_context true
-      end
-    end
-    
-    # === MESSAGE QUEUES ===
-    message_queues do
-      # Kafka
-      kafka do
-        enabled true
-        inject_headers true
-        extract_headers true
-        header_name 'traceparent'
-      end
-      
-      # RabbitMQ
-      rabbitmq do
-        enabled true
-        inject_headers true
-        extract_headers true
-      end
-      
-      # AWS SQS
-      sqs do
-        enabled true
-        use_message_attributes true
-      end
-    end
-    
-    # === SERVICE MESH ===
-    service_mesh do
-      auto_detect true
-      
-      # Istio
-      istio do
-        enabled true
-        use_headers true
-      end
-      
-      # Linkerd
-      linkerd do
-        enabled true
-        use_headers true
-      end
-    end
-    
-    # === METRICS ===
-    metrics do
-      # Service-to-service latency
-      measure_service_latency true
-      
-      # Service hops
-      track_service_hops true
-      
-      # Trace duration
-      measure_trace_duration true
-    end
-  end
+  config.rails_instrumentation_enabled = true
+  config.sidekiq_enabled = true
+  config.active_job_enabled = true
+
+  # Optional outgoing Net::HTTP trace propagation (see Railtie + lib/e11y/tracing/)
+  config.enable_http_tracing = true
 end
 ```
 
+W3C `traceparent` on the **inbound** request is handled by `E11y::Middleware::Request`. Multi-protocol propagation (Faraday-only, Kafka, mesh) is described in [ADR-005](../architecture/ADR-005-tracing-context.md) and [DISTRIBUTED_TRACING.md](../DISTRIBUTED_TRACING.md)—implement or enable per what is actually merged in your `e11y` version.
+
 ---
 
 ## 📊 Monitoring Distributed Traces
@@ -936,15 +809,7 @@ histogram_quantile(0.95,
 
 ### ✅ DO
 
-**1. Use W3C Trace Context (standard)**
-```ruby
-# ✅ GOOD: Industry standard
-config.distributed_tracing do
-  propagation do
-    w3c_trace_context enabled: true
-  end
-end
-```
+**1. Use W3C Trace Context (standard)** — inbound requests are parsed in `E11y::Middleware::Request` when clients send `traceparent`; propagate the same header on outgoing HTTP where supported (`enable_http_tracing` for Net::HTTP, or your client middleware).
 
 **2. Mark service boundaries**
 ```ruby
@@ -962,11 +827,10 @@ end
 
 **3. Propagate context in async jobs**
 ```ruby
-# ✅ GOOD: Preserve trace_id in jobs
-config.background_jobs do
-  sidekiq do
-    propagate_trace_context true
-  end
+# Enable the shipped Sidekiq / Active Job integration (trace + metadata on jobs)
+E11y.configure do |config|
+  config.sidekiq_enabled = true
+  config.active_job_enabled = true
 end
 ```
 
@@ -975,20 +839,8 @@ end
 ### ❌ DON'T
 
 **1. Don't use different context formats**
-```ruby
-# ❌ BAD: Mixed formats (incompatible!)
-# Service A: W3C Trace Context
-# Service B: B3 format
-# Service C: Custom format
-# → Can't correlate!
-
-# ✅ GOOD: Single format everywhere
-config.distributed_tracing do
-  propagation do
-    w3c_trace_context enabled: true  # All services!
-  end
-end
-```
+
+Mixing B3, vendor headers, and W3C in one system breaks correlation. Standardize on **W3C `traceparent`** wherever you control the client and server.
 
 **2. Don't forget message queue propagation**
 ```ruby
@@ -1001,7 +853,7 @@ KafkaProducer.publish(
   topic: 'orders',
   value: order.to_json,
   headers: {
-    'traceparent' => E11y::TraceContext.current.to_traceparent
+    "traceparent" => E11y::Tracing::Propagator.build_traceparent
   }
 )
 ```