e11y 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (157) hide show
  1. checksums.yaml +7 -0
  2. data/.rspec +4 -0
  3. data/.rubocop.yml +69 -0
  4. data/CHANGELOG.md +26 -0
  5. data/CODE_OF_CONDUCT.md +64 -0
  6. data/LICENSE.txt +21 -0
  7. data/README.md +179 -0
  8. data/Rakefile +37 -0
  9. data/benchmarks/run_all.rb +33 -0
  10. data/config/README.md +83 -0
  11. data/config/loki-local-config.yaml +35 -0
  12. data/config/prometheus.yml +15 -0
  13. data/docker-compose.yml +78 -0
  14. data/docs/00-ICP-AND-TIMELINE.md +483 -0
  15. data/docs/01-SCALE-REQUIREMENTS.md +858 -0
  16. data/docs/ADR-001-architecture.md +2617 -0
  17. data/docs/ADR-002-metrics-yabeda.md +1395 -0
  18. data/docs/ADR-003-slo-observability.md +3337 -0
  19. data/docs/ADR-004-adapter-architecture.md +2385 -0
  20. data/docs/ADR-005-tracing-context.md +1372 -0
  21. data/docs/ADR-006-security-compliance.md +4143 -0
  22. data/docs/ADR-007-opentelemetry-integration.md +1385 -0
  23. data/docs/ADR-008-rails-integration.md +1911 -0
  24. data/docs/ADR-009-cost-optimization.md +2993 -0
  25. data/docs/ADR-010-developer-experience.md +2166 -0
  26. data/docs/ADR-011-testing-strategy.md +1836 -0
  27. data/docs/ADR-012-event-evolution.md +958 -0
  28. data/docs/ADR-013-reliability-error-handling.md +2750 -0
  29. data/docs/ADR-014-event-driven-slo.md +1533 -0
  30. data/docs/ADR-015-middleware-order.md +1061 -0
  31. data/docs/ADR-016-self-monitoring-slo.md +1234 -0
  32. data/docs/API-REFERENCE-L28.md +914 -0
  33. data/docs/COMPREHENSIVE-CONFIGURATION.md +2366 -0
  34. data/docs/IMPLEMENTATION_NOTES.md +2804 -0
  35. data/docs/IMPLEMENTATION_PLAN.md +1971 -0
  36. data/docs/IMPLEMENTATION_PLAN_ARCHITECTURE.md +586 -0
  37. data/docs/PLAN.md +148 -0
  38. data/docs/QUICK-START.md +934 -0
  39. data/docs/README.md +296 -0
  40. data/docs/design/00-memory-optimization.md +593 -0
  41. data/docs/guides/MIGRATION-L27-L28.md +692 -0
  42. data/docs/guides/PERFORMANCE-BENCHMARKS.md +434 -0
  43. data/docs/guides/README.md +44 -0
  44. data/docs/prd/01-overview-vision.md +440 -0
  45. data/docs/use_cases/README.md +119 -0
  46. data/docs/use_cases/UC-001-request-scoped-debug-buffering.md +813 -0
  47. data/docs/use_cases/UC-002-business-event-tracking.md +1953 -0
  48. data/docs/use_cases/UC-003-pattern-based-metrics.md +1627 -0
  49. data/docs/use_cases/UC-004-zero-config-slo-tracking.md +728 -0
  50. data/docs/use_cases/UC-005-sentry-integration.md +759 -0
  51. data/docs/use_cases/UC-006-trace-context-management.md +905 -0
  52. data/docs/use_cases/UC-007-pii-filtering.md +2648 -0
  53. data/docs/use_cases/UC-008-opentelemetry-integration.md +1153 -0
  54. data/docs/use_cases/UC-009-multi-service-tracing.md +1043 -0
  55. data/docs/use_cases/UC-010-background-job-tracking.md +1018 -0
  56. data/docs/use_cases/UC-011-rate-limiting.md +1906 -0
  57. data/docs/use_cases/UC-012-audit-trail.md +2301 -0
  58. data/docs/use_cases/UC-013-high-cardinality-protection.md +2127 -0
  59. data/docs/use_cases/UC-014-adaptive-sampling.md +1940 -0
  60. data/docs/use_cases/UC-015-cost-optimization.md +735 -0
  61. data/docs/use_cases/UC-016-rails-logger-migration.md +785 -0
  62. data/docs/use_cases/UC-017-local-development.md +867 -0
  63. data/docs/use_cases/UC-018-testing-events.md +1081 -0
  64. data/docs/use_cases/UC-019-tiered-storage-migration.md +562 -0
  65. data/docs/use_cases/UC-020-event-versioning.md +708 -0
  66. data/docs/use_cases/UC-021-error-handling-retry-dlq.md +956 -0
  67. data/docs/use_cases/UC-022-event-registry.md +648 -0
  68. data/docs/use_cases/backlog.md +226 -0
  69. data/e11y.gemspec +76 -0
  70. data/lib/e11y/adapters/adaptive_batcher.rb +207 -0
  71. data/lib/e11y/adapters/audit_encrypted.rb +239 -0
  72. data/lib/e11y/adapters/base.rb +580 -0
  73. data/lib/e11y/adapters/file.rb +224 -0
  74. data/lib/e11y/adapters/in_memory.rb +216 -0
  75. data/lib/e11y/adapters/loki.rb +333 -0
  76. data/lib/e11y/adapters/otel_logs.rb +203 -0
  77. data/lib/e11y/adapters/registry.rb +141 -0
  78. data/lib/e11y/adapters/sentry.rb +230 -0
  79. data/lib/e11y/adapters/stdout.rb +108 -0
  80. data/lib/e11y/adapters/yabeda.rb +370 -0
  81. data/lib/e11y/buffers/adaptive_buffer.rb +339 -0
  82. data/lib/e11y/buffers/base_buffer.rb +40 -0
  83. data/lib/e11y/buffers/request_scoped_buffer.rb +246 -0
  84. data/lib/e11y/buffers/ring_buffer.rb +267 -0
  85. data/lib/e11y/buffers.rb +14 -0
  86. data/lib/e11y/console.rb +122 -0
  87. data/lib/e11y/current.rb +48 -0
  88. data/lib/e11y/event/base.rb +894 -0
  89. data/lib/e11y/event/value_sampling_config.rb +84 -0
  90. data/lib/e11y/events/base_audit_event.rb +43 -0
  91. data/lib/e11y/events/base_payment_event.rb +33 -0
  92. data/lib/e11y/events/rails/cache/delete.rb +21 -0
  93. data/lib/e11y/events/rails/cache/read.rb +23 -0
  94. data/lib/e11y/events/rails/cache/write.rb +22 -0
  95. data/lib/e11y/events/rails/database/query.rb +45 -0
  96. data/lib/e11y/events/rails/http/redirect.rb +21 -0
  97. data/lib/e11y/events/rails/http/request.rb +26 -0
  98. data/lib/e11y/events/rails/http/send_file.rb +21 -0
  99. data/lib/e11y/events/rails/http/start_processing.rb +26 -0
  100. data/lib/e11y/events/rails/job/completed.rb +22 -0
  101. data/lib/e11y/events/rails/job/enqueued.rb +22 -0
  102. data/lib/e11y/events/rails/job/failed.rb +22 -0
  103. data/lib/e11y/events/rails/job/scheduled.rb +23 -0
  104. data/lib/e11y/events/rails/job/started.rb +22 -0
  105. data/lib/e11y/events/rails/log.rb +56 -0
  106. data/lib/e11y/events/rails/view/render.rb +23 -0
  107. data/lib/e11y/events.rb +18 -0
  108. data/lib/e11y/instruments/active_job.rb +201 -0
  109. data/lib/e11y/instruments/rails_instrumentation.rb +141 -0
  110. data/lib/e11y/instruments/sidekiq.rb +175 -0
  111. data/lib/e11y/logger/bridge.rb +205 -0
  112. data/lib/e11y/metrics/cardinality_protection.rb +172 -0
  113. data/lib/e11y/metrics/cardinality_tracker.rb +134 -0
  114. data/lib/e11y/metrics/registry.rb +234 -0
  115. data/lib/e11y/metrics/relabeling.rb +226 -0
  116. data/lib/e11y/metrics.rb +102 -0
  117. data/lib/e11y/middleware/audit_signing.rb +174 -0
  118. data/lib/e11y/middleware/base.rb +140 -0
  119. data/lib/e11y/middleware/event_slo.rb +167 -0
  120. data/lib/e11y/middleware/pii_filter.rb +266 -0
  121. data/lib/e11y/middleware/pii_filtering.rb +280 -0
  122. data/lib/e11y/middleware/rate_limiting.rb +214 -0
  123. data/lib/e11y/middleware/request.rb +163 -0
  124. data/lib/e11y/middleware/routing.rb +157 -0
  125. data/lib/e11y/middleware/sampling.rb +254 -0
  126. data/lib/e11y/middleware/slo.rb +168 -0
  127. data/lib/e11y/middleware/trace_context.rb +131 -0
  128. data/lib/e11y/middleware/validation.rb +118 -0
  129. data/lib/e11y/middleware/versioning.rb +132 -0
  130. data/lib/e11y/middleware.rb +12 -0
  131. data/lib/e11y/pii/patterns.rb +90 -0
  132. data/lib/e11y/pii.rb +13 -0
  133. data/lib/e11y/pipeline/builder.rb +155 -0
  134. data/lib/e11y/pipeline/zone_validator.rb +110 -0
  135. data/lib/e11y/pipeline.rb +12 -0
  136. data/lib/e11y/presets/audit_event.rb +65 -0
  137. data/lib/e11y/presets/debug_event.rb +34 -0
  138. data/lib/e11y/presets/high_value_event.rb +51 -0
  139. data/lib/e11y/presets.rb +19 -0
  140. data/lib/e11y/railtie.rb +138 -0
  141. data/lib/e11y/reliability/circuit_breaker.rb +216 -0
  142. data/lib/e11y/reliability/dlq/file_storage.rb +277 -0
  143. data/lib/e11y/reliability/dlq/filter.rb +117 -0
  144. data/lib/e11y/reliability/retry_handler.rb +207 -0
  145. data/lib/e11y/reliability/retry_rate_limiter.rb +117 -0
  146. data/lib/e11y/sampling/error_spike_detector.rb +225 -0
  147. data/lib/e11y/sampling/load_monitor.rb +161 -0
  148. data/lib/e11y/sampling/stratified_tracker.rb +92 -0
  149. data/lib/e11y/sampling/value_extractor.rb +82 -0
  150. data/lib/e11y/self_monitoring/buffer_monitor.rb +79 -0
  151. data/lib/e11y/self_monitoring/performance_monitor.rb +97 -0
  152. data/lib/e11y/self_monitoring/reliability_monitor.rb +146 -0
  153. data/lib/e11y/slo/event_driven.rb +150 -0
  154. data/lib/e11y/slo/tracker.rb +119 -0
  155. data/lib/e11y/version.rb +9 -0
  156. data/lib/e11y.rb +283 -0
  157. metadata +452 -0
data/docs/PLAN.md ADDED
@@ -0,0 +1,148 @@
1
+ # ✅ TeamTab Plan Successfully Created!
2
+
3
+ **Date:** 2026-01-17
4
+ **Status:** Complete (100%)
5
+
6
+ ---
7
+
8
+ ## 📊 CREATED PLANS SUMMARY
9
+
10
+ ### **7 Root Tasks (Phases):**
11
+
12
+ 1. **FEAT-4711:** PHASE 0: Gem Setup & Best Practices (Week -1)
13
+ - 3 L3 components + 20 L4 subtasks
14
+ - Root task (created separately)
15
+
16
+ 2. **FEAT-4738:** PHASE 1: Foundation (Weeks 1-4)
17
+ - 6 L3 components + 19 L4 subtasks
18
+ - Root task (created separately)
19
+
20
+ 3. **FEAT-4770:** PHASE 2: Core Features (Weeks 5-10)
21
+ - 4 L3 components + 12 L4 subtasks
22
+ - Root task (created separately)
23
+
24
+ 4. **FEAT-4837:** PHASE 2.8: Advanced Sampling Strategies ⚡ *NEW*
25
+ - 5 L3 components + 14 L4 subtasks
26
+ - Root task (created 2026-01-20)
27
+ - **Deferred from L2.7** - Advanced adaptive sampling (error-based, load-based, value-based, stratified)
28
+
29
+ 5. **FEAT-4792:** PHASE 3: Rails Integration (Weeks 11-14)
30
+ - 3 L3 components + 6 L4 subtasks
31
+ - Root task (created separately)
32
+
33
+ 6. **FEAT-4805:** PHASE 4: Production Hardening (Weeks 15-20)
34
+ - 4 L3 components + 8 L4 subtasks
35
+ - Root task (created separately)
36
+
37
+ 7. **FEAT-4822:** PHASE 5: Scale & Optimization (Weeks 21-26)
38
+ - 4 L3 components + 8 L4 subtasks
39
+ - Root task (created separately)
40
+
41
+ ---
42
+
43
+ ## 📈 STATISTICS
44
+
45
+ **Total Created:**
46
+ - **7 root tasks** (phases) - *+1 added 2026-01-20*
47
+ - **29 L3 components** (main subtasks) - *+5 from Phase 2.8*
48
+ - **87 L4 subtasks** (detailed tasks) - *+14 from Phase 2.8*
49
+ - **~120+ L5-L6 verification tasks** (in task descriptions)
50
+
51
+ **Timeline:**
52
+ - Week -1: Phase 0 (Research & Setup)
53
+ - Weeks 1-4: Phase 1 (Foundation)
54
+ - Weeks 5-10: Phase 2 (Core Features)
55
+ - Weeks 11-14: Phase 3 (Rails Integration)
56
+ - Weeks 15-20: Phase 4 (Production Hardening)
57
+ - Weeks 21-26: Phase 5 (Scale & Optimization)
58
+ - **Total Duration:** 23-27 weeks
59
+
60
+ **Team:**
61
+ - Peak parallelization: 6 developers simultaneously
62
+ - Total estimate: ~1000+ person-hours
63
+
64
+ ---
65
+
66
+ ## ✅ WHAT'S INCLUDED IN THE PLAN
67
+
68
+ ### **Phase 0: Gem Setup** ✅
69
+ - Best practices research (Devise, Sidekiq, Puma, Dry-rb, Yabeda, Sentry)
70
+ - Project skeleton setup (Zeitwerk, CI/CD, Docker Compose)
71
+ - Team alignment
72
+
73
+ ### **Phase 1: Foundation** ✅
74
+ - Event::Base (zero-allocation, schema DSL, presets)
75
+ - Buffers (RingBuffer, AdaptiveBuffer, RequestScoped)
76
+ - Middleware Pipeline (zones, ordering)
77
+
78
+ ### **Phase 2: Core Features** ✅
79
+ - PII Filtering & Security (3-tier, audit pipeline)
80
+ - Adapter Architecture (6 adapters: Stdout, File, Loki, Sentry, Elasticsearch, InMemory)
81
+ - Metrics & Yabeda (pattern-based, cardinality protection)
82
+ - Sampling & Cost Optimization (Basic - L2.7)
83
+
84
+ ### **Phase 2.8: Advanced Sampling Strategies** ⚡ *NEW* ⏳
85
+ - Error-Based Adaptive Sampling (↑ rate during incidents)
86
+ - Load-Based Adaptive Sampling (↓ rate during overload)
87
+ - Value-Based Sampling (↑ rate for high-value events)
88
+ - Stratified Sampling for SLO Accuracy (C11)
89
+ - **Status:** Planned (2026-01-20), awaiting approval to start
90
+ - **Depends On:** L2.7 (Basic Sampling - completed)
91
+
92
+ ### **Phase 3: Rails Integration** ✅
93
+ - Railtie & Auto-Configuration
94
+ - Rails Instrumentation (unidirectional ASN → E11y)
95
+ - Sidekiq/ActiveJob Integration (hybrid tracing)
96
+ - Rails.logger Migration
97
+
98
+ ### **Phase 4: Production Hardening** ✅
99
+ - Reliability & Error Handling (DLQ, rate limiting, retry)
100
+ - OpenTelemetry Integration (logs, baggage PII protection)
101
+ - Event Versioning & Schema Migrations
102
+ - SLO Tracking & Self-Monitoring
103
+
104
+ ### **Phase 5: Scale & Optimization** ✅
105
+ - High Cardinality Protection (dynamic actions)
106
+ - Tiered Storage Migration (hot/warm/cold)
107
+ - Performance Optimization (1K → 10K → 100K events/sec)
108
+ - Documentation & Gem Release
109
+
110
+ ---
111
+
112
+ ## 🎯 KEY ACHIEVEMENTS
113
+
114
+ ✅ **All ADRs covered** (ADR-001 to ADR-016)
115
+ ✅ **All Use Cases covered** (UC-001 to UC-022)
116
+ ✅ **All critical resolutions** (C01 to C20)
117
+ ✅ **Detailed DoD for each task**
118
+ ✅ **[P] markers for parallelization**
119
+ ✅ **Quality Gates for milestone tasks**
120
+ ✅ **Verification steps (L6) in descriptions**
121
+
122
+ ---
123
+
124
+ ## 🚀 NEXT STEPS
125
+
126
+ ### **Current Status (2026-01-20):**
127
+ - ✅ **L2.7 (Basic Sampling)** - COMPLETED
128
+ - ⏳ **FEAT-4837 (Phase 2.8)** - AWAITING APPROVAL
129
+ - Plan created for advanced sampling strategies
130
+ - Need human approval to start execution
131
+
132
+ ### **To Continue:**
133
+ 1. **Return to main plan:** Continue with original Phase 2 tasks (L2.8+)
134
+ 2. **Approve Phase 2.8:** Review and approve FEAT-4837 when ready to implement advanced sampling
135
+ 3. **Track progress:** Monitor via TeamTab dashboard
136
+
137
+ ---
138
+
139
+ ## 📝 REFERENCES
140
+
141
+ - **Main Plan:** `docs/IMPLEMENTATION_PLAN.md`
142
+ - **Architecture:** `docs/IMPLEMENTATION_PLAN_ARCHITECTURE.md`
143
+ - **ADRs:** `docs/ADR-001-*.md` to `docs/ADR-016-*.md`
144
+ - **Use Cases:** `docs/use_cases/UC-001-*.md` to `docs/use_cases/UC-022-*.md`
145
+
146
+ ---
147
+
148
+ **🎉 Plan is ready for execution! All 6 phases created with detailed structure, DoD, and clear dependencies.**