e11y 0.2.0 → 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (230) hide show
  1. checksums.yaml +4 -4
  2. data/.rubocop.yml +130 -10
  3. data/CHANGELOG.md +56 -1
  4. data/CLAUDE.md +168 -0
  5. data/CONTRIBUTING.md +640 -0
  6. data/README.md +134 -702
  7. data/RELEASE.md +18 -3
  8. data/Rakefile +108 -29
  9. data/config/README.md +1 -1
  10. data/config/loki-local-config.yaml +12 -0
  11. data/config/otel-collector-config.yaml +44 -0
  12. data/cucumber.yml +1 -0
  13. data/docker-compose.yml +18 -2
  14. data/docs/ADAPTERS.md +76 -0
  15. data/docs/ADAPTIVE_SAMPLING.md +59 -0
  16. data/docs/COMPARISON.md +104 -0
  17. data/docs/CONFIGURATION.md +52 -0
  18. data/docs/DISTRIBUTED_TRACING.md +44 -0
  19. data/docs/LIMITATIONS.md +13 -0
  20. data/docs/METRICS_DSL.md +84 -0
  21. data/docs/PERFORMANCE.md +60 -0
  22. data/docs/PII_FILTERING.md +40 -0
  23. data/docs/PRESETS.md +65 -0
  24. data/docs/QUICK-START.md +546 -587
  25. data/docs/RAILS_INTEGRATION.md +29 -0
  26. data/docs/SCHEMA_VALIDATION.md +63 -0
  27. data/docs/SLO-PROMQL-ALERTS.md +161 -0
  28. data/docs/TESTING.md +69 -0
  29. data/docs/{ADR-001-architecture.md → architecture/ADR-001-architecture.md} +35 -64
  30. data/docs/{ADR-002-metrics-yabeda.md → architecture/ADR-002-metrics-yabeda.md} +62 -236
  31. data/docs/{ADR-003-slo-observability.md → architecture/ADR-003-slo-observability.md} +27 -466
  32. data/docs/{ADR-004-adapter-architecture.md → architecture/ADR-004-adapter-architecture.md} +163 -146
  33. data/docs/{ADR-005-tracing-context.md → architecture/ADR-005-tracing-context.md} +10 -9
  34. data/docs/{ADR-006-security-compliance.md → architecture/ADR-006-security-compliance.md} +184 -191
  35. data/docs/{ADR-007-opentelemetry-integration.md → architecture/ADR-007-opentelemetry-integration.md} +3 -21
  36. data/docs/{ADR-008-rails-integration.md → architecture/ADR-008-rails-integration.md} +209 -339
  37. data/docs/{ADR-009-cost-optimization.md → architecture/ADR-009-cost-optimization.md} +45 -54
  38. data/docs/architecture/ADR-010-developer-experience.md +522 -0
  39. data/docs/{ADR-011-testing-strategy.md → architecture/ADR-011-testing-strategy.md} +41 -83
  40. data/docs/{ADR-013-reliability-error-handling.md → architecture/ADR-013-reliability-error-handling.md} +37 -12
  41. data/docs/{ADR-014-event-driven-slo.md → architecture/ADR-014-event-driven-slo.md} +12 -24
  42. data/docs/{ADR-015-middleware-order.md → architecture/ADR-015-middleware-order.md} +23 -41
  43. data/docs/{ADR-016-self-monitoring-slo.md → architecture/ADR-016-self-monitoring-slo.md} +52 -349
  44. data/docs/{ADR-017-multi-rails-compatibility.md → architecture/ADR-017-multi-rails-compatibility.md} +4 -11
  45. data/docs/architecture/ADR-018-memory-optimization.md +366 -0
  46. data/docs/{ADR-INDEX.md → architecture/ADR-INDEX.md} +11 -6
  47. data/docs/{00-ICP-AND-TIMELINE.md → prd/00-ICP-AND-TIMELINE.md} +6 -6
  48. data/docs/{01-SCALE-REQUIREMENTS.md → prd/01-SCALE-REQUIREMENTS.md} +6 -6
  49. data/docs/prd/01-overview-vision.md +19 -14
  50. data/docs/use_cases/README.md +22 -23
  51. data/docs/use_cases/UC-001-request-scoped-debug-buffering.md +50 -44
  52. data/docs/use_cases/UC-002-business-event-tracking.md +26 -95
  53. data/docs/use_cases/UC-003-event-metrics.md +66 -0
  54. data/docs/use_cases/UC-004-zero-config-slo-tracking.md +42 -101
  55. data/docs/use_cases/UC-005-sentry-integration.md +13 -15
  56. data/docs/use_cases/UC-006-trace-context-management.md +30 -28
  57. data/docs/use_cases/UC-007-pii-filtering.md +35 -87
  58. data/docs/use_cases/UC-008-opentelemetry-integration.md +51 -89
  59. data/docs/use_cases/UC-009-multi-service-tracing.md +4 -4
  60. data/docs/use_cases/UC-010-background-job-tracking.md +5 -5
  61. data/docs/use_cases/UC-011-rate-limiting.md +95 -168
  62. data/docs/use_cases/UC-012-audit-trail.md +21 -46
  63. data/docs/use_cases/UC-013-high-cardinality-protection.md +29 -167
  64. data/docs/use_cases/UC-014-adaptive-sampling.md +2 -2
  65. data/docs/use_cases/UC-015-cost-optimization.md +46 -99
  66. data/docs/use_cases/UC-016-rails-logger-migration.md +39 -213
  67. data/docs/use_cases/UC-017-local-development.md +203 -777
  68. data/docs/use_cases/UC-018-testing-events.md +3 -3
  69. data/docs/use_cases/UC-019-retention-based-routing.md +53 -106
  70. data/docs/use_cases/UC-020-event-versioning.md +8 -9
  71. data/docs/use_cases/UC-021-error-handling-retry-dlq.md +18 -22
  72. data/docs/use_cases/UC-022-event-registry.md +15 -21
  73. data/docs/use_cases/backlog.md +119 -87
  74. data/e11y.gemspec +2 -2
  75. data/gems/e11y-devtools/README.md +136 -0
  76. data/gems/e11y-devtools/config/routes.rb +8 -0
  77. data/gems/e11y-devtools/e11y-devtools.gemspec +25 -0
  78. data/gems/e11y-devtools/exe/e11y +34 -0
  79. data/gems/e11y-devtools/lib/e11y/devtools/mcp/server.rb +96 -0
  80. data/gems/e11y-devtools/lib/e11y/devtools/mcp/tool_base.rb +25 -0
  81. data/gems/e11y-devtools/lib/e11y/devtools/mcp/tools/clear.rb +31 -0
  82. data/gems/e11y-devtools/lib/e11y/devtools/mcp/tools/errors.rb +35 -0
  83. data/gems/e11y-devtools/lib/e11y/devtools/mcp/tools/event_detail.rb +33 -0
  84. data/gems/e11y-devtools/lib/e11y/devtools/mcp/tools/events_by_trace.rb +33 -0
  85. data/gems/e11y-devtools/lib/e11y/devtools/mcp/tools/interactions.rb +40 -0
  86. data/gems/e11y-devtools/lib/e11y/devtools/mcp/tools/recent_events.rb +34 -0
  87. data/gems/e11y-devtools/lib/e11y/devtools/mcp/tools/search.rb +34 -0
  88. data/gems/e11y-devtools/lib/e11y/devtools/mcp/tools/stats.rb +30 -0
  89. data/gems/e11y-devtools/lib/e11y/devtools/overlay/assets/overlay.js +115 -0
  90. data/gems/e11y-devtools/lib/e11y/devtools/overlay/controller.rb +54 -0
  91. data/gems/e11y-devtools/lib/e11y/devtools/overlay/engine.rb +26 -0
  92. data/gems/e11y-devtools/lib/e11y/devtools/overlay/middleware.rb +80 -0
  93. data/gems/e11y-devtools/lib/e11y/devtools/overlay/rails_controller.rb +42 -0
  94. data/gems/e11y-devtools/lib/e11y/devtools/tui/app.rb +262 -0
  95. data/gems/e11y-devtools/lib/e11y/devtools/tui/grouping.rb +66 -0
  96. data/gems/e11y-devtools/lib/e11y/devtools/tui/widgets/event_detail.rb +62 -0
  97. data/gems/e11y-devtools/lib/e11y/devtools/tui/widgets/event_list.rb +70 -0
  98. data/gems/e11y-devtools/lib/e11y/devtools/tui/widgets/interaction_list.rb +47 -0
  99. data/gems/e11y-devtools/lib/e11y/devtools/version.rb +8 -0
  100. data/gems/e11y-devtools/lib/e11y/devtools.rb +13 -0
  101. data/gems/e11y-devtools/spec/e11y/devtools/mcp/tools_spec.rb +107 -0
  102. data/gems/e11y-devtools/spec/e11y/devtools/overlay/controller_spec.rb +58 -0
  103. data/gems/e11y-devtools/spec/e11y/devtools/overlay/middleware_spec.rb +46 -0
  104. data/gems/e11y-devtools/spec/e11y/devtools/tui/app_spec.rb +85 -0
  105. data/gems/e11y-devtools/spec/e11y/devtools/tui/grouping_spec.rb +64 -0
  106. data/gems/e11y-devtools/spec/spec_helper.rb +5 -0
  107. data/gems/e11y-devtools/spec/tui/widgets/event_list_spec.rb +44 -0
  108. data/gems/e11y-devtools/spec/tui/widgets/interaction_list_spec.rb +62 -0
  109. data/lib/e11y/adapters/audit_encrypted.rb +53 -11
  110. data/lib/e11y/adapters/base.rb +33 -34
  111. data/lib/e11y/adapters/dev_log/file_store.rb +143 -0
  112. data/lib/e11y/adapters/dev_log/query.rb +219 -0
  113. data/lib/e11y/adapters/dev_log.rb +118 -0
  114. data/lib/e11y/adapters/file.rb +3 -6
  115. data/lib/e11y/adapters/in_memory.rb +52 -5
  116. data/lib/e11y/adapters/in_memory_test.rb +29 -0
  117. data/lib/e11y/adapters/loki.rb +58 -23
  118. data/lib/e11y/adapters/null.rb +82 -0
  119. data/lib/e11y/adapters/opentelemetry_collector.rb +183 -0
  120. data/lib/e11y/adapters/otel_logs.rb +136 -23
  121. data/lib/e11y/adapters/sentry.rb +4 -7
  122. data/lib/e11y/adapters/stdout.rb +73 -7
  123. data/lib/e11y/adapters/yabeda.rb +153 -29
  124. data/lib/e11y/buffers/adaptive_buffer.rb +3 -17
  125. data/lib/e11y/buffers/{request_scoped_buffer.rb → ephemeral_buffer.rb} +72 -58
  126. data/lib/e11y/buffers/ring_buffer.rb +3 -16
  127. data/lib/e11y/configuration.rb +272 -0
  128. data/lib/e11y/console.rb +10 -17
  129. data/lib/e11y/current.rb +53 -1
  130. data/lib/e11y/debug/pipeline_inspector.rb +96 -0
  131. data/lib/e11y/documentation/generator.rb +48 -0
  132. data/lib/e11y/event/base.rb +176 -82
  133. data/lib/e11y/event/value_sampling_config.rb +1 -5
  134. data/lib/e11y/events/rails/database/query.rb +1 -4
  135. data/lib/e11y/events/rails/job/failed.rb +2 -0
  136. data/lib/e11y/instruments/active_job.rb +46 -12
  137. data/lib/e11y/instruments/rails_instrumentation.rb +49 -24
  138. data/lib/e11y/instruments/sidekiq.rb +137 -31
  139. data/lib/e11y/linters/base.rb +11 -0
  140. data/lib/e11y/linters/pii/pii_declaration_linter.rb +120 -0
  141. data/lib/e11y/linters/slo/config_consistency_linter.rb +76 -0
  142. data/lib/e11y/linters/slo/explicit_declaration_linter.rb +36 -0
  143. data/lib/e11y/linters/slo/slo_status_from_linter.rb +41 -0
  144. data/lib/e11y/logger/bridge.rb +26 -7
  145. data/lib/e11y/metrics/cardinality_protection.rb +10 -15
  146. data/lib/e11y/metrics/cardinality_tracker.rb +16 -6
  147. data/lib/e11y/metrics/registry.rb +3 -5
  148. data/lib/e11y/metrics/test_backend.rb +62 -0
  149. data/lib/e11y/metrics.rb +56 -10
  150. data/lib/e11y/middleware/adapter_resolver.rb +40 -0
  151. data/lib/e11y/middleware/audit_signing.rb +43 -6
  152. data/lib/e11y/middleware/baggage_protection.rb +75 -0
  153. data/lib/e11y/middleware/dev_log_source.rb +24 -0
  154. data/lib/e11y/middleware/event_slo.rb +23 -9
  155. data/lib/e11y/middleware/otel_span.rb +23 -0
  156. data/lib/e11y/middleware/pii_filter.rb +104 -75
  157. data/lib/e11y/middleware/rate_limiting.rb +54 -27
  158. data/lib/e11y/middleware/request.rb +70 -23
  159. data/lib/e11y/middleware/routing.rb +78 -21
  160. data/lib/e11y/middleware/sampling.rb +66 -17
  161. data/lib/e11y/middleware/self_monitoring_emit.rb +39 -0
  162. data/lib/e11y/middleware/trace_context.rb +45 -10
  163. data/lib/e11y/middleware/track_latency.rb +34 -0
  164. data/lib/e11y/middleware/validation.rb +7 -16
  165. data/lib/e11y/middleware/versioning.rb +26 -22
  166. data/lib/e11y/opentelemetry/semantic_conventions.rb +109 -0
  167. data/lib/e11y/opentelemetry/span_creator.rb +142 -0
  168. data/lib/e11y/pii/patterns.rb +12 -1
  169. data/lib/e11y/pipeline/builder.rb +1 -1
  170. data/lib/e11y/presets/audit_event.rb +13 -2
  171. data/lib/e11y/railtie.rb +52 -15
  172. data/lib/e11y/registry.rb +306 -0
  173. data/lib/e11y/reliability/circuit_breaker.rb +19 -21
  174. data/lib/e11y/reliability/dlq/base.rb +71 -0
  175. data/lib/e11y/reliability/dlq/file_adapter.rb +301 -0
  176. data/lib/e11y/reliability/dlq/file_storage.rb +63 -34
  177. data/lib/e11y/reliability/dlq/filter.rb +37 -54
  178. data/lib/e11y/reliability/retry_handler.rb +26 -29
  179. data/lib/e11y/reliability/retry_rate_limiter.rb +3 -11
  180. data/lib/e11y/sampling/error_spike_detector.rb +0 -2
  181. data/lib/e11y/sampling/load_monitor.rb +5 -9
  182. data/lib/e11y/sampling/stratified_tracker.rb +18 -0
  183. data/lib/e11y/self_monitoring/buffer_monitor.rb +2 -0
  184. data/lib/e11y/self_monitoring/performance_monitor.rb +19 -61
  185. data/lib/e11y/self_monitoring/reliability_monitor.rb +4 -74
  186. data/lib/e11y/slo/config_loader.rb +40 -0
  187. data/lib/e11y/slo/config_validator.rb +58 -0
  188. data/lib/e11y/slo/dashboard_generator.rb +122 -0
  189. data/lib/e11y/slo/event_driven.rb +8 -0
  190. data/lib/e11y/slo/tracker.rb +31 -4
  191. data/lib/e11y/testing/have_tracked_event_matcher.rb +190 -0
  192. data/lib/e11y/testing/rspec_matchers.rb +21 -0
  193. data/lib/e11y/testing/snapshot_matcher.rb +86 -0
  194. data/lib/e11y/trace_context/sampler.rb +35 -0
  195. data/lib/e11y/tracing/faraday_middleware.rb +31 -0
  196. data/lib/e11y/tracing/net_http_patch.rb +33 -0
  197. data/lib/e11y/tracing/propagator.rb +116 -0
  198. data/lib/e11y/tracing.rb +47 -0
  199. data/lib/e11y/version.rb +1 -1
  200. data/lib/e11y/versioning/version_extractor.rb +32 -0
  201. data/lib/e11y.rb +141 -265
  202. data/lib/generators/e11y/event/event_generator.rb +22 -0
  203. data/lib/generators/e11y/event/templates/event.rb.tt +16 -0
  204. data/lib/generators/e11y/grafana_dashboard/grafana_dashboard_generator.rb +30 -0
  205. data/lib/generators/e11y/grafana_dashboard/templates/e11y_dashboard.json +81 -0
  206. data/lib/generators/e11y/install/install_generator.rb +34 -0
  207. data/lib/generators/e11y/install/templates/e11y.rb +239 -0
  208. data/lib/generators/e11y/prometheus_alerts/prometheus_alerts_generator.rb +29 -0
  209. data/lib/generators/e11y/prometheus_alerts/templates/e11y_alerts.yml +28 -0
  210. data/lib/tasks/e11y_docs.rake +30 -0
  211. data/lib/tasks/e11y_events.rake +71 -0
  212. data/lib/tasks/e11y_lint.rake +91 -0
  213. data/lib/tasks/e11y_slo.rake +29 -0
  214. metadata +129 -39
  215. data/docs/ADR-010-developer-experience.md +0 -2166
  216. data/docs/API-REFERENCE-L28.md +0 -914
  217. data/docs/COMPREHENSIVE-CONFIGURATION.md +0 -2366
  218. data/docs/CONTRIBUTING.md +0 -312
  219. data/docs/IMPLEMENTATION_NOTES.md +0 -2804
  220. data/docs/IMPLEMENTATION_PLAN.md +0 -1971
  221. data/docs/IMPLEMENTATION_PLAN_ARCHITECTURE.md +0 -586
  222. data/docs/PLAN.md +0 -148
  223. data/docs/README.md +0 -296
  224. data/docs/design/00-memory-optimization.md +0 -593
  225. data/docs/guides/MIGRATION-L27-L28.md +0 -692
  226. data/docs/guides/PERFORMANCE-BENCHMARKS.md +0 -434
  227. data/docs/guides/README.md +0 -44
  228. data/docs/use_cases/UC-003-pattern-based-metrics.md +0 -1627
  229. data/lib/e11y/adapters/registry.rb +0 -141
  230. /data/docs/{ADR-012-event-evolution.md → architecture/ADR-012-event-evolution.md} +0 -0
@@ -0,0 +1,366 @@
1
+ # ADR-018: Memory Optimization Strategy (Zero-Allocation Pattern)
2
+
3
+ **Status:** Accepted
4
+ **Date:** January 12, 2026
5
+ **Covers:** Event tracking memory efficiency, GC pressure reduction, performance targets
6
+ **Depends On:** ADR-001 (Architecture), ADR-004 (Adapters)
7
+
8
+ ---
9
+
10
+ ## 📋 Table of Contents
11
+
12
+ 1. [Context & Problem](#1-context--problem)
13
+ 2. [Decision](#2-decision)
14
+ 3. [Architecture](#3-architecture)
15
+ 4. [Implementation](#4-implementation)
16
+ - 4.1. Event Class (Zero-Allocation Design)
17
+ - 4.2. Collector (Hash-Based Processing)
18
+ - 4.3. Buffer (Hash-Based Storage)
19
+ - 4.4. Adapters (Hash-Based Serialization)
20
+ 5. [Performance Comparison](#5-performance-comparison)
21
+ 6. [Additional Optimizations](#6-additional-optimizations)
22
+ 7. [Testing Memory Efficiency](#7-testing-memory-efficiency)
23
+ 8. [Trade-offs](#8-trade-offs)
24
+ 9. [See Also](#9-see-also)
25
+
26
+ ---
27
+
28
+ ## 1. Context & Problem
29
+
30
+ ### 1.1. Problem Statement
31
+
32
+ **Naive Implementation (Bad):**
33
+
34
+ ```ruby
35
+ class Events::OrderPaid < E11y::Event
36
+ def self.track(**attributes)
37
+ event = new(attributes) # ← Allocates instance object
38
+ E11y::Collector.collect(event)
39
+ end
40
+ end
41
+
42
+ # Result: 10,000 events/sec = 10,000 object allocations/sec
43
+ # Memory pressure → GC overhead → latency spikes
44
+ ```
45
+
46
+ **Memory Impact:**
47
+ - Ruby object: ~40 bytes base
48
+ - Instance variables: ~8 bytes each
49
+ - Event payload hash: ~200-500 bytes
50
+ - **Total per event: ~300-600 bytes**
51
+ - **10k events/sec = 3-6 MB/sec allocation rate**
52
+ - **GC frequency: every 2-3 seconds**
53
+
54
+ ### 1.2. Key Insight
55
+
56
+ > **Events are immutable data** — don't need object identity, just data structure.
57
+
58
+ ---
59
+
60
+ ## 2. Decision
61
+
62
+ **Adopt Class-Method Pipeline (Zero Instance Allocation):**
63
+
64
+ - Events are represented as **hashes**, not object instances
65
+ - All processing via **class methods** (`Event.track(...)`), never `new()`
66
+ - Pipeline passes **hash through** — no wrapping, no object creation
67
+ - Collector, Buffer, Adapters operate on **hash data** exclusively
68
+
69
+ ---
70
+
71
+ ## 3. Architecture
72
+
73
+ ```
74
+ Events::OrderPaid.track(...)
75
+
76
+ [Class Method] Validate attributes
77
+
78
+ [Class Method] Build event hash (reusable structure)
79
+
80
+ [Class Method] Enrich context
81
+
82
+ [Class Method] Pass to collector (NO INSTANCE CREATED)
83
+
84
+ E11y::Collector.collect(event_hash)
85
+ ```
86
+
87
+ ---
88
+
89
+ ## 4. Implementation
90
+
91
+ ### 4.1. Event Class (Zero-Allocation Design)
92
+
93
+ ```ruby
94
+ # lib/e11y/event.rb
95
+ module E11y
96
+ class Event
97
+ class << self
98
+ def track(**attributes, &block)
99
+ return if filtered_by_severity?
100
+ validate_attributes!(attributes)
101
+ event_data = build_event_data(attributes, &block)
102
+ E11y::Collector.collect(event_data)
103
+ end
104
+
105
+ private
106
+
107
+ def build_event_data(attributes, &block)
108
+ event_data = {
109
+ event_class: name,
110
+ event_name: event_name,
111
+ severity: default_severity,
112
+ timestamp: Time.now,
113
+ payload: attributes.dup,
114
+ context: {},
115
+ duration_ms: nil,
116
+ trace_id: nil,
117
+ event_id: nil
118
+ }
119
+
120
+ if block
121
+ start = Process.clock_gettime(Process::CLOCK_MONOTONIC, :millisecond)
122
+ block.call
123
+ event_data[:duration_ms] = Process.clock_gettime(Process::CLOCK_MONOTONIC, :millisecond) - start
124
+ end
125
+
126
+ event_data
127
+ end
128
+ end
129
+ end
130
+ end
131
+ ```
132
+
133
+ ### 4.2. Collector (Hash-Based Processing)
134
+
135
+ ```ruby
136
+ # lib/e11y/collector.rb
137
+ module E11y
138
+ class Collector
139
+ class << self
140
+ def collect(event_data)
141
+ enrich_context!(event_data)
142
+ event_data[:event_id] = generate_event_id
143
+ process!(event_data)
144
+
145
+ if request_scoped? && event_data[:severity] == :debug
146
+ E11y::RequestScope.buffer_event(event_data)
147
+ else
148
+ send_to_adapters(event_data)
149
+ end
150
+ end
151
+
152
+ private
153
+
154
+ def enrich_context!(event_data)
155
+ event_data[:context].merge!(E11y.config.global_context)
156
+ event_data[:trace_id] = E11y::TraceId.extract
157
+ end
158
+ end
159
+ end
160
+ end
161
+ ```
162
+
163
+ ### 4.3. Buffer (Hash-Based Storage)
164
+
165
+ ```ruby
166
+ # lib/e11y/buffer/ring_buffer.rb
167
+ module E11y
168
+ module Buffer
169
+ class RingBuffer
170
+ def push(event_data)
171
+ return false if full?
172
+ pos = @write_pos.value
173
+ @buffer[pos] = event_data # Store hash directly (no wrapping)
174
+ @write_pos.value = (pos + 1) % @capacity
175
+ @size.increment
176
+ true
177
+ end
178
+
179
+ def pop_batch(max_size = 500)
180
+ batch = []
181
+ while batch.size < max_size && !empty?
182
+ batch << pop if event_data = pop
183
+ end
184
+ batch
185
+ end
186
+ end
187
+ end
188
+ end
189
+ ```
190
+
191
+ ### 4.4. Adapters (Hash-Based Serialization)
192
+
193
+ ```ruby
194
+ # lib/e11y/adapters/loki_adapter.rb
195
+ module E11y
196
+ module Adapters
197
+ class LokiAdapter < Base
198
+ def send_batch(events)
199
+ # events = array of hashes (not instances!)
200
+ streams = events.group_by { |e| extract_labels(e) }.map do |labels, events|
201
+ {
202
+ stream: @default_labels.merge(labels),
203
+ values: events.map { |e| [timestamp_ns(e), format_event(e)] }
204
+ }
205
+ end
206
+ @client.post('/loki/api/v1/push', json: { streams: streams })
207
+ end
208
+ end
209
+ end
210
+ end
211
+ ```
212
+
213
+ ---
214
+
215
+ ## 5. Performance Comparison
216
+
217
+ ### 5.1. Memory Allocation
218
+
219
+ | Approach | Allocations/event | Memory/event | GC Pressure |
220
+ |----------|-------------------|--------------|-------------|
221
+ | **Instance-based** | 1 object + 1 hash | ~400 bytes | High |
222
+ | **Hash-based** | 1 hash (reused structure) | ~200 bytes | Low |
223
+ | **Improvement** | 50% fewer allocations | 50% less memory | 3x less GC |
224
+
225
+ ### 5.2. Benchmark Results
226
+
227
+ ```ruby
228
+ # Instance-based (naive)
229
+ Benchmark.memory do |x|
230
+ x.report('instance-based') do
231
+ 10_000.times { Events::OrderPaid.new(order_id: '123', amount: 99.99) }
232
+ end
233
+ end
234
+ # Result: 10,000 objects + 10,000 hashes = 4 MB allocated
235
+
236
+ # Hash-based (optimized)
237
+ Benchmark.memory do |x|
238
+ x.report('hash-based') do
239
+ 10_000.times { Events::OrderPaid.track(order_id: '123', amount: 99.99) }
240
+ end
241
+ end
242
+ # Result: 10,000 hashes = 2 MB allocated
243
+ ```
244
+
245
+ ### 5.3. Performance Target Achievement
246
+
247
+ | Target | Hash-Based | Status |
248
+ |--------|------------|--------|
249
+ | <1ms p99 latency | 0.8ms | ✅ |
250
+ | 10k+ events/sec | 15k/sec | ✅ |
251
+ | <5% GC overhead | 3% | ✅ |
252
+
253
+ ---
254
+
255
+ ## 6. Additional Optimizations
256
+
257
+ ### 6.1. Symbol Reuse
258
+
259
+ ```ruby
260
+ # BAD: String allocations
261
+ event_data[:event_name] = 'order.paid' # New string each time
262
+
263
+ # GOOD: Cache symbols
264
+ def event_name
265
+ @event_name ||= name.demodulize.underscore.gsub('_', '.').to_sym
266
+ end
267
+ ```
268
+
269
+ ### 6.2. Hash Pre-Allocation
270
+
271
+ ```ruby
272
+ # GOOD: Pre-allocate with all keys (no reallocation)
273
+ event_data = {
274
+ event_class: nil,
275
+ event_name: nil,
276
+ severity: nil,
277
+ timestamp: nil,
278
+ payload: nil,
279
+ context: nil,
280
+ duration_ms: nil,
281
+ trace_id: nil,
282
+ event_id: nil
283
+ }
284
+ ```
285
+
286
+ ### 6.3. Lazy Serialization
287
+
288
+ ```ruby
289
+ # DON'T serialize until needed (in adapter, not in collector)
290
+
291
+ # BAD: Serialize in collector
292
+ def collect(event_data)
293
+ json = event_data.to_json # ← Too early! (string allocation)
294
+ send_to_adapters(json)
295
+ end
296
+
297
+ # GOOD: Serialize in adapter (just before sending)
298
+ def send_batch(events)
299
+ payload = events.map(&:to_json).join("\n")
300
+ @client.post(payload)
301
+ end
302
+ ```
303
+
304
+ ---
305
+
306
+ ## 7. Testing Memory Efficiency
307
+
308
+ ```ruby
309
+ # spec/performance/memory_spec.rb
310
+ RSpec.describe 'Memory Efficiency' do
311
+ it 'does not allocate event instances' do
312
+ before_count = ObjectSpace.count_objects[:T_OBJECT]
313
+ 1_000.times { Events::OrderPaid.track(order_id: '123', amount: 99.99) }
314
+ after_count = ObjectSpace.count_objects[:T_OBJECT]
315
+ expect(after_count - before_count).to be < 10
316
+ end
317
+
318
+ it 'allocates minimal memory per event' do
319
+ require 'memory_profiler'
320
+ report = MemoryProfiler.report do
321
+ 1_000.times { Events::OrderPaid.track(order_id: '123', amount: 99.99, currency: 'USD') }
322
+ end
323
+ expect(report.total_allocated_memsize).to be < 300_000 # 300 KB
324
+ end
325
+ end
326
+ ```
327
+
328
+ ---
329
+
330
+ ## 8. Trade-offs
331
+
332
+ ### 8.1. Pros ✅
333
+
334
+ 1. **50% less memory allocation** — fewer objects created
335
+ 2. **3x less GC pressure** — major latency improvement
336
+ 3. **Simpler serialization** — hash → JSON (no object marshaling)
337
+ 4. **Cache-friendly** — hash structure is contiguous in memory
338
+ 5. **Thread-safe** — immutable data passed around
339
+
340
+ ### 8.2. Cons ❌
341
+
342
+ 1. **No method delegation** — can't call `event.order_id`, must use `event[:payload][:order_id]`
343
+ 2. **No type safety** — hash can have any keys (validation at entry point compensates)
344
+ 3. **Less OOP** — functional style (hash pipeline) vs OOP (object methods)
345
+
346
+ ### 8.3. Decision Rationale
347
+
348
+ **Pros outweigh cons significantly:**
349
+ - Performance is critical (10k+ events/sec)
350
+ - Events are immutable data (no behavior needed)
351
+ - Validation at entry point ensures correctness
352
+ - Type safety via dry-struct schema at `track()` call
353
+
354
+ ---
355
+
356
+ ## 9. See Also
357
+
358
+ - **ADR-001: Architecture** — §5 Memory Optimization Strategy (summary), §8 Performance Requirements
359
+ - **ADR-004: Adapter Architecture** — Hash-based adapter contract
360
+ - **ADR-009: Cost Optimization** — Related performance strategies
361
+ - **docs/design/00-memory-optimization.md** — Original design document (superseded by this ADR)
362
+
363
+ ---
364
+
365
+ **Status:** ✅ Accepted
366
+ **Next Review:** After MVP implementation
@@ -15,22 +15,26 @@ This document provides an index of all architectural decisions made for the E11y
15
15
  | [ADR-007](ADR-007-opentelemetry-integration.md) | OpenTelemetry Integration | ✅ Accepted | 3 |
16
16
  | [ADR-008](ADR-008-rails-integration.md) | Rails Integration Strategy | ✅ Accepted | 3 |
17
17
  | [ADR-009](ADR-009-cost-optimization.md) | Cost Optimization Strategies | ✅ Accepted | 4 |
18
- | [ADR-010](ADR-010-developer-experience.md) | Developer Experience (DX) | ✅ Accepted | 5 |
18
+ | [ADR-010](ADR-010-developer-experience.md) | Developer Experience: DevLog adapter, TUI (ratatui_ruby), Browser Overlay, MCP Server (Hub-and-Spoke) | ✅ Accepted | 5 |
19
19
  | [ADR-011](ADR-011-testing-strategy.md) | Testing Strategy | ✅ Accepted | 5 |
20
20
  | [ADR-012](ADR-012-event-evolution.md) | Event Schema Evolution | ✅ Accepted | 1 |
21
21
  | [ADR-013](ADR-013-reliability-error-handling.md) | Reliability & Error Handling | ✅ Accepted | 4 |
22
22
  | [ADR-014](ADR-014-event-driven-slo.md) | Event-Driven SLO Tracking | ✅ Accepted | 3 |
23
23
  | [ADR-015](ADR-015-middleware-order.md) | Middleware Execution Order | ✅ Accepted | 2 |
24
24
  | [ADR-016](ADR-016-self-monitoring-slo.md) | Self-Monitoring SLO | ✅ Accepted | 4 |
25
+ | [ADR-017](ADR-017-multi-rails-compatibility.md) | Multi-Rails Compatibility | ✅ Accepted | 2 |
26
+ | [ADR-018](ADR-018-memory-optimization.md) | Memory Optimization (Zero-Allocation) | ✅ Accepted | 0 |
25
27
 
26
28
  ## 🎯 Key Decisions by Topic
27
29
 
28
30
  ### Architecture & Design
29
- - **ADR-001**: Core architecture principles, zero-allocation pattern, convention over configuration
31
+ - **ADR-001**: Core architecture principles, convention over configuration
30
32
  - **ADR-012**: Event schema evolution strategy with versioning
33
+ - **ADR-018**: Memory optimization (zero-allocation pattern, hash-based events)
31
34
 
32
35
  ### Performance & Scale
33
36
  - **ADR-001 §5**: Performance requirements (1K/10K/100K events/sec)
37
+ - **ADR-018**: Memory optimization (zero-allocation, hash-based events)
34
38
  - **ADR-009**: Cost optimization strategies (adaptive sampling, compression, tiered storage)
35
39
 
36
40
  ### Reliability & Operations
@@ -52,7 +56,7 @@ This document provides an index of all architectural decisions made for the E11y
52
56
  - **ADR-005**: Trace context propagation
53
57
 
54
58
  ### Developer Experience
55
- - **ADR-010**: Developer experience priorities (5-min setup, conventions)
59
+ - **ADR-010**: Hub-and-Spoke devtools JSONL DevLog adapter + TUI (ratatui_ruby) + Browser Overlay (Rails Engine + Shadow DOM badge) + MCP Server (8 tools, stdio/HTTP transport, AI integration)
56
60
  - **ADR-011**: Testing strategy (RSpec, integration tests, benchmarks)
57
61
  - **ADR-015**: Middleware execution order guarantees
58
62
 
@@ -87,9 +91,10 @@ Review:
87
91
 
88
92
  ### For Performance Tuning
89
93
  See:
90
- 1. [ADR-001 §5](ADR-001-architecture.md) - Performance requirements
91
- 2. [ADR-009](ADR-009-cost-optimization.md) - Optimization strategies
92
- 3. [docs/guides/performance-tuning.md](guides/performance-tuning.md) - Tuning guide
94
+ 1. [ADR-018](ADR-018-memory-optimization.md) - Zero-allocation pattern, memory efficiency
95
+ 2. [ADR-001 §5](ADR-001-architecture.md) - Performance requirements
96
+ 3. [ADR-009](ADR-009-cost-optimization.md) - Optimization strategies
97
+ 4. [docs/guides/performance-tuning.md](guides/performance-tuning.md) - Tuning guide
93
98
 
94
99
  ## 🔗 Related Documentation
95
100
 
@@ -39,7 +39,7 @@
39
39
  - High cardinality costs (Datadog, New Relic bills exploding)
40
40
 
41
41
  **What They Need:**
42
- - ✅ Pattern-based metrics (reduce boilerplate)
42
+ - ✅ Event-level metrics DSL (reduce boilerplate)
43
43
  - ✅ Cardinality protection (prevent cost explosions)
44
44
  - ✅ Multi-adapter support (integrate with existing stack)
45
45
  - ✅ Team-wide conventions (event schemas, PII filtering)
@@ -135,10 +135,10 @@
135
135
 
136
136
  ### Phase 2: Yabeda Integration (Weeks 9-12)
137
137
  **Target:** April 2025
138
- **Goal:** Pattern-based metrics automation
138
+ **Goal:** Event-level metrics automation
139
139
 
140
- **Week 9-10: Pattern Engine**
141
- - [ ] Pattern matching (`*`, `order.*`, `*.paid`)
140
+ **Week 9-10: Metrics DSL**
141
+ - [ ] Event-level `metrics do ... end` DSL
142
142
  - [ ] Label extraction from events
143
143
  - [ ] Counter metrics
144
144
  - [ ] Histogram metrics (with buckets)
@@ -261,7 +261,7 @@
261
261
  **Focus:** Feature-complete release candidate
262
262
 
263
263
  **New Features:**
264
- - ✅ Pattern-based metrics (Yabeda)
264
+ - ✅ Event-level metrics (Yabeda)
265
265
  - ✅ Cardinality protection
266
266
  - ✅ SLO tracking (zero-config)
267
267
  - ✅ OpenTelemetry integration
@@ -424,7 +424,7 @@
424
424
  - ✅ Rails-first design (vs OTel's language-agnostic complexity)
425
425
  - ✅ Zero-config SLO tracking (unique feature)
426
426
  - ✅ Request-scoped debug buffering (unique feature)
427
- - ✅ Pattern-based metrics (less boilerplate)
427
+ - ✅ Event-level metrics DSL (less boilerplate)
428
428
  - ✅ Cost optimization built-in (vs expensive SaaS)
429
429
 
430
430
  ---
@@ -66,11 +66,11 @@ E11y.configure do |config|
66
66
  worker_threads 2 # Multiple workers
67
67
  end
68
68
 
69
- # Adaptive sampling для cost control
69
+ # Adaptive sampling for cost control
70
70
  config.sampling do
71
71
  strategy :adaptive
72
72
  target_samples_per_second 200 # Cap at 200/sec
73
- min_rate 0.1 # Minimum 10% даже при высокой нагрузке
73
+ min_rate 0.1 # Minimum 10% even under high load
74
74
  end
75
75
  end
76
76
  ```
@@ -112,13 +112,13 @@ E11y.configure do |config|
112
112
  worker_threads 4 # Multiple workers
113
113
  end
114
114
 
115
- # Агрессивный sampling
115
+ # Aggressive sampling
116
116
  config.sampling do
117
117
  strategy :adaptive
118
118
  target_samples_per_second 1_000 # Cap at 1k/sec
119
- min_rate 0.01 # 1% минимум
119
+ min_rate 0.01 # 1% minimum
120
120
 
121
- # Tail-based sampling для критичных событий
121
+ # Tail-based sampling for critical events
122
122
  tail do
123
123
  enabled true
124
124
  sample_if do |events|
@@ -481,7 +481,7 @@ E11y.configure do |config|
481
481
  target_samples_per_second 1_000
482
482
  min_rate 0.01 # 1% minimum
483
483
 
484
- # Tail-based sampling для критичных событий
484
+ # Tail-based sampling for critical events
485
485
  tail do
486
486
  enabled true
487
487
  sample_if do |events|
@@ -9,15 +9,15 @@
9
9
 
10
10
  ## 📋 Executive Summary
11
11
 
12
- **E11y** (easy telemetry) - production-ready Ruby gem для структурированных бизнес-событий с уникальными killer features:
12
+ **E11y** (easy telemetry) - production-ready Ruby gem for structured business events with unique killer features:
13
13
 
14
- 1. **Request-scoped debug buffering** - debug events только при ошибках (89% reduction)
15
- 2. **Pattern-based metrics** - автоматические метрики без boilerplate
16
- 3. **Zero-config SLO tracking** - built-in monitoring одной строкой конфига
14
+ 1. **Request-scoped debug buffering** - debug events only on errors (89% reduction)
15
+ 2. **Event-level metrics DSL** - automatic metrics without boilerplate
16
+ 3. **Zero-config SLO tracking** - built-in monitoring with a single config line
17
17
 
18
18
  **Target Market:** Ruby/Rails teams (5-100 engineers)
19
- **Problem:** Observability сложна, дорога и перегружена шумом
20
- **Solution:** Простой, Rails-first gem с production-ready defaults
19
+ **Problem:** Observability is complex, expensive, and overloaded with noise
20
+ **Solution:** Simple, Rails-first gem with production-ready defaults
21
21
 
22
22
  ---
23
23
 
@@ -160,14 +160,19 @@ E11y.configure { |config| config.slo_tracking = true }
160
160
 
161
161
  #### 3. Automatic Metrics
162
162
 
163
- **Pattern-based metrics:**
163
+ **Event-level metrics DSL:**
164
164
  ```ruby
165
- # Define event once
166
- Events::OrderPaid.track(order_id: '123', amount: 99, currency: 'USD')
165
+ # Define event once with metrics
166
+ class Events::OrderPaid < E11y::Event::Base
167
+ schema { required(:order_id).filled(:string); required(:amount).filled(:float); optional(:currency).maybe(:string) }
168
+ metrics do
169
+ counter :orders_paid_total, tags: [:currency]
170
+ histogram :order_amount, value: :amount, tags: [:currency]
171
+ end
172
+ end
167
173
 
168
- # Get metrics automatically (configured via patterns)
169
- # - orders.paid.total{currency="USD"} = 1
170
- # - orders.paid.amount{currency="USD"} = 99
174
+ Events::OrderPaid.track(order_id: '123', amount: 99, currency: 'USD')
175
+ # orders_paid_total{currency="USD"} = 1, order_amount{currency="USD"} = 99
171
176
  ```
172
177
 
173
178
  **Result:** No boilerplate, no duplication, consistent.
@@ -243,7 +248,7 @@ end
243
248
  | **Setup complexity** | High (5+ pages) | Low (1 line) |
244
249
  | **Rails integration** | Manual | Automatic |
245
250
  | **Request-scoped buffering** | ❌ | ✅ |
246
- | **Pattern-based metrics** | ❌ | ✅ |
251
+ | **Event-level metrics DSL** | ❌ | ✅ |
247
252
  | **SLO tracking** | Manual setup | One-line config |
248
253
  | **Target audience** | Polyglot teams | Rails teams |
249
254
 
@@ -363,7 +368,7 @@ end
363
368
  ✅ <1ms p99 latency
364
369
 
365
370
  **v1.0 (Phase 5):**
366
- Pattern-based metrics (Yabeda)
371
+ Event-level metrics (Yabeda)
367
372
  ✅ Zero-config SLO tracking
368
373
  ✅ OpenTelemetry integration
369
374
  ✅ Cardinality protection
@@ -1,18 +1,18 @@
1
1
  # E11y Use Cases Documentation
2
2
 
3
- Эта папка содержит детальные use cases для различных сценариев использования E11y gem.
3
+ This folder contains detailed use cases for various E11y gem usage scenarios.
4
4
 
5
- ## 📁 Структура
5
+ ## 📁 Structure
6
6
 
7
7
  ### Core Use Cases
8
- - **[UC-001: Request-Scoped Debug Buffering](./UC-001-request-scoped-debug-buffering.md)** - Killer feature: debug events только при ошибках
9
- - **[UC-002: Business Event Tracking](./UC-002-business-event-tracking.md)** - Структурированные бизнес-события
10
- - **[UC-003: Pattern-Based Metrics](./UC-003-pattern-based-metrics.md)** - Автоматические метрики из событий
8
+ - **[UC-001: Request-Scoped Debug Buffering](./UC-001-request-scoped-debug-buffering.md)** - Killer feature: debug events only on errors
9
+ - **[UC-002: Business Event Tracking](./UC-002-business-event-tracking.md)** - Structured business events
10
+ - **[UC-003: Event Metrics](./UC-003-event-metrics.md)** - Metrics in event classes
11
11
  - **[UC-004: Zero-Config SLO Tracking](./UC-004-zero-config-slo-tracking.md)** - Built-in SLO monitoring
12
12
 
13
13
  ### Integration Use Cases
14
- - **[UC-005: Sentry Integration](./UC-005-sentry-integration.md)** - Error tracking с автоматическими breadcrumbs
15
- - **[UC-006: Trace Context Management](./UC-006-trace-context-management.md)** - Автоматическая корреляция через trace_id
14
+ - **[UC-005: Sentry Integration](./UC-005-sentry-integration.md)** - Error tracking with automatic breadcrumbs
15
+ - **[UC-006: Trace Context Management](./UC-006-trace-context-management.md)** - Automatic correlation via trace_id
16
16
  - **[UC-007: PII Filtering](./UC-007-pii-filtering.md)** - Rails-compatible PII filtering
17
17
  - **[UC-008: OpenTelemetry Integration](./UC-008-opentelemetry-integration.md)** - OTel compatibility
18
18
  - **[UC-009: Multi-Service Tracing](./UC-009-multi-service-tracing.md)** - Distributed tracing
@@ -34,7 +34,7 @@
34
34
  - **[UC-018: Testing Events](./UC-018-testing-events.md)** - Test strategies
35
35
  - **[UC-020: Event Versioning](./UC-020-event-versioning.md)** - Schema evolution & backward compatibility
36
36
  - **[UC-021: Error Handling & DLQ](./UC-021-error-handling-retry-dlq.md)** - Retry policy & dead letter queue
37
- - **[UC-022: Event Registry](./UC-022-event-registry.md)** - Event introspection & discovery
37
+ - **[UC-022: Event Registry](./UC-022-event-registry.md)** - Event discovery (find, event_classes, where)
38
38
 
39
39
  ## 🎯 Use Case Categories
40
40
 
@@ -42,15 +42,15 @@
42
42
 
43
43
  **Ruby/Rails Developers:**
44
44
  - UC-002 (Business Event Tracking)
45
- - UC-014 (Rails Logger Migration)
46
- - UC-015 (Local Development)
47
- - UC-016 (Testing Events)
45
+ - UC-016 (Rails Logger Migration)
46
+ - UC-017 (Local Development)
47
+ - UC-018 (Testing Events)
48
48
 
49
49
  **DevOps/SRE Engineers:**
50
50
  - UC-001 (Request-Scoped Debug Buffering)
51
51
  - UC-004 (Zero-Config SLO Tracking)
52
52
  - UC-008 (OpenTelemetry Integration)
53
- - UC-011 (Adaptive Sampling)
53
+ - UC-014 (Adaptive Sampling)
54
54
 
55
55
  **Security/Compliance Teams:**
56
56
  - UC-007 (PII Filtering)
@@ -60,7 +60,7 @@
60
60
  **Engineering Managers/CTOs:**
61
61
  - UC-015 (Cost Optimization)
62
62
  - UC-013 (High Cardinality Protection)
63
- - UC-003 (Pattern-Based Metrics)
63
+ - UC-003 (Event Metrics)
64
64
 
65
65
  ### By Complexity
66
66
 
@@ -72,7 +72,7 @@
72
72
 
73
73
  **Intermediate (15-60 min setup):**
74
74
  - UC-001 (Request-Scoped Debug Buffering)
75
- - UC-003 (Pattern-Based Metrics)
75
+ - UC-003 (Event Metrics)
76
76
  - UC-004 (Zero-Config SLO Tracking)
77
77
  - UC-006 (Trace Context Management)
78
78
  - UC-007 (PII Filtering)
@@ -89,9 +89,9 @@
89
89
 
90
90
  ### For New Users
91
91
  Start with:
92
- 1. UC-002 (Business Event Tracking) - понять основы
93
- 2. UC-015 (Local Development) - настроить локально
94
- 3. UC-004 (Zero-Config SLO Tracking) - получить метрики
92
+ 1. UC-002 (Business Event Tracking) - learn the basics
93
+ 2. UC-017 (Local Development) - set up locally
94
+ 3. UC-004 (Zero-Config SLO Tracking) - get metrics
95
95
 
96
96
  ### For Production Deployment
97
97
  Review:
@@ -103,15 +103,14 @@ Review:
103
103
  ### For Migration from Existing Tools
104
104
  See:
105
105
  1. UC-016 (Rails Logger Migration)
106
- 2. UC-008 (OpenTelemetry Integration) - если уже используете OTel
107
- 3. UC-009 (Multi-Service Tracing) - если microservices
106
+ 2. UC-008 (OpenTelemetry Integration) - if you already use OTel
107
+ 3. UC-009 (Multi-Service Tracing) - for microservices
108
108
 
109
109
  ## 🔗 Related Documentation
110
110
 
111
- - **[Quick Start Guide](../E11Y-QUICK-START.md)** - 5-minute setup
112
- - **[API Reference](../api/README.md)** - Detailed API docs
113
- - **[Architecture Overview](../architecture/README.md)** - System design
114
- - **[Configuration Guide](../configuration/README.md)** - All config options
111
+ - **[Quick Start Guide](../QUICK-START.md)** - 5-minute setup
112
+ - **[Architecture Overview](../architecture/ADR-INDEX.md)** - System design
113
+ - **[Configuration Guide](../CONFIGURATION.md)** - All config options
115
114
 
116
115
  ---
117
116