opentrace 0.8.0 → 0.13.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: fd043296872a0f5241acc4f8331bc2d6c6f7e39628e0501fc6e0480af7c423f9
4
- data.tar.gz: b37915c9ed52898d6607a668c264c573f8d2012d937ec4053712ff2c65fb741e
3
+ metadata.gz: efabcbbf59438deda1b76fcaa0953218bd2ab1144388e0abd20cef1c77d42582
4
+ data.tar.gz: 6df6eef94307d9ee5ab79869cd785abea16464dd7488fe7c52288a3f65f326a8
5
5
  SHA512:
6
- metadata.gz: a9567eefab80204781b112f43f915a755ad55327eef2255b7b9c655da9eb36e4eed55c6617e1711d00bbd74ec9dd1f88332b0f0c0c4e2e0562e75a1fdcb8a9fe
7
- data.tar.gz: 11bf43482a27798a4bcf20c7cb700a565692d2b863d3011feb403df97e039b2b75cd35e0e53f12d7b17a107b7d07ef1959c0539d9507397376bb8901afcb9346
6
+ metadata.gz: ac55ce5f272fea2dfa221b9a08283f0d70a534f09fe4119fece6a2e953231f863f346c95997f685f65f47ec4ffa9e48af7b5dba6445ca9e21d8ce162d980034a
7
+ data.tar.gz: 927293c59438670c43b58a486355bb227b988bad34a3532b234d538fc871ec541d62279668c2c495eacfc54d1ef5a7eef1c778238d30906b19356020452e1f15
data/README.md CHANGED
@@ -11,6 +11,7 @@ A thin, safe Ruby client that forwards structured application logs to an [OpenTr
11
11
 
12
12
  ## Features
13
13
 
14
+ ### Core
14
15
  - **Zero-risk integration** -- all errors swallowed, never raises to host app
15
16
  - **Async dispatch** -- logs are queued in-memory and sent via a background thread
16
17
  - **Batch sending** -- groups logs into configurable batches for efficient network usage
@@ -19,28 +20,59 @@ A thin, safe Ruby client that forwards structured application logs to an [OpenTr
19
20
  - **Works with any server** -- Puma (threads), Unicorn (forks), Passenger, and Falcon (fibers)
20
21
  - **Fork safe** -- detects forked worker processes and re-initializes cleanly
21
22
  - **Fiber safe** -- uses `Fiber[]` storage for correct request isolation in fiber-based servers
22
- - **Rails integration** -- auto-instruments controllers, SQL queries, ActiveJob, views, cache, and more
23
- - **Rack middleware** -- propagates `request_id` via fiber-local storage
24
- - **Logger wrapper** -- drop-in replacement that forwards to OpenTrace while keeping your original logger
25
- - **Rails 7.1+ BroadcastLogger** -- native support via `broadcast_to`
26
- - **TaggedLogging** -- preserves `ActiveSupport::TaggedLogging` tags in metadata
27
- - **Context support** -- attach global metadata to every log via Hash or Proc
28
- - **Business events** -- `OpenTrace.event` sends typed events (e.g. `payment.completed`) that bypass level filtering
29
23
  - **Level filtering** -- `min_level` threshold or `allowed_levels` list to control which severities are forwarded
30
24
  - **Auto-enrichment** -- every log includes `hostname`, `pid`, and `git_sha` automatically
31
- - **Exception helper** -- `OpenTrace.error` captures class, message, cleaned backtrace, and error fingerprint
32
25
  - **Runtime controls** -- enable/disable logging at runtime without restarting
33
26
  - **Graceful shutdown** -- pending logs are flushed automatically on process exit
34
- - **N+1 query detection** -- warns when a request exceeds 20 SQL queries
27
+ - **Adaptive sampling** -- graduated backpressure reduces overhead under load (configurable `sample_rate`)
28
+ - **Deferred payloads** -- request thread pushes frozen arrays; heavy work runs on background thread
29
+
30
+ ### Instrumentation
31
+ - **Custom instrumentation** -- `OpenTrace.trace("stripe.charge") { ... }` with nested spans and timing
32
+ - **Exception helper** -- `OpenTrace.error` captures class, message, cleaned backtrace, cause chain, and error fingerprint
33
+ - **Exception cause chaining** -- walks `exception.cause` up to 5 levels deep
34
+ - **Breadcrumbs** -- `OpenTrace.add_breadcrumb` records a trail of events attached to errors
35
+ - **Source code context** -- captures surrounding source lines at the error origin (opt-in)
36
+ - **Local variables capture** -- `OpenTrace.capture_binding(e, binding)` snapshots variables at crash point (opt-in)
37
+ - **Transaction naming** -- `OpenTrace.set_transaction_name` for custom grouping
38
+ - **Business events** -- `OpenTrace.event` sends typed events (e.g. `payment.completed`) that bypass level filtering
39
+ - **Context support** -- attach global metadata to every log via Hash or Proc
40
+
41
+ ### Rails Integration
42
+ - **Auto-instrumentation** -- controllers, SQL queries, ActiveJob, views, cache, deprecation warnings
43
+ - **Rack middleware** -- propagates `request_id` via fiber-local storage
35
44
  - **Per-request summary** -- one rich log per request with SQL, view, cache breakdown and timeline
45
+ - **N+1 query detection** -- warns when a request exceeds 20 SQL queries
46
+ - **Duplicate query detection** -- fingerprints SQL queries to find repeated patterns
47
+ - **SQL normalization** -- replaces literals with `?` for grouping; generates stable fingerprints
48
+ - **EXPLAIN plan capture** -- runs EXPLAIN on slow queries asynchronously on background thread (opt-in)
49
+ - **Log trace injection** -- injects `[trace_id=xxx request_id=yyy]` into Rails logger output (opt-in)
50
+ - **Session tracking** -- extracts session ID from rack session or cookies (opt-in)
51
+ - **Logger wrapper** -- drop-in replacement that forwards to OpenTrace while keeping your original logger
52
+ - **Rails 7.1+ BroadcastLogger** -- native support via `broadcast_to`
53
+ - **TaggedLogging** -- preserves `ActiveSupport::TaggedLogging` tags in metadata
36
54
  - **Error fingerprinting** -- stable fingerprint for grouping identical errors across requests
37
55
  - **Deprecation tracking** -- captures Rails deprecation warnings with callsite
56
+
57
+ ### Data Protection
58
+ - **PII scrubbing** -- automatic detection and redaction of emails, credit cards, SSNs, tokens, passwords (opt-in)
59
+ - **Lifecycle callbacks** -- `on_error`, `after_send`, `before_breadcrumb`, `before_send` hooks
60
+ - **Before-send filter** -- drop or modify payloads before delivery
61
+
62
+ ### Monitoring
38
63
  - **DB pool monitoring** -- background thread reports connection pool saturation (opt-in)
39
64
  - **Job queue depth** -- monitors Sidekiq, GoodJob, or SolidQueue queue sizes (opt-in)
40
65
  - **Memory delta tracking** -- snapshots process RSS before/after each request (opt-in)
41
66
  - **External HTTP tracking** -- captures outbound Net::HTTP calls with timing (opt-in)
42
- - **Version negotiation** -- startup compatibility check with capability-based feature detection
67
+ - **GC/Runtime metrics** -- periodic collection of GC stats, thread count, and process RSS (opt-in)
68
+
69
+ ### Delivery
43
70
  - **Distributed tracing** -- W3C Trace Context (`traceparent`) propagation across services with span IDs
71
+ - **Unix socket transport** -- 2-5x faster delivery for co-located servers with automatic HTTP fallback (opt-in)
72
+ - **Gzip compression** -- automatic payload compression for bandwidth reduction
73
+ - **Version negotiation** -- startup compatibility check with capability-based feature detection
74
+ - **Circuit breaker** -- stops sending when server is unreachable, resumes after cooldown
75
+ - **Exponential backoff** -- retries with jitter on server errors
44
76
 
45
77
  ## Installation
46
78
 
@@ -89,7 +121,7 @@ OpenTrace.configure do |c|
89
121
  c.environment = "production" # default: nil
90
122
  c.timeout = 1.0 # HTTP timeout in seconds (default: 1.0)
91
123
  c.enabled = true # default: true
92
- c.min_level = :info # minimum level to forward (default: :debug)
124
+ c.min_level = :info # minimum level to forward (default: :info)
93
125
  c.allowed_levels = [:warn, :error] # explicit level list (overrides min_level, default: nil)
94
126
  c.batch_size = 50 # logs per batch (default: 50)
95
127
  c.flush_interval = 5.0 # seconds between flushes (default: 5.0)
@@ -105,15 +137,15 @@ OpenTrace.configure do |c|
105
137
  c.git_sha = ENV["REVISION"] # checks REVISION, GIT_SHA, HEROKU_SLUG_COMMIT
106
138
 
107
139
  # SQL logging (Rails only)
108
- c.sql_logging = true # default: true
140
+ c.sql_logging = false # forward individual SQL queries (default: false)
109
141
  c.sql_duration_threshold_ms = 100.0 # only log queries slower than this (default: 0.0 = all)
110
142
 
111
- # Path filtering
112
- c.ignore_paths = ["/health", %r{\A/assets/}] # skip noisy paths (default: [])
143
+ # Path filtering (defaults include /up, /health, /healthz, /ping, /ready, /livez, /readyz)
144
+ c.ignore_paths = ["/health", %r{\A/assets/}] # customize paths to skip
113
145
 
114
146
  # Per-request summary (Rails only)
115
147
  c.request_summary = true # accumulate events into one rich log (default: true)
116
- c.timeline = true # include event timeline in summary (default: true)
148
+ c.timeline = false # include event timeline in summary (default: false)
117
149
  c.timeline_max_events = 200 # cap timeline entries (default: 200)
118
150
 
119
151
  # Background monitors (opt-in)
@@ -125,6 +157,42 @@ OpenTrace.configure do |c|
125
157
  # Advanced opt-in features
126
158
  c.memory_tracking = false # RSS delta per request (default: false)
127
159
  c.http_tracking = false # external HTTP call tracking (default: false)
160
+
161
+ # Sampling & performance
162
+ c.sample_rate = 1.0 # 0.0-1.0, fraction of requests to trace (default: 1.0)
163
+ c.sampler = ->(env) { 0.1 } # dynamic per-endpoint sampler (default: nil)
164
+ c.before_send = ->(payload) { payload } # filter/drop payloads before delivery (default: nil)
165
+
166
+ # SQL normalization (default: true)
167
+ c.sql_normalization = true # replace SQL literals with ? for grouping
168
+
169
+ # Instrumentation
170
+ c.source_context = false # capture source code around errors (default: false)
171
+ c.local_vars_capture = false # enable OpenTrace.capture_binding (default: false)
172
+ c.log_trace_injection = false # inject trace_id into Rails logger (default: false)
173
+ c.session_tracking = false # extract session ID from cookies (default: false)
174
+
175
+ # EXPLAIN plan capture
176
+ c.explain_slow_queries = false # run EXPLAIN on slow queries (default: false)
177
+ c.explain_threshold_ms = 100.0 # threshold in ms (default: 100.0)
178
+
179
+ # PII protection
180
+ c.pii_scrubbing = false # scrub PII from metadata (default: false)
181
+ c.pii_patterns = [/CUST-\d{8}/] # additional patterns (default: nil)
182
+ c.pii_disabled_patterns = [:phone] # disable built-in patterns (default: nil)
183
+
184
+ # Lifecycle callbacks
185
+ c.on_error = ->(exc, meta) { } # called on error capture (default: nil)
186
+ c.after_send = ->(batch_size, bytes) { } # called after delivery (default: nil)
187
+ c.before_breadcrumb = ->(crumb) { crumb } # filter breadcrumbs (default: nil)
188
+
189
+ # GC/Runtime metrics
190
+ c.runtime_metrics = false # collect GC/thread/memory stats (default: false)
191
+ c.runtime_metrics_interval = 30 # seconds between collections (default: 30)
192
+
193
+ # Unix socket transport
194
+ c.transport = :http # :http or :unix_socket (default: :http)
195
+ c.socket_path = "/tmp/opentrace.sock" # path to Unix socket (default: "/tmp/opentrace.sock")
128
196
  end
129
197
  ```
130
198
 
@@ -185,6 +253,96 @@ This captures:
185
253
  - `exception_message` -- truncated to 500 characters
186
254
  - `backtrace` -- cleaned (Rails backtrace cleaner or gem-filtered), limited to 15 frames
187
255
  - `error_fingerprint` -- 12-char hash for grouping identical errors (stable across line number changes)
256
+ - `exception_causes` -- full cause chain (up to 5 levels via `exception.cause`)
257
+ - `breadcrumbs` -- trail of events leading up to the error (if any were added)
258
+ - `source_context` -- surrounding source code lines at the error origin (when enabled)
259
+ - `local_variables` -- variable state at crash point (when `capture_binding` was called)
260
+
261
+ ### Custom Instrumentation
262
+
263
+ Trace any block of code with automatic timing:
264
+
265
+ ```ruby
266
+ OpenTrace.trace("stripe.charge", resource: "Invoice") do |span|
267
+ span.set_tag(:amount, 2000)
268
+ Stripe::Charge.create(amount: 2000, currency: "usd")
269
+ end
270
+ ```
271
+
272
+ Spans can be nested -- child spans automatically track their parent:
273
+
274
+ ```ruby
275
+ OpenTrace.trace("checkout.process") do
276
+ OpenTrace.trace("checkout.validate") { validate_cart }
277
+ OpenTrace.trace("checkout.charge") { charge_card }
278
+ OpenTrace.trace("checkout.fulfill") { create_order }
279
+ end
280
+ ```
281
+
282
+ Each span emits a log entry with `span_operation`, `span_duration_ms`, and parent/child IDs. When a `RequestCollector` is active, spans also appear in the request summary.
283
+
284
+ ### Breadcrumbs
285
+
286
+ Record a trail of events leading up to an error:
287
+
288
+ ```ruby
289
+ OpenTrace.add_breadcrumb("auth", "User logged in", { provider: "google" })
290
+ OpenTrace.add_breadcrumb("nav", "Visited /settings")
291
+ OpenTrace.add_breadcrumb("action", "Changed password")
292
+ ```
293
+
294
+ Breadcrumbs are stored per-request (Fiber-local, max 25) and automatically attached to error payloads. They are cleared after each request.
295
+
296
+ Filter breadcrumbs with a callback:
297
+
298
+ ```ruby
299
+ OpenTrace.configure do |c|
300
+ c.before_breadcrumb = ->(crumb) {
301
+ crumb.category == "noisy" ? nil : crumb # return nil to drop
302
+ }
303
+ end
304
+ ```
305
+
306
+ ### Local Variables Capture
307
+
308
+ Capture the state of local variables when an error occurs:
309
+
310
+ ```ruby
311
+ OpenTrace.configure do |c|
312
+ c.local_vars_capture = true
313
+ end
314
+
315
+ def update_profile(user, params)
316
+ user.update!(params)
317
+ rescue ActiveRecord::RecordInvalid => e
318
+ OpenTrace.capture_binding(e, binding) # explicit capture
319
+ OpenTrace.error(e, { action: "update_profile" })
320
+ raise
321
+ end
322
+ ```
323
+
324
+ This produces:
325
+
326
+ ```json
327
+ {
328
+ "local_variables": [
329
+ { "name": "user", "value": "#<User id: 42, name: nil>", "type": "User" },
330
+ { "name": "params", "value": "{\"name\"=>\"\"}", "type": "Hash" }
331
+ ]
332
+ }
333
+ ```
334
+
335
+ Capped at 10 variables, 500 chars per value. No global VM hooks -- zero overhead unless you explicitly call `capture_binding`.
336
+
337
+ ### Transaction Naming
338
+
339
+ Override the auto-detected transaction name for custom grouping:
340
+
341
+ ```ruby
342
+ OpenTrace.set_transaction_name("API::V2::Users#search")
343
+ ```
344
+
345
+ The transaction name appears in the request log message and metadata, enabling grouping by business operation instead of route.
188
346
 
189
347
  ### Business Events
190
348
 
@@ -383,7 +541,7 @@ Configure SQL logging:
383
541
  ```ruby
384
542
  OpenTrace.configure do |c|
385
543
  # ...
386
- c.sql_logging = true # enable/disable (default: true)
544
+ c.sql_logging = true # enable/disable (default: false)
387
545
  c.sql_duration_threshold_ms = 100.0 # only log slow queries (default: 0.0 = all)
388
546
  end
389
547
  ```
@@ -543,6 +701,185 @@ A recursion guard prevents OpenTrace's own HTTP calls to the server from being t
543
701
 
544
702
  **Note**: This works by prepending a module to `Net::HTTP`. Libraries that use `Net::HTTP` internally (Faraday, HTTParty, RestClient) are automatically captured.
545
703
 
704
+ ### Source Code Context
705
+
706
+ When enabled, error logs include the surrounding source lines at the crash location:
707
+
708
+ ```ruby
709
+ OpenTrace.configure do |c|
710
+ c.source_context = true
711
+ end
712
+ ```
713
+
714
+ Produces:
715
+
716
+ ```json
717
+ {
718
+ "source_context": {
719
+ "file": "app/models/order.rb",
720
+ "line": 42,
721
+ "context": {
722
+ "39": " def total",
723
+ "40": " items.sum(:price) +",
724
+ "41": " tax_amount +",
725
+ "42": " shipping_cost",
726
+ "43": " end"
727
+ }
728
+ }
729
+ }
730
+ ```
731
+
732
+ Files are cached (LRU, 50 files max). Only reads files under `/app/`, `/lib/`, or `/config/` and smaller than 100KB.
733
+
734
+ ### SQL Normalization
735
+
736
+ Enabled by default. Replaces SQL literals with `?` placeholders for grouping:
737
+
738
+ ```
739
+ SELECT * FROM users WHERE id = 42 AND email = 'alice@example.com'
740
+ => SELECT * FROM users WHERE id = ? AND email = ?
741
+ ```
742
+
743
+ Each normalized query gets a 12-char fingerprint (MD5) for fast grouping. The request summary includes `top_duplicates` with the most repeated query patterns.
744
+
745
+ ### Duplicate Query Detection
746
+
747
+ When a `RequestCollector` is active, SQL queries are fingerprinted and counted. The request summary includes:
748
+
749
+ - `duplicate_queries` -- number of fingerprints seen more than once
750
+ - `worst_duplicate_count` -- highest repeat count
751
+ - `top_duplicates` -- top 3 repeated queries with count and fingerprint
752
+ - `n_plus_one_warning` -- `true` when worst duplicate exceeds 5
753
+
754
+ ### Log Trace Injection
755
+
756
+ Injects trace context into your existing Rails logger output:
757
+
758
+ ```ruby
759
+ OpenTrace.configure do |c|
760
+ c.log_trace_injection = true
761
+ end
762
+ ```
763
+
764
+ Before: `Processing by UsersController#show`
765
+ After: `[trace_id=abc123 request_id=req-456] Processing by UsersController#show`
766
+
767
+ ### Session Tracking
768
+
769
+ Extracts session ID from the Rack session or session cookie:
770
+
771
+ ```ruby
772
+ OpenTrace.configure do |c|
773
+ c.session_tracking = true
774
+ end
775
+ ```
776
+
777
+ The session ID appears in request metadata, enabling session-level analysis.
778
+
779
+ ### PII Scrubbing
780
+
781
+ Automatically detects and redacts sensitive data before sending:
782
+
783
+ ```ruby
784
+ OpenTrace.configure do |c|
785
+ c.pii_scrubbing = true
786
+ end
787
+ ```
788
+
789
+ Built-in patterns detect: credit card numbers, email addresses, SSNs, phone numbers, bearer tokens, and API keys. Sensitive keys (`password`, `secret`, `token`, `api_key`, `authorization`) are always redacted.
790
+
791
+ Customize patterns:
792
+
793
+ ```ruby
794
+ OpenTrace.configure do |c|
795
+ c.pii_scrubbing = true
796
+ c.pii_patterns = [/CUST-\d{8}/] # add custom patterns
797
+ c.pii_disabled_patterns = [:phone] # disable built-in patterns
798
+ end
799
+ ```
800
+
801
+ PII scrubbing runs on the background thread -- zero request-thread overhead.
802
+
803
+ ### EXPLAIN Plan Capture
804
+
805
+ Automatically captures EXPLAIN output for slow SQL queries:
806
+
807
+ ```ruby
808
+ OpenTrace.configure do |c|
809
+ c.explain_slow_queries = true
810
+ c.explain_threshold_ms = 50.0 # queries slower than 50ms
811
+ end
812
+ ```
813
+
814
+ The SQL text is captured on the request thread (zero DB overhead). EXPLAIN is executed asynchronously on the background thread using a separate DB connection. Max 3 EXPLAIN queries per request.
815
+
816
+ ```json
817
+ {
818
+ "explain_plans": [{
819
+ "sql": "SELECT * FROM orders WHERE user_id = 42 ORDER BY created_at DESC",
820
+ "duration_ms": 87.3,
821
+ "explain_plan": "Seq Scan on orders (cost=0.00..45892.00 rows=12 width=380)\n Filter: (user_id = 42)"
822
+ }]
823
+ }
824
+ ```
825
+
826
+ ### GC/Runtime Metrics
827
+
828
+ Background thread collects Ruby runtime stats at a configurable interval:
829
+
830
+ ```ruby
831
+ OpenTrace.configure do |c|
832
+ c.runtime_metrics = true
833
+ c.runtime_metrics_interval = 30 # seconds (default: 30)
834
+ end
835
+ ```
836
+
837
+ Metrics collected: `gc_count`, `gc_major_count`, `gc_minor_count`, `gc_heap_live_slots`, `gc_heap_free_slots`, `gc_malloc_increase_bytes`, `thread_count`, `process_rss_mb`, `process_pid`. Sent as `event_type: "runtime.metrics"`.
838
+
839
+ ### Unix Socket Transport
840
+
841
+ For deployments where the OpenTrace server runs on the same host:
842
+
843
+ ```ruby
844
+ OpenTrace.configure do |c|
845
+ c.transport = :unix_socket
846
+ c.socket_path = "/var/run/opentrace.sock"
847
+ end
848
+ ```
849
+
850
+ 2-5x faster than HTTP for local delivery (no TCP/TLS overhead). Falls back to HTTP automatically if the socket is unavailable. Same batching, compression, and retry guarantees.
851
+
852
+ ### Lifecycle Callbacks
853
+
854
+ Hook into key lifecycle events:
855
+
856
+ ```ruby
857
+ OpenTrace.configure do |c|
858
+ # Called when OpenTrace.error captures an exception
859
+ c.on_error = ->(exception, metadata) {
860
+ Sentry.capture_exception(exception) if metadata[:exception_class] == "CriticalError"
861
+ }
862
+
863
+ # Called after each successful batch delivery
864
+ c.after_send = ->(batch_size, bytes) {
865
+ StatsD.histogram("opentrace.batch_size", batch_size)
866
+ }
867
+
868
+ # Filter or modify breadcrumbs
869
+ c.before_breadcrumb = ->(crumb) {
870
+ crumb.category == "secret" ? nil : crumb # return nil to drop
871
+ }
872
+
873
+ # Filter or modify entire payloads before delivery
874
+ c.before_send = ->(payload) {
875
+ payload[:metadata].delete(:internal_debug) if Rails.env.production?
876
+ payload # return nil to drop the entire payload
877
+ }
878
+ end
879
+ ```
880
+
881
+ All callbacks are wrapped in rescue -- a broken callback will never affect the host app.
882
+
546
883
  ## Runtime Controls
547
884
 
548
885
  ```ruby
@@ -0,0 +1,46 @@
1
+ # frozen_string_literal: true
2
+
3
+ module OpenTrace
4
+ class Breadcrumb
5
+ attr_reader :category, :message, :data, :timestamp, :level
6
+
7
+ def initialize(category:, message:, data: nil, level: "info")
8
+ @category = category.to_s
9
+ @message = message.to_s
10
+ @data = data
11
+ @level = level.to_s
12
+ @timestamp = Process.clock_gettime(Process::CLOCK_REALTIME)
13
+ end
14
+
15
+ def to_h
16
+ h = { category: @category, message: @message, level: @level, timestamp: @timestamp }
17
+ h[:data] = @data if @data
18
+ h
19
+ end
20
+ end
21
+
22
+ class BreadcrumbBuffer
23
+ MAX_BREADCRUMBS = 25
24
+
25
+ def initialize
26
+ @buffer = []
27
+ end
28
+
29
+ def add(breadcrumb)
30
+ @buffer.shift if @buffer.size >= MAX_BREADCRUMBS
31
+ @buffer << breadcrumb
32
+ end
33
+
34
+ def to_a
35
+ @buffer.map(&:to_h)
36
+ end
37
+
38
+ def empty?
39
+ @buffer.empty?
40
+ end
41
+
42
+ def size
43
+ @buffer.size
44
+ end
45
+ end
46
+ end
@@ -16,8 +16,9 @@ module OpenTrace
16
16
 
17
17
  attr_reader :stats
18
18
 
19
- def initialize(config)
19
+ def initialize(config, sampler: nil)
20
20
  @config = config
21
+ @sampler = sampler
21
22
  @queue = Thread::Queue.new
22
23
  @mutex = Mutex.new
23
24
  @thread = nil
@@ -140,6 +141,16 @@ module OpenTrace
140
141
  next if batch.empty?
141
142
 
142
143
  send_batch(batch)
144
+
145
+ # Adjust backpressure based on queue depth
146
+ if @sampler
147
+ queue_pct = @queue.size.to_f / MAX_QUEUE_SIZE
148
+ if queue_pct > 0.75
149
+ @sampler.increase_backpressure!
150
+ elsif queue_pct < 0.25
151
+ @sampler.decrease_backpressure!
152
+ end
153
+ end
143
154
  end
144
155
  rescue Exception # rubocop:disable Lint/RescueException
145
156
  # Swallow all errors including thread kill
@@ -201,7 +212,9 @@ module OpenTrace
201
212
  nil
202
213
  end
203
214
 
204
- def send_batch(batch)
215
+ MAX_BATCH_SPLIT_DEPTH = 5
216
+
217
+ def send_batch(batch, depth: 0)
205
218
  # Circuit breaker: skip if server is known-down
206
219
  unless @circuit_breaker.allow_request?
207
220
  @stats.increment(:dropped_circuit_open, batch.size)
@@ -212,22 +225,48 @@ module OpenTrace
212
225
  # Disable HTTP tracking for our own calls to prevent infinite recursion
213
226
  Fiber[:opentrace_http_tracking_disabled] = true
214
227
 
215
- # Apply per-payload truncation
216
- batch = batch.map { |p| fit_payload(p) }.compact
228
+ # Materialize deferred entries + apply before_send filter + truncate
229
+ batch = batch.filter_map do |item|
230
+ payload = PayloadBuilder.materialize(item, @config)
231
+ next nil unless payload
232
+ if @config.before_send
233
+ payload = @config.before_send.call(payload) rescue payload
234
+ unless payload
235
+ @stats.increment(:dropped_filtered)
236
+ next nil
237
+ end
238
+ end
239
+ # PII scrubbing (runs on background thread)
240
+ if @config.pii_scrubbing && payload[:metadata]
241
+ active_patterns = build_pii_patterns
242
+ PiiScrubber.scrub!(payload[:metadata], patterns: active_patterns)
243
+ end
244
+
245
+ fit_payload(payload)
246
+ end
217
247
  return if batch.empty?
218
248
 
219
249
  json = JSON.generate(batch)
220
250
 
221
- # If entire batch exceeds limit, split and retry
251
+ # If entire batch exceeds limit, split and retry (with depth guard)
222
252
  if json.bytesize > @config.max_payload_bytes
253
+ if depth >= MAX_BATCH_SPLIT_DEPTH
254
+ @stats.increment(:dropped_oversized, batch.size)
255
+ fire_on_drop(batch.size, :oversized)
256
+ return
257
+ end
223
258
  @stats.increment(:payload_splits)
224
259
  mid = batch.size / 2
225
- send_batch(batch[0...mid]) if mid > 0
226
- send_batch(batch[mid..]) if mid < batch.size
260
+ send_batch(batch[0...mid], depth: depth + 1) if mid > 0
261
+ send_batch(batch[mid..], depth: depth + 1) if mid < batch.size
227
262
  return
228
263
  end
229
264
 
230
- response = send_with_retry(json)
265
+ response = if @config.transport == :unix_socket
266
+ unix_socket_send(json)
267
+ else
268
+ send_with_retry(json)
269
+ end
231
270
  handle_response(response, batch, json.bytesize)
232
271
  rescue StandardError
233
272
  @circuit_breaker.record_failure
@@ -251,6 +290,7 @@ module OpenTrace
251
290
  @stats.increment(:delivered, batch.size)
252
291
  @stats.increment(:batches_sent)
253
292
  @stats.increment(:bytes_sent, bytes)
293
+ @config.after_send&.call(batch.size, bytes) rescue nil
254
294
  when Net::HTTPTooManyRequests
255
295
  handle_rate_limit(response, batch)
256
296
  when Net::HTTPUnauthorized
@@ -351,6 +391,32 @@ module OpenTrace
351
391
  persistent_http.request(request)
352
392
  end
353
393
 
394
+ def unix_socket_send(json)
395
+ payload = if @config.compression && json.bytesize > @config.compression_threshold
396
+ gzip_compress(json)
397
+ else
398
+ json
399
+ end
400
+
401
+ socket = UNIXSocket.new(@config.socket_path)
402
+ # Protocol: 4-byte big-endian length prefix + payload
403
+ socket.write([payload.bytesize].pack("N"))
404
+ socket.write(payload)
405
+ socket.flush
406
+
407
+ # Read 4-byte status code response
408
+ response_data = socket.read(4)
409
+ status = response_data&.unpack1("N") || 500
410
+ socket.close
411
+
412
+ UnixSocketResponse.new(status)
413
+ rescue Errno::ECONNREFUSED, Errno::ENOENT, Errno::ENOTSOCK
414
+ # Socket not available — fall back to HTTP
415
+ send_with_retry(json)
416
+ rescue StandardError
417
+ nil
418
+ end
419
+
354
420
  def retryable_response?(response)
355
421
  response.code.to_i >= 500
356
422
  end
@@ -394,6 +460,22 @@ module OpenTrace
394
460
  io.string
395
461
  end
396
462
 
463
+ def build_pii_patterns
464
+ patterns = PiiScrubber::PATTERNS.dup
465
+ # Remove disabled patterns
466
+ if @config.pii_disabled_patterns
467
+ @config.pii_disabled_patterns.each { |name| patterns.delete(name) }
468
+ end
469
+ result = patterns.values
470
+ # Add custom patterns
471
+ if @config.pii_patterns
472
+ result.concat(@config.pii_patterns)
473
+ end
474
+ result
475
+ rescue StandardError
476
+ PiiScrubber::PATTERNS.values
477
+ end
478
+
397
479
  def fire_on_drop(count, reason)
398
480
  @config.on_drop&.call(count, reason)
399
481
  rescue StandardError
@@ -458,6 +540,20 @@ module OpenTrace
458
540
  http.request(request)
459
541
  end
460
542
 
543
+ # Adapts a numeric status code from Unix socket into Net::HTTP response duck type
544
+ UnixSocketResponse = Struct.new(:code) do
545
+ def is_a?(klass)
546
+ c = code.to_i
547
+ case klass.name
548
+ when "Net::HTTPSuccess" then c >= 200 && c < 300
549
+ when "Net::HTTPTooManyRequests" then c == 429
550
+ when "Net::HTTPUnauthorized" then c == 401
551
+ when "Net::HTTPServerError" then c >= 500 && c < 600
552
+ else super
553
+ end
554
+ end
555
+ end
556
+
461
557
  def truncate_payload(payload)
462
558
  meta = payload[:metadata]&.dup || {}
463
559