foam-otel 1.8.1 → 1.9.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/GOTCHAS.md +126 -0
- data/README.md +5 -0
- data/lib/foam/otel/before_send.rb +36 -0
- data/lib/foam/otel/ingest.rb +6 -4
- data/lib/foam/otel/init.rb +10 -0
- data/lib/foam/otel/isolated_exporters.rb +239 -0
- data/lib/foam/otel/isolated_http_client.rb +586 -0
- data/lib/foam/otel/pipelines.rb +10 -3
- data/lib/foam/otel/prepend_collision_shield.rb +193 -0
- data/lib/foam/otel/version.rb +42 -1
- data/lib/foam/otel.rb +1 -0
- metadata +4 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: a2e506ae900650923638df263e5775efa20dd8ab7a63540eefb9d5156c1d008c
|
|
4
|
+
data.tar.gz: 64ea4091c85c9541ec1f08dd5560608e560ec80e8ee6918f95098b976b0ec82e
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 4c776696c10dc442394e5b023db7b47d178ec4b328addd693ba72b60e726c19f923533f0f1645e7b62ec322e63d6fcd0b39b1ca917cd9aa66403295c0b67e266
|
|
7
|
+
data.tar.gz: 615ec0ee4400e6e11baa6a6572fcafdccdeb6764f9bb2e65afcf4ff72a8185e9ba45086cd5e8e039934ea1043fe165d94784983539ac0d3e361a6b28f72a986e
|
data/GOTCHAS.md
CHANGED
|
@@ -866,6 +866,132 @@ exfiltratable — the value-pattern secret layer is the required second control
|
|
|
866
866
|
"REAL OTLP encode", "aggregates hook faults", "arity", "in-place value
|
|
867
867
|
mutation never bleeds"); `spec/before_send_champ_scenario_spec.rb` (the
|
|
868
868
|
export-thread / stamped-context proof).
|
|
869
|
+
## F17: There is no unwrapped Net::HTTP instance — foam's export must own its transport (export isolation, 2026-07-29)
|
|
870
|
+
|
|
871
|
+
- **Trap**: Ruby APM agents instrument by PREPENDING modules onto `Net::HTTP`
|
|
872
|
+
itself, so every instance in the process — including the one inside the
|
|
873
|
+
upstream OTLP exporter — dispatches through the foreign prepend chain, and
|
|
874
|
+
private helper names from different vendors resolve against each other on
|
|
875
|
+
the shared ancestry (the wrong-arity `annotate_span_with_response!`
|
|
876
|
+
collision class: every export crashes, zero rows land, the app looks
|
|
877
|
+
healthy). The upstream exporter's `untraced` suppression does NOT protect
|
|
878
|
+
the send — foreign agents never check OTel's flag; the exporter died with
|
|
879
|
+
it active. Reproduced at production parity against a real proprietary
|
|
880
|
+
auto-instrumenting agent (docs/decisions/export-isolation-ruby.md, with
|
|
881
|
+
before/after crash and rows-landed counts).
|
|
882
|
+
- **Sources**:
|
|
883
|
+
- Installed source — the vulnerable transport: `Net::HTTP.new` inside
|
|
884
|
+
`http_connection` and the `Net::HTTPResponse`-matching send loop
|
|
885
|
+
(opentelemetry-exporter-otlp `exporter.rb:129,150-258`; same shape in
|
|
886
|
+
-otlp-logs `logs_exporter.rb` and -otlp-metrics `metrics_exporter.rb` —
|
|
887
|
+
with drifting private `backoff?` signatures across the three gems).
|
|
888
|
+
- Design reference (strongest industry mechanism, isolation by
|
|
889
|
+
construction): the Go OTLP exporter's owned internal transport and its
|
|
890
|
+
documented warning against injecting an instrumented transport; the
|
|
891
|
+
error-tracking vendors' native-fetch browser transports.
|
|
892
|
+
- The prior collision incident class: the upstream contrib `net_http`
|
|
893
|
+
patch's private helper namespace
|
|
894
|
+
(opentelemetry-instrumentation-net_http `patches/stable/instrumentation.rb:81`).
|
|
895
|
+
- **Decision & why**: Isolation by construction, layered over suppression —
|
|
896
|
+
never suppression alone. `Foam::Otel::IsolatedHttpClient` is a hand-written
|
|
897
|
+
HTTP/1.1 sender over foam-owned `TCPSocket`/`OpenSSL::SSL::SSLSocket`
|
|
898
|
+
(VERIFY_PEER + hostname verification, CA/mTLS options, HTTP(S)_PROXY /
|
|
899
|
+
NO_PROXY with CONNECT tunneling, deadline IO, keep-alive with one
|
|
900
|
+
stale-socket retransmit, chunked/`Connection: close` reading, pid-keyed
|
|
901
|
+
reconnect for fork safety). `IsolatedExporters::{Trace,Logs,Metrics}Exporter`
|
|
902
|
+
subclass the upstream exporters, replacing exactly `http_connection` and
|
|
903
|
+
`send_bytes` (foam-owned retry loop, upstream semantics verbatim: 429/503
|
|
904
|
+
honor Retry-After, 408/502/504 + transient socket errors back off with
|
|
905
|
+
jitter, other 4xx drop; `untraced` kept as layer two). All six construction
|
|
906
|
+
sites (door 1 + door 2 × three signals) — the package's entire outbound
|
|
907
|
+
surface — build these subclasses. Capture is untouched; and foam's own
|
|
908
|
+
prepends (LLM shims, Logger bridge, fork hook) define ONLY the intercepted
|
|
909
|
+
public method, so foam can never be the colliding party on anyone's class.
|
|
910
|
+
- **Mitigation**: a foreign prepend chain on `Net::HTTP` structurally cannot
|
|
911
|
+
sit in foam's delivery path; a hostile wrong-arity prepend that crashes
|
|
912
|
+
every ordinary `Net::HTTP` caller leaves foam's rows landing.
|
|
913
|
+
- **Test**: `spec/export_isolation_spec.rb` (hostile prepend in a forked
|
|
914
|
+
child + spy-prepend never-dispatches proof, 503/429 Retry-After retries,
|
|
915
|
+
CONNECT-tunnel and absolute-form proxy paths, NO_PROXY, TLS wrong-hostname
|
|
916
|
+
rejection, chunked/keep-alive/stale-socket/deadline/fork-safety units, the
|
|
917
|
+
untraced layer-two pin, and the zero-collidable-private-helpers invariant);
|
|
918
|
+
`spec/export_isolation_hardening_spec.rb` (protocol edges: 1xx skip,
|
|
919
|
+
duplicate/trailered/extension-chunked responses, EOF-framed bodies,
|
|
920
|
+
keep-alive expiry, malformed status lines, IPv6 Host framing, 8-thread
|
|
921
|
+
concurrency; retry accounting with the backoff clock stubbed: exact
|
|
922
|
+
retry-cap exhaustion, Retry-After delta AND HTTP-date, 408/502/504,
|
|
923
|
+
redirects never followed, zero/mid-loop budget exhaustion; compression
|
|
924
|
+
none + its env var; loud non-protobuf 5xx handling; shutdown lifecycle;
|
|
925
|
+
mTLS e2e via the spec CERTIFICATE/CLIENT_CERTIFICATE/CLIENT_KEY env vars
|
|
926
|
+
incl. the rejected no-client-cert half; proxy credentials on CONNECT,
|
|
927
|
+
refused CONNECT, lowercase env twins, the NO_PROXY shape matrix; all
|
|
928
|
+
three signals through a full init, door-2 tap e2e, and a REAL fork);
|
|
929
|
+
`spec/ingest_spec.rb` (case 4 half 3: zero feedback spans with BOTH
|
|
930
|
+
cooperative layers defeated); `spec/transport_spec.rb` (the exact isolated
|
|
931
|
+
classes on all six sites); and the REAL-agent coexistence gate
|
|
932
|
+
`test-apps/ruby-coexistence/run-verify.sh` (two postures booting the
|
|
933
|
+
actual commercial agents in auto-instrument mode, hard-gated on the
|
|
934
|
+
vendor module being observed live ahead of Net::HTTP in the running
|
|
935
|
+
process's ancestor chain, verdict = protobuf-decoded rows on all three
|
|
936
|
+
signals with zero export errors).
|
|
937
|
+
|
|
938
|
+
---
|
|
939
|
+
|
|
940
|
+
## F18: One private-method namespace per class — cross-vendor helper collisions crash the APP, and foam shields them (coexistence mandate 2026-07-29)
|
|
941
|
+
|
|
942
|
+
- **Trap**: Every module prepended onto a class shares ONE private-method
|
|
943
|
+
namespace. When two observability vendors prepend same-named private
|
|
944
|
+
helpers at different signatures onto Net::HTTP, method lookup hands EVERY
|
|
945
|
+
caller the frontmost definition — one vendor's wrapper invokes the other
|
|
946
|
+
vendor's helper and raises ArgumentError on EVERY Net::HTTP request the
|
|
947
|
+
app makes. This is live in the wild today: a proprietary agent's wrapper
|
|
948
|
+
and the upstream OTel contrib net_http patch both define
|
|
949
|
+
`annotate_span_with_response!` (3-arg vs 2-arg). Foam ships that contrib
|
|
950
|
+
patch for capture, so a foam + agent process CONTAINS the collision — and
|
|
951
|
+
foam's promise is coexistence: the app must keep working.
|
|
952
|
+
- **Sources**:
|
|
953
|
+
- Installed source — the two colliding definitions:
|
|
954
|
+
opentelemetry-instrumentation-net_http 0.29.0
|
|
955
|
+
`patches/stable/instrumentation.rb:81` (`annotate_span_with_response!(span, response)`)
|
|
956
|
+
and the proprietary agent's contrib HTTP instrumentation calling its own
|
|
957
|
+
3-arg spelling of the same private name (verified live in the
|
|
958
|
+
coexistence gate; backtrace pinned in
|
|
959
|
+
docs/decisions/export-isolation-ruby.md §A).
|
|
960
|
+
- Ruby semantics: prepended modules join the class's ancestor chain and
|
|
961
|
+
private methods resolve through the SAME chain for every caller — there
|
|
962
|
+
is no per-module helper namespace.
|
|
963
|
+
- **Decision & why**: foam cannot rename the upstream helper (its own caller
|
|
964
|
+
resolves the name through the same shared chain — a rename breaks the
|
|
965
|
+
gem being "fixed"), and foam must never edit another vendor's module. So
|
|
966
|
+
foam owns the collision instead: after the instrumentation sweep, init
|
|
967
|
+
scans Net::HTTP's prepend chain for private-helper names defined by two
|
|
968
|
+
or more foreign modules at NON-identical positional signatures (names the
|
|
969
|
+
class itself defines are super-chains, never shielded) and prepends ONE
|
|
970
|
+
frontmost dispatch method per colliding name — routing by caller source
|
|
971
|
+
gem first, then by signature fit, falling back to the previously-frontmost
|
|
972
|
+
definition (status quo ante; the shield never makes a broken chain
|
|
973
|
+
worse). Its dispatch methods accept (*args, **kwargs, &block), so nothing
|
|
974
|
+
can arity-crash against the shield itself. No collision → NOTHING is
|
|
975
|
+
installed. Every shielded name is announced loudly (rule 15). This is the
|
|
976
|
+
ONE deliberate exception to foam's own-prepend hygiene rule (F17's
|
|
977
|
+
"foam's prepends carry zero private helpers"): the shield's whole job is
|
|
978
|
+
to own the already-colliding names.
|
|
979
|
+
- **Mitigation**: with the shield installed the coexistence gate's
|
|
980
|
+
patched-client probe went from 3/3 crashes to 3/3 passes under the real
|
|
981
|
+
agent, with both vendors' helpers verifiably executing again.
|
|
982
|
+
- **Residual**: an agent that patches AFTER foam's init lands ahead of the
|
|
983
|
+
shield and re-exposes the raw collision until a later init/fork re-scan;
|
|
984
|
+
agents overwhelmingly boot first (initializer/preload) — the order the
|
|
985
|
+
coexistence gate proves. Identical-signature collisions are left alone
|
|
986
|
+
(no crash class; dispatch could not disambiguate semantics).
|
|
987
|
+
- **Test**: `spec/prepend_collision_shield_spec.rb` (the unshielded RED
|
|
988
|
+
control, both-vendors-work dispatch, idempotence, no-op purity on
|
|
989
|
+
collision-free chains, super-chain and identical-signature exclusions,
|
|
990
|
+
fall-through to the real error, rule-9 scan degradation, caller-source
|
|
991
|
+
dispatch across two on-disk gem roots, and the real Net::HTTP + real
|
|
992
|
+
contrib patch field scenario red→green in a forked child);
|
|
993
|
+
`test-apps/ruby-coexistence/run-verify.sh` (the patched-client probe is a
|
|
994
|
+
REQUIRED pass on every vendor posture).
|
|
869
995
|
|
|
870
996
|
---
|
|
871
997
|
|
data/README.md
CHANGED
|
@@ -672,6 +672,11 @@ The per-record contract (Sentry `beforeSend` semantics):
|
|
|
672
672
|
in place, they are shared structures. Foam re-normalizes the record's
|
|
673
673
|
`total_recorded_*` bookkeeping after your hooks run, so adding/deleting
|
|
674
674
|
attributes never corrupts the wire's dropped-count accounting.
|
|
675
|
+
`record.resource` and `record.instrumentation_scope` are **identity, not
|
|
676
|
+
payload**: read them freely, but edits and replacements are discarded —
|
|
677
|
+
the original identity objects always ship (they are process-global and
|
|
678
|
+
the encoder groups by them; a writable resource would also bypass
|
|
679
|
+
redaction, which never masks resource attributes).
|
|
675
680
|
* **return `nil`** → the record is dropped, silently (that is the filter
|
|
676
681
|
mechanism, not an error).
|
|
677
682
|
* **raise, or return a foreign object** → that record is dropped
|
|
@@ -62,6 +62,7 @@ module Foam
|
|
|
62
62
|
# flood an adversary-shaped record stream controls — rule 15's
|
|
63
63
|
# "loud but never a flood" posture).
|
|
64
64
|
def run(hooks, record, signal, faults)
|
|
65
|
+
identity = identity_of(record)
|
|
65
66
|
current, counters = thaw(record)
|
|
66
67
|
hooks.each do |hook|
|
|
67
68
|
result = hook.call(current)
|
|
@@ -74,6 +75,7 @@ module Foam
|
|
|
74
75
|
current = result
|
|
75
76
|
end
|
|
76
77
|
restore_counters!(current, counters)
|
|
78
|
+
restore_identity!(current, identity)
|
|
77
79
|
current
|
|
78
80
|
rescue StandardError, SystemStackError => e
|
|
79
81
|
faults << e.class.name
|
|
@@ -110,9 +112,43 @@ module Foam
|
|
|
110
112
|
copy.attributes = attributes.nil? ? {} : attributes.transform_values { |v| thaw_value(v) }
|
|
111
113
|
end
|
|
112
114
|
copy.body = thaw_body(copy.body) if copy.respond_to?(:body) && copy.respond_to?(:body=)
|
|
115
|
+
# The scope handed to the hook is a COPY: InstrumentationScope is a
|
|
116
|
+
# mutable Struct, and an in-place `scope.name =` on the shared
|
|
117
|
+
# object would corrupt process-global identity (see identity_of).
|
|
118
|
+
if copy.respond_to?(:instrumentation_scope) && copy.respond_to?(:instrumentation_scope=)
|
|
119
|
+
copy.instrumentation_scope = copy.instrumentation_scope&.dup
|
|
120
|
+
end
|
|
113
121
|
[copy, sdk_drop_counters(record)]
|
|
114
122
|
end
|
|
115
123
|
|
|
124
|
+
# resource / instrumentation_scope are process-global IDENTITY objects
|
|
125
|
+
# shared by every record the tracer/logger emits — and Ruby's
|
|
126
|
+
# InstrumentationScope is a plain MUTABLE Struct. A hook that renames
|
|
127
|
+
# the scope would corrupt the identity of EVERY subsequent span/log
|
|
128
|
+
# from that instrument (proven live in the 2026-07-29 retro-probe),
|
|
129
|
+
# and a wholesale `record.resource =` replacement would ship
|
|
130
|
+
# hook-authored resource attributes RAW — redaction never masks
|
|
131
|
+
# resource, so that path could widen what ships. Both are closed the
|
|
132
|
+
# way the Go port does it: identity always comes from the ORIGINAL
|
|
133
|
+
# record — captured here, force-restored onto the survivor after the
|
|
134
|
+
# pipeline (hook edits to identity are discarded; the encoder's
|
|
135
|
+
# group-by-identity stays intact). Documented in the README:
|
|
136
|
+
# resource/instrumentation_scope are informational, not hook-writable.
|
|
137
|
+
def identity_of(record)
|
|
138
|
+
{
|
|
139
|
+
resource: (record.resource if record.respond_to?(:resource)),
|
|
140
|
+
scope: (record.instrumentation_scope if record.respond_to?(:instrumentation_scope)),
|
|
141
|
+
}
|
|
142
|
+
end
|
|
143
|
+
|
|
144
|
+
def restore_identity!(record, identity)
|
|
145
|
+
record.resource = identity[:resource] if record.respond_to?(:resource=)
|
|
146
|
+
if record.respond_to?(:instrumentation_scope=)
|
|
147
|
+
record.instrumentation_scope = identity[:scope]
|
|
148
|
+
end
|
|
149
|
+
record
|
|
150
|
+
end
|
|
151
|
+
|
|
116
152
|
def thaw_value(value)
|
|
117
153
|
case value
|
|
118
154
|
when String then value.dup
|
data/lib/foam/otel/ingest.rb
CHANGED
|
@@ -285,23 +285,25 @@ module Foam
|
|
|
285
285
|
# CWE-295 — see pipelines.rb setup_traces): door 2 carries the same
|
|
286
286
|
# fleet ingest token over the same TLS connection, so the silent
|
|
287
287
|
# OTEL_RUBY_EXPORTER_OTLP_SSL_VERIFY_NONE downgrade is pinned out
|
|
288
|
-
# here identically.
|
|
288
|
+
# here identically. All three are IsolatedExporters subclasses —
|
|
289
|
+
# door 2's delivery must be exactly as immune to foreign Net::HTTP
|
|
290
|
+
# prepends as door 1's (export-isolation design, 2026-07-29).
|
|
289
291
|
def build_otlp_span_exporter(endpoint, headers)
|
|
290
|
-
|
|
292
|
+
IsolatedExporters::TraceExporter.new(
|
|
291
293
|
endpoint: "#{endpoint}/v1/traces", headers: headers,
|
|
292
294
|
ssl_verify_mode: OpenSSL::SSL::VERIFY_PEER
|
|
293
295
|
)
|
|
294
296
|
end
|
|
295
297
|
|
|
296
298
|
def build_otlp_log_exporter(endpoint, headers)
|
|
297
|
-
|
|
299
|
+
IsolatedExporters::LogsExporter.new(
|
|
298
300
|
endpoint: "#{endpoint}/v1/logs", headers: headers,
|
|
299
301
|
ssl_verify_mode: OpenSSL::SSL::VERIFY_PEER
|
|
300
302
|
)
|
|
301
303
|
end
|
|
302
304
|
|
|
303
305
|
def build_otlp_metric_exporter(endpoint, headers)
|
|
304
|
-
|
|
306
|
+
IsolatedExporters::MetricsExporter.new(
|
|
305
307
|
endpoint: "#{endpoint}/v1/metrics", headers: headers,
|
|
306
308
|
ssl_verify_mode: OpenSSL::SSL::VERIFY_PEER
|
|
307
309
|
)
|
data/lib/foam/otel/init.rb
CHANGED
|
@@ -528,6 +528,10 @@ module Foam
|
|
|
528
528
|
require "opentelemetry-exporter-otlp"
|
|
529
529
|
require "opentelemetry-exporter-otlp-logs"
|
|
530
530
|
require "opentelemetry-exporter-otlp-metrics"
|
|
531
|
+
# The isolated transport subclasses (export-isolation design) load
|
|
532
|
+
# HERE — after the upstream exporter gems they subclass — so the lazy
|
|
533
|
+
# rule-41 posture holds: a disabled boot never pays their cost.
|
|
534
|
+
require_relative "isolated_exporters"
|
|
531
535
|
end
|
|
532
536
|
|
|
533
537
|
# The central RAW-capture activation config (raised-floor ruling
|
|
@@ -610,6 +614,12 @@ module Foam
|
|
|
610
614
|
FLOOR_INSTRUMENTATION_CONFIG.merge(SUPERSEDED_INSTRUMENTATION_CONFIG)
|
|
611
615
|
)
|
|
612
616
|
Diagnostics.info("instrumentations installed: #{gems.length} bundled gem(s)")
|
|
617
|
+
# Coexistence shield (2026-07-29, GOTCHAS F18): with the sweep done,
|
|
618
|
+
# every prepend that will sit on Net::HTTP in the realistic boot
|
|
619
|
+
# order (agent first, foam second) is in place — scan for cross-
|
|
620
|
+
# vendor private-helper collisions and shim them. A collision-free
|
|
621
|
+
# chain (the common case) installs NOTHING.
|
|
622
|
+
PrependCollisionShield.install!(::Net::HTTP) if defined?(::Net::HTTP)
|
|
613
623
|
rescue StandardError => e
|
|
614
624
|
Diagnostics.warn("instrumentation activation failed: #{e.class}: #{e.message}")
|
|
615
625
|
end
|
|
@@ -0,0 +1,239 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# The isolated OTLP exporters (export-isolation design, 2026-07-29). The
|
|
4
|
+
# upstream opentelemetry-exporter-otlp{,-logs,-metrics} gems each build a
|
|
5
|
+
# Net::HTTP inside #http_connection and drive it from #send_bytes — which puts
|
|
6
|
+
# every foreign APM's Net::HTTP prepend chain (Datadog, New Relic, Sentry…)
|
|
7
|
+
# INSIDE foam's delivery path. These subclasses keep everything else the
|
|
8
|
+
# upstream classes do — endpoint/headers/compression/timeout resolution
|
|
9
|
+
# including the spec OTEL_EXPORTER_OTLP_* env vars, protobuf encoding, the
|
|
10
|
+
# ssl/mTLS options — and replace exactly two private methods:
|
|
11
|
+
#
|
|
12
|
+
# * #http_connection → returns Foam::Otel::IsolatedHttpClient (an owned,
|
|
13
|
+
# hand-written HTTP/1.1 sender; see isolated_http_client.rb);
|
|
14
|
+
# * #send_bytes → a foam-owned send loop that reimplements the upstream
|
|
15
|
+
# retry semantics VERBATIM (backoff with jitter on 429/503 honoring
|
|
16
|
+
# Retry-After, backoff on 408/502/504 and transient socket/SSL errors,
|
|
17
|
+
# drop on other 4xx/5xx) against the isolated client. Reimplemented, not
|
|
18
|
+
# inherited, because the upstream loop pattern-matches Net::HTTPResponse
|
|
19
|
+
# classes and rescues Net::* exceptions — and because its private helper
|
|
20
|
+
# signatures drift across the three gems (logs' backoff? has no reason:).
|
|
21
|
+
#
|
|
22
|
+
# Suppression stays as LAYER TWO: the send runs inside
|
|
23
|
+
# Common::Utilities.untraced, so cooperative instrumentation (foam's own
|
|
24
|
+
# included) never traces foam's export even if a future capture hook could
|
|
25
|
+
# see it. Layer one — the reason the Datadog crash cannot recur — is that the
|
|
26
|
+
# bytes never enter Net::HTTP at all.
|
|
27
|
+
require "zlib"
|
|
28
|
+
|
|
29
|
+
require_relative "diagnostics"
|
|
30
|
+
require_relative "isolated_http_client"
|
|
31
|
+
|
|
32
|
+
module Foam
|
|
33
|
+
module Otel
|
|
34
|
+
module IsolatedExporters
|
|
35
|
+
# Export result codes; == OpenTelemetry::SDK::*::Export::SUCCESS/FAILURE
|
|
36
|
+
# in every signal's SDK (integers, so this module never reaches into the
|
|
37
|
+
# per-signal constant trees).
|
|
38
|
+
SUCCESS = 0
|
|
39
|
+
FAILURE = 1
|
|
40
|
+
RETRY_COUNT = 5 # upstream's cap, kept verbatim
|
|
41
|
+
|
|
42
|
+
# The shared send path, included by the three per-signal subclasses
|
|
43
|
+
# below. Relies only on ivars every upstream constructor sets (@http,
|
|
44
|
+
# @path, @headers, @timeout, @compression, @uri) plus the optional
|
|
45
|
+
# @metrics_reporter (absent in the logs exporter — every use is nil-safe).
|
|
46
|
+
module SendPath
|
|
47
|
+
private
|
|
48
|
+
|
|
49
|
+
def http_connection(uri, ssl_verify_mode, certificate_file, client_certificate_file, client_key_file)
|
|
50
|
+
IsolatedHttpClient.new(
|
|
51
|
+
uri,
|
|
52
|
+
ssl_verify_mode: ssl_verify_mode,
|
|
53
|
+
certificate_file: certificate_file,
|
|
54
|
+
client_certificate_file: client_certificate_file,
|
|
55
|
+
client_key_file: client_key_file
|
|
56
|
+
)
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
def send_bytes(bytes, timeout:) # rubocop:disable Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity
|
|
60
|
+
return FAILURE if bytes.nil?
|
|
61
|
+
|
|
62
|
+
report_value("otel.otlp_exporter.message.uncompressed_size", bytes.bytesize)
|
|
63
|
+
if @compression == "gzip"
|
|
64
|
+
body = Zlib.gzip(bytes)
|
|
65
|
+
report_value("otel.otlp_exporter.message.compressed_size", body.bytesize)
|
|
66
|
+
else
|
|
67
|
+
body = bytes
|
|
68
|
+
end
|
|
69
|
+
request_headers = @headers.merge("Content-Type" => "application/x-protobuf")
|
|
70
|
+
request_headers["Content-Encoding"] = "gzip" if @compression == "gzip"
|
|
71
|
+
|
|
72
|
+
retry_count = 0
|
|
73
|
+
timeout ||= @timeout
|
|
74
|
+
start_time = OpenTelemetry::Common::Utilities.timeout_timestamp
|
|
75
|
+
|
|
76
|
+
# Layer two: the cooperative suppression flag, kept even though the
|
|
77
|
+
# isolated client makes it structurally unnecessary for delivery —
|
|
78
|
+
# it also stops foam ever tracing its own export.
|
|
79
|
+
OpenTelemetry::Common::Utilities.untraced do
|
|
80
|
+
loop do
|
|
81
|
+
remaining_timeout = OpenTelemetry::Common::Utilities.maybe_timeout(timeout, start_time)
|
|
82
|
+
return FAILURE if remaining_timeout.zero?
|
|
83
|
+
|
|
84
|
+
response = @http.post(@path, body: body, headers: request_headers, timeout: remaining_timeout)
|
|
85
|
+
|
|
86
|
+
case response.code
|
|
87
|
+
when 200..299
|
|
88
|
+
return SUCCESS
|
|
89
|
+
when 429, 503
|
|
90
|
+
next if foam_backoff?(retry_after: response["retry-after"],
|
|
91
|
+
retry_count: retry_count += 1, reason: response.code.to_s)
|
|
92
|
+
|
|
93
|
+
# Retry exhaustion is announced (fleet-review parity fix:
|
|
94
|
+
# upstream's per-retry handle_http_error kept failures
|
|
95
|
+
# visible; foam announces once, at the point data drops).
|
|
96
|
+
foam_log_request_failure(response.code)
|
|
97
|
+
return FAILURE
|
|
98
|
+
when 408, 502, 504
|
|
99
|
+
next if foam_backoff?(retry_count: retry_count += 1, reason: response.code.to_s)
|
|
100
|
+
|
|
101
|
+
foam_log_request_failure(response.code)
|
|
102
|
+
return FAILURE
|
|
103
|
+
when 404
|
|
104
|
+
foam_log_request_failure(response.code)
|
|
105
|
+
return FAILURE
|
|
106
|
+
when 300..399
|
|
107
|
+
# Upstream parity: redirects are not followed (its
|
|
108
|
+
# handle_redirect is a TODO); drop the connection and back off.
|
|
109
|
+
@http.finish
|
|
110
|
+
next if foam_backoff?(retry_after: 0, retry_count: retry_count += 1,
|
|
111
|
+
reason: response.code.to_s)
|
|
112
|
+
|
|
113
|
+
return FAILURE
|
|
114
|
+
when 400..599
|
|
115
|
+
foam_log_status(response.body)
|
|
116
|
+
add_to_failure_counter(response.code.to_s)
|
|
117
|
+
return FAILURE
|
|
118
|
+
else
|
|
119
|
+
@http.finish
|
|
120
|
+
foam_log_request_failure(response.code)
|
|
121
|
+
return FAILURE
|
|
122
|
+
end
|
|
123
|
+
rescue IsolatedHttpClient::TimeoutError
|
|
124
|
+
# Deliberate divergence from upstream (documented): upstream
|
|
125
|
+
# lets Net::WriteTimeout fall to the fatal catch-all while
|
|
126
|
+
# retrying read timeouts; a stalled WRITE is just as transient
|
|
127
|
+
# (proxy hiccup, slow-start collector), so foam retries both
|
|
128
|
+
# within the same budget/backoff caps.
|
|
129
|
+
next if foam_backoff?(retry_count: retry_count += 1, reason: "timeout")
|
|
130
|
+
|
|
131
|
+
return FAILURE
|
|
132
|
+
rescue OpenSSL::SSL::SSLError => e
|
|
133
|
+
next if foam_backoff?(retry_count: retry_count += 1, reason: "openssl_error")
|
|
134
|
+
|
|
135
|
+
OpenTelemetry.handle_error(exception: e, message: "SSL error in isolated OTLP send_bytes")
|
|
136
|
+
return FAILURE
|
|
137
|
+
rescue SocketError, IsolatedHttpClient::ProxyConnectError
|
|
138
|
+
next if foam_backoff?(retry_count: retry_count += 1, reason: "socket_error")
|
|
139
|
+
|
|
140
|
+
return FAILURE
|
|
141
|
+
rescue SystemCallError => e
|
|
142
|
+
next if foam_backoff?(retry_count: retry_count += 1, reason: e.class.name)
|
|
143
|
+
|
|
144
|
+
return FAILURE
|
|
145
|
+
rescue EOFError
|
|
146
|
+
next if foam_backoff?(retry_count: retry_count += 1, reason: "eof_error")
|
|
147
|
+
|
|
148
|
+
return FAILURE
|
|
149
|
+
rescue Zlib::DataError
|
|
150
|
+
next if foam_backoff?(retry_count: retry_count += 1, reason: "zlib_error")
|
|
151
|
+
|
|
152
|
+
return FAILURE
|
|
153
|
+
rescue StandardError => e
|
|
154
|
+
OpenTelemetry.handle_error(exception: e, message: "unexpected error in isolated OTLP send_bytes")
|
|
155
|
+
add_to_failure_counter(e.class.to_s)
|
|
156
|
+
return FAILURE
|
|
157
|
+
end
|
|
158
|
+
end
|
|
159
|
+
end
|
|
160
|
+
|
|
161
|
+
# Upstream backoff semantics verbatim (red-flag 2): count the failure,
|
|
162
|
+
# give up past RETRY_COUNT, honor Retry-After as delta-seconds or
|
|
163
|
+
# HTTP-date, else exponential jitter — foam-prefixed (rule: foam is
|
|
164
|
+
# never the colliding party on anyone's class, its own included).
|
|
165
|
+
def foam_backoff?(retry_count:, reason:, retry_after: nil)
|
|
166
|
+
add_to_failure_counter(reason)
|
|
167
|
+
return false if retry_count > RETRY_COUNT
|
|
168
|
+
|
|
169
|
+
sleep_interval = nil
|
|
170
|
+
unless retry_after.nil?
|
|
171
|
+
sleep_interval = Integer(retry_after, exception: false)
|
|
172
|
+
sleep_interval ||=
|
|
173
|
+
begin
|
|
174
|
+
Time.httpdate(retry_after) - Time.now
|
|
175
|
+
rescue StandardError
|
|
176
|
+
nil
|
|
177
|
+
end
|
|
178
|
+
sleep_interval = nil unless sleep_interval&.positive?
|
|
179
|
+
end
|
|
180
|
+
sleep_interval ||= rand(2**retry_count)
|
|
181
|
+
|
|
182
|
+
sleep(sleep_interval)
|
|
183
|
+
true
|
|
184
|
+
end
|
|
185
|
+
|
|
186
|
+
def foam_log_status(body)
|
|
187
|
+
status = Google::Rpc::Status.decode(body.to_s)
|
|
188
|
+
details = status.details.filter_map do |detail|
|
|
189
|
+
klass = ::Google::Protobuf::DescriptorPool.generated_pool.lookup(detail.type_name).msgclass
|
|
190
|
+
detail.unpack(klass) if klass
|
|
191
|
+
end
|
|
192
|
+
OpenTelemetry.handle_error(
|
|
193
|
+
message: "OTLP exporter received rpc.Status{message=#{status.message}, details=#{details}} for uri=#{@uri}"
|
|
194
|
+
)
|
|
195
|
+
rescue StandardError
|
|
196
|
+
# A body that isn't a valid rpc.Status (HTML error page from a
|
|
197
|
+
# proxy, empty 500…) still gets a readable failure line.
|
|
198
|
+
OpenTelemetry.handle_error(message: "OTLP exporter received an error response for uri=#{@uri}")
|
|
199
|
+
end
|
|
200
|
+
|
|
201
|
+
def foam_log_request_failure(response_code)
|
|
202
|
+
OpenTelemetry.handle_error(
|
|
203
|
+
message: "OTLP exporter received http.code=#{response_code} for uri='#{@uri}' in isolated send_bytes"
|
|
204
|
+
)
|
|
205
|
+
add_to_failure_counter(response_code.to_s)
|
|
206
|
+
end
|
|
207
|
+
|
|
208
|
+
def add_to_failure_counter(reason)
|
|
209
|
+
@metrics_reporter&.add_to_counter("otel.otlp_exporter.failure", labels: { "reason" => reason })
|
|
210
|
+
rescue StandardError
|
|
211
|
+
nil
|
|
212
|
+
end
|
|
213
|
+
|
|
214
|
+
def report_value(metric, value)
|
|
215
|
+
@metrics_reporter&.record_value(metric, value: value)
|
|
216
|
+
rescue StandardError
|
|
217
|
+
nil
|
|
218
|
+
end
|
|
219
|
+
end
|
|
220
|
+
|
|
221
|
+
# The three per-signal exporters. Constructors, protobuf encoding,
|
|
222
|
+
# spec env-var handling, force_flush/shutdown all inherit from upstream;
|
|
223
|
+
# only the transport-touching privates are replaced by SendPath (module
|
|
224
|
+
# ancestry: subclass → SendPath → upstream class, so the upstream
|
|
225
|
+
# constructor's http_connection call dispatches into SendPath's).
|
|
226
|
+
class TraceExporter < ::OpenTelemetry::Exporter::OTLP::Exporter
|
|
227
|
+
include SendPath
|
|
228
|
+
end
|
|
229
|
+
|
|
230
|
+
class LogsExporter < ::OpenTelemetry::Exporter::OTLP::Logs::LogsExporter
|
|
231
|
+
include SendPath
|
|
232
|
+
end
|
|
233
|
+
|
|
234
|
+
class MetricsExporter < ::OpenTelemetry::Exporter::OTLP::Metrics::MetricsExporter
|
|
235
|
+
include SendPath
|
|
236
|
+
end
|
|
237
|
+
end
|
|
238
|
+
end
|
|
239
|
+
end
|