foam-otel 1.8.0 → 1.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3e034361554ad48530554a1bdc62d75adf1899e3ef57104a367abd2c412d2f93
-  data.tar.gz: 40b6db9d7f11ed9105b00c9998b87da51e0822480d97da52be766ddea24435ca
+  metadata.gz: d27bb6b38712364503503fecd37ea383f9a7e3fb3648da8ef41524c575ef9302
+  data.tar.gz: 0c9fee962999043120904c8ffe85621e32ae04fcb300280ec8af0b2c77291a0d
 SHA512:
-  metadata.gz: 9c71df1b9ce568dabd996b659d13e0bc05e93a81e070376a77e8c09ccd6a6fc51dcaa028c19a904e78d38daae532c6bc75077e3fa5331047aa1c64e08a9609ee
-  data.tar.gz: 90623d934372eb908d8c9d13e5ddd4108faad54bea4c5a647e7934959562ff130c2cdfdc2f0c5c5cc00a1a86eb5fcc4ee03c745c69c8c258fced2caf6ceb2f31
+  metadata.gz: 8bdcb84ca5d9becc44e9a181512ac19c9c8c24f5035e7da9da09a1ab59a4c09cd96b9cde80a8b5d073001ae6f36d9f4e1b3e1a3e640c4eafa6d30ac34800e47c
+  data.tar.gz: b09d59343870e8d16ef9e4c0794e7ec78486d5ab9c3eea339d99c4f8b4fa46f58ca5246ca3df70ed8a82c795fd4108c296c3bb19e720640c5acce080f0e90a98

data/GOTCHAS.md

@@ -808,6 +808,193 @@ exfiltratable — the value-pattern secret layer is the required second control
 
 ---
 
+## F16: before_send hooks run inside the export lock, and record bookkeeping must be re-normalized behind them
+
+- **Trap**: Three traps share the seam. (1) The batch processors call
+  `exporter.export` inside `@export_mutex.synchronize` with no lock timeout —
+  a customer `before_send` hook that blocks (DB lookup, un-timed HTTP, sleep)
+  wedges the worker thread AND any concurrent `force_flush`/`shutdown`
+  acquiring the same mutex: the process can hang at exit until SIGKILL.
+  (2) The OTLP exporters encode `dropped_attributes_count` as
+  `total_recorded_attributes - attributes.size` into a protobuf **uint32** —
+  a hook that ADDS an attribute without the total being fixed up makes the
+  subtraction negative, protobuf raises `RangeError`, `encode` rescues to
+  nil, and the WHOLE export request — every healthy record batched beside
+  the touched one — is dropped, silently, forever (the same trap
+  session_stitching already documents for foam's own stamped attributes; a
+  hand-built replacement record with nil totals is the NoMethodError variant
+  of the same encode failure). (3) A hook fault marks the whole batch
+  FAILURE (redacting-exporter parity), and `force_flush` aborts its drain
+  loop on the first non-SUCCESS batch — so a hook that faults
+  DETERMINISTICALLY on some record shape makes the at-exit flush abandon
+  everything queued behind the first faulted batch.
+- **Sources**:
+  - Installed source: `OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor`
+    `#export_batch` inside `@export_mutex.synchronize`
+    (opentelemetry-sdk `trace/export/batch_span_processor.rb:189`), same shape
+    in `logs/export/batch_log_record_processor.rb:186`; `force_flush`'s
+    `return result_code unless result_code == SUCCESS` drain-abort
+    (`batch_span_processor.rb:113-115`).
+  - Installed source: `dropped_attributes_count: span_data.total_recorded_attributes - span_data.attributes&.size.to_i`
+    (opentelemetry-exporter-otlp `exporter.rb:343`; logs twin
+    `logs_exporter.rb:311`); `encode` rescues `StandardError` → nil →
+    `send_bytes(nil)` returns FAILURE for the whole request.
+  - In-repo precedent: `lib/foam/otel/session_stitching.rb` bumps
+    `@total_recorded_attributes` when foam adds its own attribute, guarding
+    against exactly trap (2).
+- **Decision & why**: Hooks stay ON the export thread (any queue/thread
+  indirection would break the drop-before-serialize guarantee and add an
+  unbounded buffer); foam re-normalizes `total_recorded_*` on every
+  surviving record after the pipeline (`BeforeSend.restore_counters!` —
+  size + SDK-dropped delta captured pre-hook, nil-safe), hands hooks COPIED
+  attribute values and log bodies so in-place mutation cannot bleed into
+  the live record or a tenant's buffered view, and keeps FAILURE accounting
+  for faulted records (parity with the redacting exporters) while warning
+  ONCE per batch with error classes only. The blocking-hook risk is
+  accepted and documented (README "The `before_send` hook") — it is the
+  customer's own code on their own pipeline, the same trust grade as a
+  tenant processor, and rule 49's DoS bounds govern foam's code, not the
+  customer's.
+- **Mitigation**: README pins the contract: non-blocking hooks only; stamp
+  request/job-thread context onto attributes at capture; replace (never
+  mutate) events/links/status; deterministic-fault hooks are a customer bug
+  surfaced by the aggregated per-batch `[foam]` warning + batch FAILURE.
+  Boot-time arity validation rejects hooks that cannot take one positional
+  argument (the Sentry two-arg port) so the fault-every-record shape is
+  caught on the engineer's machine (rule 10).
+- **Test**: `spec/before_send_spec.rb` ("re-normalizes total_recorded",
+  "REAL OTLP encode", "aggregates hook faults", "arity", "in-place value
+  mutation never bleeds"); `spec/before_send_champ_scenario_spec.rb` (the
+  export-thread / stamped-context proof).
+## F17: There is no unwrapped Net::HTTP instance — foam's export must own its transport (export isolation, 2026-07-29)
+
+- **Trap**: Ruby APM agents instrument by PREPENDING modules onto `Net::HTTP`
+  itself, so every instance in the process — including the one inside the
+  upstream OTLP exporter — dispatches through the foreign prepend chain, and
+  private helper names from different vendors resolve against each other on
+  the shared ancestry (the wrong-arity `annotate_span_with_response!`
+  collision class: every export crashes, zero rows land, the app looks
+  healthy). The upstream exporter's `untraced` suppression does NOT protect
+  the send — foreign agents never check OTel's flag; the exporter died with
+  it active. Reproduced at production parity against a real proprietary
+  auto-instrumenting agent (docs/decisions/export-isolation-ruby.md, with
+  before/after crash and rows-landed counts).
+- **Sources**:
+  - Installed source — the vulnerable transport: `Net::HTTP.new` inside
+    `http_connection` and the `Net::HTTPResponse`-matching send loop
+    (opentelemetry-exporter-otlp `exporter.rb:129,150-258`; same shape in
+    -otlp-logs `logs_exporter.rb` and -otlp-metrics `metrics_exporter.rb` —
+    with drifting private `backoff?` signatures across the three gems).
+  - Design reference (strongest industry mechanism, isolation by
+    construction): the Go OTLP exporter's owned internal transport and its
+    documented warning against injecting an instrumented transport; the
+    error-tracking vendors' native-fetch browser transports.
+  - The prior collision incident class: the upstream contrib `net_http`
+    patch's private helper namespace
+    (opentelemetry-instrumentation-net_http `patches/stable/instrumentation.rb:81`).
+- **Decision & why**: Isolation by construction, layered over suppression —
+  never suppression alone. `Foam::Otel::IsolatedHttpClient` is a hand-written
+  HTTP/1.1 sender over foam-owned `TCPSocket`/`OpenSSL::SSL::SSLSocket`
+  (VERIFY_PEER + hostname verification, CA/mTLS options, HTTP(S)_PROXY /
+  NO_PROXY with CONNECT tunneling, deadline IO, keep-alive with one
+  stale-socket retransmit, chunked/`Connection: close` reading, pid-keyed
+  reconnect for fork safety). `IsolatedExporters::{Trace,Logs,Metrics}Exporter`
+  subclass the upstream exporters, replacing exactly `http_connection` and
+  `send_bytes` (foam-owned retry loop, upstream semantics verbatim: 429/503
+  honor Retry-After, 408/502/504 + transient socket errors back off with
+  jitter, other 4xx drop; `untraced` kept as layer two). All six construction
+  sites (door 1 + door 2 × three signals) — the package's entire outbound
+  surface — build these subclasses. Capture is untouched; and foam's own
+  prepends (LLM shims, Logger bridge, fork hook) define ONLY the intercepted
+  public method, so foam can never be the colliding party on anyone's class.
+- **Mitigation**: a foreign prepend chain on `Net::HTTP` structurally cannot
+  sit in foam's delivery path; a hostile wrong-arity prepend that crashes
+  every ordinary `Net::HTTP` caller leaves foam's rows landing.
+- **Test**: `spec/export_isolation_spec.rb` (hostile prepend in a forked
+  child + spy-prepend never-dispatches proof, 503/429 Retry-After retries,
+  CONNECT-tunnel and absolute-form proxy paths, NO_PROXY, TLS wrong-hostname
+  rejection, chunked/keep-alive/stale-socket/deadline/fork-safety units, the
+  untraced layer-two pin, and the zero-collidable-private-helpers invariant);
+  `spec/export_isolation_hardening_spec.rb` (protocol edges: 1xx skip,
+  duplicate/trailered/extension-chunked responses, EOF-framed bodies,
+  keep-alive expiry, malformed status lines, IPv6 Host framing, 8-thread
+  concurrency; retry accounting with the backoff clock stubbed: exact
+  retry-cap exhaustion, Retry-After delta AND HTTP-date, 408/502/504,
+  redirects never followed, zero/mid-loop budget exhaustion; compression
+  none + its env var; loud non-protobuf 5xx handling; shutdown lifecycle;
+  mTLS e2e via the spec CERTIFICATE/CLIENT_CERTIFICATE/CLIENT_KEY env vars
+  incl. the rejected no-client-cert half; proxy credentials on CONNECT,
+  refused CONNECT, lowercase env twins, the NO_PROXY shape matrix; all
+  three signals through a full init, door-2 tap e2e, and a REAL fork);
+  `spec/ingest_spec.rb` (case 4 half 3: zero feedback spans with BOTH
+  cooperative layers defeated); `spec/transport_spec.rb` (the exact isolated
+  classes on all six sites); and the REAL-agent coexistence gate
+  `test-apps/ruby-coexistence/run-verify.sh` (two postures booting the
+  actual commercial agents in auto-instrument mode, hard-gated on the
+  vendor module being observed live ahead of Net::HTTP in the running
+  process's ancestor chain, verdict = protobuf-decoded rows on all three
+  signals with zero export errors).
+
+---
+
+## F18: One private-method namespace per class — cross-vendor helper collisions crash the APP, and foam shields them (coexistence mandate 2026-07-29)
+
+- **Trap**: Every module prepended onto a class shares ONE private-method
+  namespace. When two observability vendors prepend same-named private
+  helpers at different signatures onto Net::HTTP, method lookup hands EVERY
+  caller the frontmost definition — one vendor's wrapper invokes the other
+  vendor's helper and raises ArgumentError on EVERY Net::HTTP request the
+  app makes. This is live in the wild today: a proprietary agent's wrapper
+  and the upstream OTel contrib net_http patch both define
+  `annotate_span_with_response!` (3-arg vs 2-arg). Foam ships that contrib
+  patch for capture, so a foam + agent process CONTAINS the collision — and
+  foam's promise is coexistence: the app must keep working.
+- **Sources**:
+  - Installed source — the two colliding definitions:
+    opentelemetry-instrumentation-net_http 0.29.0
+    `patches/stable/instrumentation.rb:81` (`annotate_span_with_response!(span, response)`)
+    and the proprietary agent's contrib HTTP instrumentation calling its own
+    3-arg spelling of the same private name (verified live in the
+    coexistence gate; backtrace pinned in
+    docs/decisions/export-isolation-ruby.md §A).
+  - Ruby semantics: prepended modules join the class's ancestor chain and
+    private methods resolve through the SAME chain for every caller — there
+    is no per-module helper namespace.
+- **Decision & why**: foam cannot rename the upstream helper (its own caller
+  resolves the name through the same shared chain — a rename breaks the
+  gem being "fixed"), and foam must never edit another vendor's module. So
+  foam owns the collision instead: after the instrumentation sweep, init
+  scans Net::HTTP's prepend chain for private-helper names defined by two
+  or more foreign modules at NON-identical positional signatures (names the
+  class itself defines are super-chains, never shielded) and prepends ONE
+  frontmost dispatch method per colliding name — routing by caller source
+  gem first, then by signature fit, falling back to the previously-frontmost
+  definition (status quo ante; the shield never makes a broken chain
+  worse). Its dispatch methods accept (*args, **kwargs, &block), so nothing
+  can arity-crash against the shield itself. No collision → NOTHING is
+  installed. Every shielded name is announced loudly (rule 15). This is the
+  ONE deliberate exception to foam's own-prepend hygiene rule (F17's
+  "foam's prepends carry zero private helpers"): the shield's whole job is
+  to own the already-colliding names.
+- **Mitigation**: with the shield installed the coexistence gate's
+  patched-client probe went from 3/3 crashes to 3/3 passes under the real
+  agent, with both vendors' helpers verifiably executing again.
+- **Residual**: an agent that patches AFTER foam's init lands ahead of the
+  shield and re-exposes the raw collision until a later init/fork re-scan;
+  agents overwhelmingly boot first (initializer/preload) — the order the
+  coexistence gate proves. Identical-signature collisions are left alone
+  (no crash class; dispatch could not disambiguate semantics).
+- **Test**: `spec/prepend_collision_shield_spec.rb` (the unshielded RED
+  control, both-vendors-work dispatch, idempotence, no-op purity on
+  collision-free chains, super-chain and identical-signature exclusions,
+  fall-through to the real error, rule-9 scan degradation, caller-source
+  dispatch across two on-disk gem roots, and the real Net::HTTP + real
+  contrib patch field scenario red→green in a forked child);
+  `test-apps/ruby-coexistence/run-verify.sh` (the patched-client probe is a
+  REQUIRED pass on every vendor posture).
+
+---
+
 ## General gotchas (applicable to Ruby)
 
 - **G1 — init after target import / pre-init no-op**: the API's proxy providers

data/README.md

@@ -331,7 +331,7 @@ not captured — inbound only.
 | `additional_metric_readers:` | Array | no | `[]` | Tenant seam, metrics. |
 | `additional_instrumentations:` | Array | no | `[]` | Constructed tier-2 instrumentation instances to register (fault-isolated: one that throws is skipped with a `[foam]` warning). |
 | `ignored_outbound_hosts:` | Array<String> | no | `[]` | Hosts whose outbound calls produce no spans — EXTENDS the built-in export-loop guard. For a co-resident agent's intake or a tenant exporter's endpoint. Applies to the clients whose official instrumentation supports host suppression: **Net::HTTP and Excon**. Faraday/HTTP (httprb)/HTTPX have no upstream `untraced_hosts` option — foam warns loudly at init when one of those is bundled (GOTCHAS G13); foam's own export loop is guarded for every client regardless. |
-| `before_send:` | Proc/Array | no | `[]` | Export hook (1.8.0): one callable — or an Array run as a pipeline — called with every **span and log record** at the export boundary, after batching and BEFORE foam's redaction pass and serialization, so nothing a hook drops ever leaves the process. Return the record to export it (mutating it in place is supported — `record.attributes` arrives unfrozen), or `nil` to drop it. A hook that raises (or returns anything else) drops **that record** loudly and the batch reports failure — fail-closed, never a raise into your threads. Runs BEFORE redaction, so the credential floor and your redact lists still apply to whatever the hook returns. Spans and logs only (metrics are aggregated state, not records). A non-callable raises at boot. Process-global: a second `init` never changes it. See "The `before_send` hook" below. |
+| `before_send:` | Proc/Array | no | `[]` | Export hook (1.8.0): one callable — or an Array run as a pipeline — called with every **span and log record** at the export boundary, after batching and BEFORE foam's redaction pass and serialization, so nothing a hook drops ever leaves the process. Return the record (or a same-type replacement) to export it — in-place mutation supported; return `nil` to drop it; a raise or foreign return drops **that record** loudly, fail-closed, never a raise into your threads. Runs BEFORE redaction — a hook can never widen what ships. Spans and logs only; door 1 only. A non-callable (or a one-arg-incompatible lambda) raises at boot. Process-global. Hooks run on the export thread — keep them non-blocking. Full contract: "The `before_send` hook" below. |
 | `capture_payloads:` | Symbol/String | no | `:off` | Inbound HTTP **body** capture on the rack server span (see "Payload (body) capture"). `:off` = zero teeing, zero middleware (the default); `:errors` = tee per request, attach only on exception/5xx; `:always` = attach on every request. Equivalent strings accepted; anything else raises at boot. Overridden in both directions by the `FOAM_CAPTURE_PAYLOADS` env clamp (env table below). Textual payloads only, 8192-char cap + `…[truncated]`, true `.size` attributes; sizes-only for binary/compressed. |
 | `diagnostics:` | Boolean | no | false | Verbose `[foam]` self-reporting of init/health. Warnings and errors are always loud regardless. |
 | `secret_heuristics:` | Boolean | no | true | The value-pattern secret layer's HEURISTIC tier (generic keyword+entropy detection, see "The value-pattern secret layer"). `false` disables ONLY the heuristics — the named provider patterns, the credential floor and the redaction-coverage contract have no off switch. Disabling logs one loud `[foam]` line at init (an explicit, audited decision for telemetry whose legitimate values collide with the heuristics). |
@@ -661,17 +661,42 @@ set.
 
 The per-record contract (Sentry `beforeSend` semantics):
 
-* **return the record** → it continues to the next hook / to export.
-  Mutating it in place is supported — `record.attributes` arrives as an
-  unfrozen copy, so `record.attributes["k"] = v` and
-  `record.attributes.delete("k")` just work.
+* **return the record — or a replacement of the same type** → it continues
+  to the next hook / to export. Mutating it in place is supported —
+  `record.attributes` arrives as your own unfrozen copy (values included),
+  so `record.attributes["k"] = v`, `record.attributes.delete("k")`,
+  `record.attributes["k"].gsub!(...)` and (for logs) editing `record.body`
+  all just work without bleeding into the span your app still holds or the
+  view a tenant processor saw. Events, links and status are the exception:
+  **replace them wholesale** (`record.events = [...]`) — never mutate them
+  in place, they are shared structures. Foam re-normalizes the record's
+  `total_recorded_*` bookkeeping after your hooks run, so adding/deleting
+  attributes never corrupts the wire's dropped-count accounting.
 * **return `nil`** → the record is dropped, silently (that is the filter
   mechanism, not an error).
-* **raise, or return anything else** → that record is dropped **loudly**
-  (`[foam]` warning) and the batch reports failure — fail-closed: a faulted
-  hook never ships a record you may have meant to scrub, and never raises
-  into your application or export threads. Healthy records in the same batch
-  still export.
+* **raise, or return a foreign object** → that record is dropped
+  **loudly** — one aggregated `[foam]` warning per batch, error classes
+  only (messages can carry your data) — and the batch reports failure:
+  fail-closed, a faulted hook never ships a record you may have meant to
+  scrub, and never raises into your application or export threads. Healthy
+  records in the same batch still export. Because a fault marks the whole
+  batch failed, a hook that faults **deterministically** also makes the
+  at-exit flush report failure — fix faulting hooks promptly (GOTCHAS F16).
+
+**Hooks run on foam's export thread, inside the batch processor's export
+lock.** Keep them fast and non-blocking: no network calls, no DB lookups,
+no sleeping — a blocked hook stalls the export pipeline and can wedge the
+at-exit flush. Anything a hook needs from the request/job thread
+(tenant context, user identity) must be stamped onto the record as an
+attribute at capture time (see the `additional_span_processors` stamper
+pattern); thread-locals are gone by the time the hook runs.
+
+Boot validation is strict (rule 10): a non-callable — or a **lambda/Method
+whose signature cannot accept one positional argument** (e.g. a Sentry-port
+`->(event, hint) { ... }` with two required params, or a required keyword)
+— raises `ArgumentError` at `init`, because it would otherwise fault on
+every record and ship zero telemetry from a green boot. Make the extra
+Sentry `hint` param optional (`->(event, hint = nil)`) and it is accepted.
 
 Scope: **spans and log records** (the two record-shaped signals — the hook
 receives `SpanData` or `LogRecordData`; distinguish them with `is_a?` or

data/lib/foam/otel/before_send.rb

@@ -1,5 +1,6 @@
 # frozen_string_literal: true
 
+require_relative "constants"
 require_relative "diagnostics"
 
 module Foam
@@ -53,37 +54,134 @@ module Foam
 
       # Run the hook pipeline over one record: each hook receives the
       # previous hook's return. Returns the surviving record, nil for an
-      # intentional drop, or ERROR for a faulted one.
-      def run(hooks, record, signal)
-        current = thaw(record)
+      # intentional drop, or ERROR for a faulted one. Fault DETAIL is
+      # appended to `faults` (error class / return class only — never a
+      # message, which can embed record values) so the exporter can warn
+      # ONCE per batch instead of once per record (the redacting exporters'
+      # aggregation convention; an unthrottled per-record warn is a stderr
+      # flood an adversary-shaped record stream controls — rule 15's
+      # "loud but never a flood" posture).
+      def run(hooks, record, signal, faults)
+        current, counters = thaw(record)
         hooks.each do |hook|
           result = hook.call(current)
           return nil if result.nil?
 
           unless result.is_a?(record.class)
-            Diagnostics.warn("before_send returned a #{result.class} for a #{signal} record — record " \
-                             "dropped (return the record, a replacement of the same type, or nil to drop; " \
-                             "fail-closed, rule 14)")
+            faults << "hook returned #{result.class}"
             return ERROR
           end
           current = result
         end
+        restore_counters!(current, counters)
         current
       rescue StandardError, SystemStackError => e
-        Diagnostics.warn("before_send raised for a #{signal} record — record dropped, never exported " \
-                         "half-transformed (fail-closed, rule 14): #{e.class}: #{e.message}")
+        faults << e.class.name
+        # Full detail (message included) only under diagnostics: true —
+        # e.message can carry attribute values.
+        Diagnostics.info("before_send fault detail (#{signal}): #{e.class}: #{e.message}")
         ERROR
       end
 
-      # A shallow dup with an unfrozen attributes copy — the hook gets its
-      # own mutable record (span attributes arrive frozen; GOTCHAS F1).
+      # A dup the hook can safely mutate IN PLACE: the struct and the
+      # attributes hash are copied, and so are the mutable attribute VALUES
+      # (strings, and strings inside string arrays — the OTel attribute
+      # types) plus a structured log body — so `record.attributes["k"] = v`,
+      # `attributes["k"].gsub!(...)` and `record.body["k"] = v` never bleed
+      # into the live span/record the customer still holds or the masked
+      # view a tenant processor buffered (pipelines.rb), and never hit a
+      # frozen value. Events/links/status are NOT deep-copied: replace them
+      # wholesale (`record.events = [...]`), never mutate them in place —
+      # documented in the README.
+      #
+      # Also returns the record's SDK-drop counters, captured BEFORE the
+      # hooks run: the OTLP exporters encode dropped_attributes_count as
+      # `total_recorded_attributes - attributes.size` (a protobuf uint32),
+      # so a hook that ADDS an attribute without the total being fixed up
+      # afterwards makes the count NEGATIVE and the encoder drops the WHOLE
+      # export request — every healthy record in the batch, silently (the
+      # same trap session_stitching.rb documents and guards for foam's own
+      # added attributes). restore_counters! below re-normalizes the
+      # survivor from these.
       def thaw(record)
         copy = record.dup
         if copy.respond_to?(:attributes) && copy.respond_to?(:attributes=)
           attributes = copy.attributes
-          copy.attributes = attributes.nil? ? {} : attributes.dup
+          copy.attributes = attributes.nil? ? {} : attributes.transform_values { |v| thaw_value(v) }
         end
-        copy
+        copy.body = thaw_body(copy.body) if copy.respond_to?(:body) && copy.respond_to?(:body=)
+        [copy, sdk_drop_counters(record)]
+      end
+
+      def thaw_value(value)
+        case value
+        when String then value.dup
+        when Array then value.map { |e| e.is_a?(String) ? e.dup : e }
+        else value
+        end
+      end
+
+      # Bounded deep copy of a log body (String / Hash / Array / scalar) so
+      # in-place body mutation is safe too; the depth guard mirrors the
+      # redaction engine's (rule 9 — an adversarial structure never drives
+      # a stack overflow).
+      def thaw_body(body, depth = 0)
+        return body if depth > MAX_REDACT_DEPTH
+
+        case body
+        when String then body.dup
+        when Hash then body.each_with_object({}) { |(k, v), out| out[k] = thaw_body(v, depth + 1) }
+        when Array then body.map { |v| thaw_body(v, depth + 1) }
+        else body
+        end
+      end
+
+      # How many attributes/events/links the SDK itself dropped (limits)
+      # BEFORE the hooks ran — nil-safe, clamped at zero.
+      def sdk_drop_counters(record)
+        {
+          attributes: sdk_dropped(record, :total_recorded_attributes, :attributes),
+          events: sdk_dropped(record, :total_recorded_events, :events),
+          links: sdk_dropped(record, :total_recorded_links, :links),
+        }
+      end
+
+      def sdk_dropped(record, total_getter, list_getter)
+        return 0 unless record.respond_to?(total_getter)
+
+        total = record.public_send(total_getter).to_i
+        size = collection_size(record, list_getter)
+        total > size ? total - size : 0
+      end
+
+      def collection_size(record, list_getter)
+        return 0 unless record.respond_to?(list_getter)
+
+        list = record.public_send(list_getter)
+        list.respond_to?(:size) ? list.size.to_i : 0
+      rescue StandardError
+        0
+      end
+
+      # Re-normalize the survivor's total_recorded_* bookkeeping after the
+      # hooks ran: total = what the record NOW carries + what the SDK had
+      # already dropped. Without this, a hook-ADDED attribute makes the
+      # OTLP encoder's `total - size` subtraction negative (protobuf uint32
+      # → RangeError → the encoder returns nil → the ENTIRE batch is
+      # dropped, silently, forever), and a hand-built replacement record
+      # with nil totals raises NoMethodError at the same line. Also keeps a
+      # hook DELETE honest (the wire no longer claims an SDK-limit drop).
+      def restore_counters!(record, counters)
+        if record.respond_to?(:total_recorded_attributes=)
+          record.total_recorded_attributes = collection_size(record, :attributes) + counters[:attributes]
+        end
+        if record.respond_to?(:total_recorded_events=)
+          record.total_recorded_events = collection_size(record, :events) + counters[:events]
+        end
+        if record.respond_to?(:total_recorded_links=)
+          record.total_recorded_links = collection_size(record, :links) + counters[:links]
+        end
+        record
       end
 
       # The span-side wrapper (composition over the redacting exporter, same
@@ -97,18 +195,19 @@ module Foam
 
         def export(span_data, timeout: nil)
           batch = span_data.to_a
-          faulted = 0
+          faults = []
           kept = batch.filter_map do |sd|
-            result = BeforeSend.run(@hooks, sd, :span)
-            if result.equal?(ERROR)
-              faulted += 1
-              nil
-            else
-              result
-            end
+            result = BeforeSend.run(@hooks, sd, :span, faults)
+            result.equal?(ERROR) ? nil : result
+          end
+          # ONE aggregated warn per batch (the redacting exporters'
+          # convention) — class names only, never messages/values.
+          unless faults.empty?
+            Diagnostics.warn("before_send faulted for #{faults.length} span record(s) — dropped, never " \
+                             "exported half-transformed (fail-closed, rule 14): #{faults.uniq.take(3).join(', ')}")
           end
           status = kept.empty? ? OpenTelemetry::SDK::Trace::Export::SUCCESS : @inner.export(kept, timeout: timeout)
-          faulted.positive? ? OpenTelemetry::SDK::Trace::Export::FAILURE : status
+          faults.empty? ? status : OpenTelemetry::SDK::Trace::Export::FAILURE
         rescue StandardError, SystemStackError
           # Fail closed past the per-record guards, loudly (rule 15) —
           # never raise into the batch thread.
@@ -128,18 +227,17 @@ module Foam
 
         def export(log_record_data, timeout: nil)
           batch = log_record_data.to_a
-          faulted = 0
+          faults = []
           kept = batch.filter_map do |lrd|
-            result = BeforeSend.run(@hooks, lrd, :log)
-            if result.equal?(ERROR)
-              faulted += 1
-              nil
-            else
-              result
-            end
+            result = BeforeSend.run(@hooks, lrd, :log, faults)
+            result.equal?(ERROR) ? nil : result
+          end
+          unless faults.empty?
+            Diagnostics.warn("before_send faulted for #{faults.length} log record(s) — dropped, never " \
+                             "exported half-transformed (fail-closed, rule 14): #{faults.uniq.take(3).join(', ')}")
           end
           status = kept.empty? ? OpenTelemetry::SDK::Logs::Export::SUCCESS : @inner.export(kept, timeout: timeout)
-          faulted.positive? ? OpenTelemetry::SDK::Logs::Export::FAILURE : status
+          faults.empty? ? status : OpenTelemetry::SDK::Logs::Export::FAILURE
         rescue StandardError, SystemStackError
           Diagnostics.warn("before_send pass raised past its guards — log batch dropped (fail-closed, rule 14/15)")
           OpenTelemetry::SDK::Logs::Export::FAILURE

data/lib/foam/otel/config.rb

@@ -108,22 +108,65 @@ module Foam
       # #call — a lambda, a Proc, a Method, or any hook object) or an Array
       # of callables run as a pipeline. Anything else — including a nil or
       # non-callable INSIDE the array — raises here, never a silent no-op
-      # that leaves the customer believing their filter runs.
+      # that leaves the customer believing their filter runs. The array is
+      # COPIED before freezing (never freeze the caller's own object).
+      #
+      # Arity is validated too, where Ruby lets us introspect it: a lambda
+      # or Method that cannot accept ONE positional argument (a Sentry-port
+      # `->(event, hint) { }`, a forgotten-param `-> { }`, a required
+      # keyword) would pass a bare respond_to?(:call) check and then fault
+      # on EVERY record at export — a green boot that ships zero telemetry
+      # forever. That is exactly the boot-detectable programming error rule
+      # 10 exists for. Non-lambda Procs are arity-forgiving by language
+      # semantics and uninspectable callables get the benefit of the doubt.
       def validate_before_send!(value, context: "Foam::Otel.init")
         hooks = if value.nil?
                   []
                 else
-                  value.is_a?(Array) ? value : [value]
+                  value.is_a?(Array) ? value.dup : [value]
                 end
         hooks.each_with_index do |hook, index|
-          next if hook.respond_to?(:call)
+          unless hook.respond_to?(:call)
+            raise ArgumentError, "#{context} before_send: must be a callable (responds to #call) or an " \
+                                 "Array of callables, got #{hook.class} at position #{index}"
+          end
+          next if before_send_arity_ok?(hook)
 
-          raise ArgumentError, "#{context} before_send: must be a callable (responds to #call) or an " \
-                               "Array of callables, got #{hook.class} at position #{index}"
+          raise ArgumentError, "#{context} before_send: the callable at position #{index} cannot be " \
+                               "called with one record argument (it takes a different number of " \
+                               "required positional parameters, or requires keywords) — before_send " \
+                               "hooks are called as hook.call(record)"
         end
         hooks.freeze
       end
 
+      # True when the hook can be invoked as hook.call(record). Lambdas and
+      # Methods enforce their signatures, so they are checked; non-lambda
+      # Procs coerce arguments (always fine); anything whose #call cannot
+      # be introspected passes (never a false boot failure).
+      def before_send_arity_ok?(hook)
+        params = if hook.is_a?(Proc)
+                   return true unless hook.lambda?
+
+                   hook.parameters
+                 elsif hook.is_a?(Method) || hook.is_a?(UnboundMethod)
+                   hook.parameters
+                 else
+                   begin
+                     hook.method(:call).parameters
+                   rescue StandardError
+                     return true
+                   end
+                 end
+        return false if params.any? { |kind, _| kind == :keyreq }
+
+        required = params.count { |kind, _| kind == :req }
+        flexible = params.any? { |kind, _| kind == :opt || kind == :rest }
+        required == 1 || (required.zero? && flexible)
+      rescue StandardError
+        true
+      end
+
       # ---- the capture_payloads option (payload-capture mandate 2026-07-28)
       # Canonicalize a mode value: the three symbols, or their string
       # spellings trimmed + case-insensitive, map to the canonical symbol;

data/lib/foam/otel/ingest.rb

@@ -285,23 +285,25 @@ module Foam
         # CWE-295 — see pipelines.rb setup_traces): door 2 carries the same
         # fleet ingest token over the same TLS connection, so the silent
         # OTEL_RUBY_EXPORTER_OTLP_SSL_VERIFY_NONE downgrade is pinned out
-        # here identically.
+        # here identically. All three are IsolatedExporters subclasses —
+        # door 2's delivery must be exactly as immune to foreign Net::HTTP
+        # prepends as door 1's (export-isolation design, 2026-07-29).
         def build_otlp_span_exporter(endpoint, headers)
-          OpenTelemetry::Exporter::OTLP::Exporter.new(
+          IsolatedExporters::TraceExporter.new(
             endpoint: "#{endpoint}/v1/traces", headers: headers,
             ssl_verify_mode: OpenSSL::SSL::VERIFY_PEER
           )
         end
 
         def build_otlp_log_exporter(endpoint, headers)
-          OpenTelemetry::Exporter::OTLP::Logs::LogsExporter.new(
+          IsolatedExporters::LogsExporter.new(
             endpoint: "#{endpoint}/v1/logs", headers: headers,
             ssl_verify_mode: OpenSSL::SSL::VERIFY_PEER
           )
         end
 
         def build_otlp_metric_exporter(endpoint, headers)
-          OpenTelemetry::Exporter::OTLP::Metrics::MetricsExporter.new(
+          IsolatedExporters::MetricsExporter.new(
             endpoint: "#{endpoint}/v1/metrics", headers: headers,
             ssl_verify_mode: OpenSSL::SSL::VERIFY_PEER
           )

data/lib/foam/otel/init.rb

@@ -528,6 +528,10 @@ module Foam
         require "opentelemetry-exporter-otlp"
         require "opentelemetry-exporter-otlp-logs"
         require "opentelemetry-exporter-otlp-metrics"
+        # The isolated transport subclasses (export-isolation design) load
+        # HERE — after the upstream exporter gems they subclass — so the lazy
+        # rule-41 posture holds: a disabled boot never pays their cost.
+        require_relative "isolated_exporters"
       end
 
       # The central RAW-capture activation config (raised-floor ruling
@@ -610,6 +614,12 @@ module Foam
           FLOOR_INSTRUMENTATION_CONFIG.merge(SUPERSEDED_INSTRUMENTATION_CONFIG)
         )
         Diagnostics.info("instrumentations installed: #{gems.length} bundled gem(s)")
+        # Coexistence shield (2026-07-29, GOTCHAS F18): with the sweep done,
+        # every prepend that will sit on Net::HTTP in the realistic boot
+        # order (agent first, foam second) is in place — scan for cross-
+        # vendor private-helper collisions and shim them. A collision-free
+        # chain (the common case) installs NOTHING.
+        PrependCollisionShield.install!(::Net::HTTP) if defined?(::Net::HTTP)
       rescue StandardError => e
         Diagnostics.warn("instrumentation activation failed: #{e.class}: #{e.message}")
       end