foam-otel 1.7.0 → 1.8.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 48166d05b120dd8bdca15a9b45db5b4c16a6574fcb6890f4caff2c17c2ae9add
-  data.tar.gz: 2dcd4b3b146a0221c231ba5efc057085418b74e21b24d8441396d47cfc0a8288
+  metadata.gz: 6ef766d3509a7ee5daa25c7f53dce352b9c67c8ed22283099c72f3d6f0b8e957
+  data.tar.gz: caca5adca96f4e3dab3d0610ce54000858cb231cbf167e5bb6dd0c6f1dcd6a9b
 SHA512:
-  metadata.gz: 4e92c7af8cc5cf7cc177b792f87a8107b918035efdce080e2a974750180ebbb9aac4a5460cb08372f3ded2724d136804bcb8f0574ee4685ebbdd94fde7afb3e8
-  data.tar.gz: ddd7b6e431034663ec94dacd8b7b051ee63bfecdbeb2a38d383c58531854ba0580f34c9a04ed56473b654b89155fc9a3b7ce140b073ba346d90a5a801b9c6d4b
+  metadata.gz: 55da8fd55a07d287d81fddf5a65eeb8b7e0f743d1ec8b034362972f5c02a15a84bab18119aacf9e881b4deefc61d6958e062990fe3b205f66f2a88fd11ba514b
+  data.tar.gz: 70e46934c6250308cb8d15c90a160eeab5ce916b294cfc7395e46fc07e0d8d2027ded17f3f21cdcf3125c35773fb899324eb0374c6326c04d73a2908e0e1024d

data/GOTCHAS.md

@@ -808,6 +808,67 @@ exfiltratable — the value-pattern secret layer is the required second control
 
 ---
 
+## F16: before_send hooks run inside the export lock, and record bookkeeping must be re-normalized behind them
+
+- **Trap**: Three traps share the seam. (1) The batch processors call
+  `exporter.export` inside `@export_mutex.synchronize` with no lock timeout —
+  a customer `before_send` hook that blocks (DB lookup, un-timed HTTP, sleep)
+  wedges the worker thread AND any concurrent `force_flush`/`shutdown`
+  acquiring the same mutex: the process can hang at exit until SIGKILL.
+  (2) The OTLP exporters encode `dropped_attributes_count` as
+  `total_recorded_attributes - attributes.size` into a protobuf **uint32** —
+  a hook that ADDS an attribute without the total being fixed up makes the
+  subtraction negative, protobuf raises `RangeError`, `encode` rescues to
+  nil, and the WHOLE export request — every healthy record batched beside
+  the touched one — is dropped, silently, forever (the same trap
+  session_stitching already documents for foam's own stamped attributes; a
+  hand-built replacement record with nil totals is the NoMethodError variant
+  of the same encode failure). (3) A hook fault marks the whole batch
+  FAILURE (redacting-exporter parity), and `force_flush` aborts its drain
+  loop on the first non-SUCCESS batch — so a hook that faults
+  DETERMINISTICALLY on some record shape makes the at-exit flush abandon
+  everything queued behind the first faulted batch.
+- **Sources**:
+  - Installed source: `OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor`
+    `#export_batch` inside `@export_mutex.synchronize`
+    (opentelemetry-sdk `trace/export/batch_span_processor.rb:189`), same shape
+    in `logs/export/batch_log_record_processor.rb:186`; `force_flush`'s
+    `return result_code unless result_code == SUCCESS` drain-abort
+    (`batch_span_processor.rb:113-115`).
+  - Installed source: `dropped_attributes_count: span_data.total_recorded_attributes - span_data.attributes&.size.to_i`
+    (opentelemetry-exporter-otlp `exporter.rb:343`; logs twin
+    `logs_exporter.rb:311`); `encode` rescues `StandardError` → nil →
+    `send_bytes(nil)` returns FAILURE for the whole request.
+  - In-repo precedent: `lib/foam/otel/session_stitching.rb` bumps
+    `@total_recorded_attributes` when foam adds its own attribute, guarding
+    against exactly trap (2).
+- **Decision & why**: Hooks stay ON the export thread (any queue/thread
+  indirection would break the drop-before-serialize guarantee and add an
+  unbounded buffer); foam re-normalizes `total_recorded_*` on every
+  surviving record after the pipeline (`BeforeSend.restore_counters!` —
+  size + SDK-dropped delta captured pre-hook, nil-safe), hands hooks COPIED
+  attribute values and log bodies so in-place mutation cannot bleed into
+  the live record or a tenant's buffered view, and keeps FAILURE accounting
+  for faulted records (parity with the redacting exporters) while warning
+  ONCE per batch with error classes only. The blocking-hook risk is
+  accepted and documented (README "The `before_send` hook") — it is the
+  customer's own code on their own pipeline, the same trust grade as a
+  tenant processor, and rule 49's DoS bounds govern foam's code, not the
+  customer's.
+- **Mitigation**: README pins the contract: non-blocking hooks only; stamp
+  request/job-thread context onto attributes at capture; replace (never
+  mutate) events/links/status; deterministic-fault hooks are a customer bug
+  surfaced by the aggregated per-batch `[foam]` warning + batch FAILURE.
+  Boot-time arity validation rejects hooks that cannot take one positional
+  argument (the Sentry two-arg port) so the fault-every-record shape is
+  caught on the engineer's machine (rule 10).
+- **Test**: `spec/before_send_spec.rb` ("re-normalizes total_recorded",
+  "REAL OTLP encode", "aggregates hook faults", "arity", "in-place value
+  mutation never bleeds"); `spec/before_send_champ_scenario_spec.rb` (the
+  export-thread / stamped-context proof).
+
+---
+
 ## General gotchas (applicable to Ruby)
 
 - **G1 — init after target import / pre-init no-op**: the API's proxy providers

data/README.md

@@ -331,6 +331,7 @@ not captured — inbound only.
 | `additional_metric_readers:` | Array | no | `[]` | Tenant seam, metrics. |
 | `additional_instrumentations:` | Array | no | `[]` | Constructed tier-2 instrumentation instances to register (fault-isolated: one that throws is skipped with a `[foam]` warning). |
 | `ignored_outbound_hosts:` | Array<String> | no | `[]` | Hosts whose outbound calls produce no spans — EXTENDS the built-in export-loop guard. For a co-resident agent's intake or a tenant exporter's endpoint. Applies to the clients whose official instrumentation supports host suppression: **Net::HTTP and Excon**. Faraday/HTTP (httprb)/HTTPX have no upstream `untraced_hosts` option — foam warns loudly at init when one of those is bundled (GOTCHAS G13); foam's own export loop is guarded for every client regardless. |
+| `before_send:` | Proc/Array | no | `[]` | Export hook (1.8.0): one callable — or an Array run as a pipeline — called with every **span and log record** at the export boundary, after batching and BEFORE foam's redaction pass and serialization, so nothing a hook drops ever leaves the process. Return the record (or a same-type replacement) to export it — in-place mutation supported; return `nil` to drop it; a raise or foreign return drops **that record** loudly, fail-closed, never a raise into your threads. Runs BEFORE redaction — a hook can never widen what ships. Spans and logs only; door 1 only. A non-callable (or a one-arg-incompatible lambda) raises at boot. Process-global. Hooks run on the export thread — keep them non-blocking. Full contract: "The `before_send` hook" below. |
 | `capture_payloads:` | Symbol/String | no | `:off` | Inbound HTTP **body** capture on the rack server span (see "Payload (body) capture"). `:off` = zero teeing, zero middleware (the default); `:errors` = tee per request, attach only on exception/5xx; `:always` = attach on every request. Equivalent strings accepted; anything else raises at boot. Overridden in both directions by the `FOAM_CAPTURE_PAYLOADS` env clamp (env table below). Textual payloads only, 8192-char cap + `…[truncated]`, true `.size` attributes; sizes-only for binary/compressed. |
 | `diagnostics:` | Boolean | no | false | Verbose `[foam]` self-reporting of init/health. Warnings and errors are always loud regardless. |
 | `secret_heuristics:` | Boolean | no | true | The value-pattern secret layer's HEURISTIC tier (generic keyword+entropy detection, see "The value-pattern secret layer"). `false` disables ONLY the heuristics — the named provider patterns, the credential floor and the redaction-coverage contract have no off switch. Disabling logs one loud `[foam]` line at init (an explicit, audited decision for telemetry whose legitimate values collide with the heuristics). |
@@ -388,6 +389,24 @@ Foam::Otel.init(name: "checkout-api", environment: env, enabled: true, token: EN
                 ignored_outbound_hosts: ["ingest.eval-tool.example"]) # the tenant loop guard — required
 ```
 
+```ruby
+# The before_send export hook (1.8.0): transform or drop spans/logs right
+# before they are exported to foam — after batching, BEFORE redaction and
+# serialization (a dropped record never leaves the process). Return the
+# record (mutate in place freely) or nil to drop; a raising hook drops ONLY
+# its record, loudly, and never raises into your threads (fail-closed).
+Foam::Otel.init(name: "checkout-api", environment: env, enabled: true, token: ENV.fetch("FOAM_OTEL_TOKEN"),
+                before_send: lambda { |record|
+                  next nil if record.respond_to?(:name) && record.name == "GET /healthz" # drop noise
+
+                  record.attributes.delete("internal.debug.blob") # trim before it ships
+                  record
+                })
+# Or an Array pipeline — hooks run in order, each receiving the previous
+# hook's return; the first nil drops the record and short-circuits:
+# Foam::Otel.init(..., before_send: [DropHealthchecks, TrimDebugAttrs, StampTeam])
+```
+
 ```ruby
 # Verbose self-reporting while wiring foam up (warnings are loud regardless):
 Foam::Otel.init(name: "checkout-api", environment: "development", enabled: true,
@@ -616,6 +635,79 @@ processor = Foam::Otel.create_ingest_span_processor(
 
 ---
 
+## The `before_send` hook
+
+`before_send:` is the last word your code gets on every span and log record
+before it is exported to foam. It runs at the **export boundary** — after the
+batch processor hands foam the finished records, immediately **before**
+foam's redaction pass and the OTLP serialization — so a record your hook
+drops is never serialized and never leaves the process. (This is the same
+seam redaction lives at, and for the same Ruby SDK reason: a span's
+attributes freeze at finish before any `on_finish` processor runs, so a
+`SpanProcessor` could neither mutate nor drop. The export boundary is the
+first point where the record is a mutable struct.)
+
+The pipeline order is pinned:
+
+```
+BatchProcessor → before_send hooks → redaction (floor + your lists) → OTLP wire
+```
+
+Your hooks run **first**, on the raw record (it is your own data, still
+in-process), and redaction runs on whatever they return — so `before_send`
+can never widen what ships: an attribute a hook adds is masked by the
+credential floor and your redact lists exactly like one an instrumentation
+set.
+
+The per-record contract (Sentry `beforeSend` semantics):
+
+* **return the record — or a replacement of the same type** → it continues
+  to the next hook / to export. Mutating it in place is supported —
+  `record.attributes` arrives as your own unfrozen copy (values included),
+  so `record.attributes["k"] = v`, `record.attributes.delete("k")`,
+  `record.attributes["k"].gsub!(...)` and (for logs) editing `record.body`
+  all just work without bleeding into the span your app still holds or the
+  view a tenant processor saw. Events, links and status are the exception:
+  **replace them wholesale** (`record.events = [...]`) — never mutate them
+  in place, they are shared structures. Foam re-normalizes the record's
+  `total_recorded_*` bookkeeping after your hooks run, so adding/deleting
+  attributes never corrupts the wire's dropped-count accounting.
+* **return `nil`** → the record is dropped, silently (that is the filter
+  mechanism, not an error).
+* **raise, or return a foreign object** → that record is dropped
+  **loudly** — one aggregated `[foam]` warning per batch, error classes
+  only (messages can carry your data) — and the batch reports failure:
+  fail-closed, a faulted hook never ships a record you may have meant to
+  scrub, and never raises into your application or export threads. Healthy
+  records in the same batch still export. Because a fault marks the whole
+  batch failed, a hook that faults **deterministically** also makes the
+  at-exit flush report failure — fix faulting hooks promptly (GOTCHAS F16).
+
+**Hooks run on foam's export thread, inside the batch processor's export
+lock.** Keep them fast and non-blocking: no network calls, no DB lookups,
+no sleeping — a blocked hook stalls the export pipeline and can wedge the
+at-exit flush. Anything a hook needs from the request/job thread
+(tenant context, user identity) must be stamped onto the record as an
+attribute at capture time (see the `additional_span_processors` stamper
+pattern); thread-locals are gone by the time the hook runs.
+
+Boot validation is strict (rule 10): a non-callable — or a **lambda/Method
+whose signature cannot accept one positional argument** (e.g. a Sentry-port
+`->(event, hint) { ... }` with two required params, or a required keyword)
+— raises `ArgumentError` at `init`, because it would otherwise fault on
+every record and ship zero telemetry from a green boot. Make the extra
+Sentry `hint` param optional (`->(event, hint = nil)`) and it is accepted.
+
+Scope: **spans and log records** (the two record-shaped signals — the hook
+receives `SpanData` or `LogRecordData`; distinguish them with `is_a?` or
+`respond_to?`). Metrics are aggregated state, not discrete records, and are
+deliberately not routed through `before_send`. Door-2 ingest taps are
+unchanged. Like `enabled:` and the endpoint, `before_send` is process-global:
+the exporter chain is built at the first `init`, and a second `init` never
+changes it.
+
+---
+
 ## The helpers
 
 All helpers never raise, and no-op silently before `init` and when disabled.

data/lib/foam/otel/before_send.rb

@@ -0,0 +1,251 @@
+# frozen_string_literal: true
+
+require_relative "constants"
+require_relative "diagnostics"
+
+module Foam
+  module Otel
+    # The customer's `before_send:` hook(s) (1.8.0), run at the EXPORT
+    # BOUNDARY — after batching, immediately BEFORE foam's redaction pass and
+    # the OTLP serialization, so a record a hook drops is never serialized
+    # and never leaves the process. Why not a SpanProcessor: the same Ruby
+    # SDK constraint that forced redaction to the exporter boundary — a span
+    # freezes its attributes at finish BEFORE any on_finish processor runs
+    # (GOTCHAS F1), so a processor could neither mutate nor reliably drop.
+    # The mutable SpanData / LogRecordData Structs the batch processors hand
+    # the exporter are the first (and last) point where per-record transform
+    # + drop is possible before the wire.
+    #
+    # Composition order (pinned, pipelines.rb):
+    #   BatchProcessor -> BeforeSend::*Exporter -> Redacting*Exporter -> OTLP
+    # The hooks run FIRST so the credential floor, the value-pattern secret
+    # layer and the customer's key lists still apply to whatever a hook
+    # returns — before_send can never widen what ships past redaction (an
+    # attribute a hook ADDS is masked exactly like one an instrumentation
+    # set).
+    #
+    # Contract per record (Sentry beforeSend semantics — the spec's named
+    # fail-closed precedent, BASE_PACKAGE_SPEC D6):
+    #   * return the record (mutating it in place is fine) → next hook /
+    #     export;
+    #   * return nil → record DROPPED (the intentional filter mechanism —
+    #     success, not a failure);
+    #   * raise, or return a foreign object → record DROPPED, loudly, and
+    #     the batch reports FAILURE (fail-closed: a hook fault must never
+    #     ship a record the customer may have meant to scrub, and must never
+    #     raise into the export thread — rules 9/14/15). SystemStackError is
+    #     rescued explicitly (not a StandardError): a hook recursing on a
+    #     poisoned payload kills neither the batch thread nor the caller.
+    # The record reaches the first hook as a shallow dup with an UNFROZEN
+    # attributes copy (span attributes freeze at finish — GOTCHAS F1), so
+    # `record.attributes["k"] = v` just works; the dup also keeps the
+    # customer's mutations off the struct any other consumer might hold.
+    #
+    # Scope: spans and logs (the two record-shaped signals). Metric data is
+    # aggregated state, not records — deliberately NOT routed through
+    # before_send (documented in the README).
+    module BeforeSend
+      # Internal sentinel distinguishing a FAULTED drop (counts toward batch
+      # FAILURE) from an intentional nil drop. A hook can never return it
+      # legitimately — any non-record return is itself classified a fault.
+      ERROR = Object.new.freeze
+
+      module_function
+
+      # Run the hook pipeline over one record: each hook receives the
+      # previous hook's return. Returns the surviving record, nil for an
+      # intentional drop, or ERROR for a faulted one. Fault DETAIL is
+      # appended to `faults` (error class / return class only — never a
+      # message, which can embed record values) so the exporter can warn
+      # ONCE per batch instead of once per record (the redacting exporters'
+      # aggregation convention; an unthrottled per-record warn is a stderr
+      # flood an adversary-shaped record stream controls — rule 15's
+      # "loud but never a flood" posture).
+      def run(hooks, record, signal, faults)
+        current, counters = thaw(record)
+        hooks.each do |hook|
+          result = hook.call(current)
+          return nil if result.nil?
+
+          unless result.is_a?(record.class)
+            faults << "hook returned #{result.class}"
+            return ERROR
+          end
+          current = result
+        end
+        restore_counters!(current, counters)
+        current
+      rescue StandardError, SystemStackError => e
+        faults << e.class.name
+        # Full detail (message included) only under diagnostics: true —
+        # e.message can carry attribute values.
+        Diagnostics.info("before_send fault detail (#{signal}): #{e.class}: #{e.message}")
+        ERROR
+      end
+
+      # A dup the hook can safely mutate IN PLACE: the struct and the
+      # attributes hash are copied, and so are the mutable attribute VALUES
+      # (strings, and strings inside string arrays — the OTel attribute
+      # types) plus a structured log body — so `record.attributes["k"] = v`,
+      # `attributes["k"].gsub!(...)` and `record.body["k"] = v` never bleed
+      # into the live span/record the customer still holds or the masked
+      # view a tenant processor buffered (pipelines.rb), and never hit a
+      # frozen value. Events/links/status are NOT deep-copied: replace them
+      # wholesale (`record.events = [...]`), never mutate them in place —
+      # documented in the README.
+      #
+      # Also returns the record's SDK-drop counters, captured BEFORE the
+      # hooks run: the OTLP exporters encode dropped_attributes_count as
+      # `total_recorded_attributes - attributes.size` (a protobuf uint32),
+      # so a hook that ADDS an attribute without the total being fixed up
+      # afterwards makes the count NEGATIVE and the encoder drops the WHOLE
+      # export request — every healthy record in the batch, silently (the
+      # same trap session_stitching.rb documents and guards for foam's own
+      # added attributes). restore_counters! below re-normalizes the
+      # survivor from these.
+      def thaw(record)
+        copy = record.dup
+        if copy.respond_to?(:attributes) && copy.respond_to?(:attributes=)
+          attributes = copy.attributes
+          copy.attributes = attributes.nil? ? {} : attributes.transform_values { |v| thaw_value(v) }
+        end
+        copy.body = thaw_body(copy.body) if copy.respond_to?(:body) && copy.respond_to?(:body=)
+        [copy, sdk_drop_counters(record)]
+      end
+
+      def thaw_value(value)
+        case value
+        when String then value.dup
+        when Array then value.map { |e| e.is_a?(String) ? e.dup : e }
+        else value
+        end
+      end
+
+      # Bounded deep copy of a log body (String / Hash / Array / scalar) so
+      # in-place body mutation is safe too; the depth guard mirrors the
+      # redaction engine's (rule 9 — an adversarial structure never drives
+      # a stack overflow).
+      def thaw_body(body, depth = 0)
+        return body if depth > MAX_REDACT_DEPTH
+
+        case body
+        when String then body.dup
+        when Hash then body.each_with_object({}) { |(k, v), out| out[k] = thaw_body(v, depth + 1) }
+        when Array then body.map { |v| thaw_body(v, depth + 1) }
+        else body
+        end
+      end
+
+      # How many attributes/events/links the SDK itself dropped (limits)
+      # BEFORE the hooks ran — nil-safe, clamped at zero.
+      def sdk_drop_counters(record)
+        {
+          attributes: sdk_dropped(record, :total_recorded_attributes, :attributes),
+          events: sdk_dropped(record, :total_recorded_events, :events),
+          links: sdk_dropped(record, :total_recorded_links, :links),
+        }
+      end
+
+      def sdk_dropped(record, total_getter, list_getter)
+        return 0 unless record.respond_to?(total_getter)
+
+        total = record.public_send(total_getter).to_i
+        size = collection_size(record, list_getter)
+        total > size ? total - size : 0
+      end
+
+      def collection_size(record, list_getter)
+        return 0 unless record.respond_to?(list_getter)
+
+        list = record.public_send(list_getter)
+        list.respond_to?(:size) ? list.size.to_i : 0
+      rescue StandardError
+        0
+      end
+
+      # Re-normalize the survivor's total_recorded_* bookkeeping after the
+      # hooks ran: total = what the record NOW carries + what the SDK had
+      # already dropped. Without this, a hook-ADDED attribute makes the
+      # OTLP encoder's `total - size` subtraction negative (protobuf uint32
+      # → RangeError → the encoder returns nil → the ENTIRE batch is
+      # dropped, silently, forever), and a hand-built replacement record
+      # with nil totals raises NoMethodError at the same line. Also keeps a
+      # hook DELETE honest (the wire no longer claims an SDK-limit drop).
+      def restore_counters!(record, counters)
+        if record.respond_to?(:total_recorded_attributes=)
+          record.total_recorded_attributes = collection_size(record, :attributes) + counters[:attributes]
+        end
+        if record.respond_to?(:total_recorded_events=)
+          record.total_recorded_events = collection_size(record, :events) + counters[:events]
+        end
+        if record.respond_to?(:total_recorded_links=)
+          record.total_recorded_links = collection_size(record, :links) + counters[:links]
+        end
+        record
+      end
+
+      # The span-side wrapper (composition over the redacting exporter, same
+      # wrap-never-subclass posture as redacting_exporter.rb: only the public
+      # export/force_flush/shutdown contract the batch processors call).
+      class SpanExporter
+        def initialize(inner, config)
+          @inner = inner
+          @hooks = config.before_send
+        end
+
+        def export(span_data, timeout: nil)
+          batch = span_data.to_a
+          faults = []
+          kept = batch.filter_map do |sd|
+            result = BeforeSend.run(@hooks, sd, :span, faults)
+            result.equal?(ERROR) ? nil : result
+          end
+          # ONE aggregated warn per batch (the redacting exporters'
+          # convention) — class names only, never messages/values.
+          unless faults.empty?
+            Diagnostics.warn("before_send faulted for #{faults.length} span record(s) — dropped, never " \
+                             "exported half-transformed (fail-closed, rule 14): #{faults.uniq.take(3).join(', ')}")
+          end
+          status = kept.empty? ? OpenTelemetry::SDK::Trace::Export::SUCCESS : @inner.export(kept, timeout: timeout)
+          faults.empty? ? status : OpenTelemetry::SDK::Trace::Export::FAILURE
+        rescue StandardError, SystemStackError
+          # Fail closed past the per-record guards, loudly (rule 15) —
+          # never raise into the batch thread.
+          Diagnostics.warn("before_send pass raised past its guards — span batch dropped (fail-closed, rule 14/15)")
+          OpenTelemetry::SDK::Trace::Export::FAILURE
+        end
+
+        def force_flush(timeout: nil) = @inner.force_flush(timeout: timeout)
+        def shutdown(timeout: nil) = @inner.shutdown(timeout: timeout)
+      end
+
+      class LogRecordExporter
+        def initialize(inner, config)
+          @inner = inner
+          @hooks = config.before_send
+        end
+
+        def export(log_record_data, timeout: nil)
+          batch = log_record_data.to_a
+          faults = []
+          kept = batch.filter_map do |lrd|
+            result = BeforeSend.run(@hooks, lrd, :log, faults)
+            result.equal?(ERROR) ? nil : result
+          end
+          unless faults.empty?
+            Diagnostics.warn("before_send faulted for #{faults.length} log record(s) — dropped, never " \
+                             "exported half-transformed (fail-closed, rule 14): #{faults.uniq.take(3).join(', ')}")
+          end
+          status = kept.empty? ? OpenTelemetry::SDK::Logs::Export::SUCCESS : @inner.export(kept, timeout: timeout)
+          faults.empty? ? status : OpenTelemetry::SDK::Logs::Export::FAILURE
+        rescue StandardError, SystemStackError
+          Diagnostics.warn("before_send pass raised past its guards — log batch dropped (fail-closed, rule 14/15)")
+          OpenTelemetry::SDK::Logs::Export::FAILURE
+        end
+
+        def force_flush(timeout: nil) = @inner.force_flush(timeout: timeout)
+        def shutdown(timeout: nil) = @inner.shutdown(timeout: timeout)
+      end
+    end
+  end
+end

data/lib/foam/otel/config.rb

@@ -13,6 +13,7 @@ module Foam
       :name, :environment, :version, :enabled,
       :redact_keys, :redact_pii_keys, :redact_detect, :ignored_outbound_hosts,
       :diagnostics, :endpoint, :secret_heuristics, :capture_payloads,
+      :before_send,
       keyword_init: true
     )
 
@@ -40,7 +41,8 @@ module Foam
           redact_detect: [].freeze,
           ignored_outbound_hosts: [].freeze,
           diagnostics: false, endpoint: FOAM_OTEL_ENDPOINT,
-          secret_heuristics: true, capture_payloads: :off
+          secret_heuristics: true, capture_payloads: :off,
+          before_send: [].freeze
         ).freeze
       end
 
@@ -53,7 +55,7 @@ module Foam
       def resolve_config(name:, environment:, version:, enabled:,
                          redact_keys:, redact_pii_keys:, ignored_outbound_hosts:,
                          diagnostics:, endpoint:, secret_heuristics: true, redact: nil,
-                         capture_payloads: :off)
+                         capture_payloads: :off, before_send: nil)
         keys, pii, detect = merged_redact_lists(redact_keys, redact_pii_keys, redact)
         # capture_payloads is validated LOUDLY at boot (rule 10, exactly like
         # the redact: object): a value outside CAPTURE_PAYLOAD_MODES (or its
@@ -89,12 +91,82 @@ module Foam
           # — zero teeing, zero middleware) | :errors | :always, canonical
           # symbol form. init.rb applies the FOAM_CAPTURE_PAYLOADS operator
           # env clamp BEFORE this resolves.
-          capture_payloads: capture_mode
+          capture_payloads: capture_mode,
+          # The customer's export hook(s) (1.8.0): one callable or an Array
+          # pipeline, validated LOUDLY at boot (rule 10 — a non-callable is a
+          # programming error). Stored as a frozen Array; empty (the default)
+          # means the exporter chain gains no before_send stage at all.
+          before_send: validate_before_send!(before_send)
         ).freeze
       end
 
       private
 
+      # ---- the before_send option (1.8.0) ------------------------------------
+      # Loud at boot (rule 10, same posture as capture_payloads and the
+      # redact: object): before_send must be one callable (responds to
+      # #call — a lambda, a Proc, a Method, or any hook object) or an Array
+      # of callables run as a pipeline. Anything else — including a nil or
+      # non-callable INSIDE the array — raises here, never a silent no-op
+      # that leaves the customer believing their filter runs. The array is
+      # COPIED before freezing (never freeze the caller's own object).
+      #
+      # Arity is validated too, where Ruby lets us introspect it: a lambda
+      # or Method that cannot accept ONE positional argument (a Sentry-port
+      # `->(event, hint) { }`, a forgotten-param `-> { }`, a required
+      # keyword) would pass a bare respond_to?(:call) check and then fault
+      # on EVERY record at export — a green boot that ships zero telemetry
+      # forever. That is exactly the boot-detectable programming error rule
+      # 10 exists for. Non-lambda Procs are arity-forgiving by language
+      # semantics and uninspectable callables get the benefit of the doubt.
+      def validate_before_send!(value, context: "Foam::Otel.init")
+        hooks = if value.nil?
+                  []
+                else
+                  value.is_a?(Array) ? value.dup : [value]
+                end
+        hooks.each_with_index do |hook, index|
+          unless hook.respond_to?(:call)
+            raise ArgumentError, "#{context} before_send: must be a callable (responds to #call) or an " \
+                                 "Array of callables, got #{hook.class} at position #{index}"
+          end
+          next if before_send_arity_ok?(hook)
+
+          raise ArgumentError, "#{context} before_send: the callable at position #{index} cannot be " \
+                               "called with one record argument (it takes a different number of " \
+                               "required positional parameters, or requires keywords) — before_send " \
+                               "hooks are called as hook.call(record)"
+        end
+        hooks.freeze
+      end
+
+      # True when the hook can be invoked as hook.call(record). Lambdas and
+      # Methods enforce their signatures, so they are checked; non-lambda
+      # Procs coerce arguments (always fine); anything whose #call cannot
+      # be introspected passes (never a false boot failure).
+      def before_send_arity_ok?(hook)
+        params = if hook.is_a?(Proc)
+                   return true unless hook.lambda?
+
+                   hook.parameters
+                 elsif hook.is_a?(Method) || hook.is_a?(UnboundMethod)
+                   hook.parameters
+                 else
+                   begin
+                     hook.method(:call).parameters
+                   rescue StandardError
+                     return true
+                   end
+                 end
+        return false if params.any? { |kind, _| kind == :keyreq }
+
+        required = params.count { |kind, _| kind == :req }
+        flexible = params.any? { |kind, _| kind == :opt || kind == :rest }
+        required == 1 || (required.zero? && flexible)
+      rescue StandardError
+        true
+      end
+
       # ---- the capture_payloads option (payload-capture mandate 2026-07-28)
       # Canonicalize a mode value: the three symbols, or their string
       # spellings trimmed + case-insensitive, map to the canonical symbol;
@@ -212,7 +284,11 @@ module Foam
           # the first init: middleware insertion already happened (or
           # deliberately did not — an :off boot shipped none), so a second
           # init cannot meaningfully flip it. Carried forward unchanged.
-          capture_payloads: existing.capture_payloads
+          capture_payloads: existing.capture_payloads,
+          # before_send is process-global too: the exporter chain was built
+          # (with or without its before_send stage) at the first init and a
+          # second init never rebuilds pipelines. Carried forward unchanged.
+          before_send: existing.before_send
         ).freeze
       end
     end

data/lib/foam/otel/init.rb

@@ -59,7 +59,8 @@ module Foam
                ignored_outbound_hosts: nil,
                diagnostics: false,
                secret_heuristics: true,
-               capture_payloads: :off)
+               capture_payloads: :off,
+               before_send: nil)
         # Identity is validated at boot (rule 10): a blank required value is a
         # programming error the engineer must catch on their machine.
         validate_present!(:name, name)
@@ -96,7 +97,10 @@ module Foam
           # FOAM_CAPTURE_PAYLOADS operator env clamp applied first (a valid
           # env value overrides the option in BOTH directions; an invalid
           # one warns + falls back — never a crashed boot).
-          capture_payloads: resolve_capture_payloads(capture_payloads)
+          capture_payloads: resolve_capture_payloads(capture_payloads),
+          # The customer's export hook(s): a non-callable raises INSIDE
+          # resolve_config, loudly, at boot (rule 10).
+          before_send: before_send
         )
         # The heuristic-tier opt-out is an explicit, audited customer decision
         # (security-fixes-design §V.8) — always one loud line, never silent.

data/lib/foam/otel/pipelines.rb

@@ -8,6 +8,7 @@ require "delegate"
 require "openssl"
 
 require_relative "redacting_exporter"
+require_relative "before_send"
 require_relative "session_stitching"
 
 module Foam
@@ -33,6 +34,11 @@ module Foam
           ssl_verify_mode: OpenSSL::SSL::VERIFY_PEER
         )
         exporter = RedactingSpanExporter.new(inner, config)
+        # The customer's before_send hooks (1.8.0) wrap OUTSIDE redaction:
+        # batch → before_send → redaction → OTLP, so the hooks see the record
+        # first and the credential floor still masks whatever they return
+        # (before_send.rb). No hooks configured → no stage at all.
+        exporter = BeforeSend::SpanExporter.new(exporter, config) unless config.before_send.empty?
         # Pin the sampler explicitly: the TracerProvider constructor otherwise
         # reads OTEL_TRACES_SAMPLER, which an operator (or a stale fleet-wide
         # var) could set to always_off and silently drop EVERY foam span
@@ -87,6 +93,8 @@ module Foam
           ssl_verify_mode: OpenSSL::SSL::VERIFY_PEER # F-RB1 pin — see setup_traces
         )
         exporter = RedactingLogRecordExporter.new(inner, config)
+        # before_send stage, same shape and ordering as setup_traces above.
+        exporter = BeforeSend::LogRecordExporter.new(exporter, config) unless config.before_send.empty?
         # Pin the log-record limits at the SDK defaults — LogRecordLimits
         # otherwise reads OTEL_LOGRECORD_ATTRIBUTE_COUNT_LIMIT /
         # OTEL_(LOGRECORD_)ATTRIBUTE_VALUE_LENGTH_LIMIT (same env family as

data/lib/foam/otel/version.rb

@@ -109,6 +109,42 @@ module Foam
     #   customer redact lists reach inside the body JSON). Rails auto-wiring
     #   (Railtie + init fallback) inserts the middleware ONLY when the mode
     #   is not :off; plain Rack adds one `use` line.
-    VERSION = "1.7.0"
+    # 1.8.0: `before_send:` EXPORT HOOK (MINOR — one additive init option,
+    #   default absent, zero default-path behavior change). One callable or
+    #   an Array pipeline, run per record at the EXPORT BOUNDARY — after
+    #   batching, BEFORE foam's redaction pass and the OTLP serialization —
+    #   so a dropped record never leaves the process, and whatever a hook
+    #   returns still rides the full credential floor / value layer /
+    #   customer-key pass (before_send can never widen what ships). Spans
+    #   and logs only (metrics are aggregated state, not records). Return
+    #   the record (in-place mutation supported — attributes arrive
+    #   unfrozen) to keep it, nil to drop it; a hook that raises or returns
+    #   a foreign object drops the record LOUDLY and the batch reports
+    #   FAILURE (fail-closed, Sentry beforeSend precedent, spec D6). A
+    #   non-callable raises at boot; process-global like enabled/endpoint
+    #   (a second init never changes it). Door-1 only — the door-2 ingest
+    #   taps are unchanged.
+    # 1.8.1: before_send REVIEW HARDENING (adversarial review, 21 confirmed
+    #   findings; PATCH — bugfix-only, no API change). CRITICAL: the OTLP
+    #   encoders derive dropped_attributes_count as total_recorded - size
+    #   (protobuf uint32) — a hook-ADDED attribute made the count negative,
+    #   the encoder raised, and the WHOLE batch (healthy siblings included)
+    #   was silently dropped; foam now re-normalizes total_recorded_* on
+    #   every surviving record (nil-safe for hand-built replacement records;
+    #   hook DELETES no longer fake an SDK-limit drop) — GOTCHAS F16. thaw
+    #   now copies attribute VALUES and deep-copies structured log bodies
+    #   (in-place mutation never bleeds into the live record or the tenant
+    #   masked view; frozen values never fault). Boot validation gains an
+    #   arity check: a lambda/Method that cannot take one positional arg
+    #   (the Sentry ->(event, hint) port) raises at init instead of
+    #   faulting on every record. Hook faults warn ONCE per batch (error
+    #   classes only — messages can carry values; detail under
+    #   diagnostics). validate_before_send! copies the caller's array
+    #   (never freezes customer-owned input). Same-type replacement records
+    #   documented (always accepted). Champ scenario spec: CI
+    #   service-container fallback, pg_isready gate, leak cleanup, numeric
+    #   version pick. Hooks run on the export thread — non-blocking only
+    #   (documented, GOTCHAS F16).
+    VERSION = "1.8.1"
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: foam-otel
 version: !ruby/object:Gem::Version
-  version: 1.7.0
+  version: 1.8.1
 platform: ruby
 authors:
 - Foam
@@ -378,6 +378,7 @@ files:
 - lib/foam-otel.rb
 - lib/foam/otel.rb
 - lib/foam/otel/api.rb
+- lib/foam/otel/before_send.rb
 - lib/foam/otel/classifier.rb
 - lib/foam/otel/config.rb
 - lib/foam/otel/constants.rb