foam-otel 1.7.0 → 1.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 48166d05b120dd8bdca15a9b45db5b4c16a6574fcb6890f4caff2c17c2ae9add
-  data.tar.gz: 2dcd4b3b146a0221c231ba5efc057085418b74e21b24d8441396d47cfc0a8288
+  metadata.gz: 3e034361554ad48530554a1bdc62d75adf1899e3ef57104a367abd2c412d2f93
+  data.tar.gz: 40b6db9d7f11ed9105b00c9998b87da51e0822480d97da52be766ddea24435ca
 SHA512:
-  metadata.gz: 4e92c7af8cc5cf7cc177b792f87a8107b918035efdce080e2a974750180ebbb9aac4a5460cb08372f3ded2724d136804bcb8f0574ee4685ebbdd94fde7afb3e8
-  data.tar.gz: ddd7b6e431034663ec94dacd8b7b051ee63bfecdbeb2a38d383c58531854ba0580f34c9a04ed56473b654b89155fc9a3b7ce140b073ba346d90a5a801b9c6d4b
+  metadata.gz: 9c71df1b9ce568dabd996b659d13e0bc05e93a81e070376a77e8c09ccd6a6fc51dcaa028c19a904e78d38daae532c6bc75077e3fa5331047aa1c64e08a9609ee
+  data.tar.gz: 90623d934372eb908d8c9d13e5ddd4108faad54bea4c5a647e7934959562ff130c2cdfdc2f0c5c5cc00a1a86eb5fcc4ee03c745c69c8c258fced2caf6ceb2f31

data/README.md

@@ -331,6 +331,7 @@ not captured — inbound only.
 | `additional_metric_readers:` | Array | no | `[]` | Tenant seam, metrics. |
 | `additional_instrumentations:` | Array | no | `[]` | Constructed tier-2 instrumentation instances to register (fault-isolated: one that throws is skipped with a `[foam]` warning). |
 | `ignored_outbound_hosts:` | Array<String> | no | `[]` | Hosts whose outbound calls produce no spans — EXTENDS the built-in export-loop guard. For a co-resident agent's intake or a tenant exporter's endpoint. Applies to the clients whose official instrumentation supports host suppression: **Net::HTTP and Excon**. Faraday/HTTP (httprb)/HTTPX have no upstream `untraced_hosts` option — foam warns loudly at init when one of those is bundled (GOTCHAS G13); foam's own export loop is guarded for every client regardless. |
+| `before_send:` | Proc/Array | no | `[]` | Export hook (1.8.0): one callable — or an Array run as a pipeline — called with every **span and log record** at the export boundary, after batching and BEFORE foam's redaction pass and serialization, so nothing a hook drops ever leaves the process. Return the record to export it (mutating it in place is supported — `record.attributes` arrives unfrozen), or `nil` to drop it. A hook that raises (or returns anything else) drops **that record** loudly and the batch reports failure — fail-closed, never a raise into your threads. Runs BEFORE redaction, so the credential floor and your redact lists still apply to whatever the hook returns. Spans and logs only (metrics are aggregated state, not records). A non-callable raises at boot. Process-global: a second `init` never changes it. See "The `before_send` hook" below. |
 | `capture_payloads:` | Symbol/String | no | `:off` | Inbound HTTP **body** capture on the rack server span (see "Payload (body) capture"). `:off` = zero teeing, zero middleware (the default); `:errors` = tee per request, attach only on exception/5xx; `:always` = attach on every request. Equivalent strings accepted; anything else raises at boot. Overridden in both directions by the `FOAM_CAPTURE_PAYLOADS` env clamp (env table below). Textual payloads only, 8192-char cap + `…[truncated]`, true `.size` attributes; sizes-only for binary/compressed. |
 | `diagnostics:` | Boolean | no | false | Verbose `[foam]` self-reporting of init/health. Warnings and errors are always loud regardless. |
 | `secret_heuristics:` | Boolean | no | true | The value-pattern secret layer's HEURISTIC tier (generic keyword+entropy detection, see "The value-pattern secret layer"). `false` disables ONLY the heuristics — the named provider patterns, the credential floor and the redaction-coverage contract have no off switch. Disabling logs one loud `[foam]` line at init (an explicit, audited decision for telemetry whose legitimate values collide with the heuristics). |
@@ -388,6 +389,24 @@ Foam::Otel.init(name: "checkout-api", environment: env, enabled: true, token: EN
                 ignored_outbound_hosts: ["ingest.eval-tool.example"]) # the tenant loop guard — required
 ```
 
+```ruby
+# The before_send export hook (1.8.0): transform or drop spans/logs right
+# before they are exported to foam — after batching, BEFORE redaction and
+# serialization (a dropped record never leaves the process). Return the
+# record (mutate in place freely) or nil to drop; a raising hook drops ONLY
+# its record, loudly, and never raises into your threads (fail-closed).
+Foam::Otel.init(name: "checkout-api", environment: env, enabled: true, token: ENV.fetch("FOAM_OTEL_TOKEN"),
+                before_send: lambda { |record|
+                  next nil if record.respond_to?(:name) && record.name == "GET /healthz" # drop noise
+
+                  record.attributes.delete("internal.debug.blob") # trim before it ships
+                  record
+                })
+# Or an Array pipeline — hooks run in order, each receiving the previous
+# hook's return; the first nil drops the record and short-circuits:
+# Foam::Otel.init(..., before_send: [DropHealthchecks, TrimDebugAttrs, StampTeam])
+```
+
 ```ruby
 # Verbose self-reporting while wiring foam up (warnings are loud regardless):
 Foam::Otel.init(name: "checkout-api", environment: "development", enabled: true,
@@ -616,6 +635,54 @@ processor = Foam::Otel.create_ingest_span_processor(
 
 ---
 
+## The `before_send` hook
+
+`before_send:` is the last word your code gets on every span and log record
+before it is exported to foam. It runs at the **export boundary** — after the
+batch processor hands foam the finished records, immediately **before**
+foam's redaction pass and the OTLP serialization — so a record your hook
+drops is never serialized and never leaves the process. (This is the same
+seam redaction lives at, and for the same Ruby SDK reason: a span's
+attributes freeze at finish before any `on_finish` processor runs, so a
+`SpanProcessor` could neither mutate nor drop. The export boundary is the
+first point where the record is a mutable struct.)
+
+The pipeline order is pinned:
+
+```
+BatchProcessor → before_send hooks → redaction (floor + your lists) → OTLP wire
+```
+
+Your hooks run **first**, on the raw record (it is your own data, still
+in-process), and redaction runs on whatever they return — so `before_send`
+can never widen what ships: an attribute a hook adds is masked by the
+credential floor and your redact lists exactly like one an instrumentation
+set.
+
+The per-record contract (Sentry `beforeSend` semantics):
+
+* **return the record** → it continues to the next hook / to export.
+  Mutating it in place is supported — `record.attributes` arrives as an
+  unfrozen copy, so `record.attributes["k"] = v` and
+  `record.attributes.delete("k")` just work.
+* **return `nil`** → the record is dropped, silently (that is the filter
+  mechanism, not an error).
+* **raise, or return anything else** → that record is dropped **loudly**
+  (`[foam]` warning) and the batch reports failure — fail-closed: a faulted
+  hook never ships a record you may have meant to scrub, and never raises
+  into your application or export threads. Healthy records in the same batch
+  still export.
+
+Scope: **spans and log records** (the two record-shaped signals — the hook
+receives `SpanData` or `LogRecordData`; distinguish them with `is_a?` or
+`respond_to?`). Metrics are aggregated state, not discrete records, and are
+deliberately not routed through `before_send`. Door-2 ingest taps are
+unchanged. Like `enabled:` and the endpoint, `before_send` is process-global:
+the exporter chain is built at the first `init`, and a second `init` never
+changes it.
+
+---
+
 ## The helpers
 
 All helpers never raise, and no-op silently before `init` and when disabled.

data/lib/foam/otel/before_send.rb

@@ -0,0 +1,153 @@
+# frozen_string_literal: true
+
+require_relative "diagnostics"
+
+module Foam
+  module Otel
+    # The customer's `before_send:` hook(s) (1.8.0), run at the EXPORT
+    # BOUNDARY — after batching, immediately BEFORE foam's redaction pass and
+    # the OTLP serialization, so a record a hook drops is never serialized
+    # and never leaves the process. Why not a SpanProcessor: the same Ruby
+    # SDK constraint that forced redaction to the exporter boundary — a span
+    # freezes its attributes at finish BEFORE any on_finish processor runs
+    # (GOTCHAS F1), so a processor could neither mutate nor reliably drop.
+    # The mutable SpanData / LogRecordData Structs the batch processors hand
+    # the exporter are the first (and last) point where per-record transform
+    # + drop is possible before the wire.
+    #
+    # Composition order (pinned, pipelines.rb):
+    #   BatchProcessor -> BeforeSend::*Exporter -> Redacting*Exporter -> OTLP
+    # The hooks run FIRST so the credential floor, the value-pattern secret
+    # layer and the customer's key lists still apply to whatever a hook
+    # returns — before_send can never widen what ships past redaction (an
+    # attribute a hook ADDS is masked exactly like one an instrumentation
+    # set).
+    #
+    # Contract per record (Sentry beforeSend semantics — the spec's named
+    # fail-closed precedent, BASE_PACKAGE_SPEC D6):
+    #   * return the record (mutating it in place is fine) → next hook /
+    #     export;
+    #   * return nil → record DROPPED (the intentional filter mechanism —
+    #     success, not a failure);
+    #   * raise, or return a foreign object → record DROPPED, loudly, and
+    #     the batch reports FAILURE (fail-closed: a hook fault must never
+    #     ship a record the customer may have meant to scrub, and must never
+    #     raise into the export thread — rules 9/14/15). SystemStackError is
+    #     rescued explicitly (not a StandardError): a hook recursing on a
+    #     poisoned payload kills neither the batch thread nor the caller.
+    # The record reaches the first hook as a shallow dup with an UNFROZEN
+    # attributes copy (span attributes freeze at finish — GOTCHAS F1), so
+    # `record.attributes["k"] = v` just works; the dup also keeps the
+    # customer's mutations off the struct any other consumer might hold.
+    #
+    # Scope: spans and logs (the two record-shaped signals). Metric data is
+    # aggregated state, not records — deliberately NOT routed through
+    # before_send (documented in the README).
+    module BeforeSend
+      # Internal sentinel distinguishing a FAULTED drop (counts toward batch
+      # FAILURE) from an intentional nil drop. A hook can never return it
+      # legitimately — any non-record return is itself classified a fault.
+      ERROR = Object.new.freeze
+
+      module_function
+
+      # Run the hook pipeline over one record: each hook receives the
+      # previous hook's return. Returns the surviving record, nil for an
+      # intentional drop, or ERROR for a faulted one.
+      def run(hooks, record, signal)
+        current = thaw(record)
+        hooks.each do |hook|
+          result = hook.call(current)
+          return nil if result.nil?
+
+          unless result.is_a?(record.class)
+            Diagnostics.warn("before_send returned a #{result.class} for a #{signal} record — record " \
+                             "dropped (return the record, a replacement of the same type, or nil to drop; " \
+                             "fail-closed, rule 14)")
+            return ERROR
+          end
+          current = result
+        end
+        current
+      rescue StandardError, SystemStackError => e
+        Diagnostics.warn("before_send raised for a #{signal} record — record dropped, never exported " \
+                         "half-transformed (fail-closed, rule 14): #{e.class}: #{e.message}")
+        ERROR
+      end
+
+      # A shallow dup with an unfrozen attributes copy — the hook gets its
+      # own mutable record (span attributes arrive frozen; GOTCHAS F1).
+      def thaw(record)
+        copy = record.dup
+        if copy.respond_to?(:attributes) && copy.respond_to?(:attributes=)
+          attributes = copy.attributes
+          copy.attributes = attributes.nil? ? {} : attributes.dup
+        end
+        copy
+      end
+
+      # The span-side wrapper (composition over the redacting exporter, same
+      # wrap-never-subclass posture as redacting_exporter.rb: only the public
+      # export/force_flush/shutdown contract the batch processors call).
+      class SpanExporter
+        def initialize(inner, config)
+          @inner = inner
+          @hooks = config.before_send
+        end
+
+        def export(span_data, timeout: nil)
+          batch = span_data.to_a
+          faulted = 0
+          kept = batch.filter_map do |sd|
+            result = BeforeSend.run(@hooks, sd, :span)
+            if result.equal?(ERROR)
+              faulted += 1
+              nil
+            else
+              result
+            end
+          end
+          status = kept.empty? ? OpenTelemetry::SDK::Trace::Export::SUCCESS : @inner.export(kept, timeout: timeout)
+          faulted.positive? ? OpenTelemetry::SDK::Trace::Export::FAILURE : status
+        rescue StandardError, SystemStackError
+          # Fail closed past the per-record guards, loudly (rule 15) —
+          # never raise into the batch thread.
+          Diagnostics.warn("before_send pass raised past its guards — span batch dropped (fail-closed, rule 14/15)")
+          OpenTelemetry::SDK::Trace::Export::FAILURE
+        end
+
+        def force_flush(timeout: nil) = @inner.force_flush(timeout: timeout)
+        def shutdown(timeout: nil) = @inner.shutdown(timeout: timeout)
+      end
+
+      class LogRecordExporter
+        def initialize(inner, config)
+          @inner = inner
+          @hooks = config.before_send
+        end
+
+        def export(log_record_data, timeout: nil)
+          batch = log_record_data.to_a
+          faulted = 0
+          kept = batch.filter_map do |lrd|
+            result = BeforeSend.run(@hooks, lrd, :log)
+            if result.equal?(ERROR)
+              faulted += 1
+              nil
+            else
+              result
+            end
+          end
+          status = kept.empty? ? OpenTelemetry::SDK::Logs::Export::SUCCESS : @inner.export(kept, timeout: timeout)
+          faulted.positive? ? OpenTelemetry::SDK::Logs::Export::FAILURE : status
+        rescue StandardError, SystemStackError
+          Diagnostics.warn("before_send pass raised past its guards — log batch dropped (fail-closed, rule 14/15)")
+          OpenTelemetry::SDK::Logs::Export::FAILURE
+        end
+
+        def force_flush(timeout: nil) = @inner.force_flush(timeout: timeout)
+        def shutdown(timeout: nil) = @inner.shutdown(timeout: timeout)
+      end
+    end
+  end
+end

data/lib/foam/otel/config.rb

@@ -13,6 +13,7 @@ module Foam
       :name, :environment, :version, :enabled,
       :redact_keys, :redact_pii_keys, :redact_detect, :ignored_outbound_hosts,
       :diagnostics, :endpoint, :secret_heuristics, :capture_payloads,
+      :before_send,
       keyword_init: true
     )
 
@@ -40,7 +41,8 @@ module Foam
           redact_detect: [].freeze,
           ignored_outbound_hosts: [].freeze,
           diagnostics: false, endpoint: FOAM_OTEL_ENDPOINT,
-          secret_heuristics: true, capture_payloads: :off
+          secret_heuristics: true, capture_payloads: :off,
+          before_send: [].freeze
         ).freeze
       end
 
@@ -53,7 +55,7 @@ module Foam
       def resolve_config(name:, environment:, version:, enabled:,
                          redact_keys:, redact_pii_keys:, ignored_outbound_hosts:,
                          diagnostics:, endpoint:, secret_heuristics: true, redact: nil,
-                         capture_payloads: :off)
+                         capture_payloads: :off, before_send: nil)
         keys, pii, detect = merged_redact_lists(redact_keys, redact_pii_keys, redact)
         # capture_payloads is validated LOUDLY at boot (rule 10, exactly like
         # the redact: object): a value outside CAPTURE_PAYLOAD_MODES (or its
@@ -89,12 +91,39 @@ module Foam
           # — zero teeing, zero middleware) | :errors | :always, canonical
           # symbol form. init.rb applies the FOAM_CAPTURE_PAYLOADS operator
           # env clamp BEFORE this resolves.
-          capture_payloads: capture_mode
+          capture_payloads: capture_mode,
+          # The customer's export hook(s) (1.8.0): one callable or an Array
+          # pipeline, validated LOUDLY at boot (rule 10 — a non-callable is a
+          # programming error). Stored as a frozen Array; empty (the default)
+          # means the exporter chain gains no before_send stage at all.
+          before_send: validate_before_send!(before_send)
         ).freeze
       end
 
       private
 
+      # ---- the before_send option (1.8.0) ------------------------------------
+      # Loud at boot (rule 10, same posture as capture_payloads and the
+      # redact: object): before_send must be one callable (responds to
+      # #call — a lambda, a Proc, a Method, or any hook object) or an Array
+      # of callables run as a pipeline. Anything else — including a nil or
+      # non-callable INSIDE the array — raises here, never a silent no-op
+      # that leaves the customer believing their filter runs.
+      def validate_before_send!(value, context: "Foam::Otel.init")
+        hooks = if value.nil?
+                  []
+                else
+                  value.is_a?(Array) ? value : [value]
+                end
+        hooks.each_with_index do |hook, index|
+          next if hook.respond_to?(:call)
+
+          raise ArgumentError, "#{context} before_send: must be a callable (responds to #call) or an " \
+                               "Array of callables, got #{hook.class} at position #{index}"
+        end
+        hooks.freeze
+      end
+
       # ---- the capture_payloads option (payload-capture mandate 2026-07-28)
       # Canonicalize a mode value: the three symbols, or their string
       # spellings trimmed + case-insensitive, map to the canonical symbol;
@@ -212,7 +241,11 @@ module Foam
           # the first init: middleware insertion already happened (or
           # deliberately did not — an :off boot shipped none), so a second
           # init cannot meaningfully flip it. Carried forward unchanged.
-          capture_payloads: existing.capture_payloads
+          capture_payloads: existing.capture_payloads,
+          # before_send is process-global too: the exporter chain was built
+          # (with or without its before_send stage) at the first init and a
+          # second init never rebuilds pipelines. Carried forward unchanged.
+          before_send: existing.before_send
         ).freeze
       end
     end

data/lib/foam/otel/init.rb

@@ -59,7 +59,8 @@ module Foam
                ignored_outbound_hosts: nil,
                diagnostics: false,
                secret_heuristics: true,
-               capture_payloads: :off)
+               capture_payloads: :off,
+               before_send: nil)
         # Identity is validated at boot (rule 10): a blank required value is a
         # programming error the engineer must catch on their machine.
         validate_present!(:name, name)
@@ -96,7 +97,10 @@ module Foam
           # FOAM_CAPTURE_PAYLOADS operator env clamp applied first (a valid
           # env value overrides the option in BOTH directions; an invalid
           # one warns + falls back — never a crashed boot).
-          capture_payloads: resolve_capture_payloads(capture_payloads)
+          capture_payloads: resolve_capture_payloads(capture_payloads),
+          # The customer's export hook(s): a non-callable raises INSIDE
+          # resolve_config, loudly, at boot (rule 10).
+          before_send: before_send
         )
         # The heuristic-tier opt-out is an explicit, audited customer decision
         # (security-fixes-design §V.8) — always one loud line, never silent.

data/lib/foam/otel/pipelines.rb

@@ -8,6 +8,7 @@ require "delegate"
 require "openssl"
 
 require_relative "redacting_exporter"
+require_relative "before_send"
 require_relative "session_stitching"
 
 module Foam
@@ -33,6 +34,11 @@ module Foam
           ssl_verify_mode: OpenSSL::SSL::VERIFY_PEER
         )
         exporter = RedactingSpanExporter.new(inner, config)
+        # The customer's before_send hooks (1.8.0) wrap OUTSIDE redaction:
+        # batch → before_send → redaction → OTLP, so the hooks see the record
+        # first and the credential floor still masks whatever they return
+        # (before_send.rb). No hooks configured → no stage at all.
+        exporter = BeforeSend::SpanExporter.new(exporter, config) unless config.before_send.empty?
         # Pin the sampler explicitly: the TracerProvider constructor otherwise
         # reads OTEL_TRACES_SAMPLER, which an operator (or a stale fleet-wide
         # var) could set to always_off and silently drop EVERY foam span
@@ -87,6 +93,8 @@ module Foam
           ssl_verify_mode: OpenSSL::SSL::VERIFY_PEER # F-RB1 pin — see setup_traces
         )
         exporter = RedactingLogRecordExporter.new(inner, config)
+        # before_send stage, same shape and ordering as setup_traces above.
+        exporter = BeforeSend::LogRecordExporter.new(exporter, config) unless config.before_send.empty?
         # Pin the log-record limits at the SDK defaults — LogRecordLimits
         # otherwise reads OTEL_LOGRECORD_ATTRIBUTE_COUNT_LIMIT /
         # OTEL_(LOGRECORD_)ATTRIBUTE_VALUE_LENGTH_LIMIT (same env family as

data/lib/foam/otel/version.rb

@@ -109,6 +109,21 @@ module Foam
     #   customer redact lists reach inside the body JSON). Rails auto-wiring
     #   (Railtie + init fallback) inserts the middleware ONLY when the mode
     #   is not :off; plain Rack adds one `use` line.
-    VERSION = "1.7.0"
+    # 1.8.0: `before_send:` EXPORT HOOK (MINOR — one additive init option,
+    #   default absent, zero default-path behavior change). One callable or
+    #   an Array pipeline, run per record at the EXPORT BOUNDARY — after
+    #   batching, BEFORE foam's redaction pass and the OTLP serialization —
+    #   so a dropped record never leaves the process, and whatever a hook
+    #   returns still rides the full credential floor / value layer /
+    #   customer-key pass (before_send can never widen what ships). Spans
+    #   and logs only (metrics are aggregated state, not records). Return
+    #   the record (in-place mutation supported — attributes arrive
+    #   unfrozen) to keep it, nil to drop it; a hook that raises or returns
+    #   a foreign object drops the record LOUDLY and the batch reports
+    #   FAILURE (fail-closed, Sentry beforeSend precedent, spec D6). A
+    #   non-callable raises at boot; process-global like enabled/endpoint
+    #   (a second init never changes it). Door-1 only — the door-2 ingest
+    #   taps are unchanged.
+    VERSION = "1.8.0"
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: foam-otel
 version: !ruby/object:Gem::Version
-  version: 1.7.0
+  version: 1.8.0
 platform: ruby
 authors:
 - Foam
@@ -378,6 +378,7 @@ files:
 - lib/foam-otel.rb
 - lib/foam/otel.rb
 - lib/foam/otel/api.rb
+- lib/foam/otel/before_send.rb
 - lib/foam/otel/classifier.rb
 - lib/foam/otel/config.rb
 - lib/foam/otel/constants.rb