foam-otel 3.0.0.alpha.12 → 3.0.0.alpha.13

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b67fa02fd12d317a0bceff3c596f63d928b0187995d02e95191f9ecb05b7383e
-  data.tar.gz: 6933cac173a8df6fa83fe474f06fd59569d0da61eb4a78f16919c695561b63af
+  metadata.gz: '026729cab71cf3ea55acaf802d5efc5f6e2d71e6de6ae72f5ed57c1181f11dd2'
+  data.tar.gz: 1dc901d627cd9e326cfc5fa5f9339d53e2d9af3c9de3b2e14f64a9d4784dcd77
 SHA512:
-  metadata.gz: a7080090c2024fda65b0bf8d76edbe311961ae7c26814cd3154c11e9c7735e9730c10e100468d1046acc94e1cb2b2d1518b50cdc72b631201187b271b167c722
-  data.tar.gz: 184764049f1395f94f270cec7054138cfd78c8f6b40c93c3cafafdcb4cd7a3754bfa1eed0851e47d1f96258f10e1dea51889624862e07f86579e42d73a26353a
+  metadata.gz: e037404d73d8dcc83dbadb700b5395c41097706e2b2772597c33a577fb5b5a806df0eb0546d09dc05202ab56e49e3df0f7b2ea1f55025156691d0d014feecbf6
+  data.tar.gz: 64bcbecd4e4c539bc4190ba35c9a7845778775857180190408371c75321fdd17434e5e6698d97ab9a69dc2f1c5859d569ba69d4b027ae9f3ffb0c7598916c32e

data/README.md

@@ -23,54 +23,62 @@ bundle install
 
 ## Initialize
 
-Call `init` once, after the app's gems are loaded and before the app runs. Set `enabled: false` to turn Foam off.
+Call `init` once, after the app's gems are loaded and before the app runs. Set `enabled` to `true` only in the deployed environments that send telemetry to Foam, such as production.
 
 Store the ingest token as `FOAM_OTEL_TOKEN` and pass `ENV["FOAM_OTEL_TOKEN"]` to `init`.
 
 ### Rails
 
-Create `config/initializers/foam.rb`:
+Call `init` after every file in `config/initializers` and before Rails builds the middleware stack. Add an initializer to the application class in `config/application.rb`:
 
 ```ruby
-Foam.init(
-  name: "checkout-api",
-  environment: Rails.env,
-  enabled: true,
-  token: ENV["FOAM_OTEL_TOKEN"]
-)
+module CheckoutApi
+  class Application < Rails::Application
+    initializer "foam.init", after: :load_config_initializers do
+      Foam.init(
+        name: "checkout-api",
+        environment: Rails.env,
+        enabled: Rails.env.production?,
+        token: ENV["FOAM_OTEL_TOKEN"]
+      )
+    end
+  end
+end
 ```
 
-`init` must run before the app initializes, either with an initializer file or in `config/application.rb`. An `init` from `after_initialize` or later warns on stderr and adds no middleware. During boot it inserts `Foam::Rack::Middleware` directly after the OpenTelemetry Rack middleware. `config.ru` stays as generated.
+`init` inserts `Foam::Rack::Middleware` directly after the OpenTelemetry Rack middleware. `config.ru` stays as generated. An `init` from `after_initialize` or later warns on stderr and adds no middleware.
 
 Do not load Foam with a `RUBYOPT` preload. It runs before Rails loads, so it installs no Rails instrumentation and adds no middleware.
 
 ### Plain Ruby and Rack
 
-Call `init` at the top of `config.ru`, then add the OpenTelemetry Rack middleware and Foam's:
+Call `init` in `config.ru` after the app's requires, then add Foam's middleware:
 
 ```ruby
 require "foam/otel"
+require_relative "app"
 
 Foam.init(
   name: "checkout-api",
   environment: ENV["RACK_ENV"],
-  enabled: true,
+  enabled: ENV["RACK_ENV"] == "production",
   token: ENV["FOAM_OTEL_TOKEN"]
 )
 
-use(*OpenTelemetry::Instrumentation::Rack::Instrumentation.instance.middleware_args)
 use Foam::Rack::Middleware
 
 run App
 ```
 
+`Foam::Rack::Middleware` runs the OpenTelemetry Rack middleware, which opens the server span. When `init` is disabled or stops early, the middleware passes every request through.
+
 Foam's middleware records the `content-type`, `content-length`, and `content-encoding` headers as server span attributes. In `:advanced` mode it also sends the bodies as log records correlated to the span. See Network capture.
 
 ## Instrumentation
 
 `init` installs OpenTelemetry's Ruby instrumentation bundle, `opentelemetry-instrumentation-all`, and the standard library logger bridge, `opentelemetry-instrumentation-logger`. An instrumentation installs only when its library is loaded before `init`, so call `init` after `Bundler.require` or after the app's own requires. A library loaded after `init` is not instrumented, and a warning is written to stderr once when `state` finds one. `state[:instrumentations]` lists what installed.
 
-The bundle covers: action_mailer, action_pack, action_view, active_job, active_model_serializers, active_record, active_storage, active_support, anthropic, aws_lambda, aws_sdk, bunny, concurrent_ruby, dalli, delayed_job, ethon, excon, faraday, grape, graphql, grpc, gruf, http, http_client, httpx, koala, lmdb, mongo, mysql2, net_http, pg, que, racecar, rack, rails, rake, rdkafka, redis, resque, restclient, ruby_kafka, sidekiq, sinatra, trilogy.
+The bundle covers: action_mailer, action_pack, action_view, active_job, active_model_serializers, active_record, active_storage, active_support, anthropic, aws_lambda, aws_sdk, bunny, concurrent_ruby, dalli, delayed_job, ethon, excon, faraday, grape, graphql, grpc, gruf, http, http_client, httpx, koala, lmdb, mongo, mysql2, net_http, openai, pg, que, racecar, rack, rails, rake, rdkafka, redis, resque, restclient, ruby_kafka, sidekiq, sinatra, trilogy.
 
 To install an instrumentation outside the bundle, list it in `additional_instrumentations`.
 
@@ -88,11 +96,19 @@ To disable one instrumentation or change its options, pass it in `additional_ins
 
 TODO(jhartquist): `db_capture` init option. The `pg`, `mysql2`, and `trilogy` instrumentations can include the statement (`db_statement: :include`) and `pg` can report row counts (`db_response_returned_rows`, `db_response_affected_rows`), but none hands over results. Result capture needs Foam patches on the drivers, opt-in, through the same redaction and size caps as bodies. See Expanded data capture at the end of this document.
 
-TODO(jhartquist): Gen AI providers beyond `anthropic`. There is no OpenTelemetry Ruby instrumentation for the `openai` or `ruby-openai` gems.
+TODO(jhartquist): Gen AI providers beyond `anthropic` and `openai`. There is no OpenTelemetry Ruby instrumentation for the `ruby-openai` gem.
 
 ## Public API
 
-Every method lives on the `Foam` module. A missing or invalid value is reported in Foam's log output instead of raising; only an unknown keyword raises `ArgumentError`. `set_baggage`, the metric helpers, and `log` need a Foam export path to function properly, so they are no-ops before `init`; the trace context helpers and `record_exception` work on the current OpenTelemetry context whether or not `init` ran.
+Every method lives on the `Foam` module. A call with wrong arguments raises `ArgumentError`: an unknown keyword, a missing required keyword, or a wrong argument count. Any other problem, such as a blank or invalid value, is written to stderr with a `[foam-otel]` prefix, and the method returns without raising.
+
+These helpers do nothing until Foam has a way to export:
+
+- `set_baggage` works after `init`.
+- The metric helpers work after `init`, or with `create_foam_ingest_metric_reader` attached.
+- `log` works after `init`, or once `create_foam_ingest_log_record_processor` is built.
+
+The trace context helpers and `record_exception` work on the current OpenTelemetry context, with or without `init`.
 
 ### `init`
 
@@ -140,7 +156,7 @@ Foam.init(
 
 - `name` - the service name, sent as `service.name`.
 - `environment` - the deployment environment, sent as `deployment.environment.name`.
-- `enabled` - `false` turns Foam off; `init` returns without configuring OpenTelemetry. With a token, it still reports the outcome (see below).
+- `enabled` - `false` turns Foam off; `init` returns without configuring OpenTelemetry. With a token, it still reports the outcome (see below). `OTEL_SDK_DISABLED=true` also turns Foam off.
 - `token` - the Foam ingest token, sent as a bearer token on every export. Required when Foam is enabled; a missing token is reported on stderr and leaves Foam uninitialized.
 - `version` - the service version, sent as `service.version` when given.
 - `sample_rate` - the fraction of root traces kept, from `0.0` to `1.0`; child spans follow their parent. Leave it at `1.0` unless volume forces sampling. `OTEL_TRACES_SAMPLER` is not consulted.
@@ -164,11 +180,11 @@ Every signal carries a resource with these attributes:
 - `host.name`, `host.arch`, `host.id` when it can be read, `os.type`, and `os.version`.
 - The process and SDK attributes the OpenTelemetry SDK adds.
 
-A process forked after `init` carries its own `service.instance.id` and `process.pid`. `OTEL_RESOURCE_ATTRIBUTES` can add further attributes. It cannot override any of the attributes above, and `OTEL_SERVICE_NAME` cannot override `name`.
+A process forked after `init` carries its own `service.instance.id` and `process.pid`. `OTEL_RESOURCE_ATTRIBUTES` can add further attributes and override the process and SDK attributes the OpenTelemetry SDK adds, except `process.pid`. It cannot override the other attributes above, and `OTEL_SERVICE_NAME` cannot override `name`.
 
-`init` raises only `ArgumentError`, for a missing `name` or `environment` or an unknown keyword. Any other failure, or an option holding an invalid value, is reported on stderr with a `[foam-otel]` prefix and leaves `state` uninitialized.
+`init` raises `ArgumentError` only when the `name` or `environment` keyword is left out or a keyword is unknown. A blank value, an invalid option value, or any other failure is written to stderr with a `[foam-otel]` prefix and leaves `state` uninitialized.
 
-`init` also reports its outcome to Foam: one log record with the message and the `state`, sent straight to the endpoint with the token. The report is sent in the background and never blocks the app. When the process exits before the report has been delivered, exit waits up to two seconds for the send to finish; a report that cannot be delivered is dropped. Nothing is sent without a token.
+`init` also reports its outcome to Foam: one log record with the message and the `state`, sent straight to the endpoint with the token. `init` resolves the endpoint's host before it returns, which takes up to 1 second when DNS is slow. The report itself is sent in the background. When the process exits before the report has been delivered, exit waits up to two seconds for the send to finish; a report that cannot be delivered is dropped. Nothing is sent without a token.
 
 ### `set_endpoint(url)`
 
@@ -219,15 +235,20 @@ meter_provider.add_metric_reader(reader)
 
 All three export to the endpoint set by `set_endpoint` and take the same `redact` option as `init`. The span processor and the log record processor also take the same `before_send` option; the metric reader does not. Each one records its signal as `ingest` in `state` and reports the attachment to Foam.
 
-Foam identifies the service by the `name` and `environment` passed in: on the copy sent to Foam they overwrite the provider's `service.name` and `deployment.environment.name`, and `foam.ingest.host` names the host of the endpoint the copy goes to. The provider's own telemetry is left unchanged. Pass the same values to all three.
+Foam identifies the service by the `name` and `environment` passed in. On the copy sent to Foam, they overwrite the provider's `service.name` and `deployment.environment.name`. The copy also gets `foam.ingest.host`, the host of the endpoint it goes to. The provider's own telemetry stays unchanged. Pass the same values to all three.
 
-A missing or blank `name`, `environment`, or `token`, or an invalid `redact` or `before_send` value, is reported in Foam's log output and turns off just that helper: it returns a component that exports nothing, which the app adds to its provider as usual, and the signal stays `none` in `state`, so an unset environment variable never stops the app from booting.
+Only that helper turns off when it gets one of these values:
+
+- a missing or blank `name`, `environment`, or `token`
+- an invalid `redact` or `before_send` value
+
+It writes the problem to stderr and returns a component that exports nothing. The app adds that component to its provider as usual, and the signal stays `none` in `state`. An unset environment variable never stops the app from booting.
 
 Do not combine `init` and an ingest component for the same signal. The ingest components do not install instrumentations; register the ones the app needs with the OpenTelemetry SDK directly.
 
 ### `inject_trace_context(carrier)`
 
-Writes the W3C `traceparent`, `tracestate`, and `baggage` headers for the current span into `carrier`, a hash of string keys. A `tracestate` the carrier already holds is kept, since Foam adds no entry of its own and another SDK may have added one.
+Writes the W3C `traceparent`, `tracestate`, and `baggage` headers for the current span into `carrier`, a hash of string keys. A `tracestate` the carrier already holds is kept, since Foam adds no entry of its own and another SDK may have added one. A `traceparent` the carrier already holds is overwritten.
 
 Use it when the app sends a message itself (a Kafka record, a queue payload, a custom socket). The HTTP instrumentations already inject these headers; do not call it on ordinary HTTP requests. Without an active span it writes only `baggage`, and only when baggage is set.
 
@@ -237,8 +258,6 @@ Foam.inject_trace_context(headers)
 producer.publish(topic, payload, headers: headers)
 ```
 
-TODO(jhartquist): a carrier that already holds a `traceparent` has it overwritten. Decide whether to keep the existing one, replace it, or link the two.
-
 ### `extract_trace_context(carrier)`
 
 Reads those headers from `carrier` and returns an OpenTelemetry context. Run the receiving work inside it so its spans continue the sender's trace. Returns the current context when the carrier has no trace headers.
@@ -335,11 +354,17 @@ Foam.flush(timeout: 5)
 
 ### `shutdown(timeout: 10)`
 
-Flushes and shuts down Foam's providers, including the one behind `log` when an ingest component created it, and the ingest components, and releases the global slots Foam claimed. `init` and each ingest factory register it with `at_exit`, so most apps never call it, and an app that never shuts its own providers down still delivers the ingest components' last batch. An app calls it before a process is killed without a normal exit.
+Flushes and shuts down Foam's providers and the ingest components, and releases the global slots Foam claimed. The providers include the one behind `log` when an ingest component created it. The first `init` or ingest factory call registers it with `at_exit`, once per process. Most apps never call it. An app that never shuts its own providers down still delivers the ingest components' last batch. An app calls it before a process is killed without a normal exit.
 
 `timeout` is in seconds and bounds the whole call. A provider that runs out of time is reported on stderr.
 
-`at_exit` covers a normal exit, an unhandled exception, and a signal the process handles by exiting. The last batch is lost when the process ends without running `at_exit`, as with `exit!`, a `KILL` signal, or a crash of the Ruby process itself, and when a batch outlasts the timeout. An exception outside any span is invisible unless the app logs it.
+`at_exit` covers a normal exit, an unhandled exception, and a signal the process handles by exiting. The last batch is lost only in these cases:
+
+- The process ends without running `at_exit`: `exit!`, a `KILL` signal, or a crash of the Ruby process itself.
+- A batch outlasts the timeout.
+- The export fails, for example when the endpoint is unreachable.
+
+An exception outside any span is invisible unless the app logs it.
 
 Foam's span processor holds 4096 finished spans and drops the oldest when full; `OTEL_BSP_MAX_QUEUE_SIZE` overrides the size. The ingest span processor and the log record processors keep the OpenTelemetry default of 2048.
 
@@ -365,7 +390,7 @@ Foam.state
 # }
 ```
 
-`instrumentations` lists the instrumentations that installed, by gem name without the `opentelemetry-instrumentation-` prefix, with the version of the library each one instruments (`net_http` reports the Ruby version; `aws_lambda` has no version). An instrumentation whose library is absent or unsupported is left out.
+`instrumentations` lists each instrumentation that installed, by gem name without the `opentelemetry-instrumentation-` prefix. Each entry carries the version of the library it instruments, as `name@version`. `net_http` reports the Ruby version, and `aws_lambda` has no version. An instrumentation whose library is absent or unsupported is left out.
 
 A library loaded after `init` is not instrumented and stays out of the list; a warning is written to stderr the first time a `state` call finds one. See Instrumentation for the bundle.
 
@@ -390,15 +415,28 @@ TODO(jhartquist): custom spans and the OpenTelemetry API: getting a tracer, `in_
 
 - `:off` keeps the standard OpenTelemetry HTTP telemetry, with no extra capture.
 - `:basic` adds the `content-type`, `content-length`, and `content-encoding` headers of the request and the response as `http.request.header.<name>` and `http.response.header.<name>` span attributes. No other header attribute is added.
-- `:advanced` adds the request and response bodies as well.
+- `:advanced` adds the request and response bodies as well, and turns on the `openai` instrumentation's `capture_content`: prompts and completions arrive as `gen_ai` log records, redacted like bodies. `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` takes precedence. The `anthropic` instrumentation has no content capture; its prompts arrive only as captured `Net::HTTP` request bodies.
 
 There is no option to record other headers.
 
-Server spans get the capture from `Foam::Rack::Middleware`, placed after the OpenTelemetry Rack middleware (added by `init` in Rails apps). Client spans get it from `Net::HTTP`, Faraday with any adapter, Excon, and HTTPClient when their gem is loaded before `init`. RestClient and HTTParty are captured on the `Net::HTTP` span beneath them. `http`, `httpx`, and `ethon` get no capture, and neither does a Faraday connection built before `init`.
+Server spans get the capture from `Foam::Rack::Middleware`, which runs inside the OpenTelemetry Rack middleware (added by `init` in Rails apps). Client spans get it from `Net::HTTP`, Faraday with any adapter, Excon, and HTTPClient when their gem is loaded before `init`. RestClient and HTTParty are captured on the `Net::HTTP` span beneath them. `http`, `httpx`, and `ethon` get no capture, and neither does a Faraday connection built before `init`.
+
+Foam captures these streamed bodies:
+
+- a Rack response body the server sends through `each`
+- a Rack request body, including the part the app leaves unread when `rack.input` can seek
+- a `Net::HTTP` request body sent through `body_stream`
+- an HTTPClient response read in a block
 
-Streamed bodies are captured for a Rack response the server iterates, a Rack request body as the app reads it, a `Net::HTTP` request sent through `body_stream`, and an HTTPClient response read in a block. Other streamed bodies are not, such as a Rack `call` body, a file sent by path, a hijacked connection, a client request body sent from an IO, or a client response read in a block or through a streaming callback.
+Foam captures no other streamed body. For example:
 
-TODO(jhartquist): cover `http`, `httpx`, `ethon`, and `restclient`.
+- a Rack response body that responds to `call`
+- a file the server sends by path (`to_path`)
+- a hijacked connection (`rack.hijack`)
+- an Excon, Faraday, or HTTPClient request body sent from an IO
+- an Excon `request_block` or an HTTPClient multipart body
+- a `Net::HTTP` response read in a block or into an IO
+- an Excon response read through `response_block` or a Faraday response read through `on_data`
 
 ### Bodies
 
@@ -407,7 +445,7 @@ TODO(jhartquist): cover `http`, `httpx`, `ethon`, and `restclient`.
 - At most 1 MiB of a body is kept, sent as records of at most 64 KiB each. The rest counts toward the size.
 - Bodies are parsed by content type and redacted with the same keys as span attributes. See Redaction.
 - A server request body is captured whether or not the app reads it when the input is rewindable, as it is under Puma. Otherwise only what the app reads is captured.
-- A streamed server response is observed as it is sent, without buffering or delaying it. Its records are emitted when the body closes; a stream idle for over 60 seconds is closed as incomplete and the rest is dropped.
+- A streamed server response is observed as it is sent, without buffering or delaying it. Its records are emitted when the body closes. A stream that stalls for over 60 seconds can be recorded as incomplete.
 
 ### Known issue
 
@@ -437,13 +475,39 @@ Secrets in a log line are masked before export. See Redaction.
 
 ## AWS Lambda
 
+Wrap the function's handler with the `aws_lambda` instrumentation's handler, and call `init` in the wrapper file after the handler file loads.
+
+1. Add `foam_lambda.rb` next to the handler file:
+
+   ```ruby
+   require "foam/otel"
+   require_relative "function"
+
+   Foam.init(name: "checkout-fn", environment: "production", token: ENV["FOAM_OTEL_TOKEN"])
+
+   FOAM_HANDLER = OpenTelemetry::Instrumentation::AwsLambda::Handler.new
+
+   def foam_handler(event:, context:)
+     FOAM_HANDLER.call_wrapped(event: event, context: context)
+   end
+   ```
+
+2. Set the function's handler to `foam_lambda.foam_handler`.
+3. Set the `ORIG_HANDLER` environment variable to the original handler, such as `function.handler`.
+
 The invocation span carries `faas.trigger` for API Gateway, ALB, SNS, SQS, S3, DynamoDB streams, and EventBridge schedule events.
 
-Traces, metrics, and logs are flushed before the handler returns (including failing invocations).
+Traces, metrics, and logs are flushed before the handler returns, including failing invocations.
 
 ## Pre-forking servers and threads
 
-Call `init` once, in the process that forks: the master of Puma in cluster mode with `preload_app!` (the default since Puma 7), the master of Unicorn with `preload_app true`, or a worker that forks a child per job. Each child exports its own traces, logs, and metrics, carrying its own `process.pid`.
+Call `init` once, in the process that forks:
+
+- the master of Puma in cluster mode with `preload_app!` (the default since Puma 7)
+- the master of Unicorn with `preload_app true`
+- a worker that forks a child per job
+
+Each child exports its own traces, logs, and metrics, with its own `process.pid`.
 
 The current span and baggage are stored per fiber. A `Thread.new` or `Fiber.new` starts with none, so spans started inside it begin a new trace. To continue the trace, capture the context first and make it current inside the thread:
 
@@ -501,11 +565,11 @@ Foam.init(
 Sentry sends its events to the DSN's host over HTTP, which Foam would otherwise record as spans. Pass that address and Sentry's request header to `init`. This works whether Sentry sends inline or from its background worker:
 
 ```ruby
-dsn = Sentry.configuration.dsn
+dsn = Sentry.configuration&.dsn
 
 Foam.init(
   name: "checkout-api", environment: "production", token: ENV["FOAM_OTEL_TOKEN"],
-  ignored_outbound_hosts: ["#{dsn.host}:#{dsn.port}"],
+  ignored_outbound_hosts: dsn ? ["#{dsn.host}:#{dsn.port}"] : [],
   ignored_outbound_headers: ["X-Sentry-Auth"]
 )
 ```
@@ -521,7 +585,7 @@ OpenTelemetry.propagation = OpenTelemetry::Context::Propagation::CompositeTextMa
 
 ### New Relic
 
-New Relic's agent sends its data to its collector over HTTP, and probes the AWS, Azure and GCP metadata endpoints when it connects, which Foam would otherwise record as spans. Pass the collector's address and the metadata hosts to `init`. The agent's requests carry no header of its own, so there is none to pass:
+New Relic's agent sends its data to its collector over HTTP. When it connects, it also probes the AWS, Azure, and GCP metadata endpoints. Foam records each of these requests as a span unless its host is ignored. The agent's requests carry no header of their own, so there is no header to ignore. Pass the collector's address and the metadata hosts to `init`:
 
 ```ruby
 collector = "#{NewRelic::Agent.config[:host]}:#{NewRelic::Agent.config[:port]}"
@@ -536,6 +600,8 @@ Foam.init(
 )
 ```
 
+TODO(jhartquist): confirm the host New Relic's agent sends its data to. The agent sends `preconnect` to `Agent.config[:host]` and every later request to the `redirect_host` in the reply, while the list above names the configured host only. New Relic's network docs list exact collector hosts per region, so the redirect likely names the same host, but mosh's receiver never redirects. Check with a real license key and a debug agent log, in the US and EU regions.
+
 Start New Relic before `init`, as Rails does through the agent's Railtie. If the agent instruments Net::HTTP in chain mode and starts after `init`, every Net::HTTP request raises `SystemStackError`. The agent uses chain mode when `instrumentation.net_http` is `chain`, or on its own next to ScoutApm, Rack::MiniProfiler, or Airbrake before 10.0.2. Where the order cannot change, set `instrumentation.net_http` to `prepend`.
 
 If the app turns on New Relic's OpenTelemetry bridge (`opentelemetry.enabled`), New Relic takes the tracer provider and propagator, and Foam records no traces. Logs and metrics still reach Foam.

data/lib/foam/otel/internal/before_send.rb

@@ -19,10 +19,10 @@ module Foam
           freeze
         end
 
-        # Called by the exporters with each hook event they built; returns what to send, or nil to drop it.
-        # A hook that raises, returns something else, or gets an event that cannot be copied changes
-        # nothing, so a broken hook never loses telemetry. The block receives what the hook raised, so
-        # the exporter can say so once for its batch.
+        # Returns what to send for a hook event the exporters built, or nil to drop it. The event goes
+        # out unchanged when the hook raises, returns something other than an event of the same type,
+        # or gets an event that cannot be copied. A broken hook never loses telemetry. The block
+        # receives what the hook raised, and the exporter reports it once for its batch.
         def event_to_export(hook_event)
           return hook_event unless @hook
 
@@ -59,17 +59,24 @@ module Foam
             result[:links].is_a?(Array) && result[:links].all? { |entry| span_link_entry?(entry) }
         end
 
+        # Attributes that are not a Hash, or a timestamp that is not an Integer, would fail the export
+        # of the whole batch.
         def span_event_entry?(entry)
-          entry.is_a?(Hash) && entry[:name].is_a?(String)
+          entry.is_a?(Hash) && entry[:name].is_a?(String) &&
+            optional?(entry[:attributes], Hash) && optional?(entry[:timestamp], Integer)
         end
 
         def span_link_entry?(entry)
-          entry.is_a?(Hash) && entry[:span_context].is_a?(OpenTelemetry::Trace::SpanContext)
+          entry.is_a?(Hash) && entry[:span_context].is_a?(OpenTelemetry::Trace::SpanContext) &&
+            optional?(entry[:attributes], Hash)
         end
 
         def log_fields?(result)
-          (result[:severity_text].nil? || result[:severity_text].is_a?(String)) &&
-            (result[:severity_number].nil? || result[:severity_number].is_a?(Integer))
+          optional?(result[:severity_text], String) && optional?(result[:severity_number], Integer)
+        end
+
+        def optional?(value, type)
+          value.nil? || value.is_a?(type)
         end
 
         NONE = new(nil)

data/lib/foam/otel/internal/body_capture.rb

@@ -78,8 +78,9 @@ module Foam
           # - Foam has no logger provider
           # - the limit is reached
           # The block returns the headers as lowercase names to values; it runs at finalize, so
-          # headers set after the body started still count.
-          def start(span, body_redactor, side:, direction:, &headers)
+          # headers set after the body started still count. body_redactor defaults to the one init
+          # built.
+          def start(span, side:, direction:, body_redactor: State.body_redactor, &headers)
             Safely.call("#{direction} body capture") do
               return unless body_redactor
               return unless span.recording? && span.context.trace_flags.sampled?
@@ -97,6 +98,14 @@ module Foam
             end
           end
 
+          # Starts the request capture of a client span and gives it the whole body. Returns the
+          # capture, or nil when start captures nothing.
+          def client_request(span, body, &headers)
+            capture = start(span, side: :client, direction: :request, &headers)
+            capture&.observe(body)
+            capture
+          end
+
           # The clock the deadline is measured on, in monotonic seconds.
           def now
             Process.clock_gettime(Process::CLOCK_MONOTONIC)
@@ -123,13 +132,23 @@ module Foam
           # - the current span is not a recording client span, as for a sampled-out request or a
           #   host the instrumentation leaves untraced
           # - a middleware above the transport owns the span and captures on it itself
+          # - the lookup failed
           def client_span
-            return if OwnRequest.current?
+            Safely.call("body capture") do
+              return if OwnRequest.current?
+
+              span = OpenTelemetry::Trace.current_span
+              return unless span.recording? && span.kind == :client
 
-            span = OpenTelemetry::Trace.current_span
-            return unless span.recording? && span.kind == :client
+              span unless OpenTelemetry::Context.current.value(OWNER)&.span.equal?(span)
+            end
+          end
 
-            span unless OpenTelemetry::Context.current.value(OWNER)&.span.equal?(span)
+          # Returns the safe headers that are present, as lowercase names to values. The block finds
+          # the value of each name. Without a block, the lookup in hash ignores the case of names.
+          def safe_headers(hash = nil, &lookup)
+            lookup ||= ->(name) { hash[name] || hash.find { |key, _value| key.casecmp?(name) }&.last }
+            Constants::SAFE_NETWORK_HEADERS.to_h { |name| [name, lookup.call(name)] }.compact
           end
 
           # Sets the safe headers of one direction on the span, in the semantic conventions' dash

data/lib/foam/otel/internal/body_decoder.rb

@@ -9,8 +9,6 @@ module Foam
       # encoding within a bound, reads the charset, cuts the text at the capture limit, and splits
       # it into chunks. A body that is not text yields no text, and only its metadata is recorded.
       module BodyDecoder
-        INFLATED_ENCODINGS = %w[gzip x-gzip deflate].freeze
-
         # How much of a body of unknown type is looked at to tell whether it is text.
         SNIFF_BYTES = 1024
         # Control characters other than tab, line feed, form feed, and carriage return.
@@ -33,9 +31,11 @@ module Foam
         end
 
         # Returns the body as UTF-8 text and whether it was cut at the capture limit. Invalid
-        # bytes are replaced and the cut lands on a character boundary. Returns nil when the
-        # content type is binary, or unknown or missing and the bytes do not read as text, or the
-        # bytes are a file whatever the type says. A multipart body's parts are judged on their own.
+        # bytes are replaced, and the cut lands on a character boundary. Returns nil when:
+        # - the content type is binary
+        # - the content type is unknown or missing, and the bytes do not read as text
+        # - the bytes are a file, whatever the type says
+        # A multipart body is never judged a file as a whole; its parts are judged on their own.
         def text(bytes, content_type)
           readable = content_type.text? || (!content_type.binary? && textual?(bytes))
           readable &&= content_type.multipart? || !file?(bytes)
@@ -84,7 +84,7 @@ module Foam
         # framing alike, and its block form hands over the output in pieces, so inflating stops
         # as soon as the bound is passed.
         def inflate(data, encoding)
-          return unless INFLATED_ENCODINGS.include?(encoding)
+          return unless Constants::INFLATED_ENCODINGS.include?(encoding)
 
           inflater = Zlib::Inflate.new(Zlib::MAX_WBITS + 32)
           output = +""

data/lib/foam/otel/internal/body_redactor.rb

@@ -48,9 +48,9 @@ module Foam
           freeze
         end
 
-        # Masks the values under sensitive keys as the content type is read; a body of unknown or
-        # missing type is read as plain text, a binary type is left alone, and a failure masks the
-        # whole body rather than leak it.
+        # Masks the values under sensitive keys, reading the body by its content type. A body of
+        # unknown or missing type is read as plain text, and a binary type is left alone. A failure
+        # masks the whole body, so nothing leaks.
         def redact(text, content_type)
           type = ContentType.parse(content_type)
           redacted =

data/lib/foam/otel/internal/clients/excon.rb

@@ -13,9 +13,6 @@ module Foam
           # response or the error.
           REQUEST_CAPTURE = :foam_request_capture
 
-          # The encodings Excon's Decompress middleware inflates.
-          DECOMPRESSED = %w[gzip x-gzip deflate].freeze
-
           def self.install
             return unless Instrumentations.installed?(INSTRUMENTATION)
 
@@ -33,7 +30,7 @@ module Foam
           # Captures a String request body before it is written and keeps the capture on the
           # datum until the response or the error arrives.
           def request_call(datum)
-            span = Safely.call("body capture") { BodyCapture.client_span }
+            span = BodyCapture.client_span
             if span
               datum[REQUEST_CAPTURE] =
                 Safely.call("request body capture") { capture_request(datum, span) }
@@ -46,15 +43,18 @@ module Foam
           # headers of both directions, now that Excon has written the request's.
           def response_call(datum)
             datum.delete(REQUEST_CAPTURE)&.finalize(complete: true)
-            if (span = Safely.call("body capture") { BodyCapture.client_span })
+            span = BodyCapture.client_span
+            if span
               Safely.call("request header capture") do
-                BodyCapture.set_header_attributes(span, :request, headers(datum[:headers]))
+                headers = BodyCapture.safe_headers(datum[:headers])
+                BodyCapture.set_header_attributes(span, :request, headers)
               end
               Safely.call("response header capture") do
-                BodyCapture.set_header_attributes(span, :response, headers(datum[:response][:headers]))
+                headers = BodyCapture.safe_headers(datum[:response][:headers])
+                BodyCapture.set_header_attributes(span, :response, headers)
               end
             end
-            Safely.call("response body capture") { capture_response(datum) }
+            Safely.call("response body capture") { capture_response(datum, span) }
             @stack.response_call(datum)
           end
 
@@ -70,25 +70,20 @@ module Foam
             body = datum[:body]
             return unless body.is_a?(String) && !datum.key?(:request_block)
 
-            capture = BodyCapture.start(
-              span, State.body_redactor, side: :client, direction: :request
-            ) { headers(datum[:headers]) }
-            capture&.observe(body)
-            capture
+            BodyCapture.client_request(span, body) { BodyCapture.safe_headers(datum[:headers]) }
           end
 
           # Captures the parsed response body. A response handed to a response block never reaches
           # the datum, so it is skipped. When a middleware above owns the request, it is told
           # instead what Decompress is about to inflate.
-          def capture_response(datum)
-            span = BodyCapture.client_span
+          def capture_response(datum, span)
             body = datum.dig(:response, :body)
             return unless body.is_a?(String) && !datum.key?(:response_block)
             return report_inflating(datum, body) unless span
 
-            capture = BodyCapture.start(
-              span, State.body_redactor, side: :client, direction: :response
-            ) { headers(datum[:response][:headers]) }
+            capture = BodyCapture.start(span, side: :client, direction: :response) do
+              BodyCapture.safe_headers(datum[:response][:headers])
+            end
             return unless capture
 
             capture.observe(body)
@@ -101,19 +96,12 @@ module Foam
           def report_inflating(datum, body)
             return unless datum[:middlewares].include?(::Excon::Middleware::Decompress)
 
-            encoding = headers(datum[:response][:headers])["content-encoding"]&.split(",")&.last&.strip
-            return unless DECOMPRESSED.include?(encoding&.downcase)
+            header = BodyCapture.safe_headers(datum[:response][:headers])["content-encoding"]
+            encoding = header&.split(",")&.last&.strip
+            return unless Constants::INFLATED_ENCODINGS.include?(encoding&.downcase)
 
             BodyCapture.inflated_below(encoding, body.bytesize)
           end
-
-          # Matched without regard to case, since the request headers keep whatever case the app
-          # wrote.
-          def headers(hash)
-            Constants::SAFE_NETWORK_HEADERS.to_h do |name|
-              [name, hash.find { |key, _value| key.casecmp?(name) }&.last]
-            end.compact
-          end
         end
       end
     end