foam-otel 3.0.0.alpha.6 → 3.0.0.alpha.9

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4458e9959661525a33f70692beaeba2f32df91a68b01b7e611ec2ae2e2901b39
-  data.tar.gz: bb9127c1d897dec9f1a8268a338382fa1ffdfe772918abdf920d413ff10b986e
+  metadata.gz: 240906547c665dd40513921db5ab9f00d65febef81d40848faa91a1f5ae02e15
+  data.tar.gz: 9a2c4c5045ba080c21b165e1a87836cd9288408343b049c293352162e9be7083
 SHA512:
-  metadata.gz: d10e3d644eee2036278ae2c5310fc1afd33ec9acda7b27189247a0cb1dd88e1ebc433538cca528567278f97fa9103319a2e573d97f2ca201e24338b50f1c2764
-  data.tar.gz: 96eefe317fbf6eb8906e30242618fe69d90d303f0b3d2e74ceb008c968adc94884a39279b0a887e8e1ab7e5c1b4595dfd8aaf6e31eaeca123b451a002438f52f
+  metadata.gz: f891d7d8a749ce8a9042f08183581ec43924445d21137b863619a912cd0be0bb516e340d01e2ed6e238356a3b61f3cedb5739b769b04e38213d785712616cb9f
+  data.tar.gz: dd19f649f254cff7a00da71b14f158cc3d3d370a3affcc587ec86156d5439aaecd956564e106c54f0c0b2d229195065f69c03ab3eae9058459265ea479b71aaf

data/README.md

@@ -40,9 +40,9 @@ Foam.init(
 )
 ```
 
-During boot, `init` adds Foam's Rack middleware to the app directly after the OpenTelemetry Rack middleware, so server spans get the same header and body capture as under plain Rack. `init` must run before the app initializes: an initializer file or `config/application.rb`. An `init` from `after_initialize` or later warns on stderr and adds no middleware.
+`init` must run before the app initializes, either with an initializer file or in `config/application.rb`. An `init` from `after_initialize` or later warns on stderr and adds no middleware. During boot it inserts `Foam::Rack::Middleware` directly after the OpenTelemetry Rack middleware. `config.ru` stays as generated.
 
-Nothing else changes; `config.ru` stays as generated. A `RUBYOPT` preload runs before Rails loads, so it installs no Rails instrumentation and adds no middleware.
+Do not load Foam with a `RUBYOPT` preload. It runs before Rails loads, so it installs no Rails instrumentation and adds no middleware.
 
 ### Plain Ruby and Rack
 
@@ -64,7 +64,7 @@ use Foam::Rack::Middleware
 run App
 ```
 
-Foam's middleware sets the `content-type`, `content-length`, and `content-encoding` headers on server spans and, in `:advanced` mode, captures bodies. See Network capture.
+Foam's middleware records the `content-type`, `content-length`, and `content-encoding` headers as server span attributes. In `:advanced` mode it also sends the bodies as log records correlated to the span. See Network capture.
 
 ## Instrumentation
 
@@ -78,17 +78,13 @@ TODO(jhartquist): Revisit and review this compatibility guidance.
 
 ### Instrumentation versions
 
-Bundler keeps one version of each gem in the application. By the time Foam starts, those versions are already fixed. `additional_instrumentations` only configures an instrumentation that Bundler installed and Ruby loaded; it cannot choose or install another version.
+Bundler keeps one version of each gem, fixed before Foam starts. Each instrumentation checks its library's version when `init` runs and is left out of `state[:instrumentations]` when the version is unsupported. `additional_instrumentations` cannot bypass the check, choose another version, or install a gem.
 
-If a bundled instrumentation does not support the library version your app uses, pin a compatible `opentelemetry-instrumentation-*` gem in the `Gemfile` and run `bundle install` again.
-
-If Bundler accepts the pin, restart the app and the whole bundle will use that instrumentation version. If it conflicts with `foam-otel` or `opentelemetry-instrumentation-all`, those versions cannot live in the same bundle. At that point, change the application library, use a compatible Foam release, or accept the telemetry gap: the app can keep running, but Foam will not receive the spans that instrumentation would have produced.
-
-Even a successful `bundle install` only means the declared gem dependencies agree. The instrumentation checks the target library when `init` runs. If the target is unsupported, Foam leaves that instrumentation out of `state[:instrumentations]`. `additional_instrumentations` cannot bypass the check or load a second version of the gem.
+For an unsupported library version, change the app's library, use a compatible Foam release, or accept the gap. The app keeps running, but Foam receives no spans from that library.
 
 The `pg`, `mysql2`, and `trilogy` instrumentations obfuscate SQL statements (`db_statement: :obfuscate`, their default), replacing every literal with `?`, so `WHERE email = 'ada@example.com'` is exported as `WHERE email = ?`. A statement over 2000 characters is exported as a placeholder instead. The other instrumentations keep their upstream defaults.
 
-To disable one instrumentation or change its options, pass it in `additional_instrumentations` or use the upstream environment variables, which take precedence: `OTEL_RUBY_INSTRUMENTATION_<NAME>_ENABLED=false` and `OTEL_RUBY_INSTRUMENTATION_<NAME>_CONFIG_OPTS`, where `<NAME>` is the instrumentation's class name after `OpenTelemetry::Instrumentation::`, upper-cased, with `::` replaced by `_`: `PG`, `NET_HTTP`, `ACTIVERECORD`.
+To disable one instrumentation or change its options, pass it in `additional_instrumentations` or set the upstream environment variables `OTEL_RUBY_INSTRUMENTATION_<NAME>_ENABLED=false` and `OTEL_RUBY_INSTRUMENTATION_<NAME>_CONFIG_OPTS`, which take precedence. `<NAME>` is the instrumentation's registry name after `OpenTelemetry::Instrumentation::`, upper-cased, with `::` replaced by `_`, such as `PG`, `NET_HTTP`, or `ACTIVERECORD`.
 
 TODO(jhartquist): `db_capture` init option. The `pg`, `mysql2`, and `trilogy` instrumentations can include the statement (`db_statement: :include`) and `pg` can report row counts (`db_response_returned_rows`, `db_response_affected_rows`), but none hands over results. Result capture needs Foam patches on the drivers, opt-in, through the same redaction and size caps as bodies. See Expanded data capture at the end of this document.
 
@@ -96,13 +92,13 @@ TODO(jhartquist): Gen AI providers beyond `anthropic`. There is no OpenTelemetry
 
 ## Public API
 
-Every method lives on the `Foam` module. Other than the ingest factories, none of them raise exceptions. `set_baggage`, the metric helpers, and `log` need a Foam export path to function properly, so they are no-ops before `init`; the trace context helpers and `record_exception` work on the current OpenTelemetry context whether or not `init` ran.
+Every method lives on the `Foam` module. A missing or invalid value is reported in Foam's log output instead of raising; only an unknown keyword raises `ArgumentError`. `set_baggage`, the metric helpers, and `log` need a Foam export path to function properly, so they are no-ops before `init`; the trace context helpers and `record_exception` work on the current OpenTelemetry context whether or not `init` ran.
 
 ### `init`
 
 Sets up tracing, metrics, logs, automatic instrumentation, W3C `tracecontext` and `baggage` propagation, and OTLP export to Foam.
 
-`init` claims the four OpenTelemetry global slots only when they are free: the tracer provider, meter provider, logger provider, and propagator. A slot another SDK set before `init` stays with that SDK. Foam warns once on stderr, reports the signal in `state`, and builds no pipeline for it, so any `additional_*` components for that signal are unused. Foam always builds its own logger provider, so `log` delivers either way. `shutdown` releases the slots Foam claimed.
+`init` claims the four OpenTelemetry global slots only when they are free: the tracer provider, meter provider, logger provider, and propagator. A slot another SDK sets before `init` stays with that SDK. Foam warns once on stderr, reports the signal in `state`, and builds no pipeline for it, so any `additional_*` components for that signal are unused. Foam always builds its own logger provider, so `log` delivers either way. `shutdown` releases the slots that Foam claimed.
 
 Call this once. Use it when Foam should run OpenTelemetry for the process.
 
@@ -148,11 +144,11 @@ Foam.init(
 - `token` - the Foam ingest token, sent as a bearer token on every export. Required when Foam is enabled; a missing token is reported on stderr and leaves Foam uninitialized.
 - `version` - the service version, sent as `service.version` when given.
 - `sample_rate` - the fraction of root traces kept, from `0.0` to `1.0`; child spans follow their parent. Leave it at `1.0` unless volume forces sampling. `OTEL_TRACES_SAMPLER` is not consulted.
-- `additional_span_processors`, `additional_log_record_processors`, `additional_metric_readers` - the app's own OpenTelemetry components, added to Foam's providers after Foam's own exporters.
+- `additional_span_processors`, `additional_log_record_processors`, `additional_metric_readers` - span processors, log record processors, and metric readers the app builds itself, such as an exporter to another backend. Foam adds them to its providers after its own exporters.
 - `additional_resource_attributes` - attributes merged into the resource of every signal. `service.name`, `service.version`, and `deployment.environment.name` come from the options above and `service.instance.id` from Foam; a value for one of them here is reported on stderr and ignored.
 - `additional_instrumentations` - instrumentations to install next to the bundle. An entry is an instrumentation's registry name, which is its class name without the trailing `::Instrumentation` (`"OpenTelemetry::Instrumentation::Logger"`), or a `[name, config]` pair to install it with options (`["OpenTelemetry::Instrumentation::PG", {db_statement: :include}]`). Require the instrumentation's gem before `init`; an entry whose instrumentation is not loaded fails `init`. An entry naming a bundled instrumentation replaces Foam's configuration for it.
 - `ignored_outbound_hosts` - skips tracing of outbound HTTP requests to these hosts, each written `host` or `host:port`. The endpoint host is always skipped. Use it for hosts whose traffic would drown the traces, such as a sidecar health check on `localhost`, another SDK's agent on `127.0.0.1:8126`, or another vendor's OTLP ingest host. Hosts are compared as written; `localhost` and `127.0.0.1` are different hosts, and an entry with a port matches that port only. A request to a skipped host still carries a `traceparent` marked not sampled, so an OpenTelemetry service behind it records nothing for that request.
-- `ignored_outbound_headers` - skips tracing of outbound Net::HTTP requests carrying any of these request headers, compared case-insensitively. Another SDK's transport stamps its own header on every request to its backend, such as `Datadog-Meta-Tracer-Version`, which tells that traffic apart when it has no host to ignore, as over a Unix socket. A connection opened before the request, as `Net::HTTP.start` does, has its connect span before any header is seen, so over TCP ignore the host as well.
+- `ignored_outbound_headers` - skips tracing of outbound Net::HTTP requests carrying any of these request headers, compared case-insensitively. Use it for another SDK's traffic to its backend when there is no host to ignore, as over a Unix socket. A connection opened before the request, as `Net::HTTP.start` does, has its connect span before any header is seen, so over TCP ignore the host as well.
 - `disable_log_sending` - `true` stops shipping the app's `Logger` lines to Foam. `Foam.log` is unaffected. See Loggers.
 - `network_capture` - `:off`, `:basic`, or `:advanced`. Use `:advanced` unless security and compliance rule it out. See Network capture.
 - `redact` - keys to mask beyond the built-in list, as `{secrets: [...], pii: [...]}`. A value under a `secrets` key keeps its last four characters when it is 12 characters or longer; a value under a `pii` key is fully masked. See Redaction.
@@ -170,15 +166,15 @@ Every signal carries a resource with these attributes:
 
 A process forked after `init` carries its own `service.instance.id` and `process.pid`. `OTEL_RESOURCE_ATTRIBUTES` can add further attributes. It cannot override any of the attributes above, and `OTEL_SERVICE_NAME` cannot override `name`.
 
-`init` never raises. A failure or an option holding an invalid value is reported on stderr with a `[foam-otel]` prefix and leaves `state` uninitialized.
+`init` raises only `ArgumentError`, for a missing `name` or `environment` or an unknown keyword. Any other failure, or an option holding an invalid value, is reported on stderr with a `[foam-otel]` prefix and leaves `state` uninitialized.
 
 `init` also reports its outcome to Foam: one log record with the message and the `state`, sent straight to the endpoint with the token. The report is sent in the background and never blocks the app. When the process exits before the report has been delivered, exit waits up to two seconds for the send to finish; a report that cannot be delivered is dropped. Nothing is sent without a token.
 
 ### `set_endpoint(url)`
 
-For tests and local development only. Sets the OTLP/HTTP base URL, without a signal path, that every export and report goes to, such as a local collector. Production apps do not call it: the default is Foam's ingest endpoint, `https://otel.api.foam.ai`.
+For tests and local development only. Call it before `init` to send every export to another OTLP/HTTP base URL, without a signal path, such as a local collector. The default is Foam's ingest endpoint, `https://otel.api.foam.ai`.
 
-Call it before `init` or before creating an ingest component. Exporters keep the endpoint they were built with, so a later call changes only later reports and ingest components; after `init` it is also reported on stderr. `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_HEADERS`, and their per-signal forms are not consulted. The upstream exporters still read the timeout, compression, and certificate variables.
+`OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_HEADERS`, and their per-signal forms are not consulted. The upstream exporters still read the timeout, compression, and certificate variables.
 
 ### `create_foam_ingest_span_processor(name, environment, token, redact: nil, before_send: nil)`
 
@@ -225,7 +221,7 @@ All three export to the endpoint set by `set_endpoint` and take the same `redact
 
 Foam identifies the service by the `name` and `environment` passed in: on the copy sent to Foam they overwrite the provider's `service.name` and `deployment.environment.name`, and `foam.ingest.host` names the host of the endpoint the copy goes to. The provider's own telemetry is left unchanged. Pass the same values to all three.
 
-Unlike every other public method, these are not guarded: they raise `ArgumentError` when `name`, `environment`, or `token` is blank or `redact` or `before_send` is invalid, and let any other failure through. Returning nothing instead would fail later, inside the app's provider, with less to go on.
+A missing or blank `name`, `environment`, or `token`, or an invalid `redact` or `before_send` value, is reported in Foam's log output and turns off just that helper: it returns a component that exports nothing, which the app adds to its provider as usual, and the signal stays `none` in `state`, so an unset environment variable never stops the app from booting.
 
 Do not combine `init` and an ingest component for the same signal. The ingest components do not install instrumentations; register the ones the app needs with the OpenTelemetry SDK directly.
 
@@ -256,7 +252,7 @@ end
 
 `inject_trace_context` and `extract_trace_context` always use the W3C `traceparent`, `tracestate`, and `baggage` headers, whatever propagator holds the global slot. Foam does not read `OTEL_PROPAGATORS`. An app that needs a different propagator registers it before `init`, and Foam leaves it in place.
 
-On the outbound requests Foam instruments, the propagation headers are written last, at the transport, after any other SDK's: Net::HTTP, HTTPClient, Excon, http.rb, Ethon, gRPC, and Sidekiq job payloads, and Faraday, RestClient, and Typhoeus through those. Another SDK's own propagation headers are left as written.
+On outbound requests, Foam writes its propagation headers after any other SDK's. Another SDK's own propagation headers are left as written.
 
 ### `set_baggage(key, value)`
 
@@ -284,13 +280,13 @@ Foam.set_metric("cache.size", 1024, unit: "By", description: "Entries in the cac
 - `add_up_down_counter(name, value, ...)` adds to a counter that can go down; pass a negative value to decrement.
 - `set_metric(name, value, ...)` sets a gauge to its latest value.
 
-Every helper takes `attributes: {}`, `unit: nil`, and `description: nil`. The instrument is created on the first call for a name, and `unit` and `description` apply then; later calls reuse it. Values are exported on Foam's metrics pipeline under the `foam-otel` scope. Calls before `init`, when Foam is disabled, or in a setup that only attaches an ingest metric reader are dropped.
+Every helper takes `attributes: {}`, `unit: nil`, and `description: nil`. The instrument is created on the first call for a name, and `unit` and `description` apply then; later calls reuse it. Without `init`, when `create_foam_ingest_metric_reader` is attached to the app's own meter provider, the helpers record on the global meter provider, so their values reach Foam through that reader. Calls before `init` with no ingest metric reader, or when Foam is disabled, are dropped.
 
 TODO(jhartquist): observable (pull) instruments. The metrics SDK has observable counters, gauges, and up-down counters; the helpers cover the four synchronous kinds only. v1 used observables for runtime metrics (GC, heap, threads, RSS), which this version does not send.
 
 ### `log(body, severity: :info, attributes: {})`
 
-Sends a log record to Foam on Foam's own export path. The record goes only to Foam: nothing is written to the app's log output, and nothing is emitted through a logger provider Foam does not own. A record emitted inside a request carries the trace and span ids of the current span.
+Sends a log record to Foam on Foam's own export path. The record goes only to Foam: nothing is written to the app's log output, and nothing is emitted through a logger provider Foam does not own. A record emitted inside a request carries the trace and span IDs of the current span.
 
 Use it to send a log directly, for example when the app's logger is not captured automatically (see Loggers).
 
@@ -318,7 +314,7 @@ The span gets an `exception` event with the type, message, and stack trace, plus
 
 ### `flush(timeout: 10)`
 
-Exports everything Foam's providers still hold. Call it before a point where the process may be killed without a normal exit, for example at the end of a job on a runner that stops the process.
+Exports everything Foam's providers and ingest components still hold. Call it before a point where the process may be killed without a normal exit, for example at the end of a job on a runner that stops the process.
 
 ```ruby
 Foam.flush(timeout: 5)
@@ -328,11 +324,11 @@ Foam.flush(timeout: 5)
 
 ### `shutdown(timeout: 10)`
 
-Flushes and shuts down Foam's providers, including the one behind `log` when an ingest component created it, and releases the global slots Foam claimed. Both `init` and the ingest log factory register it with `at_exit`, so most apps never call it. An app calls it before a process is killed without a normal exit.
+Flushes and shuts down Foam's providers, including the one behind `log` when an ingest component created it, and the ingest components, and releases the global slots Foam claimed. `init` and each ingest factory register it with `at_exit`, so most apps never call it, and an app that never shuts its own providers down still delivers the ingest components' last batch. An app calls it before a process is killed without a normal exit.
 
 `timeout` is in seconds and bounds the whole call. A provider that runs out of time is reported on stderr.
 
-`at_exit` covers a normal exit, an unhandled exception, and a signal the process handles by exiting. `exit!`, `KILL`, and a crash lose the last batch, and so does a batch that outlasts the timeout. An exception outside any span is invisible unless the app logs it.
+`at_exit` covers a normal exit, an unhandled exception, and a signal the process handles by exiting. The last batch is lost when the process ends without running `at_exit`, as with `exit!`, a `KILL` signal, or a crash of the Ruby process itself, and when a batch outlasts the timeout. An exception outside any span is invisible unless the app logs it.
 
 Foam's span processor holds 4096 finished spans and drops the oldest when full; `OTEL_BSP_MAX_QUEUE_SIZE` overrides the size. The ingest span processor and the log record processors keep the OpenTelemetry default of 2048.
 
@@ -371,7 +367,7 @@ Each `signals` value names the source of the Foam export path for that signal:
 - `:local` - logs only: another SDK owns the global provider and Foam keeps its own so `log` still delivers.
 - `:none` - no Foam export path.
 
-A slot another SDK held before `init` stays with it and reports `:none` (`:local` for logs) from the start. Foam builds no provider for that signal, so the metric helpers and `set_baggage` do nothing while `log` still delivers. `:global` is checked against the live OpenTelemetry global on each call, so a slot another SDK takes after `init` reports the same way; in that case only the slot changes hands, and the helpers keep using Foam's own providers. Both cases are warned about once on stderr. Body capture is off while another SDK holds the logs slot. `baggage` is `:global` or `:none`. `profile` is not supported and always reports `:none`.
+A slot another SDK held before `init` stays with it and reports `:none` (`:local` for logs), or `:ingest` when a `create_foam_ingest_*` object was already built for that signal. Foam builds no provider for that signal, so the metric helpers and `set_baggage` do nothing while `log` still delivers. A slot another SDK takes after `init` reports the same way, but Foam's helpers keep exporting through Foam's own providers. Both cases are warned about once on stderr. Body capture is off while another SDK holds the logs slot. `baggage` is `:global` or `:none`. `profile` is not supported and always reports `:none`.
 
 ## Custom spans and other OpenTelemetry APIs
 
@@ -387,56 +383,38 @@ TODO(jhartquist): custom spans and the OpenTelemetry API: getting a tracer, `in_
 
 There is no option to record other headers.
 
-### Header capture
-
-Server spans get the headers from `Foam::Rack::Middleware`, placed after the OpenTelemetry Rack middleware (added by `init` in Rails apps).
+Server spans get the capture from `Foam::Rack::Middleware`, placed after the OpenTelemetry Rack middleware (added by `init` in Rails apps). Client spans get it from `Net::HTTP`, Faraday with any adapter, Excon, and HTTPClient when their gem is loaded before `init`. RestClient and HTTParty are captured on the `Net::HTTP` span beneath them. `http`, `httpx`, and `ethon` get no capture, and neither does a Faraday connection built before `init`.
 
-Client spans get the headers from Foam's hooks on `Net::HTTP`, Excon, HTTPClient, and Faraday, applied unless `network_capture` is `:off`. The `Net::HTTP` hook is always installed; the others install only when their gem is loaded before `init`, and Faraday's covers connections built after `init`. Faraday is covered on the Faraday span whatever the adapter. `http`, `httpx`, `ethon`, and `restclient` are not covered.
-
-Faraday over the `httpclient` adapter sends each request with the trace marked not sampled, so a Foam server on the other side records no span for it. The issue is in the upstream HTTPClient instrumentation, which ignores the untraced context Faraday runs its adapter in; Net::HTTP and Excon honor it.
+Streamed bodies are captured for a Rack response the server iterates, a Rack request body as the app reads it, a `Net::HTTP` request sent through `body_stream`, and an HTTPClient response read in a block. Other streamed bodies are not, such as a Rack `call` body, a file sent by path, a hijacked connection, a client request body sent from an IO, or a client response read in a block or through a streaming callback.
 
 TODO(jhartquist): cover `http`, `httpx`, `ethon`, and `restclient`.
 
-TODO(jhartquist): the HTTPClient trace break above. The fix is `return super if OpenTelemetry::Common::Utilities.untraced?` at the top of the instrumentation's `do_get_block`, upstream or redefined at init until a release carries it.
-
-### Body capture
-
-In `:advanced` mode each captured body arrives as `foam.http.body.chunk` log records correlated to the span, and the span carries the body's size and content and whether the capture was complete.
+### Bodies
 
+- Each captured body arrives as `foam.http.body.chunk` log records correlated to the span, each carrying a chunk of the body's text as its log body. The span carries the body's size and whether the capture was complete.
 - Bodies with a textual content type, or one declaring a charset, are captured. gzip and deflate are decoded. Other encodings and binary types yield one record with the size only; an empty body yields none.
 - At most 1 MiB of a body is kept, sent as records of at most 64 KiB each. The rest counts toward the size.
-- Bodies are parsed by content type and redacted with the same keys as span attributes.
-
-### Rack bodies
-
-Server bodies need `Foam::Rack::Middleware` after the OpenTelemetry Rack middleware, as shown under "Plain Ruby and Rack" (added by `init` in Rails apps).
+- Bodies are parsed by content type and redacted with the same keys as span attributes. See Redaction.
+- A server request body is captured whether or not the app reads it when the input is rewindable, as it is under Puma. Otherwise only what the app reads is captured.
+- A streamed server response is observed as it is sent, without buffering or delaying it. Its records are emitted when the body closes; a stream idle for over 60 seconds is closed as incomplete and the rest is dropped.
 
-- The request body is captured whether or not the app reads it when the input is rewindable, as it is under Puma, which buffers request bodies. Otherwise what the app reads is captured.
-- A streamed response is observed as the server sends it, without buffering or delaying it. The records are emitted when the body closes; a stream idle for over 60 seconds is closed as incomplete and the rest is dropped.
-- A streaming body (`call`), a file sent by path (`to_path`), and a hijacked connection are not captured.
+### Known issue
 
-### Client bodies
+Faraday over the `httpclient` adapter sends each request with the trace marked not sampled, so a Foam service on the other side records no span for it. Faraday over `Net::HTTP` or Excon is unaffected.
 
-Client bodies are captured in `:advanced` by the same hooks that set the headers.
-
-- `Net::HTTP`: a request is captured whether sent as a string or through `body_stream`. A response read in a block or into an IO is not captured.
-- Faraday: bodies are captured on the Faraday span through Foam's Faraday middleware, whatever the adapter. A request whose body is an IO and a response streamed with `on_data` are not captured.
-- Excon: a request sent with `request_block` and a response received with `response_block` are not captured.
-- HTTPClient: a request whose body is an IO or multipart parts is not captured. A response read in a block is captured.
-- RestClient and HTTParty: captured on the `Net::HTTP` span beneath them.
-- `http`, `httpx`, and `ethon`: no bodies, since they get no header capture either.
+TODO(jhartquist): the HTTPClient trace break above. The fix is `return super if OpenTelemetry::Common::Utilities.untraced?` at the top of the instrumentation's `do_get_block`, upstream or redefined at init until a release carries it.
 
 TODO(jhartquist): capture the streamed bodies listed above. For `Net::HTTP`, prepend on `Net::HTTPResponse#read_body` and tee the block or IO the app passes.
 
 ## Loggers
 
-Foam installs `opentelemetry-instrumentation-logger`, which hooks `Logger#format_message`, so every logger built on `Logger` is covered, including one created before `init`. Each line becomes one log record: the body is the formatted line and the severity follows the `Logger` level, except that `unknown` arrives with no severity. The app's own log output is unchanged; the trace and span IDs are on the record sent to Foam, not written into the app's lines. The record carries the resource attributes but none of its own: a JSON-formatted line arrives as one string body, with its fields inside the text. Use `Foam.log` to send a record with attributes.
+Foam installs `opentelemetry-instrumentation-logger`, so every logger built on `Logger` is covered, including one created before `init`. Each line becomes one log record: the body is the formatted line and the severity follows the `Logger` level, except that `unknown` arrives with no severity. The app's own log output is unchanged; the trace and span IDs are on the record sent to Foam, not written into the app's lines. The record carries the resource attributes but none of its own: a JSON-formatted line arrives as one string body, with its fields inside the text. Use `Foam.log` to send a record with attributes.
 
 Covered:
 
 - `Logger`: every line except `Logger#<<`, which writes to the device directly.
 - Rails 7.1 and later: `Rails.logger`, `ActiveSupport::Logger`, `TaggedLogging`, and `BroadcastLogger`. A broadcast sends one record per line, tags included in the body, provided ActiveSupport is loaded before `init`; otherwise every sink sends its own record.
-- `Sidekiq::Logger`, MonoLogger, LogStashLogger, Ougai, and fluent-logger build on `Logger` and should be covered; Lograge formats `Rails.logger` lines and is covered through it. LogStashLogger's `MultiLogger` should send one record per logger it wraps. None of these is tested here.
+- `Sidekiq::Logger`, MonoLogger, LogStashLogger, Ougai, and fluent-logger build on `Logger` and are covered; Lograge formats `Rails.logger` lines and is covered through it. LogStashLogger's `MultiLogger` sends one record per logger it wraps.
 
 Not covered, since they do not build on `Logger`: Semantic Logger, `logging`, log4r, yell, and Puma's own server lines. Use `Foam.log` for a record from these. Semantic Logger also ships an `:open_telemetry` appender. `puts` and direct writes to `$stdout` or `$stderr` are never captured, nor is a `Logger` whose device is `nil` or `File::NULL`.
 
@@ -486,19 +464,40 @@ Foam.init(
 )
 ```
 
+If the app loads Datadog's OpenTelemetry support (`require "datadog/opentelemetry"`) and configures the OpenTelemetry SDK itself, `OpenTelemetry::SDK.configure` installs Datadog's span processor on the app's tracer provider, next to the ingest helpers. Two Datadog bugs then affect what Foam receives:
+
+- Once a child span ends, spans started afterwards under the same parent begin new traces. Foam receives the trace split at that point, and logs written afterwards are not correlated to the parent. Datadog's own copy of the trace is complete.
+- `force_flush` and `shutdown` on the app's tracer provider raise inside Datadog's span processor, and the processors after it, the ingest span processor included, are skipped. The app does not need either call for Foam: Foam flushes the ingest components at exit.
+
+Where the app does not need to own the OpenTelemetry SDK, call `init` instead and do not call `OpenTelemetry::SDK.configure`. Datadog's span processor is then not installed, so neither bug applies. Spans made through the OpenTelemetry API go to Foam only, and Datadog keeps what its own integrations record.
+
+### Rollbar
+
+Rollbar sends each report to its API over HTTP, which Foam would otherwise record as spans. Pass Rollbar's endpoint and request header to `init`. This works whether Rollbar reports inline, from a background thread, or from a Sidekiq job:
+
+```ruby
+rollbar = URI(Rollbar.configuration.endpoint)
+
+Foam.init(
+  name: "checkout-api", environment: "production", token: ENV["FOAM_OTEL_TOKEN"],
+  ignored_outbound_hosts: ["#{rollbar.host}:#{rollbar.port}"],
+  ignored_outbound_headers: ["X-Rollbar-Access-Token"]
+)
+```
+
 ## Redaction
 
-Foam masks the values of sensitive keys in everything it exports: span, span event, and span link attributes; log bodies and attributes; metric data point attributes; and resource attributes. A built-in list of credential and personal-data keys always applies, matched by exact name or by a sensitive segment of the key (`legacy_api_key_2`, `stripeToken`). Keys match after normalization: case, `-`, `.`, and camelCase boundaries are folded, so `apiKey`, `api-key`, and `API_KEY` are one key. Header attributes match on the header name. SQL statements arrive from the database instrumentations with every literal replaced with `?` (see Instrumentation) and are redacted like any other string. Captured HTTP bodies are masked by content type with the same keys, with their framing kept. When masking a body fails, the whole body is replaced with `********`.
+Foam masks the values of sensitive keys in everything it exports: span, span event, and span link attributes; log bodies and attributes; metric data point attributes; and resource attributes. A built-in list of credential and personal-data keys always applies, matched by exact name or by a sensitive segment of the key (`legacy_api_key_2`, `stripeToken`). Keys match after normalization: case, `-`, `.`, and camelCase boundaries are folded, so `apiKey`, `api-key`, and `API_KEY` are one key. Header attributes match on the header name. SQL statements arrive from the database instrumentations with every literal replaced with `?` (see Instrumentation) and are redacted like any other string. Captured HTTP bodies are masked by content type with the same keys. When masking a body fails, the whole body is replaced with `********`.
 
 Every other string is scanned as text, log bodies included.
 
 - A string that is a whole JSON document is masked by key at every depth. A list or object under a sensitive key becomes one mask.
-- Any other string has the value after each sensitive key masked. The key may be quoted as JSON writes it or bare before `=`, `:`, or `=>`, in a query string, a header line, or prose. The value ends at the next delimiter, and a quoted URL keeps its closing quote.
+- Any other string has the value after each sensitive key masked. The key may be quoted as JSON writes it or bare before `=`, `:`, or `=>`, in a query string, a header line, or prose. The value ends at the next delimiter.
 - `url.query` is masked pair by pair. Names such as `code` and `sig` are sensitive only inside a query string.
 
 Scanning stops after 1 MiB, and everything past that point is replaced with a single mask. Masking is value-only: the key and the surrounding text remain intact.
 
-The `redact` option adds the app's own keys, matched exactly after the same normalization. A value under a built-in or `pii` key becomes `[REDACTED]`. A value under a `secrets` key becomes `********`, followed by its last four characters when it is 12 characters or longer. That tail is kept only where the value is read whole under its key: an attribute, a hash or array, or a whole JSON document. A value found by scanning text or a `url.query` pair becomes `[REDACTED]` whatever its key.
+The `redact` option adds the app's own keys, matched exactly after the same normalization. A value under a built-in or `pii` key becomes `[REDACTED]`. A value under a `secrets` key becomes `********`, followed by its last four characters when it is 12 characters or longer.
 
 Masking changes values only: keys stay present, and span names, event names, and severity are untouched. It is applied after `before_send`, to the copy sent to Foam, through `init` and through the ingest components alike. The app's own span and log objects, and the copies handed to other processors and exporters, are unchanged.
 
@@ -598,18 +597,15 @@ OpenTelemetry packages outside the instrumentation bundle that Foam does not wir
 
 ## TODO(jhartquist): Safely
 
-Every public method except the ingest factories, and every capture hook, runs under `Internal::Safely`: a failure inside the gem is written to stderr with the `[foam-otel]` prefix and the call returns a fallback, so nothing reaches the app.  One broad boundary per public call and per hook, instead of a wrap at each site, would give one place to rate-limit and report, at the cost of skipping the rest of a hook after its first failure. Additionally, a failure could also be reported to Foam over the path `init` uses for its own outcome.
+Every public method, and every capture hook, runs under `Internal::Safely`: a failure inside the gem is written to stderr with the `[foam-otel]` prefix and the call returns a fallback, so nothing reaches the app.  One broad boundary per public call and per hook, instead of a wrap at each site, would give one place to rate-limit and report, at the cost of skipping the rest of a hook after its first failure. Additionally, a failure could also be reported to Foam over the path `init` uses for its own outcome.
 
 ## Development
 
 ```sh
 bundle install
 bundle exec rspec             # unit tests
-bundle exec rake e2e          # end-to-end tests
 bundle exec standardrb --fix  # lint and autoformat
-bundle exec rake              # unit, e2e, then lint (what CI checks)
+bundle exec rake              # unit, then lint (what CI checks)
 ```
 
-The end-to-end tests run a real OTLP collector and the fixture apps under `e2e/apps/` with puma, then assert on what the collector received. `rake e2e` downloads the pinned `otelcol-contrib` release to `~/.cache/foam-otel` the first time (about 350 MB; set `FOAM_OTELCOL_DIR` to cache it elsewhere), and the suite installs a fixture app's bundle when it starts the app. Run one file with `bundle exec rspec e2e/spec/rack_init_spec.rb`; that does not download the collector, so run `rake e2e:install` once first.
 
-Each run writes the collector config, the received telemetry, and the output of both processes to `tmp/e2e/`. A failure prints the tail of both stderr files.

data/lib/foam/otel/internal/before_send.rb

@@ -12,34 +12,38 @@ module Foam
           "before_send must respond to call, got #{hook.inspect}"
         end
 
-        # Takes the before_send option as init or an ingest factory received it; nil means no hook.
+        # Takes the before_send option as init or an ingest factory received it, after checking it
+        # with invalid_option; nil means no hook.
         def initialize(hook)
-          problem = BeforeSend.invalid_option(hook)
-          raise ArgumentError, problem if problem
-
           @hook = hook
           freeze
         end
 
         # Called by the exporters with each hook event they built; returns what to send, or nil to drop it.
         # A hook that raises, returns something else, or gets an event that cannot be copied changes
-        # nothing, so a broken hook never loses telemetry.
+        # nothing, so a broken hook never loses telemetry. The block receives what the hook raised, so
+        # the exporter can say so once for its batch.
         def event_to_export(hook_event)
           return hook_event unless @hook
 
-          result = @hook.call(deep_copy(hook_event))
+          copy = deep_copy(hook_event) or return hook_event
+          result = @hook.call(copy)
           return if result.nil?
 
           hook_event?(result, hook_event[:type]) ? result : hook_event
-        rescue
+        rescue => error
+          yield error
           hook_event
         end
 
         private
 
-        # Marshal thaws the SDK's frozen attributes, so the hook can edit in place.
+        # Marshal thaws the SDK's frozen attributes, so the hook can edit in place. nil for an event
+        # holding something Marshal cannot dump, such as an IO.
         def deep_copy(hook_event)
           Marshal.load(Marshal.dump(hook_event))
+        rescue
+          nil
         end
 
         def hook_event?(result, type)

data/lib/foam/otel/internal/body_capture.rb

@@ -19,18 +19,22 @@ module Foam
         CHUNK_COUNT = "foam.http.body.chunk.count"
         CHUNK_INDEX = "foam.http.body.chunk.index"
         CHUNK_BYTES = "foam.http.body.chunk.bytes"
-        CHUNK_CONTENT = "foam.http.body.chunk.content"
 
-        # The context key a capturing middleware marks its span under, so the transport patches
-        # beneath it leave that span alone.
-        OWNED_SPAN = OpenTelemetry::Context.create_key("foam-otel.owned-span")
+        # Where a middleware that captures above its transport, such as Faraday, puts its Owner in
+        # the context. The transport patches beneath it then skip its span and report to it what
+        # they inflated.
+        OWNER = OpenTelemetry::Context.create_key("foam-otel.owner")
 
-        # The span attributes of each direction. Size, content, and the header prefix are the
-        # semantic conventions' names; the rest are Foam's.
+        # The span a middleware captures on, above its transport. When the transport inflates the
+        # response, it sets encoding and wire_bytes to the response's encoding and size on the wire;
+        # otherwise both stay nil.
+        Owner = Struct.new(:span, :encoding, :wire_bytes)
+
+        # The span attributes of each direction. Size and the header prefix are the semantic
+        # conventions' names; the rest are Foam's.
         SPAN_ATTRIBUTES = %i[request response].to_h do |direction|
           [direction, {
             size: "http.#{direction}.body.size",
-            content: "http.#{direction}.body.content",
             capture_id: "foam.http.#{direction}.body.capture_id",
             complete: "foam.http.#{direction}.body.complete",
             truncated: "foam.http.#{direction}.body.truncated",
@@ -98,10 +102,19 @@ module Foam
             Process.clock_gettime(Process::CLOCK_MONOTONIC)
           end
 
-          # Runs the block with span owned by the caller, so the transport patches inside it
-          # capture nothing on that span and a library capturing above its transport captures once.
-          def owning(span, &block)
-            OpenTelemetry::Context.with_value(OWNED_SPAN, span, &block)
+          # Runs the block under the owner, so the transport patches inside it capture nothing on
+          # its span and each request is captured once.
+          def owning(owner, &block)
+            OpenTelemetry::Context.with_value(OWNER, owner, &block)
+          end
+
+          # Called by a transport patch that inflated a response: records the response's encoding
+          # and size on the wire on the middleware that owns the request. Does nothing when no
+          # middleware owns it.
+          def inflated_below(encoding, wire_bytes)
+            owner = OpenTelemetry::Context.current.value(OWNER)
+            owner&.encoding = encoding
+            owner&.wire_bytes = wire_bytes
           end
 
           # Returns the recording client span an outbound request belongs to, or nil when nothing is
@@ -116,7 +129,7 @@ module Foam
             span = OpenTelemetry::Trace.current_span
             return unless span.recording? && span.kind == :client
 
-            span unless OpenTelemetry::Context.current.value(OWNED_SPAN).equal?(span)
+            span unless OpenTelemetry::Context.current.value(OWNER)&.span.equal?(span)
           end
 
           # Sets the safe headers of one direction on the span, in the semantic conventions' dash
@@ -142,6 +155,7 @@ module Foam
           @chunks = []
           @observed_bytes = 0
           @kept_bytes = 0
+          @wire_bytes = nil
           @truncated = false
           @open = true
           @lock = Mutex.new
@@ -159,12 +173,16 @@ module Foam
             @last_activity = BodyCapture.now
             @observed_bytes += bytes.bytesize
             room = Constants::BODY_CAPTURE_BYTES - @kept_bytes
+            # The caller's string is copied, since the app may still write to it; a slice is a
+            # fresh string already, so it is relabeled as bytes rather than copied again.
             if bytes.bytesize > room
               @truncated = true
-              bytes = bytes.byteslice(0, room)
+              kept = bytes.byteslice(0, room).force_encoding(Encoding::BINARY)
+            else
+              kept = bytes.b
             end
-            @chunks << bytes.b unless bytes.empty?
-            @kept_bytes += bytes.bytesize
+            @chunks << kept unless kept.empty?
+            @kept_bytes += kept.bytesize
             nil
           end
         end
@@ -181,6 +199,13 @@ module Foam
           end
         end
 
+        # Tells the capture that the client library already inflated the observed bytes, and that
+        # wire_bytes is the body's size on the wire. The bytes are then read as text without
+        # decoding, and the size attributes report wire_bytes.
+        def inflated(wire_bytes)
+          @wire_bytes = wire_bytes
+        end
+
         # Ends the capture and emits its span attributes and records. Runs once; later calls do
         # nothing. The records are emitted even when the span has already ended.
         def finalize(complete:)
@@ -215,12 +240,12 @@ module Foam
           text = @body_redactor.redact(text, headers["content-type"]) if text
           truncated = @truncated || cut
           outcome = outcome_of(complete)
+          size = @wire_bytes || @observed_bytes
 
           if @span.recording?
             @span.set_attribute(@attributes[:complete], complete)
             @span.set_attribute(@attributes[:truncated], truncated)
-            @span.set_attribute(@attributes[:size], @observed_bytes)
-            @span.set_attribute(@attributes[:content], text) if text
+            @span.set_attribute(@attributes[:size], size)
           end
           return if outcome == "empty"
 
@@ -232,8 +257,8 @@ module Foam
             COMPLETE => complete,
             TRUNCATED => truncated,
             OUTCOME => outcome,
-            @attributes[:size] => @observed_bytes,
-            CAPTURED_BYTES => text ? text.bytesize : @kept_bytes,
+            @attributes[:size] => size,
+            CAPTURED_BYTES => text ? text.bytesize : 0,
             CHUNK_COUNT => chunks.size
           }.merge(header_attributes(headers))
 
@@ -243,7 +268,7 @@ module Foam
           records.each_with_index do |chunk, index|
             logger.on_emit(
               event_name: Constants::BODY_CHUNK_EVENT,
-              body: Constants::BODY_CHUNK_EVENT_BODY,
+              body: chunk,
               severity_number: Logs::SEVERITIES[:info],
               severity_text: "INFO",
               attributes: chunk ? common.merge(chunk_attributes(chunk, index)) : common,
@@ -256,7 +281,8 @@ module Foam
         # body cannot be read as text: an unknown or corrupt encoding, one inflating past the
         # bound, or a binary content type. Only the metadata is exported then, never the bytes.
         def decoded_text(headers)
-          decoded = BodyDecoder.decode(@chunks.join, headers["content-encoding"])
+          kept = (@chunks.size == 1) ? @chunks.first : @chunks.join
+          decoded = @wire_bytes ? kept : BodyDecoder.decode(kept, headers["content-encoding"])
           return [nil, false] unless decoded
 
           BodyDecoder.text(decoded, ContentType.parse(headers["content-type"])) || [nil, false]
@@ -283,7 +309,7 @@ module Foam
         end
 
         def chunk_attributes(chunk, index)
-          {CHUNK_INDEX => index, CHUNK_BYTES => chunk.bytesize, CHUNK_CONTENT => chunk}
+          {CHUNK_INDEX => index, CHUNK_BYTES => chunk.bytesize}
         end
       end
     end

data/lib/foam/otel/internal/body_decoder.rb

@@ -11,6 +11,14 @@ module Foam
       module BodyDecoder
         INFLATED_ENCODINGS = %w[gzip x-gzip deflate].freeze
 
+        # How much of a body of unknown type is looked at to tell whether it is text.
+        SNIFF_BYTES = 1024
+        # Control characters other than tab, line feed, form feed, and carriage return.
+        CONTROL = /[\x00-\x08\x0B\x0E-\x1F\x7F]/
+        # The start of an archive, image, executable, or PDF, long enough that no text starts that
+        # way. A Windows executable is left to the NUL check, since its two-letter MZ is ordinary text.
+        FILE_SIGNATURE = /\A(?:PK\x03\x04|\x1F\x8B|%PDF-|\x89PNG|\xFF\xD8\xFF|GIF8[79]a|Rar!\x1A\x07|7z\xBC\xAF|BZh[1-9]1AY&SY|\x7FELF)/n
+
         module_function
 
         # Undoes the content encoding and returns the decoded bytes. The header lists encodings in
@@ -26,21 +34,39 @@ module Foam
 
         # Returns the body as UTF-8 text and whether it was cut at the capture limit. Invalid
         # bytes are replaced and the cut lands on a character boundary. Returns nil when the
-        # content type is not text.
+        # content type is binary, or unknown or missing and the bytes do not read as text, or the
+        # bytes are a file whatever the type says. A multipart body's parts are judged on their own.
         def text(bytes, content_type)
-          return unless content_type.text?
+          readable = content_type.text? || (!content_type.binary? && textual?(bytes))
+          readable &&= content_type.multipart? || !file?(bytes)
+          return unless readable
 
           cut = bytes.bytesize > Constants::BODY_CAPTURE_BYTES
-          text = bytes.byteslice(0, Constants::BODY_CAPTURE_BYTES)
+          text = cut ? bytes.byteslice(0, Constants::BODY_CAPTURE_BYTES) : bytes.dup
           text.force_encoding(encoding_for(content_type))
           text = if text.encoding == Encoding::UTF_8
-            whole_utf8(text).scrub
+            whole = whole_utf8(text)
+            whole.valid_encoding? ? whole : whole.scrub
           else
             text.encode(Encoding::UTF_8, invalid: :replace, undef: :replace)
           end
           [text, cut]
         end
 
+        # Whether the first KiB holds a NUL or starts with a file signature.
+        def file?(bytes)
+          start = bytes.byteslice(0, SNIFF_BYTES).b
+          start.include?("\x00") || FILE_SIGNATURE.match?(start)
+        end
+
+        # Whether the start of the bytes is UTF-8 without control characters, allowing a
+        # character cut off where the look ends before the bytes do.
+        def textual?(bytes)
+          start = bytes.byteslice(0, SNIFF_BYTES).force_encoding(Encoding::UTF_8)
+          start = whole_utf8(start) if bytes.bytesize > SNIFF_BYTES
+          start.valid_encoding? && !CONTROL.match?(start)
+        end
+
         # Splits the text into pieces of at most the chunk size that end on character boundaries.
         def chunks(text)
           pieces = []