foam-otel 3.0.0.alpha.3 → 3.0.0.alpha.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ec08cbf830c20cd83d38b22e77be69cd7ccabb17cc11742a09ce2c1a13c8f668
-  data.tar.gz: ece6b5b1553b38efc87a6b9d73219230a54a7ebd29d32d74dd05cd202882ab56
+  metadata.gz: 34ec502fdc38ac38099bba65f1fc37c91f1edb79476bff598c6a0dda7245fcf0
+  data.tar.gz: 9596d8c1d746b66103b87587f5c52006e024d0498d93816912ed916014e563e4
 SHA512:
-  metadata.gz: 350906e9ca2b1542cc1c3f7fe3ca8f3f4c15cfc190e7c50c2c793454dbd512b16a0c54d3b30c7ef78f5c3698ee53dae61d516d9f73b97564cfb6552913ba2c53
-  data.tar.gz: 58d4f62efe472391622739bb85bb86fc554fa92d4d3942f9317bedfaf3542d8697bf80ba86a55bb3acdce62cbbdc36a722c7a47378256ba959b0c4f690ac22ac
+  metadata.gz: 85207e2233f8a986bb874f68215960f20216e010978e5d1355808ec2ed6bfb44f19c9b8f8947cc3164cca65864326dab8b5ef95fb8ef6c37a1150fe06923ae2a
+  data.tar.gz: 3a3bc463b06b32e1d1bd4b6c439e5f8a92b031859301f8681e78a32e4e1db655ef73d4439a0755e336abc9fc1b068bca48049172046732da4adb514eb12c00cb

data/README.md

@@ -29,7 +29,20 @@ Store the ingest token as `FOAM_OTEL_TOKEN` and pass `ENV["FOAM_OTEL_TOKEN"]` to
 
 ### Rails
 
-TODO(jhartquist): Rails setup. The Railtie that adds Foam's Rack middleware is not written, so a Rails app is not covered by header and body capture on server spans until it is.
+Create `config/initializers/foam.rb`:
+
+```ruby
+Foam.init(
+  name: "checkout-api",
+  environment: Rails.env,
+  enabled: true,
+  token: ENV["FOAM_OTEL_TOKEN"]
+)
+```
+
+During boot, `init` adds Foam's Rack middleware to the app directly after the OpenTelemetry Rack middleware, so server spans get the same header and body capture as under plain Rack. `init` must run before the app initializes: an initializer file or `config/application.rb`. An `init` from `after_initialize` or later warns on stderr and adds no middleware.
+
+Nothing else changes; `config.ru` stays as generated. A `RUBYOPT` preload runs before Rails loads, so it installs no Rails instrumentation and adds no middleware.
 
 ### Plain Ruby and Rack
 
@@ -89,6 +102,8 @@ Every method lives on the `Foam` module. Other than the ingest factories, none o
 
 Sets up tracing, metrics, logs, automatic instrumentation, W3C `tracecontext` and `baggage` propagation, and OTLP export to Foam.
 
+`init` claims the four OpenTelemetry global slots only when they are free: the tracer provider, meter provider, logger provider, and propagator. A slot another SDK set before `init` stays with that SDK. Foam warns once on stderr, reports the signal in `state`, and builds no pipeline for it, so any `additional_*` components for that signal are unused. Foam always builds its own logger provider, so `log` delivers either way. `shutdown` releases the slots Foam claimed.
+
 Call this once. Use it when Foam should run OpenTelemetry for the process.
 
 ```ruby
@@ -133,7 +148,7 @@ Foam.init(
 - `version` - the service version, sent as `service.version` when given.
 - `sample_rate` - the fraction of root traces kept, from `0.0` to `1.0`; child spans follow their parent. Leave it at `1.0` unless volume forces sampling. `OTEL_TRACES_SAMPLER` is not consulted.
 - `additional_span_processors`, `additional_log_record_processors`, `additional_metric_readers` - the app's own OpenTelemetry components, added to Foam's providers after Foam's own exporters.
-- `additional_resource_attributes` - attributes merged into the resource of every signal. `service.name`, `service.version`, and `deployment.environment.name` come from the options above; a value for one of them here is reported on stderr and ignored.
+- `additional_resource_attributes` - attributes merged into the resource of every signal. `service.name`, `service.version`, and `deployment.environment.name` come from the options above and `service.instance.id` from Foam; a value for one of them here is reported on stderr and ignored.
 - `additional_instrumentations` - instrumentations to install next to the bundle. An entry is an instrumentation's registry name, which is its class name without the trailing `::Instrumentation` (`"OpenTelemetry::Instrumentation::Logger"`), or a `[name, config]` pair to install it with options (`["OpenTelemetry::Instrumentation::PG", {db_statement: :include}]`). Require the instrumentation's gem before `init`; an entry whose instrumentation is not loaded fails `init`. An entry naming a bundled instrumentation replaces Foam's configuration for it.
 - `ignored_outbound_hosts` - skips tracing of outbound HTTP requests to these hostnames. The endpoint host is always skipped. Use it for hosts whose traffic would drown the traces, such as a sidecar health check on `localhost` or another vendor's OTLP ingest host. Hosts are compared as written; `localhost` and `127.0.0.1` are different hosts. A request to a skipped host still carries a `traceparent` marked not sampled, so an OpenTelemetry service behind it records nothing for that request.
 - `disable_log_sending` - `true` stops shipping the app's `Logger` lines to Foam. `Foam.log` is unaffected. See Loggers.
@@ -143,11 +158,19 @@ Foam.init(
 
 TODO(jhartquist): `propagation_targets`, an allowlist of the hosts that receive `traceparent`, `tracestate`, and `baggage`. Spans stay on for every other host; only the headers are withheld. Today `ignored_outbound_hosts` is the only lever and it drops the spans too. The client patches see the host and can strip the headers after the instrumentation injects them. The endpoint host stays fully ignored.
 
-Every signal carries a resource with the service identity from the options above, `telemetry.distro.name` and `telemetry.distro.version` naming this gem, `host.name`, `host.arch`, `host.id` when it can be read, `os.type`, and `os.version`, and the process and SDK attributes the OpenTelemetry SDK adds. `OTEL_RESOURCE_ATTRIBUTES` can add further attributes. It cannot override any of the attributes above, and `OTEL_SERVICE_NAME` cannot override `name`.
+Every signal carries a resource with these attributes:
+
+- The service identity from the options above: `service.name`, `deployment.environment.name`, and `service.version` when given.
+- `service.instance.id`, a UUID made for the process at `init`.
+- `telemetry.distro.name` and `telemetry.distro.version`, naming this gem.
+- `host.name`, `host.arch`, `host.id` when it can be read, `os.type`, and `os.version`.
+- The process and SDK attributes the OpenTelemetry SDK adds.
+
+A process forked after `init` carries its own `service.instance.id` and `process.pid`. `OTEL_RESOURCE_ATTRIBUTES` can add further attributes. It cannot override any of the attributes above, and `OTEL_SERVICE_NAME` cannot override `name`.
 
 `init` never raises. A failure or an option holding an invalid value is reported on stderr with a `[foam-otel]` prefix and leaves `state` uninitialized.
 
-`init` also reports its outcome to Foam: one log record with the message and the `state`, sent straight to the endpoint with the token. It is sent in the background, never blocks the app, and is dropped when it cannot be delivered. Nothing is sent without a token.
+`init` also reports its outcome to Foam: one log record with the message and the `state`, sent straight to the endpoint with the token. The report is sent in the background and never blocks the app. When the process exits before the report has been delivered, exit waits up to two seconds for the send to finish; a report that cannot be delivered is dropped. Nothing is sent without a token.
 
 ### `set_endpoint(url)`
 
@@ -229,7 +252,7 @@ OpenTelemetry::Context.with_current(context) do
 end
 ```
 
-Foam's helpers always speak W3C, whichever propagator the SDK configured globally (see `OTEL_PROPAGATORS`).
+`inject_trace_context` and `extract_trace_context` always use the W3C `traceparent`, `tracestate`, and `baggage` headers, whatever propagator holds the global slot. Foam does not read `OTEL_PROPAGATORS`. An app that needs a different propagator registers it before `init`, and Foam leaves it in place.
 
 ### `set_baggage(key, value)`
 
@@ -287,9 +310,9 @@ rescue PaymentError => e
 end
 ```
 
-The span gets an `exception` event with the type, message, and stack trace, plus any `attributes`, and its status becomes error with the exception's message. The active span is used whichever SDK started it. Without an active span nothing is recorded.
+The span gets an `exception` event with the type, message, and stack trace, plus any `attributes`, and its status becomes error with the exception's message. A value that is not an `Exception` is recorded by its string form as the message, with no type or stack trace. The active span is used whichever SDK started it. Without an active span nothing is recorded.
 
-### `flush(timeout: 30)`
+### `flush(timeout: 10)`
 
 Exports everything Foam's providers still hold. Call it before a point where the process may be killed without a normal exit, for example at the end of a job on a runner that stops the process.
 
@@ -299,14 +322,16 @@ Foam.flush(timeout: 5)
 
 `timeout` is in seconds and bounds the whole call. A provider that runs out of time is reported on stderr.
 
-### `shutdown(timeout: 30)`
+### `shutdown(timeout: 10)`
 
-Flushes and shuts down Foam's providers, including the one behind `log` when an ingest component created it. Both `init` and the ingest log factory register it with `at_exit`, so most apps never call it. Call it yourself before a process is killed without a normal exit.
+Flushes and shuts down Foam's providers, including the one behind `log` when an ingest component created it, and releases the global slots Foam claimed. Both `init` and the ingest log factory register it with `at_exit`, so most apps never call it. An app calls it before a process is killed without a normal exit.
 
 `timeout` is in seconds and bounds the whole call. A provider that runs out of time is reported on stderr.
 
 `at_exit` covers a normal exit, an unhandled exception, and a signal the process handles by exiting. `exit!`, `KILL`, and a crash lose the last batch, and so does a batch that outlasts the timeout. An exception outside any span is invisible unless the app logs it.
 
+Foam's span processor holds 4096 finished spans and drops the oldest when full; `OTEL_BSP_MAX_QUEUE_SIZE` overrides the size. The ingest span processor and the log record processors keep the OpenTelemetry default of 2048.
+
 ### `state`
 
 Returns whether Foam initialized, which instrumentations registered, which signals have an export path, and the identity `init` was given.
@@ -342,7 +367,7 @@ Each `signals` value names the source of the Foam export path for that signal:
 - `:local` - logs only: another SDK owns the global provider and Foam keeps its own so `log` still delivers.
 - `:none` - no Foam export path.
 
-`:global` is checked against the live OpenTelemetry global at each call. When another SDK takes a slot after `init`, that signal reports `:none` (`:local` for logs) and a warning is written to stderr once. Only the slot is affected: the Foam helpers keep their own references and keep working. `baggage` is `:global` or `:none`. `profile` is not supported and always reports `:none`.
+A slot another SDK held before `init` stays with it and reports `:none` (`:local` for logs) from the start. Foam builds no provider for that signal, so the metric helpers and `set_baggage` do nothing while `log` still delivers. `:global` is checked against the live OpenTelemetry global on each call, so a slot another SDK takes after `init` reports the same way; in that case only the slot changes hands, and the helpers keep using Foam's own providers. Both cases are warned about once on stderr. Body capture is off while another SDK holds the logs slot. `baggage` is `:global` or `:none`. `profile` is not supported and always reports `:none`.
 
 ## Custom spans and other OpenTelemetry APIs
 
@@ -360,7 +385,7 @@ There is no option to record other headers.
 
 ### Header capture
 
-Server spans get the headers from `Foam::Rack::Middleware`, placed after the OpenTelemetry Rack middleware. A Rails app is not covered yet.
+Server spans get the headers from `Foam::Rack::Middleware`, placed after the OpenTelemetry Rack middleware (added by `init` in Rails apps).
 
 Client spans get the headers from Foam's hooks on `Net::HTTP`, Excon, HTTPClient, and Faraday, applied unless `network_capture` is `:off`. The `Net::HTTP` hook is always installed; the others install only when their gem is loaded before `init`, and Faraday's covers connections built after `init`. Faraday is covered on the Faraday span whatever the adapter. `http`, `httpx`, `ethon`, and `restclient` are not covered.
 
@@ -380,7 +405,7 @@ In `:advanced` mode each captured body arrives as `foam.http.body.chunk` log rec
 
 ### Rack bodies
 
-Server bodies need `Foam::Rack::Middleware` after the OpenTelemetry Rack middleware, as shown under "Plain Ruby and Rack". A Rails app is not covered yet.
+Server bodies need `Foam::Rack::Middleware` after the OpenTelemetry Rack middleware, as shown under "Plain Ruby and Rack" (added by `init` in Rails apps).
 
 - The request body is captured whether or not the app reads it when the input is rewindable, as it is under Puma, which buffers request bodies. Otherwise what the app reads is captured.
 - A streamed response is observed as the server sends it, without buffering or delaying it. The records are emitted when the body closes; a stream idle for over 60 seconds is closed as incomplete and the rest is dropped.
@@ -390,13 +415,14 @@ Server bodies need `Foam::Rack::Middleware` after the OpenTelemetry Rack middlew
 
 Client bodies are captured in `:advanced` by the same hooks that set the headers.
 
-- `Net::HTTP`: a request sent through `body_stream` and a response read in a block or into an IO are not captured.
+- `Net::HTTP`: a request is captured whether sent as a string or through `body_stream`. A response read in a block or into an IO is not captured.
 - Faraday: bodies are captured on the Faraday span through Foam's Faraday middleware, whatever the adapter. A request whose body is an IO and a response streamed with `on_data` are not captured.
 - Excon: a request sent with `request_block` and a response received with `response_block` are not captured.
 - HTTPClient: a request whose body is an IO or multipart parts is not captured. A response read in a block is captured.
-- RestClient, `http`, `httpx`, and `ethon`: no bodies, since they get no header capture either.
+- RestClient and HTTParty: captured on the `Net::HTTP` span beneath them.
+- `http`, `httpx`, and `ethon`: no bodies, since they get no header capture either.
 
-TODO(jhartquist): capture streamed bodies: a request sent from an IO and a response the client streams to the app.
+TODO(jhartquist): capture the streamed bodies listed above. For `Net::HTTP`, prepend on `Net::HTTPResponse#read_body` and tee the block or IO the app passes.
 
 ## Loggers
 
@@ -414,6 +440,12 @@ If another SDK owns logs, set `disable_log_sending: true` so records do not arri
 
 Secrets in a log line are masked before export. See Redaction.
 
+## AWS Lambda
+
+The invocation span carries `faas.trigger` for API Gateway, ALB, SNS, SQS, S3, DynamoDB streams, and EventBridge schedule events.
+
+Traces, metrics, and logs are flushed before the handler returns (including failing invocations).
+
 ## Pre-forking servers and threads
 
 TODO(jhartquist): pre-forking servers (Puma in cluster mode, Unicorn, Pitchfork, Resque): whether to call `init` before or after the fork, and what the SDK's batch processors and the metric reader do in the child. Foam registers no fork hook today.
@@ -431,7 +463,13 @@ end
 
 Foam masks the values of sensitive keys in everything it exports: span, span event, and span link attributes; log bodies and attributes; metric data point attributes; and resource attributes. A built-in list of credential and personal-data keys always applies, matched by exact name or by a sensitive segment of the key (`legacy_api_key_2`, `stripeToken`). Keys match after normalization: case, `-`, `.`, and camelCase boundaries are folded, so `apiKey`, `api-key`, and `API_KEY` are one key. Header attributes match on the header name. SQL statements arrive from the database instrumentations with every literal replaced with `?` (see Instrumentation) and are redacted like any other string. Captured HTTP bodies are masked by content type with the same keys, with their framing kept. When masking a body fails, the whole body is replaced with `********`.
 
-Every other string is scanned as text, a log body included. A string that is a whole JSON document is masked by key at every depth, so a list or object under a sensitive key becomes one mask. Any other string has the value after each sensitive key masked, whether the key is quoted as JSON writes it or bare before `=`, `:`, or `=>`, and whether it sits in a query string, a header line, or prose. The value ends at the next delimiter, so the rest of the line is kept, a quoted URL's closing quote included. Such a string is scanned up to 1 MiB; the rest collapses into one mask. `url.query` is masked pair by pair, and names such as `code` and `sig` count as sensitive only inside a query string. Masking is value-only: the key and the surrounding text stay.
+Every other string is scanned as text, log bodies included.
+
+- A string that is a whole JSON document is masked by key at every depth. A list or object under a sensitive key becomes one mask.
+- Any other string has the value after each sensitive key masked. The key may be quoted as JSON writes it or bare before `=`, `:`, or `=>`, in a query string, a header line, or prose. The value ends at the next delimiter, and a quoted URL keeps its closing quote.
+- `url.query` is masked pair by pair. Names such as `code` and `sig` are sensitive only inside a query string.
+
+Scanning stops after 1 MiB, and everything past that point is replaced with a single mask. Masking is value-only: the key and the surrounding text remain intact.
 
 The `redact` option adds the app's own keys, matched exactly after the same normalization. A value under a built-in or `pii` key becomes `[REDACTED]`. A value under a `secrets` key becomes `********`, followed by its last four characters when it is 12 characters or longer. That tail is kept only where the value is read whole under its key: an attribute, a hash or array, or a whole JSON document. A value found by scanning text or a `url.query` pair becomes `[REDACTED]` whatever its key.
 
@@ -528,7 +566,7 @@ The `bunny`, `rdkafka`, `ruby_kafka`, `racecar`, `sidekiq`, `resque`, `que`, `de
 OpenTelemetry packages outside the instrumentation bundle that Foam does not wire yet.
 
 - Resource detectors. Foam's resource carries the service identity, the distro, host, and OS attributes, and the process and SDK attributes the OpenTelemetry SDK adds; `OTEL_RESOURCE_ATTRIBUTES` adds to it. The contrib detectors, `opentelemetry-resource-detector-aws`, `-azure`, `-container`, and `-google_cloud_platform`, are not run. The cloud ones probe metadata endpoints, so they belong off the `init` critical path.
-- Propagators. `init` keeps the SDK's default propagators, W3C `tracecontext` and `baggage`. The SDK honors `OTEL_PROPAGATORS` for `b3`, `b3multi`, `jaeger`, `xray`, and `ottrace` when the propagator gem is present, but the gems are not dependencies of `foam-otel`, and `inject_trace_context` and `extract_trace_context` speak W3C whatever is configured.
+- Propagators. `init` registers W3C `tracecontext` and `baggage` when the propagator slot is free and does not read `OTEL_PROPAGATORS`. The contrib propagator gems (`b3`, `b3multi`, `jaeger`, `xray`, `ottrace`) are not dependencies of `foam-otel`; an app that needs one registers it before `init`. `inject_trace_context` and `extract_trace_context` always use the W3C headers.
 - Samplers. `sample_rate` builds a parent-based ratio sampler and `OTEL_TRACES_SAMPLER` is not consulted. `opentelemetry-sampler-xray` is the only contrib sampler; there is no remote sampler for Jaeger.
 
 ## TODO(jhartquist): Safely

data/lib/foam/otel/internal/aws_lambda.rb

@@ -0,0 +1,69 @@
+# frozen_string_literal: true
+
+module Foam
+  module Otel
+    module Internal
+      # Foam's additions to the aws_lambda instrumentation, prepended on its Wrap module by init
+      # after installation:
+      # 1. Fills in faas.trigger from the event's shape for the AWS event sources the
+      #    instrumentation does not derive it for.
+      # 2. Flushes Foam's logger provider where the wrapper flushes the tracer and meter providers,
+      #    so an invocation's log records leave before the handler returns.
+      module AwsLambda
+        INSTRUMENTATION = "OpenTelemetry::Instrumentation::AwsLambda"
+        TRIGGER = "faas.trigger"
+
+        # A record's event source -> the semantic conventions' trigger for it.
+        RECORD_TRIGGERS = {
+          "aws:sns" => "pubsub",
+          "aws:sqs" => "pubsub",
+          "aws:s3" => "datasource",
+          "aws:dynamodb" => "datasource"
+        }.freeze
+
+        def self.install
+          return unless Instrumentations.instance(INSTRUMENTATION).installed?
+
+          ::OpenTelemetry::Instrumentation::AwsLambda::Wrap.prepend(Wrap)
+        end
+
+        # The trigger the event's shape names, or nil for a shape this does not know. SNS spells
+        # its record key EventSource; the other record sources spell it eventSource.
+        def self.trigger(event)
+          record = event["Records"]&.first
+          if record
+            RECORD_TRIGGERS[record["eventSource"] || record["EventSource"]]
+          elsif event["source"] == "aws.events" && event["detail-type"] == "Scheduled Event"
+            "timer"
+          end
+        end
+
+        module Wrap
+          def wrap_lambda(
+            event:, context:, handler:,
+            flush_timeout: ::OpenTelemetry::Instrumentation::AwsLambda::Wrap::DEFAULT_FLUSH_TIMEOUT
+          )
+            super
+          ensure
+            # Runs for an error too, so the records of a failed invocation leave before the error
+            # reaches the runtime. The timeout is the one the wrapper gives the other providers.
+            Safely.call("aws_lambda log flush") do
+              State.logger_provider&.force_flush(timeout: flush_timeout)
+            end
+          end
+
+          private
+
+          def otel_attributes(event, context)
+            attributes = super
+            return attributes if attributes.key?(TRIGGER)
+
+            trigger = Safely.call("faas.trigger") { AwsLambda.trigger(event) }
+            attributes[TRIGGER] = trigger if trigger
+            attributes
+          end
+        end
+      end
+    end
+  end
+end

data/lib/foam/otel/internal/clients/net_http.rb

@@ -6,10 +6,9 @@ module Foam
       module Clients
         # Sets the safe headers on Net::HTTP client spans and, in advanced network capture,
         # captures their request and response bodies. Prepended on Net::HTTP by init unless network
-        # capture is off. Captures a body only when Net::HTTP holds it as a String, so these are
-        # not captured:
-        # - a request sent through body_stream
-        # - a response read in a block or into an IO
+        # capture is off. A request body is captured whether set as a String or streamed through
+        # body_stream. A response body is captured only when Net::HTTP reads it into a String; a
+        # response the app reads itself, in a block or into an IO, is not.
         module NetHttp
           private
 
@@ -35,6 +34,7 @@ module Foam
                 yield response if block_given?
               end
             ensure
+              req.body_stream = req.body_stream.stream if req.body_stream.is_a?(ReadTee)
               request&.finalize(complete: !res.nil?)
               Safely.call("request header capture") do
                 BodyCapture.set_header_attributes(span, :request, body_headers(req))
@@ -44,15 +44,24 @@ module Foam
             res
           end
 
-          # Only a String body is captured: a stream or multipart data goes straight to the socket.
+          # A String body is observed in one piece. A body_stream is wrapped in a ReadTee, which
+          # observes each chunk as Net::HTTP copies it to the socket; transport_request unwraps it
+          # afterwards. Multipart form data is not captured.
           def capture_request(req, span, body_redactor)
             body = req.body
-            return unless body.is_a?(String)
+            stream = req.body_stream
+            return unless body.is_a?(String) || stream.respond_to?(:read)
 
             capture = BodyCapture.start(span, body_redactor, side: :client, direction: :request) do
               body_headers(req)
             end
-            capture&.observe(body)
+            return unless capture
+
+            if body
+              capture.observe(body)
+            else
+              req.body_stream = ReadTee.new(stream, capture)
+            end
             capture
           end
 

data/lib/foam/otel/internal/clients/read_tee.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+module Foam
+  module Otel
+    module Internal
+      module Clients
+        # Wraps a request body_stream so the bytes Net::HTTP sends can be captured without buffering
+        # them. Net::HTTP copies the stream to the socket with IO.copy_stream, so the read is the
+        # only place to see them.
+        class ReadTee
+          # The wrapped stream, for putting it back on the request once it is sent.
+          attr_reader :stream
+
+          def initialize(stream, capture)
+            @stream = stream
+            @capture = capture
+          end
+
+          def read(*args)
+            bytes = @stream.read(*args)
+            @capture.observe(bytes) if bytes
+            bytes
+          end
+        end
+      end
+    end
+  end
+end

data/lib/foam/otel/internal/constants.rb

@@ -52,15 +52,24 @@ module Foam
         # The log body of every body chunk record; the chunk's text goes in its attributes.
         BODY_CHUNK_EVENT_BODY = "HTTP body chunk"
 
-        # The mask for a value. A secret key's value read whole gets FULL_MASK and a tail instead.
+        # Replaces a redacted value.
         REDACTED_VALUE = "[REDACTED]"
 
-        # The mask for a value under a secret key that has no tail to keep, and for one redaction failed on.
+        # Replaces a secret: all of it when short or not text, otherwise all but its last four
+        # characters. Also stands in for a value whose redaction failed.
         FULL_MASK = "********"
 
-        # The length from which a value under a secret key keeps its last four characters.
+        # Secrets at least this long keep their last four characters unmasked.
         TAIL_MASK_THRESHOLD = 12
 
+        # Finished spans Foam's span processor holds before dropping the oldest. The SDK default
+        # of 2048 loses spans when a loop finishes more than that before the export thread runs.
+        SPAN_QUEUE_SIZE = 4096
+
+        # Seconds flush and shutdown wait by default. It is the OTLP exporters' own request timeout,
+        # and it keeps the at_exit shutdown inside a 30 second deploy grace period.
+        LIFECYCLE_TIMEOUT = 10
+
         module_function
 
         def bearer_headers(token)

data/lib/foam/otel/internal/fork_hooks.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+module Foam
+  module Otel
+    module Internal
+      # Runs State.after_fork in every child this process forks, by wrapping Process._fork, the
+      # method fork and Process.fork call. Preforking servers init Foam in the master, and a
+      # forked child gets the parent's memory but none of its threads, so the periodic metric
+      # reader's export thread is gone and the child's metrics would never leave it, and the
+      # providers' resource still names the parent process.
+      module ForkHooks
+        def _fork
+          parent = Process.pid
+          super.tap { State.after_fork unless Process.pid == parent }
+        end
+
+        def self.attach
+          Process.singleton_class.prepend(self)
+        end
+      end
+    end
+  end
+end

data/lib/foam/otel/internal/init.rb

@@ -18,6 +18,10 @@ module Foam
           # Every outcome is reported with the same identity, including the ones that stop early.
           identity = options.slice(:name, :environment, :token)
 
+          # Recorded before the checks, so state names the app after a disabled or failed init too.
+          # The token stays out of state on purpose; it is read by health checks and tests.
+          State.record_params(options.slice(:name, :environment))
+
           # Teams may turn the SDK off in specific environments.
           unless options[:enabled]
             Report.call(severity: :info, message: "SDK disabled", **identity)
@@ -63,27 +67,39 @@ module Foam
           State.take_provider(:logger)&.shutdown(timeout: nil)
 
           redactor = Redactor.from_option(options[:redact])
-          configure(instrumentation_config: instrumentation_config, redactor: redactor, **options)
-
-          # The client patches set the safe headers on every client span and, in advanced mode,
-          # capture bodies too. Bodies are redacted by the same rules as attributes, so only
-          # advanced builds a body redactor; its presence is what switches body capture on.
-          body_redactor = BodyRedactor.new(redactor) if options[:network_capture] == :advanced
-          install_client_patches unless options[:network_capture] == :off
-
-          # Recorded last so a failed init leaves the state uninitialized.
-          # The token stays out of state on purpose; it is read by health checks and tests.
-          State.record_init(
-            params: options.slice(:name, :environment),
-            tracer_provider: OpenTelemetry.tracer_provider,
-            meter_provider: OpenTelemetry.meter_provider,
-            logger_provider: OpenTelemetry.logger_provider,
-            propagator: OpenTelemetry.propagation,
-            instrumentations: Instrumentations.installed(instrumentation_config),
-            instrumentation_config: instrumentation_config,
-            body_redactor: body_redactor,
-            network_capture: options[:network_capture]
+          providers, claimed_slots = configure(
+            instrumentation_config: instrumentation_config, redactor: redactor, **options
           )
+          begin
+            ForkHooks.attach
+
+            # The client patches set the safe headers on every client span and, in advanced mode,
+            # capture bodies too. Bodies are redacted by the same rules as attributes, so only
+            # advanced builds a body redactor; its presence is what switches body capture on.
+            body_redactor = BodyRedactor.new(redactor) if options[:network_capture] == :advanced
+            install_client_patches unless options[:network_capture] == :off
+
+            # Adds faas.trigger for the event sources the aws_lambda instrumentation skips, and flushes
+            # Foam's logger provider where it flushes the tracer and meter providers.
+            Safely.call("aws_lambda") { AwsLambda.install }
+
+            # A Rails app cannot add the middleware in config.ru the way a plain Rack app does, so
+            # Foam adds it through Rails itself.
+            Safely.call("rails middleware") { RailsMiddleware.install } unless options[:network_capture] == :off
+
+            # Recorded last so a failed init leaves the state uninitialized.
+            State.record_init(
+              providers: providers,
+              claimed_slots: claimed_slots,
+              instrumentations: Instrumentations.installed(instrumentation_config),
+              instrumentation_config: instrumentation_config,
+              body_redactor: body_redactor,
+              network_capture: options[:network_capture]
+            )
+          rescue
+            roll_back(providers, claimed_slots)
+            raise
+          end
 
           # Flush whatever the batch processors still hold before the process exits.
           at_exit { Foam.shutdown }
@@ -94,66 +110,111 @@ module Foam
           nil
         end
 
+        # Builds Foam's pipelines and claims the global slots the API still holds itself. Returns
+        # the providers built, by kind, and the slots claimed, by signal.
         def configure(
           token:, name:, environment:, sample_rate:, instrumentation_config:, additional_span_processors:,
           additional_metric_readers:, additional_log_record_processors:, additional_resource_attributes:,
           ignored_outbound_hosts:, redactor:, before_send:, version: nil, **
         )
-          # Declared up front so the rescue below can stop whichever of these exist when something fails.
-          span_processor = nil
-          metric_reader = nil
-          log_record_processor = nil
+          # Declared up front so the rescue below can stop whichever providers exist when
+          # something fails.
+          providers = {}
 
-          sampler = IgnoredHosts.new(
-            ignored_hosts(ignored_outbound_hosts),
-            OpenTelemetry::SDK::Trace::Samplers.parent_based(
-              root: OpenTelemetry::SDK::Trace::Samplers.trace_id_ratio_based(sample_rate)
-            )
+          # The SDK's own switch, honored the way SDK.configure honors it: nothing is installed.
+          raise "OTEL_SDK_DISABLED is true" if ENV["OTEL_SDK_DISABLED"] == "true"
+
+          sampler = sampler(sample_rate, ignored_outbound_hosts)
+          resource = OpenTelemetry::SDK::Resources::Resource.default.merge(
+            Resource.create(name, environment, version, additional_resource_attributes)
           )
-          resource = Resource.create(name, environment, version, additional_resource_attributes)
           before_send = BeforeSend.new(before_send)
 
-          trace_exporter = Exporters::TraceExporter.new(
+          # A slot another SDK owns is left in place, and Foam builds no pipeline for that signal,
+          # since nothing would feed one. The logger provider is built either way, so log still
+          # delivers whoever holds the slot.
+          free = Slots::SIGNALS.select { |signal| Slots.free?(signal) }
+          (Slots::SIGNALS - free).each do |signal|
+            Diagnostics.warn("#{signal} global slot taken by #{Slots.live(signal).class}")
+          end
+          if free.include?(:traces)
+            exporter = Exporters::TraceExporter.new(
+              token: token, redactor: redactor, before_send: before_send
+            )
+            providers[:tracer] = tracer_provider(
+              resource, sampler, exporter, additional_span_processors
+            )
+          end
+          if free.include?(:metrics)
+            exporter = Exporters::MetricExporter.new(token: token, redactor: redactor)
+            providers[:meter] = meter_provider(resource, exporter, additional_metric_readers)
+          end
+          exporter = Exporters::LogExporter.new(
             token: token, redactor: redactor, before_send: before_send
           )
-          metrics_exporter = Exporters::MetricExporter.new(token: token, redactor: redactor)
-          logs_exporter = Exporters::LogExporter.new(
-            token: token, redactor: redactor, before_send: before_send
+          providers[:logger] = logger_provider(
+            resource, exporter, additional_log_record_processors
           )
 
-          span_processor = OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(trace_exporter)
-          metric_reader = OpenTelemetry::SDK::Metrics::Export::PeriodicMetricReader.new(exporter: metrics_exporter)
-          log_record_processor = OpenTelemetry::SDK::Logs::Export::BatchLogRecordProcessor.new(logs_exporter)
-
-          # SDK.configure swallows its own exceptions; an unchanged slot is how we know it did not finish.
-          previous_tracer_provider = OpenTelemetry.tracer_provider
-
-          # configure installs the global providers and sets up the used instrumentations.
-          OpenTelemetry::SDK.configure do |c|
-            c.resource = resource
-
-            c.add_span_processor(span_processor)
-            additional_span_processors.each { |processor| c.add_span_processor(processor) }
+          claimed_slots = {
+            traces: providers[:tracer],
+            metrics: providers[:meter],
+            logs: (providers[:logger] if free.include?(:logs)),
+            baggage: (Propagation::PROPAGATOR if free.include?(:baggage))
+          }.compact
+          claimed_slots.each { |signal, object| Slots.claim(signal, object) }
+
+          # Installs the bundle and the caller's entries, those whose library is already loaded; the
+          # rest are skipped. After the slots, so each one binds its tracer to whoever holds traces.
+          OpenTelemetry::Instrumentation.registry.install(
+            instrumentation_config.keys, instrumentation_config
+          )
+          [providers, claimed_slots]
+        rescue
+          roll_back(providers, claimed_slots)
+          raise
+        end
 
-            c.add_metric_reader(metric_reader)
-            additional_metric_readers.each { |reader| c.add_metric_reader(reader) }
+        # Frees the slots and stops the providers a failed init built, so the next init starts clean.
+        def roll_back(providers, claimed_slots)
+          claimed_slots&.each { |signal, object| Slots.release(signal, object) }
+          Lifecycle.each(:shutdown, providers.values, timeout: nil)
+        end
 
-            c.add_log_record_processor(log_record_processor)
-            additional_log_record_processors.each { |processor| c.add_log_record_processor(processor) }
+        # Drops spans headed for an ignored host and samples the rest per trace at sample_rate.
+        def sampler(sample_rate, ignored_outbound_hosts)
+          IgnoredHosts.new(
+            ignored_hosts(ignored_outbound_hosts),
+            OpenTelemetry::SDK::Trace::Samplers.parent_based(
+              root: OpenTelemetry::SDK::Trace::Samplers.trace_id_ratio_based(sample_rate)
+            )
+          )
+        end
 
-            # Installs the bundle and the caller's entries, those whose library is already loaded; the rest are skipped.
-            instrumentation_config.each { |name, config| c.use(name, config) }
-          end
+        def tracer_provider(resource, sampler, exporter, additional)
+          provider = OpenTelemetry::SDK::Trace::TracerProvider.new(
+            resource: resource, sampler: sampler
+          )
+          foam = OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(
+            exporter,
+            max_queue_size: Integer(ENV.fetch("OTEL_BSP_MAX_QUEUE_SIZE", Constants::SPAN_QUEUE_SIZE))
+          )
+          [foam, *additional].each { |processor| provider.add_span_processor(processor) }
+          provider
+        end
 
-          if OpenTelemetry.tracer_provider.equal?(previous_tracer_provider)
-            raise "no tracer provider was installed"
-          end
+        def meter_provider(resource, exporter, additional)
+          provider = OpenTelemetry::SDK::Metrics::MeterProvider.new(resource: resource)
+          foam = OpenTelemetry::SDK::Metrics::Export::PeriodicMetricReader.new(exporter: exporter)
+          [foam, *additional].each { |reader| provider.add_metric_reader(reader) }
+          provider
+        end
 
-          # The configurator builds the provider with the SDK's default sampler and takes no other.
-          OpenTelemetry.tracer_provider.sampler = sampler
-        rescue
-          Lifecycle.each(:shutdown, [span_processor, metric_reader, log_record_processor], timeout: nil)
-          raise
+        def logger_provider(resource, exporter, additional)
+          provider = OpenTelemetry::SDK::Logs::LoggerProvider.new(resource: resource)
+          foam = OpenTelemetry::SDK::Logs::Export::BatchLogRecordProcessor.new(exporter)
+          [foam, *additional].each { |processor| provider.add_log_record_processor(processor) }
+          provider
         end
 
         # Net::HTTP is patched without a check, since foam-otel loads it for its own exporters; the