foam-otel 3.0.0.alpha.3 → 3.0.0.alpha.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ec08cbf830c20cd83d38b22e77be69cd7ccabb17cc11742a09ce2c1a13c8f668
-  data.tar.gz: ece6b5b1553b38efc87a6b9d73219230a54a7ebd29d32d74dd05cd202882ab56
+  metadata.gz: 1c7362be48b291f5a1c52aca227430f7ded8a223dfd289108f1bf97b3254b184
+  data.tar.gz: b876b8f462eccad1ce7f39e4ced08da44243755c0e615109a89891ef563078d6
 SHA512:
-  metadata.gz: 350906e9ca2b1542cc1c3f7fe3ca8f3f4c15cfc190e7c50c2c793454dbd512b16a0c54d3b30c7ef78f5c3698ee53dae61d516d9f73b97564cfb6552913ba2c53
-  data.tar.gz: 58d4f62efe472391622739bb85bb86fc554fa92d4d3942f9317bedfaf3542d8697bf80ba86a55bb3acdce62cbbdc36a722c7a47378256ba959b0c4f690ac22ac
+  metadata.gz: 157e5f359ca4d872a7106377d0315c767ff6fc624952408ce85383a8b8c2a4a2e0de62b97c15fc3d8fd261ea09b276b79c0b1b0c4205105a6ab44b45043ea5ab
+  data.tar.gz: c731cca87d8ccf41524ae89da935e8878efb45d3d7ae11c84b7172ddc490ac579da0cd363079fbce6ed8dd24750fa939fa65d30fc2c8611ec8f1b0ce341a2b9d

data/README.md

@@ -89,6 +89,8 @@ Every method lives on the `Foam` module. Other than the ingest factories, none o
 
 Sets up tracing, metrics, logs, automatic instrumentation, W3C `tracecontext` and `baggage` propagation, and OTLP export to Foam.
 
+`init` claims the four OpenTelemetry global slots only when they are free: the tracer provider, meter provider, logger provider, and propagator. A slot another SDK set before `init` stays with that SDK. Foam warns once on stderr, reports the signal in `state`, and builds no pipeline for it, so any `additional_*` components for that signal are unused. Foam always builds its own logger provider, so `log` delivers either way. `shutdown` releases the slots Foam claimed.
+
 Call this once. Use it when Foam should run OpenTelemetry for the process.
 
 ```ruby
@@ -133,7 +135,7 @@ Foam.init(
 - `version` - the service version, sent as `service.version` when given.
 - `sample_rate` - the fraction of root traces kept, from `0.0` to `1.0`; child spans follow their parent. Leave it at `1.0` unless volume forces sampling. `OTEL_TRACES_SAMPLER` is not consulted.
 - `additional_span_processors`, `additional_log_record_processors`, `additional_metric_readers` - the app's own OpenTelemetry components, added to Foam's providers after Foam's own exporters.
-- `additional_resource_attributes` - attributes merged into the resource of every signal. `service.name`, `service.version`, and `deployment.environment.name` come from the options above; a value for one of them here is reported on stderr and ignored.
+- `additional_resource_attributes` - attributes merged into the resource of every signal. `service.name`, `service.version`, and `deployment.environment.name` come from the options above and `service.instance.id` from Foam; a value for one of them here is reported on stderr and ignored.
 - `additional_instrumentations` - instrumentations to install next to the bundle. An entry is an instrumentation's registry name, which is its class name without the trailing `::Instrumentation` (`"OpenTelemetry::Instrumentation::Logger"`), or a `[name, config]` pair to install it with options (`["OpenTelemetry::Instrumentation::PG", {db_statement: :include}]`). Require the instrumentation's gem before `init`; an entry whose instrumentation is not loaded fails `init`. An entry naming a bundled instrumentation replaces Foam's configuration for it.
 - `ignored_outbound_hosts` - skips tracing of outbound HTTP requests to these hostnames. The endpoint host is always skipped. Use it for hosts whose traffic would drown the traces, such as a sidecar health check on `localhost` or another vendor's OTLP ingest host. Hosts are compared as written; `localhost` and `127.0.0.1` are different hosts. A request to a skipped host still carries a `traceparent` marked not sampled, so an OpenTelemetry service behind it records nothing for that request.
 - `disable_log_sending` - `true` stops shipping the app's `Logger` lines to Foam. `Foam.log` is unaffected. See Loggers.
@@ -143,11 +145,19 @@ Foam.init(
 
 TODO(jhartquist): `propagation_targets`, an allowlist of the hosts that receive `traceparent`, `tracestate`, and `baggage`. Spans stay on for every other host; only the headers are withheld. Today `ignored_outbound_hosts` is the only lever and it drops the spans too. The client patches see the host and can strip the headers after the instrumentation injects them. The endpoint host stays fully ignored.
 
-Every signal carries a resource with the service identity from the options above, `telemetry.distro.name` and `telemetry.distro.version` naming this gem, `host.name`, `host.arch`, `host.id` when it can be read, `os.type`, and `os.version`, and the process and SDK attributes the OpenTelemetry SDK adds. `OTEL_RESOURCE_ATTRIBUTES` can add further attributes. It cannot override any of the attributes above, and `OTEL_SERVICE_NAME` cannot override `name`.
+Every signal carries a resource with these attributes:
+
+- The service identity from the options above: `service.name`, `deployment.environment.name`, and `service.version` when given.
+- `service.instance.id`, a UUID made for the process at `init`.
+- `telemetry.distro.name` and `telemetry.distro.version`, naming this gem.
+- `host.name`, `host.arch`, `host.id` when it can be read, `os.type`, and `os.version`.
+- The process and SDK attributes the OpenTelemetry SDK adds.
+
+A process forked after `init` carries its own `service.instance.id` and `process.pid`. `OTEL_RESOURCE_ATTRIBUTES` can add further attributes. It cannot override any of the attributes above, and `OTEL_SERVICE_NAME` cannot override `name`.
 
 `init` never raises. A failure or an option holding an invalid value is reported on stderr with a `[foam-otel]` prefix and leaves `state` uninitialized.
 
-`init` also reports its outcome to Foam: one log record with the message and the `state`, sent straight to the endpoint with the token. It is sent in the background, never blocks the app, and is dropped when it cannot be delivered. Nothing is sent without a token.
+`init` also reports its outcome to Foam: one log record with the message and the `state`, sent straight to the endpoint with the token. The report is sent in the background and never blocks the app. When the process exits before the report has been delivered, exit waits up to two seconds for the send to finish; a report that cannot be delivered is dropped. Nothing is sent without a token.
 
 ### `set_endpoint(url)`
 
@@ -229,7 +239,7 @@ OpenTelemetry::Context.with_current(context) do
 end
 ```
 
-Foam's helpers always speak W3C, whichever propagator the SDK configured globally (see `OTEL_PROPAGATORS`).
+`inject_trace_context` and `extract_trace_context` always use the W3C `traceparent`, `tracestate`, and `baggage` headers, whatever propagator holds the global slot. Foam does not read `OTEL_PROPAGATORS`. An app that needs a different propagator registers it before `init`, and Foam leaves it in place.
 
 ### `set_baggage(key, value)`
 
@@ -287,9 +297,9 @@ rescue PaymentError => e
 end
 ```
 
-The span gets an `exception` event with the type, message, and stack trace, plus any `attributes`, and its status becomes error with the exception's message. The active span is used whichever SDK started it. Without an active span nothing is recorded.
+The span gets an `exception` event with the type, message, and stack trace, plus any `attributes`, and its status becomes error with the exception's message. A value that is not an `Exception` is recorded by its string form as the message, with no type or stack trace. The active span is used whichever SDK started it. Without an active span nothing is recorded.
 
-### `flush(timeout: 30)`
+### `flush(timeout: 10)`
 
 Exports everything Foam's providers still hold. Call it before a point where the process may be killed without a normal exit, for example at the end of a job on a runner that stops the process.
 
@@ -299,14 +309,16 @@ Foam.flush(timeout: 5)
 
 `timeout` is in seconds and bounds the whole call. A provider that runs out of time is reported on stderr.
 
-### `shutdown(timeout: 30)`
+### `shutdown(timeout: 10)`
 
-Flushes and shuts down Foam's providers, including the one behind `log` when an ingest component created it. Both `init` and the ingest log factory register it with `at_exit`, so most apps never call it. Call it yourself before a process is killed without a normal exit.
+Flushes and shuts down Foam's providers, including the one behind `log` when an ingest component created it, and releases the global slots Foam claimed. Both `init` and the ingest log factory register it with `at_exit`, so most apps never call it. An app calls it before a process is killed without a normal exit.
 
 `timeout` is in seconds and bounds the whole call. A provider that runs out of time is reported on stderr.
 
 `at_exit` covers a normal exit, an unhandled exception, and a signal the process handles by exiting. `exit!`, `KILL`, and a crash lose the last batch, and so does a batch that outlasts the timeout. An exception outside any span is invisible unless the app logs it.
 
+Foam's span processor holds 4096 finished spans and drops the oldest when full; `OTEL_BSP_MAX_QUEUE_SIZE` overrides the size. The ingest span processor and the log record processors keep the OpenTelemetry default of 2048.
+
 ### `state`
 
 Returns whether Foam initialized, which instrumentations registered, which signals have an export path, and the identity `init` was given.
@@ -342,7 +354,7 @@ Each `signals` value names the source of the Foam export path for that signal:
 - `:local` - logs only: another SDK owns the global provider and Foam keeps its own so `log` still delivers.
 - `:none` - no Foam export path.
 
-`:global` is checked against the live OpenTelemetry global at each call. When another SDK takes a slot after `init`, that signal reports `:none` (`:local` for logs) and a warning is written to stderr once. Only the slot is affected: the Foam helpers keep their own references and keep working. `baggage` is `:global` or `:none`. `profile` is not supported and always reports `:none`.
+A slot another SDK held before `init` stays with it and reports `:none` (`:local` for logs) from the start. Foam builds no provider for that signal, so the metric helpers and `set_baggage` do nothing while `log` still delivers. `:global` is checked against the live OpenTelemetry global on each call, so a slot another SDK takes after `init` reports the same way; in that case only the slot changes hands, and the helpers keep using Foam's own providers. Both cases are warned about once on stderr. Body capture is off while another SDK holds the logs slot. `baggage` is `:global` or `:none`. `profile` is not supported and always reports `:none`.
 
 ## Custom spans and other OpenTelemetry APIs
 
@@ -390,13 +402,14 @@ Server bodies need `Foam::Rack::Middleware` after the OpenTelemetry Rack middlew
 
 Client bodies are captured in `:advanced` by the same hooks that set the headers.
 
-- `Net::HTTP`: a request sent through `body_stream` and a response read in a block or into an IO are not captured.
+- `Net::HTTP`: a request is captured whether sent as a string or through `body_stream`. A response read in a block or into an IO is not captured.
 - Faraday: bodies are captured on the Faraday span through Foam's Faraday middleware, whatever the adapter. A request whose body is an IO and a response streamed with `on_data` are not captured.
 - Excon: a request sent with `request_block` and a response received with `response_block` are not captured.
 - HTTPClient: a request whose body is an IO or multipart parts is not captured. A response read in a block is captured.
-- RestClient, `http`, `httpx`, and `ethon`: no bodies, since they get no header capture either.
+- RestClient and HTTParty: captured on the `Net::HTTP` span beneath them.
+- `http`, `httpx`, and `ethon`: no bodies, since they get no header capture either.
 
-TODO(jhartquist): capture streamed bodies: a request sent from an IO and a response the client streams to the app.
+TODO(jhartquist): capture the streamed bodies listed above. For `Net::HTTP`, prepend on `Net::HTTPResponse#read_body` and tee the block or IO the app passes.
 
 ## Loggers
 
@@ -414,6 +427,12 @@ If another SDK owns logs, set `disable_log_sending: true` so records do not arri
 
 Secrets in a log line are masked before export. See Redaction.
 
+## AWS Lambda
+
+The invocation span carries `faas.trigger` for API Gateway, ALB, SNS, SQS, S3, DynamoDB streams, and EventBridge schedule events.
+
+Traces, metrics, and logs are flushed before the handler returns (including failing invocations).
+
 ## Pre-forking servers and threads
 
 TODO(jhartquist): pre-forking servers (Puma in cluster mode, Unicorn, Pitchfork, Resque): whether to call `init` before or after the fork, and what the SDK's batch processors and the metric reader do in the child. Foam registers no fork hook today.
@@ -431,7 +450,13 @@ end
 
 Foam masks the values of sensitive keys in everything it exports: span, span event, and span link attributes; log bodies and attributes; metric data point attributes; and resource attributes. A built-in list of credential and personal-data keys always applies, matched by exact name or by a sensitive segment of the key (`legacy_api_key_2`, `stripeToken`). Keys match after normalization: case, `-`, `.`, and camelCase boundaries are folded, so `apiKey`, `api-key`, and `API_KEY` are one key. Header attributes match on the header name. SQL statements arrive from the database instrumentations with every literal replaced with `?` (see Instrumentation) and are redacted like any other string. Captured HTTP bodies are masked by content type with the same keys, with their framing kept. When masking a body fails, the whole body is replaced with `********`.
 
-Every other string is scanned as text, a log body included. A string that is a whole JSON document is masked by key at every depth, so a list or object under a sensitive key becomes one mask. Any other string has the value after each sensitive key masked, whether the key is quoted as JSON writes it or bare before `=`, `:`, or `=>`, and whether it sits in a query string, a header line, or prose. The value ends at the next delimiter, so the rest of the line is kept, a quoted URL's closing quote included. Such a string is scanned up to 1 MiB; the rest collapses into one mask. `url.query` is masked pair by pair, and names such as `code` and `sig` count as sensitive only inside a query string. Masking is value-only: the key and the surrounding text stay.
+Every other string is scanned as text, log bodies included.
+
+- A string that is a whole JSON document is masked by key at every depth. A list or object under a sensitive key becomes one mask.
+- Any other string has the value after each sensitive key masked. The key may be quoted as JSON writes it or bare before `=`, `:`, or `=>`, in a query string, a header line, or prose. The value ends at the next delimiter, and a quoted URL keeps its closing quote.
+- `url.query` is masked pair by pair. Names such as `code` and `sig` are sensitive only inside a query string.
+
+Scanning stops after 1 MiB, and everything past that point is replaced with a single mask. Masking is value-only: the key and the surrounding text remain intact.
 
 The `redact` option adds the app's own keys, matched exactly after the same normalization. A value under a built-in or `pii` key becomes `[REDACTED]`. A value under a `secrets` key becomes `********`, followed by its last four characters when it is 12 characters or longer. That tail is kept only where the value is read whole under its key: an attribute, a hash or array, or a whole JSON document. A value found by scanning text or a `url.query` pair becomes `[REDACTED]` whatever its key.
 
@@ -528,7 +553,7 @@ The `bunny`, `rdkafka`, `ruby_kafka`, `racecar`, `sidekiq`, `resque`, `que`, `de
 OpenTelemetry packages outside the instrumentation bundle that Foam does not wire yet.
 
 - Resource detectors. Foam's resource carries the service identity, the distro, host, and OS attributes, and the process and SDK attributes the OpenTelemetry SDK adds; `OTEL_RESOURCE_ATTRIBUTES` adds to it. The contrib detectors, `opentelemetry-resource-detector-aws`, `-azure`, `-container`, and `-google_cloud_platform`, are not run. The cloud ones probe metadata endpoints, so they belong off the `init` critical path.
-- Propagators. `init` keeps the SDK's default propagators, W3C `tracecontext` and `baggage`. The SDK honors `OTEL_PROPAGATORS` for `b3`, `b3multi`, `jaeger`, `xray`, and `ottrace` when the propagator gem is present, but the gems are not dependencies of `foam-otel`, and `inject_trace_context` and `extract_trace_context` speak W3C whatever is configured.
+- Propagators. `init` registers W3C `tracecontext` and `baggage` when the propagator slot is free and does not read `OTEL_PROPAGATORS`. The contrib propagator gems (`b3`, `b3multi`, `jaeger`, `xray`, `ottrace`) are not dependencies of `foam-otel`; an app that needs one registers it before `init`. `inject_trace_context` and `extract_trace_context` always use the W3C headers.
 - Samplers. `sample_rate` builds a parent-based ratio sampler and `OTEL_TRACES_SAMPLER` is not consulted. `opentelemetry-sampler-xray` is the only contrib sampler; there is no remote sampler for Jaeger.
 
 ## TODO(jhartquist): Safely

data/lib/foam/otel/internal/aws_lambda.rb

@@ -0,0 +1,69 @@
+# frozen_string_literal: true
+
+module Foam
+  module Otel
+    module Internal
+      # Foam's additions to the aws_lambda instrumentation, prepended on its Wrap module by init
+      # after installation:
+      # 1. Fills in faas.trigger from the event's shape for the AWS event sources the
+      #    instrumentation does not derive it for.
+      # 2. Flushes Foam's logger provider where the wrapper flushes the tracer and meter providers,
+      #    so an invocation's log records leave before the handler returns.
+      module AwsLambda
+        INSTRUMENTATION = "OpenTelemetry::Instrumentation::AwsLambda"
+        TRIGGER = "faas.trigger"
+
+        # A record's event source -> the semantic conventions' trigger for it.
+        RECORD_TRIGGERS = {
+          "aws:sns" => "pubsub",
+          "aws:sqs" => "pubsub",
+          "aws:s3" => "datasource",
+          "aws:dynamodb" => "datasource"
+        }.freeze
+
+        def self.install
+          return unless Instrumentations.instance(INSTRUMENTATION).installed?
+
+          ::OpenTelemetry::Instrumentation::AwsLambda::Wrap.prepend(Wrap)
+        end
+
+        # The trigger the event's shape names, or nil for a shape this does not know. SNS spells
+        # its record key EventSource; the other record sources spell it eventSource.
+        def self.trigger(event)
+          record = event["Records"]&.first
+          if record
+            RECORD_TRIGGERS[record["eventSource"] || record["EventSource"]]
+          elsif event["source"] == "aws.events" && event["detail-type"] == "Scheduled Event"
+            "timer"
+          end
+        end
+
+        module Wrap
+          def wrap_lambda(
+            event:, context:, handler:,
+            flush_timeout: ::OpenTelemetry::Instrumentation::AwsLambda::Wrap::DEFAULT_FLUSH_TIMEOUT
+          )
+            super
+          ensure
+            # Runs for an error too, so the records of a failed invocation leave before the error
+            # reaches the runtime. The timeout is the one the wrapper gives the other providers.
+            Safely.call("aws_lambda log flush") do
+              State.logger_provider&.force_flush(timeout: flush_timeout)
+            end
+          end
+
+          private
+
+          def otel_attributes(event, context)
+            attributes = super
+            return attributes if attributes.key?(TRIGGER)
+
+            trigger = Safely.call("faas.trigger") { AwsLambda.trigger(event) }
+            attributes[TRIGGER] = trigger if trigger
+            attributes
+          end
+        end
+      end
+    end
+  end
+end

data/lib/foam/otel/internal/clients/net_http.rb

@@ -6,10 +6,9 @@ module Foam
       module Clients
         # Sets the safe headers on Net::HTTP client spans and, in advanced network capture,
         # captures their request and response bodies. Prepended on Net::HTTP by init unless network
-        # capture is off. Captures a body only when Net::HTTP holds it as a String, so these are
-        # not captured:
-        # - a request sent through body_stream
-        # - a response read in a block or into an IO
+        # capture is off. A request body is captured whether set as a String or streamed through
+        # body_stream. A response body is captured only when Net::HTTP reads it into a String; a
+        # response the app reads itself, in a block or into an IO, is not.
         module NetHttp
           private
 
@@ -35,6 +34,7 @@ module Foam
                 yield response if block_given?
               end
             ensure
+              req.body_stream = req.body_stream.stream if req.body_stream.is_a?(ReadTee)
               request&.finalize(complete: !res.nil?)
               Safely.call("request header capture") do
                 BodyCapture.set_header_attributes(span, :request, body_headers(req))
@@ -44,15 +44,24 @@ module Foam
             res
           end
 
-          # Only a String body is captured: a stream or multipart data goes straight to the socket.
+          # A String body is observed in one piece. A body_stream is wrapped in a ReadTee, which
+          # observes each chunk as Net::HTTP copies it to the socket; transport_request unwraps it
+          # afterwards. Multipart form data is not captured.
           def capture_request(req, span, body_redactor)
             body = req.body
-            return unless body.is_a?(String)
+            stream = req.body_stream
+            return unless body.is_a?(String) || stream.respond_to?(:read)
 
             capture = BodyCapture.start(span, body_redactor, side: :client, direction: :request) do
               body_headers(req)
             end
-            capture&.observe(body)
+            return unless capture
+
+            if body
+              capture.observe(body)
+            else
+              req.body_stream = ReadTee.new(stream, capture)
+            end
             capture
           end
 

data/lib/foam/otel/internal/clients/read_tee.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+module Foam
+  module Otel
+    module Internal
+      module Clients
+        # Wraps a request body_stream so the bytes Net::HTTP sends can be captured without buffering
+        # them. Net::HTTP copies the stream to the socket with IO.copy_stream, so the read is the
+        # only place to see them.
+        class ReadTee
+          # The wrapped stream, for putting it back on the request once it is sent.
+          attr_reader :stream
+
+          def initialize(stream, capture)
+            @stream = stream
+            @capture = capture
+          end
+
+          def read(*args)
+            bytes = @stream.read(*args)
+            @capture.observe(bytes) if bytes
+            bytes
+          end
+        end
+      end
+    end
+  end
+end

data/lib/foam/otel/internal/constants.rb

@@ -52,15 +52,24 @@ module Foam
         # The log body of every body chunk record; the chunk's text goes in its attributes.
         BODY_CHUNK_EVENT_BODY = "HTTP body chunk"
 
-        # The mask for a value. A secret key's value read whole gets FULL_MASK and a tail instead.
+        # Replaces a redacted value.
         REDACTED_VALUE = "[REDACTED]"
 
-        # The mask for a value under a secret key that has no tail to keep, and for one redaction failed on.
+        # Replaces a secret: all of it when short or not text, otherwise all but its last four
+        # characters. Also stands in for a value whose redaction failed.
         FULL_MASK = "********"
 
-        # The length from which a value under a secret key keeps its last four characters.
+        # Secrets at least this long keep their last four characters unmasked.
         TAIL_MASK_THRESHOLD = 12
 
+        # Finished spans Foam's span processor holds before dropping the oldest. The SDK default
+        # of 2048 loses spans when a loop finishes more than that before the export thread runs.
+        SPAN_QUEUE_SIZE = 4096
+
+        # Seconds flush and shutdown wait by default. It is the OTLP exporters' own request timeout,
+        # and it keeps the at_exit shutdown inside a 30 second deploy grace period.
+        LIFECYCLE_TIMEOUT = 10
+
         module_function
 
         def bearer_headers(token)

data/lib/foam/otel/internal/fork_hooks.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+module Foam
+  module Otel
+    module Internal
+      # Runs State.after_fork in every child this process forks, by wrapping Process._fork, the
+      # method fork and Process.fork call. Preforking servers init Foam in the master, and a
+      # forked child gets the parent's memory but none of its threads, so the periodic metric
+      # reader's export thread is gone and the child's metrics would never leave it, and the
+      # providers' resource still names the parent process.
+      module ForkHooks
+        def _fork
+          parent = Process.pid
+          super.tap { State.after_fork unless Process.pid == parent }
+        end
+
+        def self.attach
+          Process.singleton_class.prepend(self)
+        end
+      end
+    end
+  end
+end

data/lib/foam/otel/internal/init.rb

@@ -18,6 +18,10 @@ module Foam
           # Every outcome is reported with the same identity, including the ones that stop early.
           identity = options.slice(:name, :environment, :token)
 
+          # Recorded before the checks, so state names the app after a disabled or failed init too.
+          # The token stays out of state on purpose; it is read by health checks and tests.
+          State.record_params(options.slice(:name, :environment))
+
           # Teams may turn the SDK off in specific environments.
           unless options[:enabled]
             Report.call(severity: :info, message: "SDK disabled", **identity)
@@ -63,27 +67,35 @@ module Foam
           State.take_provider(:logger)&.shutdown(timeout: nil)
 
           redactor = Redactor.from_option(options[:redact])
-          configure(instrumentation_config: instrumentation_config, redactor: redactor, **options)
-
-          # The client patches set the safe headers on every client span and, in advanced mode,
-          # capture bodies too. Bodies are redacted by the same rules as attributes, so only
-          # advanced builds a body redactor; its presence is what switches body capture on.
-          body_redactor = BodyRedactor.new(redactor) if options[:network_capture] == :advanced
-          install_client_patches unless options[:network_capture] == :off
-
-          # Recorded last so a failed init leaves the state uninitialized.
-          # The token stays out of state on purpose; it is read by health checks and tests.
-          State.record_init(
-            params: options.slice(:name, :environment),
-            tracer_provider: OpenTelemetry.tracer_provider,
-            meter_provider: OpenTelemetry.meter_provider,
-            logger_provider: OpenTelemetry.logger_provider,
-            propagator: OpenTelemetry.propagation,
-            instrumentations: Instrumentations.installed(instrumentation_config),
-            instrumentation_config: instrumentation_config,
-            body_redactor: body_redactor,
-            network_capture: options[:network_capture]
+          providers, claimed_slots = configure(
+            instrumentation_config: instrumentation_config, redactor: redactor, **options
           )
+          begin
+            ForkHooks.attach
+
+            # The client patches set the safe headers on every client span and, in advanced mode,
+            # capture bodies too. Bodies are redacted by the same rules as attributes, so only
+            # advanced builds a body redactor; its presence is what switches body capture on.
+            body_redactor = BodyRedactor.new(redactor) if options[:network_capture] == :advanced
+            install_client_patches unless options[:network_capture] == :off
+
+            # Adds faas.trigger for the event sources the aws_lambda instrumentation skips, and flushes
+            # Foam's logger provider where it flushes the tracer and meter providers.
+            Safely.call("aws_lambda") { AwsLambda.install }
+
+            # Recorded last so a failed init leaves the state uninitialized.
+            State.record_init(
+              providers: providers,
+              claimed_slots: claimed_slots,
+              instrumentations: Instrumentations.installed(instrumentation_config),
+              instrumentation_config: instrumentation_config,
+              body_redactor: body_redactor,
+              network_capture: options[:network_capture]
+            )
+          rescue
+            roll_back(providers, claimed_slots)
+            raise
+          end
 
           # Flush whatever the batch processors still hold before the process exits.
           at_exit { Foam.shutdown }
@@ -94,66 +106,111 @@ module Foam
           nil
         end
 
+        # Builds Foam's pipelines and claims the global slots the API still holds itself. Returns
+        # the providers built, by kind, and the slots claimed, by signal.
         def configure(
           token:, name:, environment:, sample_rate:, instrumentation_config:, additional_span_processors:,
           additional_metric_readers:, additional_log_record_processors:, additional_resource_attributes:,
           ignored_outbound_hosts:, redactor:, before_send:, version: nil, **
         )
-          # Declared up front so the rescue below can stop whichever of these exist when something fails.
-          span_processor = nil
-          metric_reader = nil
-          log_record_processor = nil
+          # Declared up front so the rescue below can stop whichever providers exist when
+          # something fails.
+          providers = {}
 
-          sampler = IgnoredHosts.new(
-            ignored_hosts(ignored_outbound_hosts),
-            OpenTelemetry::SDK::Trace::Samplers.parent_based(
-              root: OpenTelemetry::SDK::Trace::Samplers.trace_id_ratio_based(sample_rate)
-            )
+          # The SDK's own switch, honored the way SDK.configure honors it: nothing is installed.
+          raise "OTEL_SDK_DISABLED is true" if ENV["OTEL_SDK_DISABLED"] == "true"
+
+          sampler = sampler(sample_rate, ignored_outbound_hosts)
+          resource = OpenTelemetry::SDK::Resources::Resource.default.merge(
+            Resource.create(name, environment, version, additional_resource_attributes)
           )
-          resource = Resource.create(name, environment, version, additional_resource_attributes)
           before_send = BeforeSend.new(before_send)
 
-          trace_exporter = Exporters::TraceExporter.new(
+          # A slot another SDK owns is left in place, and Foam builds no pipeline for that signal,
+          # since nothing would feed one. The logger provider is built either way, so log still
+          # delivers whoever holds the slot.
+          free = Slots::SIGNALS.select { |signal| Slots.free?(signal) }
+          (Slots::SIGNALS - free).each do |signal|
+            Diagnostics.warn("#{signal} global slot taken by #{Slots.live(signal).class}")
+          end
+          if free.include?(:traces)
+            exporter = Exporters::TraceExporter.new(
+              token: token, redactor: redactor, before_send: before_send
+            )
+            providers[:tracer] = tracer_provider(
+              resource, sampler, exporter, additional_span_processors
+            )
+          end
+          if free.include?(:metrics)
+            exporter = Exporters::MetricExporter.new(token: token, redactor: redactor)
+            providers[:meter] = meter_provider(resource, exporter, additional_metric_readers)
+          end
+          exporter = Exporters::LogExporter.new(
             token: token, redactor: redactor, before_send: before_send
           )
-          metrics_exporter = Exporters::MetricExporter.new(token: token, redactor: redactor)
-          logs_exporter = Exporters::LogExporter.new(
-            token: token, redactor: redactor, before_send: before_send
+          providers[:logger] = logger_provider(
+            resource, exporter, additional_log_record_processors
           )
 
-          span_processor = OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(trace_exporter)
-          metric_reader = OpenTelemetry::SDK::Metrics::Export::PeriodicMetricReader.new(exporter: metrics_exporter)
-          log_record_processor = OpenTelemetry::SDK::Logs::Export::BatchLogRecordProcessor.new(logs_exporter)
-
-          # SDK.configure swallows its own exceptions; an unchanged slot is how we know it did not finish.
-          previous_tracer_provider = OpenTelemetry.tracer_provider
-
-          # configure installs the global providers and sets up the used instrumentations.
-          OpenTelemetry::SDK.configure do |c|
-            c.resource = resource
-
-            c.add_span_processor(span_processor)
-            additional_span_processors.each { |processor| c.add_span_processor(processor) }
+          claimed_slots = {
+            traces: providers[:tracer],
+            metrics: providers[:meter],
+            logs: (providers[:logger] if free.include?(:logs)),
+            baggage: (Propagation::PROPAGATOR if free.include?(:baggage))
+          }.compact
+          claimed_slots.each { |signal, object| Slots.claim(signal, object) }
+
+          # Installs the bundle and the caller's entries, those whose library is already loaded; the
+          # rest are skipped. After the slots, so each one binds its tracer to whoever holds traces.
+          OpenTelemetry::Instrumentation.registry.install(
+            instrumentation_config.keys, instrumentation_config
+          )
+          [providers, claimed_slots]
+        rescue
+          roll_back(providers, claimed_slots)
+          raise
+        end
 
-            c.add_metric_reader(metric_reader)
-            additional_metric_readers.each { |reader| c.add_metric_reader(reader) }
+        # Frees the slots and stops the providers a failed init built, so the next init starts clean.
+        def roll_back(providers, claimed_slots)
+          claimed_slots&.each { |signal, object| Slots.release(signal, object) }
+          Lifecycle.each(:shutdown, providers.values, timeout: nil)
+        end
 
-            c.add_log_record_processor(log_record_processor)
-            additional_log_record_processors.each { |processor| c.add_log_record_processor(processor) }
+        # Drops spans headed for an ignored host and samples the rest per trace at sample_rate.
+        def sampler(sample_rate, ignored_outbound_hosts)
+          IgnoredHosts.new(
+            ignored_hosts(ignored_outbound_hosts),
+            OpenTelemetry::SDK::Trace::Samplers.parent_based(
+              root: OpenTelemetry::SDK::Trace::Samplers.trace_id_ratio_based(sample_rate)
+            )
+          )
+        end
 
-            # Installs the bundle and the caller's entries, those whose library is already loaded; the rest are skipped.
-            instrumentation_config.each { |name, config| c.use(name, config) }
-          end
+        def tracer_provider(resource, sampler, exporter, additional)
+          provider = OpenTelemetry::SDK::Trace::TracerProvider.new(
+            resource: resource, sampler: sampler
+          )
+          foam = OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(
+            exporter,
+            max_queue_size: Integer(ENV.fetch("OTEL_BSP_MAX_QUEUE_SIZE", Constants::SPAN_QUEUE_SIZE))
+          )
+          [foam, *additional].each { |processor| provider.add_span_processor(processor) }
+          provider
+        end
 
-          if OpenTelemetry.tracer_provider.equal?(previous_tracer_provider)
-            raise "no tracer provider was installed"
-          end
+        def meter_provider(resource, exporter, additional)
+          provider = OpenTelemetry::SDK::Metrics::MeterProvider.new(resource: resource)
+          foam = OpenTelemetry::SDK::Metrics::Export::PeriodicMetricReader.new(exporter: exporter)
+          [foam, *additional].each { |reader| provider.add_metric_reader(reader) }
+          provider
+        end
 
-          # The configurator builds the provider with the SDK's default sampler and takes no other.
-          OpenTelemetry.tracer_provider.sampler = sampler
-        rescue
-          Lifecycle.each(:shutdown, [span_processor, metric_reader, log_record_processor], timeout: nil)
-          raise
+        def logger_provider(resource, exporter, additional)
+          provider = OpenTelemetry::SDK::Logs::LoggerProvider.new(resource: resource)
+          foam = OpenTelemetry::SDK::Logs::Export::BatchLogRecordProcessor.new(exporter)
+          [foam, *additional].each { |processor| provider.add_log_record_processor(processor) }
+          provider
         end
 
         # Net::HTTP is patched without a check, since foam-otel loads it for its own exporters; the

data/lib/foam/otel/internal/instrumentations.rb

@@ -88,9 +88,9 @@ module Foam
 
         module_function
 
-        # What init installs, name -> config: the bundle, with the logger bridge disabled when log sending
-        # is, then the caller's entries, so an entry replaces Foam's config for the same name. One whose
-        # instrumentation is not loaded stops init before anything is built.
+        # The instrumentation configs init installs, by name: the bundle, then the caller's entries,
+        # which override it. Disabling log sending disables the logger bridge. An entry naming an
+        # instrumentation that is not loaded fails init before anything is built.
         def config_map(additional, disable_log_sending: false)
           bundle = TABLE.keys.to_h { |name| [name, nil] }.merge(CONFIG)
           bundle[LOGGER] = {enabled: false} if disable_log_sending

data/lib/foam/otel/internal/lifecycle.rb

@@ -19,13 +19,21 @@ module Foam
           each(:shutdown, providers, timeout: timeout)
         end
 
-        # Providers, processors, and readers all flush and shut down the same way; each on its own so
-        # one failure does not skip the others, and each with what is left of one shared timeout.
+        # Providers, processors, and readers all flush and shut down the same way; each in its own
+        # thread, so one that hangs or fails holds up neither the others nor the caller. The caller is
+        # back within one shared timeout; a thread still running then finishes on its own.
         def each(operation, components, timeout:)
           start = OpenTelemetry::Common::Utilities.timeout_timestamp
-          components.compact.each do |component|
+          threads = components.compact.to_h do |component|
+            [component, Thread.new do
+              # The joining thread reports the failure, so the thread itself stays quiet.
+              Thread.current.report_on_exception = false
+              component.public_send(operation, timeout: timeout)
+            end]
+          end
+          threads.each do |component, thread|
             remaining = OpenTelemetry::Common::Utilities.maybe_timeout(timeout, start)
-            code = component.public_send(operation, timeout: remaining)
+            code = thread.join(remaining) ? thread.value : TIMEOUT
 
             # Failures are logged by the exporter or provider themselves; a timeout is silent upstream.
             Diagnostics.warn("#{operation}: #{component.class.name} timed out") if code == TIMEOUT

data/lib/foam/otel/internal/otlp.rb

@@ -2,6 +2,8 @@
 
 require "json"
 require "net/http"
+require "socket"
+require "timeout"
 require "opentelemetry/common"
 
 module Foam
@@ -12,24 +14,47 @@ module Foam
       # correlation, one request per record.
       module Otlp
         TIMEOUT = 2
+        # Seconds the caller spends resolving the endpoint host before a send.
+        RESOLVE_TIMEOUT = 1
+
+        @pending = []
+        @mutex = Mutex.new
 
         module_function
 
         # Sends in the background so the caller never waits on the network; delivery is best effort.
         def send_log(token:, resource_attributes:, scope_name:, severity:, body:)
           request = JSON.generate(log_request(resource_attributes, scope_name, severity, body))
-          url = Endpoint.url(Constants::LOGS_PATH)
-          Thread.new do
+          uri = URI(Endpoint.url(Constants::LOGS_PATH))
+          address = resolve(uri)
+          thread = Thread.new do
             # Untraced so the request never becomes a span once Net::HTTP is instrumented, and
             # marked as Foam's own so the client patches leave it alone.
             OwnRequest.mark do
-              OpenTelemetry::Common::Utilities.untraced { post(url, request, token) }
+              OpenTelemetry::Common::Utilities.untraced { post(uri, address, request, token) }
             end
           rescue
             nil
           end
+          @mutex.synchronize do
+            @pending.select!(&:alive?)
+            @pending << thread
+          end
+          thread
+        end
+
+        # Waits for the sends still in flight, each with what is left of one shared timeout.
+        def drain(timeout: TIMEOUT)
+          start = OpenTelemetry::Common::Utilities.timeout_timestamp
+          @mutex.synchronize { @pending.dup }.each do |thread|
+            thread.join(OpenTelemetry::Common::Utilities.maybe_timeout(timeout, start))
+          end
+          nil
         end
 
+        # A process that exits right after init would otherwise lose its report.
+        at_exit { drain }
+
         def log_request(resource_attributes, scope_name, severity, body)
           {
             resourceLogs: [{
@@ -49,13 +74,25 @@ module Foam
           }
         end
 
-        def post(url, request, token)
-          uri = URI(url)
+        # Resolve on the calling thread. On Ruby 3.3, forking while a background thread is mid
+        # lookup leaves the child's resolver deadlocked.
+        def resolve(uri)
+          Timeout.timeout(RESOLVE_TIMEOUT) do
+            Addrinfo.getaddrinfo(uri.hostname, uri.port, nil, :STREAM).first.ip_address
+          end
+        rescue
+          nil
+        end
+
+        def post(uri, address, request, token)
           headers = Constants.bearer_headers(token).merge("Content-Type" => "application/json")
-          Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == "https",
+          Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == "https", ipaddr: address,
             open_timeout: TIMEOUT, read_timeout: TIMEOUT, write_timeout: TIMEOUT) do |http|
             http.post(uri.path, request, headers)
           end
+        rescue SystemCallError, Net::OpenTimeout
+          # The address is the first Net::HTTP would try itself; a retry without it tries the rest.
+          address ? post(uri, nil, request, token) : raise
         end
       end
     end

data/lib/foam/otel/internal/redactor.rb

@@ -40,6 +40,9 @@ module Foam
         # keeps its quotes or runs to the next delimiter; an auth scheme word like Bearer is part of it.
         JSON_KEY = /"((?:[^"\\\n]|\\.)*+)"(\s*(?:=>|:)\s*)/
         BARE_KEY = /(?<![\w.%])([A-Za-z_][\w.\-%]*+)([ \t]*(?:=>|={1,3}|:)[ \t]*)/
+        # Both key shapes in one pattern, so one pass visits every key in source order. Groups 1 and 2
+        # are a quoted key and its separator, groups 3 and 4 a bare one.
+        KEY = Regexp.union(JSON_KEY, BARE_KEY)
         QUOTED_VALUE = /"(?:[^"\\\n]|\\.)*+"?|'(?:[^'\\\n]|\\.)*+'?/
         BARE_VALUE = /(?![\[{(])[^\s&"'<>,;)\]}]+/
         VALUE = /(?:(?:bearer|basic|digest|token)[ \t]+)?(?:#{QUOTED_VALUE}|#{BARE_VALUE})/i
@@ -126,7 +129,9 @@ module Foam
 
         # [REDACTED] for a built-in or pii key; a secret key keeps the last four characters of a String,
         # Symbol, or number of 12 or more characters, so a value stays recognizable without being usable.
+        # nil stays nil: there is no value to protect.
         def mask(kind, value)
+          return if value.nil?
           return Constants::REDACTED_VALUE if value == Constants::REDACTED_VALUE
           return Constants::REDACTED_VALUE unless kind == :secret
           return Constants::FULL_MASK unless value.is_a?(String) || value.is_a?(Symbol) || value.is_a?(Numeric)
@@ -211,8 +216,7 @@ module Foam
           mask_kind(name) || (:floor if HTTP_HEADERS.include?(normalize_key(name)))
         end
 
-        # Arrays are masked element by element so their length survives; nil is masked like any value,
-        # so the export shows the key was set.
+        # Arrays are masked element by element so their length survives.
         def mask_elements(kind, value)
           value.is_a?(Array) ? value.map { |item| mask(kind, item) } : mask(kind, value)
         end
@@ -232,20 +236,21 @@ module Foam
         end
 
         # Masks the value after each sensitive key, quoted as JSON writes it or bare. Past 1 MiB the rest
-        # collapses into one mask, which bounds the work on a hostile line.
+        # collapses into one mask, which bounds the work on a hostile line; a value the cut falls
+        # inside already ends in that mask.
         def scan_text(bytes)
-          return scan_pairs(scan_pairs(bytes, JSON_KEY), BARE_KEY) if bytes.bytesize <= SCAN_BYTE_CAP
+          return scan_pairs(bytes) if bytes.bytesize <= SCAN_BYTE_CAP
 
-          head = bytes.byteslice(0, SCAN_BYTE_CAP)
-          scan_pairs(scan_pairs(head, JSON_KEY), BARE_KEY) << Constants::REDACTED_VALUE
+          head = scan_pairs(bytes.byteslice(0, SCAN_BYTE_CAP))
+          head.end_with?(Constants::REDACTED_VALUE) ? head : head << Constants::REDACTED_VALUE
         end
 
-        # Finds each key with the pattern and masks the value after it when the key is sensitive.
-        def scan_pairs(bytes, key_pattern)
+        # Finds each key, quoted or bare, and masks the value after it when the key is sensitive.
+        def scan_pairs(bytes)
           scanner = StringScanner.new(bytes, fixed_anchor: true)
           out = nil
           last = 0
-          while scanner.scan_until(key_pattern)
+          while scanner.scan_until(KEY)
             next unless masked_key?(scanner, bytes)
 
             start = scanner.pos
@@ -260,13 +265,13 @@ module Foam
         # A key is masked on its name, or on a query-only name when it follows ?, &, or # with a bare =
         # after it: code, sig and the like are credentials only in a query string.
         def masked_key?(scanner, bytes)
-          key = decode_key(scanner[1])
+          key = decode_key(scanner[1] || scanner[3])
           return true if mask_kind(key)
 
           start = scanner.pos - scanner.matched_size
           return false unless start > 0 && QUERY_CONTEXT.include?(bytes.getbyte(start - 1))
 
-          scanner[2] == "=" && query_key?(key)
+          scanner[4] == "=" && query_key?(key)
         end
 
         # The mask in the quotes the value came with; a quote left open stays open.

data/lib/foam/otel/internal/resource.rb

@@ -2,27 +2,31 @@
 
 require "etc"
 require "rbconfig"
+require "securerandom"
 require "socket"
 require "opentelemetry/sdk"
 require "opentelemetry-semantic_conventions"
 require "opentelemetry/semconv/incubating/host"
 require "opentelemetry/semconv/incubating/os"
+require "opentelemetry/semconv/incubating/process"
 
 module Foam
   module Otel
     module Internal
       # A resource is the set of attributes describing the process that produced the telemetry:
       # which service, environment, host, and OS it came from. Every span, log, and metric
-      # carries it, and Foam reads the identity from it. This builds Foam's part; the
-      # configurator merges it over the SDK default, which already holds the process attributes
+      # carries it, and Foam reads the identity from it. This builds Foam's part; init
+      # merges it over the SDK default, which already holds the process attributes
       # and OTEL_SERVICE_NAME / OTEL_RESOURCE_ATTRIBUTES.
       module Resource
         HOST = OpenTelemetry::SemConv::Incubating::HOST
         OS = OpenTelemetry::SemConv::Incubating::OS
+        PROCESS = OpenTelemetry::SemConv::Incubating::PROCESS
 
         IDENTITY_ATTRIBUTES = [
           OpenTelemetry::SemConv::SERVICE::SERVICE_NAME,
           OpenTelemetry::SemConv::SERVICE::SERVICE_VERSION,
+          OpenTelemetry::SemConv::SERVICE::SERVICE_INSTANCE_ID,
           OpenTelemetry::SemConv::DEPLOYMENT::DEPLOYMENT_ENVIRONMENT_NAME
         ].freeze
 
@@ -49,11 +53,12 @@ module Foam
 
         module_function
 
-        # Foam's identity merges last so it wins over the caller's attributes and the env.
+        # Foam's identity and process merge last, so they win over the caller's attributes and env.
         def create(name, environment, version, additional_attributes)
           attributes = host_and_os
             .merge(without_identity(additional_attributes))
             .merge(identity(name, environment, version))
+            .merge(process)
           OpenTelemetry::SDK::Resources::Resource.create(attributes)
         end
 
@@ -67,11 +72,29 @@ module Foam
           }.compact
         end
 
+        # The attributes that name the process rather than the service: its pid and an instance id
+        # Foam generates, since nothing in the environment tells two runs of one service apart.
+        # A forked child inherits its parent's, so it sets both again.
+        def process
+          {
+            PROCESS::PROCESS_PID => Process.pid,
+            OpenTelemetry::SemConv::SERVICE::SERVICE_INSTANCE_ID => SecureRandom.uuid
+          }
+        end
+
+        # Sets the process attributes again on the resource of each provider, for the process
+        # that holds the providers now. A provider takes its resource once, at construction.
+        def refresh(providers)
+          update = OpenTelemetry::SDK::Resources::Resource.create(process)
+          providers.each do |provider|
+            resource = provider.instance_variable_get(:@resource).merge(update)
+            provider.instance_variable_set(:@resource, resource)
+          end
+        end
+
         def without_identity(attributes)
           (attributes.keys & IDENTITY_ATTRIBUTES).each do |key|
-            Diagnostics.warn(
-              "additional_resource_attributes #{key} is set from the init options; ignored"
-            )
+            Diagnostics.warn("additional_resource_attributes #{key} is set by Foam; ignored")
           end
           attributes.except(*IDENTITY_ATTRIBUTES)
         end

data/lib/foam/otel/internal/slots.rb

@@ -0,0 +1,52 @@
+# frozen_string_literal: true
+
+require "opentelemetry/sdk"
+require "opentelemetry-logs-sdk"
+require "opentelemetry-metrics-sdk"
+
+module Foam
+  module Otel
+    module Internal
+      # The OpenTelemetry globals an SDK registers itself in: the tracer, meter, and logger
+      # providers and the propagator. A slot is free while it holds the API's own no-op.
+      module Slots
+        ACCESSORS = {
+          traces: :tracer_provider,
+          metrics: :meter_provider,
+          logs: :logger_provider,
+          baggage: :propagation
+        }.freeze
+
+        SIGNALS = ACCESSORS.keys.freeze
+
+        DEFAULTS = {
+          traces: OpenTelemetry::Internal::ProxyTracerProvider,
+          metrics: OpenTelemetry::Internal::ProxyMeterProvider,
+          logs: OpenTelemetry::Internal::ProxyLoggerProvider,
+          baggage: OpenTelemetry::Context::Propagation::NoopTextMapPropagator
+        }.freeze
+
+        module_function
+
+        def live(signal)
+          OpenTelemetry.public_send(ACCESSORS.fetch(signal))
+        end
+
+        def free?(signal)
+          live(signal).instance_of?(DEFAULTS.fetch(signal))
+        end
+
+        def claim(signal, object)
+          OpenTelemetry.public_send(:"#{ACCESSORS.fetch(signal)}=", object)
+          nil
+        end
+
+        # Puts the API's no-op back while the slot still holds object, so the next registration,
+        # or a second init, finds the slot free. A slot another SDK claimed since is left in place.
+        def release(signal, object)
+          claim(signal, DEFAULTS.fetch(signal).new) if live(signal).equal?(object)
+        end
+      end
+    end
+  end
+end

data/lib/foam/otel/internal/state.rb

@@ -12,7 +12,7 @@ module Foam
         @initialized = false
         @signals = NO_SIGNALS
         @providers = {}
-        @propagator = nil
+        @claimed_slots = {}
         @taken = []
         @instrumentations = []
         @instrumentation_config = {}
@@ -39,19 +39,32 @@ module Foam
             @signals[signal] != :none
           end
 
+          # Records the app's name and environment at the start of init, so state names the app
+          # even when init stops early.
+          def record_params(params)
+            @mutex.synchronize { @params = params unless @initialized }
+          end
+
+          # Records a finished init and marks the state initialized. providers holds what init
+          # built, keyed :tracer, :meter and :logger, for flush and shutdown to run over.
+          # claimed_slots holds the object init placed in each global slot, keyed by signal; a
+          # signal reports :global only while that object is still in its slot.
           def record_init(
-            params:, tracer_provider:, meter_provider:, logger_provider:, propagator:,
-            instrumentations:, instrumentation_config:, body_redactor:, network_capture:
+            providers:, claimed_slots:, instrumentations:, instrumentation_config:,
+            body_redactor:, network_capture:
           )
             @mutex.synchronize do
-              @params = params
-              @providers = {tracer: tracer_provider, meter: meter_provider, logger: logger_provider}
-              @propagator = propagator
+              @providers = providers
+              @claimed_slots = claimed_slots
               @instrumentations = instrumentations
               @instrumentation_config = instrumentation_config
               @body_redactor = body_redactor
               @network_capture = network_capture
-              @signals = @signals.merge(traces: :global, metrics: :global, logs: :global, baggage: :global)
+              @signals = @signals.merge(
+                Slots::SIGNALS.to_h do |signal|
+                  [signal, claimed_slots.key?(signal) ? :global : without_slot(signal)]
+                end
+              )
               @initialized = true
             end
           end
@@ -72,13 +85,15 @@ module Foam
             @mutex.synchronize { @providers.delete(kind) }
           end
 
-          # Shutdown hands the providers back and leaves the state as it was before init, so the
-          # helpers go quiet again and state stops reporting slots we no longer hold.
+          # Shutdown hands the providers back, gives each slot Foam still holds back to the API, and
+          # leaves the state as it was before init, so the helpers go quiet again and state stops
+          # reporting slots we no longer hold.
           def take_providers
             @mutex.synchronize do
+              @claimed_slots.each { |signal, object| Slots.release(signal, object) }
               providers = @providers.values.compact
               @providers = {}
-              @propagator = nil
+              @claimed_slots = {}
               @instrumentations = []
               @instrumentation_config = {}
               @late = []
@@ -104,10 +119,10 @@ module Foam
             @mutex.synchronize { @providers[:logger] }
           end
 
-          # The switch for body capture: set only by an init in advanced mode, so the middleware and
-          # the client patches capture no bodies before init, in basic mode, and after shutdown.
+          # The switch for body capture: set only by an init in advanced mode, and off while another
+          # SDK holds the logs slot, the rule the Node package follows too.
           def body_redactor
-            @mutex.synchronize { @body_redactor }
+            @mutex.synchronize { @body_redactor unless %i[local none].include?(source(:logs)) }
           end
 
           # The network capture mode, :off until init runs and again after shutdown, so the Rack
@@ -116,6 +131,22 @@ module Foam
             @mutex.synchronize { @network_capture }
           end
 
+          # A forked child inherits the parent's providers. Their resource names the parent process,
+          # and the periodic metric reader's export thread did not survive the fork, so the child
+          # gets its own process attributes and a running reader. The batch processors reset on
+          # their own when the pid changes.
+          def after_fork
+            Safely.call("after fork") do
+              Resource.refresh(providers)
+              meter = @mutex.synchronize { @providers[:meter] }
+              next unless meter
+
+              meter.metric_readers.each do |reader|
+                reader.after_fork if reader.respond_to?(:after_fork)
+              end
+            end
+          end
+
           private
 
           # A library loaded after init is never instrumented, and upstream says nothing about it at the
@@ -129,44 +160,28 @@ module Foam
             end
           end
 
-          # Anyone can overwrite the OpenTelemetry globals after init, so :global is checked against
-          # the live slot at call time rather than trusted from init.
           def live_signals
-            @signals.to_h do |signal, source|
-              # We only need to check signals marked as :global
-              next [signal, source] unless source == :global
-
-              # Make sure that the source is actually in the live slot
-              live = live_slot(signal)
-              next [signal, source] if live.equal?(installed(signal))
-
-              # Store overwritten slots in @taken so we only send a warning the first time
-              unless @taken.include?(signal)
-                @taken << signal
-                Diagnostics.warn("#{signal} global slot taken by #{live.class}")
-              end
-
-              # Foam's logger provider keeps working without the slot, so log still delivers.
-              [signal, (signal == :logs) ? :local : :none]
-            end
+            @signals.to_h { |signal, _| [signal, source(signal)] }
           end
 
-          def live_slot(signal)
-            case signal
-            when :traces then OpenTelemetry.tracer_provider
-            when :metrics then OpenTelemetry.meter_provider
-            when :logs then OpenTelemetry.logger_provider
-            when :baggage then OpenTelemetry.propagation
+          # Anyone can overwrite the OpenTelemetry globals after init, so a signal recorded as
+          # :global is checked against the live slot at call time, and warned about once when
+          # displaced.
+          def source(signal)
+            return @signals[signal] unless @signals[signal] == :global
+            return :global if Slots.live(signal).equal?(@claimed_slots[signal])
+
+            unless @taken.include?(signal)
+              @taken << signal
+              Diagnostics.warn("#{signal} global slot taken by #{Slots.live(signal).class}")
             end
+            without_slot(signal)
           end
 
-          def installed(signal)
-            case signal
-            when :traces then @providers[:tracer]
-            when :metrics then @providers[:meter]
-            when :logs then @providers[:logger]
-            when :baggage then @propagator
-            end
+          # What a signal reports without its slot. Foam's logger provider keeps working, so log
+          # still delivers; the other helpers go quiet.
+          def without_slot(signal)
+            (signal == :logs) ? :local : :none
           end
         end
       end

data/lib/foam/otel/internal/traces.rb

@@ -12,8 +12,16 @@ module Foam
           span = OpenTelemetry::Trace.current_span
           return unless span.recording?
 
-          span.record_exception(exception, attributes: attributes)
-          span.status = OpenTelemetry::Trace::Status.error(exception.message)
+          if exception.is_a?(Exception)
+            span.record_exception(exception, attributes: attributes)
+            message = exception.message
+          else
+            # Anything else has no type or backtrace to record, so only its text lands on the event.
+            message = exception.to_s
+            event_attributes = {"exception.message" => message}.merge(attributes.to_h)
+            span.add_event("exception", attributes: event_attributes)
+          end
+          span.status = OpenTelemetry::Trace::Status.error(message)
           nil
         end
       end

data/lib/foam/otel/version.rb

@@ -2,6 +2,6 @@
 
 module Foam
   module Otel
-    VERSION = "3.0.0.alpha.3"
+    VERSION = "3.0.0.alpha.4"
   end
 end

data/lib/foam/otel.rb

@@ -17,13 +17,17 @@ require_relative "otel/internal/body_capture"
 require_relative "otel/internal/rack/input_tee"
 require_relative "otel/internal/rack/body_tee"
 require_relative "rack/middleware"
+require_relative "otel/internal/clients/read_tee"
 require_relative "otel/internal/clients/net_http"
 require_relative "otel/internal/clients/excon"
 require_relative "otel/internal/clients/http_client"
 require_relative "otel/internal/clients/faraday"
+require_relative "otel/internal/aws_lambda"
 require_relative "otel/internal/before_send"
 require_relative "otel/internal/instrumentations"
 require_relative "otel/internal/ignored_hosts"
+require_relative "otel/internal/slots"
+require_relative "otel/internal/fork_hooks"
 require_relative "otel/internal/resource"
 require_relative "otel/internal/init"
 require_relative "otel/internal/exporters"
@@ -186,13 +190,13 @@ module Foam
         end
       end
 
-      def flush(timeout: 30)
+      def flush(timeout: Internal::Constants::LIFECYCLE_TIMEOUT)
         Internal::Safely.call("flush") do
           Internal::Lifecycle.flush(timeout: timeout)
         end
       end
 
-      def shutdown(timeout: 30)
+      def shutdown(timeout: Internal::Constants::LIFECYCLE_TIMEOUT)
         Internal::Safely.call("shutdown") do
           Internal::Lifecycle.shutdown(timeout: timeout)
         end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: foam-otel
 version: !ruby/object:Gem::Version
-  version: 3.0.0.alpha.3
+  version: 3.0.0.alpha.4
 platform: ruby
 authors:
 - Foam
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-09-14 00:00:00.000000000 Z
+date: 2026-09-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: opentelemetry-exporter-otlp
@@ -160,6 +160,7 @@ files:
 - lib/foam-otel.rb
 - lib/foam/otel.rb
 - lib/foam/otel/internal.rb
+- lib/foam/otel/internal/aws_lambda.rb
 - lib/foam/otel/internal/before_send.rb
 - lib/foam/otel/internal/body_capture.rb
 - lib/foam/otel/internal/body_decoder.rb
@@ -168,11 +169,13 @@ files:
 - lib/foam/otel/internal/clients/faraday.rb
 - lib/foam/otel/internal/clients/http_client.rb
 - lib/foam/otel/internal/clients/net_http.rb
+- lib/foam/otel/internal/clients/read_tee.rb
 - lib/foam/otel/internal/constants.rb
 - lib/foam/otel/internal/content_type.rb
 - lib/foam/otel/internal/diagnostics.rb
 - lib/foam/otel/internal/endpoint.rb
 - lib/foam/otel/internal/exporters.rb
+- lib/foam/otel/internal/fork_hooks.rb
 - lib/foam/otel/internal/ignored_hosts.rb
 - lib/foam/otel/internal/ingest.rb
 - lib/foam/otel/internal/init.rb
@@ -190,6 +193,7 @@ files:
 - lib/foam/otel/internal/report.rb
 - lib/foam/otel/internal/resource.rb
 - lib/foam/otel/internal/safely.rb
+- lib/foam/otel/internal/slots.rb
 - lib/foam/otel/internal/state.rb
 - lib/foam/otel/internal/traces.rb
 - lib/foam/otel/version.rb