foam-otel 2.0.0 → 2.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 767e21dce7e53eb8a74e430c736f2ba94e22c882022733fff7fb8edb058c118e
-  data.tar.gz: 4d9e7b6789d79daec4eb0482a22ac68520263c0a267e1b4e4def7eae97d9e642
+  metadata.gz: 894986849774ca1e6ee37ed18eccb168fe0b026a438abdd53432b46a5de4b9d4
+  data.tar.gz: c59a29ea7223dc7a92500303a2615d03d048fcf70927c12e37250d21a8785f57
 SHA512:
-  metadata.gz: 4512202149fa0ee2b50093b7485b07373a09ed29315b6489d3e59d0c111ecf3d023e77e49f9af271ca97df33210e2028ba3376e4a43516284bd6f0fe195a8627
-  data.tar.gz: 9d15c2c183dfdac3ca20f0a3eccae5c6f2e45134206a697604a1b70e7a9046c775d0b5e805662a162aee2b0d89e62f810cdea8e43a72b9f15db783f1b07ccb0d
+  metadata.gz: 5cddf45218f81f9c18ab2557c7737a53da552bdf429524e6ef1f1cb64fbc84e19757de30d370e6e1217cf2a5217d271b3c4a19b6b8610d36f1846699e96a2cf6
+  data.tar.gz: 2199d01ee1f3cb89ae50d721ad6110c6c803256ac5093ba81dfc070e8cdb3af460ff13d653e6f9dd307f5a5005673e178693ca5a630459e698adc7e29a00b76e

data/GOTCHAS.md

@@ -542,9 +542,16 @@ anthropic 1.59.0, gemini-ai 4.3.0, ruby_llm 1.16.0).
   own HTTP send must not produce a SECOND span under the gen_ai span.
 - **Sources**: installed source — openai `resources/chat/completions.rb:119`
   / `resources/responses.rb:96`, anthropic `resources/messages.rb:68`,
-  gemini-ai `controllers/client.rb:151`, ruby_llm `provider.rb:44` (the
-  patched call sites); opentelemetry-common `utilities.rb:104-117`
-  (`untraced` — the suppression context the http contrib honors). Ruling:
+  gemini-ai `controllers/client.rb:151`, ruby_llm `provider.rb:44`,
+  ruby-openai `lib/openai/client.rb` (`#chat(parameters:)`/`#embeddings` —
+  the COMMUNITY gem sharing the official gem's OpenAI namespace and
+  lib/openai.rb entrypoint: presence is discriminated by the
+  `parameters:`-keyword SHAPE, never the bare method name), http (HTTP.rb)
+  `lib/http/client.rb#request` (the hand-rolled Anthropic transport seam,
+  shape-checked per call); opentelemetry-common `utilities.rb:104-117`
+  (`untraced` — the suppression context net_http/excon honor; contrib
+  `-http` does NOT consult it (G13/UNGUARDABLE) — its transport span NESTS
+  under the gen_ai span, the contract's sanctioned rendering). Ruling:
   LLM provider parity + testing-mandatory (decision log 2026-07-26).
 - **Decision & why**: every patch routes through ONE guarded core
   (`Foam::Otel::LLM.observe`, `lib/foam/otel/llm.rb`): request extraction
@@ -555,10 +562,23 @@ anthropic 1.59.0, gemini-ai 4.3.0, ruby_llm 1.16.0).
   Presence-checked constants + explicit version windows per SDK; emission
   gated on foam owning traces. Wire names are the GenAI semconv strings the
   fleet's js/python gap-fillers emit — content attributes included, RAW.
+  The transport shim (HTTP.rb) additionally must never be the FIRST reader
+  of the app's response: content is parsed only for bounded JSON bodies
+  (Content-Length ≤ its byte cap) and the drained bytes are re-wrapped onto
+  the response so app-side `#to_s`/`#each`/`#readpartial` are untouched;
+  SSE/chunked/oversized bodies skip content capture entirely. HTTP.rb never
+  raises on 4xx/5xx, so `LLM.observe`'s `error_detector:` turns the
+  OBSERVED status into the ERROR span (`error.type` = the bare code string).
 - **Test**: `spec/llm_openai_spec.rb`, `spec/llm_anthropic_spec.rb`,
-  `spec/llm_gemini_spec.rb`, `spec/llm_ruby_llm_spec.rb` — real SDKs against
-  a local fake provider (only the 3rd party mocked), per-provider activity +
-  RAW-content assertions, error-path re-raise, and the fail-to-dark set.
+  `spec/llm_gemini_spec.rb`, `spec/llm_ruby_llm_spec.rb`,
+  `spec/llm_ruby_openai_spec.rb` (isolated subprocess — the two openai gems
+  can never share an interpreter; both cross-gem discriminator negatives
+  pinned here and in `spec/llm_openai_spec.rb`),
+  `spec/llm_http_anthropic_spec.rb` (streaming-reader safety, SSE
+  never-consume, byte-cap, near-miss-path negatives, persistent-client) —
+  real SDKs against a local fake provider (only the 3rd party mocked),
+  per-provider activity + RAW-content assertions, error-path re-raise, and
+  the fail-to-dark set.
 
 ## F12: Session stamping must beat the freeze (spans) and the buffer (logs)
 
@@ -937,7 +957,7 @@ exfiltratable — the value-pattern secret layer is the required second control
 
 ---
 
-## F18: One private-method namespace per class — cross-vendor helper collisions crash the APP, and foam shields them (coexistence mandate 2026-07-29)
+## F18: One private-method namespace per class — cross-vendor helper collisions crash the APP, and foam shields them (coexistence mandate 2026-07-29; HTTP::Client twin added in 2.1.0)
 
 - **Trap**: Every module prepended onto a class shares ONE private-method
   namespace. When two observability vendors prepend same-named private
@@ -949,6 +969,15 @@ exfiltratable — the value-pattern secret layer is the required second control
   `annotate_span_with_response!` (3-arg vs 2-arg). Foam ships that contrib
   patch for capture, so a foam + agent process CONTAINS the collision — and
   foam's promise is coexistence: the app must keep working.
+  THE SAME COLLISION EXISTS ON A SECOND CLASS: dd-trace-rb's httprb patch
+  and the contrib `opentelemetry-instrumentation-http` patch both define
+  `annotate_span_with_response!` on `HTTP::Client` (3-arg vs 2-arg) — the
+  colliding definitions verified at source (below). A
+  `datadog/auto_instrument` +
+  contrib-http process crashes every HTTP.rb request the same way.
+  (The upstream issue numbers the introducing PR cited for this twin could
+  NOT be verified as tracking it and are withdrawn — the colliding source
+  definitions are the primary evidence.)
 - **Sources**:
   - Installed source — the two colliding definitions:
     opentelemetry-instrumentation-net_http 0.29.0
@@ -957,6 +986,12 @@ exfiltratable — the value-pattern secret layer is the required second control
     3-arg spelling of the same private name (verified live in the
     coexistence gate; backtrace pinned in
     docs/decisions/export-isolation-ruby.md §A).
+  - The HTTP::Client twin: dd-trace-rb
+    `lib/datadog/tracing/contrib/httprb/instrumentation.rb`
+    (3-arg `annotate_span_with_response!(span, response, options)`) vs
+    contrib opentelemetry-instrumentation-http
+    `lib/opentelemetry/instrumentation/http/patches/stable/client.rb`
+    (2-arg) — both definitions verified at source.
   - Ruby semantics: prepended modules join the class's ancestor chain and
     private methods resolve through the SAME chain for every caller — there
     is no per-module helper namespace.
@@ -964,7 +999,8 @@ exfiltratable — the value-pattern secret layer is the required second control
   resolves the name through the same shared chain — a rename breaks the
   gem being "fixed"), and foam must never edit another vendor's module. So
   foam owns the collision instead: after the instrumentation sweep, init
-  scans Net::HTTP's prepend chain for private-helper names defined by two
+  scans the prepend chains of Net::HTTP AND `HTTP::Client` (when the http
+  gem is loaded) for private-helper names defined by two
   or more foreign modules at NON-identical positional signatures (names the
   class itself defines are super-chains, never shielded) and prepends ONE
   frontmost dispatch method per colliding name — routing by caller source
@@ -979,17 +1015,27 @@ exfiltratable — the value-pattern secret layer is the required second control
 - **Mitigation**: with the shield installed the coexistence gate's
   patched-client probe went from 3/3 crashes to 3/3 passes under the real
   agent, with both vendors' helpers verifiably executing again.
-- **Residual**: an agent that patches AFTER foam's init lands ahead of the
-  shield and re-exposes the raw collision until a later init/fork re-scan;
-  agents overwhelmingly boot first (initializer/preload) — the order the
-  coexistence gate proves. Identical-signature collisions are left alone
+- **Residual**: the scan is one-shot at init and only shields collisions
+  BOTH of whose definitions are already on the chain — an agent that
+  patches AFTER foam's init lands ahead of the shield and re-exposes the
+  raw collision until a later init/fork re-scan. For Net::HTTP agents
+  overwhelmingly boot first (initializer/preload — the order the
+  coexistence gate proves). The HTTP::Client twin is MORE order-sensitive:
+  `defined?(::HTTP::Client)` must be true at init (the http gem required
+  first — Bundler.require satisfies this) and both vendor patches must
+  already be installed; a `Foam::Otel.init` that runs before
+  `Datadog.configure` leaves that collision unshielded — initialize foam
+  AFTER the vendor agents (the recommended order for every coexistence
+  bridge). Identical-signature collisions are left alone
   (no crash class; dispatch could not disambiguate semantics).
 - **Test**: `spec/prepend_collision_shield_spec.rb` (the unshielded RED
   control, both-vendors-work dispatch, idempotence, no-op purity on
   collision-free chains, super-chain and identical-signature exclusions,
   fall-through to the real error, rule-9 scan degradation, caller-source
-  dispatch across two on-disk gem roots, and the real Net::HTTP + real
-  contrib patch field scenario red→green in a forked child);
+  dispatch across two on-disk gem roots, the real Net::HTTP + real
+  contrib patch field scenario red→green in a forked child, and the real
+  `HTTP::Client` field scenario — 2-arg × 3-arg vendor-shaped collision on
+  the real class, red→green in a forked child);
   `test-apps/ruby-coexistence/run-verify.sh` (the patched-client probe is a
   REQUIRED pass on every vendor posture).
 

data/README.md

@@ -100,7 +100,7 @@ or erase further specific fields (see below).
 > Set the standard contrib lever:
 > `OTEL_RUBY_INSTRUMENTATION_SIDEKIQ_CONFIG_OPTS='propagation_style=child'`.
 
-### The LLM surface — OpenAI, Anthropic, Gemini (and ruby_llm)
+### The LLM surface — OpenAI, Anthropic, Gemini (plus ruby_llm, ruby-openai, and hand-rolled HTTP.rb Anthropic)
 
 The Ruby contrib registry ships no LLM instrumentation, so foam does: thin,
 presence-checked shims over the SDKs' public call sites, emitting the standard
@@ -117,11 +117,22 @@ js/python Gemini paths (**wire change in 1.2.1**: Gemini spans previously
 said `chat`). Covered: the official `openai` SDK
 (chat.completions.create + responses.create), the official `anthropic` SDK
 (messages.create), Gemini via the `gemini-ai` gem (Google ships no official
-Ruby SDK), and `ruby_llm` (one seam covering all its providers). Absent SDK =
+Ruby SDK), `ruby_llm` (one seam covering all its providers), the community
+`ruby-openai` gem (`client.chat` / `client.embeddings` — presence is
+discriminated from the official gem by call shape, since both reopen the
+`OpenAI` namespace; embeddings emit tokens + models, never the input
+documents), and hand-rolled HTTP.rb Anthropic clients (`HTTP.post` /
+`HTTP.persistent(...).post` to `api.anthropic.com/v1/messages` — exactly
+that endpoint; HTTP.rb's non-raising 4xx/5xx become ERROR spans with the
+observed status, and foam never consumes a response stream: SSE, chunked,
+and oversized bodies pass through untouched, and buffered JSON bodies are
+handed back readable). Absent SDK =
 inert; version-guarded; fail-to-dark — a shim failure can NEVER break the
 model call, and the SDK's own error re-raises identically. One model call =
 one `gen_ai` client span (the SDK's HTTP send rides the same suppression
-context foam's exporters use). Streaming call sites (`#stream`/`#stream_raw`
+context foam's exporters use; transport gems that ignore that context, like
+the contrib `http` instrumentation, nest beneath it instead). Streaming
+call sites (`#stream`/`#stream_raw`
 on the official SDKs) are a recorded follow-up.
 
 ### Session stitching — browser `session.id` on backend telemetry

data/RESEARCH.md

@@ -246,8 +246,16 @@ SDK — foam ships its own thin gap-filler shims (LANDED,
 shims patch the SDKs' public call sites — official `openai`
 (chat.completions.create + responses.create), official `anthropic`
 (messages.create), `gemini-ai` (generate_content / stream_generate_content;
-Google ships no official Ruby SDK), and `ruby_llm` (Provider#complete, one
-seam covering all its providers) — emitting the same GenAI semconv attribute
+Google ships no official Ruby SDK), `ruby_llm` (Provider#complete, one
+seam covering all its providers), `ruby-openai` (GRADUATED from tier-3 in
+2.1.0: the long-dominant COMMUNITY OpenAI client, `OpenAI::Client#chat
+(parameters:)`/`#embeddings` — it shares the official gem's `OpenAI`
+namespace and `lib/openai.rb` entrypoint, so the two gems can never load in
+one process and presence is discriminated by the `parameters:`-keyword
+SHAPE, never the bare method name; specs run in an isolated subprocess for
+the same reason, `spec/llm_ruby_openai_spec.rb`), and `http` (HTTP.rb —
+the hand-rolled Anthropic transport seam, `HTTP::Client#request`, below) —
+emitting the same GenAI semconv attribute
 set the fleet's js/python gap-fillers emit. `gen_ai.operation.name` matches
 the fleet per API shape: `chat` on the chat seams, `generate_content` on the
 Gemini shim (aligned in 1.2.1 — Ruby previously emitted `chat` for Gemini, a
@@ -267,7 +275,11 @@ the model call is never breakable by telemetry (GOTCHAS F11). LLM testing is
 MANDATORY and landed: per-provider specs against the REAL SDKs with only the
 3rd party mocked (`spec/llm_openai_spec.rb`, `spec/llm_anthropic_spec.rb`,
 `spec/llm_gemini_spec.rb`, `spec/llm_ruby_llm_spec.rb` — the last is the
-wire-level Gemini proof over a local fake provider).
+wire-level Gemini proof over a local fake provider —
+`spec/llm_ruby_openai_spec.rb`, and `spec/llm_http_anthropic_spec.rb`,
+which additionally pins the transport shim's never-first-reader guarantee:
+streaming reads survive capture, SSE and oversized bodies are never
+consumed, and the observed-status error path).
 
 - `-anthropic` (in the registry / `all` meta-gem) is **context-propagation ONLY**
   — its sole patch stores/restores OTel context across the SDK's connection pool;
@@ -278,12 +290,26 @@ wire-level Gemini proof over a local fake provider).
   but is NOT bundled: foam's own `ruby_llm` shim emits the fleet-identical
   attribute set instead, avoiding a non-registry supply-chain dependency and
   a cross-language semconv divergence.
-- `ruby-openai` (community gem), `langchainrb`, AWS Bedrock (via `-aws_sdk`,
+- `http` (HTTP.rb) hand-rolled Anthropic clients: **TRANSPORT-LEVEL SHIM,
+  DELIBERATELY BOUNDED (2.1.0).** SDK-less apps that POST
+  `api.anthropic.com/v1/messages` directly (a live customer pattern) get the
+  full gen_ai set via a per-call shape check — known host + EXACT
+  distinctive path + POST (contract SPEC §10's transport-hook rules;
+  near-miss endpoints like `/v1/messages/count_tokens` and
+  `/v1/messages/batches` are pinned NEGATIVE, never fabricated). Scope is a
+  closed set — ONE provider, ONE transport: foam-otel 0.1.x shipped broad
+  transport classifiers and RETIRED them (the ecosystem churns too fast to
+  chase from a shared gem), and that ruling stands; extending this seam to
+  more hosts/transports (Net::HTTP, Faraday hand-rolls) is a per-addition
+  fleet ruling with this same census entry updated, never a silent patch.
+  Embeddings note (fleet parity): NO shim captures embeddings INPUT —
+  python's contrib openai-v2 path emits none; tokens + models only.
+- `langchainrb`, AWS Bedrock (via `-aws_sdk`,
   which emits no gen_ai attributes), `llm.rb`: **TIER3-recipe** —
   hand-written spans via helpers + passthroughs with `gen_ai.*` semconv + token
   counts + trace/log correlation. An uncovered library the FDE hits becomes a
   filed package need (the FDE activates, never authors); until then the tier-3
-  recipe covers it.
+  recipe covers it — the path `ruby-openai` itself took to graduation.
 - No mainstream library is CANNOT-COVER — no finding against the helper set.
 
 Sources: contrib `-anthropic` CHANGELOG; https://github.com/thoughtbot/opentelemetry-instrumentation-ruby_llm ;

data/lib/foam/otel/init.rb

@@ -620,6 +620,12 @@ module Foam
         # vendor private-helper collisions and shim them. A collision-free
         # chain (the common case) installs NOTHING.
         PrependCollisionShield.install!(::Net::HTTP) if defined?(::Net::HTTP)
+        # The HTTP.rb ("http" gem) twin of the same collision: dd-trace-rb's
+        # httprb patch and the contrib http patch both define the private
+        # helper `annotate_span_with_response!` on HTTP::Client at
+        # non-identical signatures (3-arg vs 2-arg) — without the shield,
+        # EVERY HTTP.rb request crashes once both agents are loaded.
+        PrependCollisionShield.install!(::HTTP::Client) if defined?(::HTTP::Client)
       rescue StandardError => e
         Diagnostics.warn("instrumentation activation failed: #{e.class}: #{e.message}")
       end

data/lib/foam/otel/llm/http_anthropic_shim.rb

@@ -0,0 +1,318 @@
+# frozen_string_literal: true
+
+require "stringio"
+
+module Foam
+  module Otel
+    module LLM
+      # Gap-filler for Anthropic API calls made DIRECTLY over the `http` gem
+      # (HTTP.rb) — the hand-rolled-client pattern (HTTP.headers(...).post(
+      # "https://api.anthropic.com/v1/messages", json: body)) that apps use
+      # instead of an SDK. Presence-checked on HTTP.rb and shape-checked per
+      # call (host + EXACT path + POST), so it is inert for every other
+      # HTTP.rb request in the process and in any app without the gem. The
+      # host may come from the request uri or, for the idiomatic persistent
+      # client (HTTP.persistent(origin).post("/v1/messages", ...)), from the
+      # client's persistent origin.
+      #
+      # ONE gen_ai span per logical call: the request runs inside
+      # LLM.observe's suppression context. NOTE the contrib http
+      # instrumentation does NOT consult the untraced context (GOTCHAS G13,
+      # init.rb's UNGUARDABLE_HTTP_CLIENT_INSTRUMENTATIONS) — when that gem
+      # is active its transport span NESTS beneath the gen_ai span instead
+      # of being suppressed, the contract's sanctioned rendering (SPEC §10:
+      # a different layer, like rails-inside-rack).
+      #
+      # The app's response stream is NEVER taken away from it: response
+      # content is read only when the body is bounded (Content-Length ≤ the
+      # parse cap) AND can be handed back — the drained bytes are re-wrapped
+      # onto the response so #to_s, #each and #readpartial all still work.
+      # Anything else (SSE/event-stream, chunked, oversized, restore-probe
+      # failure) skips content capture: activity + observed status survive,
+      # the stream is untouched. Error responses carry their OBSERVED status
+      # (rule 25); HTTP.rb never raises on 4xx/5xx, so the status IS the
+      # error signal — success spans carry NO http.* (fleet gen_ai
+      # vocabulary, llm.rb).
+      module HttpAnthropicShim
+        SUPPORTED_BELOW = Gem::Version.new("6.0.0")
+        HOSTS = ["api.anthropic.com"].freeze
+        MESSAGES_PATH = "/v1/messages"
+        # Transport twin of PayloadCapture's byte cap: bodies above this are
+        # never materialized or parsed by the shim (rule 49 — bounded work
+        # on the caller thread). Content is omitted, activity survives.
+        BODY_PARSE_BYTE_CAP = 32_768
+
+        class << self
+          def present?
+            defined?(::HTTP::Client) &&
+              ::HTTP::Client.method_defined?(:request) &&
+              supported_version?
+          end
+
+          def install!
+            return true if @installed
+
+            ::HTTP::Client.prepend(RequestPatch)
+            Diagnostics.info("LLM shim installed: http-rb anthropic (POST #{HOSTS.join(', ')}#{MESSAGES_PATH})")
+            @installed = true
+          end
+
+          def installed? = @installed
+
+          def supported_version?
+            spec = Gem.loaded_specs["http"]
+            return true if spec.nil?
+
+            supported = spec.version < SUPPORTED_BELOW
+            unless supported || @version_warned
+              @version_warned = true
+              Diagnostics.warn("http #{spec.version} is outside foam's LLM-shim window (< #{SUPPORTED_BELOW}) " \
+                               "— the HTTP.rb Anthropic shim stays dark; file this with foam support")
+            end
+            supported
+          rescue StandardError
+            true
+          end
+
+          # Shape check: is this ONE call an Anthropic Messages API request?
+          # uri arrives as whatever the caller passed (String / URI /
+          # HTTP::URI — possibly a bare path on a persistent client, whose
+          # origin then supplies the host). The path must match EXACTLY:
+          # /v1/messages/count_tokens, /v1/messages/batches and friends are
+          # real non-inference Anthropic endpoints under the same prefix,
+          # and a prefix match would fabricate chat spans for them
+          # (contract SPEC §10: an ordinary REST path must never fabricate
+          # a gen_ai span).
+          def anthropic_messages?(verb, uri, client = nil)
+            return false unless verb.to_s.downcase == "post"
+
+            parsed = uri.is_a?(String) ? URI.parse(uri) : uri
+            host = parsed.respond_to?(:host) ? parsed.host : nil
+            path = parsed.respond_to?(:path) ? parsed.path : nil
+            host ||= persistent_host(client)
+            HOSTS.include?(host) && path.to_s.chomp("/") == MESSAGES_PATH
+          rescue StandardError
+            false
+          end
+
+          # The persistent-client idiom passes a bare path; the origin lives
+          # on the client's default options (client.rb joins them only
+          # inside #request, after this check runs).
+          def persistent_host(client)
+            opts = client.respond_to?(:default_options) ? client.default_options : nil
+            base = opts.respond_to?(:persistent) ? opts.persistent : nil
+            base ? URI.parse(base.to_s).host : nil
+          rescue StandardError
+            nil
+          end
+
+          # The request body: the patch sees opts BEFORE Client#request
+          # normalizes it — usually a plain Hash ({json: {...}} from the
+          # chainable), sometimes an HTTP::Options (structured :json
+          # pre-encoding) — or an encoded string body, parsed best-effort
+          # under the byte cap. NEVER consumes streams (file/IO bodies
+          # degrade to the bare operation/provider pair).
+          def request_body(opts)
+            json, body =
+              if opts.is_a?(Hash)
+                [opts[:json] || opts["json"], opts[:body] || opts["body"]]
+              else
+                [opts.respond_to?(:json) ? opts.json : nil,
+                 opts.respond_to?(:body) ? opts.body : nil]
+              end
+            return json if json.is_a?(Hash)
+            return nil unless body.is_a?(String) && body.bytesize <= BODY_PARSE_BYTE_CAP && body.start_with?("{")
+
+            JSON.parse(body)
+          rescue StandardError
+            nil
+          end
+
+          def request_attributes(body)
+            body = body.is_a?(Hash) ? body : {}
+            LLM.request_attributes(
+              provider: "anthropic", operation: "chat",
+              model: body["model"] || body[:model],
+              max_tokens: body["max_tokens"] || body[:max_tokens],
+              input_messages: body["messages"] || body[:messages],
+              system_instructions: body["system"] || body[:system]
+            )
+          end
+
+          # Reads the finished JSON body for capture WITHOUT taking anything
+          # away from the app: only when the body is bounded (declared
+          # Content-Length within the cap, no Content-Encoding — a gzip
+          # body's declared length is the COMPRESSED size, and the inflated
+          # stream could dwarf the cap) and a replacement Body can be
+          # built (probed BEFORE consuming — if HTTP.rb's internals ever
+          # drift, foam skips capture rather than risk the app's stream).
+          # The drained bytes are re-wrapped in a fresh HTTP::Response::Body
+          # over an EOF-compatible buffer and swapped back, so app-side
+          # #to_s, #each and #readpartial behave exactly as if foam had
+          # never read the response. A transport failure MID-READ is
+          # re-armed: the swapped-in body re-raises the ORIGINAL exception
+          # on the app's own read — never a misleading StateError. The read
+          # itself inherits the app's HTTP.rb timeout configuration (bytes
+          # are capped here; deadlines belong to the client's .timeout).
+          def readable_json_body(response)
+            length = content_length(response)
+            return nil unless length&.positive? && length <= BODY_PARSE_BYTE_CAP
+            return nil if content_encoded?(response)
+            return nil unless response.instance_variable_defined?(:@body)
+            return nil unless build_body("") # constructor probe, pre-consumption
+
+            contents = begin
+              response.body.to_s
+            rescue StandardError, SystemStackError => e
+              rearm_failed_body(response, e)
+              return nil
+            end
+            replacement = build_body(contents) ||
+                          build_body(contents.dup.force_encoding(Encoding::BINARY))
+            # No hand-back, no capture: foam never profits from a stream it
+            # failed to return to the app.
+            return nil unless replacement
+
+            response.instance_variable_set(:@body, replacement)
+            # Belt over the declared length: never parse more than the cap
+            # actually materialized (a lying Content-Length header).
+            contents.bytesize <= BODY_PARSE_BYTE_CAP ? contents : nil
+          rescue StandardError
+            nil
+          end
+
+          def content_length(response)
+            raw = response.respond_to?(:headers) ? response.headers["Content-Length"] : nil
+            raw.nil? ? nil : Integer(raw.to_s, 10)
+          rescue StandardError
+            nil
+          end
+
+          def content_encoded?(response)
+            return false unless response.respond_to?(:headers)
+
+            encoding = response.headers["Content-Encoding"].to_s
+            !encoding.empty? && encoding.downcase != "identity"
+          rescue StandardError
+            true # unknown => assume encoded, skip capture
+          end
+
+          def build_body(contents)
+            ::HTTP::Response::Body.new(BufferedBodyStream.new(contents), encoding: contents.encoding)
+          rescue StandardError
+            nil
+          end
+
+          # A read that died under foam must die IDENTICALLY under the app:
+          # swap in a body whose reads re-raise the original transport error.
+          def rearm_failed_body(response, error)
+            replacement = ::HTTP::Response::Body.new(FailingBodyStream.new(error), encoding: Encoding::BINARY)
+            response.instance_variable_set(:@body, replacement)
+          rescue StandardError
+            nil
+          end
+        end
+
+        # Matches HTTP::Connection's readpartial contract (nil at EOF, never
+        # EOFError) so a restored Body streams exactly like a live one.
+        class BufferedBodyStream
+          CHUNK = 16_384
+
+          def initialize(contents)
+            @io = StringIO.new(contents)
+          end
+
+          def readpartial(size = CHUNK, outbuf = nil)
+            return nil if @io.eof?
+
+            outbuf ? @io.readpartial(size, outbuf) : @io.readpartial(size)
+          end
+        end
+
+        # Re-raises the transport error foam swallowed mid-capture, so the
+        # app's own read fails with the REAL exception class and message.
+        class FailingBodyStream
+          def initialize(error)
+            @error = error
+          end
+
+          def readpartial(*_args)
+            raise @error
+          end
+        end
+
+        # Non-streaming JSON Message → response attributes; usage rides
+        # {input_tokens, output_tokens} exactly like the SDK shim.
+        RESPONSE = lambda do |response|
+          h = response.is_a?(Hash) ? response : {}
+          usage = h["usage"].is_a?(Hash) ? h["usage"] : {}
+          LLM.response_attributes(
+            model: h["model"],
+            id: h["id"],
+            finish_reasons: h["stop_reason"] ? [h["stop_reason"]] : nil,
+            input_tokens: usage["input_tokens"],
+            output_tokens: usage["output_tokens"],
+            output_messages: h["content"]
+          )
+        end
+
+        module RequestPatch
+          def request(verb, uri, opts = {})
+            unless Foam::Otel::LLM.emit? && HttpAnthropicShim.anthropic_messages?(verb, uri, self)
+              return super
+            end
+
+            body = HttpAnthropicShim.request_body(opts)
+            Foam::Otel::LLM.observe(
+              provider: "anthropic",
+              request: Foam::Otel::LLM.safe_request(provider: "anthropic") do
+                HttpAnthropicShim.request_attributes(body)
+              end,
+              error_detector: HttpAnthropicShim::ERROR_DETECTOR,
+              response_extractor: HttpAnthropicShim::EXTRACTOR
+            ) { super }
+          end
+        end
+
+        # Observed-status error detection: HTTP.rb never raises on 4xx/5xx —
+        # the status IS the error signal (rule 25: observed, never fabricated).
+        ERROR_DETECTOR = lambda do |response|
+          status = response.respond_to?(:status) ? response.status.to_i : 0
+          status >= 400 ? status : nil
+        end
+
+        # Response attributes. The status code is stamped ONLY on the error
+        # path (the observed error signal — fleet gen_ai spans carry no
+        # http.* on success, the posture python's gap-filler also converged
+        # on). error.type carries the bare
+        # status string ("400"), matching the span status description and
+        # semconv's well-known values — never an invented format.
+        EXTRACTOR = lambda do |response|
+          attrs = {}
+          status = response.respond_to?(:status) ? response.status.to_i : 0
+          if status >= 400
+            attrs[LLM::HTTP_RESPONSE_STATUS_CODE] = status
+            attrs[LLM::ERROR_TYPE] = status.to_s
+          end
+          # HTTP::ContentType#to_s is the INSPECT string, not the media
+          # type — read #mime_type (via Response#mime_type), falling back
+          # to the raw header.
+          mime = response.respond_to?(:mime_type) ? response.mime_type.to_s : ""
+          if mime.empty? && response.respond_to?(:headers)
+            mime = response.headers["Content-Type"].to_s
+          end
+          if mime.include?("json")
+            begin
+              raw = HttpAnthropicShim.readable_json_body(response)
+              parsed = raw ? JSON.parse(raw) : nil
+              attrs.merge!(RESPONSE.call(parsed)) if parsed.is_a?(Hash)
+            rescue StandardError
+              nil # unparseable body: activity + status survive, content lost
+            end
+          end
+          attrs
+        end
+      end
+    end
+  end
+end

data/lib/foam/otel/llm/ruby_openai_shim.rb

@@ -0,0 +1,138 @@
+# frozen_string_literal: true
+
+module Foam
+  module Otel
+    module LLM
+      # Gap-filler for the `ruby-openai` gem (the long-standing community
+      # OpenAI client: `OpenAI::Client#chat`, NOT the official `openai`
+      # gem's resource layer OpenAIShim covers). Plenty of production Rails
+      # apps still ride it — presence-checked and version-windowed like its
+      # siblings so it is inert anywhere the gem is absent, and dark outside
+      # the supported window.
+      module RubyOpenAIShim
+        SUPPORTED_BELOW = Gem::Version.new("9.0.0")
+
+        class << self
+          # The official `openai` gem ALSO defines OpenAI::Client#chat — but
+          # as a zero-argument RESOURCE ACCESSOR (client.chat.completions…).
+          # ruby-openai's is the request method chat(parameters:). The shape
+          # check (the parameters: keyword) is the ONLY reliable discriminator
+          # between the two gems sharing this namespace — never the bare
+          # method name.
+          def present?
+            return false unless defined?(::OpenAI::Client) && ::OpenAI::Client.method_defined?(:chat)
+
+            chat = ::OpenAI::Client.instance_method(:chat)
+            takes_parameters = chat.parameters.any? { |type, name| %i[key keyreq].include?(type) && name == :parameters }
+            takes_parameters && supported_version?
+          rescue StandardError
+            false
+          end
+
+          def install!
+            return true if @installed
+
+            ::OpenAI::Client.prepend(ChatPatch)
+            if ::OpenAI::Client.method_defined?(:embeddings)
+              ::OpenAI::Client.prepend(EmbeddingsPatch)
+            end
+            Diagnostics.info("LLM shim installed: ruby-openai (client.chat, client.embeddings)")
+            @installed = true
+          end
+
+          def installed? = @installed
+
+          def supported_version?
+            spec = Gem.loaded_specs["ruby-openai"]
+            return true if spec.nil? # vendored/unknown: the shape checks decide
+
+            supported = spec.version < SUPPORTED_BELOW
+            unless supported || @version_warned
+              @version_warned = true
+              Diagnostics.warn("ruby-openai #{spec.version} is outside foam's LLM-shim window (< #{SUPPORTED_BELOW}) " \
+                               "— the ruby-openai shim stays dark; file this with foam support")
+            end
+            supported
+          rescue StandardError
+            true
+          end
+
+          def chat_request(params)
+            params = params.is_a?(Hash) ? params : {}
+            LLM.request_attributes(
+              provider: "openai", operation: "chat",
+              model: params[:model] || params["model"],
+              max_tokens: params[:max_tokens] || params["max_tokens"] ||
+                          params[:max_completion_tokens] || params["max_completion_tokens"],
+              input_messages: params[:messages] || params["messages"]
+            )
+          end
+
+          def embeddings_request(params)
+            params = params.is_a?(Hash) ? params : {}
+            # Embeddings INPUT is deliberately NOT captured (fleet parity:
+            # the python core rides contrib openai-v2, which emits no
+            # embeddings input under any attribute — document chunks are a
+            # classic PII carrier, and gen_ai.input.messages is defined for
+            # chat messages, not document batches). Tokens + models carry
+            # the signal; widening this is a fleet ruling, not a shim choice.
+            LLM.request_attributes(
+              provider: "openai", operation: "embeddings",
+              model: params[:model] || params["model"]
+            )
+          end
+        end
+
+        # ruby-openai returns plain parsed-JSON Hashes (string keys):
+        # {"id"=>..., "model"=>..., "choices"=>[{"message"=>{...},
+        # "finish_reason"=>...}], "usage"=>{"prompt_tokens"=>N, ...}}.
+        CHAT_RESPONSE = lambda do |response|
+          h = response.is_a?(Hash) ? response : {}
+          usage = h["usage"].is_a?(Hash) ? h["usage"] : {}
+          choices = h["choices"].is_a?(Array) ? h["choices"] : []
+          LLM.response_attributes(
+            model: h["model"],
+            id: h["id"],
+            finish_reasons: choices.map { |c| c.is_a?(Hash) ? c["finish_reason"] : nil }.compact,
+            input_tokens: usage["prompt_tokens"],
+            output_tokens: usage["completion_tokens"],
+            output_messages: choices.map { |c| c.is_a?(Hash) ? c["message"] : nil }.compact
+          )
+        end
+
+        # Embeddings response: usage.prompt_tokens only; the vectors are
+        # NOT content (and would be enormous) — count them, never ship them.
+        EMBEDDINGS_RESPONSE = lambda do |response|
+          h = response.is_a?(Hash) ? response : {}
+          usage = h["usage"].is_a?(Hash) ? h["usage"] : {}
+          LLM.response_attributes(
+            model: h["model"],
+            input_tokens: usage["prompt_tokens"]
+          )
+        end
+
+        module ChatPatch
+          def chat(parameters: {})
+            Foam::Otel::LLM.observe(
+              provider: "openai",
+              request: Foam::Otel::LLM.safe_request(provider: "openai") { RubyOpenAIShim.chat_request(parameters) },
+              response_extractor: RubyOpenAIShim::CHAT_RESPONSE
+            ) { super }
+          end
+        end
+
+        module EmbeddingsPatch
+          def embeddings(parameters: {})
+            Foam::Otel::LLM.observe(
+              provider: "openai", operation: "embeddings",
+              request: Foam::Otel::LLM.safe_request(provider: "openai", operation: "embeddings") do
+                RubyOpenAIShim.embeddings_request(parameters)
+              end,
+              response_extractor: RubyOpenAIShim::EMBEDDINGS_RESPONSE
+            ) { super }
+          end
+        end
+      end
+    end
+  end
+end

data/lib/foam/otel/llm.rb

@@ -13,13 +13,22 @@ module Foam
     # `_openai_responses.py` and js/otel's anthropic gap-filler — over the
     # SDKs' public call sites:
     #
-    #   * `openai`     (official OpenAI Ruby SDK)  — chat.completions.create +
-    #                                                responses.create
-    #   * `anthropic`  (official Anthropic Ruby SDK) — messages.create
-    #   * `gemini-ai`  (the de-facto Gemini Ruby SDK; Google ships no official
-    #                   Ruby SDK) — generate_content / stream_generate_content
-    #   * `ruby_llm`   (multi-provider client) — Provider#complete, one seam
-    #                   covering its OpenAI/Anthropic/Gemini/... providers
+    #   * `openai`      (official OpenAI Ruby SDK)  — chat.completions.create +
+    #                                                 responses.create
+    #   * `anthropic`   (official Anthropic Ruby SDK) — messages.create
+    #   * `gemini-ai`   (the de-facto Gemini Ruby SDK; Google ships no official
+    #                    Ruby SDK) — generate_content / stream_generate_content
+    #   * `ruby_llm`    (multi-provider client) — Provider#complete, one seam
+    #                    covering its OpenAI/Anthropic/Gemini/... providers
+    #   * `ruby-openai` (the long-standing COMMUNITY OpenAI client —
+    #                    OpenAI::Client#chat(parameters:) / #embeddings; shares
+    #                    the OpenAI namespace with the official gem, so presence
+    #                    is checked by the parameters:-keyword SHAPE, never the
+    #                    bare method name) — chat + embeddings
+    #   * `http`        (HTTP.rb) — hand-rolled-provider-client pattern: POSTs
+    #                    to api.anthropic.com/v1/messages from SDK-less apps;
+    #                    shape-checked per call so every other HTTP.rb request
+    #                    in the process keeps its plain transport span
     #
     # Contract invariants (identical to the python/js gap-fillers):
     #   * Content is CAPTURED, then redacted BEFORE serialization (coverage
@@ -33,12 +42,15 @@ module Foam
     #     fail-to-dark; the wrapper NEVER throws into app code or fails the
     #     customer's model call — the SDK call's own error re-raises
     #     IDENTICALLY, telemetry failures degrade to an unspanned call.
-    #   * ONE logical LLM call = ONE foam client span: the SDK's own HTTP send
-    #     runs inside the OTel suppression context the official http
-    #     instrumentations honor (Common::Utilities.untraced — the same
-    #     mechanism the exporters use, GOTCHAS G4), so the transport is never
-    #     double-spanned. Feature-detected; absent → the call runs in the
-    #     gen_ai span's context and the transport span parents under it.
+    #   * ONE logical LLM call = ONE foam gen_ai span: the SDK's own HTTP
+    #     send runs inside the OTel suppression context (Common::Utilities
+    #     .untraced — the same mechanism the exporters use, GOTCHAS G4).
+    #     Instrumentations that honor it (net_http, excon) emit no transport
+    #     span at all; those that DON'T (contrib -http — GOTCHAS G13, the
+    #     UNGUARDABLE list) emit a transport span that NESTS beneath the
+    #     gen_ai span — the contract's sanctioned rendering (SPEC §10: a
+    #     different layer, like rails-inside-rack). Either way exactly one
+    #     gen_ai span. Feature-detected; Utilities absent → nesting.
     #   * Wire vocabulary (rule 26): current GenAI semconv exactly as the
     #     fleet's other packages emit it — span name "{operation} {model}",
     #     gen_ai.operation.name / gen_ai.provider.name / request+response
@@ -69,7 +81,7 @@ module Foam
 
       class << self
         def adapters
-          [OpenAIShim, AnthropicShim, GeminiShim, RubyLLMShim]
+          [OpenAIShim, AnthropicShim, GeminiShim, RubyLLMShim, RubyOpenAIShim, HttpAnthropicShim]
         end
 
         # Presence-checked, fault-isolated activation — one broken adapter
@@ -118,7 +130,7 @@ module Foam
         # value passes through untouched, its error re-raises identically.
         # Every telemetry step is individually guarded — a shim failure can
         # only ever mean a missing span, never a broken model call.
-        def observe(provider:, request:, operation: "chat", response_extractor: nil, &block)
+        def observe(provider:, request:, operation: "chat", response_extractor: nil, error_detector: nil, &block)
           return yield unless emit?
 
           span = start_span(provider, operation, request)
@@ -132,6 +144,13 @@ module Foam
             raise
           end
           begin
+            # Non-raising transports (HTTP.rb: a 4xx/5xx is a RETURN VALUE,
+            # not an exception) surface their error through the detector —
+            # the observed status becomes the span status (rule 25).
+            detected = error_detector&.call(result)
+            if detected
+              span.status = OpenTelemetry::Trace::Status.error(detected.to_s)
+            end
             response = response_extractor&.call(result)
             apply_attributes(span, response) if response
           rescue StandardError, SystemStackError
@@ -237,7 +256,10 @@ module Foam
 
         def start_span(provider, operation, request)
           model = request[GEN_AI_REQUEST_MODEL]
-          name = model ? "#{operation} #{model}" : operation
+          # Contract SPEC §10 span-name shape "<operation> <model|provider>":
+          # when the model is unextractable the provider stands in — never a
+          # bare operation name.
+          name = "#{operation} #{model || provider}"
           OpenTelemetry.tracer_provider.tracer(TRACER_NAME).start_span(
             name, kind: :client, attributes: request
           )
@@ -276,8 +298,13 @@ module Foam
           span.status = OpenTelemetry::Trace::Status.error(error.class.to_s)
           apply_attributes(span, ERROR_TYPE => error.class.to_s)
           # The SDKs' API errors carry the REAL transport status — emitted
-          # only when OBSERVED (rule 25: never fabricate).
-          status = error.respond_to?(:status) ? error.status : nil
+          # only when OBSERVED (rule 25: never fabricate). Faraday-backed
+          # SDKs (ruby-openai) expose it as #response_status.
+          status = if error.respond_to?(:status)
+                     error.status
+                   elsif error.respond_to?(:response_status)
+                     error.response_status
+                   end
           apply_attributes(span, HTTP_RESPONSE_STATUS_CODE => status) if status.is_a?(Integer)
           span.record_exception(error) if error.is_a?(Exception)
         rescue StandardError, SystemStackError
@@ -298,3 +325,5 @@ require_relative "llm/openai_shim"
 require_relative "llm/anthropic_shim"
 require_relative "llm/gemini_shim"
 require_relative "llm/ruby_llm_shim"
+require_relative "llm/ruby_openai_shim"
+require_relative "llm/http_anthropic_shim"

data/lib/foam/otel/version.rb

@@ -194,6 +194,80 @@ module Foam
     #   a combination never tested. MAJOR: narrowing the supported-runtime
     #   label is breaking for installs on the dropped versions. No wire/API
     #   change — every 1.x contract above still holds.
-    VERSION = "2.0.0"
+    # 2.1.0: LLM GAP-FILLER SHIMS + HTTP::Client COEXISTENCE SHIELD (MINOR —
+    #   additive instrumentations, zero default-path behavior change for apps
+    #   without the gems). Two new presence-checked, version-windowed LLM
+    #   adapters closing the SDK-less-call gap: RubyOpenAIShim for the
+    #   ruby-openai community client (OpenAI::Client#chat(parameters:) /
+    #   #embeddings — presence by the parameters:-keyword SHAPE, never the
+    #   bare method name: the official openai gem shares the OpenAI
+    #   namespace and defines Client#chat as a zero-arg resource accessor,
+    #   so a name check misfires on official-gem apps; both cross-gem
+    #   negatives are pinned. Embeddings emit tokens + models ONLY — input
+    #   documents are never captured, matching the python core's contrib
+    #   openai-v2 path), and
+    #   HttpAnthropicShim for hand-rolled HTTP.rb Anthropic clients (POST
+    #   api.anthropic.com/v1/messages — EXACT path match; near-miss
+    #   endpoints such as /v1/messages/count_tokens and
+    #   /v1/messages/batches are pinned negative, never fabricated —
+    #   including the persistent-client idiom
+    #   HTTP.persistent(origin).post("/v1/messages", ...)), shape-checked
+    #   per call so every other HTTP.rb request keeps only its plain
+    #   transport span.
+    #   Both inherit the LLM contract: RAW content capture redacted before
+    #   serialization, one gen_ai span per logical call (transport spans
+    #   from untraced-blind gems nest beneath — GOTCHAS G13),
+    #   fail-to-dark everywhere. The transport shim is never the FIRST
+    #   reader of the app's response: bounded JSON bodies
+    #   (Content-Length ≤ 32 KiB) are captured and handed back re-readable
+    #   (to_s/each/readpartial all intact); SSE (text/event-stream),
+    #   chunked, and oversized bodies are never materialized (rule 49 cap;
+    #   content omitted, activity + observed status survive). LLM.observe
+    #   gains an optional error_detector: hook — HTTP.rb never raises on
+    #   4xx/5xx, so the shim turns the OBSERVED status into an ERROR span
+    #   (rule 25); the status code rides http.response.status_code on the
+    #   ERROR path ONLY (success gen_ai spans stay http.*-free, the fleet
+    #   vocabulary) and error.type is the bare status string ("400").
+    #   Degraded gen_ai span names now fall back to
+    #   "<operation> <provider>" (contract SPEC §10 shape) instead of a
+    #   bare operation, and record_failure reads Faraday-style
+    #   #response_status so ruby-openai provider errors carry their
+    #   observed status. ALSO
+    #   (coexistence mandate, GOTCHAS F18 twin): the prepend-collision shield
+    #   now also scans HTTP::Client — dd-trace-rb's httprb patch and the
+    #   contrib http patch define annotate_span_with_response! at
+    #   non-identical signatures (3-arg vs 2-arg), the exact F18 Net::HTTP
+    #   collision on a second class, live in the wild under
+    #   datadog/auto_instrument (both colliding definitions verified at
+    #   source — GOTCHAS F18); the
+    #   signature-dispatch shim lets both agents keep working WHEN both
+    #   vendor patches precede foam's init-time scan (F18 residual:
+    #   initialize foam after the vendor agents; a later-arriving patch is
+    #   unshielded until a re-init/fork re-scan).
+    # 2.1.1: LLM CAPTURE HARDENING + RESPONSE-STYLE MATRICES (PATCH — bug
+    #   fixes and tests only, no new surface). HttpAnthropicShim's body
+    #   capture: content-encoded (gzip) responses are never captured (the
+    #   declared Content-Length is the COMPRESSED size — the 32 KiB cap
+    #   could be bypassed by inflation; a post-materialization bytesize
+    #   guard also covers lying Content-Length); a transport failure DURING
+    #   capture now re-raises the ORIGINAL error on the app's own read
+    #   (was: a misleading HTTP::StateError); if the body hand-back cannot
+    #   be rebuilt, capture is skipped entirely. Degraded ruby-openai
+    #   embeddings spans keep gen_ai.operation.name="embeddings" (the
+    #   fallback stamped "chat" on a span named "embeddings openai").
+    #   GOTCHAS F18 citation hygiene: the three upstream issue links cited
+    #   for the HTTP::Client collision could not be verified and are
+    #   withdrawn; the colliding source definitions are the evidence.
+    #   Tests: full response-style matrices for both providers — every
+    #   documented Anthropic /v1/messages shape (parallel tool_use,
+    #   thinking/redacted_thinking, all stop_reasons incl.
+    #   model_context_window_exceeded, citation splitting, server-tool
+    #   round trips, error envelopes 400-529, non-object/invalid JSON) and
+    #   every documented OpenAI chat/embeddings shape (all finish_reasons
+    #   + unknown pass-through, parallel/custom/legacy tool calls,
+    #   refusal, n>1 choices, usage-detail variants, Faraday error
+    #   classes with observed status, base64 embeddings) — plus gzip
+    #   skip-capture, mid-read re-arm, and restored-body #each pins.
+    VERSION = "2.1.1"
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: foam-otel
 version: !ruby/object:Gem::Version
-  version: 2.0.0
+  version: 2.1.1
 platform: ruby
 authors:
 - Foam
@@ -395,8 +395,10 @@ files:
 - lib/foam/otel/llm.rb
 - lib/foam/otel/llm/anthropic_shim.rb
 - lib/foam/otel/llm/gemini_shim.rb
+- lib/foam/otel/llm/http_anthropic_shim.rb
 - lib/foam/otel/llm/openai_shim.rb
 - lib/foam/otel/llm/ruby_llm_shim.rb
+- lib/foam/otel/llm/ruby_openai_shim.rb
 - lib/foam/otel/logger_bridge.rb
 - lib/foam/otel/metrics.rb
 - lib/foam/otel/payload_capture.rb