foam-otel 1.0.2 → 1.2.0

data/GOTCHAS.md

@@ -8,7 +8,10 @@ those tests.
 
 Installed-source refs are rooted at the lockfile-pinned gems (opentelemetry-sdk
 1.13.0, -logs-sdk 0.6.1, -metrics-sdk 0.15.0, -api 1.11.0, -exporter-otlp 0.34.1,
--instrumentation-rack 0.31.1, -instrumentation-rails 0.42.0).
+-instrumentation-rack 0.31.1, -instrumentation-rails 0.42.0, and the floor
+contrib pins: -net_http 0.29.0, -faraday 0.33.0, -pg 0.36.0, -mysql2 0.34.0,
+-redis 0.29.0, -mongo 0.26.0, -sidekiq 0.29.0; LLM shim refs at openai 0.72.0,
+anthropic 1.59.0, gemini-ai 4.3.0, ruby_llm 1.16.0).
 
 ---
 
@@ -36,13 +39,21 @@ Installed-source refs are rooted at the lockfile-pinned gems (opentelemetry-sdk
   span/log in the child — foam relies on upstream and never reimplements it. The
   PeriodicMetricReader does not, and foam bypasses `SDK.configure`, so foam
   installs its OWN `Process._fork` hook to restart foam's readers in the child.
-- **Mitigation**: `lib/foam/otel/fork_hooks.rb` prepends `Process._fork` and
-  calls `Foam::Otel.after_fork!` in the child (restarts tracked metric readers,
-  `lib/foam/otel/api.rb`). Also exposed for `Puma on_worker_boot` /
-  `Unicorn after_fork`. The README's cluster-mode recipe calls `init` per worker
-  so each worker also mints a fresh `service.instance.id`.
-- **Test**: `spec/fork_spec.rb` — "a real fork does not crash and the child can
-  flush", "restarts every tracked metric reader and never raises".
+- **Mitigation**: `lib/foam/otel/fork_hooks.rb` prepends `Process._fork` and, in
+  the child, calls `Foam::Otel.after_fork!` (restarts tracked metric readers,
+  `lib/foam/otel/api.rb`) and clears the init idempotency guard
+  (`mark_forked!`), so the README's cluster-mode `init`-per-worker recipe is a
+  REAL re-init: it retires the inherited pipelines, re-owns foam's own slots
+  (`classify_for_registration`, `lib/foam/otel/init.rb` — a genuinely FOREIGN
+  provider is never touched), and mints a fresh `service.instance.id` per worker
+  (section 0 resource table). `after_fork!` is also exposed for
+  `Puma on_worker_boot` / `Unicorn after_fork` without re-init.
+- **Test**: `spec/fork_spec.rb` — "B3: a worker re-init after fork mints a NEW
+  service.instance.id", "the Process._fork hook fires in the CHILD only …
+  restarts the PeriodicMetricReader thread in the child", "the child's re-init
+  re-owns FOAM's inherited slots but NEVER a foreign provider", "init() itself
+  installs the fork hook", plus the child-flush smoke; conformance FORK phase
+  (distinct per-worker instance ids on the wire).
 
 ## R2: Signal maturity is uneven — logs & metrics SDKs are pre-1.0
 
@@ -52,12 +63,18 @@ Installed-source refs are rooted at the lockfile-pinned gems (opentelemetry-sdk
   — https://opentelemetry.io/docs/languages/ruby/ ; the metrics-sdk README states
   "alpha … things may break and APIs may change". Installed versions:
   logs-sdk 0.6.1, metrics-sdk 0.15.0.
-- **Decision & why**: Pin with pessimistic constraints (`~> 0.6`, `~> 0.15`) and
-  wrap both signals behind stable foam helpers (`log`, the four metric helpers)
-  so app code never touches the churning API directly.
-- **Mitigation**: gemspec pins + the helper wrappers (`lib/foam/otel/api.rb`,
-  `lib/foam/otel/metrics.rb`); the lockfile freezes exact versions.
-- **Test**: `spec/version_spec.rb` — pins the supported SDK lines and proves the
+- **Decision & why**: CAP every pre-1.0 range at the next minor above the
+  version the suite + conformance gate last proved (rule 40: a loose
+  two-segment `~> 0.x` would admit breaking 0.x minors into customer installs
+  — the lockfile only protects foam's own CI, never a customer resolution),
+  and wrap both signals behind stable foam helpers (`log`, the four metric
+  helpers) so app code never touches the churning API directly. Widening a cap
+  is a deliberate rule-44 re-prove.
+- **Mitigation**: capped gemspec ranges (e.g. `>= 0.6, < 0.7` logs-sdk,
+  `>= 0.15, < 0.16` metrics-sdk — reasons at each pin) + the helper wrappers
+  (`lib/foam/otel/api.rb`, `lib/foam/otel/metrics.rb`).
+- **Test**: `spec/gemspec_spec.rb` ("rule 40 — capped ranges, never an open
+  floor"), `spec/version_spec.rb` — pins the supported SDK lines and proves the
   helpers work over the pinned pre-1.0 SDKs.
 
 ## R3: Contrib instrumentations can themselves raise
@@ -105,35 +122,152 @@ Installed-source refs are rooted at the lockfile-pinned gems (opentelemetry-sdk
   real Rails app boots with foam and the official rack span carries the inbound
   telemetry (`/ok`, `/fail`, `/reject` assertions).
 
+## R5: Door-2 ingest taps ride the alpha logs/metrics SDKs — SHIPPED, every behavioral reliance pinned (fleet audit R8)
+
+- **Trap**: the log and metric ingest entries attach to upstream SDKs that are
+  pre-1.0 ("development"/alpha). Four load-bearing behaviors could drift under
+  a 0.x minor: (1) `MeterProvider#add_metric_reader` BACK-FILLS instruments
+  created before attach; (2) `MetricReader#metric_store` is the seam
+  `add_metric_reader` registers instruments through (why the tap SUBCLASSES
+  `PeriodicMetricReader` instead of composing); (3) collected data points are
+  SHALLOW dups whose `attributes` hash is the LIVE aggregation key — in-place
+  masking would corrupt the customer's aggregation state; (4)
+  `LogRecord#resource` carries the emitting provider's resource (the
+  self-ingest detection read).
+- **Sources**: installed source — `meter_provider.rb:110-121` (back-fill walk
+  + `register_synchronous_instrument` fan-out), `metric_reader.rb:15-22`
+  (`metric_store` seam), `aggregation/sum.rb:44-50` (`ndp.dup` shallow
+  copies), `logs/log_record.rb:16-26` (`attr_accessor :resource`);
+  maturity statement — https://opentelemetry.io/docs/languages/ruby/
+- **Decision & why**: the entries are SHIPPED (README Door 2 section); the
+  pre-1.0 logs/metrics SDKs are a version-pin caveat, not a deferral. Keep every
+  reliance inside the already-capped gemspec ranges (R2), and pin each with a
+  test so a 0.x bump is a deliberate rule-44 re-prove (any widening re-runs the
+  door-2 e2e), never a silent drift.
+- **Mitigation**: capped ranges (`foam-otel.gemspec`); masking rebuilds point
+  COPIES with fresh attribute hashes (`Redaction.mask_metric_data`,
+  `lib/foam/otel/redaction.rb`); subclassing isolated to
+  `lib/foam/otel/ingest_metric_reader.rb`.
+- **Test**: `spec/ingest_spec.rb` — "instruments created BEFORE attach flow to
+  foam (back-fill)" (reliances 1-2), "mask_metric_data is FAIL-CLOSED … and
+  never mutates the customer's point" (reliance 3), the case-2 log test +
+  self-ingest cases (reliance 4).
+
+## R6: PeriodicMetricReader thread death — a raising exporter or collect kills door-2 metrics silently
+
+- **Trap**: `PeriodicMetricReader#export` has NO rescue: one exception from
+  `collect` or the exporter chain kills the reader's ticker thread, and the
+  metrics tap goes dark forever with nothing but a dead thread (rule 15's
+  silent-loss shape, on someone else's pipeline).
+- **Sources**: installed source — `periodic_metric_reader.rb:105-131`
+  (`export` calls `collect` + `@exporter.export` unrescued inside the ticker
+  loop).
+- **Decision & why**: close both halves inside foam. The tap's whole exporter
+  chain (stamp → mask → OTLP) rescues `StandardError, SystemStackError` to
+  `FAILURE`, and the reader subclass overrides `collect` with a `[]`-on-error
+  guard — the thread can never see an exception from foam's side.
+- **Mitigation**: `lib/foam/otel/ingest.rb` (`MetricStampExporter`),
+  `lib/foam/otel/redacting_exporter.rb` (`RedactingMetricsExporter`),
+  `lib/foam/otel/ingest_metric_reader.rb` (`collect` guard).
+- **Test**: `spec/ingest_spec.rb` — "the reader thread survives a poisoned
+  collect ([] on error) and a raising exporter chain (GOTCHAS R6)".
+
+## R7: The door-2 fork registry — tap readers must survive fork AND a foam re-init (@ingest_taps ≠ @metric_readers)
+
+- **Trap**: two ways a door-2 metrics tap dies in cluster mode. (1) The
+  reader thread is not restarted after fork: upstream wires `after_fork` only
+  through `SDK.configure`'s ForkHooks, which door 2 never runs (the tap rides
+  the CUSTOMER'S manual setup). (2) If tap readers were tracked in foam's own
+  `@metric_readers`, the post-fork re-init path (`retire_foam_pipelines!`)
+  would CLEAR them — orphaning readers that belong to the customer's
+  pipeline, dark in every worker with no warning.
+- **Sources**: installed source — `periodic_metric_reader.rb:82-87`
+  (`after_fork` restarts the thread + resets the exporter),
+  `metrics/configurator_patch.rb:57` (upstream's hook is SDK.configure-only);
+  foam's own `init.rb` `retire_foam_pipelines!` (clears `@metric_readers`).
+- **Decision & why**: the ingest factory installs foam's `Process._fork` hook
+  (idempotent) and registers tap readers in a SEPARATE module registry,
+  `@ingest_taps`, walked by `Foam::Otel.after_fork!` beside foam's own
+  readers and deliberately untouched by pipeline retirement. The exporter
+  wrappers forward `reset` so `after_fork` reaches the OTLP connection.
+- **Mitigation**: `lib/foam/otel/init.rb` (`register_ingest_tap!`,
+  `retire_foam_pipelines!` comment), `lib/foam/otel/api.rb` (`after_fork!`
+  walks both), `lib/foam/otel/ingest.rb` (registration + `reset` forwarding).
+- **Test**: `spec/ingest_spec.rb` — "retire_foam_pipelines! clears
+  @metric_readers but NEVER @ingest_taps", "a REAL fork: foam's Process._fork
+  hook restarts the tap's reader thread in the child", "the exporter chain
+  forwards reset for after_fork".
+
+## R8: The tap's loop story — untraced coverage, the REQUIRED README step, and the echo filter
+
+- **Trap**: the tap's exports to foam are outbound HTTP THEIR instrumentation
+  will span, and their pipeline hands those spans back to the tap — a
+  feedback loop foam's own rule-24 guard cannot reach (foam does not own
+  their instrumentation). The SDK suppression context alone is defeated by
+  non-contrib patches, contrib versions predating the `untraced?` check, and
+  proprietary agents wrapping Net::HTTP; the documented `untraced_hosts` step
+  is fail-open by definition (an FDE who forgets it ships the loop).
+- **Sources**: installed source — exporters send inside
+  `Common::Utilities.untraced` (otlp `exporter.rb:146-147,171`, logs
+  `logs_exporter.rb:136-137`, metrics `util.rb:29-31`); the contrib bail-out
+  (net_http `patches/*/instrumentation.rb` `untraced?` /
+  `untraced_hosts`); spec rule 24 ("not optional and not configurable off").
+- **Decision & why**: three layers. (1) The suppression context, automatic —
+  foam's exporters all send through Net::HTTP inside `untraced`. (2) The
+  README-REQUIRED `untraced_hosts` step in THEIR http instrumentation (door
+  2's mirror of the 18 C ignore entry; the tap cannot wire it itself — door 2
+  never mutates their instrumentation). (3) The ECHO FILTER, structural: a
+  client/producer span naming the ACTIVE export host is telemetry about
+  foam's own export call — the tap refuses to re-export its own echo, warns
+  once naming the missing step (template W5), and leaves THEIR copy of the
+  span untouched (additive reader intact).
+- **Mitigation**: `lib/foam/otel/ingest.rb` (`SpanProcessor#echo_span?`,
+  `Ingest.echo_warning`); README "Door 2 — the required loop step".
+- **Test**: `spec/ingest_spec.rb` case 4 — halves 1/2 (each layer alone yields
+  zero export spans, with a live positive control), half 3 (BOTH defeated →
+  the feedback span appears — red without the guards — and the echo filter
+  bounds it, proven against the wire bytes), and the echo-filter unit case.
+
 ---
 
-## F1: Span attributes freeze at finish — the redaction floor cannot be a processor
+## F1: Span attributes freeze at finish — the redaction key pass cannot be a processor
 
-- **Trap**: The obvious place to mask attributes is a SpanProcessor, but the Ruby
-  SDK freezes a span's attributes at `finish` BEFORE any `on_finish` processor
-  runs — so a processor cannot mutate them, and neither can a tenant processor
-  see a masked view.
+- **Trap**: The obvious place to apply the redaction key pass is a SpanProcessor,
+  but the Ruby SDK freezes a span's attributes at `finish` BEFORE any `on_finish`
+  processor runs — so a processor cannot mutate them, and neither can a tenant
+  processor see a masked view.
 - **Sources**: installed source — `Span#finish` does
   `@attributes = validated_attributes(@attributes).freeze` then sets `@ended`
   (opentelemetry-sdk `trace/span.rb:277,280`); `to_span_data` returns the frozen
   `@attributes` by reference (`trace/span.rb:296-307`).
-- **Decision & why**: foam runs the always-on floor (rule 14) at the EXPORTER
-  boundary — a `RedactingSpanExporter`/`RedactingLogRecordExporter` that rebuilds
-  the mutable `SpanData`/`LogRecordData` Structs with masked attributes/events/
-  body before serialization. This guarantees no raw secret leaves the process in
-  foam's own export.
-- **Divergence (TODO pcga11)**: because attributes freeze at finish, a tenant
-  `additional_span_processor` receives span attributes UNMASKED. Masking is
-  EXPORT-only in Ruby, so the rule-18 C "tenant sees already-masked" guarantee
-  holds ONLY for values the caller pre-masks with `redact()` — NOT for
-  third-party instrumentation attributes AND NOT for values passed to
-  `set_attribute`/`set_attributes`. Foam's own export is always masked. Recorded
-  honestly, not papered over; surfaced to pcga11 as spec-gap material (Ruby
-  offers no safe pre-freeze hook — the `on_finishing` hook runs inside the span
-  mutex, so a processor calling `set_attribute` there would deadlock).
-- **Mitigation**: `lib/foam/otel/redacting_exporter.rb` + `lib/foam/otel/redaction.rb`.
-- **Test**: `spec/redacting_exporter_spec.rb` (masking on the wire),
-  `spec/redaction_spec.rb` (the floor, tested exhaustively).
+- **Opt-in redaction (holistic redesign, 2026-07-26)**: foam does NO redaction by
+  default — with no `redact_keys`/`redact_pii_keys` every value is captured RAW
+  (no secret floor, no value-pattern masking). When keys ARE configured, foam
+  applies the key pass at the EXPORTER boundary — a `RedactingSpanExporter`/
+  `RedactingLogRecordExporter` that rebuilds the mutable `SpanData`/
+  `LogRecordData` Structs with masked attributes/events/body before
+  serialization. With no keys configured the exporter is a true pass-through.
+- **Divergence (NARROWED)**: when keys are configured foam's HELPERS apply the
+  key pass AT CAPTURE (`api.rb` `stringify`/`set_attribute`/`log`; `metrics.rb`
+  always did), so a tenant `additional_*` instance receives foam-helper data
+  ALREADY-MASKED (rule 18 C.3) — the pass is idempotent, so the wire value is
+  unchanged. The residual divergence is THIRD-PARTY INSTRUMENTATION attributes
+  only: those freeze at finish before any processor runs, so a tenant span
+  processor sees them as foam captured them and foam can only apply the key pass
+  at the exporter boundary. Ruby offers no safe pre-freeze hook — the
+  `on_finishing` hook runs inside the span mutex, so a processor calling
+  `set_attribute` there would deadlock. Foam's own export applies the configured
+  key pass, and the exporters FAIL CLOSED once a pass has begun: a struct whose
+  rebuild raises is DROPPED with a loud warning (`Redaction.mask_*` return nil on
+  hard failure), never exported half-masked.
+- **Mitigation**: capture-time key pass in `lib/foam/otel/api.rb` +
+  `lib/foam/otel/redacting_exporter.rb` + `lib/foam/otel/redaction.rb`.
+- **Test**: `spec/tenant_seam_spec.rb` (tenant processors receive helper-set
+  LISTED keys ALREADY-MASKED, free-text bodies RAW, the byte-identical
+  with/without-tenant proofs for spans, logs, AND metrics),
+  `spec/redacting_exporter_spec.rb` (opt-in masking on the wire; "a POISONED
+  SpanData (dup raises) is DROPPED — never exported raw"), `spec/redaction_spec.rb`
+  (default zero-redaction + the two opt-in key lists; capture+exporter idempotence).
 
 ## F2: OTEL_SDK_DISABLED and OTEL_PROPAGATORS are only honored by SDK.configure
 
@@ -145,18 +279,88 @@ Installed-source refs are rooted at the lockfile-pinned gems (opentelemetry-sdk
   read only by `Configurator#configure_propagation`
   (`sdk/configurator.rb:209-225`, `none` → NoopTextMapPropagator). Spec:
   https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/
-- **Decision & why**: foam implements both itself. `OTEL_SDK_DISABLED=true`
-  (exact string) supersedes `enabled: true` and goes fully inert.
-  `OTEL_PROPAGATORS=none` installs a no-op propagator (injection off, telemetry
-  keeps flowing); any other value warns-and-ignores (foam's propagator set is
-  fixed).
-- **Mitigation**: `lib/foam/otel/init.rb` (`killed = ENV["OTEL_SDK_DISABLED"] == "true"`,
-  `configure_propagation`).
-- **Test**: `spec/init_spec.rb` (OTEL_SDK_DISABLED precedence + exact-string),
-  `spec/env_vars_spec.rb` (OTEL_PROPAGATORS none / default / ignored).
+- **Decision & why**: foam implements both itself. `OTEL_SDK_DISABLED` supersedes
+  `enabled: true` and goes fully inert when it reads (trimmed, case-insensitive)
+  `true` — DELIBERATELY looser than the upstream Ruby SDK's exact-string check,
+  so the fleet's one emergency switch (`TRUE`, ` true `) behaves identically
+  across languages (JS/Python already trim + downcase; the OTel boolean-env
+  convention is case-insensitive). `OTEL_PROPAGATORS=none` installs a no-op
+  propagator (injection off, telemetry keeps flowing); any other value
+  warns-and-ignores (foam's propagator set is fixed).
+- **Mitigation**: `lib/foam/otel/init.rb` (`kill_switch_active?` — trimmed,
+  case-insensitive `"true"`; shared by init and the door-2 ingest factory —
+  and `configure_propagation`).
+- **Test**: `spec/init_spec.rb` (OTEL_SDK_DISABLED precedence + trimmed
+  case-insensitive), `spec/env_vars_spec.rb` (OTEL_PROPAGATORS none / default /
+  ignored).
+- **Divergence (TODO pcga11)**: the spec's env table reads
+  "`OTEL_EXPORTER_OTLP_ENDPOINT` (and per-signal variants) | HONORED where
+  present", but foam deliberately pins the per-signal variants
+  (`OTEL_EXPORTER_OTLP_{TRACES,METRICS,LOGS}_ENDPOINT`) INERT: init resolves
+  ONE endpoint and passes it to every exporter constructor explicitly, and
+  the installed exporters consult the per-signal vars only when `endpoint:`
+  is nil (otlp `exporter.rb` `prepare_endpoint`) — so a per-signal var can
+  never silently split foam's export across destinations (or dodge the
+  export-loop guard, which is fed the ONE resolved host). A set-but-inert
+  per-signal var warns loudly at init (rule 15: never a silent no-op of
+  operator intent). Surfaced here for a fleet ruling; README env table names
+  all three. *Test*: `spec/env_vars_spec.rb` ("per-signal OTLP endpoint
+  variants — pinned INERT": no-redirect + loud-warning examples).
 
 ---
 
+## F4: Pre-init proxy loggers poison the logs-API upgrade (L1) — one early log() would kill the whole log pipeline
+
+- **Trap**: `opentelemetry-logs-api 0.4.1` `ProxyLoggerProvider#delegate=`
+  upgrades every CACHED proxy logger with a POSITIONAL
+  `provider.logger(key.name, key.version)` call, but the logs SDK's
+  `LoggerProvider#logger` is KEYWORD-ONLY (`logger(name:, version: nil)`) —
+  so if ANY logger was created through the pre-init proxy, the global
+  assignment `OpenTelemetry.logger_provider = provider` raises `ArgumentError`
+  MID-setter and the slot stays the inert proxy: the process exports NO logs,
+  ever, and the failure is one swallowed warning at init.
+- **Sources**: installed source — `proxy_logger_provider.rb:42`
+  (`logger.delegate = provider.logger(key.name, key.version)`, positional) vs
+  logs-sdk `logger_provider.rb:44` (`def logger(name:, version: nil)`);
+  the setter shape in `opentelemetry-logs-api.rb:26-34` (delegate installed
+  BEFORE the raise, assignment after).
+- **Decision & why**: two layers. (1) foam's `get_logger` NEVER registers into
+  the pre-init proxy registry — pre-init it returns the API's plain no-op
+  `OpenTelemetry::Logs::Logger` (pre-init logs are a silent no-op by contract
+  anyway), so foam's own surface can never poison the upgrade. (2) init's logs
+  setup rescues the `ArgumentError` and assigns AGAIN — the delegate was
+  installed before the raise, so the second assignment is accepted
+  (`delegate=` ignores the reset) and foam's provider lands; loggers a
+  CUSTOMER cached pre-init stay no-op, warned loudly.
+- **Mitigation**: `lib/foam/otel/api.rb` (`get_logger` no-op-logger path),
+  `lib/foam/otel/pipelines.rb` (`setup_logs` rescue+retry).
+- **Test**: `spec/init_spec.rb` ("ordering — LOG helper used BEFORE init":
+  rebind + held-logger cases), `spec/public_api_spec.rb` ("get_meter /
+  get_logger no-op in a clean process" — pins that no ProxyLogger is created).
+
+## F5: The propagation slot is last-write-wins — foam installs only over the API default
+
+- **Trap**: `OpenTelemetry.propagation` is a plain `attr_writer` global
+  (last-write-wins). Unconditionally assigning foam's W3C composite at init
+  would silently REPLACE a propagator a foreign SDK installed first — severing
+  the customer's B3/Jaeger/X-Ray propagation (inbound extraction AND outbound
+  injection) in their own product. "Two SDKs racing the context manager breaks
+  propagation for BOTH products — worse than losing a signal" (rule 18 B).
+- **Sources**: `opentelemetry-api opentelemetry.rb:27` (`attr_writer
+  :propagation`), `:69-71` (the reader memoizes the default
+  `NoopTextMapPropagator`).
+- **Decision & why**: default-detection, the same shape as the provider
+  classifier: foam installs its W3C tracecontext+baggage set ONLY when the
+  slot holds the API's built-in `NoopTextMapPropagator` default; anything else
+  is a foreign owner and is KEPT, with one honest `[foam]` warning. (A
+  customer who deliberately installed their own Noop instance is
+  indistinguishable from the default — foam then installs W3C; the warning
+  path names whatever class it kept.)
+- **Mitigation**: `lib/foam/otel/init.rb` (`configure_propagation` owner check).
+- **Test**: `spec/propagation_spec.rb` ("does not clobber a FOREIGN propagator
+  installed before init", "installs foam's W3C set when the slot still holds
+  the API DEFAULT").
+
 ## F3: Last-write-wins globals — foam cannot defend a slot claimed AFTER init
 
 - **Trap**: Ruby's OTel global setters are last-write-wins — a foreign SDK that
@@ -180,6 +384,190 @@ Installed-source refs are rooted at the lockfile-pinned gems (opentelemetry-sdk
 - **Test**: `spec/coexistence_spec.rb` — "warns ONCE when a foreign SDK displaces
   a slot after init".
 
+## F6: `ignored_outbound_hosts` cannot reach Faraday / HTTP (httprb) / HTTPX (G13)
+
+- **Trap**: Only the Net::HTTP and Excon contrib instrumentations support host
+  suppression. Faraday, HTTP (httprb) and HTTPX declare NO `untraced_hosts`
+  option, and `Base#config_options` silently DROPS unknown config keys — so
+  pre-installing them with foam's host list would install them UNGUARDED with
+  the option discarded: a tenant exporter or agent egress using those clients
+  still produces spans (the feedback-loop class rule 24's guard exists to
+  prevent), silently.
+- **Sources**: installed source — net_http `instrumentation.rb:30`
+  (`option :untraced_hosts, default: []`); Excon includes
+  `Concerns::UntracedHosts`; contrib source for faraday/http/httpx
+  instrumentation declares no such option; `instrumentation-base base.rb:309-310`
+  (unknown config keys dropped with only an upstream-logger warning);
+  `base.rb:218-219` (install is idempotent — a bad first install can never be
+  re-applied by `install_all`).
+- **Decision & why**: never silently not-apply a documented option. The
+  pre-install list is LIMITED to the two clients that honor it, and when the
+  FDE wires `ignored_outbound_hosts` while an unguardable client gem is
+  bundled, init warns loudly naming the client. Foam's OWN export loop is safe
+  for every client regardless (G4: the OTLP exporters send inside
+  `Common::Utilities.untraced`). Upstream need (untraced_hosts on
+  faraday/http/httpx) is flagged for filing.
+- **Mitigation**: `lib/foam/otel/init.rb` (`HTTP_CLIENT_INSTRUMENTATIONS` /
+  `UNGUARDABLE_HTTP_CLIENT_INSTRUMENTATIONS`, `warn_unguardable_clients`);
+  README option-table + scenario 2/3 note.
+- **Test**: `spec/loop_guard_spec.rb` ("lists the Instrumentation *class* path
+  for exactly the clients whose upstream SUPPORTS untraced_hosts",
+  "unguardable clients (GOTCHAS G13)…" warning cases).
+
+## F7: The classifier's FREE predicate reads `OpenTelemetry::Internal::Proxy*` (G14, rule 42 fenced reach)
+
+- **Trap**: Ruby's provider setters are last-write-wins with no read-back of
+  "still default" — the ONLY non-displacing way to classify a slot is to
+  pre-read it against the API's internal `Internal::ProxyTracerProvider` /
+  `ProxyMeterProvider` / `ProxyLoggerProvider` classes (the exact check the
+  API's own setters use). BREAKS IF upstream renames/moves those constants:
+  the feature-detect (`defined?`) then makes every slot UNVERIFIABLE.
+- **Sources**: installed source — `opentelemetry-api opentelemetry.rb:53`
+  (setter's own `instance_of? Internal::ProxyTracerProvider` check);
+  metrics-api / logs-api twins; RESEARCH.md §2 (why attempt-then-read is
+  impossible in Ruby).
+- **Decision & why**: keep the pre-read (it mirrors upstream's own predicate,
+  pinned by the version-capped `opentelemetry-api ~> 1.1` range in the
+  gemspec), and DEGRADE HONESTLY: a missing constant yields a distinct
+  UNCLASSIFIABLE verdict — foam registers nothing (fail-safe) and warns "foam
+  cannot classify the <signal> slot on this OpenTelemetry API version", never
+  fabricating a foreign-owner name. Upstream need (a public "is the global
+  still default" predicate) is flagged for filing.
+- **Mitigation**: `lib/foam/otel/classifier.rb` (`Verdict#unclassifiable?`,
+  `UNCLASSIFIABLE`), `lib/foam/otel/init.rb` (`warn_unclassifiable`).
+- **Test**: `spec/classifier_spec.rb` — "internal Proxy-class read degradation
+  (rule 42 fence)" (both examples: fail-safe verdicts + the cannot-classify
+  warning with no fabricated owner).
+
+## F8: The classify→register gap — Ruby has no atomic set-once attempt
+
+- **Trap**: all three Ruby setters assign UNCONDITIONALLY (last-write-wins
+  under a mutex that guards only the assignment — no compare-and-set), so
+  foam's pre-read classify and its own setter cannot be one atomic attempt
+  (rule 18 step 4 rests the no-race intent on attempt atomicity, which Ruby
+  does not offer). A foreign SDK registering in the gap between foam's
+  pre-read of a slot and foam's setter would be silently DISPLACED — and the
+  late-arrival check (`warn_if_displaced`) could never fire, because foam's
+  write lands last and the live provider IS foam's.
+- **Sources**: installed source — opentelemetry-api `opentelemetry.rb:52-60`
+  (tracer setter: mutex + unconditional assignment; metrics-/logs-api twins
+  `opentelemetry-metrics-api.rb:26-34` / `opentelemetry-logs-api.rb:26-34`);
+  RESEARCH.md §2 ("no Ruby setter refuses global replacement").
+- **Decision & why**: MINIMIZE the window — each signal is classified
+  IMMEDIATELY before its own setter (`register_free_signal` →
+  `classify_for_registration`), never as an up-front classify-all pass, so
+  the residual race is a few instructions wide instead of spanning three
+  whole pipeline builds. The platform offers no primitive to close it
+  entirely; a foreign SDK racing inside that residual window on another
+  thread is displaced with no warning — recorded honestly here (the same
+  honesty F3 gives the after-init displacement). Boot-time init on the main
+  thread (README: init FIRST, before the app boots) makes the residual
+  window practically unreachable.
+- **Mitigation**: `lib/foam/otel/init.rb` (`register_free_signal` /
+  `classify_for_registration` — the per-signal classify-then-set pairing).
+- **Test**: `spec/coexistence_spec.rb` — "TOCTOU window (GOTCHAS F8): a
+  foreign SDK claiming LOGS while foam registers TRACES is kept and warned,
+  never silently displaced".
+
+## F9: The upstream datastore instrumentations SANITIZE by default — db_statement :obfuscate would stand silently
+
+- **Trap**: pg/mysql2/redis/mongo contrib all declare
+  `option :db_statement, default: :obfuscate` — an `install_all` with no
+  config ships OBFUSCATED statements, violating the zero-default-redaction
+  ruling with no error anywhere. Worse, `Base#config_options` silently
+  REPLACES an invalid option value with the default — so a contrib bump that
+  renamed/dropped `:include` would silently fall BACK to `:obfuscate`.
+- **Sources**: installed source — pg `instrumentation.rb:29`, mysql2
+  `instrumentation.rb:24`, redis `instrumentation.rb:24`, mongo
+  `instrumentation.rb:36` (the `option :db_statement, default: :obfuscate`
+  lines); `instrumentation-base base.rb:309-310` (invalid values replaced
+  with the default, warning goes to the upstream logger only). Ruling:
+  docs/decisions/holistic-redesign-decision-log.md (raised floor, raw
+  capture 2026-07-26).
+- **Decision & why**: activation passes a CENTRAL config map
+  (`FLOOR_INSTRUMENTATION_CONFIG`, `lib/foam/otel/init.rb`) setting
+  `db_statement: :include` for all four datastore instrumentations — raw
+  statements are the floor's point; masking is opt-in via redact keys at
+  foam's redaction stage. The option's continued upstream validity is pinned
+  by a test so a contrib rename is a CI failure, never a silent sanitize.
+- **Test**: `spec/floor_spec.rb` — "install_all receives the CENTRAL
+  raw-capture config", ":include is a declared db_statement option on every
+  configured floor gem"; presence checks stay proven there too.
+
+## F10: The stdlib Logger bridge — two feedback loops and the lazy-block contract
+
+- **Trap**: bridging `::Logger` naively (1) turns the OTel SDK's OWN internal
+  logger into a producer — every export failure logs, which emits a record,
+  which must be exported: unbounded growth the per-stack re-entrancy flag
+  cannot see because the batch thread is a different stack; (2) evaluates a
+  caller's lazy `logger.debug { expensive }` block even when the severity is
+  filtered (stdlib's block form exists precisely to avoid that cost), or
+  evaluates an accepted block TWICE (once for the bridge, once in `super`).
+- **Sources**: installed source — opentelemetry-api `opentelemetry.rb:30-31`
+  (`OpenTelemetry.logger` memoized `Logger.new($stdout)`) and `:37` (the
+  error handler logs through it — the export-failure feedback edge); stdlib
+  logger gem `logger.rb:675` (`Logger#add`, the single funnel; lazy block
+  resolution order); foam's emit path `lib/foam/otel/logger_bridge.rb`.
+- **Decision & why**: prepend ONE patch on `Logger#add`; skip
+  `OpenTelemetry.logger` by live identity; hold a thread-local re-entrancy
+  flag around the emit; resolve an accepted block exactly once and hand the
+  RESULT down to `super`; return `super` untouched for filtered severities
+  (block unevaluated). Emission is gated on foam OWNING the logs slot (rule
+  18 B) and every bridge failure degrades to the stdlib write alone (rule 9).
+- **Test**: `spec/logger_bridge_spec.rb` — the two loop-guard examples, the
+  lazy-block pair, the foreign-slot darkness, and "NEVER breaks the host's
+  log call".
+
+## F11: LLM shims patch third-party SDK call sites — the model call is sacred
+
+- **Trap**: no official Ruby contrib instrumentation exists for ANY LLM SDK,
+  so foam patches the SDKs' public methods directly. A raising extractor, a
+  broken tracer, or a semantic drift in the SDK's params/response shapes
+  must never fail the customer's model call (a broken instrumentation that
+  drops a production LLM request is the worst possible trade), and the SDK's
+  own HTTP send must not produce a SECOND span under the gen_ai span.
+- **Sources**: installed source — openai `resources/chat/completions.rb:119`
+  / `resources/responses.rb:96`, anthropic `resources/messages.rb:68`,
+  gemini-ai `controllers/client.rb:151`, ruby_llm `provider.rb:44` (the
+  patched call sites); opentelemetry-common `utilities.rb:104-117`
+  (`untraced` — the suppression context the http contrib honors). Ruling:
+  LLM provider parity + testing-mandatory (decision log 2026-07-26).
+- **Decision & why**: every patch routes through ONE guarded core
+  (`Foam::Otel::LLM.observe`, `lib/foam/otel/llm.rb`): request extraction
+  inside `safe_request` (degrades to bare operation/provider), span start
+  fail-to-dark (call runs unspanned), the SDK call's error re-raised
+  IDENTICALLY with `error.type` (+ the OBSERVED http status only), the SDK's
+  transport run inside `Common::Utilities.untraced` (one call = one span).
+  Presence-checked constants + explicit version windows per SDK; emission
+  gated on foam owning traces. Wire names are the GenAI semconv strings the
+  fleet's js/python gap-fillers emit — content attributes included, RAW.
+- **Test**: `spec/llm_openai_spec.rb`, `spec/llm_anthropic_spec.rb`,
+  `spec/llm_gemini_spec.rb`, `spec/llm_ruby_llm_spec.rb` — real SDKs against
+  a local fake provider (only the 3rd party mocked), per-provider activity +
+  RAW-content assertions, error-path re-raise, and the fail-to-dark set.
+
+## F12: Session stamping must beat the freeze (spans) and the buffer (logs)
+
+- **Trap**: the two signals offer different safe write points. Span
+  attributes FREEZE at finish (F1) — an on_finish stamp is impossible; log
+  records are buffered LIVE by the batch processor and serialized later on
+  another thread — a stamp must land on the emitting thread before export
+  can convert the record.
+- **Sources**: installed source — opentelemetry-sdk `trace/span.rb:277-280`
+  (freeze at finish), logs-sdk `batch_log_record_processor.rb:70-83`
+  (`on_emit` buffers the live record) + `export` (LogRecordData conversion
+  at export time). Ruling: session stitching (decision log 2026-07-26).
+- **Decision & why**: spans stamp in `on_start` (order-independent — the
+  attribute exists long before finish); logs stamp in `on_emit` on the
+  emitting thread, registered right after foam's batch processor (which
+  stays FIRST — pinned invariant). Both read `session.id` /
+  `session.previous_id` from the CONTEXT's baggage (foam's W3C propagator
+  set already extracts it), stamp verbatim, no-op when absent, never raise.
+  Door-2 taps are untouched — foam never mutates the customer's records.
+- **Test**: `spec/session_stitching_spec.rb` — span + log stamping, the
+  rack end-to-end with real `traceparent`+`baggage` headers, the no-op and
+  never-raise cases.
+
 ## General gotchas (applicable to Ruby)
 
 - **G1 — init after target import / pre-init no-op**: the API's proxy providers
@@ -189,29 +577,110 @@ Installed-source refs are rooted at the lockfile-pinned gems (opentelemetry-sdk
 - **G2 — process ends before flush**: batch processors buffer; `flush` is public
   and the README wires it at handler end / worker exit.
   *Test*: `spec/fork_spec.rb` (child flush).
+- **G3 — batch queue silently drops**: a full `BatchSpanProcessor`/
+  `BatchLogRecordProcessor` queue drops telemetry with no signal by default
+  (installed source: `batch_span_processor.rb` `report_dropped_spans`, reasons
+  `buffer-full`/`terminating`, visible only through the SDK's internal metrics
+  reporter / `OTEL_LOG_LEVEL=debug` error handler). Foam keeps upstream's
+  buffering untouched (rule 4: no client-side shedding decisions of its own)
+  and makes loss VISIBLE instead: `flush()` is public for burst/exit paths and
+  the failure-modes/diagnostics story is README surface; drops in foam's OWN
+  fail-closed path (redaction hard failure) always warn loudly.
+  *Test*: `spec/redacting_exporter_spec.rb` (loud drop warnings);
+  `spec/public_api_spec.rb` ("actually drains a pending span"); conformance
+  FLUSH phase (10-minute batch delays — only flush/shutdown deliver).
 - **G4 — exporter self-tracing loop**: the OTLP exporter sends inside an
   `OpenTelemetry::Common::Utilities.untraced` context the official
   instrumentation respects (opentelemetry-exporter-otlp `exporter.rb:147`), so
   export never traces itself; `ignored_outbound_hosts` + the endpoint host also
-  feed the http-client instrumentations' `untraced_hosts`
+  feed the http-client instrumentations' `untraced_hosts` where upstream
+  supports the option — Net::HTTP and Excon only, see F6/G13
   (`lib/foam/otel/init.rb` `configure_loop_guard`).
-  *Test*: `test-apps/ruby-rails` gate (no feedback spans appear).
+  *Test*: `spec/loop_guard_spec.rb` (behavioral zero-spans proof);
+  `test-apps/ruby-rails` gate (no feedback spans appear).
 - **G5 — context across async boundaries**: `Context.current` is fiber-local; a
   raw `Thread.new` loses the span. foam relies on the official
   `-concurrent_ruby` and `-active_job` instrumentations (bundled via `-rails`) to
   carry context across Rails' pools and jobs; a hand-rolled thread is the FDE's
-  to wrap. *Documented*; conformance covers the rack-context path.
+  to wrap. *Test*: `spec/gotcha_spec.rb` ("the bundled concurrent_ruby instrumentation
+  carries the active span across a Promises thread pool"); conformance covers
+  the rack-context path.
+- **G6 — customer infrastructure strips propagation headers**: gateways/LBs
+  strip or rewrite `traceparent`, and a format mismatch (W3C sender, B3-only
+  receiver) fails extraction SILENTLY — every service becomes its own root.
+  Foam cannot fix their infra; foam keeps extraction non-fabricating (a fresh
+  root, never invented ids), keeps the W3C set the default, and the README
+  troubleshooting entry ("a downstream rejects traceparent" /
+  `OTEL_PROPAGATORS=none`) carries the operator lever. When the incumbent owns
+  the propagator, foam YIELDS (F5) so the formats the customer's infra expects
+  keep flowing.
+  *Test*: `spec/propagation_spec.rb` (inbound respected, injection exact);
+  conformance MAIN phase (two-hop continuation with fixed W3C ids) and
+  PROPNONE phase (extraction+injection stop, telemetry flows).
 - **G7 — metric cardinality explosion**: the metric helpers forward
   caller-supplied `attributes` verbatim; an unbounded attribute value (user id,
-  raw path) explodes the series count. The README carries the warning with the
-  classic bad example. *Test*: `spec/metrics_spec.rb` (attribute passthrough).
+  raw path) explodes the series count. There is deliberately NO code
+  mitigation (foam never rewrites or drops metric dimensions — rule 4/26);
+  the mitigation is the README metrics warning with the classic bad example —
+  *Documented* (README "Metrics" section). *Test*: `spec/metrics_spec.rb`
+  ("captures a metric attribute RAW by default; masks it only when the key is in
+  redact_keys" — pins that unlisted attributes pass through VERBATIM).
+- **G8 — never-ended spans leak memory**: a span started and never ended holds
+  memory and context forever (and never exports — the batch processor only
+  sees FINISHED spans, `on_finish`). `Foam::Otel.span {}` always ends the span
+  by construction (rule 9), proven even when the block raises; `get_tracer`
+  passthrough users own their `span.finish` — README note ("the span always
+  ends") rather than code.
+  *Test*: `spec/public_api_spec.rb` ("re-raises the block's error identically
+  AND records it, always ending the span" — the span appears in
+  finished_spans, i.e. it was ended).
 - **G9 — a second SDK / injected agent**: the classifier detects a foreign
-  provider per signal and warns by name; foam registers only into free slots.
+  provider per signal and warns by name; foam registers only into free slots —
+  and installs NO instrumentation into a foreign-owned trace pipeline
+  (`spec/activation_spec.rb` claimed-traces cases).
   *Test*: `spec/coexistence_spec.rb` (8-permutation matrix), `spec/classifier_spec.rb`.
+- **G10 — instrumentor version-range drift**: a contrib gem supports library
+  vN; the customer upgrades to vN+1 and the instrumentation goes silent (each
+  contrib gem's `compatible?`/`present?` gates install) — or, worse, an
+  UPGRADED contrib gem changes what it emits (rule 26: contrib pins set the
+  semconv). Foam caps every contrib range at the proven minor in the gemspec
+  (`-rack >= 0.26, < 0.32`, `-rails >= 0.31, < 0.43` — reasons at the pins) so
+  drift is a deliberate foam release with the fixtures re-proven, never an
+  in-place `bundle update` surprise; the supported matrix is README surface.
+  *Test*: `spec/gemspec_spec.rb` ("rule 40 — capped ranges"); conformance gate
+  pins the emitted rack/action_pack span shape against the locked pair.
+- **G11 — MetricReader temporality and Views**: multiple readers per
+  MeterProvider are spec-legal; each reader triggers its OWN collection
+  (`MetricReader#collect` pulls the shared `@metric_store`,
+  metrics-sdk `export/metric_reader.rb:21`), and the Ruby metrics SDK's
+  temporality/Views story is the least-documented corner upstream (alpha —
+  see R2). Foam's own reader is the stock `PeriodicMetricReader` with upstream
+  defaults (no Views, no temporality override — thin passthrough, rule 26);
+  a tenant reader supplied via `additional_metric_readers` runs its collect
+  loop on ITS thread (tenant code, not foam-invoked — the never-throw guard
+  covers foam-invoked flush/shutdown only, README says so), and foam's export
+  is byte-identical with and without it.
+  *Test*: `spec/tenant_seam_spec.rb` ("foam's METRICS export is IDENTICAL with
+  and without a tenant metric reader", "a tenant metric reader raising in
+  force_flush/shutdown never breaks foam-invoked flush or shutdown").
 - **G12 — two copies of the OTel API**: two gem copies → two global registries →
-  silent no-op or doubled telemetry. Cannot be seen at build time; the README
+  silent no-op or doubled telemetry. Ruby's Bundler makes duplicate same-name
+  gems unresolvable (one activated copy per process — a second copy raises
+  `Gem::LoadError` loudly), so the trap is a REBUNDLED/vendored copy; the README
   troubleshooting entry says to check `bundle list | grep opentelemetry-api` for
-  one copy first. *Documented* (README).
+  one copy first. *Test*: `spec/gotcha_spec.rb` ("exactly ONE opentelemetry-api gem is
+  active", "the API entry file was loaded from exactly ONE gem copy"),
+  `spec/packaging_spec.rb` ("activating a conflicting opentelemetry-api
+  version raises Gem::LoadError at boot").
+- **G15 — runtime metrics are asynchronous-instrument callbacks**: the
+  hand-written GC/thread/RSS readings run on the PeriodicMetricReader's
+  thread at COLLECT time (no extra timer thread; fork recovery rides R1's
+  reader restart). A raising reading (missing GC.stat key, unreadable /proc)
+  must degrade to nil — the SDK ignores non-numeric callback results — so one
+  bad reading can never kill the collect loop or the healthy instruments
+  beside it (`lib/foam/otel/runtime_metrics.rb` `guarded_callback`).
+  *Test*: `spec/runtime_metrics_spec.rb` ("a poisoned reading is
+  fail-to-dark", the foreign-slot and idempotence cases).
 
 ---