foam-otel 1.0.2 → 1.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/README.md CHANGED
@@ -2,9 +2,11 @@
2
2
 
3
3
  Foam's OpenTelemetry core for Ruby services. A thin, safe wrapper over the
4
4
  official OpenTelemetry libraries: foam owns the pipeline (providers, batch
5
- processors, OTLP export to the foam fleet endpoint, an always-on redaction
6
- floor), turns on automatic tier-1/2 instrumentation, and hands you a small set
7
- of never-throw helpers — and nothing else.
5
+ processors, OTLP export to the foam fleet endpoint), turns on automatic
6
+ tier-1/2 instrumentation, and hands you a small set of never-throw helpers —
7
+ and nothing else. **Redaction is fully opt-in: by default foam captures every
8
+ value RAW** (no secret floor, no value-pattern masking); you enumerate the
9
+ fields to mask via `redact_keys`/`redact_pii_keys`.
8
10
 
9
11
  Built to `docs/BASE_PACKAGE_SPEC.md`. This README is the manual you integrate
10
12
  from.
@@ -19,12 +21,16 @@ Foam builds on the OpenTelemetry authors' work (https://opentelemetry.io) — se
19
21
  ```ruby
20
22
  # Gemfile
21
23
  gem "foam-otel"
22
- # Bundle the official instrumentation for whatever your app uses; foam
23
- # auto-activates each when present (presence-checked). Rails + Rack come with
24
- # foam-otel; add others as needed:
25
- gem "opentelemetry-instrumentation-pg" # e.g. Postgres
26
- gem "opentelemetry-instrumentation-redis" # e.g. Redis
27
- gem "opentelemetry-instrumentation-sidekiq" # e.g. Sidekiq
24
+ # foam-otel bundles the UNIVERSAL FLOOR and auto-activates each piece when its
25
+ # target library is present (presence-checked, and gated on foam owning the
26
+ # slot it produces into): Rack + the Rails family, the HTTP clients (Net::HTTP,
27
+ # Faraday), the primary datastores (pg, mysql2, redis, mongo) with RAW
28
+ # db.statement capture, Sidekiq, the stdlib Logger bridge, hand-written Ruby
29
+ # runtime + GC metrics, session stitching (browser session.id via baggage),
30
+ # and the LLM shims (OpenAI, Anthropic, Gemini, ruby_llm — activity + raw
31
+ # content). Add only the niche long tail your app needs beyond the floor:
32
+ gem "opentelemetry-instrumentation-graphql" # e.g. GraphQL
33
+ gem "opentelemetry-instrumentation-resque" # e.g. Resque
28
34
  ```
29
35
 
30
36
  **Supported versions** (tested in CI, `spec/version_spec.rb`): Ruby **>= 3.1**;
@@ -54,10 +60,53 @@ Foam::Otel.init(
54
60
  ```
55
61
 
56
62
  **Outcome**: foam registers the tracer/meter/logger providers, exports OTLP to
57
- the foam fleet endpoint, and auto-activates the official rack/rails
58
- instrumentation (and any other bundled `opentelemetry-instrumentation-*`). Every
59
- inbound request, DB call, HTTP call, and job the official instrumentations cover
60
- now flows to foam, with the redaction floor applied on the wire.
63
+ the foam fleet endpoint, and auto-activates the **universal floor** — the
64
+ official rack/rails instrumentation plus the bundled HTTP (Net::HTTP/Faraday),
65
+ datastore (pg/mysql2/redis/mongo), and Sidekiq instrumentations — each
66
+ presence-gated, and gated on foam owning the traces slot (a foreign-owned
67
+ tracer stands the activation down). The datastore set is activated with
68
+ **`db_statement: :include`** — foam explicitly overrides the upstream
69
+ `:obfuscate` default so RAW SQL / redis args / mongo commands land on the wire.
70
+ On top of the contrib set the floor adds foam's own pieces: the **stdlib
71
+ Logger bridge** (every host `Logger` line also lands as a trace-correlated
72
+ OTel log record, body raw), hand-written Ruby **runtime + GC metrics**
73
+ (GC counts + timing, heap slots, allocation, thread count, RSS — Ruby has no
74
+ drop-in upstream metrics gem, so foam ships its own collector),
75
+ **session stitching** (inbound `baggage: session.id` from the foam browser
76
+ package is stamped onto every span and log record), and the **LLM shims**
77
+ (below). Every inbound request, DB call, HTTP call, job, log line, and model
78
+ call now flows to foam. **Redaction is opt-in: by default every value —
79
+ attributes, URLs/query strings, DB statement text, log bodies, LLM content —
80
+ is captured RAW.** Pass `redact_keys`/`redact_pii_keys` to mask or erase
81
+ specific fields (see below).
82
+
83
+ ### The LLM surface — OpenAI, Anthropic, Gemini (and ruby_llm)
84
+
85
+ The Ruby contrib registry ships no LLM instrumentation, so foam does: thin,
86
+ presence-checked shims over the SDKs' public call sites, emitting the standard
87
+ `gen_ai.*` semantic conventions — activity (model, response id, finish
88
+ reasons, token usage, latency) AND prompt/response **content, RAW**
89
+ (`gen_ai.input.messages` / `gen_ai.output.messages` /
90
+ `gen_ai.system_instructions`). Covered: the official `openai` SDK
91
+ (chat.completions.create + responses.create), the official `anthropic` SDK
92
+ (messages.create), Gemini via the `gemini-ai` gem (Google ships no official
93
+ Ruby SDK), and `ruby_llm` (one seam covering all its providers). Absent SDK =
94
+ inert; version-guarded; fail-to-dark — a shim failure can NEVER break the
95
+ model call, and the SDK's own error re-raises identically. One model call =
96
+ one `gen_ai` client span (the SDK's HTTP send rides the same suppression
97
+ context foam's exporters use). Streaming call sites (`#stream`/`#stream_raw`
98
+ on the official SDKs) are a recorded follow-up.
99
+
100
+ ### Session stitching — browser `session.id` on backend telemetry
101
+
102
+ The foam browser package stamps `session.id` on every browser signal and
103
+ sends it to your API as W3C Baggage beside `traceparent`. This gem's floor
104
+ copies `session.id` (and `session.previous_id` across a rotation) from the
105
+ inbound baggage onto every span and log record foam emits — browser session →
106
+ backend traces/logs, stitched with no ingest-side join. No-op when the
107
+ baggage is absent. **CORS note (the FDE wires this on every frontend-called
108
+ API):** the allowlist must include BOTH headers —
109
+ `Access-Control-Allow-Headers: traceparent, baggage`.
61
110
 
62
111
  ---
63
112
 
@@ -70,15 +119,56 @@ now flows to foam, with the redaction floor applied on the wire.
70
119
  | `enabled:` | Boolean | yes | — | The config switch. `false` = fully inert (no SDK, no providers, no network, helpers no-op). `true` = export, in EVERY environment. No default — you write the logic. |
71
120
  | `token:` | String | when enabled | — | `Authorization: Bearer` for export. Required (and validated) only when `enabled: true`; wired explicitly from your secret source — never an env fallback read by the gem. **Fleet convention: read it from the `FOAM_OTEL_TOKEN` env var.** Blank while enabled → raises at boot. |
72
121
  | `version:` | String | no | nil | `service.version`, verbatim (git SHA recommended). Missing → warns and continues. Never detected at runtime. |
73
- | `redact_keys:` | Array<String> | no | `[]` | Extra field names treated as SECRETS (tail-masked). EXTENDS the always-on floor. |
74
- | `redact_pii_keys:` | Array<String> | no | `[]` | Field names treated as PII (full `[REDACTED]`, no tail). Foam ships no PII preset — this is your enumeration. |
122
+ | `redact_keys:` | Array<String> | no | `[]` | The ONLY field names tail-masked (shape-preserving, e.g. `********f456`). Opt-in — empty (the default) means NO masking. Matches by case-insensitive substring, including keys nested in structured values and query-string keys. |
123
+ | `redact_pii_keys:` | Array<String> | no | `[]` | Field names fully erased to `[REDACTED]` (no tail). Opt-in — empty (the default) means NO erasure. Foam ships no preset; this is your enumeration. |
75
124
  | `additional_span_processors:` | Array | no | `[]` | Tenant seam: constructed SpanProcessor instances added to foam's pipeline (additive; never replace foam's export). See coexistence. |
76
125
  | `additional_log_record_processors:` | Array | no | `[]` | Tenant seam, logs. |
77
126
  | `additional_metric_readers:` | Array | no | `[]` | Tenant seam, metrics. |
78
127
  | `additional_instrumentations:` | Array | no | `[]` | Constructed tier-2 instrumentation instances to register (fault-isolated: one that throws is skipped with a `[foam]` warning). |
79
- | `ignored_outbound_hosts:` | Array<String> | no | `[]` | Hosts whose outbound calls produce no spans — EXTENDS the built-in export-loop guard. For a co-resident agent's intake or a tenant exporter's endpoint. |
128
+ | `ignored_outbound_hosts:` | Array<String> | no | `[]` | Hosts whose outbound calls produce no spans — EXTENDS the built-in export-loop guard. For a co-resident agent's intake or a tenant exporter's endpoint. Applies to the clients whose official instrumentation supports host suppression: **Net::HTTP and Excon**. Faraday/HTTP (httprb)/HTTPX have no upstream `untraced_hosts` option — foam warns loudly at init when one of those is bundled (GOTCHAS G13); foam's own export loop is guarded for every client regardless. |
80
129
  | `diagnostics:` | Boolean | no | false | Verbose `[foam]` self-reporting of init/health. Warnings and errors are always loud regardless. |
81
130
 
131
+ Examples for the remaining options (each is runnable as-is):
132
+
133
+ ```ruby
134
+ # Redaction is OPT-IN (default: nothing is masked). List the fields to redact:
135
+ Foam::Otel.init(name: "checkout-api", environment: env, enabled: true, token: ENV.fetch("FOAM_OTEL_TOKEN"),
136
+ redact_keys: %w[password api_key internal_ref voucher_code], # tail-masked
137
+ redact_pii_keys: %w[customer_email full_name]) # fully [REDACTED]
138
+ ```
139
+
140
+ ```ruby
141
+ # Constructed tier-2 instrumentation instances (fault-isolated; one that
142
+ # throws is skipped with a [foam] warning, never a crashed boot):
143
+ Foam::Otel.init(name: "checkout-api", environment: env, enabled: true, token: ENV.fetch("FOAM_OTEL_TOKEN"),
144
+ additional_instrumentations: [
145
+ OpenTelemetry::Instrumentation::Sidekiq::Instrumentation.instance,
146
+ [OpenTelemetry::Instrumentation::PG::Instrumentation.instance, { db_statement: :omit }],
147
+ ])
148
+ ```
149
+
150
+ ```ruby
151
+ # Tenant seam, logs + metrics (spans are shown in scenario 3 below):
152
+ Foam::Otel.init(name: "checkout-api", environment: env, enabled: true, token: ENV.fetch("FOAM_OTEL_TOKEN"),
153
+ additional_log_record_processors: [EvalTool::LogRecordProcessor.new],
154
+ additional_metric_readers: [EvalTool::MetricReader.new],
155
+ ignored_outbound_hosts: ["ingest.eval-tool.example"]) # the tenant loop guard — required
156
+ ```
157
+
158
+ ```ruby
159
+ # Verbose self-reporting while wiring foam up (warnings are loud regardless):
160
+ Foam::Otel.init(name: "checkout-api", environment: "development", enabled: true,
161
+ token: ENV.fetch("FOAM_OTEL_TOKEN"), diagnostics: true)
162
+ ```
163
+
164
+ And the unhappy path — a blank required argument fails AT BOOT, on your
165
+ machine, never silently:
166
+
167
+ ```ruby
168
+ Foam::Otel.init(name: "", environment: "production", enabled: true, token: ENV.fetch("FOAM_OTEL_TOKEN"))
169
+ # => raises ArgumentError: Foam::Otel.init requires a non-empty name
170
+ ```
171
+
82
172
  **Deliberately absent** (and why): `endpoint`, `cadence`/`sampling`,
83
173
  `is_production`, `force_export`, `disabled_environments`, and any
84
174
  processor/exporter injection beyond the tenant seam. Customers identify their
@@ -159,14 +249,14 @@ Foam::Otel.get_logger("my.lib")
159
249
  non-destructive thing. The classifier's output is information for you, never a
160
250
  decision it makes for you.
161
251
 
162
- ### Scenario 2 — beside a proprietary APM agent (e.g. Datadog)
252
+ ### Scenario 2 — beside a proprietary APM agent
163
253
 
164
254
  Disjoint pipelines: foam runs its own beside the agent, touching nothing. Add the
165
255
  agent's intake host so foam's outbound instrumentation doesn't shadow its egress.
166
256
 
167
257
  ```ruby
168
258
  Foam::Otel.init(name: "checkout-api", environment: env, enabled: true, token: FOAM_OTEL_TOKEN,
169
- ignored_outbound_hosts: ["trace.agent.datadoghq.com"])
259
+ ignored_outbound_hosts: ["agent-intake.vendor.example"])
170
260
  ```
171
261
 
172
262
  ### Scenario 3 — a scoped tenant rides foam's pipeline (LLM-eval, AI-observability)
@@ -180,17 +270,79 @@ Foam::Otel.init(name: "checkout-api", environment: env, enabled: true, token: FO
180
270
  ignored_outbound_hosts: ["ingest.eval-tool.example"])
181
271
  ```
182
272
 
183
- > Ruby note: because the Ruby SDK freezes span attributes at finish, a tenant
184
- > processor sees third-party instrumentation attributes UNMASKED (foam's floor
185
- > runs at the exporter boundary). Foam's own export is always masked.
186
-
187
- ### Scenario 4 — a foreign OTel SDK already owns a signal
273
+ > Ruby note: when you configure `redact_keys`/`redact_pii_keys`, foam's HELPERS
274
+ > apply the key pass at capture, so a tenant processor receives foam-helper data
275
+ > (span/log helper attributes, log bodies, metric labels) ALREADY masked.
276
+ > Because the Ruby SDK freezes span attributes at finish, a tenant
277
+ > processor sees third-party instrumentation attributes UNMASKED (for those,
278
+ > foam's key pass runs at the exporter boundary — GOTCHAS F1). With no keys
279
+ > configured (the default) nothing is masked on either path.
280
+ >
281
+ > Loop-guard note: `ignored_outbound_hosts` suppresses spans for tenant/agent
282
+ > egress made through **Net::HTTP or Excon**. A tenant exporter using
283
+ > Faraday/HTTP (httprb)/HTTPX cannot be host-suppressed today (no upstream
284
+ > `untraced_hosts` option — foam warns at init; GOTCHAS G13).
285
+
286
+ ### Scenario 4 — a foreign OTel SDK already owns a signal (door 2)
188
287
 
189
288
  If another OTel SDK registered (say) the tracer before foam, foam does NOT
190
- displace it. It warns once, naming the owner, and is DARK for that signal — your
191
- tier-3 telemetry for it flows through that SDK, not to foam. Foam ingest for a
192
- claimed signal is not yet available; feed foam server-side from that pipeline, or
193
- file the need.
289
+ displace it. `init` warns once, naming the owner, and foam is dark for that
290
+ signal — your tier-3 telemetry for it rides the foreign SDK to ITS backend.
291
+ To ALSO send that pipeline to foam, add ONE LINE per claimed signal in THEIR
292
+ setup — the ingest entries (see "Door 2 — the ingest entries" below). YOU
293
+ place the tap: foam never attaches itself to a pipeline it does not own
294
+ (their data moves only when a human says so), and the placement line doubles
295
+ as the off-switch — gate it on the same condition you would give `enabled:`.
296
+
297
+ ```ruby
298
+ # config/initializers/opentelemetry.rb — THEIR setup, plus the foam lines
299
+ require "opentelemetry/sdk"
300
+ require "foam/otel"
301
+
302
+ OpenTelemetry::SDK.configure do |c|
303
+ c.service_name = "their-service"
304
+ c.use_all
305
+ # REQUIRED loop step (see "Door 2 — the required loop step" below):
306
+ c.use "OpenTelemetry::Instrumentation::Net::HTTP", { untraced_hosts: ["otel.api.foam.ai"] }
307
+ c.add_span_processor(their_vendor_processor) # their export, untouched
308
+ if ENV["APP_ENV"] == "production" # your consent AND your off-switch
309
+ c.add_span_processor(Foam::Otel.create_ingest_span_processor(
310
+ token: ENV.fetch("FOAM_OTEL_TOKEN"), environment: ENV.fetch("APP_ENV")))
311
+ end
312
+ end
313
+ ```
314
+
315
+ **Outcome**: their pipeline keeps working exactly as before — their data,
316
+ their resource, their export, byte-identical — and foam ALSO receives the
317
+ signal, carrying THEIR resource identity plus foam's export-time stamp and
318
+ any per-tap `redact_keys`/`redact_pii_keys` you set (opt-in — none by default;
319
+ details in the Door 2 section).
320
+
321
+ **Unhappy paths**:
322
+
323
+ ```ruby
324
+ Foam::Otel.create_ingest_span_processor(token: "", environment: "production")
325
+ # => raises ArgumentError: [foam] Foam::Otel.create_ingest_span_processor requires a non-empty token
326
+ ```
327
+
328
+ - `OTEL_SDK_DISABLED=true` at construction → the factory returns an inert
329
+ no-op instance (their setup line keeps working, nothing is sent) and warns:
330
+ `[foam] OTEL_SDK_DISABLED=true — this foam ingest entry is disabled by the
331
+ operator kill switch and will send nothing.`
332
+ - Tap wired onto FOAM'S OWN pipeline (self-ingest wiring bug) → one loud
333
+ warning and the tap STANDS DOWN — door 1 already exports that data, so
334
+ nothing is lost and nothing is doubled.
335
+ - `init()` already exports a signal AND a tap sits on a separate foreign
336
+ pipeline for the same signal → one loud warning; BOTH keep flowing
337
+ (different pipelines; `foam.ingest.tier` distinguishes the copies).
338
+
339
+ When the TRACES slot is foreign-owned, foam also skips its automatic
340
+ tier-1/2 instrumentation activation (it never adds span producers or Rack
341
+ middleware to a pipeline it does not own) and warns once;
342
+ `additional_instrumentations` you pass explicitly are still installed. The
343
+ same holds for the process-wide propagator: a propagator the incumbent
344
+ installed first is KEPT (foam installs its W3C set only over the API default),
345
+ with one warning.
194
346
 
195
347
  ### Scenario 5 — per-signal composition
196
348
 
@@ -200,6 +352,13 @@ metrics and logs on the full contract:
200
352
  ```ruby
201
353
  Foam::Otel.init(name: "checkout-api", environment: env, enabled: true, token: FOAM_OTEL_TOKEN)
202
354
  # → warns: traces foreign-owned (foam dark for traces); metrics + logs → foam, full contract.
355
+
356
+ # The doors COMPOSE: add the traces tap in THEIR setup and foam receives
357
+ # traces too — the tap rides THEIR tracer, and trace ids correlate with
358
+ # foam-exported logs/metrics through the shared context:
359
+ their_provider.add_span_processor(Foam::Otel.create_ingest_span_processor(
360
+ token: ENV.fetch("FOAM_OTEL_TOKEN"), environment: ENV.fetch("APP_ENV")))
361
+ # (no double-claim warning: init() serves metrics+logs, the tap serves traces)
203
362
  ```
204
363
 
205
364
  ### Scenario 6 — tests / CI (fully inert, no token needed)
@@ -236,6 +395,143 @@ end
236
395
 
237
396
  ---
238
397
 
398
+ ## Door 2 — the ingest entries
399
+
400
+ When a foreign OTel SDK owns a signal (scenario 4), the ingest entries are
401
+ how that pipeline's data ALSO reaches foam: constructed instances you add —
402
+ one line per claimed signal — to the CUSTOMER'S OWN OTel setup. Additive
403
+ readers only: they never mutate their data, their resource, or their export.
404
+
405
+ > **All three entries are SHIPPED.** The traces entry rides the stable 1.x SDK;
406
+ > the log and metric entries ride upstream's pre-1.0 SDKs (logs "development",
407
+ > metrics alpha), pinned to this gem's capped ranges (GOTCHAS R2/R5) so a 0.x
408
+ > drift is a deliberate re-prove, never a silent change. The pre-1.0 SDK
409
+ > maturity is a version-pin caveat, not a deferral — the entries are available
410
+ > and supported today.
411
+ >
412
+ > **Version-pin posture:** because the log and metric taps ride those pre-1.0
413
+ > upstream SDKs, they are version-pinned and re-proven on every 0.x bump
414
+ > rather than assumed stable — shipped and supported either way.
415
+
416
+ ### The three entries
417
+
418
+ All three share one signature (`token:` and `environment:` required; the
419
+ redaction kwargs are the ONLY keys that tap redacts — opt-in, none by default,
420
+ scoped to that tap; `diagnostics:` is tap-scoped narration). Construction
421
+ validates loudly at boot and NEVER
422
+ throws afterwards — a failure inside a tap can never break their pipeline.
423
+ Fleet canon mapping (pinned by `spec/export_surface_spec.rb`):
424
+ `create_ingest_span_processor` = `createFoamIngestSpanProcessor`,
425
+ `create_ingest_log_record_processor` = `createFoamIngestLogRecordProcessor`,
426
+ `create_ingest_metric_reader` = `createFoamIngestMetricReader`.
427
+
428
+ **Traces** — a SpanProcessor for THEIR TracerProvider (scenario 4 snippet):
429
+
430
+ ```ruby
431
+ processor = Foam::Otel.create_ingest_span_processor(
432
+ token: ENV.fetch("FOAM_OTEL_TOKEN"), environment: ENV.fetch("APP_ENV"),
433
+ redact_keys: %w[voucher_code], # optional: extra secret keys, this tap only
434
+ redact_pii_keys: %w[customer_email], # optional: your PII enumeration, this tap only
435
+ diagnostics: false) # optional: per-tap [foam] narration
436
+ their_tracer_provider.add_span_processor(processor)
437
+ # returns a new instance per call; no-op inert instance under OTEL_SDK_DISABLED=true;
438
+ # raises ArgumentError at construction for a blank token/environment
439
+ ```
440
+
441
+ **Logs** — a LogRecordProcessor for THEIR LoggerProvider. The tap's
442
+ `redact_keys`/`redact_pii_keys` (if any) match over record attributes and keys
443
+ nested in structured bodies; free-text bodies ride RAW (no value-pattern pass):
444
+
445
+ ```ruby
446
+ provider = OpenTelemetry::SDK::Logs::LoggerProvider.new(resource: their_resource)
447
+ provider.add_log_record_processor(their_log_processor)
448
+ provider.add_log_record_processor(Foam::Otel.create_ingest_log_record_processor(
449
+ token: ENV.fetch("FOAM_OTEL_TOKEN"), environment: ENV.fetch("APP_ENV")))
450
+ OpenTelemetry.logger_provider = provider
451
+ ```
452
+
453
+ **Metrics** — a PeriodicMetricReader for THEIR MeterProvider. Attach order is
454
+ forgiving: `add_metric_reader` back-fills instruments created before the tap
455
+ attached. Foam's reader collects on its own schedule beside theirs:
456
+
457
+ ```ruby
458
+ OpenTelemetry.meter_provider.add_metric_reader(Foam::Otel.create_ingest_metric_reader(
459
+ token: ENV.fetch("FOAM_OTEL_TOKEN"), environment: ENV.fetch("APP_ENV")))
460
+ ```
461
+
462
+ ### Door 2 — the required loop step
463
+
464
+ The tap's exports to foam are outbound HTTP calls THEIR instrumentation will
465
+ span, and their pipeline hands those spans back to the tap — a feedback loop
466
+ in THEIR pipeline. Three layers keep it closed:
467
+
468
+ 1. **The SDK suppression context** (automatic): foam's exporters send inside
469
+ `untraced`, which the official http instrumentations respect.
470
+ 2. **REQUIRED — suppress foam's endpoint host in THEIR http instrumentation**
471
+ (belt for non-contrib patches and older contrib versions): add
472
+ `untraced_hosts: ["otel.api.foam.ai"]` to their Net::HTTP/Excon config
473
+ (scenario 4 snippet). When the `OTEL_EXPORTER_OTLP_ENDPOINT` override is
474
+ active, list THAT host instead. Faraday/HTTP (httprb)/HTTPX cannot take
475
+ the option, but they also never see the tap's exports — foam's exporters
476
+ use Net::HTTP only.
477
+ 3. **The echo filter** (structural backstop): a client-kind span naming the
478
+ ACTIVE foam export host is telemetry about foam's own export call — the
479
+ tap refuses to re-export its own echo, with one loud warning naming the
480
+ missing step. Their pipeline keeps that span untouched.
481
+
482
+ ### Identity on the wire
483
+
484
+ | | their pipeline | foam's copy |
485
+ | --- | --- | --- |
486
+ | `service.name` / `service.version` / `service.instance.id` | theirs | theirs, untouched (the data carries THEIR identity) |
487
+ | all attributes / events / bodies | raw, byte-identical | your per-tap `redact_keys`/`redact_pii_keys` applied (default: none → identical to theirs) |
488
+ | `deployment.environment.name` | theirs (if any) | the tap's `environment:` argument (stamped at export time) |
489
+ | `foam.ingest.tier` | absent | `"external"` (the wire-contract marker) |
490
+ | `telemetry.distro.name` / `telemetry.distro.version` | absent | `"foam"` / the gem version (support reads the tap's presence and version from telemetry alone) |
491
+
492
+ The stamp is merged onto FOAM'S COPY at export serialization — never onto
493
+ the shared objects; it changes exactly those four keys and nothing else.
494
+
495
+ ### External-tier caveats (recorded, never fought)
496
+
497
+ - THEIR sampler gates what the traces tap sees (unsampled spans never reach
498
+ processors).
499
+ - THEIR Views shape what the metrics tap receives; per-reader temporality is
500
+ not a Ruby knob — aggregations default cumulative (or the operator's
501
+ `OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE`), and cumulative totals
502
+ start at attach time.
503
+ - Lifecycle rides THEIR pipeline: their provider's `shutdown`/`force_flush`
504
+ drives the tap's. Serverless mirror: flush THEIR provider at handler end
505
+ (the platform freezes the process before batch timers fire).
506
+ - Kill switch and endpoint override apply to taps exactly as to `init`:
507
+ `OTEL_SDK_DISABLED=true` at construction → inert instance + one warning;
508
+ `OTEL_EXPORTER_OTLP_ENDPOINT` honored with the loud warning; per-signal
509
+ OTLP endpoint variants stay INERT + warned.
510
+
511
+ ### Puma / Unicorn note (door 2)
512
+
513
+ Construct taps where THEIR setup runs — with `SDK.configure` in an
514
+ initializer under cluster preload, taps are inherited by workers. The span
515
+ and log taps self-heal in the child (the stock batch processors reset on
516
+ first use after a fork); the metrics tap is restarted by foam's own
517
+ `Process._fork` hook, which the factory installs (a separate registry from
518
+ `init`'s own readers, so a per-worker re-init never orphans a tap — GOTCHAS
519
+ R7). `Foam::Otel.after_fork!` covers both if you wire it manually.
520
+
521
+ ### Tests / CI
522
+
523
+ Gate the add-line on the same condition you would give `enabled:` — the
524
+ placement line IS the switch (no foam-invented env var):
525
+
526
+ ```ruby
527
+ if ENV["APP_ENV"] == "production"
528
+ c.add_span_processor(Foam::Otel.create_ingest_span_processor(
529
+ token: ENV.fetch("FOAM_OTEL_TOKEN"), environment: ENV.fetch("APP_ENV")))
530
+ end
531
+ ```
532
+
533
+ ---
534
+
239
535
  ## Failure modes — what happens and where the warning appears
240
536
 
241
537
  | Situation | Behavior |
@@ -244,8 +540,13 @@ end
244
540
  | `enabled: true` but blank `token` | raises `ArgumentError` at boot |
245
541
  | `enabled: false` | fully inert; helpers no-op silently |
246
542
  | missing `version` | `[foam]` warning, continues |
247
- | a foreign OTel SDK owns a signal | `[foam]` warning once, naming the owner; foam dark for that signal |
543
+ | a foreign OTel SDK owns a signal | `[foam]` warning once, naming the owner AND the matching door-2 ingest entry; foam dark for that signal |
248
544
  | `init` called twice | `[foam]` warning; returns the first instance (providers are process-global) |
545
+ | ingest entry with blank `token:`/`environment:` | raises `ArgumentError` at construction (on your machine) |
546
+ | ingest entry under `OTEL_SDK_DISABLED=true` | inert no-op instance + one `[foam]` warning; their setup line never crashes |
547
+ | ingest tap wired onto foam's OWN pipeline | `[foam]` wiring-bug warning once; the tap STANDS DOWN (zero duplicates) |
548
+ | `init()` exports a signal AND a tap on a separate foreign pipeline | `[foam]` wiring-bug warning once; both keep flowing (`foam.ingest.tier` distinguishes them) |
549
+ | their instrumentation traces foam's export calls (missing loop step) | echo filter refuses foam's own echo + one `[foam]` warning naming the required `untraced_hosts` step |
249
550
 
250
551
  ---
251
552
 
@@ -253,11 +554,14 @@ end
253
554
 
254
555
  | Var | Posture |
255
556
  | --- | --- |
256
- | `OTEL_SDK_DISABLED=true` | HONORED — full kill switch, supersedes `enabled: true` (exact string `true`). |
257
- | `OTEL_EXPORTER_OTLP_ENDPOINT` | HONORED — the one operator-level override of the pinned fleet endpoint (for foam's own conformance rig / enterprise egress). Active → loud `[foam]` warning naming the destination. |
557
+ | `OTEL_SDK_DISABLED=true` | HONORED — full kill switch, supersedes `enabled: true` (matched trimmed + case-insensitive, so `TRUE` / ` true ` also kill — the fleet's one emergency switch behaves the same across languages). Door-2 ingest entries honor it too: construction returns an inert instance (read once at construction; changing it implies a restart). |
558
+ | `OTEL_EXPORTER_OTLP_ENDPOINT` | HONORED — the one operator-level override of the pinned fleet endpoint (for foam's own conformance rig / enterprise egress). Active → loud `[foam]` warning naming the destination. Applies to door-2 taps identically (and moves the host the required loop step must name). |
258
559
  | `OTEL_PROPAGATORS=none` | HONORED — turns trace propagation OFF (links lost) while telemetry keeps flowing; warns. Any other value warns and is ignored (foam's propagator set is fixed: W3C tracecontext + baggage). |
259
560
  | `OTEL_BSP_SCHEDULE_DELAY` / `OTEL_BLRP_SCHEDULE_DELAY` / `OTEL_METRIC_EXPORT_INTERVAL` | HONORED — batch cadence, read natively by the upstream SDK. |
260
- | every other `OTEL_*` | INERT — `OTEL_SERVICE_NAME` / `OTEL_RESOURCE_ATTRIBUTES` never override the init-declared identity; sampler/exporter vars are ignored. |
561
+ | `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | INERT — foam wires ONE resolved endpoint into all three exporters explicitly, so per-signal endpoint vars never redirect (or split) foam's export. Set → loud `[foam]` warning that it is inert. |
562
+ | `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` | INERT — as above (warns when set). |
563
+ | `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` | INERT — as above (warns when set). |
564
+ | every other `OTEL_*` | INERT — `OTEL_SERVICE_NAME` / `OTEL_RESOURCE_ATTRIBUTES` never override the init-declared identity; sampler/exporter/span-limit vars are ignored. |
261
565
 
262
566
  ---
263
567
 
@@ -269,6 +573,12 @@ end
269
573
  - **No data at all** → set `diagnostics: true` and read the `[foam]` lines; check
270
574
  `init` runs before your app code, that `enabled` is true, and that a token is
271
575
  wired.
576
+ - **No door-2 data** → check, in order: `OTEL_SDK_DISABLED` unset (a killed tap
577
+ is inert and warned once at boot); the tap is on the RIGHT provider (the one
578
+ their setup actually registers — a wiring-bug warning at first traffic means
579
+ self-ingest stand-down); the `[foam]` wiring-bug/echo warnings in the boot
580
+ log; construct with `diagnostics: true` for per-export narration; for
581
+ metrics under Puma, see the Door 2 Puma note.
272
582
  - **A downstream rejects `traceparent`** → set `OTEL_PROPAGATORS=none` to stop
273
583
  injection while keeping telemetry flowing.
274
584