foam-otel 1.2.1 → 1.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4e10efb9aaacdda1498a19efaa4af8338afbb98d89880cc6f662aa99205aab59
-  data.tar.gz: 436060cbcfc7ed33c171ab230a60d4f8b62a882ce46be22bcd81e9c497edbc84
+  metadata.gz: 180e8af99c1f5c9bfc35de221c4985fb3ebac7b52ddf6fc99d6552a52aaea836
+  data.tar.gz: 14a5d81c673af1a20a75e483e076d2c6b1820fa14e5a587e2a23451da3cccf77
 SHA512:
-  metadata.gz: 528f57356698e39e691b08bcc0b6f00bc409ad133afca0ac0bdeeb2d0caffbc052a7a19d7fe63b0285230eb16cd26117ede4bc0824c7c0f31377a4a983ad7458
-  data.tar.gz: 9e445e3ce3924ff924afd66e98c35b58fe536ae2d546a0f151d3d870f9359974b3bbfd781871fd39fef67d3d509573cab7c2ce573531067eedf3623943967f32
+  metadata.gz: a307d175e903abbbbd1c382e65694664f47aba18408fc7153f9a43e9fb72f8131fedcfc71a59e16ee4f7446dee0f3460233883f96a7048e1284fff75b54e16c7
+  data.tar.gz: 7322e0a91e04092d494150ad9b003e4a6fd61b23ebc7468a8c715eec2d5ab8aac64d1edc4c9fb3cb1a844c3d3a56c167b9bbf05bed2bb1d4cb491eba20695f5d

data/GOTCHAS.md

@@ -240,34 +240,48 @@ anthropic 1.59.0, gemini-ai 4.3.0, ruby_llm 1.16.0).
   `@attributes = validated_attributes(@attributes).freeze` then sets `@ended`
   (opentelemetry-sdk `trace/span.rb:277,280`); `to_span_data` returns the frozen
   `@attributes` by reference (`trace/span.rb:296-307`).
-- **Opt-in redaction (holistic redesign, 2026-07-26)**: foam does NO redaction by
-  default — with no `redact_keys`/`redact_pii_keys` every value is captured RAW
-  (no secret floor, no value-pattern masking). When keys ARE configured, foam
-  applies the key pass at the EXPORTER boundary — a `RedactingSpanExporter`/
-  `RedactingLogRecordExporter` that rebuilds the mutable `SpanData`/
-  `LogRecordData` Structs with masked attributes/events/body before
-  serialization. With no keys configured the exporter is a true pass-through.
-- **Divergence (NARROWED)**: when keys are configured foam's HELPERS apply the
-  key pass AT CAPTURE (`api.rb` `stringify`/`set_attribute`/`log`; `metrics.rb`
-  always did), so a tenant `additional_*` instance receives foam-helper data
+- **Redaction posture (holistic redesign 2026-07-26, amended same day by the
+  credential-floor ruling)**: foam does NO customer redaction by default — with
+  no `redact_keys`/`redact_pii_keys` every value is captured RAW (no
+  value-pattern masking) EXCEPT the ALWAYS-ON CREDENTIAL FLOOR: the frozen
+  header/key name list of `contract/credential-denylist.json` masks to
+  `[REDACTED]` unconditionally, with no off switch (README "The default
+  credential denylist"; design doc `docs/decisions/credential-denylist-design.md`).
+  The floor + any configured keys run at the EXPORTER boundary — a
+  `RedactingSpanExporter`/`RedactingLogRecordExporter` that rebuilds the
+  mutable `SpanData`/`LogRecordData` Structs with masked
+  attributes/events/body before serialization. The pass runs unconditionally
+  (the floor has no off state); values under unmatched names ride through raw.
+- **Divergence (CLOSED in 1.3.0)**: foam's HELPERS apply the floor + key pass
+  AT CAPTURE (`api.rb` `stringify`/`set_attribute`/`log`; `metrics.rb` always
+  did), so a tenant `additional_*` instance receives foam-helper data
   ALREADY-MASKED (rule 18 C.3) — the pass is idempotent, so the wire value is
-  unchanged. The residual divergence is THIRD-PARTY INSTRUMENTATION attributes
-  only: those freeze at finish before any processor runs, so a tenant span
-  processor sees them as foam captured them and foam can only apply the key pass
-  at the exporter boundary. Ruby offers no safe pre-freeze hook — the
-  `on_finishing` hook runs inside the span mutex, so a processor calling
-  `set_attribute` there would deadlock. Foam's own export applies the configured
-  key pass, and the exporters FAIL CLOSED once a pass has begun: a struct whose
-  rebuild raises is DROPPED with a loud warning (`Redaction.mask_*` return nil on
-  hard failure), never exported half-masked.
-- **Mitigation**: capture-time key pass in `lib/foam/otel/api.rb` +
+  unchanged. THIRD-PARTY INSTRUMENTATION attributes freeze at finish before
+  any processor runs (no safe pre-freeze hook — `on_finishing` runs inside
+  the span mutex, so a processor calling `set_attribute` there would
+  deadlock), so they CANNOT be masked in place — instead the guarded tenant
+  wrappers (`pipelines.rb` `GuardedSpanProcessor#on_finish` /
+  `GuardedLogRecordProcessor#on_emit`) hand tenants a READ-ONLY MASKED VIEW
+  (`MaskedSpanView`/`MaskedLogRecordView`, served through the same
+  `Redaction.mask_span_data`/`mask_attributes`/`mask_body` pass foam's export
+  applies) — a tenant never sees a raw floor (or configured-key) value, and a
+  span whose masked view cannot be built is WITHHELD loudly, never handed
+  over raw. Foam's own export applies the same pass, and the exporters FAIL
+  CLOSED once the pass has begun: a struct whose rebuild raises is DROPPED
+  with a loud warning (`Redaction.mask_*` return nil on hard failure), never
+  exported half-masked.
+- **Mitigation**: capture-time pass in `lib/foam/otel/api.rb` + the masked
+  tenant view in `lib/foam/otel/pipelines.rb` +
   `lib/foam/otel/redacting_exporter.rb` + `lib/foam/otel/redaction.rb`.
-- **Test**: `spec/tenant_seam_spec.rb` (tenant processors receive helper-set
-  LISTED keys ALREADY-MASKED, free-text bodies RAW, the byte-identical
-  with/without-tenant proofs for spans, logs, AND metrics),
-  `spec/redacting_exporter_spec.rb` (opt-in masking on the wire; "a POISONED
-  SpanData (dup raises) is DROPPED — never exported raw"), `spec/redaction_spec.rb`
-  (default zero-redaction + the two opt-in key lists; capture+exporter idempotence).
+- **Test**: `spec/credential_floor_spec.rb` + `spec/credential_floor_wire_spec.rb`
+  (the floor per signal/door; the tenant masked-view regression gate — a
+  third-party-set `authorization` attribute never reaches a tenant raw),
+  `spec/tenant_seam_spec.rb` (tenant processors receive helper-set LISTED keys
+  ALREADY-MASKED, free-text bodies RAW, the byte-identical with/without-tenant
+  proofs for spans, logs, AND metrics), `spec/redacting_exporter_spec.rb`
+  (masking on the wire; "a POISONED SpanData (dup raises) is DROPPED — never
+  exported raw"), `spec/redaction_spec.rb` (raw-above-the-floor defaults + the
+  two opt-in key lists; capture+exporter idempotence).
 
 ## F2: OTEL_SDK_DISABLED and OTEL_PROPAGATORS are only honored by SDK.configure
 
@@ -568,6 +582,59 @@ anthropic 1.59.0, gemini-ai 4.3.0, ruby_llm 1.16.0).
   rack end-to-end with real `traceparent`+`baggage` headers, the no-op and
   never-raise cases.
 
+## F13: The value-pattern secret layer + coverage fix + TLS pin (1.4.0, security ruling 2026-07-27)
+
+The credential floor masks by NAME and is value-shape-blind; a live secret
+under an innocuous key (`note`, `detail`) sailed past it. Foam telemetry is
+read downstream by LLMs, so ONE leaked credential is prompt-injection
+exfiltratable — the value-pattern secret layer is the required second control
+(`docs/decisions/security-fixes-design.md` CONTRACT V).
+
+- **What / where**: `Redaction.scan_value_secrets` masks credential-SHAPED
+  spans (`SECRET_VALUE_RULES` + the PEM/PuTTY linear scans + the H1/H2
+  heuristics) to `[REDACTED]` in EVERY leaf string the engine walks —
+  attribute values, `status.message`, exception events, log bodies (string +
+  structured leaves), SQL, LLM content (BEFORE serialization, F-RB2), URL
+  query AND fragment pair values (Contract C1), resource identity strings
+  (once at build), exemplar `filtered_attributes`. It rides the SAME exporter-
+  boundary seam as the floor — one pass, ordered floor → C1 tokenizer → value
+  scan → user keys, each hit terminal (`redaction.rb` `mask_one`/
+  `redact_leaf_string`). METRIC DATAPOINT ATTRIBUTES are the one mandated
+  exemption (`metrics.rb`/`mask_metric_data` pass `value_scan: false`) —
+  low-cardinality by construction; a value mask there would break aggregation
+  dimensions.
+- **Non-disableable Tier 1**: the rules are frozen module constants, not
+  Config — no config path can narrow them (same posture as the floor). ONLY
+  the H1/H2 heuristic tier honors `secret_heuristics: false`, logged loud at
+  init.
+- **ReDoS-bounded / fail-closed**: compile-once, a lowercase-literal anchor
+  pre-filter (no anchor → zero regexes), a 256 KiB value cap (oversize +
+  anchor ⇒ whole-value mask, never regex-scanned), a match-flood cap, a
+  256-pair query/fragment cap, a per-Regexp `timeout:` on Ruby ≥ 3.2. ANY
+  scanner fault (timeout included) masks the value, never a raw pass-through.
+- **F-RB1 TLS pin (CWE-295)**: the vendored OTLP exporter honored
+  `OTEL_RUBY_EXPORTER_OTLP_SSL_VERIFY_NONE` as a PRESENCE check — one env var
+  would silently strip cert verification off the token-bearing export
+  connection. Every exporter constructor (door 1 `pipelines.rb`, door 2
+  `ingest.rb`) now pins `ssl_verify_mode: OpenSSL::SSL::VERIFY_PEER`; the var
+  is INERT and warned about (`init.rb` `warn_inert_ssl_verify_none`). There is
+  no foam option to weaken it.
+- **Mitigation**: `lib/foam/otel/constants.rb` (the frozen ruleset),
+  `lib/foam/otel/redaction.rb` (the engine + coverage tokenizer),
+  `lib/foam/otel/pipelines.rb` + `lib/foam/otel/ingest.rb` (the TLS pin),
+  `lib/foam/otel/llm.rb` (C4 pre-serialization masking).
+- **Sources**: `docs/decisions/security-fixes-design.md`;
+  gitleaks/detect-secrets/secretlint provenance in `THIRD-PARTY-NOTICES`;
+  vendored exporter presence-check at
+  opentelemetry-exporter-otlp `exporter.rb:39-47`.
+- **Test**: `spec/value_secret_layer_spec.rb` (the per-rule corpus, caps,
+  precedence, coverage contract, off-surface pin, structural gates),
+  `spec/security_regressions_spec.rb` (F-RB1 + F-RB2 verbatim repros),
+  `spec/vuln_suite_spec.rb` (the adversarial exfil/ReDoS/bypass suite +
+  mutation spot-checks).
+
+---
+
 ## General gotchas (applicable to Ruby)
 
 - **G1 — init after target import / pre-init no-op**: the API's proxy providers
@@ -623,8 +690,10 @@ anthropic 1.59.0, gemini-ai 4.3.0, ruby_llm 1.16.0).
   mitigation (foam never rewrites or drops metric dimensions — rule 4/26);
   the mitigation is the README metrics warning with the classic bad example —
   *Documented* (README "Metrics" section). *Test*: `spec/metrics_spec.rb`
-  ("captures a metric attribute RAW by default; masks it only when the key is in
-  redact_keys" — pins that unlisted attributes pass through VERBATIM).
+  ("captures a NON-floor metric attribute RAW by default; masks it only when
+  the key is in redact_keys" — pins that unlisted, non-floor attributes pass
+  through VERBATIM; the credential floor's frozen names are the one always-on
+  exception, `spec/credential_floor_wire_spec.rb`).
 - **G8 — never-ended spans leak memory**: a span started and never ended holds
   memory and context forever (and never exports — the batch processor only
   sees FINISHED spans, `on_finish`). `Foam::Otel.span {}` always ends the span

data/README.md

@@ -4,9 +4,15 @@ Foam's OpenTelemetry core for Ruby services. A thin, safe wrapper over the
 official OpenTelemetry libraries: foam owns the pipeline (providers, batch
 processors, OTLP export to the foam fleet endpoint), turns on automatic
 tier-1/2 instrumentation, and hands you a small set of never-throw helpers —
-and nothing else. **Redaction is fully opt-in: by default foam captures every
-value RAW** (no secret floor, no value-pattern masking); you enumerate the
-fields to mask via `redact_keys`/`redact_pii_keys`.
+and nothing else. **Redaction is fully opt-in above two always-on exceptions:
+by default foam captures every value RAW** (no PII preset) **except (1) the
+credential floor** — a fixed list of credential/secret header and field NAMES
+always masked to `[REDACTED]` (see "The default credential denylist" below) —
+**and (2) the value-pattern secret layer** — credential-SHAPED value spans
+(cloud keys, tokens, JWTs, private-key blocks, `user:pass@` URLs…) masked to
+`[REDACTED]` regardless of field name (see "The value-pattern secret layer"
+below); you enumerate any further fields to mask via
+`redact_keys`/`redact_pii_keys`.
 
 Built to `docs/BASE_PACKAGE_SPEC.md`. This README is the manual you integrate
 from.
@@ -75,10 +81,12 @@ drop-in upstream metrics gem, so foam ships its own collector),
 **session stitching** (inbound `baggage: session.id` from the foam browser
 package is stamped onto every span and log record), and the **LLM shims**
 (below). Every inbound request, DB call, HTTP call, job, log line, and model
-call now flows to foam. **Redaction is opt-in: by default every value —
-attributes, URLs/query strings, DB statement text, log bodies, LLM content —
-is captured RAW.** Pass `redact_keys`/`redact_pii_keys` to mask or erase
-specific fields (see below).
+call now flows to foam. **Redaction is opt-in above the credential floor: by
+default every value — attributes, URLs/query strings, DB statement text, log
+bodies, LLM content — is captured RAW, except values whose NAME is on the
+always-on credential denylist (masked `[REDACTED]`; see "The default
+credential denylist" below).** Pass `redact_keys`/`redact_pii_keys` to mask
+or erase further specific fields (see below).
 
 > **Sidekiq trace shape** (GOTCHAS G16): the bundled instrumentation keeps
 > its upstream default `propagation_style: :link` — a performed job runs as
@@ -134,14 +142,15 @@ API):** the allowlist must include BOTH headers —
 | `enabled:` | Boolean | yes | — | The config switch. `false` = fully inert (no SDK, no providers, no network, helpers no-op). `true` = export, in EVERY environment. No default — you write the logic. |
 | `token:` | String | when enabled | — | `Authorization: Bearer` for export. Required (and validated) only when `enabled: true`; wired explicitly from your secret source — never an env fallback read by the gem. **Fleet convention: read it from the `FOAM_OTEL_TOKEN` env var.** Blank while enabled → raises at boot. |
 | `version:` | String | no | nil | `service.version`, verbatim (git SHA recommended). Missing → warns and continues. Never detected at runtime. |
-| `redact_keys:` | Array<String> | no | `[]` | The ONLY field names tail-masked (shape-preserving, e.g. `********f456`). Opt-in — empty (the default) means NO masking. Matches by case-insensitive substring, including keys nested in structured values and query-string keys. |
-| `redact_pii_keys:` | Array<String> | no | `[]` | Field names fully erased to `[REDACTED]` (no tail). Opt-in — empty (the default) means NO erasure. Foam ships no preset; this is your enumeration. |
+| `redact_keys:` | Array<String> | no | `[]` | The ONLY field names tail-masked (shape-preserving, e.g. `********f456`). Opt-in and ADDITIVE on top of the always-on credential floor — empty (the default) means no masking beyond the floor. Matches by case-insensitive substring, including keys nested in structured values and query-string keys. A key that is also on the floor stays fully `[REDACTED]` (the floor wins; it is never downgraded to a tail). |
+| `redact_pii_keys:` | Array<String> | no | `[]` | Field names fully erased to `[REDACTED]` (no tail). Opt-in and additive above the floor — empty (the default) means no erasure beyond the floor. Foam ships no PII preset; this is your enumeration. |
 | `additional_span_processors:` | Array | no | `[]` | Tenant seam: constructed SpanProcessor instances added to foam's pipeline (additive; never replace foam's export). See coexistence. |
 | `additional_log_record_processors:` | Array | no | `[]` | Tenant seam, logs. |
 | `additional_metric_readers:` | Array | no | `[]` | Tenant seam, metrics. |
 | `additional_instrumentations:` | Array | no | `[]` | Constructed tier-2 instrumentation instances to register (fault-isolated: one that throws is skipped with a `[foam]` warning). |
 | `ignored_outbound_hosts:` | Array<String> | no | `[]` | Hosts whose outbound calls produce no spans — EXTENDS the built-in export-loop guard. For a co-resident agent's intake or a tenant exporter's endpoint. Applies to the clients whose official instrumentation supports host suppression: **Net::HTTP and Excon**. Faraday/HTTP (httprb)/HTTPX have no upstream `untraced_hosts` option — foam warns loudly at init when one of those is bundled (GOTCHAS G13); foam's own export loop is guarded for every client regardless. |
 | `diagnostics:` | Boolean | no | false | Verbose `[foam]` self-reporting of init/health. Warnings and errors are always loud regardless. |
+| `secret_heuristics:` | Boolean | no | true | The value-pattern secret layer's HEURISTIC tier (generic keyword+entropy detection, see "The value-pattern secret layer"). `false` disables ONLY the heuristics — the named provider patterns, the credential floor and the redaction-coverage contract have no off switch. Disabling logs one loud `[foam]` line at init (an explicit, audited decision for telemetry whose legitimate values collide with the heuristics). |
 
 Examples for the remaining options (each is runnable as-is):
 
@@ -152,6 +161,16 @@ Foam::Otel.init(name: "checkout-api", environment: env, enabled: true, token: EN
                 redact_pii_keys: %w[customer_email full_name])               # fully [REDACTED]
 ```
 
+```ruby
+# Only if your legitimate telemetry collides with the generic secret
+# heuristics (e.g. base64 content-addressed ids masked as high-entropy
+# tokens): disable the HEURISTIC tier only. The named provider patterns and
+# the credential floor stay on — they have no off switch. Logs one loud
+# [foam] line.
+Foam::Otel.init(name: "checkout-api", environment: env, enabled: true, token: ENV.fetch("FOAM_OTEL_TOKEN"),
+                secret_heuristics: false)
+```
+
 ```ruby
 # Constructed tier-2 instrumentation instances (fault-isolated; one that
 # throws is skipped with a [foam] warning, never a crashed boot):
@@ -193,6 +212,159 @@ override is the operator-level `OTEL_EXPORTER_OTLP_ENDPOINT` env var (below).
 
 ---
 
+## The default credential denylist (the always-on floor)
+
+**Fleet ruling 2026-07-26** (binding design:
+`docs/decisions/credential-denylist-design.md`; frozen fixture:
+`contract/credential-denylist.json`). This is the ONE exception to foam's
+raw-by-default capture, and it has **no off switch** — no init option, no env
+var, and no door-2 parameter can disable, shrink, or re-spell it.
+
+**What is masked.** The VALUE of every header/attribute/field whose NAME is on
+the frozen list, in every signal (spans — events and links included — logs,
+metrics), on door 1, on all three door-2 ingest taps, and in the view handed
+to tenant `additional_*` processors:
+
+- **(a) the seven credential headers**, matched as header names wherever
+  headers are captured — including the semconv
+  `http.request.header.<name>` / `http.response.header.<name>` span-attribute
+  forms: `authorization`, `proxy-authorization`, `cookie`, `set-cookie`,
+  `x-api-key`, `x-auth-token`, `www-authenticate`;
+- **(b) the frozen 52-entry key list** of
+  `contract/credential-denylist.json` (the deduplicated union of those seven,
+  sentry-python's default denylists, and foam's documented reference roots —
+  `password`, `token`, `secret`, `api_key`, `ssn`, `jwt`, `private_key`,
+  `connect.sid`, `phpsessid`, …), matched as attribute/field names at every
+  depth the engine walks, and per-key in URL query strings
+  (`?token=x&user=bob` → `token=[REDACTED]&user=bob`).
+
+**The matching rule.** Case-insensitive, dash/underscore-normalized **EXACT
+name-equality — never substring**: `authorization` masks the header/field
+`authorization` (any casing, `-`≡`_`) and nothing else — `authorization_url`,
+`x_api_key_id`, `secretary`, and `session_count` all ride RAW. Dotted entries
+match verbatim and whole (`connect.sid` masks; `user.session` does not). This
+is deliberately STRICTER matching than your own `redact_keys` lists, which
+keep their documented substring behavior.
+
+**The mask.** The full literal `[REDACTED]` — no tail, no length
+preservation. These are credentials, not debug aids. Semconv header
+attributes (string arrays) mask per element, preserving arity.
+
+**Precedence.** The floor runs unconditionally BEFORE your
+`redact_keys`/`redact_pii_keys`, which stay purely additive on top. Listing a
+floor name in `redact_keys` does NOT downgrade it to a tail mask — the floor
+is terminal. `enabled: false` stays fully inert exactly as before (nothing
+exports at all — the floor masks data that leaves; inert mode has none).
+
+**Everything else stays RAW — above the floor and the value layer.** The
+floor matches NAMES only, never values. Credential-SHAPED value spans are the
+value-pattern secret layer's job (next section); UNSHAPED values — bodies,
+query values, SQL text, LLM content, and PII under unlisted names — export
+byte-identical raw, exactly as documented everywhere else in this README.
+
+**Why.** Coverage-over-masking remains the mission, but raw `Authorization`
+headers, cookies, and API keys on the wire are a breach in waiting for every
+customer at once — the fleet ruling carves out exactly this list (and only
+this list) as non-negotiable. The list is byte-identical in every foam core
+and gate-checked in CI against the fleet fixture
+(`spec/credential_floor_spec.rb`).
+
+**Fleet param-name canon** (the customer options above the floor, identical
+concept in every core — the floor itself has NO init surface anywhere):
+
+| Concept | js/otel | js/browser | python | ruby | java |
+|---|---|---|---|---|---|
+| secret keys (tail-mask) | `redactKeys` | `redactKeys` | `redact_keys` | `redact_keys:` | `.redactKeys(String…)` |
+| PII keys (full `[REDACTED]`) | `redactPiiKeys` | `redactPiiKeys` | `redact_pii_keys` | `redact_pii_keys:` | `.redactPiiKeys(String…)` |
+
+> **Migration note — 1.3.0 (minor): always-on credential masking.** As of
+> this version foam masks, by default and in every signal, the VALUES of a
+> fixed list of credential/secret header and field NAMES (`authorization`,
+> `cookie`, `set-cookie`, `proxy-authorization`, `x-api-key`, `x-auth-token`,
+> `www-authenticate`, and the 52-name key list in
+> `contract/credential-denylist.json` — Sentry-parity plus foam's documented
+> roots) to the literal `[REDACTED]`. Matching is exact name-equality,
+> case-insensitive, dash/underscore-insensitive — never substring:
+> `authorization_url` is untouched. Everything else still exports RAW exactly
+> as before; `redact_keys`/`redact_pii_keys` are unchanged and additive.
+> There is no off switch — if a dashboard keyed off a raw credential value
+> (it should not have), it will now see `[REDACTED]`. Customers who
+> previously received these header/field values raw stop receiving them.
+
+---
+
+## The value-pattern secret layer (always-on)
+
+The credential floor masks by NAME and is deliberately value-shape-blind — a
+live AWS key under the field name `note` would sail past it. Foam telemetry is
+read downstream by LLMs, so a leaked credential is exfiltratable by prompt
+injection; "no leakage at all" is the bar. The value-pattern secret layer is
+the second always-on control (fleet ruling 2026-07-27,
+`docs/decisions/security-fixes-design.md`):
+
+**What it masks.** Credential-SHAPED value spans, regardless of the field
+name they hide under: AWS access-key ids and keyword-anchored AWS secrets,
+GCP API keys, Azure client/storage secrets, GitHub tokens and fine-grained
+PATs, GitLab tokens, Slack tokens + webhooks, Stripe secret keys
+(publishable `pk_` keys are deliberately NOT masked), JWTs, PEM private-key
+blocks (`-----BEGIN … PRIVATE KEY` — a truncated block masks to the end of
+the value, fail-closed; `PUBLIC KEY`/`CERTIFICATE` blocks never match) and
+PuTTY PPK keys, `scheme://user:pass@` URI credentials (the whole userinfo,
+username included), `Bearer`/`Basic` auth values, and Anthropic/OpenAI API
+keys. The matched span becomes the literal `[REDACTED]` — no tail, no
+prefix, zero reconstructable bytes; surrounding text keeps its diagnostic
+value (`conn to postgres://[REDACTED]@db:5432 refused`).
+
+**The heuristic tier** additionally catches the long tail: a
+keyword-anchored generic pass (`password = hunter2SecretXyz99` inside a log
+body masks its value — gitleaks-derived, entropy-gated, with placeholder/
+stopword/key-context false-positive suppression) and a whole-value
+high-entropy base64 token pass (detect-secrets-derived; pure hex and UUIDs
+are deliberately exempt — trace ids and digests are telemetry's ambient
+vocabulary). Only this tier is disableable: `secret_heuristics: false`
+(one loud `[foam]` line; the named patterns above are not configurable).
+
+**Where it runs.** Every value-bearing string on every export path, both
+doors: span/event/link attribute values, `status.message`, exception
+messages, log bodies (string and structured leaves), SQL (`db.statement`),
+LLM content (BEFORE serialization and again at the boundary), URL query AND
+fragment pair values, baggage-derived attributes, exemplar
+`filtered_attributes`, and resource identity strings (once, at init).
+Metric datapoint attributes (metric labels) are the one exempt surface —
+low-cardinality by construction; a value mask there would break your
+aggregation dimensions.
+
+**Bounded by design.** Compile-once rules, a literal-anchor pre-filter (no
+anchor in the value → zero regexes run), a 256 KiB value cap (an oversized
+value with an anchor hit anywhere masks WHOLE, fail-closed; without one it
+passes untouched), a match-flood cap, a 256-pair query/fragment tokenizer
+cap, and a per-Regexp timeout on Ruby ≥ 3.2. ANY scanner fault masks the
+value rather than shipping it raw.
+
+**Provenance.** The ruleset derives from MIT/Apache-licensed OSS secret
+scanners (gitleaks, detect-secrets, secretlint) — see `THIRD-PARTY-NOTICES`.
+Rules are frozen module constants, byte-consistent across the foam cores,
+changed only by fleet ruling.
+
+> **Migration note — 1.4.0 (minor): value-pattern secret masking + redaction
+> coverage fix + TLS pin.** foam now masks credential-SHAPED values
+> (AWS/GCP/Azure keys, GitHub/GitLab tokens, Slack, Stripe, Anthropic/OpenAI
+> keys, JWTs, private-key blocks, `user:pass@` URI credentials, bearer/basic
+> tokens) to `[REDACTED]` in every exported string value — regardless of
+> field name — plus a generic keyword+entropy heuristic you can disable with
+> `secret_heuristics: false` (the named patterns and the credential floor
+> cannot be disabled). Redaction now also reaches URL FRAGMENTS (OAuth
+> implicit-flow tokens, hash-router params), bare `k=v` pair lists, nested
+> attribute containers (fail-closed), and structured LLM content BEFORE
+> serialization. Query/fragment tokenizing is capped at 256 pairs (the
+> overflow remainder masks whole). Metric labels are not value-scanned. TLS
+> peer verification is now PINNED on every exporter:
+> `OTEL_RUBY_EXPORTER_OTLP_SSL_VERIFY_NONE` is INERT and loudly warned about.
+> If a dashboard keyed off a raw token value (it should not have), it will
+> now see `[REDACTED]`.
+
+---
+
 ## The helpers
 
 All helpers never raise, and no-op silently before `init` and when disabled.
@@ -288,10 +460,13 @@ Foam::Otel.init(name: "checkout-api", environment: env, enabled: true, token: FO
 > Ruby note: when you configure `redact_keys`/`redact_pii_keys`, foam's HELPERS
 > apply the key pass at capture, so a tenant processor receives foam-helper data
 > (span/log helper attributes, log bodies, metric labels) ALREADY masked.
-> Because the Ruby SDK freezes span attributes at finish, a tenant
-> processor sees third-party instrumentation attributes UNMASKED (for those,
-> foam's key pass runs at the exporter boundary — GOTCHAS F1). With no keys
-> configured (the default) nothing is masked on either path.
+> The Ruby SDK freezes span attributes at finish, so historically a tenant
+> processor saw third-party instrumentation attributes UNMASKED — as of 1.3.0
+> that gap is closed: tenant span/log processors are handed a READ-ONLY MASKED
+> VIEW (the credential floor plus your configured keys, the same pass foam's
+> own export applies — GOTCHAS F1). A tenant never sees a raw floor value.
+> With no keys configured (the default) only the credential floor is masked
+> on either path.
 >
 > Loop-guard note: `ignored_outbound_hosts` suppresses spans for tenant/agent
 > egress made through **Net::HTTP or Excon**. A tenant exporter using
@@ -329,9 +504,10 @@ end
 
 **Outcome**: their pipeline keeps working exactly as before — their data,
 their resource, their export, byte-identical — and foam ALSO receives the
-signal, carrying THEIR resource identity plus foam's export-time stamp and
-any per-tap `redact_keys`/`redact_pii_keys` you set (opt-in — none by default;
-details in the Door 2 section).
+signal, carrying THEIR resource identity plus foam's export-time stamp, the
+always-on credential floor applied to foam's copy, and any per-tap
+`redact_keys`/`redact_pii_keys` you set (opt-in — none by default; details in
+the Door 2 section).
 
 **Unhappy paths**:
 
@@ -431,8 +607,10 @@ readers only: they never mutate their data, their resource, or their export.
 ### The three entries
 
 All three share one signature (`token:` and `environment:` required; the
-redaction kwargs are the ONLY keys that tap redacts — opt-in, none by default,
-scoped to that tap; `diagnostics:` is tap-scoped narration). Construction
+redaction kwargs are the only CUSTOMER keys that tap redacts — opt-in, none by
+default, scoped to that tap, additive above the always-on credential floor,
+which every tap applies with zero configuration; `diagnostics:` is tap-scoped
+narration). Construction
 validates loudly at boot and NEVER
 throws afterwards — a failure inside a tap can never break their pipeline.
 Fleet canon mapping (pinned by `spec/export_surface_spec.rb`):
@@ -453,9 +631,10 @@ their_tracer_provider.add_span_processor(processor)
 # raises ArgumentError at construction for a blank token/environment
 ```
 
-**Logs** — a LogRecordProcessor for THEIR LoggerProvider. The tap's
-`redact_keys`/`redact_pii_keys` (if any) match over record attributes and keys
-nested in structured bodies; free-text bodies ride RAW (no value-pattern pass):
+**Logs** — a LogRecordProcessor for THEIR LoggerProvider. The credential
+floor and the tap's `redact_keys`/`redact_pii_keys` (if any) match over
+record attributes and keys nested in structured bodies; free-text bodies ride
+RAW (no value-pattern pass):
 
 ```ruby
 provider = OpenTelemetry::SDK::Logs::LoggerProvider.new(resource: their_resource)
@@ -499,7 +678,7 @@ in THEIR pipeline. Three layers keep it closed:
 | | their pipeline | foam's copy |
 | --- | --- | --- |
 | `service.name` / `service.version` / `service.instance.id` | theirs | theirs, untouched (the data carries THEIR identity) |
-| all attributes / events / bodies | raw, byte-identical | your per-tap `redact_keys`/`redact_pii_keys` applied (default: none → identical to theirs) |
+| all attributes / events / bodies | raw, byte-identical | the credential floor + your per-tap `redact_keys`/`redact_pii_keys` applied (default: floor only — everything off the frozen list identical to theirs) |
 | `deployment.environment.name` | theirs (if any) | the tap's `environment:` argument (stamped at export time) |
 | `foam.ingest.tier` | absent | `"external"` (the wire-contract marker) |
 | `telemetry.distro.name` / `telemetry.distro.version` | absent | `"foam"` / the gem version (support reads the tap's presence and version from telemetry alone) |
@@ -576,6 +755,7 @@ end
 | `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | INERT — foam wires ONE resolved endpoint into all three exporters explicitly, so per-signal endpoint vars never redirect (or split) foam's export. Set → loud `[foam]` warning that it is inert. |
 | `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` | INERT — as above (warns when set). |
 | `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` | INERT — as above (warns when set). |
+| `OTEL_RUBY_EXPORTER_OTLP_SSL_VERIFY_NONE` | INERT — foam pins TLS peer verification (`ssl_verify_mode: VERIFY_PEER`) on every exporter constructor, both doors, so this vendored-exporter knob can never strip certificate verification off the credential-bearing export connection (1.4.0 security fix, CWE-295). Set → loud `[foam]` warning that it is inert; fix the trust chain (CA bundle) instead. |
 | every other `OTEL_*` | INERT — `OTEL_SERVICE_NAME` / `OTEL_RESOURCE_ATTRIBUTES` never override the init-declared identity; sampler/exporter/span-limit vars are ignored. |
 
 ---

data/RESEARCH.md

@@ -144,8 +144,11 @@ statement posture is **captured RAW, never suppressed** (2026-07-26 ruling —
 coverage is the mission), and the upstream `-pg`/`-mysql2`/`-redis`/`-mongo`
 gems all DEFAULT to `db_statement: :obfuscate` — a sanitizer — so foam
 explicitly activates them with `db_statement: :include` (GOTCHAS F9; pinned
-by `spec/floor_spec.rb`). Redaction stays fully opt-in, applied only for keys
-the customer lists in `redact_keys`/`redact_pii_keys`. An FDE can still clamp
+by `spec/floor_spec.rb`). Redaction stays fully opt-in above the always-on
+credential floor (the frozen NAME list of
+`contract/credential-denylist.json` — statement TEXT is never scanned),
+applied only for keys the customer lists in
+`redact_keys`/`redact_pii_keys`. An FDE can still clamp
 `db_statement: :omit`/`:obfuscate` per customer via the standard
 `OTEL_RUBY_INSTRUMENTATION_<NAME>_CONFIG_OPTS` env var when a compliance
 context calls for it — per-customer tightening (rule 18 / Area 4), not the
@@ -210,7 +213,8 @@ foam ships the logs pipeline + `log()` helper (trace-correlated) — the mechani
 rule 20 requires. The stdlib **Logger bridge is LANDED as foam's own code**
 (`lib/foam/otel/logger_bridge.rb`, console-logs ruling 2026-07-26): one
 prepend on `Logger#add` turns every message a host logger accepts into a
-trace-correlated OTel log record — body RAW (opt-in key-masking only),
+trace-correlated OTel log record — body RAW (name-keyed masking only: the
+always-on credential floor plus any customer keys; free text never scanned),
 severity mapped, loop-guarded (the OTel SDK's internal logger is skipped by
 identity + a thread-local re-entrancy flag), gated on foam owning the logs
 slot, and never able to break the host's log call (GOTCHAS F10; tests
@@ -226,7 +230,8 @@ The official contrib `opentelemetry-instrumentation-logger 0.4.0` bridges the
 same `Logger#add` seam and is deliberately NOT bundled — census disposition
 EXCLUDED/superseded (§3): foam's bridge additionally holds the lazy-block
 contract, the OTel-internal-logger identity guard, the re-entrancy flag, the
-logs-slot gate, and the opt-in redaction path, and shipping/sweeping both
+logs-slot gate, and the redaction path (credential floor + opt-in keys), and
+shipping/sweeping both
 would double-export every host log line (rule 12). init's sweep pins the
 contrib gem disabled even when an FDE bundles it (`spec/floor_spec.rb`).
 
@@ -289,38 +294,59 @@ https://opentelemetry.io/docs/specs/semconv/gen-ai/aws-bedrock/ ; traceloop Ruby
 
 ## 6. Redaction design record (rule 14, as amended 2026-07-26)
 
-**Redaction is FULLY OPT-IN — the default is ZERO redaction** (owner ruling
-2026-07-26, holistic-redesign decision log; `lib/foam/otel/redaction.rb`).
-With neither `redact_keys` nor `redact_pii_keys` set (or both empty) every
-value is captured RAW — attributes, headers, bodies, URL query params,
-`db.statement`, log bodies — including secrets and PII. There is NO always-on
-secrets floor and NO value-pattern auto-masking; the package auto-masks
-nothing and ships no preset. The customer owns the privacy/legal posture; the
-FDE reactive clamp is the per-customer valve.
-
-**The retired floor**: earlier revisions shipped an always-on researched
-secrets floor in `constants.rb` (roots + value-pattern pass, sourced from the
-OSS-scrubber consensus — raven-ruby, sentry, Elastic APM, Rails
-filter_parameters, gitleaks et al. — modeled on the Node core). It was
-REMOVED by the 2026-07-26 ruling; the provenance record lives in this file's
-git history and the decision log. Do not reintroduce it without a new ruling.
-
-**The current engine** (`redaction.rb`): redaction runs ONLY over the keys
-the customer enumerates. `redact_keys` matches → tail mask, shape-preserving
-(`********` + last 4 when ≥ 12 chars; the fixed 8-char body never reveals
-length; non-scalars redact in full); `redact_pii_keys` matches → full
-`[REDACTED]`, no tail. Matching is SUBSTRING, case-insensitive, over the
-lowercased key — the customer's lists, nothing inferred. The pass runs at the
-exporter boundary (`redacting_exporter.rb` — the Ruby SDK freezes span
-attributes at finish before any `on_finish` processor runs, so the exporter
-boundary is the seam; see GOTCHAS) and is a true pass-through when no keys
-are configured. The explicit `redact(value)` helper always masks (calling it
-is the opt-in). Fail-closed for listed keys (mask on any error, cycle + depth
-guarded).
-
-Proven by `spec/redaction_spec.rb` (opt-in defaults — nothing masked with no
-keys; both mask shapes; fail-closed) and `spec/redacting_exporter_spec.rb` +
-the conformance app (default-path RAW capture on the wire).
+**Redaction is FULLY OPT-IN above the CREDENTIAL FLOOR** (owner ruling
+2026-07-26, holistic-redesign decision log, AMENDED the same day by the
+credential-floor fleet ruling — `docs/decisions/credential-denylist-design.md`;
+`lib/foam/otel/redaction.rb`). With neither `redact_keys` nor
+`redact_pii_keys` set (or both empty) every value is captured RAW —
+attributes, bodies, URL query params, `db.statement`, log bodies — including
+PII, EXCEPT the one always-on exception: the CREDENTIAL FLOOR masks the
+values of the frozen credential/secret header + key NAME list
+(`contract/credential-denylist.json`: the seven headers + 52 keys, constants
+`CREDENTIAL_HEADER_DENYLIST`/`CREDENTIAL_KEY_DENYLIST` in `constants.rb`) to
+the literal `[REDACTED]`, in every signal, on every door, with NO off switch.
+There is still NO value-pattern auto-masking and NO PII preset; above the
+floor the customer owns the privacy/legal posture; the FDE reactive clamp is
+the per-customer valve.
+
+**The retired floor vs the credential floor**: earlier revisions shipped an
+always-on researched secrets floor in `constants.rb` (roots + value-pattern
+pass, sourced from the OSS-scrubber consensus — raven-ruby, sentry, Elastic
+APM, Rails filter_parameters, gitleaks et al. — modeled on the Node core). It
+was REMOVED by the 2026-07-26 zero-default ruling; the provenance record
+lives in this file's git history and the decision log. The SAME-DAY
+credential-floor ruling is the new fleet ruling that reintroduces exactly ONE
+always-on list — the frozen credential denylist (name-match only, never
+value patterns; rule-14 reference roots absorbed as entries). Its content
+changes only by a new fleet ruling and is CI-gated bit-for-bit against the
+fleet fixture (`spec/credential_floor_spec.rb`).
+
+**The current engine** (`redaction.rb`): the floor predicate
+(`floor_kind`/`floor_match?`) runs FIRST and is TERMINAL — canon-exact
+matching (trim, ASCII-lowercase, `-`→`_` fold; NEVER substring:
+`authorization` never masks `authorization_url`), including the semconv
+`http.{request,response}.header.<name>` prefix peel. Customer redaction then
+runs ONLY over the keys the customer enumerates. `redact_keys` matches → tail
+mask, shape-preserving (`********` + last 4 when ≥ 12 chars; the fixed 8-char
+body never reveals length; non-scalars redact in full); `redact_pii_keys`
+matches → full `[REDACTED]`, no tail. CUSTOMER matching is SUBSTRING,
+case-insensitive, over the lowercased key — the customer's lists, nothing
+inferred (the FLOOR deliberately does NOT ride the substring matcher — it
+would over-match). The pass runs at the exporter boundary
+(`redacting_exporter.rb` — the Ruby SDK freezes span attributes at finish
+before any `on_finish` processor runs, so the exporter boundary is the seam;
+see GOTCHAS) and runs UNCONDITIONALLY (the floor has no off state); values
+under unmatched names ride through raw, and tenant `additional_*` processors
+receive the masked VIEW (`pipelines.rb`, GOTCHAS F1). The explicit
+`redact(value)` helper always masks (calling it is the opt-in). Fail-closed
+for floor + listed keys (mask on any error, cycle + depth guarded).
+
+Proven by `spec/credential_floor_spec.rb` + `spec/credential_floor_wire_spec.rb`
+(the floor: list identity gate, matcher, every signal, doors 1+2, tenant
+view, negative controls), `spec/redaction_spec.rb` (raw-above-the-floor
+defaults — nothing else masked with no keys; both mask shapes; fail-closed)
+and `spec/redacting_exporter_spec.rb` + the conformance app (default-path RAW
+capture on the wire, floor names excepted).
 
 ---