foam-otel 1.3.0 → 1.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 82f1791c9e7d9929a655760c4e2f67295bb70f55dee75a2738bfcae33f5658f6
-  data.tar.gz: 16917b6e7a4e1dc1f0c6160f7026d4fe90c6213a16fa0664807e459359c5ea92
+  metadata.gz: 1e23dd37962a70ea36799a92d914fd2a38aa967a6cd1aface19bbaa322ab8966
+  data.tar.gz: f0018c27fb2dfff9316118438688a4a8ab2baea18810dbda522880361dfe6681
 SHA512:
-  metadata.gz: 38df8e89604a42d3db4698f5f12934f2bf3e25e288124caf10694a0e698e50f3a732fdd9f1dd2455fb52af90777cba00a3e825f23b967817d6fbda25b567fbdf
-  data.tar.gz: 5034a35f4d3a7c9d1f18fb0e7adc6691e75aacf0130eb74312ce14ad0ab902e787a25b75af45d6f618cd70a13df12b8323fc172d1cb0abee2950991bb839db6d
+  metadata.gz: 9731cb1fae1b54f236dd583102b2348ab4b45bff38745f446c584151e594ea685f2689d6a7fb4807136372006b7b10c0284b2e7c140c0c9625edb5a820294c8a
+  data.tar.gz: 91744d9fb0fb671bce62de973eec96152b147742ee5948fedf607626dfd89b8c3c6ed055522f21c4bbfd7d65ce5b558da9e67605578e3573ec15f8b0d4efeecc

data/GOTCHAS.md

@@ -582,6 +582,59 @@ anthropic 1.59.0, gemini-ai 4.3.0, ruby_llm 1.16.0).
   rack end-to-end with real `traceparent`+`baggage` headers, the no-op and
   never-raise cases.
 
+## F13: The value-pattern secret layer + coverage fix + TLS pin (1.4.0, security ruling 2026-07-27)
+
+The credential floor masks by NAME and is value-shape-blind; a live secret
+under an innocuous key (`note`, `detail`) sailed past it. Foam telemetry is
+read downstream by LLMs, so ONE leaked credential is prompt-injection
+exfiltratable — the value-pattern secret layer is the required second control
+(`docs/decisions/security-fixes-design.md` CONTRACT V).
+
+- **What / where**: `Redaction.scan_value_secrets` masks credential-SHAPED
+  spans (`SECRET_VALUE_RULES` + the PEM/PuTTY linear scans + the H1/H2
+  heuristics) to `[REDACTED]` in EVERY leaf string the engine walks —
+  attribute values, `status.message`, exception events, log bodies (string +
+  structured leaves), SQL, LLM content (BEFORE serialization, F-RB2), URL
+  query AND fragment pair values (Contract C1), resource identity strings
+  (once at build), exemplar `filtered_attributes`. It rides the SAME exporter-
+  boundary seam as the floor — one pass, ordered floor → C1 tokenizer → value
+  scan → user keys, each hit terminal (`redaction.rb` `mask_one`/
+  `redact_leaf_string`). METRIC DATAPOINT ATTRIBUTES are the one mandated
+  exemption (`metrics.rb`/`mask_metric_data` pass `value_scan: false`) —
+  low-cardinality by construction; a value mask there would break aggregation
+  dimensions.
+- **Non-disableable Tier 1**: the rules are frozen module constants, not
+  Config — no config path can narrow them (same posture as the floor). ONLY
+  the H1/H2 heuristic tier honors `secret_heuristics: false`, logged loud at
+  init.
+- **ReDoS-bounded / fail-closed**: compile-once, a lowercase-literal anchor
+  pre-filter (no anchor → zero regexes), a 256 KiB value cap (oversize +
+  anchor ⇒ whole-value mask, never regex-scanned), a match-flood cap, a
+  256-pair query/fragment cap, a per-Regexp `timeout:` on Ruby ≥ 3.2. ANY
+  scanner fault (timeout included) masks the value, never a raw pass-through.
+- **F-RB1 TLS pin (CWE-295)**: the vendored OTLP exporter honored
+  `OTEL_RUBY_EXPORTER_OTLP_SSL_VERIFY_NONE` as a PRESENCE check — one env var
+  would silently strip cert verification off the token-bearing export
+  connection. Every exporter constructor (door 1 `pipelines.rb`, door 2
+  `ingest.rb`) now pins `ssl_verify_mode: OpenSSL::SSL::VERIFY_PEER`; the var
+  is INERT and warned about (`init.rb` `warn_inert_ssl_verify_none`). There is
+  no foam option to weaken it.
+- **Mitigation**: `lib/foam/otel/constants.rb` (the frozen ruleset),
+  `lib/foam/otel/redaction.rb` (the engine + coverage tokenizer),
+  `lib/foam/otel/pipelines.rb` + `lib/foam/otel/ingest.rb` (the TLS pin),
+  `lib/foam/otel/llm.rb` (C4 pre-serialization masking).
+- **Sources**: `docs/decisions/security-fixes-design.md`;
+  gitleaks/detect-secrets/secretlint provenance in `THIRD-PARTY-NOTICES`;
+  vendored exporter presence-check at
+  opentelemetry-exporter-otlp `exporter.rb:39-47`.
+- **Test**: `spec/value_secret_layer_spec.rb` (the per-rule corpus, caps,
+  precedence, coverage contract, off-surface pin, structural gates),
+  `spec/security_regressions_spec.rb` (F-RB1 + F-RB2 verbatim repros),
+  `spec/vuln_suite_spec.rb` (the adversarial exfil/ReDoS/bypass suite +
+  mutation spot-checks).
+
+---
+
 ## General gotchas (applicable to Ruby)
 
 - **G1 — init after target import / pre-init no-op**: the API's proxy providers

data/README.md

@@ -4,12 +4,15 @@ Foam's OpenTelemetry core for Ruby services. A thin, safe wrapper over the
 official OpenTelemetry libraries: foam owns the pipeline (providers, batch
 processors, OTLP export to the foam fleet endpoint), turns on automatic
 tier-1/2 instrumentation, and hands you a small set of never-throw helpers —
-and nothing else. **Redaction is fully opt-in above one always-on exception:
-by default foam captures every value RAW** (no value-pattern masking, no PII
-preset) **except the credential floor** — a fixed list of credential/secret
-header and field NAMES that is always masked to `[REDACTED]` (see "The
-default credential denylist" below); you enumerate any further fields to mask
-via `redact_keys`/`redact_pii_keys`.
+and nothing else. **Redaction is fully opt-in above two always-on exceptions:
+by default foam captures every value RAW** (no PII preset) **except (1) the
+credential floor** — a fixed list of credential/secret header and field NAMES
+always masked to `[REDACTED]` (see "The default credential denylist" below) —
+**and (2) the value-pattern secret layer** — credential-SHAPED value spans
+(cloud keys, tokens, JWTs, private-key blocks, `user:pass@` URLs…) masked to
+`[REDACTED]` regardless of field name (see "The value-pattern secret layer"
+below); you enumerate any further fields to mask via
+`redact_keys`/`redact_pii_keys`.
 
 Built to `docs/BASE_PACKAGE_SPEC.md`. This README is the manual you integrate
 from.
@@ -139,14 +142,16 @@ API):** the allowlist must include BOTH headers —
 | `enabled:` | Boolean | yes | — | The config switch. `false` = fully inert (no SDK, no providers, no network, helpers no-op). `true` = export, in EVERY environment. No default — you write the logic. |
 | `token:` | String | when enabled | — | `Authorization: Bearer` for export. Required (and validated) only when `enabled: true`; wired explicitly from your secret source — never an env fallback read by the gem. **Fleet convention: read it from the `FOAM_OTEL_TOKEN` env var.** Blank while enabled → raises at boot. |
 | `version:` | String | no | nil | `service.version`, verbatim (git SHA recommended). Missing → warns and continues. Never detected at runtime. |
-| `redact_keys:` | Array<String> | no | `[]` | The ONLY field names tail-masked (shape-preserving, e.g. `********f456`). Opt-in and ADDITIVE on top of the always-on credential floor — empty (the default) means no masking beyond the floor. Matches by case-insensitive substring, including keys nested in structured values and query-string keys. A key that is also on the floor stays fully `[REDACTED]` (the floor wins; it is never downgraded to a tail). |
-| `redact_pii_keys:` | Array<String> | no | `[]` | Field names fully erased to `[REDACTED]` (no tail). Opt-in and additive above the floor — empty (the default) means no erasure beyond the floor. Foam ships no PII preset; this is your enumeration. |
+| `redact:` | Hash | no | nil | The grouped redaction object (1.5.0, the preferred surface): `{ secrets: [...], pii: [...], detect: [...] }` — symbol or string keys. `secrets:` has `redact_keys` semantics (tail mask) and `pii:` has `redact_pii_keys` semantics (full `[REDACTED]`); when both a legacy option and its `redact` field are set, the lists UNION. `detect:` is the opt-in PII detection tier's entity list — value-shape detection with typed placeholders (see "PII detection (opt-in)" below). An unknown field, a non-Array value, or an unknown `detect` entity name raises at boot. Absent (the default) → behavior byte-identical to before. |
+| `redact_keys:` | Array<String> | no | `[]` | Alias — the preferred spelling is `redact: { secrets: [...] }` (same semantics; the lists union when both are set). The ONLY field names tail-masked (shape-preserving, e.g. `********f456`). Opt-in and ADDITIVE on top of the always-on credential floor — empty (the default) means no masking beyond the floor. Matches by case-insensitive substring, including keys nested in structured values and query-string keys. A key that is also on the floor stays fully `[REDACTED]` (the floor wins; it is never downgraded to a tail). |
+| `redact_pii_keys:` | Array<String> | no | `[]` | Alias — the preferred spelling is `redact: { pii: [...] }` (same semantics; the lists union when both are set). Field names fully erased to `[REDACTED]` (no tail). Opt-in and additive above the floor — empty (the default) means no erasure beyond the floor. Foam ships no PII preset; this is your enumeration. |
 | `additional_span_processors:` | Array | no | `[]` | Tenant seam: constructed SpanProcessor instances added to foam's pipeline (additive; never replace foam's export). See coexistence. |
 | `additional_log_record_processors:` | Array | no | `[]` | Tenant seam, logs. |
 | `additional_metric_readers:` | Array | no | `[]` | Tenant seam, metrics. |
 | `additional_instrumentations:` | Array | no | `[]` | Constructed tier-2 instrumentation instances to register (fault-isolated: one that throws is skipped with a `[foam]` warning). |
 | `ignored_outbound_hosts:` | Array<String> | no | `[]` | Hosts whose outbound calls produce no spans — EXTENDS the built-in export-loop guard. For a co-resident agent's intake or a tenant exporter's endpoint. Applies to the clients whose official instrumentation supports host suppression: **Net::HTTP and Excon**. Faraday/HTTP (httprb)/HTTPX have no upstream `untraced_hosts` option — foam warns loudly at init when one of those is bundled (GOTCHAS G13); foam's own export loop is guarded for every client regardless. |
 | `diagnostics:` | Boolean | no | false | Verbose `[foam]` self-reporting of init/health. Warnings and errors are always loud regardless. |
+| `secret_heuristics:` | Boolean | no | true | The value-pattern secret layer's HEURISTIC tier (generic keyword+entropy detection, see "The value-pattern secret layer"). `false` disables ONLY the heuristics — the named provider patterns, the credential floor and the redaction-coverage contract have no off switch. Disabling logs one loud `[foam]` line at init (an explicit, audited decision for telemetry whose legitimate values collide with the heuristics). |
 
 Examples for the remaining options (each is runnable as-is):
 
@@ -157,6 +162,32 @@ Foam::Otel.init(name: "checkout-api", environment: env, enabled: true, token: EN
                 redact_pii_keys: %w[customer_email full_name])               # fully [REDACTED]
 ```
 
+```ruby
+# The grouped redact object (1.5.0, preferred — the flat options above are
+# aliases and UNION with it when both are set), plus the opt-in PII
+# detection tier (typed placeholders; see "PII detection (opt-in)"):
+Foam::Otel.init(name: "checkout-api", environment: env, enabled: true, token: ENV.fetch("FOAM_OTEL_TOKEN"),
+                redact: {
+                  secrets: %w[internal_ref voucher_code],   # == redact_keys (tail mask)
+                  pii: %w[customer_email full_name],        # == redact_pii_keys (full [REDACTED])
+                  detect: %w[email phone ssn credit_card ip], # value-shape detection → [EMAIL], [PHONE], …
+                })
+# An unknown field, a non-Array value, or an unknown detect entity raises at boot:
+# Foam::Otel.init(..., redact: { detect: ["name"] })
+# => raises ArgumentError: Foam::Otel.init redact: unknown detect entity "name" — the valid
+#    entity names are exactly {email, phone, ssn, credit_card, ip} (contract/pii-detect.json)
+```
+
+```ruby
+# Only if your legitimate telemetry collides with the generic secret
+# heuristics (e.g. base64 content-addressed ids masked as high-entropy
+# tokens): disable the HEURISTIC tier only. The named provider patterns and
+# the credential floor stay on — they have no off switch. Logs one loud
+# [foam] line.
+Foam::Otel.init(name: "checkout-api", environment: env, enabled: true, token: ENV.fetch("FOAM_OTEL_TOKEN"),
+                secret_heuristics: false)
+```
+
 ```ruby
 # Constructed tier-2 instrumentation instances (fault-isolated; one that
 # throws is skipped with a [foam] warning, never a crashed boot):
@@ -242,10 +273,11 @@ floor name in `redact_keys` does NOT downgrade it to a tail mask — the floor
 is terminal. `enabled: false` stays fully inert exactly as before (nothing
 exports at all — the floor masks data that leaves; inert mode has none).
 
-**Everything else stays RAW.** The floor matches NAMES only — it never scans
-values: bodies, query values, SQL text, LLM content, and PII under unlisted
-names export byte-identical raw, exactly as documented everywhere else in
-this README.
+**Everything else stays RAW — above the floor and the value layer.** The
+floor matches NAMES only, never values. Credential-SHAPED value spans are the
+value-pattern secret layer's job (next section); UNSHAPED values — bodies,
+query values, SQL text, LLM content, and PII under unlisted names — export
+byte-identical raw, exactly as documented everywhere else in this README.
 
 **Why.** Coverage-over-masking remains the mission, but raw `Authorization`
 headers, cookies, and API keys on the wire are a breach in waiting for every
@@ -278,6 +310,130 @@ concept in every core — the floor itself has NO init surface anywhere):
 
 ---
 
+## The value-pattern secret layer (always-on)
+
+The credential floor masks by NAME and is deliberately value-shape-blind — a
+live AWS key under the field name `note` would sail past it. Foam telemetry is
+read downstream by LLMs, so a leaked credential is exfiltratable by prompt
+injection; "no leakage at all" is the bar. The value-pattern secret layer is
+the second always-on control (fleet ruling 2026-07-27,
+`docs/decisions/security-fixes-design.md`):
+
+**What it masks.** Credential-SHAPED value spans, regardless of the field
+name they hide under: AWS access-key ids and keyword-anchored AWS secrets,
+GCP API keys, Azure client/storage secrets, GitHub tokens and fine-grained
+PATs, GitLab tokens, Slack tokens + webhooks, Stripe secret keys
+(publishable `pk_` keys are deliberately NOT masked), JWTs, PEM private-key
+blocks (`-----BEGIN … PRIVATE KEY` — a truncated block masks to the end of
+the value, fail-closed; `PUBLIC KEY`/`CERTIFICATE` blocks never match) and
+PuTTY PPK keys, `scheme://user:pass@` URI credentials (the whole userinfo,
+username included), `Bearer`/`Basic` auth values, and Anthropic/OpenAI API
+keys. The matched span becomes the literal `[REDACTED]` — no tail, no
+prefix, zero reconstructable bytes; surrounding text keeps its diagnostic
+value (`conn to postgres://[REDACTED]@db:5432 refused`).
+
+**The heuristic tier** additionally catches the long tail: a
+keyword-anchored generic pass (`password = hunter2SecretXyz99` inside a log
+body masks its value — gitleaks-derived, entropy-gated, with placeholder/
+stopword/key-context false-positive suppression) and a whole-value
+high-entropy base64 token pass (detect-secrets-derived; pure hex and UUIDs
+are deliberately exempt — trace ids and digests are telemetry's ambient
+vocabulary). Only this tier is disableable: `secret_heuristics: false`
+(one loud `[foam]` line; the named patterns above are not configurable).
+
+**Where it runs.** Every value-bearing string on every export path, both
+doors: span/event/link attribute values, `status.message`, exception
+messages, log bodies (string and structured leaves), SQL (`db.statement`),
+LLM content (BEFORE serialization and again at the boundary), URL query AND
+fragment pair values, baggage-derived attributes, exemplar
+`filtered_attributes`, and resource identity strings (once, at init).
+Metric datapoint attributes (metric labels) are the one exempt surface —
+low-cardinality by construction; a value mask there would break your
+aggregation dimensions.
+
+**Bounded by design.** Compile-once rules, a literal-anchor pre-filter (no
+anchor in the value → zero regexes run), a 256 KiB value cap (an oversized
+value with an anchor hit anywhere masks WHOLE, fail-closed; without one it
+passes untouched), a match-flood cap, a 256-pair query/fragment tokenizer
+cap, and a per-Regexp timeout on Ruby ≥ 3.2. ANY scanner fault masks the
+value rather than shipping it raw.
+
+**Provenance.** The ruleset derives from MIT/Apache-licensed OSS secret
+scanners (gitleaks, detect-secrets, secretlint) — see `THIRD-PARTY-NOTICES`.
+Rules are frozen module constants, byte-consistent across the foam cores,
+changed only by fleet ruling.
+
+> **Migration note — 1.4.0 (minor): value-pattern secret masking + redaction
+> coverage fix + TLS pin.** foam now masks credential-SHAPED values
+> (AWS/GCP/Azure keys, GitHub/GitLab tokens, Slack, Stripe, Anthropic/OpenAI
+> keys, JWTs, private-key blocks, `user:pass@` URI credentials, bearer/basic
+> tokens) to `[REDACTED]` in every exported string value — regardless of
+> field name — plus a generic keyword+entropy heuristic you can disable with
+> `secret_heuristics: false` (the named patterns and the credential floor
+> cannot be disabled). Redaction now also reaches URL FRAGMENTS (OAuth
+> implicit-flow tokens, hash-router params), bare `k=v` pair lists, nested
+> attribute containers (fail-closed), and structured LLM content BEFORE
+> serialization. Query/fragment tokenizing is capped at 256 pairs (the
+> overflow remainder masks whole). Metric labels are not value-scanned. TLS
+> peer verification is now PINNED on every exporter:
+> `OTEL_RUBY_EXPORTER_OTLP_SSL_VERIFY_NONE` is INERT and loudly warned about.
+> If a dashboard keyed off a raw token value (it should not have), it will
+> now see `[REDACTED]`.
+
+---
+
+## PII detection (opt-in)
+
+The floor and the secret layer protect credentials. PII in VALUES — an email
+address inside a log line, a card number inside an exception message — still
+exports RAW by default, because coverage-over-masking is the mission and only
+you know your privacy posture. As of 1.5.0 you can opt into value-shape PII
+detection per entity, via `redact: { detect: [...] }` (fleet ruling
+2026-07-28; frozen fixture: `contract/pii-detect.json` — the entity names and
+placeholders are byte-identical in every foam core and gate-checked in CI by
+`spec/pii_detect_spec.rb`). Nothing detects unless you list the entity —
+an empty/absent `detect` list is exactly today's behavior.
+
+**The exact entity set** (these five, nothing else — an unknown name raises
+at boot):
+
+| Entity | Placeholder | Catches |
+| --- | --- | --- |
+| `email` | `[EMAIL]` | `john.smith+test@example.co.uk` — requires a real TLD (`user@localhost` rides raw) |
+| `phone` | `[PHONE]` | `+1 (415) 555-0142`, `415-555-0199` — separated groups; bare digit runs, dotted versions (`2024.10.05`) and clock times ride raw |
+| `ssn` | `[SSN]` | `536-90-4399`, `536 90 4399` — delimited 3-2-4 only; structurally invalid SSNs (`000-…`, group `00`, serial `0000`) and undelimited runs ride raw |
+| `credit_card` | `[CREDIT_CARD]` | 13–19 digit PANs (spaced/dashed/contiguous) that pass **Luhn** — a card-shaped tracking id failing the checksum is never masked |
+| `ip` | `[IP]` | IPv4 (octet-validated — `999.1.1.1` and `10.4.1.2000` ride raw) and IPv6, `::`-compressed included |
+
+**How it masks.** Only the matched character span is replaced with the
+entity's typed placeholder — surrounding text keeps its diagnostic value
+(`login from [IP] flagged`). It runs everywhere the value-pattern secret
+layer runs (span/event/link attributes, `status.message`, log bodies, URL
+query and fragment leaves, nested structures, both doors), immediately AFTER
+the secret layer (a shaped credential still becomes `[REDACTED]` — detect
+never weakens it) and BEFORE your key lists; metric datapoint attributes stay
+exempt exactly like the secret layer. Detection is idempotent (placeholders
+never re-match), rides the same execution caps and fail-closed discipline
+(any detector fault masks the whole value to `[REDACTED]`, loudly), and the
+patterns follow the same bounded-quantifier authoring rules.
+
+**What it can NOT do — read this before relying on it.** This is value-SHAPE
+detection, nothing more: **person names and free-text prose are NOT
+detectable** — recognizing "the patient, John Smith, reported…" takes
+server-side NER, which is platform scope, not an SDK regex. Field-NAME-keyed
+PII (`patient_name`) is what `redact: { pii: [...] }` is for; enumerate those
+keys yourself. Foam still ships no PII preset — the detect list is your
+explicit, audited enumeration.
+
+```ruby
+# Door-2 taps take the same redact object, scoped to that tap:
+processor = Foam::Otel.create_ingest_span_processor(
+  token: ENV.fetch("FOAM_OTEL_TOKEN"), environment: ENV.fetch("APP_ENV"),
+  redact: { pii: %w[customer_email], detect: %w[email credit_card] })
+```
+
+---
+
 ## The helpers
 
 All helpers never raise, and no-op silently before `init` and when disabled.
@@ -520,7 +676,9 @@ readers only: they never mutate their data, their resource, or their export.
 ### The three entries
 
 All three share one signature (`token:` and `environment:` required; the
-redaction kwargs are the only CUSTOMER keys that tap redacts — opt-in, none by
+redaction kwargs — the grouped `redact:` object and its legacy aliases
+`redact_keys:`/`redact_pii_keys:`, union semantics exactly as on `init` —
+are the only CUSTOMER redaction that tap applies — opt-in, none by
 default, scoped to that tap, additive above the always-on credential floor,
 which every tap applies with zero configuration; `diagnostics:` is tap-scoped
 narration). Construction
@@ -668,6 +826,7 @@ end
 | `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | INERT — foam wires ONE resolved endpoint into all three exporters explicitly, so per-signal endpoint vars never redirect (or split) foam's export. Set → loud `[foam]` warning that it is inert. |
 | `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` | INERT — as above (warns when set). |
 | `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` | INERT — as above (warns when set). |
+| `OTEL_RUBY_EXPORTER_OTLP_SSL_VERIFY_NONE` | INERT — foam pins TLS peer verification (`ssl_verify_mode: VERIFY_PEER`) on every exporter constructor, both doors, so this vendored-exporter knob can never strip certificate verification off the credential-bearing export connection (1.4.0 security fix, CWE-295). Set → loud `[foam]` warning that it is inert; fix the trust chain (CA bundle) instead. |
 | every other `OTEL_*` | INERT — `OTEL_SERVICE_NAME` / `OTEL_RESOURCE_ATTRIBUTES` never override the init-declared identity; sampler/exporter/span-limit vars are ignored. |
 
 ---

data/THIRD-PARTY-NOTICES

@@ -20,6 +20,33 @@ filter_parameters (MIT). Mask FORMATS are facts, not copyrightable; no code
 was copied. None of these libraries is a runtime dependency of foam-otel.
 Provenance: RESEARCH.md.
 
+The VALUE-PATTERN SECRET LAYER ruleset (lib/foam/otel/constants.rb —
+SECRET_VALUE_RULES, SECRET_HEURISTIC_H1/H2, SECRET_VALUE_PLACEHOLDERS,
+SECRET_VALUE_STOPWORDS, SECRET_KEY_CONTEXT_ALLOWLIST; fleet ruling 2026-07-27,
+docs/decisions/security-fixes-design.md CONTRACT V) is likewise COPIED
+KNOWLEDGE derived from permissively-licensed open-source secret scanners, not
+vendored code (rule 38). Regex facts about public token formats are not
+copyrightable; the safe posture is followed anyway — every pattern derives
+from the MIT/Apache source text below or from the vendor's own public token
+format. None of these tools is a runtime dependency of foam-otel.
+
+  gitleaks       — MIT        — https://github.com/gitleaks/gitleaks (commit b58d3f1)
+                   (aws/gcp/azure/github/gitlab/slack/stripe/jwt/openai token
+                    shapes; the generic-api-key H1 template; the placeholder +
+                    stopword + key-context allowlists)
+  detect-secrets — Apache-2.0 — https://github.com/Yelp/detect-secrets (commit 5e14193)
+                   (the Base64HighEntropyString H2 entropy tier; several
+                    provider prefixes and the all-numeric entropy penalty)
+  secretlint     — MIT        — https://github.com/secretlint/secretlint (commit 7da613e)
+                   (bounded-JS-regex authoring discipline; the DB
+                    connection-string / URI-userinfo shape; the anthropic key
+                    shape)
+
+  trufflehog (AGPL-3.0) was consulted ONLY as a provider-coverage CHECKLIST —
+  NO pattern is transcribed from its source. Recorded deliberate: the AGPL
+  GCP service-account JSON pattern is NOT adopted; that secret is covered by
+  the private-key-block linear scan over its embedded PEM instead.
+
 Direct runtime dependencies (name version — license — repository):
 
   opentelemetry-api 1.11.0 — Apache-2.0 — https://github.com/open-telemetry/opentelemetry-ruby

data/lib/foam/otel/config.rb

@@ -11,11 +11,17 @@ module Foam
     # loop guard and the redacting exporters can read one authoritative value.
     Config = Struct.new(
       :name, :environment, :version, :enabled,
-      :redact_keys, :redact_pii_keys, :ignored_outbound_hosts,
-      :diagnostics, :endpoint,
+      :redact_keys, :redact_pii_keys, :redact_detect, :ignored_outbound_hosts,
+      :diagnostics, :endpoint, :secret_heuristics,
       keyword_init: true
     )
 
+    # The exact field names the grouped `redact:` option accepts (fleet ruling
+    # 2026-07-28, docs/decisions/redact-object-detect-design.md §2): `secrets`
+    # unions with the legacy redact_keys, `pii` with redact_pii_keys, `detect`
+    # opts into the PII detection tier. Anything else raises at boot.
+    REDACT_OPTION_FIELDS = %w[secrets pii detect].freeze
+
     class << self
       # The inert config the helpers read before init() runs (everything a
       # no-op needs: empty redaction lists, export disabled).
@@ -23,8 +29,10 @@ module Foam
         Config.new(
           name: "", environment: "", version: nil, enabled: false,
           redact_keys: [].freeze, redact_pii_keys: [].freeze,
+          redact_detect: [].freeze,
           ignored_outbound_hosts: [].freeze,
-          diagnostics: false, endpoint: FOAM_OTEL_ENDPOINT
+          diagnostics: false, endpoint: FOAM_OTEL_ENDPOINT,
+          secret_heuristics: true
         ).freeze
       end
 
@@ -36,7 +44,8 @@ module Foam
 
       def resolve_config(name:, environment:, version:, enabled:,
                          redact_keys:, redact_pii_keys:, ignored_outbound_hosts:,
-                         diagnostics:, endpoint:)
+                         diagnostics:, endpoint:, secret_heuristics: true, redact: nil)
+        keys, pii, detect = merged_redact_lists(redact_keys, redact_pii_keys, redact)
         Config.new(
           name: name,
           environment: environment,
@@ -46,12 +55,23 @@ module Foam
           # always-on credential floor (which is a module constant, NOT config
           # state — no config shape can alter it). Empty (the default) → no
           # customer redaction, raw capture above the floor. Stored lowercased
-          # for the engine's case-insensitive substring match.
-          redact_keys: downcase_list(redact_keys),
-          redact_pii_keys: downcase_list(redact_pii_keys),
+          # for the engine's case-insensitive substring match. The grouped
+          # `redact:` object (2026-07-28 ruling) UNIONS with the legacy flat
+          # lists; redact.detect is the opt-in PII detection tier's entity
+          # list — empty (the default) means zero PII detection.
+          redact_keys: keys,
+          redact_pii_keys: pii,
+          redact_detect: detect,
           ignored_outbound_hosts: Array(ignored_outbound_hosts).map { |h| h.to_s.downcase }.freeze,
           diagnostics: diagnostics ? true : false,
-          endpoint: endpoint
+          endpoint: endpoint,
+          # The value-pattern layer's HEURISTIC tier opt-out (security-fixes-
+          # design §V.8): false disables H1/H2 ONLY — the named Tier-1
+          # patterns, the credential floor and the coverage contract are
+          # module-constant machinery no config shape can narrow. Anything
+          # but literal false means ON (default-on preserves the fleet's
+          # no-leakage bar; disabling is an explicit, audited decision).
+          secret_heuristics: secret_heuristics == false ? false : true
         ).freeze
       end
 
@@ -61,22 +81,88 @@ module Foam
         Array(list).map { |k| k.to_s.downcase }.reject(&:empty?).uniq.freeze
       end
 
-      # A second init() refreshes ONLY the redaction lists (and diagnostics)
-      # onto the EXISTING config: enabled/endpoint/identity are process-global
-      # and immutable after the first init (see init.rb — a second init must
-      # never flip enabled and leave foam half-dark). Private: not public API
-      # surface (pinned by export_surface_spec).
-      def refresh_post_init_config(existing, redact_keys:, redact_pii_keys:, diagnostics:)
+      # ---- the grouped `redact:` object (fleet ruling 2026-07-28) -----------
+      # Parse + validate the object into its three raw lists. LOUD AT BOOT
+      # (rule 10, exactly like the name/environment validation): a non-Hash
+      # redact, an unknown field, a non-Array field value, or an unknown
+      # detect entity name raises ArgumentError here — never a silent skip,
+      # never a dark boot. Symbol AND string keys/entities are accepted
+      # (design §2 Ruby row); entities are coerced with the same downcase
+      # normalization the legacy lists use, then validated against the frozen
+      # fixture entity names (contract/pii-detect.json — the ONLY valid
+      # spellings). Called by both doors: init() via resolve_config /
+      # refresh_post_init_config, and the ingest factory's step-2 validation
+      # (with the entry name as context).
+      def parse_redact!(redact, context: "Foam::Otel.init")
+        return { secrets: nil, pii: nil, detect: nil } if redact.nil?
+
+        unless redact.is_a?(Hash)
+          raise ArgumentError, "#{context} redact: must be a Hash with fields drawn from " \
+                               "{#{REDACT_OPTION_FIELDS.join(', ')}} (e.g. redact: { detect: [\"email\"] }), " \
+                               "got #{redact.class}"
+        end
+        out = { secrets: nil, pii: nil, detect: nil }
+        redact.each do |key, value|
+          field = key.to_s
+          unless REDACT_OPTION_FIELDS.include?(field)
+            raise ArgumentError, "#{context} redact: unknown field #{key.inspect} — the valid fields are " \
+                                 "exactly {#{REDACT_OPTION_FIELDS.join(', ')}}"
+          end
+          unless value.is_a?(Array)
+            raise ArgumentError, "#{context} redact: #{field} must be an Array of strings, got #{value.class}"
+          end
+          out[field.to_sym] = Array(out[field.to_sym]) + value
+        end
+        validate_detect_entities!(out[:detect], context)
+        out
+      end
+
+      # Fixture rule "unknown_entity": the valid names are exactly the keys of
+      # contract/pii-detect.json's `entities` — anything else raises at init.
+      def validate_detect_entities!(detect, context)
+        return if detect.nil?
+
+        valid = PII_DETECT_ENTITIES.keys
+        detect.each do |entity|
+          next if valid.include?(entity.to_s.downcase)
+
+          raise ArgumentError, "#{context} redact: unknown detect entity #{entity.inspect} — the valid entity " \
+                               "names are exactly {#{valid.join(', ')}} (contract/pii-detect.json)"
+        end
+      end
+
+      # Legacy flat lists ∪ the grouped object's fields (design §2: the legacy
+      # aliases live on unchanged; when both are set, the lists UNION), all
+      # three normalized exactly as the legacy options always were.
+      def merged_redact_lists(redact_keys, redact_pii_keys, redact, context: "Foam::Otel.init")
+        parsed = parse_redact!(redact, context: context)
+        [downcase_list(Array(redact_keys) + Array(parsed[:secrets])),
+         downcase_list(Array(redact_pii_keys) + Array(parsed[:pii])),
+         downcase_list(parsed[:detect])]
+      end
+
+      # A second init() refreshes ONLY the redaction lists (and diagnostics,
+      # and the heuristic-tier flag — redaction state) onto the EXISTING
+      # config: enabled/endpoint/identity are process-global and immutable
+      # after the first init (see init.rb — a second init must never flip
+      # enabled and leave foam half-dark). The `redact:` object refreshes
+      # exactly like the legacy lists (design §2 second-init row). Private:
+      # not public API surface (pinned by export_surface_spec).
+      def refresh_post_init_config(existing, redact_keys:, redact_pii_keys:, diagnostics:,
+                                   secret_heuristics: true, redact: nil)
+        keys, pii, detect = merged_redact_lists(redact_keys, redact_pii_keys, redact)
         Config.new(
           name: existing.name,
           environment: existing.environment,
           version: existing.version,
           enabled: existing.enabled,
-          redact_keys: downcase_list(redact_keys),
-          redact_pii_keys: downcase_list(redact_pii_keys),
+          redact_keys: keys,
+          redact_pii_keys: pii,
+          redact_detect: detect,
           ignored_outbound_hosts: existing.ignored_outbound_hosts,
           diagnostics: diagnostics ? true : false,
-          endpoint: existing.endpoint
+          endpoint: existing.endpoint,
+          secret_heuristics: secret_heuristics == false ? false : true
         ).freeze
       end
     end