foam-otel 1.4.0 → 1.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 180e8af99c1f5c9bfc35de221c4985fb3ebac7b52ddf6fc99d6552a52aaea836
-  data.tar.gz: 14a5d81c673af1a20a75e483e076d2c6b1820fa14e5a587e2a23451da3cccf77
+  metadata.gz: cb5e4f91fea3170bbb206582adade6ca516824edad064a4be911ebe912ce7abf
+  data.tar.gz: 67ee9186826b4bc31c7f51deeaf8fd91bec6aaec393099f23b000384821230b9
 SHA512:
-  metadata.gz: a307d175e903abbbbd1c382e65694664f47aba18408fc7153f9a43e9fb72f8131fedcfc71a59e16ee4f7446dee0f3460233883f96a7048e1284fff75b54e16c7
-  data.tar.gz: 7322e0a91e04092d494150ad9b003e4a6fd61b23ebc7468a8c715eec2d5ab8aac64d1edc4c9fb3cb1a844c3d3a56c167b9bbf05bed2bb1d4cb491eba20695f5d
+  metadata.gz: 4c4c7288e2b049a9f686d0eae3637a74261820bc779d6e8ec4daf14347b6ed3b423648c4c0d226982663465907465fc350a44cd52b998ad97b4766b69ecb727b
+  data.tar.gz: 7eb2f27f17fc9fcd9649b72c68b59493b366ff66950c86ff42f4baa414b7ea222b9e251f30264adbfe32400fe7851b2cf64505bf666c5385546b44f4a5714768

data/GOTCHAS.md

@@ -633,6 +633,94 @@ exfiltratable — the value-pattern secret layer is the required second control
   `spec/vuln_suite_spec.rb` (the adversarial exfil/ReDoS/bypass suite +
   mutation spot-checks).
 
+## F14: Default-on header capture — the official rack options are the seam (a NAMED list, not capture-all), operator header config is never overridden, and Faraday middleware ORDER decides which span gets enriched
+
+- **Trap**: four ways foam's default-on header capture
+  (`header_capture.rb`) could go wrong. (1) The official rack gem's
+  `allowed_request_headers:`/`allowed_response_headers:` options are a
+  STATIC ALLOWLIST compiled at install time with no capture-all form — so
+  Ruby inbound cannot be capture-all like js/python; pretending otherwise
+  (or shipping an empty default) silently costs the whole signal. (2) The
+  gem applies operator env config
+  (`OTEL_RUBY_INSTRUMENTATION_RACK_CONFIG_OPTS`,
+  instrumentation-base `config_overrides_from_env`) per option at install,
+  where an env value BEATS any passed config — except the narrow-to-empty
+  spelling (`allowed_request_headers=`), which the gem's env parser DROPS
+  (`parts[1]` nil): foam blindly passing its defaults would then override
+  the operator's explicit opt-out. And `Instrumentation::Base#install` is
+  first-wins (`return true if installed?`) — foam pre-installing BEFORE the
+  operator's explicit instance would freeze the operator's header config
+  out entirely. (3) A middleware writing to
+  `OpenTelemetry::Trace.current_span` writes to WHATEVER span is ambient:
+  foam's Faraday middleware registered without `use :open_telemetry` FIRST
+  (the official faraday instrumentation auto-appends its tracer middleware
+  AFTER the customer's block, i.e. INSIDE any middleware the block added)
+  would stamp `http.request.header.*` onto the enclosing app span,
+  silently corrupting another producer's data. (4) Outbound Net::HTTP has
+  NO hook surface in its official instrumentation (a closed
+  `connect`/`request` patch, no request/response hook, no header option) —
+  the only route would be monkey-patching stdlib or gem internals.
+- **Sources**: installed source —
+  opentelemetry-instrumentation-rack-0.31.1 `instrumentation.rb:24-25`
+  (allowlist options), `instrumentation.rb:108-127` (compiled at install:
+  enumerate-up-front, no capture-all; `build_attribute_name` folds `-` to
+  `_`, so the emitted suffixes are underscored);
+  opentelemetry-instrumentation-base-0.26.1 `base.rb:218-227` (install is
+  first-wins/idempotent), `base.rb:274-313` (env override beats passed
+  config per option), `base.rb:343-363` (the `;`-separated `name=value`
+  env grammar; a valueless `name=` is dropped);
+  opentelemetry-instrumentation-faraday-0.33.0 `instrumentation.rb` (no
+  header option; `patches/stable/connection.rb:18-25` — the auto-insert
+  appends `use(:open_telemetry)` AFTER the customer's builder block) and
+  `middlewares/stable/tracer_middleware.rb:27-45` (the span is active only
+  around the handlers BELOW the tracer middleware);
+  opentelemetry-instrumentation-net_http-0.29.0 `patches/instrument.rb`
+  (no extension point).
+- **Decision & why** (user ruling 2026-07-28, superseding the first-pass
+  capture-all Rack middleware): INBOUND capture is the official rack gem's
+  OWN header options, which foam PRE-INSTALLS with documented default
+  lists (`DEFAULT_REQUEST_HEADERS`/`DEFAULT_RESPONSE_HEADERS` — the
+  standard APM set plus the seven floor headers, which arrive `[REDACTED]`
+  so presence stays visible) — the same pre-install shape as the loop
+  guard, ordered AFTER `install_additional` so an operator's explicit rack
+  instance wins entirely, and computed per option AGAINST the operator's
+  env var so an operator-named option is never filled by foam (the F-PY3
+  posture: a narrower operator list is never widened, and `name=` narrows
+  to the gem default `[]`, never to foam's list). The honest consequence
+  is documented, not papered over: a header outside the list is NOT
+  captured (README shows the env-var extension recipe). OUTBOUND stays
+  foam's Faraday middleware (Faraday's first-class public API — the
+  official faraday instrumentation has no header option), capture-all,
+  triple-gated per write: foam owns the traces slot (rule 18 B), the span
+  is recording, AND the span's `instrumentation_scope.name` is
+  "OpenTelemetry::Instrumentation::Faraday" — a mis-ordered stack captures
+  NOTHING instead of enriching the wrong span. Faraday placement stays a
+  per-connection line (`f.use :open_telemetry` then
+  `f.use :foam_otel_headers`): the only default-on mechanism is the
+  Connection prepend the official gem uses — exactly the
+  internals-patching foam never does. Net::HTTP (and Excon / httprb /
+  HTTPX) outbound headers are deliberately OUT OF SCOPE — skipped and
+  documented rather than hacked. Masking stays central: the credential
+  floor covers the emitted `http.{request,response}.header.<name>` forms
+  at the exporter boundary (dash/underscore-normalized, so the rack gem's
+  underscored suffixes match), never a key list in the capture path.
+- **Mitigation**: `lib/foam/otel/header_capture.rb` (the default lists,
+  `preinstall_rack_defaults!` + the operator-precedence parsing, the
+  Faraday middleware + public-registry registration, the shared gates);
+  `lib/foam/otel/init.rb` (`activate_instrumentations` pre-install
+  ordering — after `install_additional`, before `install_all` — gated on
+  foam owning traces; `activate_floor_extensions` Faraday registration).
+- **Test**: `spec/header_capture_spec.rb` — default-on capture both
+  directions on the official rack span, the honest negative (an unlisted
+  header is NOT captured), floor headers arrive `[REDACTED]` on the wire
+  with zero config, multi-value arity, operator env respect (named option
+  wins, narrow-to-empty honored, explicit instance wins entirely),
+  install idempotence (a second sweep never clobbers the config),
+  foreign-traces stand-down, hostile values (invalid UTF-8 / huge /
+  raising readers), Faraday scope/ownership gates + mis-ordered-stack
+  no-op + registration idempotence; `spec/conventions_spec.rb` (rule 21:
+  header capture never grows a `rack.input` body tee).
+
 ---
 
 ## General gotchas (applicable to Ruby)

data/README.md

@@ -33,8 +33,11 @@ gem "foam-otel"
 # Faraday), the primary datastores (pg, mysql2, redis, mongo) with RAW
 # db.statement capture, Sidekiq, the stdlib Logger bridge, hand-written Ruby
 # runtime + GC metrics, session stitching (browser session.id via baggage),
-# and the LLM shims (OpenAI, Anthropic, Gemini, ruby_llm — activity + raw
-# content). Add only the niche long tail your app needs beyond the floor:
+# default-on HTTP header capture (a documented default header list on the
+# official rack server span; every Faraday client header — see "Header
+# capture" below), and the LLM shims
+# (OpenAI, Anthropic, Gemini, ruby_llm — activity + raw content). Add only
+# the niche long tail your app needs beyond the floor:
 gem "opentelemetry-instrumentation-graphql"  # e.g. GraphQL
 gem "opentelemetry-instrumentation-resque"   # e.g. Resque
 ```
@@ -131,6 +134,106 @@ baggage is absent. **CORS note (the FDE wires this on every frontend-called
 API):** the allowlist must include BOTH headers —
 `Access-Control-Allow-Headers: traceparent, baggage`.
 
+### Header capture — request + response headers on the official spans
+
+**Inbound (default-on, no init option):** foam configures the OFFICIAL
+`opentelemetry-instrumentation-rack` gem's own header options
+(`allowed_request_headers:` / `allowed_response_headers:`) with a
+documented **default list** at init, so the rack SERVER span carries the
+listed request and response headers as
+`http.request.header.<name>` / `http.response.header.<name>` attributes
+(the gem's form: lowercase with `-` folded to `_`, e.g.
+`http.request.header.x_request_id`), on error requests too.
+
+> **Honest limit (deliberate divergence from foam's js/python cores):**
+> Ruby inbound captures the **named default list below, NOT all headers** —
+> the official gem's options are an enumerated allowlist with no
+> capture-all form, and that official option is the chosen seam (ruling
+> 2026-07-28). A custom header outside the list is **not captured** unless
+> you extend the list (recipe below).
+
+The default lists (`Foam::Otel::HeaderCapture::DEFAULT_REQUEST_HEADERS` /
+`DEFAULT_RESPONSE_HEADERS` — frozen, documented constants):
+
+- **request** — `content-type`, `content-length`, `content-encoding`,
+  `accept`, `accept-charset`, `accept-encoding`, `accept-language`,
+  `user-agent`, `referer`, `origin`, `host`, `cache-control`, `pragma`,
+  `if-none-match`, `if-modified-since`, `range`, `via`, `forwarded`,
+  `x-forwarded-for`, `x-forwarded-proto`, `x-forwarded-host`,
+  `x-forwarded-port`, `x-real-ip`, `x-request-id`, `x-correlation-id`,
+  plus the request half of the credential floor — `authorization`,
+  `proxy-authorization`, `cookie`, `x-api-key`, `x-auth-token`;
+- **response** — `content-type`, `content-length`, `content-encoding`,
+  `content-language`, `content-range`, `cache-control`, `pragma`,
+  `expires`, `age`, `etag`, `last-modified`, `vary`, `location`,
+  `retry-after`, `x-request-id`, `x-correlation-id`, `x-runtime`,
+  `server-timing`, plus the response half of the floor — `set-cookie`,
+  `www-authenticate`.
+
+Propagation headers (`traceparent`, `tracestate`, `baggage`) are
+deliberately not listed — they are extracted as span context, not captured
+as attributes; hop-by-hop plumbing (`connection`, `keep-alive`, …) is
+skipped too.
+
+**Extending (or narrowing) the list — the gem's own standard env var,
+always respected, never overridden by foam:**
+
+```bash
+# The value REPLACES the list for that option — name every header you want:
+export OTEL_RUBY_INSTRUMENTATION_RACK_CONFIG_OPTS='allowed_request_headers=content-type,accept,x-request-id,x-tenant-id'
+```
+
+Precedence: (1) an explicit rack instrumentation instance you pass via
+`additional_instrumentations:` installs first and its header config wins
+entirely; (2) any header option you set in
+`OTEL_RUBY_INSTRUMENTATION_RACK_CONFIG_OPTS` governs that option (foam's
+default fills only the option you did not touch — a narrower operator list
+is never widened, even the narrow-to-empty `allowed_request_headers=`
+spelling); (3) otherwise foam's defaults apply. Foam never installs the
+rack instrumentation (or its header config) when a foreign SDK owns the
+traces slot.
+
+**Floor masking, zero config:** the always-on credential floor masks the
+seven credential headers (`authorization`, `proxy-authorization`, `cookie`,
+`set-cookie`, `x-api-key`, `x-auth-token`, `www-authenticate`) to
+`[REDACTED]` in the emitted attribute forms (dash/underscore-normalized
+match) — they ride the default lists deliberately so their PRESENCE is
+visible, value never (see "The default credential denylist";
+`x-forwarded-for` / `x-real-ip` are on the 52-entry key floor and arrive
+`[REDACTED]` too). **Masking further captured headers is redaction's job,
+not a capture switch:** list the emitted name form in `redact_keys` /
+`redact: { secrets: [...] }` (tail mask) or `redact_pii_keys` /
+`redact: { pii: [...] }` (full `[REDACTED]`) — the substring key match
+reaches the attribute names (inbound names are underscored by the rack gem,
+so list e.g. `x_runtime`; Faraday's outbound names keep dashes).
+
+**Outbound (Faraday):** the official faraday instrumentation has no header
+option at all, so outbound stays foam's own Faraday middleware (its
+first-class public extension API), enriching the official CLIENT span with
+**every** request header actually sent (the injected `traceparent`
+included) and every response header received — string-array attributes,
+dash-form names. It never creates a span and stands down when foam is
+disabled/killed, when a foreign SDK owns traces, or when the current span
+is not the official faraday instrumentation's own (a mis-ordered stack
+captures nothing rather than writing onto the wrong span). One line per
+connection (Faraday has no public add-to-every-connection hook — the
+middleware name is registered by init, placement is yours; ORDER MATTERS,
+the official middleware first):
+
+```ruby
+conn = Faraday.new(url: "https://partner-api.example") do |f|
+  f.use :open_telemetry     # the official client span (explicit, so it sits OUTSIDE)
+  f.use :foam_otel_headers  # foam's header capture, directly inside it
+end
+```
+
+**Deliberately out of scope** (no standard extension point exists —
+skipped, not monkey-patched): outbound **Net::HTTP / Excon / HTTP (httprb)
+/ HTTPX** headers (their official instrumentations expose no
+request/response hook or header option) and gRPC metadata. Outbound header
+coverage today is Faraday. **Bodies/payloads are not captured** — a
+separate, future capability, not part of header capture.
+
 ---
 
 ## The `init` options
@@ -142,8 +245,9 @@ API):** the allowlist must include BOTH headers —
 | `enabled:` | Boolean | yes | — | The config switch. `false` = fully inert (no SDK, no providers, no network, helpers no-op). `true` = export, in EVERY environment. No default — you write the logic. |
 | `token:` | String | when enabled | — | `Authorization: Bearer` for export. Required (and validated) only when `enabled: true`; wired explicitly from your secret source — never an env fallback read by the gem. **Fleet convention: read it from the `FOAM_OTEL_TOKEN` env var.** Blank while enabled → raises at boot. |
 | `version:` | String | no | nil | `service.version`, verbatim (git SHA recommended). Missing → warns and continues. Never detected at runtime. |
-| `redact_keys:` | Array<String> | no | `[]` | The ONLY field names tail-masked (shape-preserving, e.g. `********f456`). Opt-in and ADDITIVE on top of the always-on credential floor — empty (the default) means no masking beyond the floor. Matches by case-insensitive substring, including keys nested in structured values and query-string keys. A key that is also on the floor stays fully `[REDACTED]` (the floor wins; it is never downgraded to a tail). |
-| `redact_pii_keys:` | Array<String> | no | `[]` | Field names fully erased to `[REDACTED]` (no tail). Opt-in and additive above the floor — empty (the default) means no erasure beyond the floor. Foam ships no PII preset; this is your enumeration. |
+| `redact:` | Hash | no | nil | The grouped redaction object (1.5.0, the preferred surface): `{ secrets: [...], pii: [...], detect: [...] }` — symbol or string keys. `secrets:` has `redact_keys` semantics (tail mask) and `pii:` has `redact_pii_keys` semantics (full `[REDACTED]`); when both a legacy option and its `redact` field are set, the lists UNION. `detect:` is the opt-in PII detection tier's entity list — value-shape detection with typed placeholders (see "PII detection (opt-in)" below). An unknown field, a non-Array value, or an unknown `detect` entity name raises at boot. Absent (the default) → behavior byte-identical to before. |
+| `redact_keys:` | Array<String> | no | `[]` | Alias — the preferred spelling is `redact: { secrets: [...] }` (same semantics; the lists union when both are set). The ONLY field names tail-masked (shape-preserving, e.g. `********f456`). Opt-in and ADDITIVE on top of the always-on credential floor — empty (the default) means no masking beyond the floor. Matches by case-insensitive substring, including keys nested in structured values and query-string keys. A key that is also on the floor stays fully `[REDACTED]` (the floor wins; it is never downgraded to a tail). |
+| `redact_pii_keys:` | Array<String> | no | `[]` | Alias — the preferred spelling is `redact: { pii: [...] }` (same semantics; the lists union when both are set). Field names fully erased to `[REDACTED]` (no tail). Opt-in and additive above the floor — empty (the default) means no erasure beyond the floor. Foam ships no PII preset; this is your enumeration. |
 | `additional_span_processors:` | Array | no | `[]` | Tenant seam: constructed SpanProcessor instances added to foam's pipeline (additive; never replace foam's export). See coexistence. |
 | `additional_log_record_processors:` | Array | no | `[]` | Tenant seam, logs. |
 | `additional_metric_readers:` | Array | no | `[]` | Tenant seam, metrics. |
@@ -161,6 +265,22 @@ Foam::Otel.init(name: "checkout-api", environment: env, enabled: true, token: EN
                 redact_pii_keys: %w[customer_email full_name])               # fully [REDACTED]
 ```
 
+```ruby
+# The grouped redact object (1.5.0, preferred — the flat options above are
+# aliases and UNION with it when both are set), plus the opt-in PII
+# detection tier (typed placeholders; see "PII detection (opt-in)"):
+Foam::Otel.init(name: "checkout-api", environment: env, enabled: true, token: ENV.fetch("FOAM_OTEL_TOKEN"),
+                redact: {
+                  secrets: %w[internal_ref voucher_code],   # == redact_keys (tail mask)
+                  pii: %w[customer_email full_name],        # == redact_pii_keys (full [REDACTED])
+                  detect: %w[email phone ssn credit_card ip], # value-shape detection → [EMAIL], [PHONE], …
+                })
+# An unknown field, a non-Array value, or an unknown detect entity raises at boot:
+# Foam::Otel.init(..., redact: { detect: ["name"] })
+# => raises ArgumentError: Foam::Otel.init redact: unknown detect entity "name" — the valid
+#    entity names are exactly {email, phone, ssn, credit_card, ip} (contract/pii-detect.json)
+```
+
 ```ruby
 # Only if your legitimate telemetry collides with the generic secret
 # heuristics (e.g. base64 content-addressed ids masked as high-entropy
@@ -365,6 +485,58 @@ changed only by fleet ruling.
 
 ---
 
+## PII detection (opt-in)
+
+The floor and the secret layer protect credentials. PII in VALUES — an email
+address inside a log line, a card number inside an exception message — still
+exports RAW by default, because coverage-over-masking is the mission and only
+you know your privacy posture. As of 1.5.0 you can opt into value-shape PII
+detection per entity, via `redact: { detect: [...] }` (fleet ruling
+2026-07-28; frozen fixture: `contract/pii-detect.json` — the entity names and
+placeholders are byte-identical in every foam core and gate-checked in CI by
+`spec/pii_detect_spec.rb`). Nothing detects unless you list the entity —
+an empty/absent `detect` list is exactly today's behavior.
+
+**The exact entity set** (these five, nothing else — an unknown name raises
+at boot):
+
+| Entity | Placeholder | Catches |
+| --- | --- | --- |
+| `email` | `[EMAIL]` | `john.smith+test@example.co.uk` — requires a real TLD (`user@localhost` rides raw) |
+| `phone` | `[PHONE]` | `+1 (415) 555-0142`, `415-555-0199` — separated groups; bare digit runs, dotted versions (`2024.10.05`) and clock times ride raw |
+| `ssn` | `[SSN]` | `536-90-4399`, `536 90 4399` — delimited 3-2-4 only; structurally invalid SSNs (`000-…`, group `00`, serial `0000`) and undelimited runs ride raw |
+| `credit_card` | `[CREDIT_CARD]` | 13–19 digit PANs (spaced/dashed/contiguous) that pass **Luhn** — a card-shaped tracking id failing the checksum is never masked |
+| `ip` | `[IP]` | IPv4 (octet-validated — `999.1.1.1` and `10.4.1.2000` ride raw) and IPv6, `::`-compressed included |
+
+**How it masks.** Only the matched character span is replaced with the
+entity's typed placeholder — surrounding text keeps its diagnostic value
+(`login from [IP] flagged`). It runs everywhere the value-pattern secret
+layer runs (span/event/link attributes, `status.message`, log bodies, URL
+query and fragment leaves, nested structures, both doors), immediately AFTER
+the secret layer (a shaped credential still becomes `[REDACTED]` — detect
+never weakens it) and BEFORE your key lists; metric datapoint attributes stay
+exempt exactly like the secret layer. Detection is idempotent (placeholders
+never re-match), rides the same execution caps and fail-closed discipline
+(any detector fault masks the whole value to `[REDACTED]`, loudly), and the
+patterns follow the same bounded-quantifier authoring rules.
+
+**What it can NOT do — read this before relying on it.** This is value-SHAPE
+detection, nothing more: **person names and free-text prose are NOT
+detectable** — recognizing "the patient, John Smith, reported…" takes
+server-side NER, which is platform scope, not an SDK regex. Field-NAME-keyed
+PII (`patient_name`) is what `redact: { pii: [...] }` is for; enumerate those
+keys yourself. Foam still ships no PII preset — the detect list is your
+explicit, audited enumeration.
+
+```ruby
+# Door-2 taps take the same redact object, scoped to that tap:
+processor = Foam::Otel.create_ingest_span_processor(
+  token: ENV.fetch("FOAM_OTEL_TOKEN"), environment: ENV.fetch("APP_ENV"),
+  redact: { pii: %w[customer_email], detect: %w[email credit_card] })
+```
+
+---
+
 ## The helpers
 
 All helpers never raise, and no-op silently before `init` and when disabled.
@@ -607,7 +779,9 @@ readers only: they never mutate their data, their resource, or their export.
 ### The three entries
 
 All three share one signature (`token:` and `environment:` required; the
-redaction kwargs are the only CUSTOMER keys that tap redacts — opt-in, none by
+redaction kwargs — the grouped `redact:` object and its legacy aliases
+`redact_keys:`/`redact_pii_keys:`, union semantics exactly as on `init` —
+are the only CUSTOMER redaction that tap applies — opt-in, none by
 default, scoped to that tap, additive above the always-on credential floor,
 which every tap applies with zero configuration; `diagnostics:` is tap-scoped
 narration). Construction
@@ -752,6 +926,7 @@ end
 | `OTEL_EXPORTER_OTLP_ENDPOINT` | HONORED — the one operator-level override of the pinned fleet endpoint (for foam's own conformance rig / enterprise egress). Active → loud `[foam]` warning naming the destination. Applies to door-2 taps identically (and moves the host the required loop step must name). |
 | `OTEL_PROPAGATORS=none` | HONORED — turns trace propagation OFF (links lost) while telemetry keeps flowing; warns. Any other value warns and is ignored (foam's propagator set is fixed: W3C tracecontext + baggage). |
 | `OTEL_BSP_SCHEDULE_DELAY` / `OTEL_BLRP_SCHEDULE_DELAY` / `OTEL_METRIC_EXPORT_INTERVAL` | HONORED — batch cadence, read natively by the upstream SDK. |
+| `OTEL_RUBY_INSTRUMENTATION_RACK_CONFIG_OPTS` | HONORED (by the contrib rack gem itself) and RESPECTED by foam — a header option you set here (`allowed_request_headers=…` / `allowed_response_headers=…`) governs that option; foam's default header list fills only the option you did not touch (never widened, never overridden — see "Header capture"). The other `OTEL_RUBY_INSTRUMENTATION_<NAME>_CONFIG_OPTS` vars are likewise the contrib gems' own standard levers (e.g. the Sidekiq `propagation_style` note above). |
 | `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | INERT — foam wires ONE resolved endpoint into all three exporters explicitly, so per-signal endpoint vars never redirect (or split) foam's export. Set → loud `[foam]` warning that it is inert. |
 | `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` | INERT — as above (warns when set). |
 | `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` | INERT — as above (warns when set). |

data/lib/foam/otel/config.rb

@@ -11,11 +11,17 @@ module Foam
     # loop guard and the redacting exporters can read one authoritative value.
     Config = Struct.new(
       :name, :environment, :version, :enabled,
-      :redact_keys, :redact_pii_keys, :ignored_outbound_hosts,
+      :redact_keys, :redact_pii_keys, :redact_detect, :ignored_outbound_hosts,
       :diagnostics, :endpoint, :secret_heuristics,
       keyword_init: true
     )
 
+    # The exact field names the grouped `redact:` option accepts (fleet ruling
+    # 2026-07-28, docs/decisions/redact-object-detect-design.md §2): `secrets`
+    # unions with the legacy redact_keys, `pii` with redact_pii_keys, `detect`
+    # opts into the PII detection tier. Anything else raises at boot.
+    REDACT_OPTION_FIELDS = %w[secrets pii detect].freeze
+
     class << self
       # The inert config the helpers read before init() runs (everything a
       # no-op needs: empty redaction lists, export disabled).
@@ -23,6 +29,7 @@ module Foam
         Config.new(
           name: "", environment: "", version: nil, enabled: false,
           redact_keys: [].freeze, redact_pii_keys: [].freeze,
+          redact_detect: [].freeze,
           ignored_outbound_hosts: [].freeze,
           diagnostics: false, endpoint: FOAM_OTEL_ENDPOINT,
           secret_heuristics: true
@@ -37,7 +44,8 @@ module Foam
 
       def resolve_config(name:, environment:, version:, enabled:,
                          redact_keys:, redact_pii_keys:, ignored_outbound_hosts:,
-                         diagnostics:, endpoint:, secret_heuristics: true)
+                         diagnostics:, endpoint:, secret_heuristics: true, redact: nil)
+        keys, pii, detect = merged_redact_lists(redact_keys, redact_pii_keys, redact)
         Config.new(
           name: name,
           environment: environment,
@@ -47,9 +55,13 @@ module Foam
           # always-on credential floor (which is a module constant, NOT config
           # state — no config shape can alter it). Empty (the default) → no
           # customer redaction, raw capture above the floor. Stored lowercased
-          # for the engine's case-insensitive substring match.
-          redact_keys: downcase_list(redact_keys),
-          redact_pii_keys: downcase_list(redact_pii_keys),
+          # for the engine's case-insensitive substring match. The grouped
+          # `redact:` object (2026-07-28 ruling) UNIONS with the legacy flat
+          # lists; redact.detect is the opt-in PII detection tier's entity
+          # list — empty (the default) means zero PII detection.
+          redact_keys: keys,
+          redact_pii_keys: pii,
+          redact_detect: detect,
           ignored_outbound_hosts: Array(ignored_outbound_hosts).map { |h| h.to_s.downcase }.freeze,
           diagnostics: diagnostics ? true : false,
           endpoint: endpoint,
@@ -69,21 +81,84 @@ module Foam
         Array(list).map { |k| k.to_s.downcase }.reject(&:empty?).uniq.freeze
       end
 
+      # ---- the grouped `redact:` object (fleet ruling 2026-07-28) -----------
+      # Parse + validate the object into its three raw lists. LOUD AT BOOT
+      # (rule 10, exactly like the name/environment validation): a non-Hash
+      # redact, an unknown field, a non-Array field value, or an unknown
+      # detect entity name raises ArgumentError here — never a silent skip,
+      # never a dark boot. Symbol AND string keys/entities are accepted
+      # (design §2 Ruby row); entities are coerced with the same downcase
+      # normalization the legacy lists use, then validated against the frozen
+      # fixture entity names (contract/pii-detect.json — the ONLY valid
+      # spellings). Called by both doors: init() via resolve_config /
+      # refresh_post_init_config, and the ingest factory's step-2 validation
+      # (with the entry name as context).
+      def parse_redact!(redact, context: "Foam::Otel.init")
+        return { secrets: nil, pii: nil, detect: nil } if redact.nil?
+
+        unless redact.is_a?(Hash)
+          raise ArgumentError, "#{context} redact: must be a Hash with fields drawn from " \
+                               "{#{REDACT_OPTION_FIELDS.join(', ')}} (e.g. redact: { detect: [\"email\"] }), " \
+                               "got #{redact.class}"
+        end
+        out = { secrets: nil, pii: nil, detect: nil }
+        redact.each do |key, value|
+          field = key.to_s
+          unless REDACT_OPTION_FIELDS.include?(field)
+            raise ArgumentError, "#{context} redact: unknown field #{key.inspect} — the valid fields are " \
+                                 "exactly {#{REDACT_OPTION_FIELDS.join(', ')}}"
+          end
+          unless value.is_a?(Array)
+            raise ArgumentError, "#{context} redact: #{field} must be an Array of strings, got #{value.class}"
+          end
+          out[field.to_sym] = Array(out[field.to_sym]) + value
+        end
+        validate_detect_entities!(out[:detect], context)
+        out
+      end
+
+      # Fixture rule "unknown_entity": the valid names are exactly the keys of
+      # contract/pii-detect.json's `entities` — anything else raises at init.
+      def validate_detect_entities!(detect, context)
+        return if detect.nil?
+
+        valid = PII_DETECT_ENTITIES.keys
+        detect.each do |entity|
+          next if valid.include?(entity.to_s.downcase)
+
+          raise ArgumentError, "#{context} redact: unknown detect entity #{entity.inspect} — the valid entity " \
+                               "names are exactly {#{valid.join(', ')}} (contract/pii-detect.json)"
+        end
+      end
+
+      # Legacy flat lists ∪ the grouped object's fields (design §2: the legacy
+      # aliases live on unchanged; when both are set, the lists UNION), all
+      # three normalized exactly as the legacy options always were.
+      def merged_redact_lists(redact_keys, redact_pii_keys, redact, context: "Foam::Otel.init")
+        parsed = parse_redact!(redact, context: context)
+        [downcase_list(Array(redact_keys) + Array(parsed[:secrets])),
+         downcase_list(Array(redact_pii_keys) + Array(parsed[:pii])),
+         downcase_list(parsed[:detect])]
+      end
+
       # A second init() refreshes ONLY the redaction lists (and diagnostics,
       # and the heuristic-tier flag — redaction state) onto the EXISTING
       # config: enabled/endpoint/identity are process-global and immutable
       # after the first init (see init.rb — a second init must never flip
-      # enabled and leave foam half-dark). Private: not public API surface
-      # (pinned by export_surface_spec).
+      # enabled and leave foam half-dark). The `redact:` object refreshes
+      # exactly like the legacy lists (design §2 second-init row). Private:
+      # not public API surface (pinned by export_surface_spec).
       def refresh_post_init_config(existing, redact_keys:, redact_pii_keys:, diagnostics:,
-                                   secret_heuristics: true)
+                                   secret_heuristics: true, redact: nil)
+        keys, pii, detect = merged_redact_lists(redact_keys, redact_pii_keys, redact)
         Config.new(
           name: existing.name,
           environment: existing.environment,
           version: existing.version,
           enabled: existing.enabled,
-          redact_keys: downcase_list(redact_keys),
-          redact_pii_keys: downcase_list(redact_pii_keys),
+          redact_keys: keys,
+          redact_pii_keys: pii,
+          redact_detect: detect,
           ignored_outbound_hosts: existing.ignored_outbound_hosts,
           diagnostics: diagnostics ? true : false,
           endpoint: existing.endpoint,

data/lib/foam/otel/constants.rb

@@ -342,5 +342,74 @@ module Foam
       yourkey your_api_key your-api-key xxxxxxxx todo
       notreal loremipsum deadbeef 0000000000 1111111111
     ].freeze
+
+    # ---- THE PII DETECT TIER (fleet ruling 2026-07-28, FULLY OPT-IN) --------
+    # docs/decisions/redact-object-detect-design.md §3; frozen fixture:
+    # contract/pii-detect.json (entity names + placeholders byte-identical,
+    # gate-checked by spec/pii_detect_spec.rb; behavior proven against the
+    # fixture's positive/negative vectors). Nothing here runs unless the
+    # customer lists the entity in init's redact.detect (or a door-2 tap's
+    # redact:) — the empty/absent list is today's raw-by-default behavior,
+    # byte-identical. The engine (redaction.rb detect_pii) runs this tier
+    # immediately AFTER the value-pattern secret layer and BEFORE customer
+    # key masking, masking ONLY the matched character span with the entity's
+    # typed :placeholder (surrounding text preserved verbatim; emitted
+    # placeholders never re-match — fixture "placeholder_idempotence").
+    #
+    # Authoring rules mirror the secret layer's (design §V.5.2 / rule 49):
+    # compile-once at require (a bad pattern fails the require, loudly),
+    # bounded quantifiers only, no lookbehind, no atomic/possessive groups;
+    # :group names the span to mask (0 = the whole match; phone masks group 1
+    # behind a consumed non-digit boundary — never a lookbehind). Patterns
+    # are byte-consistent with the python core's PII_DETECT_ENTITIES; the one
+    # recorded portability translation is phone's `^` → `\A` (Ruby's ^ is a
+    # LINE anchor — same translation note as SECRET_HEURISTIC_H2). :filter
+    # (:luhn) names a mandatory post-match validator — a card-shaped digit
+    # string failing Luhn is NEVER masked (fixture rule "checksum"). NOT
+    # detectable by design: person names and free-text prose (server-side
+    # NER scope — the README states this plainly).
+    PII_DETECT_ENTITIES = {
+      "email" => {
+        placeholder: "[EMAIL]", group: 0,
+        regex: '\b[A-Za-z0-9._%+\-]{1,64}@[A-Za-z0-9\-]{1,63}' \
+               '(?:\.[A-Za-z0-9\-]{1,63}){0,10}\.[A-Za-z]{2,24}\b',
+      }.freeze,
+      # Group 1 is the number; the leading alternation is a consumed boundary
+      # guard (no lookbehind — rule-49 discipline). Separators are space/dash
+      # only, so dotted versions (2024.10.05) and clock times (12:30:45)
+      # never trip.
+      "phone" => {
+        placeholder: "[PHONE]", group: 1,
+        regex: '(?:\A|[^0-9A-Za-z])((?:\+[0-9]{1,3}[ \-]?)?' \
+               '(?:\([0-9]{1,4}\)[ \-]?|[0-9]{1,4}[ \-])' \
+               '[0-9]{3,4}[ \-][0-9]{3,4})(?![0-9])',
+      }.freeze,
+      # Delimited 3-2-4 with a CONSISTENT delimiter (the \1 backreference —
+      # a one-char group, linear by construction) — an undelimited 9-digit
+      # run never matches. Area 000/666/9xx, group 00, and serial 0000 are
+      # structurally invalid SSNs and ride raw.
+      "ssn" => {
+        placeholder: "[SSN]", group: 0,
+        regex: '\b(?!000|666|9[0-9]{2})[0-9]{3}([ \-])' \
+               '(?!00)[0-9]{2}\1(?!0000)[0-9]{4}\b',
+      }.freeze,
+      # 13-19 digits in 4-4-4-rest grouping (space/dash/contiguous). The
+      # regex is deliberately loose on grouping; the LUHN filter is the
+      # normative gate (fixture "checksum" rule).
+      "credit_card" => {
+        placeholder: "[CREDIT_CARD]", group: 0, filter: :luhn,
+        regex: '\b[0-9]{4}(?:[ \-]?[0-9]{4}){2}[ \-]?[0-9]{1,7}\b',
+      }.freeze,
+      # IPv4 with per-octet range validation (999.1.1.1 and 4-digit octets
+      # ride raw), full 8-group IPv6, and ::-compressed IPv6.
+      "ip" => {
+        placeholder: "[IP]", group: 0,
+        regex: '\b(?:(?:25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])\.){3}' \
+               '(?:25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])\b' \
+               '|\b(?:[0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4}\b' \
+               '|\b(?:[0-9A-Fa-f]{1,4}:){1,6}:' \
+               '(?:[0-9A-Fa-f]{1,4}(?::[0-9A-Fa-f]{1,4}){0,5})?(?![0-9A-Fa-f:])',
+      }.freeze,
+    }.freeze
   end
 end