oh-my-agent-commandcode 0.0.0-stage → 20.38.4

package/skills/oma-observability/resources/boundaries/cross-application.md

@@ -0,0 +1,348 @@
+---
+otel_spec: "1.x (stable API/SDK)"
+otel_semconv: "1.43.0 (2026-07)"
+specs:
+  - "W3C Trace Context: Level 1 Recommendation 2020-02-06; Level 2 Candidate Recommendation"
+  - "W3C Baggage: Recommendation 2022-12-22"
+---
+
+# Cross-Application Observability Boundary
+
+## 1. Scope
+
+This file governs trace context, request correlation, causation chains, and baggage propagation across application boundaries; including service-to-service, cross-cloud, and cross-mesh calls.
+
+Design decision D3 absorbs two formerly separate documents into this file:
+
+- **Propagators** (formerly `propagators.md`): per-ecosystem header formats and normalization rules for W3C, B3, AWS X-Ray, GCP, Azure, Datadog, Cloudflare, Istio/Envoy, and Linkerd.
+- **DDD bounded context** (formerly `multi-domain.md`): `service.namespace` as the domain grouping attribute for multi-service architectures.
+
+Cross-references to other files in this skill use the `../` prefix convention.
+
+---
+
+## 2. 4-Layer Correlation Model
+
+Distributed systems require four distinct correlation identifiers, each serving a different audience and cardinality need. Using only `trace_id` conflates concerns; using only `request_id` loses async chain visibility.
+
+| Layer | Identifier | Propagation mechanism | Cardinality | Primary audience |
+|-------|-----------|----------------------|-------------|-----------------|
+| Infra / Trace | `trace_id` / `span_id` | W3C `traceparent` header | 1 per trace | Platform engineers, trace backends |
+| Correlation | `request_id` (`x-request-id`) | HTTP header + log field `http.request.id` | 1 per request | Customer support, error banners, frontend users |
+| Causation | `causation_id` | Message headers (Kafka, SQS, RabbitMQ) | 1 per event chain | Async workflow lineage, CQRS / event sourcing |
+| Idempotency | `idempotency_key` | HTTP header (`Idempotency-Key`) or message header | 1 per business operation | Duplicate-processing guard; distinct from causation |
+| Business | `user.id`, `tenant.id`, `order.id` | W3C Baggage (`baggage` header) | Per business entity | Product analytics, cross-cutting dashboards |
+
+> **Causation vs. idempotency are distinct**; `causation_id` expresses event ancestry (what triggered this event) and is often unique per edge in an event chain; `idempotency_key` is a business-level deduplication token shared by all retries of the same operation. Using a single header for both breaks dedup logic during retry if the causation chain diverges. Implement as two separate attributes / headers.
+
+### 2.1 trace_id
+
+Auto-generated by the OTel SDK at the trace root. 128-bit, encoded as 32 lowercase hex characters. Carried in `traceparent`. This is an **internal platform identifier**; do not expose it directly to end users because it reveals internal infrastructure topology. Its role is to be the carrier that links all spans in one distributed trace.
+
+### 2.2 request_id
+
+Human-facing correlation ID. Must appear in:
+
+- Every outbound HTTP response header (`x-request-id`)
+- Every log record as a structured field (`http.request.id` span attribute)
+- User-visible error banners on the frontend (support ticket anchor)
+
+The frontend generates a UUIDv4 `request_id` and sends it on every outbound request. Middleware sets the span attribute and logs the field. Customer support agents hand users a `request_id` from the error banner; platform engineers query logs to retrieve `trace_id` and pivot to the distributed trace waterfall.
+
+Cross-ref `../incident-forensics.md §MRA` for the support-to-trace lookup flow.
+
+### 2.3 causation_id / idempotency_key
+
+Event-driven and async workflows break the synchronous call chain. A Kafka consumer processing a payment event may run minutes after the HTTP request that produced it. `causation_id` links the new trace back to the originating event chain without forcing a parent-child span relationship.
+
+- HTTP POST: set `Idempotency-Key` request header → record as span attribute
+- Kafka producer: write `causation_id` into Kafka message header
+- Kafka consumer: extract `causation_id` → create OTel span link (not parent-child) to the originating span
+
+Span links are the correct OTel primitive for async causation. Cross-ref `../signals/traces.md §Messaging patterns`.
+
+### 2.4 Business context via Baggage
+
+Business identifiers (`user.id`, `tenant.id`, `order.id`, `feature.variant`) propagate as W3C Baggage key-value pairs. They flow through every service on the call path and appear as span attributes and log fields, enabling cross-cutting dashboard filters (e.g., "all traces for tenant ACME in the last hour").
+
+Baggage carries PII risk. See Section 4 for trust-boundary filtering rules.
+
+---
+
+## 3. Propagator Matrix
+
+The recommended default for all new deployments is **W3C Trace Context**. At ingress or multi-cloud boundaries, extract all known propagator formats and emit W3C downstream; normalization at the edge prevents split-brain traces inside the cluster.
+
+| Ecosystem | Header(s) | OTel propagator | Normalization notes |
+|-----------|-----------|----------------|---------------------|
+| **W3C Trace Context** (default) | `traceparent`, `tracestate`, `baggage` | `tracecontext`, `baggage` | L1 Rec 2020-02-06; L2 CR adds random-id flag. Mandatory baseline for all new services. |
+| **B3** (Zipkin legacy) | `x-b3-traceid`, `x-b3-spanid`, `x-b3-parentspanid`, `x-b3-sampled` | `b3multi` / `b3` (single-header) | Use multi-header format. Keep as secondary propagator for Zipkin-legacy services; migrate to W3C for new work. |
+| **AWS X-Ray** | `X-Amzn-Trace-Id` (`Root=1-<ts>-<hex>;Parent=<id>;Sampled=<0\|1>`) | `awsxray` | AWS X-Ray SDK supports W3C output since 2022. Prefer W3C; use X-Ray propagator only for Lambda or services that require X-Ray console trace linking. |
+| **GCP Cloud Trace** | `traceparent` (W3C preferred), legacy `X-Cloud-Trace-Context` | `gcp` propagator | GCP supports both formats. Emit W3C; accept legacy for backward compat. |
+| **Azure Monitor** | `traceparent` (W3C preferred), legacy `Request-Id` | W3C preferred | Application Insights SDK v3+ defaults to W3C. Treat `Request-Id` as read-only legacy input only. |
+| **Datadog APM** | `x-datadog-trace-id`, `x-datadog-parent-id`, `x-datadog-sampling-priority` | `datadog` propagator | Configure Datadog Agent to accept W3C input; use OTel Datadog exporter for span forwarding. |
+| **Cloudflare Workers** | `cf-ray` + W3C `traceparent` | Map `cf-ray` → baggage | `cf-ray` is not a trace ID. Map it to `baggage: cf.ray=<value>` for correlation. Emit W3C `traceparent` on Worker egress calls. |
+| **Istio / Envoy** | W3C `traceparent` + Envoy internal `x-request-id`, `x-ot-span-context` (legacy) | W3C (OTLP tracer, Envoy 1.29+) | `x-ot-span-context` is deprecated as of Envoy 1.29. Cross-ref `../layers/mesh.md §Propagator headers`. |
+| **Linkerd** | `l5d-ctx-trace`, `l5d-ctx-span`, `l5d-ctx-parent`, `l5d-ctx-deadline` | `linkerd` propagator | Linkerd headers are not W3C-compatible. Translate to W3C at the mesh boundary gateway. Cross-ref `../layers/mesh.md §Cross-mesh compatibility rule`. |
+
+### 3.1 Multi-propagator strategy at ingress
+
+Configure the ingress gateway or API gateway collector with a composite propagator that extracts all known formats. After extraction, emit only W3C downstream:
+
+```
+Composite extract order (ingress):
+  1. tracecontext  (W3C — primary)
+  2. b3multi       (Zipkin legacy)
+  3. awsxray       (AWS Lambda / API Gateway)
+  4. datadog       (Datadog-instrumented upstream)
+
+Inject order (egress, internal):
+  1. tracecontext  (W3C — only format emitted internally)
+  2. baggage       (W3C Baggage — always)
+```
+
+This pattern ensures that a request arriving from an AWS Lambda (X-Ray header) or a Datadog-instrumented partner API is absorbed into the W3C trace context and propagated uniformly to all internal services.
+
+---
+
+## 4. Baggage Rules and PII Guidance
+
+W3C Baggage propagates to every service on the call path. The W3C Baggage specification §Security states:
+
+> "Application owners should either ensure that no proprietary or confidential information is stored in baggage, or ensure baggage is not present in requests that cross trust boundaries."
+
+The OTel Baggage API spec does not independently repeat this warning; the normative source is the W3C parent specification. Implementers MUST read both.
+
+Cross-ref `../signals/privacy.md §Baggage rules` for the enforcement implementation.
+
+### 4.1 Trust-boundary rule
+
+At every egress trust boundary (API gateway, external webhook, third-party vendor API), apply a baggage filter:
+
+- **Default**: strip all baggage unless the receiving service is on the explicit allowlist.
+- **Allowlist**: maintained per-environment; requires security team sign-off to add entries.
+- **Internal-to-internal**: baggage passes through without filtering (intra-cluster calls within the same trust zone).
+
+### 4.2 Allowed and prohibited baggage values
+
+| Allowed (safe at any trust level) | Prohibited (must never appear in baggage) |
+|-----------------------------------|------------------------------------------|
+| `tenant.id` (opaque identifier) | `user.email` |
+| `user.tier` (tier name, not PII) | Authentication tokens or session IDs |
+| `feature.variant` | Credit card numbers or PAN |
+| `deployment.sha` | Passwords or API keys |
+| `region.hint` | Any field classified as PII under GDPR / PIPA |
+
+### 4.3 Enforcement point
+
+The mesh ingress gateway is the primary enforcement point. Configure Envoy's Lua filter or an OTel Collector `attributes` processor at the gateway to drop prohibited keys. Cross-ref `../layers/mesh.md §Baggage scrubbing`.
+
+---
+
+## 5. DDD Bounded Context via service.namespace
+
+OpenTelemetry resource attributes `service.name` and `service.namespace` map directly to DDD concepts:
+
+| OTel attribute | DDD concept | Example |
+|----------------|-------------|---------|
+| `service.namespace` | Bounded context (domain group) | `payments`, `inventory`, `identity` |
+| `service.name` | Individual microservice | `checkout-api`, `stock-api`, `auth-service` |
+
+### 5.1 Why service.namespace matters
+
+Without `service.namespace`, a system with 80+ microservices presents a flat list in trace backends and dashboards. Operators cannot distinguish domain-level KPIs from service-level noise.
+
+With `service.namespace`, dashboards can be scoped:
+
+- `service.namespace = payments` → filter all payment-domain spans, compute payment-domain SLI
+- `service.namespace = inventory` → separate inventory burn-rate alert from payments alert
+
+### 5.2 Configuration
+
+Set `service.namespace` as an OTel resource attribute at deployment time, not at instrumentation time:
+
+```yaml
+# OTel Collector resourcedetection / resource processor
+processors:
+  resource:
+    attributes:
+      - key: service.namespace
+        value: payments
+        action: upsert
+      - key: service.name
+        value: checkout-api
+        action: upsert
+```
+
+Alternatively, set via environment variable that the SDK's Environment Resource Detector picks up:
+
+```
+OTEL_RESOURCE_ATTRIBUTES=service.namespace=payments,service.name=checkout-api
+```
+
+### 5.3 Anti-pattern: flat service registry
+
+One hundred services without namespace grouping causes cognitive overload in the trace backend service map. Namespace assignment is a required step in any multi-service deployment. Assign namespaces aligned with DDD bounded contexts, not with team names or infrastructure regions.
+
+---
+
+## 6. Cross-Cloud Trace Continuity
+
+When a request crosses cloud provider boundaries (e.g., App A on AWS calls App B on GCP calls App C on-premises), trace continuity requires W3C `traceparent` on every hop.
+
+| Provider | W3C support status |
+|----------|--------------------|
+| AWS X-Ray SDK | W3C output supported since 2022 |
+| GCP Cloud Trace | W3C `traceparent` preferred; legacy `X-Cloud-Trace-Context` still accepted |
+| Azure Monitor | W3C `traceparent` preferred; Application Insights SDK v3+ default |
+| Jaeger (self-managed) | W3C Trace Context since Jaeger v1.35 |
+
+Collector bridge strategy: deploy an OTel Collector at each cloud egress point. The Collector normalizes incoming vendor headers to W3C before forwarding to the next cloud's ingress. This ensures that a trace rooted in AWS survives the GCP boundary and terminates in an on-premises Jaeger backend with an unbroken span chain.
+
+Cross-ref `../transport/collector-topology.md §Multi-cluster/regional` for Collector deployment patterns.
+
+---
+
+## 7. request_id to trace_id Integration
+
+The support escalation flow depends on reliable linkage between the user-visible `request_id` and the internal `trace_id`.
+
+**Flow:**
+
+1. Frontend generates UUIDv4 `request_id` and sends it on every outbound request.
+2. Backend middleware extracts `request_id`, sets span attribute `http.request.id`, and writes it as a structured log field.
+3. On error, the frontend displays the `request_id` to the user (never `trace_id`; the latter leaks internal topology).
+4. Customer support receives the `request_id` from the user's error screenshot.
+5. Platform engineer queries the log system: `http.request.id = "<request_id>"` → retrieves `trace_id` from the matching log record.
+6. Platform engineer opens the trace waterfall in the trace backend using `trace_id`.
+
+Cross-ref `../incident-forensics.md §MRA` for the full minimum reproducible artifact workflow.
+
+---
+
+## 8. Idempotency and Event-Driven Trace Lineage
+
+Idempotency keys and causation IDs are first-class observability primitives in event-driven architectures.
+
+**HTTP pattern (Stripe-style):**
+
+- Client sends `Idempotency-Key: <uuid>` header on POST.
+- Server records the key as span attribute `http.idempotency_key`.
+- On duplicate detection, the existing response is returned; the `trace_id` of the original attempt is logged alongside the duplicate attempt's `trace_id`.
+
+**Kafka / SQS pattern:**
+
+- Producer writes `causation_id` (UUIDv4) and original `traceparent` into message headers.
+- Consumer extracts both headers before creating the `CONSUMER` span.
+- Consumer span links to the producer span via `span.addLink(producerSpanContext)`.
+- The link: not parent-child; preserves trace lineage without forcing synchronous ordering assumptions.
+
+**DLQ replay:**
+
+- When replaying from a dead letter queue, re-inject the original `traceparent` and `causation_id` from the failed message headers. Generating a new `trace_id` at replay severs the forensic chain.
+
+Cross-ref `../signals/traces.md §Messaging patterns` for span link implementation details.
+
+---
+
+## 9. Trust Boundary Patterns
+
+| Boundary type | Trace behavior | Baggage behavior |
+|---------------|---------------|-----------------|
+| Internal service to internal service (same cluster) | Forward `traceparent` unchanged | Forward baggage unchanged |
+| Internal service to external API gateway (outbound) | Create new CLIENT span; include `traceparent` only if vendor is on allowlist | Strip all baggage unless vendor is on explicit allowlist |
+| B2B webhook (outbound to partner) | Start a new trace; log original `trace_id` as annotation in the outbound span only | Do not propagate baggage |
+| Third-party vendor API call | Do not propagate `traceparent` unless vendor explicitly supports it | Do not propagate baggage |
+| External partner calling inbound | Extract `traceparent` if present; validate and sanitize | Strip baggage at ingress gateway; reattach only allowlisted keys |
+
+---
+
+## 10. Context Inheritance in Non-Interactive Sessions
+
+CI/CD pipelines, agent workflows, and background jobs need trace context for end-to-end visibility across automated steps.
+
+A parent process may explicitly inject W3C context into `TRACEPARENT` and
+optionally `TRACESTATE`, but environment inheritance alone does not establish a
+child span parent. The Environment Resource Detector populates resource attributes;
+it does not extract this carrier as active trace context. Initialize the child's
+SDK, context manager, and W3C propagator before extracting and activating context.
+
+```js
+// After SDK/context-manager/propagator initialization in the child process.
+// runJob is the application's job implementation.
+import { context, propagation, ROOT_CONTEXT, trace } from '@opentelemetry/api';
+
+const carrier = {};
+if (process.env.TRACEPARENT) carrier.traceparent = process.env.TRACEPARENT;
+if (process.env.TRACESTATE) carrier.tracestate = process.env.TRACESTATE;
+const parent = propagation.extract(ROOT_CONTEXT, carrier);
+const tracer = trace.getTracer('background-job');
+await context.with(parent, () => tracer.startActiveSpan('job', async span => {
+  try { return await runJob(); }
+  finally { span.end(); }
+}));
+```
+
+The configured W3C propagator validates the carrier; absent/invalid trace context
+starts an independent trace. Validate inherited context against the process's
+trust boundary and verify actual parent/trace IDs in a fixture before relying on
+end-to-end correlation. Sources: <https://opentelemetry.io/docs/specs/otel/context/api-propagators/>
+and <https://github.com/open-telemetry/opentelemetry-js/tree/main/packages/opentelemetry-resources>.
+
+---
+
+## 11. Matrix Coverage
+
+These cells from `../matrix.md` are primarily driven by this file:
+
+| Layer | Boundary | Signal | Symbol | Primary artifact |
+|-------|----------|--------|--------|-----------------|
+| mesh | cross-application | traces | PASS | W3C `traceparent` in mesh proxy; zero-code injection |
+| L7-application | cross-application | traces | PASS | OTel SDK W3C propagation; DDD `service.namespace` grouping |
+| L7-application | cross-application | privacy | PASS | Baggage trust-boundary filtering at API gateway |
+| L7-application | cross-application | audit | PASS | Correlation across audit events via `trace_id` + `request_id` |
+| L3-network | cross-application | traces | PARTIAL | L3 packets carry no trace context; tagging at egress only |
+| L4-transport | cross-application | traces | PARTIAL | TCP is not trace-native; trace context begins at mesh or L7 |
+
+---
+
+## 12. Anti-Patterns
+
+The following extend `../anti-patterns.md §Cross-application`.
+
+| Anti-pattern | Consequence | Remediation |
+|---|---|---|
+| Mixed propagators without normalization at ingress | Broken traces at mesh or cloud boundary; spans appear as disconnected root spans | Apply composite propagator at ingress gateway; normalize to W3C downstream |
+| Baggage carrying PII crossing a trust boundary | W3C Baggage spec §Security violation; PII exposed to untrusted downstream services | Strip baggage at egress gateway; use allowlist; never put PII in baggage |
+| `request_id` not exposed to frontend | Customer support cannot correlate user-reported errors to backend traces | Return `x-request-id` in every response; display in error banners |
+| `service.namespace` not set | 100+ services appear as a flat unordered list; domain-level KPIs are impossible | Set `service.namespace` per DDD bounded context in OTel resource attributes |
+| B3-only propagation without W3C fallback | Traces break when crossing any non-Zipkin boundary (GCP, AWS, Azure) | Add `tracecontext` as primary propagator; keep `b3multi` as secondary for legacy |
+| New `trace_id` generated on DLQ replay | Original failure span is orphaned; forensic chain from root cause to replay is severed | Re-inject original `traceparent` and `causation_id` from failed message headers |
+
+---
+
+## References
+
+
+
+Internal cross-references:
+
+- `../standards.md`: normative semconv stability tiers and W3C spec versions
+- `../matrix.md`: full 112-cell coverage map (cross-application boundary row)
+- `../layers/mesh.md`: propagator headers per mesh; baggage scrubbing at ingress gateway
+- `../signals/traces.md`: OTel span data model, messaging span links, baggage security
+- `../signals/privacy.md`: PII classification, baggage PII redaction rules
+- `../signals/audit.md`: audit event correlation across application boundaries
+- `../transport/collector-topology.md`: multi-cluster and multi-region Collector bridge patterns
+- `../incident-forensics.md`: request_id to trace_id lookup flow (MRA section)
+
+### Primary sources
+
+- W3C Trace Context Level 1 Recommendation: <https://www.w3.org/TR/trace-context/>
+- W3C Trace Context Level 2 Candidate Recommendation: <https://www.w3.org/TR/trace-context-2/>
+- W3C Baggage Recommendation: <https://www.w3.org/TR/baggage/>
+- OTel Baggage API specification: <https://opentelemetry.io/docs/specs/otel/baggage/api/>
+- AWS X-Ray trace header: <https://docs.aws.amazon.com/xray/latest/devguide/xray-concepts.html#xray-concepts-tracingheader>
+- GCP Cloud Trace context propagation: <https://cloud.google.com/trace/docs/trace-context>
+- Datadog trace context propagation: <https://docs.datadoghq.com/tracing/trace_collection/trace_context_propagation/>
+- Linkerd distributed tracing: <https://linkerd.io/2.15/features/distributed-tracing/>

package/skills/oma-observability/resources/boundaries/multi-tenant.md

@@ -0,0 +1,290 @@
+---
+otel_spec: "1.x (stable API/SDK)"
+otel_semconv: "1.43.0 (2026-07)"
+specs:
+  - "W3C Baggage: Recommendation 2022-12-22"
+  - "GDPR: Regulation (EU) 2016/679; KR PIPA: amended 2023"
+notes:
+  - "OpenCost: CNCF Incubating (advanced 2024-10-31)"
+---
+
+# Multi-Tenant Observability
+
+## 1. Scope
+
+Multi-tenant observability covers the collection, routing, isolation, attribution, and residency of telemetry signals in a B2B SaaS platform where multiple customer tenants share underlying infrastructure.
+
+**In scope:**
+- Tenant isolation strategy across four tiers (soft / routing / dedicated-collector / dedicated-backend)
+- Tenant attribute propagation via W3C Baggage
+- Per-tenant sampling policies
+- Per-tenant retention schedules
+- Cost attribution per tenant (chargeback / showback)
+- Data residency routing for GDPR (EU) and PIPA (KR) tenants
+
+**Out of scope (related but distinct):**
+- Cross-service propagation mechanics: see `cross-application.md` (propagators, baggage rules)
+- FinOps unit economics and OpenCost metric surface; see `../signals/cost.md`
+- PII redaction and anonymization rules per tenant; see `../signals/privacy.md`
+
+---
+
+## 2. OTel Attribute Conventions for Tenant
+
+OpenTelemetry Semantic Conventions do not include a stable `tenant.*` group as of semconv 1.43.0. The attributes below are **custom application-defined** attributes. Follow the naming convention from `../standards.md §3` and prefix with the domain noun to avoid collisions with future OTel semconv additions.
+
+| Attribute | Type | Example | Status |
+|-----------|------|---------|--------|
+| `tenant.id` | string | `"acme-corp"` | Custom (not OTel Stable); use consistently across all signals |
+| `tenant.tier` | string enum | `"free"` / `"pro"` / `"enterprise"` | Custom; drives sampling and routing decisions |
+| `tenant.region` | string | `"eu-west-1"` / `"ap-northeast-2"` | Custom; drives data residency routing |
+
+**Naming rationale:** dot-separated namespace (`tenant.*`) matches OTel semconv style and avoids the underscore ambiguity common in Prometheus label names. Do not use `customer_id`, `org_id`, or `account_id` for the same concept; pick one canonical key and propagate it everywhere.
+
+---
+
+## 3. Four-Tier Isolation Strategy
+
+Most B2B SaaS organizations apply a mix of tiers: enterprise tenants get Tier 3 or 4, pro gets Tier 2, and free gets Tier 1. Select the highest tier required by the tenant's compliance obligations.
+
+| Tier | Description | Relative Cost | Isolation Strength | Compliance Fit |
+|------|-------------|---------------|--------------------|----------------|
+| 1. Soft | Shared collector + shared backend; tenants are separated only by `tenant.id` label filtering in dashboards and queries | Low | Weak; no pipeline isolation; noisy-neighbor risk | Basic B2B without data separation requirements |
+| 2. Routing | Shared collector pool; a verified routing connector or separate collector/exporter pipelines route by tenant tier; still shared backend | Medium | Medium; pipeline isolation; shared storage | Regulated tiers with data processing agreements |
+| 3. Dedicated collector | Per-tenant collector instance in a dedicated Kubernetes namespace; isolates ingestion and processing; shared or per-region backend | High | Strong; ingestion isolated; namespace-level blast radius | Enterprise tenants, HIPAA, ISO 27001 requirements |
+| 4. Dedicated backend | Per-tenant observability backend project or account (e.g., separate Grafana org, separate Datadog account, separate GCP project) | Highest | Strongest; full stack isolation from ingestion to storage | Highest compliance obligations (FedRAMP, SOC 2 Type II per tenant, GDPR Art. 28 sub-processor separation) |
+
+**Routing vs sampling:** verify the routing connector stability/configuration for the pinned Collector version. `tail_sampling` `and` policies only decide keep/drop in a shared trace pipeline; they neither route signals nor provide pipeline isolation. If connector stability is unsuitable, use separately configured collector/exporter pipelines or dedicated collectors. Use per-tier sampling only when shared-pipeline retention rates satisfy the requirement.
+
+---
+
+## 4. Tenant ID Propagation
+
+Tenant context must be carried across service boundaries so every span, metric, and log record emitted by any service is attributable to its originating tenant.
+
+**Mechanism:** W3C Baggage (`baggage` header, Recommendation 2022-12-22). Set `tenant.id` and `tenant.tier` as baggage entries at the ingress gateway. All downstream services read from the OTel Baggage API and apply these values as resource or span attributes.
+
+```
+Ingress Gateway
+  → set baggage: tenant.id=acme-corp, tenant.tier=enterprise
+      ↓
+Service A (reads baggage → sets span attribute tenant.id)
+      ↓
+Service B (reads baggage → sets span attribute tenant.id)
+```
+
+**Trust-boundary warning:** W3C Baggage is visible to every service in the propagation chain, including third-party or external services. The W3C Baggage specification (https://www.w3.org/TR/baggage/) explicitly notes that baggage values cross trust boundaries. Carrying `tenant.id` to external egress endpoints leaks customer account existence information.
+
+Rule: strip or validate `tenant.*` baggage entries at the egress gateway before forwarding to any external third-party endpoint. Internal propagation only. Cross-ref `../signals/privacy.md §Common PII in Telemetry` for baggage PII rules.
+
+---
+
+## 5. Per-Tenant Sampling
+
+Different tenant tiers justify different sampling rates. Enterprise tenants have SLA obligations and full debugging requirements; free tenants justify only ambient visibility.
+
+**Per-tier retention targets:**
+
+| Tenant Tier | Trace Sampling Rate | Rationale |
+|-------------|---------------------|-----------|
+| `enterprise` | 100% | SLA obligations, full debugging capability, compliance audit trail |
+| `pro` | 20% | Representative sample, cost-controlled |
+| `free` | 2% | Ambient visibility only |
+
+**Sampling configuration:** use the single tenant-aware YAML in
+`../transport/sampling-recipes.md §4`. Option A branches pipelines with a verified
+routing component; Option B changes keep/drop rates within a shared pipeline.
+Neither a tier label nor a tail-sampling policy provides tenant storage/RBAC
+isolation. Positive keep policies combine as OR, so the recipe's baseline also
+evaluates traces from other tiers; it is not an ordered fallback.
+
+Route by traceID to individual gateway samplers upstream of tail sampling
+(`../transport/collector-topology.md §6`). Keep the tier consistent across each trace, derive it from server-side tenant
+policy, and verify the observed retention rate and error/cost policies after
+upgrades. The table above states example rate targets, not an authorization or
+compliance requirement to collect every enterprise trace.
+
+---
+
+## 6. Per-Tenant Retention
+
+Retention schedules must be enforced per tier. Hot storage is fast-query; warm is compressed but queryable; cold is archival with retrieval latency.
+
+| Tier | Hot | Warm | Cold |
+|------|-----|------|------|
+| Enterprise | 90 days | 1 year | 3 years |
+| Pro | 30 days | 90 days | None |
+| Free | 7 days | 30 days | None |
+
+**Implementation:** Apply Kubernetes-native or backend-native lifecycle policies keyed on the `tenant.id` and `tenant.tier` labels. For shared backends (Tier 1–2), use label-based TTL rules or index lifecycle management (e.g., OpenSearch ISM policies, Loki retention rules, Thanos compactor retention). For dedicated backends (Tier 3–4), set backend project-level retention per tenant.
+
+Cross-ref `../meta-observability.md §Retention Matrix` for the full retention policy table covering all seven signals.
+
+---
+
+## 7. Cost Attribution
+
+Tenant-level cost attribution enables chargeback (billing tenants for their resource consumption) and showback (internal reporting without billing).
+
+**Kubernetes workload labeling:** maintain a verified mapping from workload labels to tenants. Aggregate cost through the OpenCost Allocation API using that mapping and an explicit window. The default `/metrics` endpoint does not automatically create `opencost_workload_cost_total` or a per-tenant cumulative cost counter; implement and document such a counter separately if needed. See `../signals/cost.md §5`.
+
+```yaml
+# Kubernetes Pod template label (applied via Helm values or admission webhook)
+metadata:
+  labels:
+    tenant.id: "acme-corp"
+    tenant.tier: "enterprise"
+```
+
+**Telemetry cost attribution:** The observability bill itself (e.g., Datadog RUM sessions per tenant, Honeycomb events per tenant) must also be attributed. Instrument the collector pipeline with throughput counters per tenant to apportion the observability bill proportionally.
+
+**Cardinality constraint:** `tenant.id` used as a metric label must be capped at the top-N active tenant count (100–1,000 tenants is safe for most TSDBs). Beyond this threshold, bucket overflow tenants under an `"other"` label to prevent metric series explosion. Cross-ref `../meta-observability.md §Cardinality Guardrails`.
+
+Cross-ref `../signals/cost.md §4 Cost Attribution by Dimension` for FinOps unit economics (per-tenant PromQL formulas, OpenCost architecture, and FOCUS spec).
+
+---
+
+## 8. Data Residency
+
+International-transfer requirements depend on the data, destination, and legal
+basis. GDPR Chapter V permits mechanisms including adequacy decisions (Art.45),
+appropriate safeguards (Art.46), and limited derogations (Art.49). Korean PIPA
+Art.28-8 has its own permitted grounds and safeguards. Neither establishes a
+blanket ban on every cross-region telemetry export. Separately enforce stricter
+contractual residency requirements where they apply.
+
+**Example adopted residency policy**, for tenants whose contracts require local
+processing (not a universal statement of GDPR/PIPA):
+
+| Tenant policy | Collector placement | Backend placement | Transfer decision |
+|---|---|---|---|
+| EU-local contract | Approved EU edge | Approved EU backend | Stay within the contract's residency boundary |
+| KR-local contract | Approved KR edge | Approved KR backend | Stay within the contract's residency boundary |
+| Transfer permitted | Approved source region | Approved destination | Document applicable transfer basis, safeguards, and contract scope |
+
+Route by a server-verified tenant policy before export, including transient
+processing locations and backups. When local residency is required, deploy
+local collectors/backends; otherwise verify and document the permitted transfer
+mechanism and destination rather than assuming that a region label authorizes it.
+
+Sources: <https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32016R0679>
+and <https://www.law.go.kr/LSW/lsLinkCommonInfo.do?chrClsCd=010202&lsJoLnkSeq=1029334957>.
+
+**Source-of-truth rule (critical)**: `tenant.region` MUST be resolved from an internal, server-side authoritative source; tenant registry service, organization metadata table, or IdP claim stamped at session start. It MUST NOT be trusted from client-supplied input (HTTP header, baggage, query string, or JWT claim the client itself controls). A misconfigured or malicious tenant could otherwise self-declare a non-EU/KR region and bypass residency routing. Enforce at the ingress gateway: reject requests where a client-declared `tenant.region` disagrees with the registry lookup keyed on `tenant.id`. In practice, strip any inbound `tenant.region` attribute and re-attach the registry-sourced value before the Collector pipeline accepts the span/log.
+
+Cross-ref `../transport/collector-topology.md §7 Federated / Multi-Cluster` for the multi-region edge topology diagram. Cross-ref `../signals/privacy.md §2 Regulatory Drivers` for GDPR and PIPA penalty context and `../signals/privacy.md §Cross-border transfer` for PII-specific cross-border rules.
+
+---
+
+## 9. Tenant Onboarding and Offboarding
+
+**Onboarding checklist:**
+1. Provision `tenant.id` and `tenant.tier` as Kubernetes labels on all tenant workloads (via Helm values or admission webhook).
+2. Create per-tenant routing rule if Tier 2+.
+3. Provision per-tenant collector namespace if Tier 3+.
+4. Create per-tenant backend project or organization if Tier 4.
+5. Create per-tenant dashboard folder with RBAC rules (see §10).
+6. Register tenant in cardinality allowlist (top-N cap enforcement).
+
+Automate steps 1–6 as code; cross-ref `../observability-as-code.md` for provisioning patterns.
+
+**Offboarding and eligible erasure:**
+
+Contract termination does not automatically require deletion of every audit,
+cost, or telemetry record. Determine which records contain personal data and
+which Art.17(1) erasure ground applies. Art.17(3) includes exceptions for legal
+obligations and legal claims; purpose-limited lawful retention may continue.
+Non-personal records and contractual retention require their own policy check.
+
+Offboarding procedure:
+1. Stop new tenant ingestion and inventory records across hot/warm/cold tiers,
+   backups, audit, and cost systems; classify personal data and applicable duties.
+2. Delete eligible records through verified backend/backup procedures. Document
+   retained classes, legal basis, approved expiry, and access restrictions.
+3. Remove active routing/dashboard access and the cardinality allowlist entry.
+4. Deprovision collector/backend resources after required deletion and lawful
+   retention are handled; preserve necessary retained records under restricted access.
+5. Emit a minimized audit event recording the action, scope, time, and operator.
+
+Source: GDPR Art.17(1), Art.17(3)(b),(e),
+<https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32016R0679>.
+
+Cross-ref `../signals/audit.md` for audit event schema for offboarding erasure events.
+
+---
+
+## 10. Dashboard Isolation
+
+| Tier | Dashboard Isolation Mechanism |
+|------|-------------------------------|
+| Tier 1 | Grafana folder per tenant; dashboard variables filter by `tenant.id` label |
+| Tier 2 | Grafana folder per tenant; Grafana RBAC restricts folder access by team |
+| Tier 3 | Grafana organization per tenant; or Honeycomb environment per tenant |
+| Tier 4 | Dedicated backend instance; tenant admin is org owner in their own account |
+
+**RBAC rule:** a tenant admin identity claim must be mapped to a Grafana or backend role that scopes data access strictly to that tenant's `tenant.id`. Cross-org or cross-tenant data leakage via dashboard query is a compliance violation.
+
+Cross-ref `../signals/privacy.md §Backend RBAC` for Grafana RBAC configuration patterns and OPA policy rules for query-level tenant isolation.
+
+---
+
+## 11. Noisy Neighbor Protection
+
+Shared infrastructure (Tier 1 and Tier 2) is vulnerable to one high-volume tenant degrading the observability pipeline for all other tenants.
+
+**Controls:**
+
+| Control | Scope | Mechanism |
+|---------|-------|-----------|
+| Per-tenant ingress rate limit | Collector receiver level | `ratelimiter` extension (alpha) or `filter` processor tied to per-tenant token-bucket state; pair with `memory_limiter` as backpressure |
+| Per-tenant cardinality quota | Metrics pipeline | Top-N series cap per `tenant.id`; overflow bucketed as `"other"` |
+| Circuit breaker on ingress | Collector pipeline | `memory_limiter` processor per-tenant pipeline; shed load when memory exceeds threshold |
+| Per-tenant queue depth limit | Exporter queue | `sending_queue` max size per tenant pipeline (Tier 2 routing) |
+
+Apply rate limits at the first collector tier (agent or edge). A tenant exceeding its quota must receive a well-defined error (e.g., OTLP `ResourceExhausted` gRPC status) rather than silently dropping data.
+
+---
+
+## 12. Matrix Cells: Multi-Tenant Row
+
+Quick navigation for multi-tenant boundary cells in `../matrix.md`:
+
+| Layer | Signal | Status | Artifact |
+|-------|--------|--------|----------|
+| L3-network | metrics | PASS | VPC flow logs per tenant; egress bytes attributed by source CIDR mapped to `tenant.id` |
+| L7-application | traces | PASS | W3C Baggage carries `tenant.id`; all spans tagged at ingress |
+| L7-application | cost | PASS | OpenCost workload attribution via `tenant.id` pod label; per-tenant PromQL aggregation |
+| L7-application | privacy | PASS | Per-tenant PII redaction rules; per-tenant residency routing |
+| L7-application | audit | PASS | Per-tenant audit trail; erasure events on offboarding |
+
+---
+
+## 13. Anti-Patterns
+
+The following are candidates for `../anti-patterns.md §Multi-Tenant`:
+
+| Anti-Pattern | Impact | Correction |
+|-------------|--------|------------|
+| `tenant.id` as metric label without top-N cap | Cardinality explosion in TSDB; ingestor OOM; query timeouts for all tenants | Enforce top-N cap (100–1,000); bucket overflow as `"other"`; cross-ref `../meta-observability.md §Cardinality Guardrails` |
+| `tenant.id` in W3C Baggage crossing trust boundaries | Tenant account existence leaks to third-party services; GDPR personal data transfer without legal basis | Strip `tenant.*` baggage at egress gateway before forwarding to any external endpoint |
+| Shared backend for regulated tiers (Tier 1 for compliance tenants) | Co-mingled data violates data processing agreements; one breach affects all tenants | Upgrade regulated tenants to Tier 3 or 4; apply isolation tier based on contractual obligation, not cost convenience |
+| Cross-border export without checking data, transfer basis, safeguards, or contract | Unauthorized personal-data transfer or residency-contract breach | Verify applicable GDPR/PIPA transfer grounds and destination; enforce stricter local-residency contracts using server-side policy |
+| No policy-based offboarding process | Eligible personal data persists or required evidence is erased prematurely | Inventory record classes; delete eligible data and retain legally required evidence under restricted access and approved expiry |
+
+---
+
+## Cross-References
+
+| Topic | File |
+|-------|------|
+| Baggage propagation mechanics and trust-boundary rules | `cross-application.md` |
+| Full FinOps cost attribution and OpenCost metric surface | `../signals/cost.md` |
+| PII redaction, anonymization, and GDPR/PIPA regulatory detail | `../signals/privacy.md` |
+| Tenant-aware sampling recipes (routing_connector, tail_sampling) | `../transport/sampling-recipes.md §4` |
+| Multi-cluster and regional collector topology | `../transport/collector-topology.md §7` |
+| Retention matrix for all seven signals | `../meta-observability.md §Retention Matrix` |
+| Cardinality guardrails and top-N cap | `../meta-observability.md §Cardinality Guardrails` |
+| Dashboard RBAC and query-level tenant isolation | `../signals/privacy.md §Backend RBAC` |
+| Audit event schema for offboarding erasure | `../signals/audit.md` |
+| Observability-as-code provisioning for tenant onboarding | `../observability-as-code.md` |
+| Full 112-cell coverage matrix | `../matrix.md` |