oh-my-agent-commandcode 0.0.0-stage → 20.38.4

package/skills/oma-observability/resources/layers/mesh.md

@@ -0,0 +1,368 @@
+---
+otel_spec: "1.x (stable API/SDK)"
+otel_semconv: "1.43.0 (2026-07)"
+tools:
+  - "Istio: 1.22+; Envoy: 1.29+; OTel Operator: v0.100+"
+---
+
+# Service Mesh Observability
+
+## 1. Why the Service Mesh Has Its Own Observability File
+
+The service mesh is classified as a **L4-L7 hybrid layer** in the taxonomy defined in `resources/standards.md §OSI Boundary Decision`. It differs from the adjacent L4-transport layer in a fundamental way: the mesh proxy (sidecar or ambient agent) operates at the HTTP/gRPC framing level, not at raw TCP socket level.
+
+Three properties make mesh observability distinct enough to warrant a dedicated file:
+
+1. **Zero-code auto-instrumentation.** Sidecar injection (or ambient mode) intercepts all inbound and outbound traffic without requiring any change to application code. Metrics, access logs, and distributed trace spans are produced automatically from day one of mesh deployment. This contrasts with `layers/L4-transport.md`, where eBPF probes attach to kernel socket events and produce transport-centric (TCP RTT, retransmit) artifacts.
+
+2. **Proxy-centric, not transport-centric.** Mesh observability is Envoy-proxy-centric or Linkerd-proxy-centric. The primary artifacts are Envoy listeners/clusters, access log records, and HTTP/gRPC span metadata; not TCP segments or IP packets. `layers/L4-transport.md` covers kernel-level transport; this file covers proxy-level L7 framing.
+
+3. **mTLS identity as a first-class signal.** The mesh terminates mTLS at the sidecar, producing certificate identity (SPIFFE SVID), cipher suite, and expiry as observable attributes alongside each request. No other layer produces this security context natively.
+
+---
+
+## 2. Service Mesh Options (as of 2026-Q2)
+
+| Mesh | CNCF Status | Proxy engine | Deployment modes | Primary differentiator |
+|------|------------|-------------|-----------------|----------------------|
+| **Istio** | Graduated | Envoy | Sidecar, Ambient (ztunnel + waypoint) | Richest Envoy telemetry; Telemetry API v2; OTLP direct export since 1.22 |
+| **Linkerd** | Graduated | linkerd2-proxy (Rust) | Sidecar only | Lightweight; minimal CPU overhead; built-in mTLS; own propagator headers |
+| **Envoy** | Graduated | Self | Standalone gateway or sidecar | Proxy engine underlying Istio; also deployed as standalone API gateway |
+| **Consul Connect** | Not CNCF | Envoy (via xDS) | Sidecar | HashiCorp ecosystem; strong multi-datacenter support |
+| **Kuma** | CNCF Sandbox | Envoy (via xDS) | Sidecar, Universal (VM) | Multi-cloud and multi-zone; Kong-backed |
+
+CNCF status source: <https://landscape.cncf.io>; verify at use time when load-bearing.
+
+---
+
+## 3. Native Telemetry from the Mesh
+
+### 3.1 Metrics: Envoy Golden Signals
+
+Envoy exposes golden signals per **listener** (inbound) and **cluster** (outbound). Istio enables a controlled subset by default to avoid cardinality explosion caused by per-endpoint label permutations.
+
+| Signal | Metric names (Istio/Envoy) | Label cardinality note |
+|--------|---------------------------|----------------------|
+| Request rate (throughput) | `istio_requests_total` | High: source × destination × method × status code |
+| Error rate | `istio_requests_total{response_code=~"5.."}` | Subset of throughput labels |
+| Latency p50/p95/p99 | `istio_request_duration_milliseconds` | Histogram; disable per-endpoint if >1000 pods |
+| Saturation (pending) | `envoy_cluster_upstream_rq_pending_total` | Per cluster |
+| mTLS handshake errors | `istio_tcp_connections_closed_total` | Proxy-level |
+
+Istio 1.22+ Telemetry API disables `destination_service_name` by default at high pod counts to cap cardinality. Override only after confirming cardinality budget in `resources/meta-observability.md`.
+
+### 3.2 Distributed Traces
+
+Envoy automatically creates **ingress and egress spans** for every HTTP/gRPC call that traverses the proxy. Each span captures:
+
+- `http.request.method`, `http.response.status_code`, `url.full` (Stable semconv)
+- `service.peer.name` derived from Envoy cluster name (Development semconv; successor to the deprecated `peer.service`)
+- mTLS peer identity (SPIFFE SVID) via `tls.*` attributes (Development semconv; see `resources/standards.md §3`)
+- `x-request-id` correlation ID (Istio-specific; carries across internal calls)
+
+Spans are created without SDK changes. The caveat is that without application-level header forwarding, each service starts a **new root span** rather than contributing to the existing trace. This is the context propagation problem addressed in Section 5.
+
+### 3.3 Access Logs
+
+Envoy access logs provide per-request metadata: source workload identity, destination service, HTTP method and status code, request duration, bytes sent and received, and upstream cluster name. Access logs complement traces; they are always available (100% sampling) even when traces are sampled at 1%.
+
+Structured access log format (JSON) is required for trace correlation:
+
+```json
+{
+  "trace_id": "%TRACE_ID%",
+  "span_id": "%SPAN_ID%",
+  "upstream_cluster": "%UPSTREAM_CLUSTER%",
+  "response_code": "%RESPONSE_CODE%",
+  "duration": "%DURATION%",
+  "source_principal": "%DOWNSTREAM_PEER_SUBJECT%"
+}
+```
+
+> Envoy command operators `%TRACE_ID%` and `%SPAN_ID%` (available since Envoy 1.25) extract the W3C `traceparent` context Envoy itself emitted. Using `%REQ(X-B3-TRACEID)%` is only correct if the upstream explicitly sent the B3 header. Since this skill standardizes on W3C (see §6 Propagator Headers), prefer the command operators.
+
+### 3.4 mTLS Certificate Observability
+
+Istio exposes certificate expiry and rotation events as Prometheus metrics:
+
+- `citadel_server_csr_count`: certificate signing request volume
+- `pilot_xds_pushes{type="sds"}`: SDS secret delivery (certificate rotation)
+- Cert expiry: scraped from `istio-proxy` via `/pki` endpoint, alertable as a PrometheusRule
+
+Cross-reference: `signals/privacy.md §Security Context (TLS attrs Development)` for attribute stability notes.
+
+---
+
+## 4. OpenTelemetry Direct Export (Envoy 1.29+ / Istio 1.22+)
+
+### 4.1 Envoy OTLP Tracer
+
+Since Envoy 1.29, the built-in OTLP tracer exports spans directly over OTLP/HTTP to any OTel-compatible backend or OTel Collector, replacing the older Zipkin and Jaeger tracers.
+
+Istio 1.22 surfaces this via the **Telemetry API** `opentelemetry` provider type, which avoids editing raw Envoy bootstrap config.
+
+### 4.2 Telemetry CR (Istio 1.22+)
+
+```yaml
+# Configure Istio tracing to export OTLP/HTTP to an OTel Collector
+apiVersion: telemetry.istio.io/v1
+kind: Telemetry
+metadata:
+  name: otel-tracing
+  namespace: istio-system
+spec:
+  tracing:
+    - providers:
+        - name: otel-tracing-provider
+      randomSamplingPercentage: 10.0
+---
+# Register the OTel tracing provider in MeshConfig
+# (set via IstioOperator or helm values: meshConfig.extensionProviders)
+# meshConfig:
+#   extensionProviders:
+#     - name: otel-tracing-provider
+#       opentelemetry:
+#         service: otel-collector.observability.svc.cluster.local
+#         port: 4318
+#         http:
+#           path: /v1/traces
+#         resourceDetectors:
+#           environment: {}
+```
+
+The `resourceDetectors.environment` block activates the **Environment Resource Detector**, which reads `OTEL_RESOURCE_ATTRIBUTES` from the proxy environment and enriches every span with `service.name`, `service.namespace`, `k8s.pod.name`, and `k8s.node.name` (Stable semconv) automatically.
+
+### 4.3 Custom Samplers (Envoy 1.29+)
+
+Envoy 1.29 introduced the OTel Sampler interface, allowing parent-based or trace-ID-ratio samplers to be configured without patching application code. Tail-based sampling decisions are still made at the Collector tier; see `resources/transport/sampling-recipes.md` for tail-sampler configuration.
+
+---
+
+## 5. Zero-Code Auto-Instrumentation Pitfall: Broken Context Propagation
+
+### 5.1 The Problem
+
+The mesh sidecar creates spans but cannot inject `traceparent` into the application's outbound HTTP calls by itself. If the application does not read the incoming `traceparent` header and forward it on outbound calls, Envoy on the egress side starts a **new trace**, breaking the trace chain. The result is a trace forest of disconnected single-hop spans rather than a unified distributed trace.
+
+This is anti-pattern #1 for mesh observability: mesh-only tracing without application header forwarding.
+
+### 5.2 Solution A: Application-Level Propagator Library
+
+Add the W3C Trace Context propagator to the application SDK. The SDK reads `traceparent` from incoming requests and automatically injects it into all outbound HTTP/gRPC calls.
+
+```python
+# Python example (opentelemetry-sdk)
+from opentelemetry.propagate import set_global_textmap
+from opentelemetry.propagators.composite import CompositePropagator
+from opentelemetry.propagators.b3 import B3MultiFormat
+from opentelemetry import propagate
+
+set_global_textmap(CompositePropagator([
+    propagate.get_global_textmap(),  # W3C TraceContext + Baggage
+    B3MultiFormat(),                  # B3 for Zipkin-legacy services
+]))
+```
+
+### 5.3 Solution B: OTel Operator Instrumentation CR
+
+The **OpenTelemetry Operator** (`github.com/open-telemetry/opentelemetry-operator`) injects the OTel SDK init container and environment variables into Pods automatically via a mutating webhook, without any application code change.
+
+Supported language runtimes: Java, NodeJS, Python, .NET, Go, Apache HTTPD, Nginx.
+
+**Step 1; Deploy the Instrumentation CR:**
+
+```yaml
+apiVersion: opentelemetry.io/v1alpha1
+kind: Instrumentation
+metadata:
+  name: default-instrumentation
+  namespace: my-app
+spec:
+  exporter:
+    endpoint: http://otel-collector.observability.svc.cluster.local:4318
+  propagators:
+    - tracecontext   # W3C traceparent / tracestate
+    - baggage        # W3C baggage
+    - b3             # B3 single-header (Zipkin legacy compatibility)
+  sampler:
+    type: parentbased_traceidratio
+    argument: "0.1"
+  java:
+    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
+  nodejs:
+    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest
+  python:
+    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest
+  dotnet:
+    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-dotnet:latest
+  go:
+    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-go:latest
+```
+
+**Step 2; Activate per Pod via annotation:**
+
+```yaml
+metadata:
+  annotations:
+    instrumentation.opentelemetry.io/inject-java: "true"
+    # or: inject-nodejs, inject-python, inject-dotnet, inject-go
+```
+
+The Operator injects the SDK agent, sets `OTEL_EXPORTER_OTLP_ENDPOINT`, and configures the propagator list. The sidecar then sees correctly formed `traceparent` headers on all outbound calls, connecting mesh spans to application spans in a single trace.
+
+---
+
+## 6. Propagator Headers per Mesh
+
+Cross-reference: `resources/boundaries/cross-application.md` for full propagator compatibility matrix.
+
+| Header | Standard / Origin | Carried by | Notes |
+|--------|------------------|-----------|-------|
+| `traceparent` | W3C Trace Context L1 | W3C-compliant SDKs, Envoy OTLP, Istio | Recommended default for all new deployments |
+| `tracestate` | W3C Trace Context L1 | Same as above | Vendor-specific trace flags; opaque to intermediaries |
+| `baggage` | W3C Baggage | OTel SDK, Istio | Key-value propagation; strip PII at ingress gateway |
+| `x-b3-traceid` | B3 (Zipkin) | Envoy B3 propagator, Linkerd (legacy compat) | 128-bit; use B3 multi-header format |
+| `x-b3-spanid` | B3 (Zipkin) | Same | 64-bit |
+| `x-b3-parentspanid` | B3 (Zipkin) | Same | 64-bit; absent on root spans |
+| `x-b3-sampled` | B3 (Zipkin) | Same | `1` = keep, `0` = drop; overrides downstream sampler |
+| `x-request-id` | Envoy / Istio internal | Istio sidecar injected | Stable correlation ID across retries; not a trace ID |
+| `x-ot-span-context` | OpenTracing (legacy) | Envoy OpenTracing tracer | Deprecated; replaced by OTLP tracer in Envoy 1.29+ |
+| `l5d-ctx-trace` | Linkerd | linkerd2-proxy | Linkerd trace context; not W3C-compatible |
+| `l5d-ctx-span` | Linkerd | linkerd2-proxy | Current span identifier in Linkerd format |
+| `l5d-ctx-parent` | Linkerd | linkerd2-proxy | Parent span reference |
+| `l5d-ctx-deadline` | Linkerd | linkerd2-proxy | Deadline propagation (timeout budget) |
+
+**Cross-mesh compatibility rule:** when Istio and Linkerd coexist (or when a service exits the mesh boundary to an external system), standardize on W3C `traceparent` / `tracestate` as the translation surface. Configure each mesh's propagator list to include W3C as the first entry. Translate Linkerd `l5d-ctx-*` headers to W3C at the mesh boundary gateway to avoid split-brain traces.
+
+---
+
+## 7. Combining Mesh and Application Telemetry
+
+The mesh alone covers **network boundary spans**: the ingress proxy creates a server-side span, the egress proxy creates a client-side span. What it cannot see is what the application does between receiving a request and making the next outbound call; database queries, cache reads, external API calls, or business-logic operations.
+
+| Source | Spans covered | Business context |
+|--------|--------------|-----------------|
+| Mesh (Envoy/Linkerd) | Ingress and egress HTTP/gRPC per service hop | None; proxy has no access to app state |
+| Application SDK | Internal operations: DB, cache, queue, business logic | Full; SDK operates inside the application process |
+| Combined via OTel Collector | All of the above, correlated by `trace_id` | Complete; best result |
+
+Recommended pipeline architecture:
+
+```
+Envoy sidecar
+  └─ OTLP/HTTP → OTel Collector (DaemonSet or sidecar)
+                    └─ traces → observability backend
+
+Application SDK (via OTel Operator Instrumentation CR)
+  └─ OTLP/gRPC → OTel Collector (same or separate instance)
+                    └─ traces → observability backend (same trace_id)
+```
+
+Both pipelines feed the same backend. The backend joins spans by `trace_id`. The result is a waterfall chart with proxy-level boundary spans and application-level internal spans in the same view.
+
+For Collector topology options see `resources/transport/collector-topology.md`.
+
+---
+
+## 8. mTLS Observability (Security Context)
+
+When Istio PeerAuthentication is set to `STRICT` mode, all pod-to-pod traffic is encrypted with mTLS. The mesh then exposes security context as observable attributes:
+
+- **TLS version**: `tls.protocol.version` (Development tier per semconv 1.43.0; see `resources/standards.md §Semconv Stability Tiers`)
+- **Cipher suite**: `tls.cipher` (Development tier)
+- **Certificate expiry**: alertable via PrometheusRule on `citadel_server_root_cert_expiry_timestamp`
+- **SPIFFE peer identity**: available in Envoy access log via `%DOWNSTREAM_PEER_SUBJECT%`; identifies source workload for zero-trust audit
+
+```yaml
+# PrometheusRule: alert when any Istio cert expires within 7 days
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: istio-cert-expiry
+  namespace: istio-system
+spec:
+  groups:
+    - name: istio.certificates
+      rules:
+        - alert: IstioCertExpiryWarning
+          expr: |
+            (citadel_server_root_cert_expiry_timestamp - time()) / 86400 < 7
+          for: 1h
+          labels:
+            severity: warning
+          annotations:
+            summary: "Istio root certificate expires in fewer than 7 days"
+```
+
+Cross-reference: `resources/signals/privacy.md §Security Context` for TLS attribute stability notes.
+
+---
+
+## 9. Sampling Considerations
+
+Istio's default sampling rate is **1%** (`randomSamplingPercentage: 1.0`). At 1%, you need 100 requests before the first trace is visible in the backend; this is insufficient for debugging low-traffic endpoints or canary deployments.
+
+| Scenario | Recommended sampling rate | Configuration location |
+|----------|--------------------------|----------------------|
+| Development / staging | 100% | Telemetry CR `randomSamplingPercentage: 100` |
+| Production (baseline) | 10% | Telemetry CR `randomSamplingPercentage: 10` |
+| High-traffic SLO critical path | Tail-based at Collector | Collector `tail_sampling` processor |
+| Canary release tracing | 100% on canary subset | Telemetry CR scoped to canary namespace or label |
+
+Tune via the Telemetry API `samplingPercentage` field (Istio 1.22+) or by configuring a custom OTel Sampler via Envoy 1.29+. For tail-based sampling (retain error traces, drop successful traces after SLO window), configure at the gateway Collector tier as documented in `resources/transport/sampling-recipes.md`.
+
+---
+
+## 10. Matrix Coverage Reference (mesh row)
+
+These cells from `resources/matrix.md` are the primary coverage drivers for this file:
+
+| matrix cell | symbol | artifact |
+|------------|--------|---------|
+| mesh × cross-application × traces | PASS | Zero-code L7 trace continuity; primary use case of the mesh layer |
+| mesh × multi-tenant × traces | PASS | `tenant.id` via W3C Baggage; proxy can enforce baggage scrubbing at gateway |
+| mesh × release × traces | PARTIAL | Canary routing rules are observable; `service.version` on spans; trace continuity requires OTel Operator CR |
+| mesh × privacy × * | PASS | mTLS config observability; baggage scrubbing; SPIFFE identity in access log |
+| mesh × slo × metrics | PASS | Envoy request rate + error rate are primary SLI sources |
+
+---
+
+## 11. Anti-Patterns
+
+The following are anti-patterns for this layer. They are candidates for inclusion in `resources/anti-patterns.md`.
+
+| # | Anti-pattern | Impact | Remedy |
+|---|-------------|--------|--------|
+| AP-M1 | Mesh-only tracing without application SDK | Spans cover network hops only; DB queries, business logic, and external API calls invisible | Deploy OTel Operator Instrumentation CR to inject SDK into all application pods |
+| AP-M2 | W3C Trace Context not standardized across meshes | Linkerd and Istio produce disconnected trace forests when a request crosses mesh boundaries | Configure W3C as first propagator in both meshes; translate at boundary gateway |
+| AP-M3 | Default 1% Istio sampling rate in production without awareness | First trace visible only after 100 requests; low-traffic endpoints never appear in trace backend | Raise to 10% baseline; use tail-based sampling at Collector for cost control |
+| AP-M4 | OTel Operator Instrumentation CR not deployed | `traceparent` not forwarded by application code; mesh spans are disconnected root spans | Deploy `Instrumentation` CR and annotate all application Deployments |
+| AP-M5 | Cardinality explosion from per-endpoint Envoy labels | `istio_requests_total` with high-cardinality `destination_service_name` × `source_workload` × URL path exceeds Prometheus cardinality budget | Use Telemetry API to suppress high-cardinality labels; aggregate at Collector metric transform processor |
+
+---
+
+## 12. References
+
+
+Internal cross-references:
+- `resources/standards.md`: normative semconv stability tiers and W3C Trace Context requirements
+- `resources/matrix.md`: full 112-cell coverage map (mesh row)
+- `resources/transport/sampling-recipes.md`: tail-based sampling at Collector tier
+- `resources/transport/collector-topology.md`: DaemonSet vs sidecar Collector topology
+- `resources/meta-observability.md`: cardinality guardrails and pipeline self-health
+- `resources/layers/L4-transport.md`: eBPF socket-level profiles for mesh sidecar overhead
+- `resources/boundaries/cross-application.md`: full propagator compatibility matrix
+- `resources/boundaries/multi-tenant.md`: baggage-based tenant attribution
+- `resources/boundaries/release.md`: canary trace routing with `service.version`
+- `resources/signals/privacy.md`: TLS attribute stability and mTLS security context
+- `resources/signals/traces.md`: OTel SDK trace patterns for application layer
+
+## References
+
+- Istio observability concepts: <https://istio.io/latest/docs/concepts/observability/>
+- Istio OpenTelemetry tracing task: <https://istio.io/latest/docs/tasks/observability/distributed-tracing/opentelemetry/>
+- Envoy + Istio OTel features (2024): <https://opentelemetry.io/blog/2024/new-otel-features-envoy-istio/>
+- OTel Operator GitHub: <https://github.com/open-telemetry/opentelemetry-operator>
+- Linkerd observability and propagator docs: <https://linkerd.io/2.15/features/distributed-tracing/>
+- W3C Trace Context L1: <https://www.w3.org/TR/trace-context/>
+- OTel semconv `tls.*`: <https://opentelemetry.io/docs/specs/semconv/attributes-registry/tls/>

package/skills/oma-observability/resources/matrix.md

@@ -0,0 +1,177 @@
+---
+otel_spec: "1.x (stable API/SDK)"
+otel_semconv: "1.43.0 (2026-07)"
+---
+
+# Observability Coverage Matrix
+
+## 1. Purpose
+
+This file is the coverage map for the `oma-observability` skill. It answers one question per cell:
+
+> "What observability artifact belongs at the intersection of this layer, this boundary, and this signal?"
+
+The matrix has three goals:
+
+1. **Prevent hand-wave instrumentation.** "Just add metrics" is not a plan. Every cell forces an explicit decision: what artifact, from which source, tagged how, stored where.
+2. **Surface N/A early.** Some combinations are physically or semantically meaningless (e.g., L3-level CPU profiling). Declaring them N/A is a correct engineering answer, not a gap.
+3. **Provide navigation.** Each covered cell links to the authoritative layer/boundary/signal file within this skill so implementers can jump directly to the detailed guidance.
+
+**Normative base:** all taxonomy, semconv stability tiers, and OSI boundary decisions are declared in `resources/standards.md`. Read that file before using this matrix.
+
+---
+
+## 2. Taxonomy
+
+### Layers (4)
+
+| Label | Meaning |
+|-------|---------|
+| L3-network | IP routing, VPC flow logs, BGP/BMP, ICMP, PMTUD |
+| L4-transport | TCP retransmits/RTT, QUIC/HTTP3 transport, eBPF (Beyla/Pixie) |
+| mesh | Istio/Linkerd/Envoy: zero-code instrumentation, mTLS metadata as security context |
+| L7-application | HTTP/gRPC/WebSocket, RUM (web + mobile), crash analytics, messaging |
+
+### Boundaries (4)
+
+| Label | Meaning |
+|-------|---------|
+| multi-tenant | Signals must be isolated, attributed, and possibly residency-constrained per tenant |
+| cross-application | Propagation across service, domain, or vendor boundaries |
+| slo | Signals that feed SLI computation, burn-rate alerting, or error-budget accounting |
+| release | Signals that are correlated to a specific deployment, canary, or feature flag state |
+
+### Signals (7)
+
+| Label | Meaning |
+|-------|---------|
+| metrics | Numeric time-series (counters, gauges, histograms) |
+| logs | Structured log records and events |
+| traces | Distributed spans and context propagation |
+| profiles | Continuous CPU/memory/off-CPU flame graphs (OTEP 0239; Development) |
+| cost | FinOps unit-economics and resource attribution |
+| audit | Immutable, tamper-evident event records for compliance |
+| privacy | PII detection, redaction, anonymization, pseudonymization controls |
+
+---
+
+## 3. How to Read the Matrix
+
+- **Row** = one boundary within a given layer.
+- **Column** = one signal.
+- **Cell content** = 1-2 word artifact description + file reference.
+
+### Cell legend
+
+| Token | Meaning |
+|-------|---------|
+| `PASS` | Covered: artifact is well-defined; see referenced file and section |
+| `PARTIAL` | Partially covered or requires caveat; see referenced file and "Caveats" section below |
+| `N/A` | Not meaningful at this layer/boundary or produces no actionable artifact |
+
+Every `PASS` and `PARTIAL` cell includes at least one file reference. File references point to files within this skill tree.
+
+---
+
+## 4. The Matrix
+
+### Layer: L3-network
+
+L3 is the IP routing layer: VPC flow logs, BGP/BMP, ICMP unreachables, and PMTUD probing. Observability at this layer is network-flow-centric. Traces are not native to L3 packets; profiles are not applicable. SLO and release boundaries are defined at the application layer and cannot be computed from IP flow data alone.
+
+| Boundary \ Signal | metrics | logs | traces | profiles | cost | audit | privacy |
+|---|---|---|---|---|---|---|---|
+| multi-tenant | PASS per-tenant VPC flow byte/packet counters → `layers/L3-network.md`, `boundaries/multi-tenant.md §Metric attribution` | PASS VPC flow log stream tagged by tenant network CIDR → `layers/L3-network.md`, `signals/logs.md` | PARTIAL trace-ID egress tagging at L3 boundary only; trace context not carried in IP headers → `layers/L3-network.md §BGP advanced`, `signals/traces.md` | N/A; CPU/memory profiling has no L3 artifact | PARTIAL egress byte attribution by tenant VPC → `signals/cost.md §egress`, `boundaries/multi-tenant.md` | PASS VPC flow audit trail tagged by tenant → `signals/audit.md`, `layers/L3-network.md` | PARTIAL IP addresses are PII in GDPR/PIPA; mask or hash before retention → `signals/privacy.md §IP addresses` |
+| cross-application | PASS inter-VPC/peering flow metrics → `layers/L3-network.md`, `signals/metrics.md` | PASS VPC flow logs across peering or transit gateway → `layers/L3-network.md`, `signals/logs.md` | PARTIAL L3 packets carry no trace context natively; use trace-ID in DNS or SNI side-channel only → `layers/L3-network.md`, `boundaries/cross-application.md` | N/A | PARTIAL cross-VPC egress cost attribution; unreliable without flow tagging → `signals/cost.md` | PASS inter-VPC flow audit for SOC2 network controls → `signals/audit.md`, `layers/L3-network.md` | PARTIAL source/destination IPs crossing application boundary are PII candidates → `signals/privacy.md §IP addresses` |
+| slo | N/A; SLO error budgets are defined at L7; L3 availability feeds infra health at most | N/A; VPC flow logs are operational, not SLO inputs | N/A | N/A | N/A | N/A | N/A |
+| release | N/A; release events carry no L3 signal | N/A | N/A | N/A | N/A | N/A | N/A |
+
+---
+
+### Layer: L4-transport
+
+L4 is the TCP/UDP/QUIC transport layer. eBPF (Beyla, Pixie) is the primary observability mechanism, producing socket-level metrics and off-CPU flame graphs without code changes. TCP retransmits, RTT, and connection states are rich signals. Trace context is not native to TCP; it lives at the application framing layer. SLO and release boundaries are application concerns.
+
+| Boundary \ Signal | metrics | logs | traces | profiles | cost | audit | privacy |
+|---|---|---|---|---|---|---|---|
+| multi-tenant | PASS per-tenant TCP retransmit rate, RTT histograms via eBPF socket filter → `layers/L4-transport.md §eBPF`, `boundaries/multi-tenant.md` | PASS TCP connection lifecycle events per tenant socket namespace → `layers/L4-transport.md`, `signals/logs.md` | PARTIAL TCP is not trace-native; mesh or L7 must carry trace context; L4 can log socket tuples for correlation → `layers/L4-transport.md`, `signals/traces.md` | PASS eBPF CPU/off-CPU profiling at socket-level covers L4 overhead; pprof-compatible output → `signals/profiles.md`, `layers/L4-transport.md §eBPF` | PARTIAL L4 byte volume per tenant as cost proxy; not a direct cost signal → `signals/cost.md` | PARTIAL connection-level audit (who connected to what port, when); useful for SOC2 network access controls → `signals/audit.md`, `layers/L4-transport.md` | PARTIAL source IPs in TCP metadata are PII; apply same IP-masking rules as L3 → `signals/privacy.md §IP addresses` |
+| cross-application | PASS cross-service TCP RTT and retransmit metrics via eBPF → `layers/L4-transport.md §eBPF`, `signals/metrics.md` | PASS TCP connection events across service socket pairs → `layers/L4-transport.md`, `signals/logs.md` | PARTIAL trace context does not exist at L4; identify cross-app flows by socket 5-tuple and correlate to L7 spans → `layers/L4-transport.md`, `boundaries/cross-application.md` | PASS eBPF off-CPU profiles covering network-wait time across application boundaries → `signals/profiles.md`, `layers/L4-transport.md §eBPF` | PARTIAL cross-application byte volume as FinOps signal; correlate with L7 cost attribution for accuracy → `signals/cost.md` | PASS TCP connection audit across trust boundaries → `signals/audit.md`, `layers/L4-transport.md` | PARTIAL connection metadata contains IPs; apply masking at pipeline ingress → `signals/privacy.md` |
+| slo | N/A; SLO windows are application-defined; L4 connection success rate may inform infra SLI but is not a canonical SLO boundary | N/A | N/A | N/A | N/A | N/A | N/A |
+| release | N/A; release events are application-layer; L4 has no deployment semantic | N/A | N/A | N/A | N/A | N/A | N/A |
+
+---
+
+### Layer: mesh
+
+The service mesh layer (Istio, Linkerd, Envoy) operates as a transparent L4-L7 proxy sidecar or ambient mode agent. It provides zero-code instrumentation, mTLS metadata as security context, and W3C Trace Context propagation without application code changes. This layer has the strongest native multi-signal coverage of any layer in this skill.
+
+| Boundary \ Signal | metrics | logs | traces | profiles | cost | audit | privacy |
+|---|---|---|---|---|---|---|---|
+| multi-tenant | PASS per-tenant RED (Rate/Error/Duration) metrics from Envoy telemetry; tagged via `tenant.id` baggage → `layers/mesh.md`, `boundaries/multi-tenant.md §Metric attribution`, `signals/metrics.md` | PASS Envoy access logs per tenant with baggage-derived tenant tag → `layers/mesh.md`, `signals/logs.md` | PASS Envoy zero-code span injection + W3C Baggage `tenant.id` propagation → `layers/mesh.md`, `boundaries/multi-tenant.md`, `signals/traces.md` | PARTIAL mesh proxies add latency overhead visible in eBPF profiles; not a mesh-native profiling source → `signals/profiles.md`, `layers/mesh.md` | PARTIAL request-level cost attribution by tenant via mesh telemetry; feed into OpenCost → `signals/cost.md`, `layers/mesh.md` | PASS mTLS identity + access log provide SOC2 accountability trail per tenant → `signals/audit.md`, `layers/mesh.md` | PASS mTLS config enforces transport encryption; baggage scrubbing at ingress gateway removes PII before propagation → `signals/privacy.md`, `layers/mesh.md`, `resources/standards.md §W3C Baggage` |
+| cross-application | PASS cross-service RED metrics at mesh proxy; primary use case for service topology mapping → `layers/mesh.md`, `boundaries/cross-application.md`, `signals/metrics.md` | PASS Envoy access logs across service-to-service calls; correlation by `trace_id` → `layers/mesh.md`, `signals/logs.md` | PASS primary trace origin for cross-application spans; Envoy injects spans without code changes; W3C Trace Context propagation → `layers/mesh.md`, `boundaries/cross-application.md`, `signals/traces.md` | PARTIAL mesh overhead visible via eBPF off-CPU profiles on sidecar process; not mesh-native → `signals/profiles.md`, `layers/L4-transport.md §eBPF` | PASS request cost attribution across services; mesh provides per-service byte/request counts → `signals/cost.md`, `layers/mesh.md` | PASS mTLS peer identity in access log provides cross-application accountability → `signals/audit.md`, `layers/mesh.md` | PASS baggage trust boundary enforced at mesh ingress gateway; strip or validate external baggage → `signals/privacy.md`, `boundaries/cross-application.md`, `resources/standards.md §W3C Baggage` |
+| slo | PASS mesh request rate and error rate are primary SLI sources for latency and availability SLOs → `layers/mesh.md`, `boundaries/slo.md`, `signals/metrics.md §SLI` | PARTIAL Envoy access logs as burn-rate event source; log-based SLI possible but metric-based preferred → `signals/logs.md`, `boundaries/slo.md` | PARTIAL trace sampling must be configured to retain error traces for SLO error-budget forensics; not a direct SLO input → `signals/traces.md`, `boundaries/slo.md` | N/A; profiles do not feed SLO computation | N/A; cost is a separate FinOps concern, not an SLO boundary input | PARTIAL SLO audit trail: record burn-rate threshold breach events as audit log entries → `signals/audit.md`, `boundaries/slo.md` | N/A |
+| release | PASS mesh request split metrics for canary traffic (Flagger/Argo Rollouts proxy rules) → `layers/mesh.md`, `boundaries/release.md`, `signals/metrics.md` | PASS deployment event injected as Envoy log entry with `service.version` tag → `layers/mesh.md`, `signals/logs.md`, `boundaries/release.md` | PARTIAL canary proxy rules at mesh layer route a percentage of traces to new version; trace header carries `service.version` → `layers/mesh.md`, `boundaries/release.md`, `signals/traces.md` | N/A | PARTIAL canary cost delta visible via per-version request counts in mesh telemetry → `signals/cost.md`, `boundaries/release.md` | PASS release event audit: mesh records version-tagged traffic split events → `signals/audit.md`, `boundaries/release.md` | N/A |
+
+---
+
+### Layer: L7-application
+
+L7 is the application layer: HTTP/gRPC, RUM (web + mobile), crash analytics, and messaging. This is the primary signal domain for this skill. All seven signals are meaningful here. Multi-tenant instrumentation is richest at L7 because application code has full access to user context, tenant identifiers, and business semantics.
+
+| Boundary \ Signal | metrics | logs | traces | profiles | cost | audit | privacy |
+|---|---|---|---|---|---|---|---|
+| multi-tenant | PASS per-tenant RED + custom business metrics tagged with `tenant.id`; histogram by tenant tier → `signals/metrics.md §tenant`, `boundaries/multi-tenant.md`, `layers/L7-application/web-rum.md` | PASS structured log stream with `tenant.id` on every record; tenant log routing via OTel Collector `routing_connector` → `signals/logs.md`, `boundaries/multi-tenant.md` | PASS W3C Baggage `tenant.id` propagated on every span; trace exported per-tenant collector pipeline if residency required → `signals/traces.md`, `boundaries/multi-tenant.md`, `resources/standards.md §W3C Baggage` | PASS per-tenant continuous profiling with Parca/Pyroscope; label by `tenant.id` for flame graph isolation → `signals/profiles.md`, `boundaries/multi-tenant.md` | PASS request-level cost attribution by tenant; feed OpenCost unit economics model → `signals/cost.md §unit economics`, `boundaries/multi-tenant.md` | PASS per-tenant audit event stream; WORM storage per tenant for SOC2 evidence → `signals/audit.md`, `boundaries/multi-tenant.md` | PASS tenant-scoped PII redaction; per-tenant privacy config for GDPR right-to-erasure → `signals/privacy.md`, `boundaries/multi-tenant.md` |
+| cross-application | PASS inter-service request rate and latency histograms; use `service.name` + `service.peer.name` labels → `signals/metrics.md`, `boundaries/cross-application.md` | PASS correlation log: `trace_id` + `span_id` on every log record enables log-trace join across applications → `signals/logs.md`, `boundaries/cross-application.md` | PASS primary trace origin; W3C Trace Context `traceparent` on all outbound calls; DDD namespace baggage for bounded-context attribution → `signals/traces.md`, `boundaries/cross-application.md`, `resources/standards.md §W3C Trace Context` | PASS application-level profiling showing cross-service call overhead; correlate with traces via `trace_id` label → `signals/profiles.md`, `boundaries/cross-application.md` | PASS request cost attribution across services; per-service unit cost model → `signals/cost.md`, `boundaries/cross-application.md` | PASS cross-application audit events carry caller identity and `trace_id` for accountability chain → `signals/audit.md`, `boundaries/cross-application.md` | PASS baggage trust boundary at API gateway; validate or strip PII-bearing baggage from external callers → `signals/privacy.md`, `boundaries/cross-application.md`, `resources/standards.md §W3C Baggage` |
+| slo | PASS SLI metric computation (availability, latency p99, error rate); SLO targets defined in OpenSLO YAML → `signals/metrics.md §SLI`, `boundaries/slo.md`, `resources/observability-as-code.md` | PARTIAL log-based SLI possible (error log count / total); valid for non-metrics-instrumented services but metric-based SLI preferred → `signals/logs.md`, `boundaries/slo.md` | PARTIAL critical path trace sampling for SLO forensics; tail-sampler keeps error traces within error budget window → `signals/traces.md`, `boundaries/slo.md`, `transport/sampling-recipes.md` | N/A; profiling does not feed SLO computation directly | N/A; cost SLO is a FinOps budget concern, not an error-budget SLO | PARTIAL SLO breach audit record: persist burn-rate threshold crossing as immutable audit event → `signals/audit.md`, `boundaries/slo.md` | N/A |
+| release | PASS release marker metric event; `service.version` label on all metrics for before/after comparison → `signals/metrics.md`, `boundaries/release.md`, `resources/observability-as-code.md` | PASS deployment event as structured log record; deployment SHA, version, and rollout strategy logged → `signals/logs.md`, `boundaries/release.md` | PASS `service.version` attribute on all spans; canary trace routing by version tag for A/B error comparison → `signals/traces.md`, `boundaries/release.md` | PARTIAL pre/post-release profile comparison for performance regression detection; Parca/Pyroscope diff view → `signals/profiles.md`, `boundaries/release.md` | PARTIAL release cost delta: compare per-request cost across canary vs stable version → `signals/cost.md`, `boundaries/release.md` | PASS release audit event: immutable record of who deployed what version when → `signals/audit.md`, `boundaries/release.md` | N/A |
+
+---
+
+## 5. Caveats
+
+Rationale for the rarer N/A / PARTIAL cells. Each entry justifies the marker and points to the authoritative file.
+
+- **C1. L3 × {SLO, release}**: N/A. SLO and release are L7 constructs (OpenSLO YAML, canary markers); L3 IP flow feeds infra health, not application error budgets. An L3 event (BGP leak, PMTUD black hole) that causes an SLO burn surfaces as an L7 error spike first; investigate via `resources/incident-forensics.md` 6-dim localization, not via L3 SLO policy. See `boundaries/slo.md`, `layers/L3-network.md`.
+- **C2. L4 × {SLO, release}**: same rationale as C1. TCP connection success rate can serve as a fallback SLI when L7 instrumentation is absent, but it is not recommended. See `boundaries/slo.md §fallback SLI sources`.
+- **C3. {L3, L4} × profiles**: profiling (Parca, Pyroscope, OTEP 0239) is process-level; L3 has no equivalent artifact. L4 × profiles PASS reflects eBPF kernel-socket and off-CPU wait measurements; the closest meaningful artifact; attributed to `layers/L4-transport.md §eBPF`.
+- **C4. {L3, L4} × traces (PARTIAL)**: W3C Trace Context lives in HTTP/gRPC headers, not IP/TCP. The PARTIAL reflects a correlation technique (log socket 5-tuple alongside trace ID) rather than native propagation. Native propagation begins at mesh or L7. See `resources/standards.md §W3C Trace Context`.
+- **C5. mesh × profiles (PARTIAL)**: sidecar proxies (Envoy, Linkerd-proxy) are separate processes; their CPU overhead is visible via eBPF on the sidecar, but that is an L4 artifact, not a mesh-native profiling signal. Mesh exposes no profiling API. See `signals/profiles.md`, `layers/L4-transport.md §eBPF`.
+- **C6. {L3, L4} × cost (PARTIAL)**: byte volume is a cost proxy sufficient for cloud egress billing but insufficient for unit economics, which requires L7 per-request attribution. See `signals/cost.md §egress attribution`.
+- **C7. {L3, L4} × privacy (PARTIAL)**: IP addresses are personal data (GDPR Art. 4(1), PIPA equivalent). VPC flow and TCP connection logs are operationally required but must be masked, hashed, or pseudonymized before long-term retention. Masking technique (prefix truncation, HMAC+salt, pseudonymization) is in `signals/privacy.md §IP addresses`.
+- **C8. {L3, L4} × {SLO, release} × audit**; audit requires an identity-bearing, time-bound subject; IP/TCP-layer events carry no such subject tied to an SLO policy or deployment action. Audit at SLO/release boundaries is always L7-originated. See `signals/audit.md`.
+
+---
+
+## 6. Cell Count Verification
+
+| Layer | Boundaries | Signals | Cells |
+|-------|-----------|---------|-------|
+| L3-network | 4 | 7 | 28 |
+| L4-transport | 4 | 7 | 28 |
+| mesh | 4 | 7 | 28 |
+| L7-application | 4 | 7 | 28 |
+| **Total** | | | **112** |
+
+All 112 cells are populated. No cell is blank.
+
+---
+
+## 7. Cross-references
+
+All files referenced in this matrix belong to the `oma-observability` skill tree.
+
+| Category | Files |
+|----------|-------|
+| Layers | `layers/L3-network.md`, `layers/L4-transport.md`, `layers/mesh.md`, `layers/L7-application/web-rum.md`, `layers/L7-application/mobile-rum.md`, `layers/L7-application/crash-analytics.md` |
+| Boundaries | `boundaries/multi-tenant.md`, `boundaries/cross-application.md`, `boundaries/slo.md`, `boundaries/release.md` |
+| Signals | `signals/metrics.md`, `signals/logs.md`, `signals/traces.md`, `signals/profiles.md`, `signals/cost.md`, `signals/audit.md`, `signals/privacy.md` |
+| Transport | `transport/sampling-recipes.md`, `transport/collector-topology.md` |
+| Resources | `resources/standards.md`, `resources/incident-forensics.md`, `resources/observability-as-code.md`, `resources/meta-observability.md` |
+
+---
+
+## 8. Coverage updates
+
+- **On semconv promotion** (Development → RC → Stable): re-evaluate PARTIAL cells that cite stability as a caveat; promote to PASS if the semconv group is now stable.
+- **On new layer or boundary addition**: add the corresponding table section and update all cross-references. Signal columns change only with a taxonomy revision.
+- **On N/A re-evaluation**: add a Caveat entry (Section 5) explaining why a previously N/A combination is now meaningful before changing the cell marker.