oh-my-agent-commandcode 0.0.0-stage → 20.38.4

package/skills/oma-observability/resources/signals/privacy.md

@@ -0,0 +1,351 @@
+---
+otel_semconv: "1.43.0 (2026-07); GDPR: 2016/679; PIPA: 2023 amendment"
+---
+
+# Privacy in Telemetry
+
+## 1. Scope and Distinction from Audit
+
+**Privacy** answers: *what personal data flows through telemetry, and how do we minimize, redact, or anonymize it?*
+
+**Audit** answers: *what actions occurred, who performed them, and can we prove it?*
+
+These are architecturally opposite concerns. See Design Decision D5 for the split rationale.
+
+| Dimension | Privacy | Audit |
+|-----------|---------|-------|
+| Goal | Collect less, protect subjects | Prove what happened |
+| Mutability | Supports deletion (right to erasure) | Immutable, append-only |
+| Retention | Short raw tier; long anonymized tier | Long-term (7–10 years typical) |
+| Consumer | DPO, privacy engineers, legal | Security ops, compliance, auditors |
+| Regulation driver | GDPR Art. 5, Art. 17; PIPA § 21 | SOX, PCI-DSS, ISO/IEC 27001 |
+
+**D5 Split rationale.** Merging privacy and audit into a single signal pipeline produces conflicting retention policies (erasure vs. immutability), access control models (broad deletion rights vs. tamper-evident storage), and regulatory obligations. Separating them lets each pipeline satisfy its own obligations without compromise. Cross-references between the two are explicit rather than implicit (see §9 and `audit.md`).
+
+Primary consumers of this file: Data Protection Officers (DPO), privacy engineers, legal counsel.
+
+---
+
+## 2. Regulatory Drivers
+
+| Regulation | Jurisdiction | Key Articles / Principles | Max Penalty |
+|------------|-------------|--------------------------|-------------|
+| GDPR | EU / EEA | Art. 5(1)(c) minimization; Art. 5(1)(e) storage limitation; Art. 17 erasure; Art. 32 security | 4% global annual turnover |
+| PIPA | South Korea | § 3 minimization; § 21 destruction; § 29 safety measures; separate sensitive data consent | KRW 300M or 3% revenue |
+| CCPA / CPRA | California, US | Opt-out of sale; right to delete; sensitive PI category | USD 7,500 per intentional violation |
+| HIPAA | United States | PHI definition; minimum necessary standard; breach notification | USD 100–50,000 per violation |
+| PIPEDA | Canada | Fair information principles; breach of security safeguards | CAD 100,000 per violation |
+| LGPD | Brazil | Art. 6 finality + necessity; Art. 18 erasure rights | 2% Brazil revenue, max BRL 50M |
+
+**Misclassification risk.** Treating identifiable pseudonymized data as anonymous can cause applicable GDPR duties to be missed. A theoretical maximum fine is not evidence that a particular misclassification received that penalty. Assess reasonable identification using §3.
+
+---
+
+## 3. Anonymization vs Pseudonymization vs Tokenization
+
+Assess whether a person can reasonably be identified, directly or indirectly, using this data and other available information. GDPR Recital 26 considers means likely to be used by the controller or another person, including singling out; inability to recover the original value alone does not establish anonymity.
+
+| Technique | Identification/linkage assessment | GDPR Applies? | Typical Tooling |
+|-----------|-------------|---------------|-----------------|
+| Anonymization | No reasonable identification, including linkage to other available data | No, only when that assessment is met | Evaluated aggregation, generalization, or privacy-preserving methods |
+| Pseudonymization | Additional information or other reasonably available means may re-identify | Yes | HMAC with separately held key, format-preserving encryption (FPE), AES-FF1 |
+| Tokenization | Vault lookup can link the token to the person | Yes | Payment token vaults, HSM-backed services |
+| Hashing (no salt, low entropy) | Enumeration/linkage can identify despite one-way hashing | Yes when reasonably identifiable | SHA-256 without salt on numeric IDs |
+
+**Decision rule.** Evaluate reasonable re-identification, linkage, and singling-out risks and record the assumptions. Treat the result as personal data if identification remains reasonably possible or the assessment is unresolved. A stable hash or deletion of a local lookup key does not by itself prove anonymity.
+
+**Key separation.** Keep additional identifying information and keys separately with independent access controls. Art. 32 requires security measures appropriate to risk; it does not mandate a different geographic region. Select the storage/access boundary according to the threat model, residency policy, and applicable transfer requirements.
+
+---
+
+## 4. Common PII in Telemetry
+
+| PII Category | Where It Appears in Telemetry | OTel Semconv Attribute |
+|-------------|------------------------------|------------------------|
+| IP address | HTTP server spans, logs, network metrics | `client.address`, `network.peer.address` |
+| User ID | Span attributes, baggage, log body | `enduser.id` (deprecated → `user.id`) |
+| Email, name, phone | Log body, error traces, baggage | None; must not be added |
+| Session token / cookie | HTTP headers, span attributes | `http.request.header.*` |
+| Browser user-agent | HTTP server spans | `user_agent.original` |
+| Geolocation | IP-derived enrichment, custom attrs | `geo.city`, `geo.country_iso_code` |
+| Query string parameters | URL full string | `url.full`, `url.query` |
+| Stack trace with user data | Exception events | `exception.stacktrace` |
+| Request body on error | Trace event attributes | Custom; avoid entirely |
+| Baggage values | W3C Baggage header, propagated downstream | See `../boundaries/cross-application.md §Baggage rules` |
+
+Baggage is especially dangerous: values propagate across service boundaries and can be logged or traced by any downstream collector. Apply trust-boundary filters before injecting user-derived values into baggage.
+
+---
+
+## 5. PII Handling Rules by Field
+
+| Field | Required Action | Technique |
+|-------|----------------|-----------|
+| IPv4 address | Mask last octet (`192.0.2.0`) or drop | Truncation / drop |
+| IPv6 address | Mask last 80 bits or drop | Truncation / drop |
+| Email address | Drop local part (`***@example.com`) or drop entirely | Redaction |
+| User ID (internal) | Salted HMAC with vault-managed key | Pseudonymization |
+| Card number, SSN | Drop entirely; never collect | Drop |
+| Phone number | Drop entirely | Drop |
+| Session token, password, `Authorization` header | Drop entirely | Drop |
+| Timestamp (high-resolution) | Truncate to minute (metrics) or hour (logs) depending on aggregation tier | Generalization |
+| Geolocation (precise) | Truncate to city or region | Generalization |
+| User-agent string | Keep major browser family only, or drop | Generalization / drop |
+| Query string | Drop or allowlist known safe params | Drop / allowlist |
+| Request body | Drop on error traces; never capture by default | Drop |
+
+---
+
+## 6. OTel Collector Processors for Redaction
+
+Use a layered pipeline: **attributes** (known PII keys) → **transform/OTTL** (pattern-based) → **redaction** (safety-net allowlist) → storage.
+
+### 6.1 `attributes` Processor: Named Key Actions
+
+```yaml
+processors:
+  attributes/minimize:
+    actions:
+      - key: user.email
+        action: delete
+      - key: user.id
+        action: hash
+      - key: http.request.header.authorization
+        action: delete
+      - key: http.request.header.cookie
+        action: delete
+      - key: url.query
+        action: delete
+```
+
+> **Warning on `action: hash`**: the Collector's built-in `hash` action does not apply a salt. For low-entropy inputs such as numeric user IDs, an unsalted hash is reversible by brute force and is treated as pseudonymization (not anonymization) under GDPR. See §8 for salted-hash alternatives at the SDK layer.
+
+### 6.2 `transform` Processor (OTTL): Pattern-Based Transformation
+
+```yaml
+processors:
+  transform/ip_mask:
+    error_mode: ignore
+    trace_statements:
+      - context: span
+        statements:
+          # IPv4: replace last octet with .0
+          - replace_pattern(attributes["client.address"], "\\.\\d+$", ".0")
+          # Drop precise geolocation
+          - delete_key(attributes, "geo.coordinates")
+          # Truncate email to domain only
+          - replace_pattern(attributes["user.email"], "^[^@]+", "***")
+```
+
+### 6.3 `redaction` Processor: Allowlist Safety Net
+
+```yaml
+processors:
+  redaction/allowlist:
+    allow_all_keys: false
+    allowed_keys:
+      - http.request.method
+      - http.response.status_code
+      - http.route
+      - service.name
+      - trace.id
+      - span.id
+      - db.system
+      - rpc.system.name
+    blocked_values:
+      - "\\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Z|a-z]{2,}\\b"   # email pattern
+      - "\\b(?:\\d[ -]?){13,16}\\b"                                    # card number pattern
+    summary: debug
+```
+
+### 6.4 Pipeline Assembly
+
+```yaml
+service:
+  pipelines:
+    traces:
+      receivers: [otlp]
+      processors:
+        - attributes/minimize
+        - transform/ip_mask
+        - redaction/allowlist
+        - batch
+      exporters: [otlp/backend]
+```
+
+---
+
+## 7. SDK-Layer Redaction (Prefer over Pipeline)
+
+Redacting at the instrumentation layer is always preferable: PII never enters the pipeline or wire.
+
+| SDK / Tool | Mechanism |
+|------------|-----------|
+| OTel SDK (all languages) | Attribute processor on `TracerProvider` at construction time |
+| Sentry | `before_send` / `before_send_transaction` callbacks |
+| Datadog | `before_send` callback; `obfuscation_config` in agent |
+| OpenTelemetry JS/Python | `SpanProcessor` with custom `onStart` / `onEnd` |
+
+**Rule.** SDK-layer redaction is the first line of defense. Collector processors are the second line. Treat collector processors as a safety net, not the primary mechanism.
+
+---
+
+## 8. Salted Hashing Caveats
+
+OTel Collector's built-in `hash` action uses SHA-256 **without salt**. On low-entropy inputs (numeric user IDs, 6-digit codes), this is reversible via brute-force rainbow tables.
+
+**Requirements for safe pseudonymization:**
+
+1. Apply salt using a vault-managed key (e.g., HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager).
+2. Store the key in an independently access-controlled vault separated from telemetry access. Geographic separation is an optional risk-based design choice, subject to residency and transfer requirements.
+3. Rotate the salt on a defined schedule (quarterly recommended); document rotation in the key management policy.
+4. Use HMAC-SHA256 with the salt, not bare SHA-256.
+5. Because the Collector `hash` action cannot inject vault-managed salts, perform HMAC at the **SDK layer** before emitting spans.
+
+```python
+# SDK-layer HMAC pseudonymization (Python example)
+import hmac, hashlib, os
+
+def pseudonymize_user_id(user_id: str, vault_key: bytes) -> str:
+    return hmac.new(vault_key, user_id.encode(), hashlib.sha256).hexdigest()
+```
+
+---
+
+## 9. Retention and Right to Erasure (GDPR Art. 17)
+
+| Tier | Content | Max Retention | Erasure Support |
+|------|---------|---------------|-----------------|
+| Raw (with PII) | Full spans, logs, unredacted | Example: purpose-justified 7 days | Delete eligible records after checking grounds/exceptions |
+| Redacted short | Direct identifiers removed; assess remaining linkage | Example: 30 days | Required when remaining data is personal and an erasure ground applies |
+| Anonymized long | Data assessed as no longer reasonably identifiable | Purpose-defined period | GDPR erasure does not apply only when the anonymity assessment is met |
+| Audit events | Access/action evidence, minimized for its purpose | Applicable law/contract and purpose-specific schedule | Assess erasure grounds and documented legal-retention/legal-claim exceptions |
+
+Erasure pipeline: request → verify identity and the applicable erasure ground → identify personal records across relevant tiers → check documented retention/legal-claim exceptions → delete eligible records under the approved schedule → record the action and justified exceptions. Audit records do not have a blanket exemption.
+
+Cross-reference: `../meta-observability.md §Retention Matrix` for full retention policy. Cross-reference: `audit.md` for erasure audit trail requirements.
+
+---
+
+## 10. Cross-Border Transfer
+
+GDPR Chapter V prohibits transfer of personal data to non-adequate countries without a legal mechanism.
+
+| Mechanism | When to Use |
+|-----------|-------------|
+| Standard Contractual Clauses (SCC) | EU → US, EU → most non-adequate countries |
+| Binding Corporate Rules (BCR) | Intra-group transfers |
+| Adequacy decision | EU → UK, Japan, Canada (PIPEDA), South Korea (partial) |
+| PIPA § 17 | KR → non-KR: contractual safeguards + PIPC notification |
+
+**Collector routing for regional compliance.**
+
+```yaml
+# Route EU telemetry to EU backend, KR to KR backend
+connectors:
+  routing:
+    default_pipelines: [traces/global]
+    table:
+      - statement: attributes["deployment.region"] == "eu-west-1"
+        pipelines: [traces/eu]
+      - statement: attributes["deployment.region"] == "ap-northeast-2"
+        pipelines: [traces/kr]
+```
+
+Cross-reference: `../transport/collector-topology.md §Multi-cluster + regional`.
+
+---
+
+## 11. TLS Context Attributes (L6 Development Status)
+
+OTel `tls.*` semantic conventions are **Development** stability tier as of semconv 1.43.0 (verified 2026-07). See `../standards.md` for stability tier definitions.
+
+| Attribute | Type | Use Case |
+|-----------|------|----------|
+| `tls.protocol.version` | string | Detect TLS 1.0/1.1 downgrade attacks |
+| `tls.cipher` | string | Flag weak ciphers (RC4, DES, 3DES) |
+| `tls.established` | boolean | Alert on handshake failures |
+| `tls.resumed` | boolean | Session resumption analysis |
+| `tls.server.certificate.expiry` | int (epoch) | Certificate expiry alerting |
+
+**Scope boundary.** These attributes provide *security context* for observability purposes (downgrade detection, cert expiry alerting). Full TLS inspection (decrypting payload for content analysis) is out of scope; use dedicated network inspection tooling.
+
+**Privacy note.** TLS cipher and protocol version are not PII. Certificate subject fields (CN, SAN) may contain hostnames; do not log end-user certificate subjects from mTLS without legal basis.
+
+---
+
+## 12. Backend Access Control (RBAC on Observability Data)
+
+Production traces and logs containing even pseudonymized data are sensitive. Access must be role-gated.
+
+| Role | Scope | Time Range | Example Tools |
+|------|-------|-----------|---------------|
+| On-call engineer | Own service traces + logs | Last 24h | Grafana folder permission |
+| Incident responder | Cross-service traces | Incident window | Datadog Teams, Honeycomb environments |
+| Security analyst | All services, security events | 30 days | Separate security-tier datasource |
+| Auditor | Audit events only (see `audit.md`) | Full retention | Read-only audit index |
+| Finance | Cost metrics only | 90 days | See `cost.md` |
+| DPO / Privacy engineer | Redaction pipeline config, raw tier | Short window only | Separate admin role |
+
+**Environment isolation.** Production backends must be strictly separated from non-production. Never allow dev/staging pipelines to receive or store production telemetry; this is a direct PII leak path.
+
+Vendor-specific RBAC: Grafana folder permissions + team sync; Datadog Teams with scoped monitors; Honeycomb environments with team-scoped API keys.
+
+---
+
+## 13. Third-Party Processor Obligations (GDPR Art. 28)
+
+Observability vendors receiving telemetry containing personal data are **data processors** under GDPR. A Data Processing Agreement (DPA) is mandatory before routing data to them.
+
+| Obligation | Action Required |
+|------------|----------------|
+| DPA in place | Signed before any data flows to vendor |
+| Sub-processor transparency | Vendor must disclose and notify of sub-processor changes |
+| Data residency | Confirm vendor's storage region matches your compliance obligations |
+| Deletion / return | Vendor must support data deletion on contract termination |
+
+Common vendors requiring DPA review: Datadog, Honeycomb, Grafana Cloud, Sentry, New Relic, Elastic Cloud. Each vendor publishes a DPA at their legal/privacy page. Reference your legal team's approved vendor list before routing production data.
+
+---
+
+## 14. Matrix Coverage Notes
+
+The following cells in `../matrix.md` have privacy-specific guidance from this file:
+
+| Layer × Boundary | Privacy Status | Notes |
+|-----------------|---------------|-------|
+| L3 (Network) × any | Warning | IP addresses are GDPR Art. 4(1) personal data identifiers; mask or drop |
+| L4 (Transport) × any | Warning | Connection 5-tuple (src IP, dst IP, ports) can identify individuals |
+| L7 (Application) × multi-tenant | Primary coverage | Per-tenant data minimization; tenant ID must not leak across boundaries |
+| L7 × cross-application | Warning | Baggage crosses trust boundaries; apply filter at ingress (§4) |
+| Release / feature flags × any | Overlap | Feature flag cohort membership may constitute profiling under GDPR Art. 4(4) |
+
+---
+
+## 15. Anti-Patterns
+
+> These entries feed into `../anti-patterns.md §Section A Privacy`.
+
+| Anti-Pattern | Risk | Remediation |
+|-------------|------|-------------|
+| Claiming anonymity from hashing or local irreversibility alone | Personal data may be misclassified and its safeguards omitted | Apply the re-identification assessment (§3); re-classify and update ROPA |
+| Hash without salt on low-entropy IDs | Rainbow-table reversal; GDPR breach | HMAC-SHA256 with vault-managed salt at SDK layer (§8) |
+| Key and telemetry accessible through the same permissions | One access compromise enables re-identification | Separate vault/IAM and access audit; choose geography under the risk and residency policy |
+| Baggage crossing trust boundary without filter | PII propagated to untrusted downstream collectors | Apply baggage allowlist at service ingress (cross-ref `../boundaries/cross-application.md`) |
+| Observability backend open to all engineers (no RBAC) | PII exposure; compliance failure | Implement role-scoped access per §12 |
+| Routing to 3rd-party vendor without DPA | GDPR Art. 28 violation | Block data flow until DPA is signed (§13) |
+| Capturing raw request body on error trace | Request bodies frequently contain credentials, PII | Drop request body by default; allowlist specific non-sensitive fields at SDK layer (§7) |
+| Cross-region telemetry routing without GDPR mechanism | GDPR Chapter V violation | Implement routing connector per §10; confirm SCC or adequacy decision |
+| OTel Collector `hash` action on production user IDs | Unsalted; brute-force reversible | Migrate to SDK-layer HMAC with vault key (§8) |
+| Logging `user_agent.original` without assessment | User-agent can be a unique identifier for profiling | Generalize to browser family only or drop (§5) |
+
+---
+
+## References
+
+- GDPR text: <https://gdpr-info.eu>
+- PIPA (Korean): <https://www.pipc.go.kr>
+- OTel sensitive data handling: <https://opentelemetry.io/docs/security/handling-sensitive-data/>
+- OTel Collector processors: <https://opentelemetry.io/docs/collector/configuration/>
+- W3C Baggage PII warning: <https://www.w3.org/TR/baggage/>
+- Cross-references: `../standards.md`, `../matrix.md`, `../meta-observability.md`, `../boundaries/cross-application.md`, `../transport/collector-topology.md`, `audit.md`, `cost.md`

package/skills/oma-observability/resources/signals/profiles.md

@@ -0,0 +1,245 @@
+---
+otel_spec: "1.x (stable API/SDK)"
+otel_semconv: "1.43.0 (2026-07)"
+notes:
+  - "Profiles signal (OTEP 0239) is Development-status; semconv in development"
+---
+
+# Profiles Signal
+
+> **EXPERIMENTAL; OTEP 0239 Development (as of 2026-Q2)**
+> Continuous profiling tooling (Parca, Pyroscope) is production-ready.
+> The OpenTelemetry profiling data model and OTLP signal are in alpha / active development; semantic conventions and wire format may change without backward compatibility.
+> Do NOT define SLOs against OTel-native profiling attributes. Treat any `profile.*` OTel semconv as experimental tier.
+> Cross-ref: `../standards.md §3 OTel semconv stability` for tier definitions.
+
+---
+
+## 1. Scope
+
+Continuous profiling; whole-cluster or whole-service CPU, memory, and allocation sampling running at low, steady overhead in production. Flamegraphs are the primary visualization artifact. The output answers the question "in which function is time or memory being spent?"
+
+This file does NOT cover ad-hoc developer profiling sessions (e.g., `go tool pprof http://localhost:6060/debug/pprof/profile` run once during a development investigation). Those are single-use diagnostics. Continuous profiling generates persistent, comparable data over time and is as much a production signal as metrics or logs.
+
+---
+
+## 2. The 5th Pillar: MELT to MELT+P
+
+Observability was originally described as three pillars (Metrics, Logs, Traces). Events as LogRecords and cost extended that. Profiles add the fifth pillar.
+
+| Signal | Answers |
+|--------|---------|
+| Metrics | What is slow / broken? (rate, error rate, latency quantile) |
+| Traces | Where is it slow? (which service, which span, which dependency) |
+| Profiles | In which function is CPU time or memory spent? (line-level attribution) |
+| Logs | Why, in context? (structured event with variables and error detail) |
+| Cost | What is the financial impact of the above? |
+
+Correlation is the mechanism that makes MELT+P actionable rather than five isolated views. Profiles share resource labels (`service.name`, `service.version`, `k8s.pod.name`) with metrics and traces, and can be linked to a specific trace by attaching a `trace_id` label to profile samples; a pattern pioneered by Grafana Tempo + Pyroscope.
+
+---
+
+## 3. Tools Landscape (as of 2026-Q2)
+
+Verify live status at <https://landscape.cncf.io>.
+
+| Tool | Category | Mechanism | CNCF Status | Notes |
+|------|----------|-----------|-------------|-------|
+| Parca | OSS, continuous | eBPF kernel-level, no agent code | Sandbox | Written in Go; Polar Signals led; whole-cluster, no language SDK required for CPU profiles |
+| Grafana Pyroscope | OSS / SaaS, continuous | Pull + push (eBPF + language SDKs) | Sandbox (verify; CNCF status uncertain post-Grafana Labs acquisition, 2023) | Formerly independent; Grafana acquired 2023; integrated into Grafana stack |
+| Polar Signals Cloud | Commercial SaaS | Parca-based, hosted | N/A | Commercial offering backed by Parca OSS; Polar Signals is the primary Parca maintainer |
+| Go `net/http/pprof` | Language-specific, on-demand | HTTP handler; sample on request | N/A | Built-in; not continuous; heavier per request; MUST NOT be left exposed on production endpoints |
+| Fgprof | Language-specific | Wall-clock profiler for Go | N/A | Supplements `pprof` by measuring off-CPU time as well as on-CPU |
+| py-spy | Language-specific, continuous | Native code (Rust); attaches without code change | N/A | Low overhead; works on CPython; no SDK injection |
+| pyroscope-python | Language-specific, continuous | Python SDK; sends to Pyroscope server | N/A | Requires SDK import; supports async runtimes |
+| async-profiler | JVM, continuous | JVMTI + `perf_events`; async-safe | N/A | Primary choice for JVM profiling; supports CPU, alloc, lock, wall modes |
+| JFR (Java Flight Recorder) | JVM, continuous | JVM-native; available from JDK 11+ | N/A | Enable with JVM flags; low overhead; integrates with async-profiler |
+| `0x` / V8 Inspector | Node.js, on-demand | V8 CPU profiler | N/A | `0x` wraps `node --prof` for flamegraph generation |
+| pyroscope-node | Node.js, continuous | Node SDK; sends to Pyroscope | N/A | Continuous alternative to `0x` |
+| stackprof | Ruby, continuous | Sampling profiler | N/A | Low overhead; supports wall, object, custom modes |
+| php-spx / phpspy | PHP, continuous | Extension (spx) or attach (spy) | N/A | `php-spx` requires extension install; `phpspy` attaches without code change |
+| Beyla / Pixie | eBPF, language-agnostic | eBPF user-space probes | Beyla CNCF Incubating (2024); Pixie CNCF Sandbox | No language integration needed; covers Go, Python, JVM, Node.js for latency profiling |
+
+**eBPF unified approach note:** For CPU and off-CPU profiling without language-level SDK integration, Parca, Beyla, and Pixie are the primary choices. They instrument at the kernel/eBPF level, making them language-agnostic for CPU time attribution.
+
+---
+
+## 4. Sampling Overhead
+
+Continuous profiling is viable in production only when overhead is provably low.
+
+| Tool / Mode | Typical CPU overhead | Sampling rate | Notes |
+|------------|---------------------|---------------|-------|
+| Parca (eBPF) | < 1% CPU | ~19 Hz per CPU core | Kernel-level; process namespaces handled correctly |
+| Pyroscope (eBPF mode) | < 1% CPU | Configurable, default ~100 Hz | Overhead scales with sampling rate; start conservatively |
+| async-profiler (CPU) | < 1% CPU | Configurable; default 100 Hz | AsyncGetCallTrace avoids safe-point bias |
+| JFR (CPU + alloc) | < 1% CPU typical | JVM-managed | Monitor GC pause impact separately |
+| py-spy | < 1% CPU | Configurable; default 100 Hz | External; no Python overhead |
+| `net/http/pprof` | ~5-10% during active profile | On-demand only | Never enable `/debug/pprof` permanently on production endpoints; see anti-patterns |
+
+Always measure baseline and profiled CPU utilization in staging before enabling continuous profiling in production. Set a budget threshold (e.g., < 1% CPU) and alert if actual overhead exceeds it.
+
+---
+
+## 5. OTEP 0239: OTel Profiling Signal Data Model
+
+Source: <https://github.com/open-telemetry/oteps/blob/main/text/profiles/0239-profiles-data-model.md>
+
+Status: **alpha / in active development (2026-Q2)**. The data model, wire format, and semantic conventions are all subject to breaking change.
+
+Key design points:
+- Profiles are transmitted as a fourth OTLP signal alongside metrics, traces, and logs. A new `ExportProfilesServiceRequest` protobuf message parallels `ExportMetricsServiceRequest`.
+- The data model derives from the pprof format (Google); a profile is a directed call graph with sample counts per stack frame.
+- Proposed profile types: CPU, Heap, Allocation, Mutex, Lock. Stored in `profile.type` attribute.
+- `profile.name` identifies the profile within a service and collection window.
+- Semantic conventions for profile attributes (e.g., mapping to `service.*`, `k8s.*`, `thread.*`) are in draft. Do not treat them as stable.
+
+```
+# Conceptual OTLP profile payload (pseudocode — not final spec)
+ProfilesData {
+  resource_profiles: [
+    ResourceProfiles {
+      resource: { attributes: [service.name, service.version, k8s.pod.name] }
+      scope_profiles: [
+        ScopeProfiles {
+          profiles: [
+            Profile {
+              profile_id: <16-byte random>
+              start_time_unix_nano: ...
+              end_time_unix_nano: ...
+              sample_type: [{ type: "cpu", unit: "nanoseconds" }]
+              sample: [{ location_id: [...], value: [...], label: [...] }]
+            }
+          ]
+        }
+      ]
+    }
+  ]
+}
+```
+
+Most production deployments (2026-Q2) use Parca's own gRPC API or Pyroscope's push API, not OTLP profiles. OTLP profile support in collectors and backends is experimental. Plan OTLP profile adoption only after the spec reaches RC status.
+
+---
+
+## 6. Use Cases
+
+| Use case | Profile type | How to investigate |
+|----------|-------------|-------------------|
+| CPU bottleneck; which function consumes CPU? | CPU | Flamegraph: wide frame = high CPU; sort by self-time |
+| Memory leak; heap growing over time? | Heap | Heap profile over hours; compare allocation counts |
+| Allocation rate; GC pressure? | Allocation | Allocation profile; identify hot allocation sites |
+| Lock contention; threads blocked? | Mutex / Lock | Mutex profile; look for frames holding locks across many samples |
+| Off-CPU; threads blocked on I/O or sleep? | Off-CPU (eBPF) | eBPF off-CPU flamegraph; I/O, syscall, and scheduler frames visible |
+
+---
+
+## 7. Correlation with Traces
+
+Profiles are the diagnostic tool you reach for after traces identify which service and span is slow. The linkage mechanism is attaching the current `trace_id` as a label on profile samples.
+
+```
+# Pyroscope Go SDK — attach trace_id to profile (conceptual)
+pyroscope.TagWrapper(ctx, pyroscope.Labels("trace_id", span.SpanContext().TraceID().String()), func() {
+    // code under profiling
+})
+```
+
+Grafana Tempo + Pyroscope integration implements this as a first-class UI feature: clicking a slow trace span offers a "View Profile" link that loads the flamegraph for that time window and service.
+
+**Limitation:** Profiles are statistical (sampled across all requests). Traces are per-request. The correlation is probabilistic; a trace_id label on a profile sample means that request was running during that sample, not that the sample is causally attributed to that exact request. This is a fundamental constraint of sampling-based profiling; document it clearly when presenting correlation results to stakeholders.
+
+---
+
+## 8. Mobile and Client-Side Profiling
+
+Client-side continuous profiling in production is not yet established practice. Platform tools provide opt-in session profiling:
+
+| Platform | Tool | Mode | Notes |
+|----------|------|------|-------|
+| iOS | Instruments (Time Profiler, Allocations) | Developer / TestFlight | Requires Xcode; not continuous in App Store builds |
+| Android | Android Studio Profiler (CPU, Memory) | Developer / debug build | Production CPU profiling requires `<profileable>` manifest flag |
+
+For production mobile observability, rely on crash analytics and RUM metrics to surface performance regressions. Cross-ref `../layers/L7-application/mobile-rum.md` for mobile RUM patterns and `../layers/L7-application/crash-analytics.md` for crash rate metrics.
+
+---
+
+## 9. Storage Backends
+
+| Backend | Primary use | Symbol resolution |
+|---------|------------|-------------------|
+| Parca server | OSS; local or k8s deployment; uses Parquet-backed storage | Upload ELF debug info / DWARF symbols; agent resolves at collection time |
+| Pyroscope OSS | Self-hosted; integrates with Grafana stack | Upload symbol files per build |
+| Grafana Cloud Profiles | Managed Pyroscope; SaaS | Upload symbols via Pyroscope API |
+| Polar Signals Cloud | Managed Parca; SaaS; automatic symbol upload | Symbols uploaded via CI step; no manual resolution needed |
+
+Symbol resolution is a critical operational concern. Without debug symbols (or source maps for Node.js), flamegraph frames appear as raw memory addresses or mangled names. Treat symbol upload as a required CI step alongside binary deployment; it has the same relationship to profiles as sourcemaps have to crash analytics.
+
+---
+
+## 10. Privacy and Security
+
+Stack traces expose function names, which may reveal proprietary algorithm structure or internal service architecture. This is especially sensitive for multi-tenant SaaS operators.
+
+- Apply access control to the profiling backend (Parca server, Pyroscope) equivalent to trace backend access control. Cross-ref `../signals/privacy.md §Backend RBAC`.
+- For multi-tenant deployments, isolate flamegraph access by tenant label. A tenant MUST NOT be able to query another tenant's flamegraph.
+- Function name redaction (stripping internal frames from exported profiles) is technically possible but rare and operationally complex. Consider it only if profile data is shared externally (e.g., vendor support handoff).
+
+---
+
+## 11. Matrix Cells: Profiles Column Summary
+
+Cross-ref: `../matrix.md` for full cell detail. Summary for the profiles signal column:
+
+| Layer × Boundary | profiles cell | Rationale |
+|-----------------|---------------|-----------|
+| L3 × all | N/A | IP routing has no process-level profiling artifact |
+| L4 × multi-tenant | PASS | eBPF CPU/off-CPU profiling covers socket-level overhead per process |
+| L4 × cross-application | PASS | eBPF off-CPU shows network-wait time between services |
+| L4 × slo | N/A | SLO is application-defined; L4 profiles are diagnostic, not SLO inputs |
+| L4 × release | N/A | Release events are L7 constructs |
+| mesh × all | PARTIAL | Proxy CPU visible via eBPF on sidecar process; not mesh-native |
+| L7 × multi-tenant | PASS | Per-tenant flamegraph via label-based isolation in Parca/Pyroscope |
+| L7 × cross-application | PASS | Application-level profiling with trace_id correlation |
+| L7 × slo | N/A | SLO is computed from metrics; profiles are diagnostic |
+| L7 × release | PARTIAL | Pre/post-release profile diff for performance regression detection |
+
+---
+
+## 12. Anti-Patterns
+
+These candidates belong in `../anti-patterns.md` and are flagged here for consolidation.
+
+| Anti-pattern | Risk | Mitigation |
+|-------------|------|-----------|
+| Continuous profiling enabled in production without overhead measurement in staging | Unexpected CPU spike > 1%; service degradation | Benchmark overhead in staging first; set alert on profiler CPU budget |
+| Symbol upload missing at deploy time | Flamegraphs show raw addresses or mangled names; engineers cannot act on data | Add symbol upload as a required CI step immediately after binary upload; gate deployment on symbol upload success |
+| `/debug/pprof` endpoint left enabled on production HTTP servers | Exposes CPU/heap profiles and internal function names to any caller who discovers the path; also a denial-of-service vector (profile requests are expensive) | Disable `net/http/pprof` import in production builds, or restrict to internal network behind authentication middleware |
+| Profile labels include high-cardinality dimensions (e.g., `request_id`, `user_id`) | Storage blowup; query performance degradation in Parca/Pyroscope backend | Label with low-cardinality dimensions only: `service.name`, `service.version`, `tenant.id` (for multi-tenant), `environment` |
+
+---
+
+## 13. References
+
+
+Internal cross-references:
+
+- `../standards.md §3`: OTel semconv stability tiers (experimental badge rationale)
+- `../matrix.md`: full 112-cell coverage matrix
+- `../layers/L4-transport.md`: eBPF profiling at socket level
+- `../layers/L7-application/mobile-rum.md`: mobile performance signals
+- `../layers/L7-application/crash-analytics.md`: crash rate and symbolication
+- `../boundaries/multi-tenant.md`: per-tenant label isolation
+- `../boundaries/release.md`: pre/post-release profile comparison
+- `../signals/privacy.md`: backend RBAC and access control
+- `../anti-patterns.md`: consolidated anti-pattern list
+
+## References
+
+- OTEP 0239 (profiling data model): <https://github.com/open-telemetry/oteps/blob/main/text/profiles/0239-profiles-data-model.md>
+- Parca: <https://parca.dev>
+- Grafana Pyroscope: <https://grafana.com/oss/pyroscope/>
+- Polar Signals Cloud: <https://polarsignals.com>
+- async-profiler: <https://github.com/jvm-profiling-tools/async-profiler>
+- py-spy: <https://github.com/benfred/py-spy>
+- CNCF Landscape (verify current CNCF status): <https://landscape.cncf.io>