oh-my-agent-commandcode 0.0.0-stage → 20.38.4

package/skills/oma-observability/resources/layers/L4-transport.md

@@ -0,0 +1,299 @@
+---
+otel_semconv: "1.43.0 (2026-07)"
+---
+
+# L4 Transport Layer Observability
+
+> **DISAMBIGUATION; read before proceeding.**
+>
+> This file covers **observing the OSI Layer 4 (TCP/UDP/QUIC) transport in your application systems**; 
+> retransmits, RTT, connection lifecycle, eBPF-based socket instrumentation, and QUIC/HTTP3 semantics.
+>
+> This file is **NOT** about the observability pipeline transport:
+> - `../transport/udp-statsd-mtu.md`; StatsD payload sizing over UDP
+> - `../transport/otlp-grpc-vs-http.md`; OTLP exporter protocol choice
+> - `../transport/collector-topology.md`; OTel Collector deployment topologies
+> - `../transport/sampling-recipes.md`; tail/head sampling strategies
+>
+> The word "transport" is overloaded. When a cross-reference says `transport/`, it refers to the
+> observability pipeline. When it says "L4", "TCP/UDP/QUIC", or this filename, it refers to this file.
+
+---
+
+## 1. Scope
+
+OSI Layer 4 observability for production systems: TCP, UDP, QUIC/HTTP3, eBPF auto-instrumentation,
+and connection lifecycle. Out of scope: L5 session (gRPC/WebSocket → L7), L6 full TLS inspection
+(use Wireshark/vendor tooling), security-focused eBPF (Cilium Tetragon/Falco → security skill).
+See `../standards.md §5` for the authoritative OSI boundary decision.
+
+---
+
+## 2. L4 Semconv (OTel)
+
+Source: <https://opentelemetry.io/docs/specs/semconv/attributes-registry/network/>
+
+| Attribute | Values | Stability | Notes |
+|-----------|--------|-----------|-------|
+| `network.transport` | `tcp`, `udp`, `unix`, `pipe`, `quic` | **Stable** | Set on spans and metrics describing a socket |
+| `network.protocol.name` | `http`, `grpc`, `amqp`, … | **Stable** | Application protocol over the transport |
+| `network.protocol.version` | `1.1`, `2`, `3` | **Stable** | `3` = HTTP/3 over QUIC |
+| `network.peer.address` | IP or hostname | **Stable** | Apply IP-masking before retention; `../signals/privacy.md §IP addresses` |
+| `network.connection.state` | `established`, `close_wait`, … | **Development** | Do not build SLOs against this; schema may change |
+| `network.connection.type` | `wifi`, `cell`, … | **Development** | Mobile/RUM only; flag in production use |
+
+`network.connection.*` attributes are Development tier (`../standards.md §3`). Use them in test
+environments and tolerate breaking changes before promoting to production dashboards.
+
+---
+
+## 3. TCP Observability
+
+### 3.1 Key Metrics
+
+| Metric | Type | Measures | Source |
+|--------|------|----------|--------|
+| `tcp_retransmits_total` | Counter | Retransmit segments; rate spike = congestion or loss | hostmetrics, eBPF |
+| `tcp_rtt_ms` (p50/p99) | Histogram | Round-trip time per connection | eBPF socket stats |
+| `tcp_active_connections` | Gauge | In-flight established connections; growth = pool leak | `/proc/net/sockstat` |
+| `tcp_time_wait_connections` | Gauge | TIME_WAIT count; high under short-lived connection churn | `/proc/net/sockstat` |
+| `tcp_syn_queue_drops_total` | Counter | SYN queue overflow; SYN flood or low `tcp_max_syn_backlog` | netlink / eBPF |
+| `tcp_close_wait_connections` | Gauge | CLOSE_WAIT growth = server not closing half-open connections | hostmetrics |
+
+```promql
+# Retransmit rate per node
+rate(tcp_retransmits_total[5m])
+
+# p99 RTT per service
+histogram_quantile(0.99, sum by (le, service_name) (rate(tcp_rtt_ms_bucket[5m])))
+
+# TIME_WAIT pressure alert
+tcp_time_wait_connections > 10000
+```
+
+### 3.2 Collection
+
+| Method | Mechanism | Privilege |
+|--------|-----------|-----------|
+| OTel hostmetrics receiver | `/proc/net/tcp`, `/proc/net/snmp`, `/proc/net/sockstat` | Root or CAP_SYS_PTRACE |
+| eBPF socket filter | kprobes on `tcp_retransmit_skb`, `tcp_sendmsg` | CAP_BPF (kernel ≥ 5.8) or root |
+| Linux netlink SOCK_DIAG | TCP socket state enumeration | Unprivileged in most distros |
+
+### 3.3 Common Pitfalls
+
+**Connection pool exhaustion.** High `tcp_active_connections` with high application latency but no TCP errors
+points to pool queue saturation, not network failure. Cross-ref: `../signals/traces.md §DB connection pool patterns`.
+
+**SYN cookie activation.** `tcp_syn_queue_drops_total` > 0 in normal traffic means `tcp_max_syn_backlog`
+is too low. SYN cookies mask drop counters in `/proc/net/snmp`; use eBPF for reliable counts.
+
+**TIME_WAIT accumulation.** Short-lived HTTP/1.1 connections exhaust ephemeral ports. Remediation:
+`SO_REUSEADDR`, wider `ip_local_port_range`, or HTTP/2 multiplexing.
+
+---
+
+## 4. UDP Observability
+
+| Metric | Source | Measures |
+|--------|--------|----------|
+| `udp_datagrams_sent_total` | `/proc/net/snmp` | Baseline send throughput |
+| `udp_receive_buffer_errors_total` | `/proc/net/snmp` `RcvbufErrors` | Socket buffer overflow; increase `net.core.rmem_max` |
+| `udp_no_ports_total` | `/proc/net/snmp` `NoPorts` | Drop because no listener on destination port |
+| `udp_checksum_errors_total` | `/proc/net/snmp` `InCsumErrors` | Corrupted datagrams; NIC or path corruption |
+
+OTel hostmetrics `network` scraper covers these counters automatically.
+Cross-ref: `../transport/udp-statsd-mtu.md` for StatsD payload sizing constraints that affect
+`udp_receive_buffer_errors_total` in high-throughput StatsD pipelines.
+
+---
+
+## 5. eBPF-Based L4 Observability
+
+eBPF instruments the Linux kernel without application code changes. Three tools are in scope.
+
+### 5.1 Tool Comparison
+
+| Tool | CNCF Status | Primary signal | OTel output | Key use case |
+|------|-------------|----------------|-------------|--------------|
+| **Grafana Beyla** | CNCF Incubating (2024) | HTTP/gRPC traces + Go/C++ runtime | OTLP natively | Zero-code HTTP/gRPC auto-instrumentation via uprobes; also emits `tcp_rtt_us` histograms. Requires kernel ≥ 5.2 with BTF enabled |
+| **Pixie** | CNCF Sandbox | Multi-protocol traces + L4 metrics via PxL | OTLP via `px-export` | Scriptable cluster-wide eBPF; auto-discovers pods |
+| **Parca** | CNCF Sandbox | Continuous CPU + off-CPU flame graphs | pprof-compatible | Whole-cluster profiling; cross-ref `../signals/profiles.md` |
+
+Sources: <https://grafana.com/oss/beyla/>, <https://px.dev/>, <https://parca.dev/>
+
+**Out of scope: Cilium Tetragon.** Security-focused runtime policy enforcement (syscall filtering, process
+exec tracing) belongs to a security skill. Falco is the comparable alternative for kernel security events.
+
+### 5.2 Kernel and Privilege Requirements
+
+| Requirement | Detail |
+|-------------|--------|
+| Minimum kernel | ≥ 4.14 for most eBPF features; ≥ 5.8 for CAP_BPF without root |
+| Capabilities | CAP_BPF + CAP_PERFMON (kernel ≥ 5.8) or root on older kernels |
+| GKE Autopilot | eBPF DaemonSets disallowed; use Standard node pools |
+| EKS Fargate | No DaemonSet support; eBPF unavailable |
+| AKS | Supported on standard node pools; verify with `uname -r` |
+
+```bash
+# Preflight kernel check (run in initContainer or node admission)
+uname -r           # must be >= 4.14, ideally >= 5.8
+capsh --print | grep cap_bpf
+
+# Beyla deploy via Helm
+helm repo add grafana https://grafana.github.io/helm-charts
+helm install beyla grafana/beyla --namespace beyla --create-namespace
+```
+
+### 5.3 Pixie PxL (Conceptual)
+
+```python
+# Conceptual PxL — TCP retransmit counts per pod
+import px
+df = px.DataFrame(table='tcp_retransmits', start_time='-5m')
+df = df.groupby(['pod', 'namespace']).agg(retransmits=('retransmits', px.sum))
+px.display(df)
+```
+
+---
+
+## 6. Connection Lifecycle Observability
+
+```
+LISTEN → SYN_RECEIVED → ESTABLISHED → (data transfer)
+  → FIN_WAIT_1 → FIN_WAIT_2 → TIME_WAIT → CLOSED   (active close)
+  → CLOSE_WAIT → LAST_ACK → CLOSED                  (passive close)
+```
+
+| State | Observable signal | Tooling |
+|-------|------------------|---------|
+| SYN → ESTABLISHED | Setup latency (SYN-ACK RTT) | eBPF `tcp_v4_connect` kprobe |
+| ESTABLISHED idle | Keepalive probe success/failure | `/proc/net/tcp` + hostmetrics |
+| FIN_WAIT / TIME_WAIT | TIME_WAIT gauge; ephemeral port pressure | hostmetrics `/proc/net/sockstat` |
+| CLOSE_WAIT growth | Server not releasing half-open connections | hostmetrics + eBPF alert |
+
+**TLS handshake boundary.** TLS sits at L4/L5. `tls.*` semconv is Development tier; do not build
+production SLOs on it. Use eBPF uprobes on OpenSSL/BoringSSL or vendor tooling for handshake latency.
+Cross-ref: `../signals/privacy.md §TLS context`.
+
+**Keepalive tuning.** TCP keepalive interval must be shorter than the upstream LB idle timeout
+(AWS ALB: 60 s; AWS NLB: 350 s; GCP CLB: 600 s default). Mismatches cause silent connection resets.
+
+**WebSocket and gRPC streams**; full lifecycle coverage is deferred to L7 (`../standards.md §5`).
+Long-lived streams mitigate TIME_WAIT accumulation but introduce reconnection-storm risk on restart.
+
+---
+
+## 7. QUIC / HTTP/3 Transport Semantics
+
+QUIC (RFC 9000) is a UDP-based transport with built-in TLS 1.3, stream multiplexing, and connection
+migration. HTTP/3 (RFC 9114) runs exclusively over QUIC.
+
+### 7.1 Observability Differences from TCP
+
+| TCP concept | QUIC equivalent | Observability impact |
+|-------------|----------------|----------------------|
+| 3-way SYN handshake | 1-RTT or 0-RTT | No SYN/FIN to observe; connection state is QUIC-internal |
+| 5-tuple for correlation | Connection ID (opaque bytes) | Connection ID survives NAT rebinding; log it on establishment |
+| `/proc/net/tcp` retransmit | QUIC loss recovery (packet number gaps) | Not visible in proc; requires QUIC-aware tooling |
+| TCP FIN / RST | QUIC GOAWAY / CONNECTION_CLOSE frames | Surfaced in Envoy access logs |
+| HTTP/2 HOL blocking | Per-stream flow control; no HOL blocking | Stream errors do not block other streams |
+
+### 7.2 OTel Semconv for QUIC
+
+```
+network.transport: quic
+network.protocol.name: http
+network.protocol.version: "3"
+```
+
+### 7.3 0-RTT Observability
+
+0-RTT sends data before the handshake completes. Emit a span attribute or metric label
+distinguishing 0-RTT from 1-RTT connections to detect replay exposure on non-idempotent endpoints.
+
+### 7.4 Firewall and Enterprise Deployment Considerations
+
+Enterprise networks frequently block outbound UDP 443 and UDP 8443 because legacy firewall rules assume HTTP(S) is TCP-only. QUIC (and therefore HTTP/3) rides on UDP and silently falls back to HTTP/1.1 or HTTP/2 when UDP is blocked; the fallback is invisible to application telemetry unless you instrument it.
+
+Detect QUIC blocking:
+- Emit `network.transport` attribute on client spans; compare ratio of `quic` vs `tcp` across deployment zones
+- CDN access logs (Cloudflare, Fastly) expose QUIC negotiation outcome per edge
+- Alert when `quic` ratio drops below baseline (for example, office network where UDP 443 is firewalled)
+
+Action: coordinate with network security to allowlist UDP 443/8443 for validated destinations; document the fallback behavior so regressions do not silently halve transport performance.
+
+### 7.5 Tool Support (2026)
+
+| Tool | QUIC support |
+|------|-------------|
+| Envoy 1.29+ | QUIC upstream and downstream; stats in admin API and access logs |
+| Istio 1.22+ | QUIC upstream via Envoy; mTLS over QUIC experimental; see `layers/mesh.md` |
+| Grafana Beyla | HTTP/3 uprobes in roadmap; verify release notes before QUIC-heavy deployments |
+| Cloudflare QUIC telemetry | <https://blog.cloudflare.com/tag/quic/>; public connection migration + 0-RTT metrics |
+| Google QUIC metrics | <https://research.google>; packet loss recovery timing |
+
+---
+
+## 8. OS-Level Integration
+
+| Source | Signal | Cross-ref |
+|--------|--------|-----------|
+| systemd-journal / syslog | TCP resets, netfilter drops, NIC errors | `../signals/logs.md §OS-level log sources` |
+| cgroup v2 net accounting | Per-container byte/packet counters | `../transport/collector-topology.md §kubernetes` |
+| nf_conntrack | NAT and stateful firewall connection table size | Requires `nf_conntrack` kernel module |
+
+---
+
+## 9. Matrix Cells: L4 Row
+
+From `../matrix.md §L4-transport`:
+
+| Cell | Symbol | Rationale |
+|------|--------|-----------|
+| L4 × cross-application × traces | PARTIAL | TCP is not trace-native; log socket 5-tuple for post-hoc correlation to L7 spans only |
+| L4 × * × profiles | PASS | eBPF off-CPU profiling via Parca captures kernel socket overhead and network-wait; pprof output |
+| L4 × privacy (all boundaries) | PARTIAL | Source/destination IPs in TCP logs are PII (GDPR Art. 4(1)); apply masking at pipeline ingress |
+| L4 × SLO and L4 × release | N/A | SLO budgets and release markers are application-layer; TCP health is infra-SLI fallback only |
+
+---
+
+## 10. Anti-Patterns: Candidates for `../anti-patterns.md §Section H`
+
+**H-L4-1: Connection pool observability absent.** Pool queue saturation is invisible in TCP metrics
+alone; neither retransmit rate nor error rate spikes until the pool times out. Instrument pool size,
+wait time, and timeout counters at the application layer. Severity: HIGH.
+
+**H-L4-2: QUIC adoption without HTTP/3 trace tooling validation.** Enabling QUIC without verifying
+that OTel SDKs emit `network.transport: quic` and `network.protocol.version: "3"` creates a
+transport-layer blind spot. Add a canary assertion in staging before production rollout. Severity: HIGH.
+
+**H-L4-3: eBPF agent deployed without kernel/capability preflight.** Beyla or Pixie DaemonSets on
+incompatible kernels fail silently with no operator-visible error. Add an `initContainer` asserting
+kernel version and CAP_BPF presence; fail non-zero if requirements are unmet. Severity: HIGH.
+
+**H-L4-4: WebSocket reconnection storm undetected.** Server restart without graceful WebSocket drain
+causes simultaneous mass-reconnect. TCP SYN counts look like normal new connections. Instrument client
+reconnection counters and alert on rate spikes; implement exponential backoff + jitter. Severity: MEDIUM.
+
+---
+
+## 11. References
+
+- OTel network attributes registry: <https://opentelemetry.io/docs/specs/semconv/attributes-registry/network/>
+- Grafana Beyla: <https://grafana.com/oss/beyla/>
+- Pixie (CNCF Sandbox): <https://px.dev/>
+- Parca (CNCF Sandbox): <https://parca.dev/>
+- QUIC RFC 9000: <https://www.rfc-editor.org/rfc/rfc9000>
+- HTTP/3 RFC 9114: <https://www.rfc-editor.org/rfc/rfc9114>
+- Cloudflare QUIC telemetry: <https://blog.cloudflare.com/tag/quic/>
+- OTel hostmetrics receiver: <https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver>
+- Linux `/proc/net/tcp` format: <https://www.kernel.org/doc/html/latest/networking/proc_net_tcp.html>
+- Semconv stability tiers: `../standards.md §3`
+- OSI boundary decisions: `../standards.md §5`
+- Coverage matrix (L4 row): `../matrix.md §L4-transport`
+- IP masking rules: `../signals/privacy.md §IP addresses`
+- Profiling (eBPF/Parca): `../signals/profiles.md`
+- OS-level logs: `../signals/logs.md §OS-level log sources`
+- Collector Kubernetes topology: `../transport/collector-topology.md`
+- StatsD UDP sizing: `../transport/udp-statsd-mtu.md`
+- Service mesh + QUIC: `layers/mesh.md`

package/skills/oma-observability/resources/layers/L7-application/crash-analytics.md

@@ -0,0 +1,359 @@
+---
+otel_semconv: "1.43.0 (2026-07)"
+---
+
+# Crash Analytics: L7 Application Layer
+
+## 1. Scope and Why Separate from RUM
+
+Crash analytics is not RUM. The distinction matters for pipeline design, KPI selection, and vendor choice.
+
+| Dimension | RUM | Crash Analytics |
+|-----------|-----|----------------|
+| Session type | Successful sessions (UX measured) | Abnormal terminations (failed sessions) |
+| Primary KPI | Core Web Vitals / INP / startup time | Crash-Free Rate (CFR), ANR Rate |
+| Pipeline | Beacon / SDK event stream → backend | Crash report queued on device → uploaded on restart |
+| Symbolication | N/A (JS source maps for errors only) | dSYM / ProGuard / R8 / Dart symbols required |
+| Offline behavior | Events dropped if offline | Report queued locally; uploaded on next launch |
+
+This file documents the common crash pipeline shared across mobile native, web, and backend uncaught exceptions. Platform-specific symbolication flows are cross-referenced:
+
+- Mobile native detail: `mobile-rum.md §Crash + ANR`
+- Web JS error detail: `web-rum.md §Error Boundary and window.onerror`
+
+Platforms in scope: iOS native, Android native, React Native, Flutter, web (JS), backend uncaught exceptions.
+
+---
+
+## 2. Core Metrics
+
+### Crash-Free Rate (CFR)
+
+CFR is the primary stability KPI for mobile and web applications.
+
+```
+Session CFR = (sessions without crash) / (total sessions)
+User CFR    = (users not experiencing any crash) / (total active users)
+```
+
+Targets by application category:
+
+| Category | Minimum Session CFR | Notes |
+|----------|--------------------|----|
+| Consumer mobile | 99.5% | Google Play bad-behavior threshold: < 99.0% triggers review |
+| Enterprise / B2B mobile | 99.7% | Internal SLA baseline |
+| Regulated (fintech, health) | 99.9%+ | Near-zero tolerance; compliance-linked |
+| Web SPA | 99.8%+ | JS errors below `Error Boundary` counted as crashes |
+
+### ANR Rate and Hang Rate
+
+Near-crash experiences that degrade user perception without a fatal signal:
+
+- **Google Play user-perceived ANR rate** = daily active users experiencing at least one user-perceived ANR / daily active users. The overall bad-behavior threshold is 0.47%; it is not an ANR-events/session threshold. Input dispatch typically times out at 5 s. Broadcast receiver timeouts are 10 s (foreground) / 60 s (background) on Android 13 and lower, and 10–20 s / 60–120 s on Android 14+, depending on CPU starvation. Keep any custom session-based stall metric separate. Sources: <https://developer.android.com/topic/performance/anrs/diagnose-and-fix-anrs> and <https://developer.android.com/google/play/vitals>.
+- **iOS Hang Rate** = main-thread hangs > 250 ms / sessions. Visible in Xcode Organizer → Hangs and via MetricKit `MXHangDiagnostic` (iOS 14+).
+
+### Supporting Metrics
+
+| Metric | Definition |
+|--------|-----------|
+| Affected Users % | `(users who hit crash X) / (DAU)`; scope of a specific crash group |
+| Time-to-Symbolication | Duration from crash report ingestion to readable stack trace |
+| Crash Volume | Raw crash event count; use with CFR, not instead of it |
+
+---
+
+## 3. Platform-Specific Symbolication Pipelines
+
+Each platform requires a distinct artifact upload strategy. Failure to automate upload results in unreadable stack traces in production.
+
+| Platform | Symbol artifact | Upload trigger | Key complexity |
+|----------|----------------|---------------|----------------|
+| iOS | dSYM bundle (per build UUID) | Xcode archive → CI upload | Bitcode deprecated in Xcode 14; no re-symbolication |
+| Android (JVM) | `mapping.txt` (ProGuard / R8) | Gradle task post-minify | Must match exact build; obfuscation mapping is one-way |
+| Android (NDK) | `.so` files with debug info stripped separately | CI symbol upload | Requires `--build-id` linkage; `.so` from APK lacks debug info |
+| React Native | JS sourcemap + iOS dSYM + Android mapping.txt | Three uploads per release | Hermes bytecode adds an extra sourcemap layer |
+| Flutter | Dart symbol files from `--split-debug-info` | CI upload after `flutter build` | Both iOS dSYM and Android mapping still needed for platform channels |
+| Web | JS sourcemap | CI deploy pipeline | `window.onerror` + `onunhandledrejection`; React/Vue Error Boundary |
+| Backend | N/A (source available) | OTel `exception.*` attributes | Sentry Python/Node/Ruby SDK for crash-like grouping |
+
+### iOS
+
+dSYM (Debug Symbols) is the binary artifact that maps obfuscated addresses to function names and line numbers. Each Xcode build generates a UUID-stamped dSYM. The UUID must match the crash report exactly; re-using a dSYM from a different build produces wrong or partial symbolication.
+
+Bitcode was deprecated in Xcode 14. Prior to Xcode 14, Apple re-compiled bitcode for App Store distribution, requiring vendors (Firebase, Sentry, Datadog) to download the re-compiled dSYM from App Store Connect via the API. Xcode 14+ removes this complexity; the Xcode-generated dSYM is final.
+
+Apple App Store Connect crash API: Firebase Crashlytics, Sentry, and Datadog can pull crash data from App Store Connect for devices where the user has opted into analytics sharing.
+
+### Android
+
+ProGuard / R8 generates `mapping.txt` which maps obfuscated class/method names back to source. The NDK layer produces native `.so` libraries; the release variant strips debug info into a separate symbol file. Upload both `mapping.txt` and NDK symbol `.so` files for full stack traces covering Java/Kotlin and C++ frames.
+
+Play Console crashes API: available for complementary data, not a replacement for vendor SDK.
+
+### React Native
+
+Three separate symbol artifacts per release:
+
+1. JS bundle sourcemap (Metro bundler output)
+2. iOS dSYM (for native frames)
+3. Android `mapping.txt` (for native frames)
+
+Hermes JS engine compiles JS to bytecode. The sourcemap chain is: Hermes bytecode address → JS sourcemap → original TS/JS source. Some vendors (Sentry, Datadog) have Hermes-aware symbolication; verify vendor documentation before assuming standard sourcemap upload suffices.
+
+### Flutter
+
+`flutter build --split-debug-info=<dir> --obfuscate` produces Dart symbol files. These must be uploaded to the crash vendor's symbol storage. Platform-channel crashes (e.g., method channel calls into Kotlin/Swift) still require iOS dSYM and Android `mapping.txt` in addition to Dart symbols.
+
+### Web
+
+Cross-reference `web-rum.md §Error tracking`. Key patterns:
+- `window.onerror` catches uncaught synchronous errors
+- `window.addEventListener('unhandledrejection', ...)` catches unhandled Promise rejections
+- React `ErrorBoundary`, Vue `app.config.errorHandler`, Svelte `<svelte:component this={ErrorBoundary}>` prevent full-page crashes and capture component-tree context
+
+### Backend
+
+Backend services rarely produce binary crash reports. OTel `exception.*` attributes (Stable semconv) capture the exception type, message, and stacktrace on spans. Sentry Python / Node.js / Ruby SDKs provide crash-like aggregation with grouping, release tracking, and breadcrumbs that mirror the mobile crash experience.
+
+---
+
+## 4. Release Tracking Integration
+
+Release tracking is the most commonly omitted step in crash pipelines. Without it, a crash spike cannot be correlated to a specific deploy.
+
+**CI must set `service.version` on every build.** The OTel semconv `service.version` attribute (Stable) is the canonical release identifier. All crash events, spans, and metrics must carry this attribute.
+
+**Symbol upload must be automated per release.** Manual upload is an anti-pattern (see Section 12). Use:
+
+- Sentry CLI: `sentry-cli releases propose-version` + `sentry-cli releases files upload-sourcemaps`
+- Fastlane plugin: `fastlane-plugin-sentry` for iOS/Android
+- Gradle plugin: `io.sentry.android.gradle` for Android automatic upload
+- Firebase: `firebase crashlytics:symbols:upload` for dSYM (Fastlane or Gradle)
+
+**Release marker events** are structured log records written at deploy time (see `../../boundaries/release.md`). Crash vendors use these to draw vertical lines on CFR trend charts, enabling before/after comparison. Flagger and Argo Rollouts canary analysis can use CFR as a promotion/rollback gate signal; cross-reference `../../boundaries/release.md §Canary analysis`.
+
+---
+
+## 5. Vendor Comparison Matrix (as of 2026-Q2)
+
+Cross-reference `../../vendor-categories.md §Category (j): Crash Analytics`.
+
+| Vendor | iOS | Android | Web JS | Backend | OSS/Free | Key differentiator |
+|--------|-----|---------|--------|---------|----------|--------------------|
+| Firebase Crashlytics | Yes | Yes | No | No | Free | Google-owned; zero cost; mobile-only; dSYM auto-upload via Xcode plugin |
+| Sentry | Yes | Yes | Yes | Yes | OSS + SaaS | Full-stack (mobile + web + backend); release tracking; OTel trace correlation |
+| Bugsnag | Yes | Yes | Yes | Yes | Commercial | Stability score; smart grouping noise reduction; breadcrumb API |
+| Embrace | Yes | Yes | No | No | Commercial | ANR/startup crash specialist; session replay; network body capture |
+| Datadog Error Tracking | Yes | Yes | Yes | Yes | Commercial | Native RUM + APM integration; unified with Datadog traces |
+
+**Key differentiators:**
+
+- **Firebase Crashlytics**: best choice when budget is zero and stack is mobile-only. No web or backend support. Google ecosystem (Analytics, BigQuery export).
+- **Sentry**: strongest cross-platform story. The only vendor in the matrix with meaningful OSS options. Release health dashboard is the most mature. OTel `trace_id` in crash context enables backend span lookup.
+- **Bugsnag**: stability metrics and smart grouping reduce alert fatigue. Breadcrumb API is the most flexible for custom event recording.
+- **Embrace**: purpose-built for mobile ANR, startup crash, and user-journey session replay. Not a general-purpose crash tool.
+- **Datadog Error Tracking**: optimal when Datadog APM is already in use. RUM + APM linkage is native; no separate SDK integration.
+
+---
+
+## 6. Crash Pipeline Flow
+
+```
+App crashes → crash report generated
+  (stack trace + breadcrumbs + device context + user context)
+      |
+      v
+  Queued locally on device (if offline — see mobile-rum.md §offline queuing)
+      |
+      v
+  Uploaded to vendor on next launch or when network available
+      |
+      v
+  Vendor-side symbolication
+  (dSYM / ProGuard / sourcemap matched by build UUID / version)
+      |
+      v
+  Deduplication by stack fingerprint
+  (vendor groups similar crashes into one issue)
+      |
+      v
+  Aggregation: Affected Users %, Event Count, CFR delta
+      |
+      v
+  Alert + ticket creation
+  (PagerDuty / OpsGenie for CFR breach; JIRA / Linear for issue triage)
+```
+
+---
+
+## 7. Breadcrumbs and Context
+
+Breadcrumbs are the sequence of events recorded before the crash. They are the primary debugging context beyond the stack trace.
+
+| Category | Examples |
+|----------|---------|
+| User actions | Button taps, navigation events, form submissions |
+| Network calls | HTTP request URL, status code, duration |
+| Log events | Application log statements recorded by SDK |
+| State changes | Authentication events, feature flag changes |
+
+Context attached to every crash report:
+
+| Field | Source | Notes |
+|-------|--------|-------|
+| `device.model` | OS API | Device model identifier |
+| `os.version` | OS API | OS version string |
+| `service.version` | Build config | Release version; must match symbol upload |
+| `screen_resolution` | OS API | Pixel dimensions |
+| `free_memory_mb` | OS API | Available RAM at crash time |
+| `user.id` | App session | Hash or opaque ID only; see Section 8 |
+
+Custom keys (e.g., `order.id`, `tenant.id`, `experiment.variant`) enable correlation with business context. Restrict to non-PII values or apply redaction before send.
+
+---
+
+## 8. Privacy and PII in Crash Reports
+
+Stack traces frequently contain PII because exception messages capture method arguments, query strings, and HTTP headers verbatim.
+
+**Common PII leaks in crash reports:**
+
+- SQL query in exception message containing `user.email` or SSN
+- HTTP response body logged in a network error containing `Authorization: Bearer <token>`
+- URL query parameters with `?email=user@example.com`
+- Custom key set with `user.email` directly
+
+**Vendor SDK redaction hooks:**
+
+Sentry `beforeSend` callback (JavaScript/TypeScript example):
+
+```typescript
+Sentry.init({
+  dsn: "...",
+  beforeSend(event) {
+    // Strip email from exception values
+    if (event.exception?.values) {
+      event.exception.values = event.exception.values.map((ex) => ({
+        ...ex,
+        value: ex.value?.replace(/[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}/g, "[email]"),
+      }));
+    }
+    // Remove auth headers from request context
+    if (event.request?.headers) {
+      delete event.request.headers["Authorization"];
+      delete event.request.headers["Cookie"];
+    }
+    return event;
+  },
+});
+```
+
+- **Firebase Crashlytics**: use `setCustomKey` with an explicit allowlist; never pass user.email or tokens as custom keys.
+- **Datadog**: `beforeSend` callback mirrors Sentry's pattern; apply to both RUM and Error Tracking initializations.
+
+Cross-reference: `../../signals/privacy.md §Crash report redaction` and `../../anti-patterns.md §Section A Privacy`.
+
+---
+
+## 9. CI Integration for Symbol Upload
+
+Symbols must never be stored in git. Upload to the vendor on every CI build that produces a release artifact.
+
+Conceptual GitHub Actions workflow (adapt per vendor):
+
+```yaml
+# .github/workflows/release.yml (conceptual — adapt version and paths)
+- name: Build iOS release
+  run: xcodebuild archive -scheme MyApp -archivePath build/MyApp.xcarchive
+
+- name: Upload dSYM to Sentry
+  env:
+    SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}
+    SENTRY_ORG: my-org
+    SENTRY_PROJECT: my-app-ios
+  run: |
+    sentry-cli releases propose-version
+    sentry-cli upload-dif build/MyApp.xcarchive/dSYMs \
+      --include-sources
+
+- name: Tag release with service.version
+  run: |
+    sentry-cli releases new "${GITHUB_REF_NAME}"
+    sentry-cli releases set-commits "${GITHUB_REF_NAME}" --auto
+    sentry-cli releases finalize "${GITHUB_REF_NAME}"
+```
+
+For Android, replace the dSYM upload step with a Gradle task invocation (`./gradlew :app:uploadSentryProguardMappings`) or `sentry-cli upload-proguard --android`.
+
+Cross-reference `../../observability-as-code.md` for observability-as-code patterns and repository structure for symbol upload configuration.
+
+---
+
+## 10. ANR and Hang Detection
+
+ANR and hangs are near-crash events that degrade UX without generating a fatal crash report. They require distinct detection and alerting strategies.
+
+### Android ANR
+
+- Definition: input-dispatch ANRs typically use a 5 s timeout; broadcast receiver timeouts depend on foreground/background priority and OS version (see §2). A 200 ms stall is not the broadcast ANR threshold.
+- Detection sources:
+  - `ApplicationExitInfo` API (Android 11+): query ANR reasons from the OS after the fact.
+  - `ANRWatchDog` library: third-party watchdog thread that detects main-thread stalls in-process.
+  - Sentry, Embrace, Datadog: vendor SDK ANR detection via watchdog thread.
+- Alert threshold: compare the Google Play user-perceived ANR rate against 0.47% of daily active users. Do not apply that threshold to a custom ANR-events/session series.
+
+### iOS Hang
+
+- Definition: main thread blocked for more than 250 ms (Xcode Organizer threshold).
+- Detection sources:
+  - Xcode Organizer → Hangs report (aggregated from devices with diagnostic sharing enabled).
+  - Sentry App Hang Detection SDK: configurable threshold, default 2 seconds; reports hangs as separate issue type.
+  - MetricKit: `MXHangDiagnostic` (iOS 14+) for on-device hang diagnostics.
+- Alert threshold: no Google Play equivalent; apply product-specific target (e.g., < 0.1% of sessions for consumer apps).
+
+---
+
+## 11. Matrix Cross-References (L7 Row)
+
+These cells from `../../matrix.md` are directly informed by this file:
+
+| Matrix cell | Artifact | Note |
+|-------------|---------|------|
+| L7 × multi-tenant × logs | Per-tenant crash rate segmentation | `tenant.id` as custom key in crash context |
+| L7 × release × traces | Release-tagged crash correlates canary rollback | `service.version` on crash event + span `trace_id` linkage |
+| L7 × privacy × logs | Crash report redaction before send | `beforeSend` hook; allowlist custom keys |
+| L7 × cross-application × traces | Backend `trace_id` in crash context | Sentry / Datadog: attach active `trace_id` to crash report for backend span lookup |
+
+---
+
+## 12. Anti-Patterns
+
+Candidates for `../../anti-patterns.md §Section G — Crash Analytics`:
+
+| Anti-pattern | Impact | Remediation |
+|-------------|--------|------------|
+| Crash report contains `user.email`, auth tokens, or card numbers without redaction filter | GDPR / PIPA breach; PII in vendor SaaS storage | Implement `beforeSend` allowlist; cross-ref Section 8 |
+| Symbol upload not automated in CI | Stack traces unreadable in production | Add symbol upload step to release pipeline; cross-ref Section 9 |
+| No release marker → cannot correlate deploy to crash spike | Crash spike investigation requires manual git bisect | Set `service.version` on every build; emit release event at deploy |
+| ANR rate unmonitored on Android | Google Play app suppression; degraded store ranking | Monitor Play user-perceived ANRs per daily active user at the 0.47% overall threshold; define custom session metrics separately |
+| Single aggregate CFR target ignoring device-tier or OS-version distribution | p10 device users (low-end hardware) masked by p90 average | Segment CFR by `device.model`, `os.version`, and network type |
+| dSYM stored in git LFS instead of vendor symbol storage | Git LFS cost; symbol-build UUID drift | Use vendor upload; never commit dSYMs |
+
+---
+
+## References
+
+
+
+
+### Primary sources
+
+- Firebase Crashlytics: <https://firebase.google.com/docs/crashlytics>
+- Sentry iOS: <https://docs.sentry.io/platforms/apple/>
+- Sentry Android: <https://docs.sentry.io/platforms/android/>
+- Sentry React Native: <https://docs.sentry.io/platforms/react-native/>
+- Sentry Flutter: <https://docs.sentry.io/platforms/flutter/>
+- Xcode Organizer Hangs: <https://developer.apple.com/documentation/xcode/understanding-user-interface-responsiveness>
+- Android ANR: <https://developer.android.com/topic/performance/vitals/anr>
+- Android ApplicationExitInfo: <https://developer.android.com/reference/android/app/ApplicationExitInfo>
+- OTel `exception.*` semconv (Stable): <https://opentelemetry.io/docs/specs/semconv/exceptions/>