oh-my-agent-commandcode 0.0.0-stage → 20.38.3

package/skills/oma-observability/resources/signals/metrics.md

@@ -0,0 +1,478 @@
+---
+otel_spec: "1.x (stable API/SDK)"
+otel_semconv: "1.43.0 (2026-07)"
+notes:
+  - "OpenMetrics: project specification (see primary source below)"
+---
+
+# Metrics Signal
+
+## 1. Scope
+
+Metrics are the "M" in MELT+P (Metrics, Events, Logs, Traces + Profiles).
+
+This file covers:
+- OTel metric instrument types and their sync/async variants
+- Prometheus exposition format and the OpenMetrics project specification
+- SLI computation patterns (Golden Signals, RED, USE) and healthcheck integration
+- `hostmetrics` receiver for host-level collection
+- Kubernetes metric sources
+- Application metric integration via OTel SDK
+- OpenCost metric surface for cost attribution
+- Cardinality budget enforcement
+- Long-term storage backend options
+
+Out of scope:
+- Full SLO framework, error budget policy, and burn-rate alert rules: see `../boundaries/slo.md`
+- Full FinOps unit economics and cost allocation model: see `cost.md`
+- Cardinality guardrail detail and pipeline remediation: see `../meta-observability.md §Section C`
+
+---
+
+## 2. OTel Metric Instrument Types (Stable)
+
+Source: <https://opentelemetry.io/docs/specs/otel/metrics/api/>
+
+Choose an OTel Meter instrument based on measurement semantics. Counter, UpDownCounter, Gauge, and Histogram have the variants below. Summary is a legacy metric data type for interoperability, not a Meter API instrument.
+
+### 2.1 Counter
+
+Monotonic, cumulative-only. Value never decreases. Measures cumulative totals.
+
+```
+Use for: requests_total, errors_total, bytes_sent_total
+Never use for: values that can decrease (use UpDownCounter)
+```
+
+Sync variant: `counter.Add(ctx, delta, attrs...)`; caller increments explicitly.
+Async variant: `ObservableCounter`; callback provides absolute cumulative value.
+
+### 2.2 UpDownCounter
+
+Monotonic: no. Can increment or decrement. Measures quantities that fluctuate.
+
+```
+Use for: queue_length, active_connections, in_flight_requests
+Never use for: strictly increasing values (use Counter)
+```
+
+Sync variant: `upDownCounter.Add(ctx, delta, attrs...)` where delta is positive or negative.
+Async variant: `ObservableUpDownCounter`; callback provides current value.
+
+### 2.3 Gauge
+
+Point-in-time observation of a current value. No accumulation.
+
+```
+Use for: cpu_usage, memory_used_bytes, temperature_celsius, cache_hit_ratio
+```
+
+Sync variant: `gauge.Record(ctx, value, attrs...)`.
+Async variant: `ObservableGauge`; preferred for values read from an external source (OS, hardware sensor).
+
+### 2.4 Histogram
+
+Distribution of values. Captures bucket counts, sum, and count. Supports quantile computation via `histogram_quantile` at query time.
+
+```
+Use for: request_duration_seconds, response_size_bytes, db_query_duration_seconds
+Prefer over Summary for all new instrumentation
+```
+
+Sync variant only: `histogram.Record(ctx, value, attrs...)`.
+Configure explicit bucket boundaries per instrument to control cardinality. Default OTel SDK buckets: `[0, 5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000]` (milliseconds; override for seconds-based metrics).
+
+### 2.5 Summary (legacy interoperability data, not a Meter instrument)
+
+Imported Prometheus/OpenCensus-style summaries contain client-computed quantiles. The OTel metric data model can carry legacy Summary points; the Meter API does not create a Summary instrument.
+
+Limitations:
+- Quantiles are computed per-process; they cannot be aggregated across replicas.
+- `p99` from three replicas cannot be merged into a fleet-level `p99`.
+- Use Histogram + `histogram_quantile` instead for any distributed system.
+
+When to keep existing Summary metrics: only if the downstream consumer cannot be migrated and the measurement is single-process.
+
+### 2.6 Temporality: Delta vs Cumulative
+
+The OTLP metric exporter defaults to `cumulative`. Delta is a configurable preference for supported instrument kinds; verify the exporter and language implementation.
+Prometheus counters expose cumulative values; gauges can increase or decrease. Choose conversion according to the instrument and backend, not a blanket monotonic rule for every metric.
+
+Set the preference via environment variable:
+
+```
+OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative
+```
+
+For delta OTLP input destined for a cumulative backend, use the Collector delta-to-cumulative processor supported by the pinned distribution/version (`delta_to_cumulative`, formerly `deltatocumulative`). `prometheusreceiver` scrapes Prometheus input; it is not a delta OTLP conversion processor. Sources: <https://opentelemetry.io/docs/specs/otel/metrics/sdk_exporters/otlp/> and <https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/deltatocumulativeprocessor>.
+
+---
+
+## 3. Prometheus Exposition Format
+
+Source: <https://prometheus.io/docs/instrumenting/exposition_formats/>
+OpenMetrics: <https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md>
+
+### 3.1 Text Format Structure
+
+```
+# HELP http_requests_total Total HTTP requests received
+# TYPE http_requests_total counter
+http_requests_total{method="GET",status="200",route="/api/v1/users"} 1234 1714521600000
+http_requests_total{method="POST",status="422",route="/api/v1/users"} 7 1714521600000
+```
+
+Rules:
+- `# HELP` line: one per metric family, human-readable description.
+- `# TYPE` line: one per metric family, declares instrument type.
+- Metric line: `name{label="value",...} value [timestamp_ms]`. Timestamp is optional.
+- Blank line separates metric families.
+
+### 3.2 Naming Convention
+
+Pattern: `{component}_{operation}_{unit}_{suffix}`
+
+| Suffix | When to use | Example |
+|--------|-------------|---------|
+| `_total` | Counters (monotonic) | `http_requests_total` |
+| `_seconds` | Duration (always seconds, not ms) | `http_request_duration_seconds` |
+| `_bytes` | Size in bytes | `response_size_bytes` |
+| `_ratio` | Fraction 0.0–1.0 | `cache_hit_ratio` |
+| `_bucket` | Histogram bucket (auto-appended) | `http_request_duration_seconds_bucket` |
+| `_count` | Histogram observation count (auto) | `http_request_duration_seconds_count` |
+| `_sum` | Histogram observation sum (auto) | `http_request_duration_seconds_sum` |
+
+### 3.3 Label Rules
+
+Low-cardinality labels only. Cross-ref `../meta-observability.md §Section C Cardinality Guardrails`.
+
+Safe label examples: `method`, `status_code`, `route` (normalized), `service`, `region`, `env`.
+Forbidden label examples: `user.id`, `request.id`, `trace.id`, `url.full` (raw), `error.message`.
+
+### 3.4 OpenMetrics Snippet
+
+OpenMetrics is a project specification for a metric exposition format. Key difference: `# EOF` terminator is required, and `_total` suffix is mandatory for counters in the `# TYPE counter` family.
+
+```
+# HELP http_requests Total HTTP requests received.
+# TYPE http_requests counter
+http_requests_total{method="GET",status="200"} 1234.0 1714521600.000
+# EOF
+```
+
+The OTel Prometheus exporter supports both Prometheus text format and OpenMetrics format; select via `Accept: application/openmetrics-text` request header.
+
+---
+
+## 4. SLI / Healthcheck Basics
+
+### 4.1 Google SRE Golden Signals
+
+Four signals for monitoring any request-serving system:
+
+| Signal | Description | Primary instrument |
+|--------|-------------|--------------------|
+| Latency | Time to serve a request (good vs error latency tracked separately) | Histogram |
+| Traffic | Volume of demand (requests/sec, queries/sec) | Counter rate |
+| Errors | Rate of failed requests (explicit 5xx, implicit latency budget exceeded) | Counter rate |
+| Saturation | How "full" the service is; utilization as fraction of capacity | Gauge or Histogram |
+
+### 4.2 RED Method (request-serving services)
+
+| Signal | PromQL sketch |
+|--------|---------------|
+| Rate | `sum(rate(http_requests_total[5m])) by (service)` |
+| Errors | `sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)` |
+| Duration | `histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))` |
+
+### 4.3 USE Method (resource-level: CPU, memory, disk)
+
+| Signal | PromQL sketch |
+|--------|---------------|
+| Utilization | `avg(system.cpu.utilization) by (host)` |
+| Saturation | `system.cpu.load_average.1m / count(system.cpu.utilization) by (host)` |
+| Errors | `rate(system.network.errors{direction="transmit"}[5m])` |
+
+### 4.4 SLI Examples (PromQL)
+
+Availability SLI (fraction of successful requests over the measurement window):
+
+```promql
+sum(rate(http_requests_total{status=~"2..|3.."}[5m]))
+  /
+sum(rate(http_requests_total[5m]))
+```
+
+Latency SLI (p99 from histogram, 5-minute rolling window):
+
+```promql
+histogram_quantile(
+  0.99,
+  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
+)
+```
+
+Cross-ref `../boundaries/slo.md` for SLO target definitions, error-budget burn-rate computation, and multi-window alert rules.
+
+### 4.5 Healthcheck Endpoints and Metrics
+
+Kubernetes readiness and liveness probes (`/readyz`, `/healthz`) are HTTP checks; they are not metrics themselves. However, the underlying health state MUST be exposed as a metric so that dashboards and SLO computation can consume it continuously:
+
+```promql
+# Health metric emitted by application (1 = healthy, 0 = unhealthy)
+up{job="my-service"}   # Prometheus scrape synthetic metric
+
+# Custom health gauge from OTel SDK
+service_health_status{service="checkout", check="database"} 1
+```
+
+Anti-pattern: a healthcheck that returns 200 without reflecting actual dependency health. If the database is down, the readiness probe MUST return a non-2xx status, and the `service_health_status` gauge MUST reflect 0. A probe that always returns 200 is not tied to an underlying SLI metric and produces false-positive availability signals.
+
+---
+
+## 5. hostmetrics Receiver (OTel Collector)
+
+Source: <https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver>
+
+The `hostmetrics` receiver collects host-level OS metrics without requiring a separate agent. It runs inside the OTel Collector process and reads from OS interfaces.
+
+### 5.1 Deployment
+
+Deploy as DaemonSet (one Collector per node) to collect per-node metrics. Cross-ref `../transport/collector-topology.md` for the two-tier topology (DaemonSet agent + gateway).
+
+```yaml
+receivers:
+  hostmetrics:
+    collection_interval: 30s
+    root_path: /hostfs    # required when running in container; mount host /proc and /sys
+    scrapers:
+      cpu: {}
+      memory: {}
+      disk: {}
+      filesystem: {}
+      network: {}
+      load: {}
+      paging: {}
+      process:
+        mute_process_name_error: true
+```
+
+### 5.2 Key Metrics
+
+| Metric | Type | Description |
+|--------|------|-------------|
+| `system.cpu.time` | Counter | CPU time by state (user, system, idle, iowait, irq, softirq, steal, nice) |
+| `system.cpu.utilization` | Gauge | CPU utilization 0.0–1.0 by state |
+| `system.memory.usage` | Gauge | Memory bytes by state (used, free, cached, buffered) |
+| `system.memory.utilization` | Gauge | Memory utilization 0.0–1.0 by state |
+| `system.disk.io` | Counter | Disk bytes read/written by device |
+| `system.disk.io_time` | Counter | Time disk is active by device (seconds) |
+| `system.filesystem.usage` | Gauge | Filesystem bytes by mount point and state |
+| `system.network.io` | Counter | Network bytes transmitted/received by interface |
+| `system.network.errors` | Counter | Network errors by interface and direction |
+| `system.load.average.1m` | Gauge | 1-minute load average |
+| `system.load.average.5m` | Gauge | 5-minute load average |
+| `system.load.average.15m` | Gauge | 15-minute load average |
+| `system.paging.usage` | Gauge | Swap/paging space by state |
+| `process.cpu.time` | Counter | Per-process CPU time by state |
+| `process.memory.physical_usage` | Gauge | Per-process RSS in bytes |
+
+### 5.3 Alternative: Prometheus Node Exporter
+
+For environments where the OTel Collector cannot run on the host, Prometheus Node Exporter is the mature alternative (CNCF Graduated). Scrape with `prometheusreceiver` and convert to OTel metrics via the Collector pipeline. Metric names differ from OTel semconv (e.g., `node_cpu_seconds_total` vs `system.cpu.time`). Prefer `hostmetrics` receiver for new deployments.
+
+---
+
+## 6. Kubernetes Metric Sources
+
+Three complementary sources cover cluster, node, pod, and container metrics:
+
+| Source | Scope | Collector component |
+|--------|-------|---------------------|
+| `kubeletstats` receiver | Pod and container metrics from kubelet API | `kubeletstatsreceiver` |
+| `k8scluster` receiver | Cluster-level state (node conditions, pod phases, deployment replicas) | `k8sclusterreceiver` |
+| Prometheus operator + ServiceMonitor CRD | Scrape application Prometheus endpoints declaratively | `prometheusreceiver` |
+
+Cross-ref `../transport/collector-topology.md §Component preferences` for the recommended configuration of each source in a DaemonSet + gateway topology.
+
+---
+
+## 7. Application Metric Integration
+
+### 7.1 Auto-instrumentation
+
+OTel SDK auto-instrumentation provides RED metrics out-of-the-box for common frameworks:
+- HTTP servers: `http.server.request.duration` (Histogram), `http.server.active_requests` (UpDownCounter)
+- HTTP clients: `http.client.request.duration` (Histogram)
+- gRPC: `rpc.server.duration` (Histogram), `rpc.client.duration` (Histogram)
+- Database clients: `db.client.operation.duration` (Histogram)
+
+These use OTel semconv stable attributes (`http.*`, `rpc.*`, `db.*` core groups). Verify your framework's OTel instrumentation library supports the attribute groups before relying on them in SLO computation.
+
+### 7.2 Custom Metrics via Meter API
+
+```python
+# Python SDK example — creating a histogram for a custom operation
+from opentelemetry import metrics
+
+meter = metrics.get_meter("com.example.checkout", version="1.0.0")
+
+payment_duration = meter.create_histogram(
+    name="payment.processing.duration",
+    unit="s",
+    description="Time to process a payment request",
+)
+
+# Recording a measurement
+payment_duration.record(0.42, {"payment.method": "card", "currency": "USD"})
+```
+
+### 7.3 Attribute Allow-listing for Cardinality Control
+
+Explicitly allow-list attributes on each instrument view to prevent label explosion. Cross-ref `../meta-observability.md §Section C`:
+
+```python
+from opentelemetry.sdk.metrics.view import View
+
+view = View(
+    instrument_name="payment.processing.duration",
+    attribute_keys={"payment.method", "currency"},  # only these two labels emitted
+)
+```
+
+---
+
+## 8. OpenCost Metric Surface
+
+Source: <https://www.opencost.io/docs/configuration/prometheus>
+
+OpenCost (CNCF Incubating) exposes cost attribution in Prometheus format at its `/metrics` endpoint. Scrape via `prometheusreceiver` to bring cost into the same TSDB as latency and error metrics, making cost a dimension alongside performance.
+
+### 8.1 Key OpenCost Metrics
+
+OpenCost exposes allocation and price gauges, not automatic namespace/workload cost counters. Multiply matching quantities and rates and aggregate by namespace; see `cost.md` for the concrete compute-cost query.
+
+| Metric | Unit | Description |
+|--------|------|-------------|
+| `node_cpu_hourly_cost` | USD / core-hour | Node CPU unit price |
+| `node_ram_hourly_cost` | USD / GiB-hour | Node memory unit price |
+| `container_cpu_allocation` | cores | Container CPU allocation |
+| `container_memory_allocation_bytes` | bytes | Container memory allocation |
+| `pod_pvc_allocation` | bytes | Pod PVC allocation |
+| `pv_hourly_cost` | USD / GiB-hour | Volume unit price |
+
+Source: <https://opencost.io/docs/integrations/metrics/>. Label-based tenant/workload totals require explicit API aggregation or verified recording rules.
+
+### 8.2 Scrape Configuration
+
+```yaml
+receivers:
+  prometheus:
+    config:
+      scrape_configs:
+        - job_name: opencost
+          scrape_interval: 60s
+          static_configs:
+            - targets: ["opencost.opencost.svc.cluster.local:9003"]
+```
+
+Cross-ref `cost.md` for the full FinOps unit-economics model, per-tenant cost attribution, and chargeback/showback patterns.
+
+---
+
+## 9. Cardinality Budget
+
+Cross-ref `../meta-observability.md §Section C Cardinality Guardrails` for full detail and remediation procedures.
+
+### 9.1 Per-Service Budget
+
+Set a hard budget per service, enforced via alert at 80% of budget:
+
+```promql
+# Alert: service approaching 5000-series budget
+count({job="my-service"}) > 4000
+```
+
+### 9.2 Forbidden Labels (Never use as metric labels)
+
+| Label | Reason |
+|-------|--------|
+| `user.id` | Unbounded; one series per user |
+| `request.id` | One series per request; instant explosion |
+| `trace.id` | One series per trace |
+| `url.full` (raw) | Query strings are unbounded |
+| `error.message` | Free-text; unbounded cardinality |
+
+### 9.3 Tenant Cap
+
+`tenant.id` label is allowed with a hard cap: top-N explicit tenants (e.g., 1000), all others mapped to label value `"other"`. Never create a new metric name per tenant; this bypasses TSDB cardinality controls entirely (anti-pattern listed in Section 12).
+
+---
+
+## 10. Long-Term Storage Backends
+
+Prometheus-compatible TSDBs for metrics retention beyond 15 days:
+
+| Backend | CNCF Status | Key Characteristic | Typical Retention |
+|---------|-------------|--------------------|-------------------|
+| Prometheus | CNCF Graduated | Local storage, no HA by default | 15d (short-term) |
+| Thanos | CNCF Incubating | Object storage long-term; Prometheus sidecar model | 1y+ |
+| Cortex | CNCF Incubating | Multi-tenant Prometheus; horizontally scalable | 1y+ |
+| Grafana Mimir | Not CNCF | Grafana Labs fork of Cortex; production-grade | 1y+ |
+| VictoriaMetrics | Not CNCF | High-performance; efficient storage compression | 1y+ |
+
+Cross-ref `../vendor-categories.md §TSDB / Long-term Metrics` for selection guidance per workload size and multi-tenancy requirements.
+
+---
+
+## 11. Matrix Coverage (metrics column)
+
+This table maps the metrics signal cells from `../matrix.md` for quick navigation.
+
+| Layer | Boundary | Coverage | Detail |
+|-------|----------|----------|--------|
+| L3-network | multi-tenant | PASS | Per-tenant VPC flow byte/packet counters |
+| L3-network | cross-application | PASS | Inter-VPC/peering flow metrics |
+| L4-transport | multi-tenant | PASS | Per-tenant TCP retransmit rate, RTT histograms via eBPF |
+| L4-transport | cross-application | PASS | Cross-service TCP RTT and retransmit metrics via eBPF |
+| mesh | multi-tenant | PASS | Per-tenant RED from Envoy telemetry; `tenant.id` via baggage |
+| mesh | cross-application | PASS | Cross-service RED metrics at mesh proxy; topology mapping |
+| mesh | slo | PASS | Request rate and error rate as primary SLI sources |
+| mesh | release | PASS | Request split metrics for canary traffic (Flagger/Argo Rollouts) |
+| L7-application | multi-tenant | PASS | Per-tenant RED + custom business metrics with `tenant.id` |
+| L7-application | cross-application | PASS | Inter-service histograms with `service.name` + `service.peer.name` |
+| L7-application | slo | PASS | SLI metric computation; SLO targets in OpenSLO YAML |
+| L7-application | release | PASS | Release marker; `service.version` label for before/after delta |
+
+---
+
+## 12. Anti-Patterns
+
+These are candidates for `../anti-patterns.md §Section B Cardinality & Cost`:
+
+| Anti-pattern | Description | Correction |
+|--------------|-------------|------------|
+| `user.id` as metric label | Creates one series per user; TSDB storage explosion | Use `user.tier` or aggregated cohort label |
+| New metric name per tenant | `http_requests_total_tenant_acme` bypasses cardinality controls | Use `http_requests_total{tenant="acme"}` with top-N cap |
+| Summary for cross-service aggregation | Client-side quantiles cannot be merged across replicas | Replace with Histogram + `histogram_quantile` at query time |
+| Healthcheck not tied to underlying SLI | `/healthz` returns 200 regardless of dependency health | Emit `service_health_status{check="db"} 0` when unhealthy; wire into SLI |
+| Raw `url.full` label | Query strings are unbounded; `?token=...` leaks secrets | Use `http.route` (normalized); apply `replace_pattern` in Collector |
+| Histogram with default bucket boundaries | Default millisecond buckets are meaningless for second-scale requests | Set explicit bucket boundaries per instrument matching the expected value range |
+
+---
+
+## Cross-References
+
+| Topic | File |
+|-------|------|
+| SLO policy, error budget, burn-rate alerts | `../boundaries/slo.md` |
+| Full FinOps / unit economics | `cost.md` |
+| Cardinality guardrails and remediation | `../meta-observability.md §Section C` |
+| Collector two-tier topology (DaemonSet + gateway) | `../transport/collector-topology.md` |
+| OTel spec and semconv stability tiers | `../standards.md` |
+| Anti-patterns (Section B) | `../anti-patterns.md` |
+| Vendor TSDB selection | `../vendor-categories.md §TSDB / Long-term Metrics` |
+| Full matrix coverage (all signals) | `../matrix.md` |
+| Logs signal | `logs.md` |
+| Traces signal | `traces.md` |
+| Profiles signal | `profiles.md` |