oh-my-agent-commandcode 0.0.0-stage → 20.38.3

package/skills/oma-observability/resources/transport/collector-topology.md

@@ -0,0 +1,308 @@
+---
+otel_spec: "1.x (stable API/SDK)"
+tools:
+  - "OTel Operator: v1beta1"
+  - "Fluent Bit: CNCF Graduated"
+  - "OTel Collector: v0.122.x"
+---
+
+# Collector Topology: Deployment Modes and Kubernetes Patterns
+
+
+---
+
+## 1. Four Deployment Modes (OTel Operator `Spec.Mode`)
+
+The OpenTelemetry Operator manages collectors via the `OpenTelemetryCollector` CRD at **v1beta1** (beta stability as of 2025). The `spec.mode` field controls the deployment strategy.
+
+| Mode | Kubernetes Kind | Replicas | Typical Use Case |
+|------|----------------|----------|-----------------|
+| `deployment` | Deployment | 1..N | Gateway, tail sampling, aggregation |
+| `daemonset` | DaemonSet | 1 per node | Node-level metrics, log collection |
+| `statefulset` | StatefulSet | 1..N | Ordered, persistent-volume workloads |
+| `sidecar` | injected container | 1 per pod | Serverless (Fargate, Cloud Run), isolation |
+
+**`Deployment` (default; Gateway use case)**
+Runs as a standard Kubernetes Deployment. Suitable for centralized processing: batching, tail sampling, exporters to backend. Scales horizontally but cannot access node-local resources (e.g., `/var/log`, `/proc`). Source: [OTel Collector Deployment docs](https://opentelemetry.io/docs/collector/deployment/).
+
+**`DaemonSet` (Agent use case; one per node)**
+Guarantees exactly one collector pod per node. Required for receivers that must run on the host: `hostmetrics`, `filelog`, `kubeletstats`. Accesses node filesystem and network namespaces directly.
+
+**`StatefulSet` (specialized)**
+Useful when collectors need stable network identity or persistent volumes (e.g., Write-Ahead Log for durability). Uncommon; prefer Deployment unless stable pod identity is mandatory.
+
+**`Sidecar` (one per pod)**
+Injected by the operator via pod annotation (`sidecar.opentelemetry.io/inject: "true"`). Runs in the same pod as the application. See Section 3 for when sidecar is appropriate.
+
+> CRD stability note: `OpenTelemetryCollector` v1beta1 is beta; `Instrumentation` CRD is also v1beta1. GA graduation is tracked in [opentelemetry-operator](https://github.com/open-telemetry/opentelemetry-operator) releases.
+
+---
+
+## 2. 2026 Recommended Pattern: Two-Tier Hybrid
+
+The standard production pattern for Kubernetes clusters is a two-tier architecture separating node-level collection from centralized processing.
+
+```
+App Pod(s)
+    │ OTLP gRPC/HTTP
+    ▼
+DaemonSet Agent (1 per node)
+  - hostmetrics / filelog / kubeletstats receivers
+  - memory_limiter and k8sattributes enrichment
+  - traces: loadbalancing exporter, routing_key: traceID
+    │ consistent traceID hash to individual gateway replicas
+    │ metrics/logs may use their separately configured normal load balancing
+    ▼
+Deployment Gateway (N tail-sampler replicas, per-pod discovery)
+  - memory_limiter → tail_sampling → batch
+  - backend exporter
+    │ OTLP / vendor protocol
+    ▼
+Observability Backend (e.g., Grafana Cloud, Jaeger, Prometheus)
+```
+
+Do not send traces through an ordinary ClusterIP/load balancer before independent
+tail samplers. It can split one trace across replicas. The traceID-aware exporter
+must be upstream of those samplers; an exporter after sampling cannot recover
+trace completeness. Resolve individual sampler replicas (for example via a
+headless Service), and validate late-span behavior during membership changes.
+
+**Agent layer responsibilities** (must be per-host):
+- `hostmetrics` receiver: CPU, memory, disk, network from `/proc`, `/sys`
+- `filelog` receiver: container log files from `/var/log/pods/`
+- `k8sattributes` processor: enriches spans/metrics with pod, namespace, node labels
+- `kubeletstats` receiver: pod/container resource metrics via Kubelet API
+
+**Gateway layer responsibilities**:
+- `batch` processor: reduce export RPCs, improve compression
+- `tail_sampling` processor: evaluate complete traces before sampling decision
+- Backend exporter: send the gateway's already-sampled traces to the selected backend. Put `loadbalancing` upstream of any further independent tail-sampler tier, not after the only sampler.
+- `memory_limiter` processor: prevent OOM under backpressure
+
+Example agent `OpenTelemetryCollector` manifest snippet for host metrics/logs only (traces additionally require an OTLP receiver and the traceID-aware `loadbalancing` exporter described above and in §6):
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  name: otel-agent
+spec:
+  mode: daemonset
+  config:
+    receivers:
+      hostmetrics:
+        collection_interval: 30s
+        scrapers:
+          cpu: {}
+          memory: {}
+          filesystem: {}
+      filelog:
+        include: [/var/log/pods/*/*/*.log]
+        include_file_path: true
+      kubeletstats:
+        collection_interval: 20s
+        auth_type: serviceAccount
+    processors:
+      k8sattributes:
+        passthrough: false
+        extract:
+          metadata: [k8s.pod.name, k8s.namespace.name, k8s.node.name]
+      memory_limiter:
+        limit_mib: 400
+        check_interval: 1s
+    exporters:
+      otlp:
+        endpoint: otel-gateway:4317
+    service:
+      pipelines:
+        metrics:
+          receivers: [hostmetrics, kubeletstats]
+          processors: [memory_limiter, k8sattributes]
+          exporters: [otlp]
+        logs:
+          receivers: [filelog]
+          processors: [memory_limiter, k8sattributes]
+          exporters: [otlp]
+```
+
+---
+
+## 3. When to Use Sidecar (NOT Default)
+
+Sidecar mode is **not recommended for standard Kubernetes nodes**. Use it only when DaemonSet is unavailable or per-pod isolation is a hard requirement.
+
+| Condition | Use Sidecar? | Reason |
+|-----------|-------------|--------|
+| AWS Fargate | Yes | DaemonSet not supported on Fargate nodes |
+| GCP Cloud Run | Yes | No DaemonSet concept; sidecar containers supported |
+| Strong per-pod isolation | Yes | Compliance or security boundary per workload |
+| Multi-tenant pod separation | Yes | Each tenant's data must not cross pod boundaries |
+| Standard Kubernetes nodes | No | N collectors for N pods; cost scales linearly |
+| Tail sampling requirement | No | Trace spans split across pods; sampler sees incomplete traces |
+
+> Warning: Enabling sidecar injection on standard Kubernetes creates one collector instance per application pod. For a 100-pod deployment, this is 100 collectors. CPU and memory costs multiply accordingly. Always prefer DaemonSet on regular nodes.
+
+---
+
+## 4. Collector Component Placement Reference
+
+Source: [OTel Kubernetes Collector Components](https://opentelemetry.io/docs/platforms/kubernetes/collector/components/)
+
+| Receiver / Processor | Preferred Mode | Notes |
+|---------------------|---------------|-------|
+| `kubeletstats` receiver | DaemonSet | Requires access to Kubelet API per node |
+| `filelog` receiver | DaemonSet | Reads `/var/log/pods/` on each node |
+| `hostmetrics` receiver | DaemonSet | Reads `/proc`, `/sys`; host-only access |
+| `k8sattributes` processor | DaemonSet or Gateway | On agent: enriches at source; on gateway: enriches late |
+| `prometheus` receiver | Deployment (with caveats) | No automatic horizontal scaling for scrape target distribution; use sharding manually |
+| `tail_sampling` processor | Deployment (Gateway) | Requires complete trace; combine with `loadbalancing` exporter |
+| `batch` processor | Both | Tuned independently per tier |
+| `memory_limiter` processor | Both (mandatory) | Always include; prevents OOM under traffic spikes |
+
+> Prometheus scraping caveat: When multiple gateway replicas each run `prometheus` receiver, scrape targets are duplicated across replicas causing duplicate metrics. Use target allocator (`spec.targetAllocator`) provided by the OTel Operator to distribute scrape targets across replicas.
+
+---
+
+## 5. Container Runtime Observability Integration
+
+**containerd and CRI-O**
+
+Both runtimes expose metrics and events relevant to infrastructure observability:
+- Image pull duration and failure rates
+- Container start/stop lifecycle events
+- Runtime panic or OOM kill events
+
+Integration approaches:
+- `hostmetrics` receiver with `process` scraper captures container process CPU/memory
+- containerd exposes metrics via its built-in Prometheus endpoint (default `:1338/metrics`)
+- CRI-O exposes metrics via `:9537/metrics`; scrape with `prometheus` receiver on the DaemonSet agent
+
+**cAdvisor integration**
+
+cAdvisor runs as part of the Kubelet on each node and exposes per-container CPU, memory, filesystem, and network metrics.
+
+```yaml
+# Scrape cAdvisor via kubeletstats receiver (preferred)
+receivers:
+  kubeletstats:
+    collection_interval: 20s
+    auth_type: serviceAccount
+    endpoint: "https://${env:K8S_NODE_IP}:10250"
+    insecure_skip_verify: true
+    metric_groups: [container, pod, node]
+```
+
+**Pod and container attribute enrichment**
+
+The `k8sattributes` processor queries the Kubernetes API to attach resource attributes to all telemetry:
+
+```yaml
+processors:
+  k8sattributes:
+    auth_type: serviceAccount
+    passthrough: false
+    extract:
+      metadata:
+        - k8s.pod.name
+        - k8s.pod.uid
+        - k8s.deployment.name
+        - k8s.namespace.name
+        - k8s.node.name
+        - k8s.container.name
+      labels:
+        - tag_name: app.label.version
+          key: app.kubernetes.io/version
+          from: pod
+    pod_association:
+      - sources:
+          - from: resource_attribute
+            name: k8s.pod.ip
+```
+
+---
+
+## 6. High-Throughput Gateway Scaling
+
+For high-volume environments, the gateway tier must scale to avoid becoming a bottleneck.
+
+**Horizontal scaling with consistent hashing**
+
+Tail sampling requires that all spans for a given `trace_id` reach the same collector replica. Use `loadbalancing` exporter in a first-tier gateway to route traces consistently:
+
+```yaml
+exporters:
+  loadbalancing:
+    protocol:
+      otlp:
+        tls:
+          insecure: true
+    resolver:
+      k8s:
+        service: otel-tail-sampler-headless
+        ports: [4317]
+    routing_key: traceID  # consistent hash on trace_id
+```
+
+**Two-tier gateway pattern for tail sampling**
+
+```
+DaemonSet Agents
+    │
+    ▼
+Gateway Tier-1 (Deployment, 3+ replicas)
+  loadbalancing exporter → consistent hash by trace_id
+    │
+    ▼
+Gateway Tier-2 (Deployment, 3+ replicas)
+  tail_sampling processor
+  batch + memory_limiter
+    │
+    ▼
+Backend
+```
+
+Tier-1 only routes; Tier-2 owns sampling decisions. This prevents trace fragmentation across replicas.
+
+---
+
+## 7. Federated / Multi-Cluster
+
+For multi-cluster and multi-region deployments:
+
+```
+Cluster A (Region US)          Cluster B (Region EU)
+  DaemonSet Agents               DaemonSet Agents
+       │                               │
+  Regional Gateway               Regional Gateway
+  (edge aggregation)             (edge aggregation)
+       │                               │
+       └──────────┬────────────────────┘
+                  ▼
+         Central Gateway (single region or multi-region active-active)
+                  │
+             Backend
+```
+
+Key considerations:
+- **Cross-cloud egress cost**: Exporting telemetry across cloud providers or regions incurs data transfer charges. Evaluate sampling aggressively at the regional edge to reduce egress volume before forwarding to the central gateway.
+- **Network latency**: Tail sampling decision wait (typically 30s) must account for cross-region span arrival delay. Increase `decision_wait` accordingly.
+- **Authentication**: Use workload identity / OIDC per cluster; avoid static credentials for cross-cluster OTLP export.
+
+---
+
+## 8. Anti-Patterns
+
+| Anti-Pattern | Why It Fails | Correct Approach |
+|-------------|-------------|-----------------|
+| Sidecar collectors on standard Kubernetes nodes | One collector per pod; CPU/memory cost scales with pod count | Use DaemonSet agent per node |
+| Tail sampling in sidecar | Sidecar sees only spans from its pod; trace is incomplete across services | Run tail_sampling in gateway tier only |
+| Single gateway replica | No high availability; single point of failure under load | Minimum 2-3 replicas; use PodDisruptionBudget |
+| Missing `memory_limiter` processor | OOM kill under traffic spikes or backpressure from backend | Always add memory_limiter as first processor in every pipeline |
+| Prometheus receiver on multiple replicas without target allocator | Duplicate scrapes = duplicate metrics in backend | Use OTel Operator target allocator for Prometheus receiver |
+| Skipping `loadbalancing` exporter before tail sampler | Spans for same trace land on different replicas; sampling decision is wrong | Always pair tail_sampling with loadbalancing exporter upstream |
+
+## References
+
+- https://opentelemetry.io/docs/platforms/kubernetes/collector/components/
+- https://github.com/open-telemetry/opentelemetry-operator
+- https://opentelemetry.io/docs/collector/deployment/

package/skills/oma-observability/resources/transport/otlp-grpc-vs-http.md

@@ -0,0 +1,256 @@
+---
+otel_spec: "1.x (stable API/SDK)"
+---
+
+# OTLP Transport: gRPC vs HTTP Decision Guide
+
+## 1. OTLP Basics
+
+The OpenTelemetry Protocol (OTLP) uses a single Protobuf schema for traces, metrics, and logs.
+It defines three wire formats over two transports:
+
+| Transport      | Content-Type                       | Port (default) |
+|----------------|------------------------------------|----------------|
+| gRPC           | `application/grpc`                 | **4317**       |
+| HTTP/Protobuf  | `application/x-protobuf`           | **4318**       |
+| HTTP/JSON      | `application/json`                 | **4318**       |
+
+All three formats carry identical semantic data. Transport choice is an operational decision.
+
+Sources:
+- OTLP specification: opentelemetry.io/docs/specs/otlp/
+- Collector configuration: opentelemetry.io/docs/collector/configuration/
+- OTLP receiver README: github.com/open-telemetry/opentelemetry-collector (receiver/otlpreceiver)
+
+---
+
+## 2. Decision Tree
+
+```
+Is the sender a browser or a runtime without gRPC support?
+  YES → HTTP/protobuf (port 4318) — smallest payload, wide compatibility
+        HTTP/JSON if debugging or if protobuf encoding unavailable
+
+  NO → Does a corporate proxy or layer-7 firewall sit between sender and collector?
+         YES → HTTP (port 4318); proxies handle HTTP/1.1 and HTTP/2 cleartext reliably
+               Confirm with: curl -v http://<collector>:4318/v1/traces
+
+         NO → Is traffic internal to a Kubernetes cluster (pod-to-pod)?
+                YES → gRPC (port 4317): multiplexed streams, lower per-RPC overhead,
+                      native health checking via grpc_health_probe
+                      Use headless Service or client-side LB (see section 6)
+
+                NO → Is volume high (>50 K spans/sec) or latency critical?
+                       YES → gRPC + gzip compression
+                       NO  → Either transport works; prefer gRPC for consistency
+```
+
+| Axis                      | Recommended transport | Reason                                          |
+|---------------------------|-----------------------|-------------------------------------------------|
+| Browser / Electron        | HTTP/JSON or HTTP/protobuf | No gRPC-Web support in standard fetch API  |
+| k8s pod-to-pod            | gRPC 4317             | Multiplexing, streaming, native health check    |
+| Corporate proxy / firewall| HTTP 4318             | Proxies often terminate gRPC streams early      |
+| VPN with deep inspection  | HTTP 4318             | gRPC ALPN negotiation blocked by some DPI       |
+| High-volume batch         | gRPC + gzip           | Header compression (HPACK), stream reuse        |
+| IoT / constrained device  | HTTP/JSON             | Simpler implementation, no Protobuf required    |
+
+---
+
+## 3. Port Conventions
+
+| Port | Protocol | Notes                                                  |
+|------|----------|--------------------------------------------------------|
+| 4317 | gRPC     | IANA-registered; TLS optional (recommended in prod)   |
+| 4318 | HTTP     | IANA-registered; paths `/v1/traces`, `/v1/metrics`, `/v1/logs` |
+
+Do not expose either port to the public internet without TLS and authentication.
+
+---
+
+## 4. Server-Side Tuning (Collector Receiver)
+
+### 4.1 gRPC Receiver
+
+```yaml
+receivers:
+  otlp:
+    protocols:
+      grpc:
+        endpoint: "0.0.0.0:4317"
+        max_recv_msg_size_mib: 16          # default 4 MiB; raise for large trace batches
+        max_concurrent_streams: 100        # per-connection stream limit
+        read_buffer_size: 524288           # 512 KiB per stream
+        keepalive:
+          server_parameters:
+            time: 30s                      # ping interval when idle
+            timeout: 5s                    # wait before closing unresponsive connection
+          enforcement_policy:
+            min_time: 10s                  # reject client pings faster than this
+            permit_without_stream: true
+```
+
+### 4.2 HTTP Receiver
+
+```yaml
+receivers:
+  otlp:
+    protocols:
+      http:
+        endpoint: "0.0.0.0:4318"
+        max_request_body_size: 16777216    # 16 MiB; matches gRPC max_recv_msg_size_mib
+        include_metadata: true             # forward request headers as resource attributes
+        cors:
+          allowed_origins: ["https://app.example.com"]
+          allowed_headers: ["*"]
+```
+
+### 4.3 Memory Limiter (Required)
+
+Place `memory_limiter` first in every pipeline; it back-pressures receivers before OOM.
+
+```yaml
+processors:
+  memory_limiter:
+    check_interval: 1s
+    limit_mib: 512
+    spike_limit_mib: 128       # refuse new data when usage > limit_mib - spike_limit_mib
+
+  batch:
+    send_batch_size: 8192
+    send_batch_max_size: 16384
+    timeout: 5s
+
+service:
+  pipelines:
+    traces:
+      receivers: [otlp]
+      processors: [memory_limiter, batch]
+      exporters: [otlp/tier2]
+```
+
+---
+
+## 5. Client-Side Tuning (Exporter)
+
+### 5.1 Compression
+
+gzip is the default and should remain enabled (typically 70–85% payload reduction).
+Do not disable it unless your TLS offload device double-compresses.
+
+```yaml
+exporters:
+  otlp:
+    endpoint: "collector:4317"
+    compression: gzip          # gzip | zstd | none
+    tls:
+      insecure: false
+      ca_file: /etc/ssl/certs/ca.crt
+```
+
+### 5.2 Sending Queue and Retry
+
+```yaml
+exporters:
+  otlp:
+    endpoint: "collector:4317"
+    compression: gzip
+    sending_queue:
+      enabled: true
+      num_consumers: 10
+      queue_size: 5000
+      storage: file_storage/otlp_queue   # persist queue to disk across restarts
+    retry_on_failure:
+      enabled: true
+      initial_interval: 5s
+      max_interval: 30s
+      max_elapsed_time: 300s             # give up after 5 min total
+```
+
+**OTLP retry semantics** (opentelemetry.io/docs/specs/otlp/): `RESOURCE_EXHAUSTED` returns
+`RetryInfo.retry_delay`; honor it. `UNAVAILABLE` → exponential backoff. `INVALID_ARGUMENT` →
+permanent failure, do not retry.
+
+### 5.3 Persistent Queue Extension
+
+```yaml
+extensions:
+  file_storage/otlp_queue:
+    directory: /var/otel/queue
+    timeout: 10s
+    compaction:
+      on_start: true
+      rebound_needed_threshold_mib: 100
+```
+
+---
+
+## 6. Load Balancing
+
+gRPC uses long-lived HTTP/2 connections, defeating L4 round-robin. Use client-side LB.
+
+**Two-tier pattern** (recommended for tail sampling):
+
+```
+[SDK exporters]
+      |  gRPC 4317
+      v
+[Tier-1: loadbalancingexporter]   ← routes by traceID for consistent routing
+      |  gRPC 4317
+      v
+[Tier-2: tailsamplingprocessor]   ← sees full trace before sampling decision
+      |  OTLP
+      v
+[Backend / storage]
+```
+
+```yaml
+exporters:
+  loadbalancing:
+    protocol:
+      otlp:
+        tls:
+          insecure: false
+        timeout: 5s
+    resolver:
+      dns:
+        hostname: otel-collector-tier2-headless.monitoring.svc.cluster.local
+        port: 4317
+        interval: 5s
+        timeout: 1s
+```
+
+For Kubernetes: use a headless Service (`clusterIP: None`) for Tier-2. The `loadbalancingexporter`
+resolves DNS and maintains one gRPC connection per pod.
+
+---
+
+## 7. Security: TLS and mTLS
+
+### Collector Receiver (server-side)
+
+```yaml
+receivers:
+  otlp:
+    protocols:
+      grpc:
+        endpoint: "0.0.0.0:4317"
+        tls:
+          cert_file: /etc/ssl/certs/collector.crt
+          key_file: /etc/ssl/private/collector.key
+          client_ca_file: /etc/ssl/certs/ca.crt   # enforce mTLS
+```
+
+### Exporter (client-side)
+
+```yaml
+exporters:
+  otlp:
+    endpoint: "collector:4317"
+    tls:
+      ca_file: /etc/ssl/certs/ca.crt
+      cert_file: /etc/ssl/certs/client.crt        # mTLS client cert
+      key_file: /etc/ssl/private/client.key
+```
+
+For Kubernetes workloads, use cert-manager to auto-rotate TLS credentials (90-day TTL).
+Use SPIFFE/SPIRE for zero-touch mTLS in multi-cluster scenarios.