oh-my-agent-commandcode 0.0.0-stage → 20.38.3

package/skills/oma-observability/resources/checklist.md

@@ -0,0 +1,123 @@
+# oma-observability Checklist
+
+> Consolidated verification checklist. Run before shipping observability changes to production.
+> Each item: action; criterion; priority.
+> Source files live under `resources/` from the skill root.
+
+## Priority legend
+
+- **P0**: block ship; do not merge or deploy without this
+- **P1**: must-before-prod; complete before promoting to production
+- **P2**: this-sprint; schedule and close within the current sprint
+
+---
+
+## Section 1: Setup validation (pre-commit / pre-merge)
+
+- [ ] Pin OTel spec version and semconv version in `standards.md` header; must match deployed SDK version (P0)
+- [ ] Define `service.name`, `service.namespace`, `service.version` on every instrumented service OTel Resource; all three attributes present and non-empty (P0)
+- [ ] Define `deployment.environment`, `cloud.provider`, `cloud.region`, `cloud.availability_zone` on every OTel Resource; required for region-dimension pivot in incident forensics (P0)
+- [ ] Define `host.id`, `k8s.pod.name`, `k8s.node.name`, `k8s.cluster.name`, `container.id` on every Kubernetes workload Resource; required for server-dimension pivot (P0)
+- [ ] Configure W3C Trace Context (`traceparent`) as default propagator; every outbound HTTP/gRPC call must forward the header; confirm by checking `trace_id` consistency across service log streams (P0)
+- [ ] Verify W3C Baggage carries only allowed keys; `tenant.id`, feature flag state, deployment SHA, region hint are allowed; user email, session tokens, and credentials are prohibited per `standards.md §W3C Baggage` (P0)
+- [ ] Strip or validate inbound baggage at ingress gateway; external callers must not inject arbitrary baggage into internal services (P0)
+- [ ] Use only stable semconv groups (`service.*`, `host.*`, `cloud.*`, `k8s.*`, `http.*`, `db.*`, `network.*` core, `error.*`) for production SLO inputs; Development-tier attributes (`tls.*`, `network.connection.*`) must not feed SLOs (P0)
+- [ ] Configure `memory_limiter` processor as the first processor in every Collector pipeline; `limit_percentage: 75`, `spike_limit_percentage: 20` (P0)
+- [ ] Place `memory_limiter` before `batch` in pipeline order; prevents OOM crash under burst (P0)
+- [ ] Validate that OTLP transport choice matches topology; gRPC port 4317 for pod-to-pod; HTTP port 4318 for browser SDKs and proxy-traversal paths (P1)
+- [ ] Verify UDP StatsD datagrams fit within path MTU; max 1472 B on standard Ethernet IPv4; use Unix Domain Socket (`unixgram`) for same-host paths (P1)
+- [ ] Version all observability artifacts in git; dashboards, alert rules, SLO definitions, and Collector configs must be applied via CI/CD, not edited in the UI (P1)
+
+---
+
+## Section 2: Pre-production readiness
+
+- [ ] Confirm matrix coverage gaps are identified and acknowledged; every uncovered cell in `matrix.md` has an explicit N/A rationale or a remediation plan (P1)
+- [ ] Set cardinality budget per service; alert at 80% of budget (default 5 000 series/service); configure `count({job="<service>"}) > 4000` alert (P1)
+- [ ] Allow-list metric attributes explicitly in the OTel SDK View; use `attribute_keys` to prevent unbounded label sets; never let raw `url.full`, `user.id`, `request.id`, `trace.id`, or `error.message` appear as metric labels (P0)
+- [ ] Normalize high-cardinality route labels; apply `transform` processor to replace numeric path segments with `/_` (e.g., `/users/42` → `/users/_`) (P1)
+- [ ] Define per-tenant sampling policy when multi-tenant; enterprise tenants may require 100% error retention + dedicated pipeline; free-tier tenants use probabilistic baseline (P1)
+- [ ] Configure four-tier tenant isolation strategy; select Tier (soft label / routing / dedicated-collector / dedicated-backend) per tenant compliance obligation and document the decision (P1)
+- [ ] Propagate `tenant.id` via W3C Baggage on all spans, logs, and metrics in multi-tenant services; use the canonical key `tenant.id`; do not use `customer_id`, `org_id`, or `account_id` as synonyms (P1)
+- [ ] Verify `tenant.region` drives data residency routing; EU and KR tenants must have telemetry routed to compliant regional Collector pipelines before reaching a cross-region backend (P0)
+- [ ] Configure two-tier Collector topology for Kubernetes; DaemonSet agent (hostmetrics, filelog, kubeletstats, k8sattributes) forwarding to Deployment gateway (batch, tail_sampling, exporters) (P1)
+- [ ] Deploy loadbalancing exporter upstream of tail_sampling processor; consistent hash by `trace_id` ensures complete traces arrive at the same gateway replica (P0)
+- [ ] Confirm `exception.type`, `exception.message`, `exception.stacktrace`, `code.function.name`, `code.file.path`, `code.line.number` are populated on every ERROR span or on exception LogRecords joinable by `trace_id`/`span_id`; `span.recordException(e)` remains valid for atomicity, but the span-event form is deprecated since semconv 1.40 — set `OTEL_SEMCONV_EXCEPTION_SIGNAL_OPT_IN=logs/dup` during migration (P0)
+- [ ] Confirm IP address logging is masked or hashed before long-term retention; IP addresses are personal data under GDPR Art. 4(1) and PIPA; apply prefix truncation or HMAC+salt at pipeline ingress (P0)
+
+---
+
+## Section 3: Production operations
+
+- [ ] Scrape OTel Collector self-metrics (`otelcol_*`) from a separate Prometheus instance or second Collector; Collector failures must not destroy their own observability (P0)
+- [ ] Check pipeline health per hop/exporter and implement completed synthetic backend-delivery probes where end-to-end verification is required; set policy-aware thresholds and monitor probe job freshness (`meta-observability.md §A6`) (P0)
+- [ ] Alert on exporter send failures above 1%; `rate(otelcol_exporter_send_failed_spans[5m]) / rate(otelcol_exporter_sent_spans[5m]) > 0.01` for 5 minutes (P0)
+- [ ] Alert on `otelcol_receiver_refused_spans > 0` sustained for 2 minutes; indicates queue full or parse failures (P0)
+- [ ] Alert on Collector heap usage exceeding 75% of container memory limit; `otelcol_process_runtime_heap_alloc_bytes > 0.75 * container_limit` (P0)
+- [ ] Configure NTP or chrony on all host VMs and Kubernetes nodes; clock drift must stay below 100 ms for reliable trace waterfall ordering (P0)
+- [ ] Alert on node clock drift exceeding 100 ms; `node_clock_drift_ms > 100` for 5 minutes triggers waterfall inversion risk (P0)
+- [ ] Emit `node_clock_drift_ms` from each host via chrony textfile collector or node exporter; required for the clock drift alert to fire (P1)
+- [ ] Configure tail_sampling policy with 100% error + 100% high-latency + 5-10% probabilistic baseline; gateway Collector only; `decision_wait: 30s` to buffer all spans (P1)
+- [ ] Apply exporter retry configuration; `initial_interval: 5s`, `max_interval: 30s`, `max_elapsed_time: 300s`, `queue_size: 1000` to handle transient backend unavailability without infinite queue growth (P1)
+- [ ] Scrape Fluent Bit self-metrics at `:2020/api/v1/metrics/prometheus`; monitor `fluentbit_output_errors_total` alongside `otelcol_*` for unified pipeline health view (P1)
+- [ ] Publish a meta-observability Grafana dashboard; must include pipeline delivery ratio panels, receiver accepted vs refused, exporter sent vs failed, queue depth, heap usage, clock drift heatmap, and top cardinality metrics table (P1)
+- [ ] Enforce retention policy per signal; metrics: 15d full-res / 90d 5m / 2y 1h; operational logs: 7d / 30d / 90d; sampled traces: 30d; profiles: 14d (P1)
+
+---
+
+## Section 4: Incident forensics readiness
+
+- [ ] Verify all MRA resource attributes are present on every signal before production; `service.name`, `service.namespace`, `service.version`, `deployment.environment`, `cloud.*`, `host.id`, `k8s.*`, `container.id` (P0)
+- [ ] Confirm `trace_id` appears on every structured log record; log-trace join is impossible without it; set via OTel SDK current span context (P0)
+- [ ] Confirm `span_id` appears on every structured log record alongside `trace_id`; required for sub-trace log correlation (P0)
+- [ ] Inject `request_id` at the API gateway; propagate as `x-request-id` header; log on every record; expose to end user for support tickets (P1)
+- [ ] Verify metric exemplars are enabled and linked to `trace_id`; required for metric-to-trace pivot in Step 2 of the 6-dimension narrowing flow (P1)
+- [ ] Confirm the 6-dimension narrowing flow can execute in under 15 minutes; run a tabletop drill: symptom capture → trace_id acquisition → region → server → service → layer → code → cross-signal validation (P1)
+- [ ] Validate vendor query patterns are documented for all backends in use; Honeycomb, Datadog, Grafana Tempo, Jaeger, Sentry, Elastic query syntax must be accessible during an incident (P1)
+- [ ] Test log-trace join by `trace_id` across at least two services in the production environment; the join must return matching records within the same request's span (P1)
+- [ ] Confirm release marker log events include `service.name`, `service.version`, `deployment.environment`, and deployment SHA; without markers, Step 5 release correlation in the forensics playbook is blind (P0)
+
+---
+
+## Section 5: Compliance & Audit
+
+- [ ] Map audit record classes to approved retention and erasure/legal-hold rules; configure tamper protection and WORM only when the selected obligation requires it (P0)
+- [ ] Implement a scheduled retention check against the approved per-class policy, including legal holds and eligible deletion; alert on a policy violation (P0)
+- [ ] Include all six mandatory audit event categories; authentication, authorization, data access, administrative, security events, system events; before claiming SOC 2 or ISO 27001 compliance (P0)
+- [ ] Carry required attributes on every audit event; `user.id` (pseudonymized), `actor.type`, `action`, `resource.type`, `resource.id`, `event.outcome` (P0)
+- [ ] Implement PII redaction at pipeline ingestion; apply Collector `transform` processor to scrub or hash PII fields before they reach any storage backend (P0)
+- [ ] Enforce GDPR storage limitation; raw personal data must not exceed the retention period justified by the processing purpose; default operational logs ≤ 7 days raw (P0)
+- [ ] Isolate record classes where their storage/access controls differ; personal audit records remain subject to privacy rules and documented erasure/retention exceptions (P0)
+- [ ] Sign a Data Processing Agreement (DPA) with every observability vendor that receives personal data; required before routing production telemetry to SaaS backends (P0)
+- [ ] Classify personal data and document permitted destinations, applicable GDPR Chapter V/PIPA Art.28-8 transfer grounds, safeguards, and any stricter residency contract before export; enforce the approved server-side policy (P0)
+- [ ] Store pseudonymization keys separately from pseudonymized telemetry data; GDPR Art. 32 requires independent access control and audit trail for the key store (P1)
+- [ ] Verify `user.id` and `user.email` are never used as metric labels; cardinality explosion and PII in TSDB are a double violation; use `filter` processor to drop any datapoint carrying these attributes (P0)
+
+---
+
+## Section 6: SLO & Release gate
+
+- [ ] Define SLO with error budget policy for every user-facing service; SLO target, SLA target (SLA < SLO), 28-day rolling window, and burn-rate alert thresholds (P1)
+- [ ] Implement multi-window burn-rate alerts; fast burn: 2% budget in 1 hour; slow burn: 5% budget in 6 hours; both required to avoid alert fatigue and missed slow leaks (P1)
+- [ ] Store SLO definitions in OpenSLO YAML in git; treated as code; PR review required for any SLO target change (P1)
+- [ ] Select SLI source from stable signals; mesh RED metrics or L7 application metrics preferred; log-based SLI is acceptable only as a fallback for un-instrumented services (P1)
+- [ ] Configure canary analysis (Flagger or Argo Rollouts) to use SLI metrics as promotion gates; success rate threshold and latency p99 threshold both required (P1)
+- [ ] Add memory saturation metric as a custom Flagger analysis gate; OOMKilled pods may not fail the success rate threshold before crashing; `container_memory_working_set_bytes` gate prevents silent OOM canary progression (P1)
+- [ ] Emit `service.version` on all metrics, logs, and spans at deploy time; required for before/after comparison in canary analysis and in Step 3c of incident forensics (P0)
+- [ ] Emit a structured release marker event at every deployment; must carry `service.name`, `service.version`, `deployment.environment`, deployment strategy, and deployment SHA (P0)
+- [ ] Record SLO burn-rate threshold crossings as immutable audit events; provides compliance evidence for availability breach; cross-ref `signals/audit.md` (P1)
+- [ ] Define rollback criteria and document the rollback procedure; include metric recovery verification steps in the runbook (P1)
+
+---
+
+## Section 7: Recovery
+
+- [ ] Document Collector queue backpressure remediation runbook; steps: identify via `otelcol_processor_queued_retry_send_queue_length`, scale gateway Collector replicas, check vendor rate limits (P1)
+- [ ] Document clock drift remediation procedure; `chronyc makestep` for immediate re-sync; escalation to PTP for sub-millisecond requirements; check NTP source reachability if drift persists (P1)
+- [ ] Document cardinality bomb mitigation procedure; identify offending metric with `topk(10, count({__name__!=""}) by (__name__))`, add `filter` processor to drop offending label, or `transform` processor to remove the label value (P1)
+- [ ] Document Collector OOM recovery path; verify `memory_limiter` is configured; if OOM still reached, increase container memory limit or reduce `limit_percentage`; never disable `memory_limiter` (P1)
+- [ ] Document vendor fallback plan for observability backend outage; identify secondary backend or local Collector buffering strategy; `sending_queue` with `queue_size` provides short-term buffering (P2)
+- [ ] Document PII incident response procedure; GDPR Art. 33 breach notification to supervisory authority within 72 hours; identify affected telemetry stores, scope the affected data subjects, and engage DPO (P0)
+- [ ] Document audit log retention violation response; verify S3/GCS Object Lock policy; confirm `retention_period` in log backend matches 7y; restore from backup if records were prematurely deleted (P0)
+- [ ] Test Flagger auto-rollback by injecting a synthetic error rate above the failure threshold in a staging environment; verify rollback completes before the `progressDeadlineSeconds` expires (P2)
+- [ ] Validate that `memory_limiter` refused spans alert fires before Collector OOM; run a controlled load test; confirm `otelcol_processor_memory_limiter_refused_spans > 0` alert fires and data loss stops short of crash (P2)

package/skills/oma-observability/resources/examples.md

@@ -0,0 +1,448 @@
+# oma-observability Examples
+
+> End-to-end walkthroughs demonstrating the skill's value.
+> Each scenario ties multiple files together into a single executable story.
+
+---
+
+## Scenario 1: Greenfield: OSS Full-Stack on Kubernetes
+
+**Situation:** Series A startup, new k8s cluster. Team wants OTel-native metrics + logs + traces + Grafana
+dashboards at zero licensing cost.
+
+**Intent:** `setup`
+
+**Files referenced:** `vendor-categories.md §(a)`, `transport/collector-topology.md §2`, `observability-as-code.md §5`, `checklist.md`
+
+### Walkthrough
+
+1. Invoke `/oma-observability "set up OTel stack on k8s"`.
+2. Intent classifier routes to `setup`.
+3. `vendor-categories.md §(a) OSS Full-Stack` selects **Grafana LGTM+** (Mimir, Loki, Tempo, Grafana)
+   for a team already familiar with Grafana dashboards.
+4. `transport/collector-topology.md §2 Two-Tier Hybrid` determines the deployment pattern:
+   DaemonSet agent per node + Deployment gateway.
+
+**Agent CRD; DaemonSet (node-level collection):**
+
+```yaml
+apiVersion: opentelemetry.io/v1beta1
+kind: OpenTelemetryCollector
+metadata:
+  name: otel-agent
+  namespace: observability
+spec:
+  mode: daemonset
+  config:
+    receivers:
+      hostmetrics:
+        collection_interval: 30s
+        scrapers: { cpu: {}, memory: {}, filesystem: {} }
+      filelog:
+        include: [/var/log/pods/*/*/*.log]
+        include_file_path: true
+      kubeletstats:
+        collection_interval: 20s
+        auth_type: serviceAccount
+    processors:
+      memory_limiter:            # mandatory: first processor in every pipeline
+        limit_mib: 400
+        check_interval: 1s
+      k8sattributes:
+        extract:
+          metadata: [k8s.pod.name, k8s.namespace.name, k8s.node.name]
+    exporters:
+      otlp:
+        endpoint: otel-gateway.observability.svc:4317
+    service:
+      pipelines:
+        metrics:
+          receivers: [hostmetrics, kubeletstats]
+          processors: [memory_limiter, k8sattributes]
+          exporters: [otlp]
+        logs:
+          receivers: [filelog]
+          processors: [memory_limiter, k8sattributes]
+          exporters: [otlp]
+```
+
+**Gateway CRD; Deployment (batching, `mode: deployment`, 2 replicas):**
+
+```yaml
+# Key fields only — full config in observability-as-code.md §5
+spec:
+  mode: deployment
+  replicas: 2
+  config:
+    receivers:
+      otlp: { protocols: { grpc: { endpoint: 0.0.0.0:4317 } } }
+    processors:
+      memory_limiter: { check_interval: 1s, limit_percentage: 75 }
+      batch: {}
+    exporters:
+      otlphttp/tempo: { endpoint: http://tempo.observability.svc:4318 }
+      prometheusremotewrite: { endpoint: http://mimir.observability.svc:9009/api/v1/push }
+      loki: { endpoint: http://loki.observability.svc:3100/loki/api/v1/push }
+    service:
+      pipelines:
+        traces: { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlphttp/tempo] }
+        metrics: { receivers: [otlp], processors: [memory_limiter, batch], exporters: [prometheusremotewrite] }
+        logs: { receivers: [otlp], processors: [memory_limiter, batch], exporters: [loki] }
+```
+
+5. Collector config is committed to git and applied via Argo CD app-of-apps per `observability-as-code.md §7`.
+6. Validation checklist (from `checklist.md §1, §3`):
+   - [ ] `memory_limiter` is the first processor in every pipeline
+   - [ ] NTP drift < 100 ms on all nodes (`chronyc tracking`)
+   - [ ] Cardinality budget set before first data ingest
+   - [ ] Gateway has `≥ 2` replicas with PodDisruptionBudget
+
+**Outcome:** 9 pods, 2 services, first trace visible in Grafana Tempo within 15 minutes of `kubectl apply`.
+
+---
+
+## Scenario 2: Incident Forensics: Payment Service 5xx Spike
+
+**Situation:** "ap-northeast-2 payment service 5xx spike at 14:20 UTC"; alert fires with no `trace_id`
+provided. On-call SRE must localize root cause in under 15 minutes.
+
+**Intent:** `investigate`
+
+**Files referenced:** `incident-forensics.md §3–§5 Scenario A`, `boundaries/slo.md §7`, `signals/traces.md`, `signals/logs.md §7`, `boundaries/release.md`
+
+### Walkthrough
+
+1. **Alert source:** multi-window burn-rate alert from `boundaries/slo.md §7` fires; 2 % budget
+   consumed in 1 h at 14.4× rate. `severity: critical`.
+2. Invoke `/oma-observability --investigate "5xx spike in ap-northeast-2"`.
+3. Router routes to `incident-forensics.md §3 Six-Dimension Narrowing Flow`.
+
+**Step 1; Acquire `trace_id` via metric exemplar (`incident-forensics.md §2`):**
+
+```promql
+histogram_quantile(0.99,
+  sum by (le, service_name) (
+    rate(http_server_request_duration_seconds_bucket{
+      deployment_environment="prod",
+      cloud_region="ap-northeast-2"
+    }[5m])
+  )
+)
+# Click a spike data point → Grafana Mimir exemplar panel shows trace_id link
+```
+
+Exemplar retrieved: `trace_id = 9a3f1c8e2b7d4e50a1c3f9d2b8e7a4c0`.
+
+**Step 2; Six-dimension narrowing:**
+
+| Dimension | Finding |
+|-----------|---------|
+| Region | `cloud.region = ap-northeast-2` only. `ap-southeast-1` healthy. Single-region blast radius. |
+| Server | Errors on 3 of 6 pods, all on `k8s.node.name = ip-10-0-2-11.ec2.internal`. |
+| Service | Earliest ERROR span: `service.name = payments-checkout`, `service.version = v2.4.1`. |
+| Layer | `span.kind = CLIENT`, `db.system = redis`; L7 DB call, not a network-layer issue. |
+| Code | `exception.type = TimeoutException`, `exception.message = "Redis pool exhausted after 5000ms"`, `code.function.name = processPayment`, `code.line.number = 287`. |
+
+**Step 3; Cross-signal validation (`incident-forensics.md §4`):**
+
+```
+# Grafana Tempo — TraceQL lookup
+{ trace:id = "9a3f1c8e2b7d4e50a1c3f9d2b8e7a4c0" }
+
+# Filter logs by trace_id to find correlated warnings
+# Loki LogQL
+{service_name="payments-checkout"} | json | trace_id="9a3f1c8e2b7d4e50a1c3f9d2b8e7a4c0"
+```
+
+Logs reveal: `WARN: connection pool at capacity (256/256)` at 14:21. Redis
+`redis_pool_active_connections / redis_pool_max_connections` metric at 100 %.
+
+**Step 4; Release correlation:**
+
+`service.version = v2.4.1` deployed at 13:57 UTC; 25 minutes before incident.
+Changelog: Redis connection pool reduced from 512 → 256 in this version.
+
+**Step 5; Remediation:**
+
+Flagger canary analysis `successRate` already < 99 % → auto-rollback to `v2.4.0` triggers
+per `boundaries/release.md §Flagger Canary CR`. Error rate returns to baseline in 3 minutes.
+Deployment event chain preserved in `signals/audit.md` for post-mortem.
+
+---
+
+## Scenario 3: Multi-Tenant Cost Attribution
+
+**Situation:** B2B SaaS now has 50+ tenants. CFO asks: "Which tenants cost most to serve, and
+how does our observability bill break down per customer?"
+
+**Intent:** `route` + `tune`
+
+**Files referenced:** `boundaries/multi-tenant.md §7`, `signals/cost.md §3–§4`, `vendor-categories.md §(f)`, `meta-observability.md §Cardinality Guardrails`
+
+### Walkthrough
+
+1. Tag every pod at deploy time with `tenant.id`; applied via Helm values or admission webhook
+   per `boundaries/multi-tenant.md §7`:
+
+```yaml
+metadata:
+  labels:
+    tenant.id: "acme-corp"
+    tenant.tier: "enterprise"
+```
+
+2. Deploy OpenCost (CNCF Incubating) with Prometheus scrape per `signals/cost.md §3`:
+
+```yaml
+# OTel Collector gateway — OpenCost scrape
+receivers:
+  prometheus:
+    config:
+      scrape_configs:
+        - job_name: opencost
+          scrape_interval: 60s
+          static_configs:
+            - targets: ["opencost.opencost.svc.cluster.local:9003"]
+```
+
+3. Obtain daily per-tenant cost from the OpenCost Allocation API using a 24-hour
+   window and aggregation by the verified workload-label mapping. The stock
+   `/metrics` endpoint does not emit `opencost_workload_cost_total`. See
+   `signals/cost.md §5` for the CPU/RAM price-times-allocation PromQL and the
+   distinction between official gauges, window allocations, and team-implemented
+   cumulative cost counters. Include storage/network/shared costs before billing.
+
+4. Cardinality guard: keep top-1 000 tenants labeled, bucket overflow as `"other"` to prevent
+   TSDB series explosion (`meta-observability.md §Cardinality Guardrails`, `boundaries/multi-tenant.md §13`):
+
+```yaml
+# OTel Collector transform processor — cardinality cap
+processors:
+  transform/cardinality:
+    metric_statements:
+      - context: datapoint
+        statements:
+          - set(attributes["tenant_id"], "other")
+              where attributes["tenant_id"] not in ${env:TOP_TENANT_LIST}
+```
+
+5. Per-tier sampling policy at the gateway to control observability bill per tenant
+   (`boundaries/multi-tenant.md §5`): `enterprise` = 100 %, `pro` = 20 %, `free` = 2 %.
+   Full `tail_sampling` YAML (with `and` sub-policies) lives in `transport/sampling-recipes.md §4`.
+
+6. Include the observability bill itself: instrument the collector pipeline with throughput counters
+   per `tenant.id` to apportion Grafana Cloud / Datadog usage proportionally
+   (`signals/cost.md §4`, `vendor-categories.md §(f)`).
+
+**Outcome:** Identified 3 free-tier tenants consuming 40 % of egress cost. Applied egress rate
+limit at collector ingress (`boundaries/multi-tenant.md §11 Noisy Neighbor Protection`).
+Those tenants were moved to a stricter free-tier cap, reducing monthly egress spend by 35 %.
+
+---
+
+## Scenario 4: Migrating off Fluentd to Fluent Bit
+
+**Situation:** Legacy Fluentd DaemonSet is consuming 400 MB RAM per node vs. Fluent Bit's typical
+80 MB in this measured workload. Use the CNCF migration guide as an informative conversion reference; Fluentd remains Graduated. Team needs zero-downtime
+migration with log parity verification before decommission.
+
+**Intent:** `migrate`
+
+**Files referenced:** `vendor-categories.md §(h)`, `signals/logs.md §6`, `transport/collector-topology.md §2`, `observability-as-code.md §3`
+
+### Walkthrough
+
+1. Invoke `/oma-observability --migrate "Fluentd to Fluent Bit"`.
+2. Router routes to `vendor-categories.md §(h) Log Pipeline` → **Fluent Bit** (CNCF Graduated,
+   C runtime). Convert Fluentd syntax/plugins/routes and validate them; configuration is not drop-in compatible.
+
+**Migration phases (zero-downtime):**
+
+**Phase 1; Dual-send:** Deploy Fluent Bit DaemonSet alongside the existing Fluentd DaemonSet.
+Fluent Bit ships logs to the same backend; Fluentd continues as primary. Confirm both streams
+reach the backend without duplicates by comparing log record counts.
+
+```ini
+# Fluent Bit DaemonSet — core pipeline (fluent-bit.conf)
+[INPUT]
+    Name           tail
+    Path           /var/log/pods/*/*/*.log
+    Parser         cri
+    Tag            kube.*
+    Mem_Buf_Limit  5MB
+
+[FILTER]
+    Name        kubernetes
+    Match       kube.*
+    Merge_Log   On
+    Labels      On
+
+[OUTPUT]
+    Name      opentelemetry
+    Match     kube.*
+    Host      otel-gateway.observability.svc
+    Port      4318
+    Logs_uri  /v1/logs
+```
+
+**Phase 2; Parity verification:** query log record count per `service.name` in both pipelines
+over a 1-hour window. Expect < 0.1 % divergence accounting for timing windows.
+
+```promql
+# Fluent Bit throughput
+sum by (service_name) (rate(fluentbit_output_proc_records_total[1h]))
+
+# Fluentd throughput (comparison)
+sum by (service_name) (rate(fluentd_output_emit_records_total[1h]))
+```
+
+**Phase 3; Decommission Fluentd:** after the agreed parity window, remove the
+Fluentd DaemonSet from its GitOps/Helm desired state and reconcile through its
+owner. A DaemonSet has no replica count and cannot be scaled to zero. For an
+unmanaged DaemonSet, deletion after verified cutover is the removal action:
+
+```bash
+kubectl delete daemonset fluentd -n logging
+```
+
+3. Commit Fluent Bit ConfigMap and DaemonSet manifests to git; apply via Argo CD per
+   `observability-as-code.md §7`.
+4. Validate with `otelcol validate` on the gateway config; run `promtool check rules` on any
+   log-based alerting rules.
+
+**Outcome:** Node-level log collection RAM drops from 400 MB (Fluentd/Ruby) to ~80 MB (Fluent Bit/C),
+a 5× reduction typical per the CNCF 2025-10 guide. CPU overhead at steady state also reduces by
+~60 %. No log gaps observed during the dual-send window.
+
+---
+
+## Scenario 5: SLO with Burn-Rate Alerts (GitOps End-to-End)
+
+**Situation:** Platform team adopts the SLO practice for the checkout service. Target: 99.9 %
+availability on a 28-day rolling window. Everything must be versioned in git and applied
+via Argo CD; no manual UI edits.
+
+**Intent:** `alert`
+
+**Files referenced:** `boundaries/slo.md §5, §7, §8`, `observability-as-code.md §4, §6, §7`, `boundaries/release.md`
+
+### Walkthrough
+
+1. Define the SLO as OpenSLO YAML and commit to the observability repo (`boundaries/slo.md §5`):
+
+```yaml
+apiVersion: openslo/v1
+kind: SLO
+metadata:
+  name: checkout-availability
+spec:
+  service: checkout
+  budgetingMethod: Occurrences
+  timeWindow:
+    - duration: 28d
+      isRolling: true
+  objectives:
+    - target: 0.999
+  indicator:
+    metadata:
+      name: checkout-success-ratio
+    spec:
+      ratioMetric:
+        counter: false # queries return window counts, not monotonic counters
+        good:
+          metricSource:
+            type: Prometheus
+            spec:
+              query: sum(increase(http_requests_total{service="checkout",status=~"2..|3.."}[28d]))
+        total:
+          metricSource:
+            type: Prometheus
+            spec:
+              query: sum(increase(http_requests_total{service="checkout"}[28d]))
+```
+
+The v1 structure follows <https://openslo.com/specification/>.
+`metricSource.spec` is implementation-defined: verify the chosen consumer's
+Prometheus adapter, window evaluation, and counter semantics. This example
+does not imply Sloth or Pyrra directly accepts OpenSLO; native definitions
+require a verified adapter/conversion if OpenSLO is the input.
+
+2. Define the equivalent SLO in Sloth native YAML (`observability-as-code.md §4.2`) and generate its PrometheusRule. Do not pass the OpenSLO document directly unless a verified adapter converts it:
+
+```bash
+sloth generate -i sloth/checkout-slo.yaml | kubectl apply -f -
+```
+
+3. The generated `PrometheusRule` includes multi-window burn-rate alerts from `boundaries/slo.md §7`
+   and `observability-as-code.md §6`:
+
+```yaml
+# Fast burn — 2 % budget in 1 h (14.4× rate), gated by 5 m short window → page
+- alert: CheckoutSLOBurnRateFast
+  expr: |
+    (sum(rate(http_requests_total{service="checkout",status=~"5.."}[1h]))
+      / sum(rate(http_requests_total{service="checkout"}[1h]))) / (1 - 0.999) > 14.4
+    and
+    (sum(rate(http_requests_total{service="checkout",status=~"5.."}[5m]))
+      / sum(rate(http_requests_total{service="checkout"}[5m]))) / (1 - 0.999) > 14.4
+  for: 2m
+  labels: { severity: critical, slo: checkout-availability }
+  annotations:
+    summary: "Checkout SLO fast burn: budget exhausting in < 1h"
+
+# Slow burn — 5 % budget in 6 h (6× rate), gated by 30 m → ticket
+# Full PromQL in observability-as-code.md §6 and boundaries/slo.md §7.4
+```
+
+4. GitOps commit flow (`observability-as-code.md §7`):
+
+```
+git PR (checkout-slo.yaml + PrometheusRule)
+  → CI: promtool check rules alerts/checkout-slo-burn-rate.yaml
+  → CI: promtool test rules alerts/checkout_slo_test.yaml
+  → peer review (SRE CODEOWNERS approval)
+  → merge to main
+  → Argo CD sync → Prometheus Operator applies PrometheusRule to cluster
+```
+
+5. Validate via synthetic error injection:
+
+```bash
+# Inject 20 % error rate for 3 minutes to trigger fast-burn alert
+kubectl run error-injector --image=curlimages/curl --restart=Never -- \
+  sh -c 'for i in $(seq 1 200); do curl -s -o /dev/null -w "%{http_code}" \
+    http://checkout-api/error-inject; done'
+```
+
+Fast-burn alert fires in < 2 minutes.
+
+6. Error budget policy (`boundaries/slo.md §8`):
+
+| Budget remaining | Action |
+|-----------------|--------|
+| > 50 % | Normal feature velocity |
+| 25–50 % | Review reliability vs feature ratio |
+| < 25 % | Reliability sprint |
+| 0 % exhausted | Deploy freeze until budget recovers |
+
+When budget reaches 0 %, Flagger canary promotion gates use the same SLI error rate to block
+new deploys (`boundaries/release.md`).
+
+---
+
+## Appendix: Vendor Query Cross-Reference
+
+From `incident-forensics.md §4`; trace lookup by `trace_id` across backends:
+
+| Vendor | Query syntax |
+|--------|-------------|
+| Grafana Tempo | `{ trace:id = "4bf92f3577b34da6a3ce929d0e0e4736" }` |
+| Honeycomb | `trace:4bf92f3577b34da6a3ce929d0e0e4736` |
+| Datadog APM | `@trace_id:4bf92f3577b34da6a3ce929d0e0e4736` |
+| Jaeger HTTP API | `GET /api/traces/4bf92f3577b34da6a3ce929d0e0e4736` |
+| Elastic APM | `trace.id: "4bf92f3577b34da6a3ce929d0e0e4736"` |
+| Sentry | `trace_id:4bf92f3577b34da6a3ce929d0e0e4736` |
+
+---
+
+Code and YAML snippets are illustrative excerpts. Use the referenced resources for full configuration.