oh-my-agent-commandcode 0.0.0-stage → 20.38.3

package/skills/oma-observability/resources/signals/audit.md

@@ -0,0 +1,310 @@
+---
+otel_semconv: "1.43.0 (2026-07); security.* namespace: Development/proposed"
+tools:
+  - "Falco: v0.38+ (CNCF Graduated); rekor: v1.x (sigstore)"
+---
+
+# Audit Signal Reference
+
+## 1. Scope & Distinction
+
+Audit trails answer **who did what when**; immutable evidence for compliance, legal hold, and forensic investigation. They are a distinct signal from both operational logs (`logs.md`) and privacy records (`privacy.md`).
+
+### Integrity and privacy requirements
+
+| Dimension        | Audit                                  | Privacy                                |
+|------------------|----------------------------------------|----------------------------------------|
+| Retention goal   | Preserve required evidence for the applicable schedule | Minimize collection and assess erasure grounds/exceptions |
+| Storage model    | Tamper protection; WORM when required by the selected control | Controls that honor eligible erasure and justified retention |
+| Consumers        | Auditors, legal, compliance officers   | Data subjects, DPO, engineering |
+| Default posture  | Collect necessary evidence; expire under a justified schedule | Collect only justified personal data |
+
+Audit and privacy are separate concerns, but both apply to personal data in an audit trail. Specify the legal basis, required evidence, access controls, retention period, and erasure exceptions together.
+
+### Distinction from operational logs (`logs.md`)
+
+| Dimension       | Operational logs                    | Audit events                           |
+|-----------------|-------------------------------------|----------------------------------------|
+| Retention       | Purpose-specific operational schedule | Law/contract/evidence-specific schedule (Section 7) |
+| Consumers       | On-call engineers, SRE              | Auditors, legal, compliance            |
+| Mutability      | May be rotated and purged           | Protected against unauthorized alteration; WORM when required |
+| Storage         | Loki / Elasticsearch / ClickHouse   | Access-controlled evidence store with selected integrity/retention controls |
+| Primary tools   | Fluent Bit, OTel Collector          | Falco, auditd, pgaudit, audit pipeline |
+
+Cross-ref `../meta-observability.md §Retention Matrix` for unified policy across all signals.
+
+---
+
+## 2. Regulatory Drivers
+
+| Framework | Relevant controls | Audit requirement |
+|-----------|-------------------|-------------------|
+| **SOC 2 (Type I/II)** | CC7.2 monitoring activities; CC7.3 incident response | Evidence supporting the selected monitoring/response controls; scope and protection depend on the engagement |
+| **ISO/IEC 27001:2022** | A.8.15 logging; A.8.16 monitoring | Log protection, log administrator access control |
+| **ISO/IEC 27002:2022** | 8.15 logging controls | Protect logs from tampering and unauthorized access |
+| **HIPAA Security Rule** | §164.312(b) audit controls; §164.316(b) documentation | Audit mechanisms for ePHI; retain required security documentation for 6 years, not a blanket 6-year rule for every raw log |
+| **PCI DSS v4.0** | Requirement 10; track and monitor all access to cardholder data | 1 year online + offline retention; tamper detection |
+| **GDPR Art. 30** | Records of processing activities | Audit of data processing operations |
+
+Cross-ref `../standards.md §ISO/IEC 27001/27002` for the normative standards baseline.
+
+Sources: [iso.org/standard/27001](https://www.iso.org/standard/27001) · [aicpa.org SOC 2](https://www.aicpa.org/soc2) · [hhs.gov HIPAA](https://www.hhs.gov/hipaa/for-professionals/security/index.html) · [pcisecuritystandards.org PCI DSS v4.0](https://www.pcisecuritystandards.org)
+
+---
+
+## 3. Audit Event Categories
+
+| Category | Examples | Regulatory driver |
+|----------|----------|-------------------|
+| **Authentication** | Login success/failure, MFA challenge, password change, session create/destroy | SOC 2 CC7.2, ISO A.8.15 |
+| **Authorization** | Permission grants, role changes, scope escalation, RBAC mutations | SOC 2 CC7.2, PCI DSS 10.2 |
+| **Data access** | Read/create/update/delete of regulated data (PII, CHD, PHI) | HIPAA §164.312(b), PCI DSS 10.2 |
+| **Administrative** | Config changes, user provisioning, API key lifecycle, certificate rotation | ISO A.5.25, SOC 2 CC7.3 |
+| **Security events** | IDS/IPS alerts, anomalous kernel syscalls (Falco), policy violations | SOC 2 CC7.3, ISO A.8.16 |
+| **System events** | Kernel audit (Linux auditd), container lifecycle events | ISO A.8.15; cross-ref `logs.md §OS-level log sources` |
+
+---
+
+## 4. Required Attributes per Audit Event
+
+Every audit event MUST carry these fields. Map to OTel `security.*` semconv namespace (Development/proposed).
+
+| Attribute | Type | Description |
+|-----------|------|-------------|
+| `user.id` | string | Authenticated identity; pseudonymized or hashed; never plain email (PII) |
+| `actor.type` | enum | `user` \| `service_account` \| `system` |
+| `action` | string | Verb: `read`, `write`, `delete`, `approve`, `login`, `logout` |
+| `resource.type` | string | What was acted on: `order`, `user_record`, `api_key`, `rbac_role` |
+| `resource.id` | string | Identifier of the affected resource |
+| `event.outcome` | enum | `success` \| `failure` \| `denied` |
+| `ip.address` | string | Source IP; may be PII; cross-ref `privacy.md §IP addresses` |
+| `timestamp` | ISO 8601 UTC | Event time; required for chain ordering (Section 6) |
+| `trace_id` | 32-char hex | Correlation with operational trace; cross-ref `../incident-forensics.md` |
+
+```json
+{
+  "timestamp": "2026-04-21T09:15:32.847Z",
+  "actor.type": "user",
+  "user.id": "sha256:a3f8c2...",
+  "action": "delete",
+  "resource.type": "user_record",
+  "resource.id": "rec_9f2k1",
+  "event.outcome": "success",
+  "ip.address": "203.0.113.42",
+  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
+  "service.name": "user-management-api",
+  "deployment.environment": "production"
+}
+```
+
+---
+
+## 5. Immutable WORM Storage (Write-Once-Read-Many)
+
+Select tamper protection and retention controls for the applicable obligation. These frameworks do not establish a universal WORM requirement for every audit log. WORM can implement a selected immutability requirement; document why it is needed before locking records.
+
+| Platform | Mechanism | Key constraint |
+|----------|-----------|----------------|
+| **AWS S3** | Object Lock; Compliance mode | Cannot delete even by root or AWS support |
+| **GCS** | Retention Policy with locked bucket | Lock is irreversible once applied |
+| **Azure Blob** | Immutability policy (time-based retention) | Legal hold override available |
+| **On-premises** | WORM tape, compliance appliances | Hardware write-protect; chain of custody required |
+
+S3 Object Lock Compliance mode example for a record class whose approved retention is 7 years (not a default for all audit data):
+
+```json
+{
+  "ObjectLockConfiguration": {
+    "ObjectLockEnabled": "Enabled",
+    "Rule": {
+      "DefaultRetention": {
+        "Mode": "COMPLIANCE",
+        "Years": 7
+      }
+    }
+  }
+}
+```
+
+`COMPLIANCE` mode prevents deletion by any principal, including the AWS account root, during the retention period. Use it only when that irreversible behavior matches the approved requirement. `GOVERNANCE` allows privileged override and may suit a different control; choose the mode explicitly.
+
+---
+
+## 6. Tamper Evidence (Cryptographic Integrity)
+
+WORM prevents deletion; tamper evidence proves no records were silently modified or omitted.
+
+**Hash chain:** each event record includes `prev_hash`; the SHA-256 of the previous event. Any insertion, deletion, or modification breaks the chain.
+
+```
+event_N.hash = SHA256(event_N.payload + event_{N-1}.hash)
+```
+
+**Merkle root anchoring:** daily, compute the Merkle root of all events in the period. Anchor this root to an external transparency log; detached from the audit system; so the anchoring timestamp is independently verifiable.
+
+| Tool | Type | Notes |
+|------|------|-------|
+| **rekor** (sigstore) | OSS transparency log | CNCF project; append-only; publicly verifiable |
+| **AWS QLDB** | Managed ledger DB | Cryptographic journal; AWS-proprietary |
+| **IBM LinuxONE** | Hardware-backed ledger | Enterprise on-prem |
+
+Source: [github.com/sigstore/rekor](https://github.com/sigstore/rekor)
+
+**Verification schedule:** run chain integrity check weekly (automated), and before any compliance audit. Verification failure is a CRITICAL security event; trigger incident response.
+
+---
+
+## 7. Evidence-Specific Retention Policy
+
+Map each record class to its legal/contractual requirement and purpose. HIPAA required security documentation, PCI audit history, and SOC 2 engagement evidence have different scopes; one period does not prove all compliance. Set an approved retention schedule and erasure/legal-hold exceptions before configuring lifecycle or Object Lock. The table below is a 7-year example only for a record class that needs that period.
+
+| Tier | Duration | Storage | Cost tier |
+|------|----------|---------|-----------|
+| Hot (recent, searchable) | 90 days | Elasticsearch / Loki | $$$ |
+| Warm (searchable, compressed) | 1 year | ClickHouse / archive index | $$ |
+| Cold (WORM, immutable) | 7 years | S3 Glacier Deep Archive (Object Lock Compliance) | $ |
+
+**Legal hold:** when an active investigation or litigation hold is active, suppress automated expiry for all affected records regardless of tier. Implement via S3 Object Lock legal hold or equivalent.
+
+**Lifecycle policy:** apply S3 Lifecycle rules to transition objects automatically. Compliance mode Object Lock must be set at write time; it cannot be applied retroactively.
+
+---
+
+## 8. Falco Integration (CNCF Graduated)
+
+Falco provides runtime security detection: anomalous kernel syscalls, container escapes, file access violations, and Kubernetes API abuse.
+
+Source: [falco.org](https://falco.org)
+
+**Deployment:** Falco DaemonSet on every node. Output JSON events → audit pipeline.
+
+Custom Falco rule example (detect secret access in production namespace):
+
+```yaml
+- rule: K8s Secret Access in Production
+  desc: Detect any read of a Kubernetes secret in the production namespace
+  condition: >
+    ka.verb=get and ka.target.resource=secrets
+    and ka.target.namespace=production
+    and not ka.user.name startswith "system:"
+  output: >
+    Secret accessed (user=%ka.user.name secret=%ka.target.name
+    ns=%ka.target.namespace ua=%ka.user-agent)
+  priority: WARNING
+  source: k8saudit
+  tags: [audit, pci_dss, hipaa]
+```
+
+**Pipeline integration:** Falco JSON output → `filelogreceiver` (OTel Collector) → audit logs pipeline → WORM cold tier. Apply the same hash chain enrichment as application audit events before writing.
+
+---
+
+## 9. Kubernetes Audit Logs
+
+Kubernetes API server audit logs record every API call; essential for cluster-admin accountability, secret access auditing, and RBAC change tracking.
+
+| Audit level | Records | Use case |
+|-------------|---------|----------|
+| `None` | Nothing | Exclude noisy, low-value paths |
+| `Metadata` | Method, URL, user, timestamp | Default for most resources |
+| `Request` | + request body | Selected non-secret mutations (e.g., RBAC), after assessing sensitive fields |
+| `RequestResponse` | + response body | Selected non-secret operations whose bodies are necessary and safe to retain |
+
+Put the Secret metadata-only exception before broader body-recording rules; Kubernetes audit policies use the first matching rule. Do not record Secret `data`/`stringData` or credential response bodies in the audit archive.
+
+```yaml
+apiVersion: audit.k8s.io/v1
+kind: Policy
+rules:
+  - level: Metadata
+    resources:
+      - group: ""
+        resources: ["secrets"]
+  # Add narrowly scoped, reviewed non-secret rules here.
+  - level: Metadata
+```
+
+Feed Kubernetes audit logs into the audit pipeline separately from operational logs. Tag with `source: k8s_apiserver` for routing. Source: <https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/>.
+
+---
+
+## 10. Database Audit Logs
+
+| Database | Extension / feature | Notes |
+|----------|---------------------|-------|
+| PostgreSQL | `pgaudit` extension | Log SELECT, DDL, DML per role |
+| MySQL | Enterprise Audit plugin | JSON output; filter by user/schema |
+| MongoDB | Audit system (Enterprise) | Filter by action type and collection |
+| AWS RDS | Built-in audit logging | Enable via parameter group |
+| GCP Cloud SQL | `cloudsql.enable_pgaudit` | Same pgaudit under the hood |
+| Azure SQL | Unified Audit Log | Native, sends to Storage/Log Analytics |
+
+Database audit events feed the same audit pipeline as application events. Normalize to the attribute schema in Section 4 before storage.
+
+---
+
+## 11. Access Control on Audit Data
+
+Audit data must be readable by auditors and legal; and not writable by anyone after initial ingest.
+
+- Separate RBAC role: `audit-reader` is distinct from `operations-engineer` and `developer`
+- Auditors have read-only access, scoped to approved time ranges
+- No single engineer can both produce audit events and modify audit storage
+- Cross-ref `privacy.md §Backend RBAC` for principle of least privilege patterns
+
+---
+
+## 12. Cross-Signal Correlation
+
+Every audit event carries `trace_id`. During an incident:
+
+1. Start from the audit event (who did what)
+2. Pivot via `trace_id` to the operational trace (what the system did)
+3. Join with logs (`logs.md §Trace ID Injection Rules`) for execution detail
+4. Reference `../incident-forensics.md` for the full MRA playbook
+
+Audit events are the authoritative source of truth for incident timelines; operational traces provide the execution context.
+
+---
+
+## 13. Matrix Coverage: Audit Column
+
+Cells from `../matrix.md` owned by this file:
+
+| Layer | Boundary | Status | Notes |
+|-------|----------|--------|-------|
+| L3-network | any | PASS | VPC flow audit trail; who connected to what |
+| L4-transport | any | PARTIAL | Limited: connection-level only, no payload |
+| L7-application | multi-tenant | PASS | Per-tenant audit trail; supports GDPR data subject right of access |
+| release | any | PASS | Deployment audit: who deployed what when (actor, SHA, timestamp) |
+| privacy | audit | overlap | See `privacy.md`; audit records data processing; privacy governs erasure |
+
+---
+
+## 14. Anti-patterns
+
+Append candidates for `../anti-patterns.md §Section F Security & Compliance`:
+
+| ID | Anti-pattern | Fix |
+|----|--------------|-----|
+| A-AU1 | Audit evidence can be altered without detection | Apply access controls and tamper protection; choose WORM only when the approved obligation requires it |
+| A-AU2 | No tamper evidence (no hash chain or notary) | Implement hash chain per Section 6; anchor Merkle root to rekor |
+| A-AU3 | PII in audit events without redaction | Hash or pseudonymize `user.id`; evaluate `ip.address` per `privacy.md` |
+| A-AU4 | Shared RBAC for operations and audit readers | Create separate `audit-reader` role; enforce in IaC |
+| A-AU5 | Retention differs from the approved record-specific schedule | Apply the documented law/contract/purpose schedule and legal holds; expire eligible records |
+| A-AU6 | Kubernetes audit logs routed to operational log store | Separate pipeline: K8s audit → WORM cold tier |
+
+---
+
+## 15. References
+
+1. ISO/IEC 27001:2022; <https://www.iso.org/standard/27001>
+2. ISO/IEC 27002:2022; <https://www.iso.org/standard/27002>
+3. AICPA SOC 2 Common Criteria: <https://www.aicpa.org/soc2>
+4. HIPAA Security Rule §164.312(b): <https://www.hhs.gov/hipaa/for-professionals/security/index.html>
+5. PCI DSS v4.0 Requirement 10: <https://www.pcisecuritystandards.org>
+6. GDPR Art. 30: <https://gdpr-info.eu/art-30-gdpr/>
+7. Falco: <https://falco.org>
+8. sigstore rekor: <https://github.com/sigstore/rekor>
+9. OTel security semconv (proposed): <https://opentelemetry.io/docs/specs/semconv/attributes-registry/security/>
+10. `../standards.md` · `../matrix.md` · `../meta-observability.md` · `../incident-forensics.md` · `privacy.md` · `logs.md`

package/skills/oma-observability/resources/signals/cost.md

@@ -0,0 +1,303 @@
+---
+otel_spec: "1.x (stable API/SDK)"
+otel_semconv: "1.43.0 (2026-07)"
+specs:
+  - "FOCUS Spec: 1.0 (FinOps Open Cost and Usage Specification)"
+notes:
+  - "OpenCost: CNCF Incubating (advanced 2024-10-31)"
+---
+
+# Cost Signal
+
+## 1. Scope & Boundary
+
+Cost is the 6th signal in the `oma-observability` skill (design decision D4: cost as 1st-class signal, standalone). It sits alongside metrics, logs, traces, profiles, audit, and privacy in the 4 × 4 × 7 coverage matrix.
+
+**In scope:**
+- Cost as a telemetry signal: metric surface (OpenCost Prometheus exposition), attribution rules, and unit economics
+- Kubernetes-native cost allocation via OpenCost (CNCF Incubating)
+- Per-tenant cost attribution (B2B SaaS) and per-feature cost deltas
+- LLM/AI token-cost observability at the span attribute level
+- Cross-cloud egress cost attribution
+- Retention policy and access-control rules for cost data
+- Cardinality rules specific to cost labels
+
+**Out of scope:**
+- Full FinOps strategy: procurement, Reserved Instances / Savings Plans, commitment analysis, contract negotiation; see [FinOps Foundation framework](https://www.finops.org/framework/)
+- Full LLM observability (prompt versioning, evals, model routing economics); see Langfuse, Arize Phoenix, LangSmith
+- Cluster cost optimization (rightsizing, spot orchestration): see Kubecost commercial tier or CloudZero
+
+**Distinct boundary with `../boundaries/multi-tenant.md §Cost Attribution`:** that file uses cost as a _routing signal_ (tenant chargeback, showback, residency). This file defines cost as a _telemetry signal_; what to collect, how to attribute it, and how to store it. Cross-reference is mandatory.
+
+---
+
+## 2. Why Cost Is a 1st-Class Signal
+
+Cost differs from metrics, logs, and traces in three structural ways:
+
+| Dimension | Metrics / Logs / Traces | Cost |
+|-----------|------------------------|------|
+| Data source | Instrumented applications + infra agents | Cloud billing APIs + Kubernetes allocation engine |
+| Primary consumer | Engineering, SRE | Finance, Platform Engineering |
+| Retention requirement | 30–90 days typical | 2 years (financial audit compliance) |
+| Granularity | Per-request, sub-second | Per-hour or per-day billing aggregates |
+
+Unit economics is the motivation: understanding `$ per request`, `$ per tenant`, and `$ per feature` requires correlating cloud billing data with application-level telemetry. Neither data source alone is sufficient.
+
+FinOps Foundation identifies cost visibility as a foundational practice across three phases: Inform → Optimize → Operate. The Inform phase is entirely an observability problem; cost data must be collected, attributed, and surfaced before optimization can occur.
+
+---
+
+## 3. OpenCost (CNCF Incubating)
+
+Sources:
+- <https://www.opencost.io/>
+- <https://www.cncf.io/blog/2024/10/31/opencost-advances-to-cncf-incubator/>
+- <https://github.com/opencost/opencost>
+
+OpenCost is the open-source Kubernetes cost monitoring standard. It advanced to CNCF Incubating status on 2024-10-31. The OSS core is maintained by IBM Kubecost (post-acquisition) and the community. Kubecost's commercial tier adds UI, recommendations, and multi-cluster federation on top of the OpenCost core.
+
+### 3.1 Architecture
+
+```
+Cloud Billing API (AWS CUR / GCP Billing Export / Azure Cost Management)
+        |
+        v
++----------------+        /metrics (Prometheus format)
+| OpenCost Pod   | -----> Prometheus scrape -----> OTel Collector prometheusreceiver
+| (namespace:    |                                          |
+|  opencost)     |                                          v
++----------------+                               Your TSDB backend
+        |                                        (Thanos / Mimir / VictoriaMetrics)
+        v
+Kubernetes API (node prices, pod labels, PVC claims)
+```
+
+The OpenCost pod reads node price data from a cloud-provider price configmap (or the cloud billing API directly), reads Kubernetes resource allocations from the kubelet and API server, and exposes Prometheus metrics at port 9003.
+
+### 3.2 Key Metrics Exposed
+
+Source: <https://opencost.io/docs/integrations/metrics/>. OpenCost exposes allocation quantities and unit/hourly prices; it does not automatically emit namespace/workload cost counters.
+
+| Metric | Unit | Purpose |
+|--------|------|---------|
+| `node_cpu_hourly_cost` | USD / core-hour | Node CPU unit price |
+| `node_ram_hourly_cost` | USD / GiB-hour | Node memory unit price |
+| `node_total_hourly_cost` | USD / hour | Total node rate |
+| `container_cpu_allocation` | cores | Allocated container CPU |
+| `container_memory_allocation_bytes` | bytes | Allocated container memory |
+| `pod_pvc_allocation` | bytes | Allocated PVC storage |
+| `pv_hourly_cost` | USD / GiB-hour | Persistent-volume unit price |
+| `kubecost_network_internet_egress_cost` | USD / GiB | Internet egress price; multiply by attributed usage |
+| `kubecost_load_balancer_cost` | USD / hour | Load-balancer rate |
+| `kubecost_cluster_management_cost` | USD / hour | Cluster management rate |
+
+### 3.3 Scrape Configuration (OTel Collector)
+
+```yaml
+receivers:
+  prometheus:
+    config:
+      scrape_configs:
+        - job_name: opencost
+          scrape_interval: 60s
+          static_configs:
+            - targets: ["opencost.opencost.svc.cluster.local:9003"]
+```
+
+### 3.4 Allocation Model
+
+Three allocation categories: compute (CPU + RAM), storage (PVC), and network (egress bytes). Each workload receives a share of node cost proportional to resource requests. Idle cost distributes across active workloads or a dedicated `__idle__` namespace via configurable rules. Cross-cloud pricing is supplied via a price configmap; on-prem environments set custom `$/CPU-hr` and `$/GB-RAM-hr` values.
+
+---
+
+## 4. Cost Attribution by Dimension
+
+| Dimension | Kubernetes mechanism | Label / attribute |
+|-----------|---------------------|-------------------|
+| Namespace / team | Namespace label → team ownership mapping | `namespace`, `team` |
+| Workload | Deployment / StatefulSet / DaemonSet name | `workload`, `controller_kind` |
+| Tenant (B2B SaaS) | `tenant.id` label on pods + W3C Baggage propagation | `tenant_id` |
+| Feature | Custom pod label (`feature.name`) | `feature_name` |
+| Per-request (LLM) | `gen_ai.cost.total_usd` span attribute | trace attribute, not metric label |
+
+**Tenant attribution** is the primary B2B SaaS use case: tag every pod with `tenant.id` at deploy time, then request the appropriate label aggregation from the OpenCost Allocation API or explicitly join allocation metrics to verified ownership metadata. Pod labels alone do not create a new cost counter at `/metrics`. Cross-ref `../boundaries/multi-tenant.md §Cost Attribution` for chargeback and showback patterns. **Feature attribution** uses a custom pod label (`feature.name`); cross-ref `../boundaries/release.md` for A/B cost comparison. **Per-request attribution** for LLM workloads uses span attributes only; writing cost to a metric label at request granularity causes a cardinality explosion (see Section 10).
+
+---
+
+## 5. Unit Economics
+
+Compute costs with matching units and attribution scope. The default OpenCost metrics support this namespace compute-cost rate (USD/hour):
+
+```promql
+sum by (namespace) (
+  container_cpu_allocation
+    * on(node) group_left node_cpu_hourly_cost
+  + container_memory_allocation_bytes
+    * on(node) group_left node_ram_hourly_cost / (1024 * 1024 * 1024)
+)
+```
+
+This covers CPU and memory only. Add PVC, attributed network usage, load-balancer, shared/idle, and management costs according to the allocation model before treating the result as a total bill. For tenant/workload daily totals, use the OpenCost Allocation API's window and aggregation controls rather than assuming a workload counter exists. Source: <https://opencost.io/docs/integrations/api/>.
+
+**Per-request cost (custom counter example, rolling 1-hour window):**
+
+The following query requires a separately implemented monotonic `workload_cost_usd_total` counter in USD and an ownership mapping to the same workload. It is a team-defined export, not an OpenCost default metric. Verify that costs and request counts cover the same scope and time window.
+
+```promql
+sum(rate(workload_cost_usd_total{workload="checkout-api"}[1h]))
+  /
+sum(rate(http_requests_total{job="checkout-api"}[1h]))
+```
+
+Both rates are per second, so the result is USD/request; do not multiply only one side by 3600. An hourly cost gauge must instead be divided by requests/hour.
+
+For gold/silver SLO cost comparisons, compare their allocated costs over the same window and attribution scope, and report the corresponding SLI measurements. Do not compare lifetime cumulative counters as a current tier-cost ratio.
+
+---
+
+## 6. LLM / AI Cost Observability
+
+Full LLM observability is out of scope for this skill. The tools for that domain are Langfuse, Arize Phoenix, LangSmith, and Braintrust (see `SKILL.md §When NOT to use`). This section covers only the intersection where LLM cost surfaces as a span attribute in the OTel pipeline.
+
+**Token-based pricing attributes** (OTel `gen_ai.*` semconv — deprecated in the core registry since semconv 1.42 and maintained in the dedicated GenAI semconv repository; treat as unstable per `../standards.md §Semconv Stability`):
+
+| Span attribute | Description |
+|----------------|-------------|
+| `gen_ai.usage.input_tokens` | Prompt tokens consumed |
+| `gen_ai.usage.output_tokens` | Completion tokens generated |
+| `gen_ai.cost.total_usd` | Computed cost in USD for the span (vendor-specific extension) |
+
+**Threshold-based sampling rule:** if `gen_ai.cost.total_usd > 0.50`, always retain the span regardless of sampling rate. This prevents silent budget overruns from being dropped before they reach the backend. Cross-ref `../transport/sampling-recipes.md §Cost-aware sampling` for the tail-sampler rule configuration.
+
+Note: `gen_ai.*` conventions moved to a dedicated GenAI semconv repository in semconv 1.42 (2026-06); the core-registry entries are deprecated. Do not use these attributes as inputs to production SLOs until they reach Stable or RC in their new home.
+
+---
+
+## 7. Cross-Cloud Cost
+
+Each cloud exposes billing data through a distinct API and export format. FOCUS (FinOps Open Cost and Usage Specification) is the data-format unification effort.
+
+| Cloud | Billing data source | Export format | Ingestion path |
+|-------|--------------------|----|------|
+| AWS | Cost & Usage Report (CUR) | Parquet / CSV → S3 | Athena query → OpenCost price configmap |
+| GCP | Billing Export | BigQuery table | BigQuery export job → OpenCost |
+| Azure | Cost Management Export | CSV → Blob Storage | Storage export → OpenCost |
+| On-prem | Manual price configmap | Custom `$/CPU-hr`, `$/GB-hr` | OpenCost configmap |
+
+**FOCUS Spec** (<https://github.com/FinOps-Open-Cost-and-Usage-Spec/FOCUS_Spec>) provides a vendor-neutral column schema (`BilledCost`, `ResourceId`, `ServiceName`) for cross-cloud cost joins. When FOCUS exports are available from a cloud provider, prefer them over native formats. For the OSS multi-cluster path, run one OpenCost instance per cluster and aggregate in Thanos or Mimir by namespace and workload labels.
+
+---
+
+## 8. Retention Policy
+
+| Data type | Retention | Rationale |
+|-----------|-----------|-----------|
+| Raw cloud billing data (CUR / GCP export) | 2 years | Financial audit compliance (SOX, ISO 27001) |
+| OpenCost Prometheus metrics (hourly resolution) | 2 years | Chargeback evidence per tenant |
+| Aggregated per-tenant / per-feature cost | Long-term (cold storage) | Historical unit economics trending |
+| Per-request span cost attributes (`gen_ai.cost.total_usd`) | 30–90 days | Operational debugging window |
+
+Cross-ref `../meta-observability.md §Retention Matrix` for the full retention policy table covering all seven signals.
+
+---
+
+## 9. Privacy & Access Control
+
+Cost per tenant reveals revenue tier, contract value, and resource consumption patterns. Restrict access by role:
+
+**RBAC separation:**
+
+| Role | Dashboard access | Data access |
+|------|-----------------|-------------|
+| Finance | Full cost by tenant, contract | Billing export (read-only) |
+| Platform Engineering | Cost by namespace, workload, cluster | OpenCost metrics (read-only) |
+| Application Engineering | Cost by their own service / namespace | Filtered by namespace label |
+| Tenant admin (self-serve) | Their own tenant cost only | Filtered by `tenant_id` claim |
+
+Cross-ref `privacy.md §Backend RBAC` for the Grafana RBAC configuration pattern and OPA policy rules for dashboard-level tenant isolation.
+
+---
+
+## 10. Anti-Cardinality Rules for Cost Labels
+
+Follow the same principles as metric labels (cross-ref `../meta-observability.md §Cardinality Guardrails`). Cost workloads add two additional risk surfaces: per-request LLM spans and dynamic tenant growth.
+
+| Rule | Applies to | Rationale |
+|------|-----------|-----------|
+| `tenant.id` allowed as metric label with top-N cap (≤ 1000) | OpenCost metrics | Bounded tenant count; use `"other"` bucket for overflow |
+| `namespace` and `workload` always allowed | OpenCost metrics | Bounded by cluster size |
+| `gen_ai.cost.total_usd` as span attribute only; never as metric label | LLM spans | One series per request = cardinality explosion |
+| `feature.name` allowed as pod label; allowed as metric label with cap | OpenCost workload attribution | Feature set is bounded; gate new features through label allowlist |
+| `request.id` never as metric label for cost | Any | Unbounded cardinality; use trace attribute only |
+
+Per-request cost labels are safe only as trace span attributes, where cardinality is handled by the trace backend (not a TSDB). Aggregate by `tenant.id`, `namespace`, and `workload` for metric surfaces.
+
+---
+
+## 11. Vendors
+
+As of 2026-Q2. Verify current status at <https://landscape.cncf.io>.
+
+Cross-ref `../vendor-categories.md §FinOps / Cost` for full selection guidance.
+
+| Vendor | Type | Notes |
+|--------|------|-------|
+| OpenCost | OSS, CNCF Incubating | Kubernetes-native; Prometheus exposition; community-maintained |
+| Kubecost | Commercial (IBM) | OpenCost OSS core + commercial UI, multi-cluster, recommendations |
+| CloudZero | Commercial SaaS | Engineering-focused cost attribution; no Kubernetes agent required |
+| Vantage | Commercial SaaS | Cross-cloud cost reporting; FOCUS spec early adopter |
+
+---
+
+## 12. Matrix Cells: Cost Column
+
+Quick navigation for cost-column cells in `../matrix.md`:
+
+| Layer | Boundary | Status | Detail |
+|-------|----------|--------|--------|
+| L3-network | multi-tenant | PARTIAL | Egress byte attribution by tenant VPC; cost proxy, not unit economics |
+| L3-network | cross-application | PARTIAL | Cross-VPC egress cost attribution; unreliable without flow tagging |
+| L4-transport | multi-tenant | PARTIAL | L4 byte volume per tenant as cost proxy; rolls up into compute |
+| L4-transport | cross-application | PARTIAL | Cross-application byte volume; correlate with L7 for accuracy |
+| mesh | multi-tenant | PARTIAL | Request-level cost attribution by tenant via mesh telemetry; feeds OpenCost |
+| mesh | cross-application | PASS | Per-service byte/request counts from Envoy; unit cost cross-service |
+| mesh | release | PARTIAL | Canary cost delta via per-version request counts in mesh telemetry |
+| L7-application | multi-tenant | PASS | Primary use case: per-tenant OpenCost unit economics with `tenant.id` pod label |
+| L7-application | cross-application | PASS | Per-service unit cost model; `gen_ai.cost.total_usd` span attribute for LLM |
+| L7-application | slo | PASS | Gold-tier vs silver-tier cost delta; cost trade-off for SLO tier selection |
+| L7-application | release | PASS | Canary cost delta: compare per-request cost across canary vs stable version |
+
+L3 and L4 cost cells are PARTIAL because they produce a cost proxy (egress bytes) that informs FinOps egress billing but is insufficient for full unit-economics modeling. See `../matrix.md §C6` for the detailed caveat.
+
+---
+
+## 13. Anti-Patterns
+
+Candidates for `../anti-patterns.md §Section B Cardinality & Cost`:
+
+| Anti-pattern | Impact | Correction |
+|-------------|--------|------------|
+| Cost label at per-request metric granularity | Cardinality explosion in TSDB; OOM on ingestor | Use `gen_ai.cost.total_usd` as span attribute; aggregate cost metrics by tenant/namespace/workload |
+| No `tenant.id` pod label | Cannot attribute cost to tenants; chargeback impossible | Apply `tenant.id` at deploy time via Helm values or admission webhook |
+| Cost dashboard with public access | Reveals tenant revenue tier and contract value | Apply Grafana RBAC; finance and engineering views are separate; no public embedding |
+| Ignoring egress cost in FinOps | Often the largest surprise cost in multi-cloud or CDN-heavy architectures | Use the applicable egress unit-price metrics and attributed network usage or Allocation API costs; set egress budget alerts |
+| LLM spans not tail-sampled on cost threshold | Silent budget blowup: expensive spans are dropped before alerting | Configure tail-sampler rule: if `gen_ai.cost.total_usd > 0.50`, always retain |
+| One metric name per tenant for cost | Bypasses TSDB cardinality controls; cannot be aggregated | Use one documented cost metric/API aggregation with a bounded tenant dimension; do not invent automatic workload cost counters |
+
+---
+
+## Cross-References
+
+| Topic | File |
+|-------|------|
+| Tenant cost chargeback and routing | `../boundaries/multi-tenant.md §Cost Attribution` |
+| Canary cost delta and A/B comparison | `../boundaries/release.md` |
+| Tail-sampler cost-aware rules | `../transport/sampling-recipes.md §Cost-aware sampling` |
+| Cardinality guardrails and top-N cap | `../meta-observability.md §Cardinality Guardrails` |
+| Retention matrix (all signals) | `../meta-observability.md §Retention Matrix` |
+| Cost dashboard RBAC | `signals/privacy.md §Backend RBAC` |
+| Vendor selection (FinOps category) | `../vendor-categories.md §FinOps / Cost` |
+| Full 112-cell matrix with cost column | `../matrix.md` |
+| OpenCost metric surface (metrics signal) | `signals/metrics.md §8 OpenCost Metric Surface` |
+| LLM / gen_ai semconv stability | `../standards.md §3 OTel Semconv Stability Tiers` |