oh-my-agent-commandcode 0.0.0-stage → 20.38.3

package/skills/oma-observability/resources/signals/traces.md

@@ -0,0 +1,360 @@
+---
+otel_spec: "1.x (stable API/SDK)"
+otel_semconv: "1.43.0 (2026-07)"
+specs:
+  - "W3C Trace Context: Level 1 Recommendation 2020-02-06"
+---
+
+# Distributed Traces Signal
+
+## 1. Scope
+
+Traces are the "T" in MELT+P; the correlation backbone of distributed observability. A single trace links every span produced by every service that participated in one request, job, or message.
+
+Covers: OTel span data model, SpanKind, W3C Trace Context, DB patterns (N+1, slow query, pool exhaustion, `db.*` Stable), messaging patterns (Kafka, Flink, Spark, DLQ, `messaging.*`), RPC (`rpc.*` RC), sampling, backends, exemplars, baggage security.
+
+Cross-references:
+- `../boundaries/cross-application.md`: propagator matrix per cloud and mesh vendor
+- `../layers/mesh.md`: zero-code auto-instrumentation; propagator headers
+
+---
+
+## 2. OTel Span Data Model (Stable)
+
+Source: <https://opentelemetry.io/docs/specs/otel/trace/api/>
+
+Every span carries the following fields:
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `TraceId` | 16-byte hex | Globally unique identifier for the entire trace |
+| `SpanId` | 8-byte hex | Unique identifier for this span |
+| `ParentSpanId` | 8-byte hex | SpanId of the parent; absent for root spans |
+| `Name` | string | Operation name (e.g., `HTTP GET /api/orders`) |
+| `Kind` | enum | SpanKind; see table below |
+| `StartTime` | timestamp | Monotonic clock at span start |
+| `EndTime` | timestamp | Monotonic clock at span end |
+| `Attributes` | key-value map | Semantic convention attributes describing the operation |
+| `Events` | list | Timestamped structured log records within the span |
+| `Links` | list | References to other spans (used for async messaging) |
+| `Status` | enum + string | `UNSET`, `OK`, or `ERROR` with optional description |
+
+### SpanKind
+
+SpanKind describes the role of the span in the larger distributed operation. Backends use it to render topology maps and assign latency accountability.
+
+| SpanKind | Role | Typical origin |
+|----------|------|----------------|
+| `INTERNAL` | Default; in-process operation with no network boundary | Business logic, cache lookup, CPU computation |
+| `SERVER` | Incoming synchronous request handled by this service | HTTP server, gRPC server handler |
+| `CLIENT` | Outgoing synchronous call from this service to another | HTTP client, DB driver, gRPC stub |
+| `PRODUCER` | Async message sent to a queue or topic | Kafka producer, RabbitMQ publisher |
+| `CONSUMER` | Async message received from a queue or topic | Kafka consumer, worker pulling from queue |
+
+Rule of thumb: sync RPC produces a `CLIENT`+`SERVER` pair sharing the same `TraceId`. Async messaging produces a `PRODUCER`+`CONSUMER` pair connected by a span link (not parent-child), because the two operations run independently.
+
+---
+
+## 3. W3C Trace Context Propagation
+
+Source: <https://www.w3.org/TR/trace-context/>
+
+### Header format
+
+```
+traceparent: 00-<trace-id>-<span-id>-<flags>
+```
+
+- `00`: version (currently always `00`)
+- `<trace-id>`: 32 hex chars (128-bit TraceId)
+- `<span-id>`: 16 hex chars (64-bit SpanId of the sending span)
+- `<flags>`: 8-bit flags; `01` = sampled, `00` = not sampled
+
+Example:
+
+```
+traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
+tracestate:  vendor1=abc123,vendor2=xyz789
+```
+
+`tracestate` carries vendor-specific opaque state as a comma-separated `key=value` list forwarded unchanged by intermediaries that do not own the key.
+
+### Propagation rules
+
+- **Inject** `traceparent` on every outbound HTTP/gRPC call; **extract** at every inbound call before creating the root span.
+- Stripping `traceparent` silently is anti-pattern #1 in `../anti-patterns.md`.
+- Cross-cloud/mesh vendor header bridging: `../boundaries/cross-application.md §Propagators`.
+- Mesh-injected headers (Envoy, Linkerd): `../layers/mesh.md §Propagator headers`.
+
+---
+
+## 4. Database Tracing Patterns (db.* semconv Stable)
+
+Source: <https://opentelemetry.io/docs/specs/semconv/database/>
+
+All DB spans use `SpanKind = CLIENT`. The span name follows the pattern `<db.operation.name> <db.collection.name>` (e.g., `SELECT orders`).
+
+### Core db.* attributes
+
+| Attribute | Stability | Example | Notes |
+|-----------|-----------|---------|-------|
+| `db.system` | Stable | `postgresql`, `mysql`, `mongodb`, `redis` | Required; identifies the DB technology |
+| `db.operation.name` | Stable | `SELECT`, `INSERT`, `HMGET` | SQL verb or command name |
+| `db.query.text` | Stable | `SELECT * FROM orders WHERE id = $1` | Parameterized query text; PII risk; see caveat |
+| `db.namespace` | Stable | `mydb` | Database name / schema |
+| `db.collection.name` | Stable | `orders` | Table or collection name |
+| `db.client.connections.used` | Development | `14` | Active connections in pool (attribute name varies by semconv draft; pin to version in `../standards.md`) |
+| `db.client.connection.pool.utilization` | Development | `0.875` | Ratio: used / max (current semconv draft name; older drafts use `db.client.connections.usage`) |
+
+`db.query.text` caveat: WHERE clause literals may contain PII (email addresses, phone numbers). Always use parameterized queries so that literals are replaced by `$1`, `?`, or `:name` placeholders. If the ORM or driver captures the raw query, apply redaction at the OTel SDK layer or in the Collector processor. Cross-ref `../signals/privacy.md`.
+
+### 4.1 N+1 Query Detection
+
+Symptom: a single parent `SERVER` span contains N child `CLIENT` spans that all share the same `db.operation.name` and `db.collection.name`, each taking 2-30 ms, producing an additive latency of N × latency_per_query.
+
+Trace waterfall example:
+
+```
+[SERVER] POST /api/orders            .................... 480ms
+  [CLIENT] SELECT orders WHERE ...   ..  18ms
+  [CLIENT] SELECT orders WHERE ...   ..  17ms
+  [CLIENT] SELECT orders WHERE ...   ..  17ms
+  [CLIENT] SELECT orders WHERE ...   ..  18ms
+  ...  (× 24 more)
+```
+
+Detection:
+
+- OTel trace visualizers (Honeycomb BubbleUp, Datadog Watchdog, Grafana Tempo TraceQL) can flag traces where `count(db.CLIENT spans per SERVER span) > N_threshold`.
+- TraceQL example (Grafana Tempo): `{ span.db.system = "postgresql" } | count() > 10`
+- Honeycomb: group by `trace.parent_id`, count `db.*` child spans, alert on p95 count > 10.
+
+Remediation: replace per-row lookups with eager loading (ORM `include`/`join`), batch queries (`SELECT ... WHERE id = ANY($1)`), or the DataLoader pattern for GraphQL.
+
+### 4.2 Slow Query Analysis
+
+Slow queries surface as `CLIENT` spans with duration above the p99 threshold.
+
+1. Index `db.query.text` + `db.operation.name` against a `span.duration` histogram in the trace backend.
+2. Flag spans where `span.duration > 100ms` at p99 (adjust threshold per SLO).
+3. Correlate `db.query.text` with `pg_stat_statements` / MySQL `slow_query_log` to obtain execution plans.
+4. Apply `EXPLAIN ANALYZE` and add the missing index or rewrite the query.
+
+Privacy: if `db.query.text` captures literal values, redact before export; see `../signals/privacy.md §query text redaction`.
+
+### 4.3 Connection Pool Exhaustion
+
+Symptom: the blocking span is on the **client side** (waiting to acquire a pool connection), not inside the DB. `CLIENT` span duration grows while actual DB response time stays flat.
+
+```
+[SERVER] POST /api/orders            .............. 5200ms
+  [CLIENT] acquire pool connection   ............  5190ms  ← pool wait, no DB call yet
+  [CLIENT] SELECT orders WHERE ...   .           10ms
+```
+
+Attributes to monitor: `db.client.connections.used` (Development), `db.client.connections.usage` (Development, alert at > 0.85). Cross-ref `../signals/metrics.md` for `pool_wait_count` and `pool_wait_duration_seconds` histograms.
+
+Remediation: tune pool size to expected concurrency (10-20 connections per pod); add circuit breaker upstream; split read/write pools for long-running read transactions.
+
+---
+
+## 5. Messaging Tracing Patterns (messaging.* semconv)
+
+Source: <https://opentelemetry.io/docs/specs/semconv/messaging/>
+
+### Core messaging.* attributes
+
+| Attribute | Example | Notes |
+|-----------|---------|-------|
+| `messaging.system` | `kafka`, `rabbitmq`, `nats` | Required |
+| `messaging.destination.name` | `orders-topic` | Topic or queue name |
+| `messaging.operation.type` | `create`, `send`, `receive`, `process`, `settle` | Operation type per semconv 1.43.0 (Development; replaces deprecated `messaging.operation`; `publish`→`send`, `deliver`→`process`). `settle` covers ack/nack; `ack` alone is NOT a valid enum value |
+| `messaging.message.id` | `msg-uuid-1234` | Message identifier for dedup |
+| `messaging.kafka.message.key` | `order-9988` | Kafka partition key |
+| `messaging.kafka.consumer.group` | `payment-consumer-group` | Kafka consumer group |
+
+### SpanKind assignment
+
+- `PRODUCER`: set when calling `producer.send()` or equivalent.
+- `CONSUMER`: set when calling `consumer.poll()` or the message handler.
+- Connect the two with a **span link** (not parent-child): async messaging breaks the synchronous call chain; span links preserve trace continuity across the queue boundary.
+
+```
+[PRODUCER] send orders-topic         ..  3ms    TraceId: aaa
+  Link → [CONSUMER] receive orders-topic  ...  220ms  TraceId: aaa
+    [CLIENT] INSERT INTO payments    ..  18ms
+```
+
+### 5.1 Kafka-Specific Patterns
+
+Inject W3C `traceparent` into Kafka message headers at the producer; extract at the consumer before creating the `CONSUMER` span. OTel Kafka instrumentation handles this automatically (`opentelemetry-instrumentation-kafka` for Java/Python).
+
+Kafka consumer lag is an external metric, not a trace attribute: MSK (`kafka.consumer_lag` CloudWatch), Confluent Cloud (built-in), self-managed (`kafka_consumer_group_lag` via JMX + Prometheus). Cross-ref `../signals/metrics.md §messaging`.
+
+**Primary-topic lag alert threshold (recommended starting point):**
+- Warn: `kafka_consumer_group_lag > 10_000 for 5m` (queue growing faster than consumer can drain)
+- Page: `kafka_consumer_group_lag > 100_000 for 10m` (backlog will breach SLO if unaddressed)
+- Calibrate per topic throughput and consumer group parallelism; high-throughput topics routinely run at 50k–500k lag without impact.
+
+### 5.2 Flink and Spark Streaming
+
+Create one span per pipeline stage; the trace covers end-to-end job from source read to sink write. Flink OTel instrumentation is emerging (not stable as of 2026-Q2); use `messaging.system = kafka` for source/sink spans and `INTERNAL` for operator chains. Spark: instrument at the `foreachBatch` boundary; each micro-batch is one `INTERNAL` span. Checkpoint metrics: cross-ref `../signals/metrics.md §streaming`.
+
+### 5.3 Dead Letter Queue (DLQ) Observability
+
+A DLQ receives messages that failed all retries. Losing trace context at DLQ ingestion prevents diagnosis.
+
+1. **Trace propagation**: copy the original `traceparent` from the failed message headers into the DLQ message headers; do not generate a new trace ID.
+2. **DLQ depth alert**: alert on `kafka_consumer_group_lag{topic="orders-dlq"}` > 0 for critical queues.
+3. **DLQ arrival span**: create a `CONSUMER` span at DLQ arrival with `messaging.destination.name = orders-dlq` and the original `trace_id`.
+4. **Replay tooling**: re-inject the original trace context into replayed messages. A new `trace_id` at replay orphans the original failure span.
+
+---
+
+## 6. RPC Tracing (rpc.* semconv RC)
+
+Source: <https://opentelemetry.io/docs/specs/semconv/rpc/grpc/>
+
+### gRPC
+
+| Attribute | Example |
+|-----------|---------|
+| `rpc.system.name` | `grpc` |
+| `rpc.service` | `com.example.OrderService` |
+| `rpc.method` | `CreateOrder` |
+
+SpanKind: `CLIENT` on the caller; `SERVER` on the callee. The `traceparent` header is propagated via gRPC metadata.
+
+### HTTP RPC
+
+Use `http.*` attributes for REST and JSON-RPC over HTTP. `rpc.*` is for binary RPC protocols only.
+
+### Error handling
+
+Set `status.code = ERROR` and `status.description` on RPC failures. Use `span.recordException(e)` to atomically populate `exception.type`, `exception.message`, and `exception.stacktrace`; required by MRA in `../incident-forensics.md §2.2`. Note: since semconv 1.40 the span-event form of exception recording is deprecated in favor of exception LogRecords (`OTEL_SEMCONV_EXCEPTION_SIGNAL_OPT_IN`); during migration prefer `logs/dup` so both the span pivot and log-based queries keep working.
+
+---
+
+## 7. Trace Sampling
+
+Cross-ref `../transport/sampling-recipes.md` for full configuration recipes.
+
+| Strategy | Pros | Cons |
+|----------|------|------|
+| Head-based | Low overhead; decision at trace root | Cannot retain traces that only show problems at a downstream hop |
+| Tail-based | Trace-complete decision; keeps all error traces | Requires buffering the full trace before deciding; higher Collector memory |
+
+Recommended recipe:
+
+- 100% of traces where `status.code = ERROR`
+- 100% of traces where business cost > $0.50 (custom attribute)
+- 5-10% tail sampling of remaining traces for baseline coverage
+
+For multi-replica tail sampling, use consistent hashing on `trace_id` via the OTel Collector `loadbalancing` exporter to route all spans of one trace to the same Collector instance.
+
+---
+
+## 8. Trace Backends
+
+Cross-ref `../vendor-categories.md` for full vendor category taxonomy.
+
+| Backend | Category | Key differentiator |
+|---------|----------|--------------------|
+| Jaeger (CNCF Graduated) | OSS full-stack | Kubernetes-native; Badger or Cassandra storage |
+| Grafana Tempo | OSS full-stack | Low-cost object storage (S3/GCS); TraceQL query language |
+| Zipkin | OSS (legacy) | Older wire format; superseded by OTel/Jaeger for new deployments |
+| Honeycomb | High-cardinality specialist | BubbleUp auto-analysis; best for high-attribute-count traces |
+| Datadog APM | Commercial SaaS | Deep integration with Datadog metrics and logs |
+| Elastic APM | Commercial SaaS | Tight Kibana/Elasticsearch integration |
+| SigNoz | OSS full-stack | ClickHouse-backed; cost-efficient at high ingestion rates |
+
+---
+
+## 9. Trace Exemplars
+
+Exemplars link a metric data point to a specific trace, enabling direct navigation from a PromQL alert to the trace that caused the spike.
+
+- Prometheus exemplar support is available since v2.31.0. Histograms and summaries can carry a `trace_id` label on selected samples.
+- OTel SDK Java, Go, and Python emit exemplars automatically when the current span context is sampled.
+- Usage: click a spike in a Grafana panel backed by a Prometheus histogram → the exemplar tooltip shows a `trace_id` link → click to open the trace in the configured trace backend (Tempo, Jaeger, etc.).
+
+```promql
+# Example: p99 latency histogram with exemplar-capable query
+histogram_quantile(0.99,
+  sum by (le, service_name) (
+    rate(http_server_request_duration_seconds_bucket{
+      deployment_environment="prod"
+    }[5m])
+  )
+)
+```
+
+Cross-ref `../incident-forensics.md §Step 2` for the exemplar-based trace acquisition flow.
+
+---
+
+## 10. Propagating Baggage (Security and Privacy)
+
+W3C Baggage (`baggage` header) carries key-value pairs alongside the trace context. It is not part of the trace itself; it is a side-channel for business attributes that downstream services need.
+
+### Allowed vs. prohibited baggage content
+
+| Allowed | Prohibited |
+|---------|-----------|
+| `tenant.id` | User email address |
+| `user.tier` (tier name, not PII) | Session tokens or auth cookies |
+| `deployment.sha` | Passwords or API keys |
+| `feature.flag.name` | Any field classified as PII under GDPR/PIPA |
+
+W3C Baggage Recommendation §3.1: baggage propagates to all downstream services within the same distributed operation. Any data placed in baggage is visible to every service on the call path, including third-party services. Data that is proprietary, confidential, or personally identifiable MUST NOT be placed in baggage without explicit downstream trust agreement.
+
+Enforcement rule: strip or validate incoming baggage at the API gateway / ingress before forwarding to internal services. The mesh ingress gateway is the correct enforcement point; see `../layers/mesh.md §Baggage scrubbing`.
+
+Cross-ref:
+
+- `../boundaries/cross-application.md`: trust-boundary baggage validation rules per cloud and mesh
+- `../signals/privacy.md`: PII classification and redaction for baggage content
+
+---
+
+## 11. Matrix Coverage (traces column)
+
+These cells from `../matrix.md` have trace-specific behavior worth noting:
+
+| Layer | Boundary | Status | Rationale |
+|-------|----------|--------|-----------|
+| L3-network | cross-application | PARTIAL | L3 packets carry no trace context natively; use trace-ID tagging at egress only |
+| L4-transport | any | PARTIAL | TCP is not trace-native; trace context begins at mesh or L7 |
+| mesh | cross-application | PASS | Primary trace origin; Envoy injects spans with zero application code changes |
+| L7-application | cross-application | PASS | Primary; W3C `traceparent` on all outbound HTTP/gRPC calls |
+| L7-application | multi-tenant | PASS | `tenant.id` propagated in W3C Baggage on every span |
+| L7-application | release | PASS | `service.version` resource attribute on all spans; canary routing by version |
+| L7-application | slo | PARTIAL | Tail sampling must retain error traces within the error budget window; traces are forensic input, not the SLI computation source |
+
+---
+
+## 12. Anti-patterns
+
+These candidates extend `../anti-patterns.md`. Each pattern breaks trace continuity or introduces compliance risk.
+
+| Anti-pattern | Consequence | Remediation |
+|---|---|---|
+| Missing parent spans (orphan spans) | Trace visualizer shows disconnected spans; root cause chain is broken | Verify `traceparent` extraction at every inbound call; check SDK auto-instrumentation coverage |
+| `db.query.text` captured with untrimmed PII | Email or phone literal in WHERE clause stored in trace backend; GDPR/PIPA violation | Parameterize all queries; add OTel SDK `db.sanitize_query` option or Collector redaction processor |
+| Async messaging without span links | `PRODUCER` and `CONSUMER` spans in separate disconnected traces; cannot follow a message end-to-end | Use OTel SDK `span.addLink()` with the producer span context when creating the consumer span |
+| DLQ drain creates a new `trace_id` | Original failure span is orphaned; forensic link to the root cause is lost | Re-inject original `traceparent` from DLQ message headers into the replayed message headers |
+| Head-based sampling on multi-service paths | A decision to not sample at the entry service drops all downstream spans, including error spans that only appear deep in the call graph | Use tail-based sampling with 100% error retention; see §7 |
+| Kafka consumer lag unmonitored | DLQ silently fills; processing backlog not detected until downstream systems starve | Alert on `kafka_consumer_group_lag` per consumer group; cross-ref `../signals/metrics.md §messaging` |
+
+---
+
+## References
+
+- OTel Trace API specification: <https://opentelemetry.io/docs/specs/otel/trace/api/>
+- OTel Database semconv: <https://opentelemetry.io/docs/specs/semconv/database/>
+- OTel Messaging semconv: <https://opentelemetry.io/docs/specs/semconv/messaging/>
+- OTel RPC/gRPC semconv: <https://opentelemetry.io/docs/specs/semconv/rpc/grpc/>
+- W3C Trace Context Level 1 Recommendation: <https://www.w3.org/TR/trace-context/>
+- W3C Baggage Recommendation: <https://www.w3.org/TR/baggage/>
+- Kafka producer/consumer instrumentation: <https://kafka.apache.org/documentation/>
+- Confluent consumer lag metrics: <https://docs.confluent.io/platform/current/kafka/monitoring.html>

package/skills/oma-observability/resources/standards.md

@@ -0,0 +1,214 @@
+---
+otel_spec: "1.x (stable API/SDK)"
+otel_semconv: "1.43.0 (2026-07)"
+specs:
+  - "W3C Trace Context: Level 1 Recommendation 2020-02-06; Level 2 Candidate Recommendation"
+  - "W3C Baggage: Recommendation 2022-12-22"
+  - "ISO/IEC 25010: 2023; ISO/IEC 27001:2022; ISO/IEC 42010:2011"
+notes:
+  - "Pinned versions (update on spec promotion or attribute deprecation; see SKILL.md §Versioning & Deprecation — no scheduled review)"
+---
+
+# Observability Standards Reference
+
+## 1. Purpose
+
+This file defines the normative standards baseline for the `oma-observability` skill. It answers:
+
+- Which specifications govern trace propagation, attribute naming, and signal transport?
+- How do ISO quality characteristics map onto observability concerns?
+- Which OSI layers are in scope and why?
+- What clock discipline is required for trustworthy timestamp ordering?
+
+All other files in this skill reference this document as the authoritative source. Implementers MUST read this before writing instrumentation, configuring collectors, or authoring SLO policies.
+
+---
+
+## 2. Primary De Facto Standards
+
+### 2.1 OpenTelemetry (CNCF Incubating)
+
+Source: <https://opentelemetry.io/docs/specs/otel/> | CNCF: <https://www.cncf.io/projects/opentelemetry/>
+
+OpenTelemetry is the vendor-neutral, CNCF-incubating specification for telemetry APIs, SDKs, semantic conventions, and the wire protocol OTLP. It is the single de facto standard this skill operates on.
+
+Four pillars in scope for this skill:
+
+| Pillar | Specification | Wire protocol |
+|--------|---------------|---------------|
+| API / SDK | opentelemetry.io/docs/specs/otel/ | N/A |
+| Semantic Conventions (semconv) | opentelemetry.io/docs/specs/semconv/ | N/A |
+| Protocol (OTLP) | opentelemetry.io/docs/specs/otlp/ | gRPC :4317 / HTTP :4318 |
+| Collector | opentelemetry.io/docs/collector/ | N/A |
+
+Key constraints:
+- Use the stable semconv groups listed in Section 3 for production instrumentation.
+- OTLP is the mandatory export format; vendor-native formats are acceptable only as secondary sinks.
+- The OTel Operator (`v1beta1`) manages Collector and auto-instrumentation CRs in Kubernetes.
+
+### 2.2 W3C Trace Context
+
+- Level 1 Recommendation: <https://www.w3.org/TR/trace-context/> (published 2020-02-06)
+- Level 2 Candidate Recommendation: <https://www.w3.org/TR/trace-context-2/>
+
+W3C Trace Context defines the `traceparent` and `tracestate` HTTP headers that establish a distributed trace across service boundaries. Level 1 is the production-stable baseline. Level 2 adds the `traceflags` precision extension and is safe to implement against CR status.
+
+Mandatory propagation rule: every outbound HTTP/gRPC call MUST forward `traceparent`. Stripping it silently is an anti-pattern.
+
+Vendor header compatibility reference (informative):
+- AWS X-Ray: `X-Amzn-Trace-Id`; compatible bridge via OTel X-Ray propagator
+- GCP Cloud Trace: prefers W3C; legacy `X-Cloud-Trace-Context` is supported for backward compat
+- Datadog: `X-Datadog-Trace-Id`; use OTel Datadog exporter for bridging
+
+### 2.3 W3C Baggage
+
+Recommendation: <https://www.w3.org/TR/baggage/> (2022-12-22)
+
+W3C Baggage defines the `baggage` HTTP header for key-value pairs propagated alongside a trace. The W3C spec itself contains trust-boundary and PII guidance: baggage is visible to all downstream services and MUST NOT carry secrets, tokens, or personally identifiable information without explicit downstream trust agreement.
+
+OTel Baggage API reference: <https://opentelemetry.io/docs/specs/otel/baggage/api/>
+
+Usage rules enforced by this skill:
+- Allowed: tenant ID, feature flag state, deployment SHA, region hint.
+- Not allowed: user email, session tokens, authentication credentials.
+- Trust boundary: strip or validate baggage at ingress gateway before forwarding to external services.
+
+---
+
+## 3. OpenTelemetry Semconv Stability Tiers
+
+Semantic convention stability determines which attributes can be used in production without risk of breaking changes. Pin the semconv version in the file header above and update on the triggers listed in §7 (spec promotion, attribute deprecation).
+
+Source: <https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/>
+
+| Tier | Groups | Production use |
+|------|--------|----------------|
+| Stable | `service.*`, `host.*`, `cloud.*`, `k8s.*`, `http.*`, `db.*` (core), `network.*` (core), `error.*` | Yes, without caveat |
+| Release Candidate (RC) | `rpc.*`, gRPC semconv | Yes, expect minor changes |
+| Development | `tls.*`, `network.connection.*` | Test environments; production use requires change-tolerance |
+| Experimental | `gen_ai.*` (deprecated in core registry since semconv 1.42; maintained in the dedicated GenAI semconv repository), profiles (OTEP 0239; Development) | Not for production SLOs |
+
+Notes:
+- `network.*` core attributes (e.g., `network.protocol.name`, `network.transport`) are Stable. `network.connection.*` (e.g., `network.connection.type`, `network.connection.subtype`) are Development.
+- `tls.*` (all) are Development as of semconv 1.43.0 (verified 2026-07). For TLS deep inspection, use Wireshark or vendor-specific TLS tooling rather than OTel attributes.
+- OTEP 0239 (profiling signal) is Development; SDK support is Java-only as of 2026-Q2. Parca and Pyroscope are in production, but the OTel profiling spec is not yet stable. Mark any profiling-related SLOs as experimental.
+
+Verified sources:
+- TLS attrs: <https://opentelemetry.io/docs/specs/semconv/attributes-registry/tls/>
+- Network attrs: <https://opentelemetry.io/docs/specs/semconv/attributes-registry/network/>
+- RPC/gRPC: <https://opentelemetry.io/docs/specs/semconv/rpc/grpc/>
+
+---
+
+## 4. ISO/IEC Indirect Mapping
+
+ISO/IEC 25010:2023 (<https://www.iso.org/standard/78176.html>) defines the Systems and Software Quality Model. There is no dedicated "Observability" quality characteristic in the 2023 edition. Observability concerns map indirectly through three characteristics:
+
+| ISO/IEC 25010:2023 characteristic | Sub-characteristic | Observability concern |
+|---|---|---|
+| Maintainability | Analysability | Can operators diagnose system state from telemetry? |
+| Security | Accountability | Are actions traceable to an authenticated identity (audit logs)? |
+| Reliability | Faultlessness | Does telemetry surface defects before users encounter them? |
+
+This mapping is informative, not normative. It is useful when presenting observability investment in terms that quality assurance or audit stakeholders recognize.
+
+Related standards:
+- **ISO/IEC 27001:2022 / 27002:2022** (<https://www.iso.org/standard/27001>); controls for information security management systems; governs log integrity, access control on observability backends, and audit trail requirements.
+- **ISO/IEC 42010:2011** (<https://www.iso.org/standard/50508.html>); architecture description standard. An "observability viewpoint" is a valid architecture viewpoint under 42010 for documenting how stakeholders inspect system internals.
+
+---
+
+## 5. OSI Boundary Decision
+
+This skill operates on OSI layers L3, L4, mesh (L4-L7 hybrid), and L7. The following table makes the in-scope and out-of-scope decisions explicit.
+
+### In scope
+
+| Layer | Label | Coverage |
+|-------|-------|----------|
+| L3 | Network | IP routing, VPC flow logs, BGP/BMP, ICMP, PMTUD |
+| L4 | Transport | TCP retransmits/RTT, eBPF (Beyla/Pixie), QUIC/HTTP3 transport |
+| L4-L7 | Service mesh | Istio/Linkerd/Envoy: zero-code instrumentation, mTLS metadata as security context |
+| L7 | Application | HTTP/gRPC/WebSocket, RUM (web + mobile), crash analytics, messaging |
+
+### Out of scope
+
+| Layer | Label | Reason | Use instead |
+|-------|-------|--------|-------------|
+| L1 | Physical | SaaS hypervisor hides; no OTel semconv coverage | Vendor DCIM tooling (Nlyte, Sunbird, Device42) |
+| L2 | Data Link | Same as L1; SNMP/IPMI are hardware domains | Vendor DCIM tooling; SNMP exporters for Prometheus if needed |
+| L5 | Session (full) | gRPC session semantics moved to L7 instrumentation; WebSocket deferred | WebSocket-specific vendor tooling |
+| L6 | Presentation (full) | TLS kept as security context attributes only (`tls.*` Development tier); full TLS inspection is not OTel's domain | Wireshark for packet-level TLS; Cloudflare Radar for TLS ecosystem data; vendor-specific TLS inspection tooling |
+
+Rationale: approximately 90% of production debugging occurs at L3, L4, mesh, and L7. L1/L2 are opaque to SaaS workloads. L5/L6 full coverage requires OTel semconv maturity that does not yet exist (Development tier).
+
+---
+
+## 6. Clock Discipline
+
+Distributed traces depend on monotonic, synchronized clocks across all nodes. Clock drift corrupts waterfall charts and makes parent-before-child ordering unreliable.
+
+### Requirements
+
+- **NTP or chrony** MUST be running and synchronized on all host VMs and container hosts.
+- Acceptable drift tolerance for trace timestamp correlation: **< 100 ms** (typical NTP accuracy on well-connected hosts).
+- For sub-millisecond precision (financial, telco, or high-frequency workloads): use **PTP (IEEE 1588)** hardware timestamping.
+
+### Cloud hypervisor time sync notes
+
+| Cloud | Mechanism | Notes |
+|-------|-----------|-------|
+| AWS | Chrony + Amazon Time Sync Service (169.254.169.123) | PTP-backed time source; verify with `chronyc tracking` and `chronyc sources -v` |
+| GCP | Internal hypervisor clock sync (`metadata.google.internal` via `time.google.com`) | GCP VMs inherit host clock. `timedatectl show` only confirms a sync daemon is active; to verify the actual offset, use `chronyc sources -v` (chrony) or `timedatectl timesync-status` (systemd-timesyncd) |
+| Azure | Hyper-V IC timesync primary; external NTP fallback only if Hyper-V IC tools absent | Azure Linux VMs with the Hyper-V Integration Services use host time as the authoritative source. External NTP (for example `time.windows.com`) as a peer source can conflict with IC timesync; treat it as a fallback, not a parallel source. Docs: <https://learn.microsoft.com/azure/virtual-machines/linux/time-sync> |
+
+### Span timestamp validation rule
+
+Validate each span against its own timestamps:
+
+```
+span.start_time <= span.end_time
+```
+
+A parent span may end before an asynchronous child completes; the OTel Trace API explicitly allows children to continue after their parent ends. Parent/child interval containment is not a validity invariant or clock-drift detector. Investigate suspicious cross-host ordering using measured clock offsets and the actual synchronous/asynchronous operation semantics. See `resources/meta-observability.md` and <https://opentelemetry.io/docs/specs/otel/trace/api/>.
+
+### Anti-pattern
+
+NTP drift left unmonitored is anti-pattern #18 in this skill: waterfall charts appear causally wrong, leading engineers to chase phantom race conditions. Monitor `otelcol_receiver_accepted_spans` and node-level NTP offset metrics together.
+
+---
+
+## 7. Versioning and Review Triggers
+
+- The spec version block in this file's header MUST be updated when:
+  - Any listed semconv group promotes from Development → RC or RC → Stable.
+  - A listed W3C document advances maturity (CR → PR → Recommendation).
+  - OTel releases a new minor version with breaking semconv changes.
+- **Review triggers only, no scheduled cadence** (see `SKILL.md` §Versioning & Deprecation): OTel spec releases and CNCF landscape changes (<https://landscape.cncf.io/>) affecting cited content; verify live when load-bearing.
+- When updating, also check and update affected signal/layer files that reference the changed attribute group.
+- Skill minor version bump required on any normative change to this file.
+
+---
+
+## 8. References
+
+Primary sources, in order of precedence:
+
+1. OpenTelemetry specification: <https://opentelemetry.io/docs/specs/otel/>
+2. OTel Semantic Conventions: <https://opentelemetry.io/docs/specs/semconv/>
+3. OTLP specification: <https://opentelemetry.io/docs/specs/otlp/>
+4. W3C Trace Context L1 Recommendation: <https://www.w3.org/TR/trace-context/>
+5. W3C Trace Context L2 Candidate Recommendation: <https://www.w3.org/TR/trace-context-2/>
+6. W3C Baggage Recommendation: <https://www.w3.org/TR/baggage/>
+7. ISO/IEC 25010:2023; <https://www.iso.org/standard/78176.html>
+8. ISO/IEC 27001:2022; <https://www.iso.org/standard/27001>
+9. ISO/IEC 42010:2011; <https://www.iso.org/standard/50508.html>
+10. CNCF Projects: <https://www.cncf.io/projects/>
+11. CNCF Landscape (authoritative vendor registry): <https://landscape.cncf.io/>
+12. OTel TLS attributes registry: <https://opentelemetry.io/docs/specs/semconv/attributes-registry/tls/>
+13. OTel network attributes registry: <https://opentelemetry.io/docs/specs/semconv/attributes-registry/network/>
+14. OTel RPC/gRPC semconv: <https://opentelemetry.io/docs/specs/semconv/rpc/grpc/>
+15. OTel Baggage API: <https://opentelemetry.io/docs/specs/otel/baggage/api/>
+16. AWS Time Sync Service: <https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html>
+17. Azure Linux time sync: <https://learn.microsoft.com/azure/virtual-machines/linux/time-sync>
+18. OTEP 0239 (profiling signal): <https://github.com/open-telemetry/oteps/blob/main/text/profiles/0239-profiles-data-model.md>