@ccoalm/ccl-skills 0.1.0

package/dist/assets/marketplace/plugins/ccl-skills/skills/platform-observability/references/metrics-conventions.md

@@ -0,0 +1,105 @@
+# Metrics Conventions
+
+## Baseline labels (attached automatically by the framework metrics client)
+
+| Label | Source | Notes |
+|---|---|---|
+| `service` (or `psm`) | env var | Service identity — PSM-style `<owner>.<class>.<env>` or `service.name` OTel attribute. |
+| `lane` / `env` | env var | Deployment environment label (`prod`, `test`, `pre`, or named lane). Mesh-routed traffic per lane → labelled lane lets you isolate per-lane regressions. |
+| `idc` / `region` | env var | Data center / cloud region. |
+| `cluster` | env var | Kubernetes cluster name (or equivalent orchestrator unit). |
+
+The framework metrics client appends these to **every** metric automatically. Service code calls `cli.Counter(ctx, name, value, kvs...)` with only domain-specific labels; baseline is invisible to the caller.
+
+**Instance identity (`pod_name`/`pod_ip`/`node_ip`/`instance`) is NOT a baseline metric label.** Pods churn under autoscaling/rollouts, so as a label they grow unbounded series over the retention window (each labelset = a separate time series). Emit instance identity as **OTel resource attributes** and **log fields**, and reach a specific instance via **exemplars** or a low-churn `*_info` target-metadata metric joined at query time — not by labelling every metric with it.
+
+## Per-metric labels — cardinality discipline
+
+Allowed:
+- API path (templated, e.g. `/v1/users/:id`, never raw URL).
+- Endpoint / handler name.
+- Method (GET/POST/...).
+- Error code class (`server_panic`, `validation_error`, `upstream_timeout` — bounded enum).
+- Status code class (`2xx`, `4xx`, `5xx`).
+
+Refuse:
+- User-id, tenant-id, account-id.
+- Raw URL with query string.
+- Free-text error message.
+- Log-id, trace-id, request-id (these are exemplar attachments, not labels).
+- Timestamps as labels.
+
+Cardinality test: `count_distinct(label_value) × count_distinct(other_labels) × series_count`. If projected > 10^6, refuse and route to exemplar traces or to a different signal type.
+
+## Instrument selection
+
+| Use case | Instrument | Notes |
+|---|---|---|
+| Request rate, error count | Counter (monotonic) | OTel `Counter` increment only — required for request/error totals so backend `rate()`/`increase()` semantics work. **不用 `UpDownCounter`**（那是非 monotonic，用于活跃连接 / 队列深度等 active count）。 |
+| Latency | Histogram | Bucket boundaries must be sensible for the SLI (e.g. 1ms, 5ms, 10ms, 50ms, 100ms, 500ms, 1s, 5s). |
+| Saturation (queue depth, connection pool) | Gauge (Observable, async) | Read-on-collect, not write-on-event. |
+| Cumulative resource | Sum | E.g. bytes transferred. |
+| One-shot probe | Histogram with single value | When you need percentile semantics. |
+
+Async observable gauges via OTel SDK reduce noise vs synchronous sets. Pattern: register one callback per metric; reduce buffered samples (avg/max/last) at collection time.
+
+## 业内方法对照（Golden Signals / RED / USE → 本 ref 已有 instruments）
+
+业内不同 framework 讲法不同，下表把外部命名映射回本 ref 已要求的 instrument，避免被术语卡住：
+
+| 外部框架 | 关注什么 | 适用对象 | 映射到本 ref instrument |
+|---|---|---|---|
+| **Google 4 Golden Signals**（SRE Book）| Latency / Traffic / Errors / Saturation | service-level（user-facing service）| Latency → Histogram；Traffic → Counter (request rate)；Errors → Counter (error count)；Saturation → Gauge (queue / pool / cpu / mem) |
+| **RED**（Tom Wilkie / Weaveworks）| Rate / Errors / Duration | request-driven service / RPC endpoint | Rate → Counter；Errors → Counter；Duration → Histogram（近似等于 request-side Golden Signals 去 Saturation；非正式 derivation）|
+| **USE**（Brendan Gregg）| Utilization / Saturation / Errors | resource（CPU / memory / disk / NIC / connection pool）| Utilization → Gauge (%) ；Saturation → Gauge (queue depth)；Errors → Counter（资源驱动而非请求驱动）|
+
+何时用哪个：
+- **service-level 默认从 Golden Signals / RED 起步**：API endpoint 健康度首选这两套
+- **resource-level 默认从 USE 起步**：DB connection pool / CPU / NIC 健康度首选这套；USE 也可用在软件资源（如 thread pool / goroutine queue）不限硬件
+- 实际仪表盘常**混用 RED + USE**（Wilkie 自己也推荐）— framework 是脚手架不是教条
+- 三套不替代 SLI/SLO 设计（参 `sli-slo-design.md`）— 是 day-to-day 仪表盘 + 报警的 baseline 共同词汇
+
+## Naming
+
+`<domain>_<noun>_<unit>_<aggregation_hint>`
+
+Examples:
+- `http_server_request_total` — request count, counter.
+- `http_server_request_duration_seconds` — latency, histogram (base-unit seconds in name, per R3; matches the OTel-translated name and the SLI query in `sli-slo-design.md`).
+- `rpc_client_call_total` — outbound RPC count.
+- `service_panic_total` — recovery counter.
+- `queue_pending_messages` — gauge.
+
+Avoid:
+- Generic names like `count`, `latency`, `total` without domain.
+- Mixed-case names.
+- Domain leakage (e.g. `school_login_total` for a non-public service skill; use `auth_login_total` instead).
+
+## SDK initialization
+
+```
+MeterProvider:
+  - Resource attrs (semconv): service.name, deployment.environment, k8s.pod.name, host.id
+  - Reader: PeriodicReader, interval 15s, OTLP gRPC exporter
+  - Exporter: insecure inside cluster (TLS at network layer)
+View overrides (optional):
+  - Custom histogram buckets per metric name.
+```
+
+The 15s reader interval matches Prometheus scrape conventions and keeps OTLP push volume sane.
+
+## Prometheus 3.0 Baseline (Nov 2024+)
+
+- **Prometheus 3.0 (released 2024-11-14, per `prometheus.io/blog/2024/11/14/prometheus-3-0/`) lifts the UTF-8 normalization requirement** — metric and label names can contain dots, slashes, colons, and other UTF-8 characters by default. This is the bridge for OpenTelemetry metrics: no more mandatory dots-to-underscores translation on ingest, so `http.server.request.duration` reaches Prometheus and PromQL queries unchanged. **OTLP receiver is native**: Prometheus exposes `/api/v1/otlp/v1/metrics` for direct OTel push; for projects that want the legacy underscore shape, opt-in `otlp.translation_strategy = UnderscoreEscapingWithSuffixes` per the migration guide. **Native histograms are first-class wire support in 3.0** — a native histogram sample is one time series carrying sparse exponential (or custom) buckets, replacing the multi-`_bucket`-series Classic histogram shape; far more efficient on cardinality and quantile accuracy; opt-in per metric via SDK; PromQL `histogram_*` functions handle both representations transparently. (Native histograms remain marked as experimental in early 3.x and stabilize in a later 3.x release per the Prometheus team — verify the specific 3.x version's stability state before production-pinning a metric to native-histogram-only.) **Remote-Write 2.0** carries native histograms + metadata + exemplars + created-timestamp in the wire protocol. **Breaking-change audit before flipping**: `remote-write-receiver` / `promql-at-modifier` / `promql-negative-offset` feature flags removed (apps depending on `--enable-feature=...` for those will fail to start); Remote-Write `enable_http2` default changed to `false`; PromQL syntax tightening for `@` and negative-offset edge cases. Pin Prometheus version in platform manifests, rehearse upgrade on a non-critical scrape target, and verify dashboards / recording rules / alerts still parse before flipping production. **Migration discipline for the UTF-8 names flip on existing deployments** — switching translation off on a running platform silently breaks every PromQL expression that references the underscore-form translated names (`http_server_request_duration_seconds_bucket` no longer exists if the OTel name `http.server.request.duration` now arrives un-normalized). Required pre-flip steps: (a) inventory every PromQL expression referencing OTel-translated metric names across dashboards / recording rules / alert rules; (b) dual-publish both forms during a bake window (Collector emits both transformed and untransformed series via a duplicating processor), or migrate query references first against a staging Prometheus before flipping production; (c) run `promtool check rules` plus a query-result dry-run on critical alerts; (d) flip the translation strategy only after the inventory shows zero unmigrated references. Underestimating this is the typical "silent dashboard outage on Tuesday morning after a platform upgrade" failure.
+
+## OpenTelemetry GenAI Semantic Conventions (Development → opt-in)
+
+- **OTel GenAI conventions (status: Development as of 2026-Q1 per `opentelemetry.io/docs/specs/semconv/gen-ai/`)** standardize LLM observability across providers. Span attributes include `gen_ai.system` (anthropic / openai / google.gemini / etc), `gen_ai.request.model`, `gen_ai.response.model`, `gen_ai.request.temperature`, `gen_ai.usage.input_tokens` (includes cached tokens per spec), `gen_ai.usage.output_tokens`; metric `gen_ai.client.operation.duration` for end-to-end latency. **The OTel GenAI spec says SHOULD NOT capture prompt/tool-argument content by default** — only metadata (model names, token counts, durations). However, **individual instrumentation libraries vary**: some community SDK middleware (LangChain instrumentations, vendor-specific tracing wrappers, OTel-contrib LLM packages) have historically defaulted to capturing prompt text in spans, contrary to the spec's spirit. Required platform-side discipline: **audit each instrumentation's prompt/event capture default before enabling GenAI telemetry on a service** (run a smoke test on a sandbox model call, inspect the emitted span attributes, confirm prompts are absent unless explicitly opted-in via the library's content-capture flag). Opt-in to content capture is a deliberate per-service decision with PII / compliance review; when enabled, capture content via event-based attributes that flow through trace events, NOT metric labels (prompt-content as a metric label would explode cardinality and likely violate the platform's label allowlist). **Stability mechanism**: instrumentations honor `OTEL_SEMCONV_STABILITY_OPT_IN` env (comma-separated; include `gen_ai_latest_experimental` to emit current conventions, else they emit the v1.36.0-frozen version). When platform-observability standardizes label / attribute allowlists across services, include the GenAI attributes alongside HTTP / RPC conventions in the bounded set; route LLM-specific span events (prompt content, tool-call payloads) through the trace backend, not metric labels, to keep metric cardinality bounded.
+
+## Common mistakes
+
+- Treating per-request fields as labels → cardinality explosion → metric backend OOMs.
+- Forgetting to add baseline labels manually because the metrics client wasn't used.
+- Using gauge for monotonic counters (loses rate semantics on restart).
+- Histogram with too few buckets (loses percentile precision) or too many (storage cost).
+- Recording metrics inside a tight loop without rate-limiting (collector receives bursts).

package/dist/assets/marketplace/plugins/ccl-skills/skills/platform-observability/references/obs-stack-architecture.md

@@ -0,0 +1,107 @@
+# Observability Stack Architecture
+
+Reference topology and component-level rules. Stack-agnostic; specific vendors named only as recurring examples.
+
+## Topology
+
+```
+          ┌──────────────────────────────────────────────────────────────┐
+          │                      Service Pod                              │
+          │  ┌─────────────┐  ┌─────────────────┐  ┌──────────────────┐ │
+          │  │ App + OTel  │  │ Mesh sidecar    │  │ Log writer       │ │
+          │  │ SDK         │  │ (if mesh used)  │  │ (zap → stdout    │ │
+          │  └────┬────────┘  └────┬────────────┘  │  + host file)    │ │
+          │       │ OTLP/gRPC      │ (mesh telem)  └────────┬─────────┘ │
+          └───────┼────────────────┼─────────────────────────┼───────────┘
+                  │                │                         │
+                  ▼                ▼                         ▼
+          ┌──────────────┐  ┌──────────────────┐    ┌──────────────────┐
+          │ OTel         │  │ Mesh control     │    │ File-collector   │
+          │ Collector    │  │ plane telemetry  │    │ DaemonSet        │
+          │ (Deployment, │  │  (Envoy stats)   │    │ (hostPath access)│
+          │  N replicas, │  └────────┬─────────┘    └─────────┬────────┘
+          │  NOT in mesh)│           │                        │
+          └──┬─────────┬─┘           │                        ▼
+             │ metrics │ traces      │              ┌──────────────────┐
+             ▼         ▼             ▼              │ Log pipeline     │
+        ┌─────────┐ ┌────────┐  ┌──────────┐        │ (logstash/vector,│
+        │ Scraper │ │ Trace  │  │ Mesh     │        │ JSON parse,      │
+        │ (Prom,  │ │ backend│  │ metrics  │        │ field promotion) │
+        │ remote_ │ │ (Jaeger│  │ scrape   │        └─────────┬────────┘
+        │ write)  │ │ /Tempo)│  └────┬─────┘                  │
+        └────┬────┘ └────┬───┘       │                        ▼
+             │           │           │              ┌──────────────────┐
+             ▼           │           │              │ Search index     │
+       ┌──────────┐      │           │              │ (ES/OpenSearch)  │
+       │ Long-term│◄─────┘           │              └─────────┬────────┘
+       │ metric   │                  │                        │
+       │ store    │◄─────────────────┘                        │
+       │ (VM/Mimir│                                           │
+       │ /Cortex) │                                           │
+       └────┬─────┘                                           │
+            │                                                 │
+            └────────────────┬────────────────────────────────┘
+                             ▼
+                  ┌──────────────────────┐
+                  │  Dashboard tool      │
+                  │  (Grafana/equivalent)│
+                  └──────────────────────┘
+```
+
+## Component Rules
+
+### OTel Collector
+- Deploy as Deployment with ≥ 3 replicas (not DaemonSet — DaemonSet wastes resources for OTLP, and you cannot tail-sample across nodes).
+- Pipelines: separate traces, metrics, logs pipelines in the collector config. Shared receivers, separate processors and exporters.
+- Processor minimum set: `batch` (always), `memory_limiter` (always), `tail_sampling` (if backend supports it), `resource` (to inject k8s.pod.name etc. when not on SDK).
+- Annotate the collector pods to **exclude from mesh** (e.g. `sidecar.istio.io/inject: "false"`). Telemetry must not depend on mesh data plane.
+- Resource limits: start at 250m CPU / 512Mi mem requests; 4 CPU / 8Gi limits. Tune from real load.
+- Listen on OTLP gRPC (4317) and OTLP HTTP (4318) at minimum.
+
+### Metrics path
+- App SDK exports via OTLP → collector → Prometheus exporter scrape OR direct remote_write to long-term store. A process picks ONE egress (OTLP push, or a Prom scrape endpoint), per the SKILL `Decision Points`; collector/scrape/remote_write is a platform back-stage choice.
+- Long-term store (VictoriaMetrics, Mimir, Cortex, or managed) MUST receive its samples from the collect/scrape layer (`remote_write` from a scraper, or collector `remote_write`/OTLP) and stay separate from it.
+- Dashboards and alerting query the long-term store, NOT the scraper. The scraper is fungible.
+- AlertManager is OPTIONAL — see `alerting-and-on-call.md` for the grafana-only pattern.
+
+### Trace path
+- SDK → collector (with batch + tail-sampling processor) → trace backend (Jaeger v2, Tempo, Zipkin, or a vendor).
+- Sampling: head-sample 1–10% in SDK; tail-sample errors + slow + named-route at 100% in collector.
+- Span attributes mandatory: `service.name`, `deployment.environment`, `k8s.pod.name`, baseline labels matching metrics.
+- Trace retention: hot 7 days, warm 30 days, cold optional. Most queries are < 24h old.
+
+### Log path
+- App writes JSON to **stdout** (preferred) OR to a known host file path (rotated).
+- File-collector DaemonSet reads from hostPath (e.g. `/opt/<org>/app/log` or `/var/log/containers`) → ships to log pipeline.
+- Log pipeline (logstash, vector, fluentbit) parses JSON, promotes nested fields to root, adds metadata (k8s namespace, pod, node).
+- Index by log-id and trace-id at minimum. Time-based index rollover (daily or larger).
+- Daemonset and collector pods MUST be excluded from mesh.
+
+### Dashboard tool
+- One tool joins metrics + traces + logs. Grafana is the de facto baseline.
+- Service overview dashboard template includes: QPS, latency p50/p90/p99, error rate, saturation (CPU/mem/queue depth), top-N hot endpoints, error budget burn.
+- Per-team dashboards inherit from the template; do not let teams rebuild the basics.
+
+## Anti-patterns to refuse
+
+- SDK → backend direct, no collector: loses batching, central sampling, retry buffer.
+- DaemonSet collector for OTLP: wastes resources, breaks tail sampling.
+- Mesh-injected collector: telemetry depends on the thing it observes.
+- Prometheus as long-term store: loses data on restart, hard to scale.
+- AlertManager + chat webhook + grafana alerting all running in parallel without ownership: alert duplication.
+- Per-pod log volume PVC: high cost, IO contention, fragility.
+
+## Cardinality 分层：alerting 用 bounded metrics，investigation 用 high-cardinality events
+
+业内 high-cardinality observability（Honeycomb 等，Majors / Fong-Jones / Miranda *Observability Engineering* 2022）核心洞察：**metrics 与 events 各管各事，不要混**。本 stack 按下面切分：
+
+- **Metrics（bounded label set）— alerting + 总览仪表盘**：label cardinality 严控（参 `metrics-conventions.md`），可聚合可比较，长期 retention 友好。回答的问题：**"系统现在 / 过去 N 段时间 是否健康"**
+- **Traces + structured log events（high-cardinality）— investigation**：可带 user_id / request_id / tenant_id / 任意 business attribute；不预聚合；按 trace_id 关联。回答的问题：**"为什么这一条请求慢 / 错了 / 为什么这个用户的工作流挂了"**
+- **Exemplars 桥接两者**：metric 数据点附带代表性 trace_id，看到 latency histogram bucket 异常时，可直接跳到一条对应的 trace 看现场。OTel metrics SDK 规范已稳定支持 exemplar；实际可用性取决于 SDK / exporter / backend 三方都启用 — 落地前确认 stack 链路完整
+
+**操作纪律**：
+- 高基数维度（user_id / tenant_id / request_id / order_id 等）**默认拒绝**进 metric label（一般会爆 cardinality）；按 `metrics-conventions.md` cardinality budget threshold 评估，仅 bounded / enumerated 维度（如 ≤ N 个固定 tenant tier）经 budget review 后允许；其余进 trace span attribute / structured log field
+- 报警基于 metric（bounded label + 聚合）；on-call 收到 alert 后用 trace / event 下钻
+- 不为"以备查问"在 metric 上预聚合每个业务维度 — 那是 trace / log 的职责
+
+参对 SLI/SLO（`sli-slo-design.md`）：SLI 是 metric 层（bounded），SLO/error budget 也是 metric 层；event-first investigation 是 SLI 报警触发后的下一步。

package/dist/assets/marketplace/plugins/ccl-skills/skills/platform-observability/references/sli-slo-design.md

@@ -0,0 +1,95 @@
+# SLI / SLO / Error Budget Design
+
+Industry baseline: Google SRE Workbook ch. 2–4 ("Implementing SLOs"). Use those formal definitions; this reference is the operational shortlist.
+
+## Define SLIs from user-visible journeys, not internal counters
+
+Bad SLI: "DB query success rate."
+Good SLI: "Successful checkout request rate (user clicks pay → response 2xx within 3 seconds)."
+
+For each top user journey, write at least:
+- **Availability SLI** — successful requests / total requests, in a measurement window.
+- **Latency SLI** — the proportion of requests faster than a threshold T: `count(latency ≤ T) / total` (from histogram buckets). A p95/p99 *value* is a dashboard aid, NOT the SLI (Google SRE: an SLI is good events / valid events).
+
+For queue-driven flows:
+- **Freshness SLI** — events processed within N seconds of arrival.
+- **Throughput SLI** — sustained events per second under acceptable backlog.
+
+## Concretize as a query against the long-term metric store
+
+Example (PromQL-style):
+
+```
+# Availability for journey X
+# Uses separate counters per framework-middleware-checklist.md:
+#   request_total       = baseline + route + method labels (no status_class)
+#   request_error_total = 5xx with error_class label
+1 - (
+  sum(rate(http_server_request_error_total{service="<svc>", route="<route>"}[5m]))
+  /
+  sum(rate(http_server_request_total{service="<svc>", route="<route>"}[5m]))
+)
+```
+
+```
+# Latency SLI: proportion of requests faster than threshold T (good / total) — NOT a percentile value
+sum(rate(http_server_request_duration_seconds_bucket{service="<svc>", route="<route>", le="<T>"}[5m]))
+/
+sum(rate(http_server_request_duration_seconds_count{service="<svc>", route="<route>"}[5m]))
+# (p95/p99 via histogram_quantile is a dashboard diagnostic, not the SLI input)
+```
+
+> The SLO threshold `T` MUST be a real explicit histogram bucket boundary (`le="<T>"` must exist); if it isn't, add/migrate the bucket boundary before claiming the SLI — otherwise the query is empty or silently uses a wrong nearby bucket.
+
+If you cannot write the query against existing metrics, the metric set is incomplete. Add the metric before claiming the SLI.
+
+## SLO target
+
+Pick a number that reflects user expectation and product maturity, not aspiration:
+- New service: 99.0% — generous.
+- Mature service: 99.9% — three nines.
+- Critical path (payment, login): 99.95% — push.
+- Never start with 99.99% without 24/7 staffing.
+
+Lower the target if every release burns the budget. Raise it only after sustained achievement.
+
+## Error budget
+
+`error_budget = 1 - SLO`. Tracked in the same window as the SLI (rolling 30 days is the default).
+
+Operational rule: if error budget is exhausted, the service stops shipping non-critical changes until the budget recovers. This makes SLOs actually mean something.
+
+## Burn-rate alerts
+
+Multi-window, multi-burn-rate per Google SRE Workbook:
+
+| Severity | Window | Burn rate | Meaning |
+|---|---|---|---|
+| Page (P0) | 1h | 14x | Will exhaust 30d budget in ~2 days at this rate |
+| Page (P0) | 5min | 14x | Same, but fast detection |
+| Notify (P1) | 6h | 6x | Will exhaust in 5 days |
+| Digest (P2) | 24h | 1x | Sustained at SLO threshold |
+
+Alert query template:
+
+```
+slo_burn_rate{...} > <threshold>
+```
+
+Each burn rate alert MUST link to a runbook entry.
+
+## Anti-patterns
+
+- SLI written as "DB up" — that is a probe, not a user-facing SLI.
+- SLO set to 100% — leaves no budget for releases.
+- Burn rate calculated over 1 minute → false-positive paging on natural noise.
+- One SLI per service — composite journeys lose visibility.
+- SLO without error-budget-driven release policy → number on a wall.
+
+## Releases consume budget
+
+The release skill (`platform-release-engineering`) MUST query the SLI / budget before promoting a canary. A canary is only promoted if:
+- Its burn rate during the bake window is within budget.
+- The cumulative budget for the release window is not exhausted.
+
+This is the handoff: observability owns the signal; release owns the gate.

package/dist/assets/marketplace/plugins/ccl-skills/skills/platform-observability/references/source-register.md

@@ -0,0 +1,11 @@
+# platform-observability — impact-chain / owner map
+
+Upstream-owner decision-surface changes record their downstream owners here
+(`upstream rule | downstream owner | expected executable behavior | status | evidence`).
+
+| upstream rule (this skill) | downstream / sibling owner | expected behavior there | status | evidence |
+|---|---|---|---|---|
+| R1 propagation trust boundary; Baggage carries non-authoritative context only; `lane`/routing derived from authenticated identity | `platform-service-connectivity` | mesh/gateway derives lane from identity; routing keys not trusted from inbound Baggage | routed (its domain) | W3C Baggage security note; OTel docs |
+| Phase C: non-mesh collector + STRICT mTLS conflict → exception required | `platform-service-connectivity` | owns the mTLS exception YAML (sidecar/ambient PeerAuthentication/DestinationRule); obs only requires it be declared | routed | Istio PeerAuthentication docs |
+| R8 SLI = good/valid events ratio; latency = threshold-bucket ratio | `platform-release-engineering` | error-budget release gate consumes these SLIs | unchanged (already references R8) | Google SRE Workbook |
+| R3 instance-identity not a metric label; instrument discipline | `go-microservice-architecture` / `python-service-architecture` | language-agnostic rule; no stack-specific glue needed | unchanged | Prometheus / OTel Metrics docs |