@ccoalm/ccl-skills 0.1.0

package/dist/assets/marketplace/plugins/ccl-skills/skills/platform-observability/references/infra-component-deployment.md

@@ -0,0 +1,268 @@
+# Infrastructure Component Deployment Patterns
+
+Concrete sizing, HA stance, retention, and in-cluster vs externally-hosted decisions for the observability infra stack. From production deployments.
+
+## Component-by-component shape
+
+| Component | Replicas | Stateful? | Storage | Retention | Mesh inject | Notes |
+|---|---|---|---|---|---|---|
+| OTel collector | 3-4 (Deployment) | No | None | n/a | NO | Multi-replica; tail-sampling possible only here, not at DaemonSet |
+| Filebeat (DaemonSet) | one per node | No | hostPath to app log dir | n/a | NO | hostPath mount required; one process per node |
+| Filebeat (sidecar, VCI mode) | one per pod | No | emptyDir | n/a | n/a (sidecar in app pod) | For serverless k8s where hostPath is unavailable |
+| Logstash | 2-5 (HPA on CPU>70% / Mem>80%) | No | None | n/a | NO | Bulk 500, 2 workers, 5s flush |
+| Elasticsearch / OpenSearch | 3+ | Yes | Persistent (often external) | per-ILM | NO | Often externally hosted (see below) |
+| Kibana | 1 | No | ConfigMap only | n/a | NO | Single replica fine — read-only UI |
+| Grafana | 1-2 | Yes (MySQL backend) | External MySQL for dashboards | dashboards forever | NO | Single replica per env in many platforms |
+| Prometheus (scrape layer) | 1-2 | Yes | Local PVC | short (days) | NO | scrape + relay only; do not use as long-term store |
+| VictoriaMetrics / Mimir / Cortex | 1+ (single-replica = dev only) | Yes | NAS RWX or block | 1 week to months | NO | Long-term store; separate from scraper; **single-replica is dev-grade only** — prod needs replication or VM-cluster mode |
+| AlertManager | 0 or 1 | No | Local | n/a | NO | Optional; grafana-only alerting is a real alternative |
+| Trace backend (Jaeger / Tempo) | 1-3 | Yes | Block storage | 7-30 days | NO | OTel-native collectors (Jaeger v2) include the backend |
+
+A few non-obvious choices:
+
+- **OTel collector as Deployment, not DaemonSet**: tail-sampling needs cross-node aggregation; DaemonSet can't see the full trace.
+- **Filebeat as DaemonSet** in node-attached k8s; as **sidecar** in serverless k8s (cloud-provider container instance). Same filebeat image, different deployment shape.
+- **Mesh injection OFF for observability infra components**: telemetry must not depend on mesh.
+
+## In-cluster vs externally-hosted
+
+Production platforms typically split:
+
+| Component | In-cluster | External | Rationale |
+|---|---|---|---|
+| OTel collector | YES | NO | App SDK → collector is in-cluster traffic |
+| Filebeat / Logstash | YES | NO | tight loop with app logs |
+| Kibana | YES | rare | UI is light |
+| Grafana | YES or external | sometimes external | If a dedicated obs team runs it, lives outside |
+| Elasticsearch | often EXTERNAL | sometimes in-cluster | High storage; many platforms run on bare metal or managed ES |
+| VictoriaMetrics | both | both | small platforms in-cluster; large platforms central VM cluster |
+| Prometheus scraper | YES | YES | sometimes a central cross-cluster scraper |
+
+External components reached via:
+
+```yaml
+# ExternalName for cloud-provider managed services (RDS, etc.)
+apiVersion: v1
+kind: Service
+metadata:
+  name: <mysql-or-equivalent>
+spec:
+  type: ExternalName
+  externalName: <cloud-provider-hostname>
+
+# Endpoints with explicit IPs for on-prem / fixed-IP services (ES bare-metal)
+apiVersion: v1
+kind: Service
+metadata:
+  name: es-cluster
+spec:
+  ports:
+    - { name: es, port: 9200, targetPort: 9200 }
+    - { name: inter-node, port: 9300, targetPort: 9300 }
+---
+apiVersion: v1
+kind: Endpoints
+metadata:
+  name: es-cluster
+subsets:
+  - addresses:
+      - { ip: <fixed-ip-1> }
+      - { ip: <fixed-ip-2> }
+      - { ip: <fixed-ip-3> }
+    ports:
+      - { name: es, port: 9200 }
+      - { name: inter-node, port: 9300 }
+```
+
+The Service + Endpoints pattern lets in-cluster pods reach external systems via the k8s service name as if they were in-cluster, without DNS resolution dependency on external DNS.
+
+## Online / Offline parallel deployments
+
+A platform may run **parallel obs stacks** for online (production) and offline (staging/dev) separation:
+
+| Concern | Online | Offline |
+|---|---|---|
+| ES cluster | external bare-metal IPs (set A) | external bare-metal IPs (set B) OR in-cluster |
+| Long-term metric store | external (separate cluster hosted at a stable hostname) | in-cluster VM single-replica |
+| Service names | `es-cluster`, `victoria-metrics-service` | `es-cluster-offline`, `victoria-metrics-service` (same name, different cluster context) |
+| Prom AlertManager | typically off (grafana-driven) | typically off |
+
+Same service names in different namespaces / environments let app config stay environment-neutral. Application reads `<service>.<infra-namespace>.svc.cluster.local:<port>` in both environments; the Service definition resolves to different upstreams.
+
+## Sizing reference
+
+Concrete starting points from real deployments (tune by load):
+
+```
+OTel collector:
+  resources:
+    requests: { cpu: 250m, memory: 512Mi }
+    limits:   { cpu: 4000m, memory: 8Gi }
+  4 replicas
+
+Filebeat DaemonSet:
+  resources: per-node, similar to app — small (1 cpu, 1Gi)
+
+Logstash:
+  resources:
+    requests: { cpu: 500m, memory: 1Gi }
+    limits:   { cpu: 2000m, memory: 2Gi }
+  2 replicas + HPA to 5
+  JVM heap: -Xms1g -Xmx1g
+  PIPELINE_WORKERS: 2
+  PIPELINE_BATCH_SIZE: 500
+  PIPELINE_BATCH_DELAY: 5
+
+Kibana:
+  resources:
+    requests: { cpu: 500m, memory: 1Gi }
+    limits:   { cpu: 1000m, memory: 2Gi }
+  1 replica
+
+Grafana:
+  resources:
+    requests: { cpu: 500m, memory: 1Gi }
+    limits:   { cpu: 1000m, memory: 2Gi }
+  1 replica per env, external MySQL backend for dashboards
+
+VictoriaMetrics (single-instance — DEV/NON-CRITICAL ONLY):
+  resources:
+    requests: { cpu: 1000m, memory: 2Gi }
+    limits:   { cpu: 4000m, memory: 8Gi }
+  Retention: 1 week to 30 days
+  Storage: 50Gi NAS RWX (allows snapshot DR)
+  WARNING: single-instance with 1-week retention loses ALL metrics on disk failure or
+  pod loss with corrupted PVC. For production, run VM-cluster mode (vmstorage 3+
+  replicas with replication factor 2+) OR send to a managed metrics service.
+  Single-instance is acceptable for dev/staging where losing a week's metrics
+  during an incident is an inconvenience, not a blocker.
+
+```
+
+## Log noise discipline: cleaner sidecar pattern
+
+Some observability infra components emit huge access / debug logs by default. Without cleanup, disk fills. Pattern:
+
+```yaml
+containers:
+  - name: <main-component>
+    # ... main spec
+    volumeMounts:
+      - { name: data, mountPath: <log-dir> }
+
+  - name: log-cleaner
+    image: <small-image-with-find-and-rm>:latest
+    resources:
+      requests: { cpu: 125m, memory: 256Mi }
+      limits:   { cpu: 500m, memory: 1Gi }
+    volumeMounts:
+      - { name: data, mountPath: <log-dir> }
+    command:
+      - "/bin/sh"
+      - "-c"
+      - |
+        while true; do
+          find <log-dir> -type f \( -name "*.log.*" -o -name "*[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9].log" \) \
+            -mtime +3 -print | while read f; do
+            rm -f "$f"
+          done
+          sleep 3600
+        done
+```
+
+Sleep 1 hour, delete files older than 3 days. The patterns match both `xxx.log.<date>.0` (logrotate format) and `xxx-<date>.log` (date-suffix format).
+
+**Critical safety**: do NOT mount the entire host log directory or share a mount with audit/security logs. The cleaner's broad glob (`*[0-9][0-9][0-9][0-9]-...`) will happily delete `audit.log.2026-01-15.0` during an incident investigation. Required restrictions:
+- Mount only the specific verbose component's log subdirectory; never a shared root.
+- Use `-xdev` (don't cross filesystem boundaries) and depth limits in `find`.
+- Maintain an allowlist of filenames the cleaner is permitted to remove, not just a date-pattern.
+- Emit a metric (`log_cleaner_files_deleted_total`) for monitoring; alert on anomalous deletion volume.
+- Document the cleaner's mount and pattern in the component's runbook so incident responders know what is auto-deleted.
+
+This is preferable to:
+- Disabling the verbose log entirely (loses incident-time data).
+- Per-deployment custom log-rotate config (drift; hard to audit).
+- Trusting the component's built-in rotation (often misconfigured; observed 1.9 GB / day from one registry component without cleanup).
+- **In-app log cleanup running as a daemon timer inside the application process** (the application starts a goroutine/thread that periodically `find <log-dir> -mtime +N -delete`). This is a recurring anti-pattern with multiple problems: it races with the platform's log rotation (logrotate, k8s log driver, fluent-bit tail), it deletes files that observability collectors are still reading and produces inconsistent log-truncation events, it requires the application container to have write+delete permission on its own log directory (broader than the standard read-by-collector posture), and it makes log-retention policy implicit in application binary instead of explicit in deployment config. Log retention is a platform concern; the application emits logs and the platform decides retention. Move app-internal log-cleanup timers to the sidecar pattern above or to the cluster's log driver.
+
+## Stateful component snapshot DR
+
+**Safety scope**: this pattern is for **whole-cluster DR** (lost data dir + lost quorum). It is NOT a single-replica repair tool. Misuse causes data corruption:
+- Single replica loses its PVC while the other replicas have newer state → init container restores stale snapshot into a live quorum → member-ID conflict, divergent log, or stale state rejoin → cluster split or data loss.
+- Therefore: gate the init container behind an **explicit DR marker** (e.g. ConfigMap key, annotation, env var set only by operator during DR scale-to-zero procedure).
+- Without the DR marker, init container exits 0 immediately. With the marker, it performs the restore — but only after confirming the StatefulSet is scaled to 0 first.
+- For single-replica repair (one pod lost PVC, others healthy): use the component's native `member remove + member add` workflow, not snapshot restore.
+
+For stateful observability infra such as a trace backend or long-term metric/log store that needs disaster recovery beyond replica redundancy:
+
+```yaml
+initContainers:
+  - name: restore-from-snapshot
+    image: <component-image>:<version>
+    env:
+      - { name: CLUSTER_SIZE, value: "3" }
+      - { name: SET_NAME, value: "<statefulset-name>" }
+      - name: NAMESPACE
+        valueFrom: { fieldRef: { fieldPath: metadata.namespace } }
+    command:
+      - "/bin/sh"
+      - "-exc"
+      - |
+        SNAPSHOT_DIR=/snapshot
+        SNAPSHOT_FILE=${SNAPSHOT_DIR}/<component>-snapshot.db
+        DATA_DIR=/data/<component>
+        RESTORE_LOCK=${SNAPSHOT_DIR}/.restore-lock
+
+        # 1. If data dir already populated, skip (normal restart).
+        if [ -d "${DATA_DIR}/member" ]; then exit 0; fi
+
+        # 2. If no snapshot, skip (first-time cluster bootstrap).
+        if [ ! -s "${SNAPSHOT_FILE}" ]; then exit 0; fi
+
+        # 3. Lock to prevent multiple pods restoring simultaneously.
+        if ! mkdir "${RESTORE_LOCK}" 2>/dev/null; then
+          # Another pod holds the lock; wait for it.
+          while [ -d "${RESTORE_LOCK}" ]; do sleep 2; done
+          if [ -d "${DATA_DIR}/member" ]; then exit 0; fi  # peer completed it
+          exit 1  # lock disappeared but data dir empty → retry
+        fi
+        trap "rm -rf \"${RESTORE_LOCK}\"" EXIT
+
+        # 4. Perform the restore.
+        <component-restore-command> "${SNAPSHOT_FILE}" --data-dir "${DATA_DIR}"
+    volumeMounts:
+      - { name: data, mountPath: /data/<component> }
+      - { name: snapshot, mountPath: /snapshot }
+
+volumes:
+  - name: snapshot
+    persistentVolumeClaim:
+      claimName: <shared-rwx-pvc>     # ReadWriteMany NAS volume
+
+volumeClaimTemplates:
+  - metadata: { name: data }
+    spec:
+      storageClassName: <nas-class>
+      accessModes: ["ReadWriteMany"]
+      resources: { requests: { storage: 50Gi } }
+```
+
+Critical patterns:
+- **DR marker gate** (see Safety scope above) before ANY restore action runs.
+- **Shared NAS PVC** (ReadWriteMany) for the snapshot file — all replicas read from the same place.
+- **File-lock for coordination** — `mkdir` is atomic; the first pod to create the lock dir wins. BUT: lock files MUST have a TTL or holder metadata. Pure `mkdir` lock is broken on holder crash: if the locking pod SIGKILLs mid-restore, peers wait forever for the lock dir to disappear. Mitigations:
+  - Write a heartbeat file inside the lock dir with timestamp + holder identity (`pod-name + uid`); peers check heartbeat age and force-release after `4× heartbeat_interval`.
+  - Or use a Kubernetes `Lease` resource (lease.coordination.k8s.io) which has built-in TTL and renewal — preferable over file-based locks for k8s-native operation.
+  - Document break-glass: when peers detect a stale lock holder (heartbeat > N minutes), operator manually deletes the lock dir after confirming the original holder is gone.
+- **Idempotent**: existing data dir means "already restored or already running" — skip.
+- **Restore command varies by component** (`etcdctl snapshot restore`, equivalent for others).
+
+Snapshot generation runs as a separate CronJob (not shown here) — periodic dump to the NAS volume.
+
+## Verification
+
+- Storage limits met: monitor disk usage per stateful component; alert at 70% / 85%.
+- Log cleaner sidecar working: file count + total size graph shows daily reset.
+- Snapshot DR drill: in non-prod, scale StatefulSet to 0, delete data PVCs, scale back; init container should restore from the latest snapshot within timeout (typically minutes).
+- Multi-replica check: scale a component to N replicas; kill 1; remaining serves; replica returns; cluster heals.
+- Cross-environment routing: same service name (e.g. `es-cluster`) resolves to different endpoints in offline vs online envs — pod's resolved IPs differ as expected.

package/dist/assets/marketplace/plugins/ccl-skills/skills/platform-observability/references/log-correlation-recipe.md

@@ -0,0 +1,124 @@
+# Log / Trace / Metrics Correlation Recipe
+
+Goal: a human (or agent) given one user complaint can find every log line, every trace span, every metric data point produced while that request was in flight — across every service hop.
+
+## The three IDs
+
+| ID | Purpose | Lifetime |
+|---|---|---|
+| `log-id` | Human-friendly correlation handle; the only id a user reports in a ticket. Generated at the edge if absent. | Per-request, immutable through all hops |
+| `trace-id` | OTel-standard 128-bit; joins spans into one trace | Per-request, OTel-managed |
+| `span-id` | Identifies one segment of work inside one service | Per-span |
+
+The platform MUST propagate all three through every transport.
+
+## Propagation transports
+
+### HTTP
+
+Inbound (server side):
+- Read `<log-id-header>` from request headers; if empty, generate (e.g. `time_ns + random + pod_suffix`).
+- Read OTel headers (`traceparent`, `tracestate`) via OTel HTTP instrumentation.
+- Inject all three into `context.Context` using platform-defined ctx keys.
+- Write `<log-id-header>` back into the response so the client can echo it in error reports.
+
+Outbound (client side):
+- Read the three IDs from `context.Context`.
+- Set `<log-id-header>` on the outgoing request.
+- Let OTel instrumentation set `traceparent`/`tracestate`.
+
+### RPC (Kitex/gRPC/Thrift)
+
+Inbound:
+- Same as HTTP, but read from RPC metadata instead of HTTP headers.
+- For Kitex specifically: the kitex-contrib OTel package handles trace headers; the platform adds log-id handling in a custom middleware.
+
+Outbound:
+- Pull the three IDs from ctx → write into RPC metadata.
+
+### Message Queue (Kafka, RocketMQ, etc.)
+
+Producer:
+- Inject the three IDs as message headers/attributes (e.g. Kafka headers, RocketMQ user properties).
+- Trace context per OTel messaging spec.
+
+Consumer:
+- Extract the three IDs from message attributes → put back into `context.Context` of the consume handler.
+- New span: kind=consumer, parent = extracted trace context.
+
+If the queue has no header support, encode as JSON envelope keys: `{ "_log_id": "...", "_traceparent": "...", "payload": {...} }`. Refuse to ship a service that swallows correlation ids at the queue boundary.
+
+### Composite propagator, format interop, and cross-stack key normalization
+
+Define the propagator **once** and install the **same** composite propagator across every stack, adapter, and transport (HTTP / RPC / MQ) — not per-adapter. A real fleet is heterogeneously instrumented, so per-adapter divergence in format support, key spelling, or propagator order silently breaks trace-join at the boundary between two services (a Go caller and a Python callee drop each other's context). Footguns, each a repeated fix:
+
+- **CompositePropagator `extract` is LAST-WINS.** When both W3C `traceparent` and `b3` arrive, the propagator listed **last** wins the conflict. The intuitive order `[TraceContext, b3, Baggage]` makes **b3** win — the opposite of "W3C wins". To make W3C win, put TraceContext **last**: `[b3, Baggage, TraceContext]`. Pin it with an explicit W3C-vs-b3 conflict test; never assume the list reads highest-priority-first.
+- **Accept every format on extract; inject W3C only — but preserve sampling and sequence the rollout.** *Extract* reads W3C `traceparent`/`tracestate` AND `b3` (single + multi header) for b3-speaking meshes / older services. *Inject* must emit **W3C only** — do NOT leave the `b3` propagator in the inject path, or a single bidirectional composite re-emits both `traceparent` and `b3`. Use an extract-only b3 wrapper (or a composite whose injector is W3C-only); on a b3-only inbound, downstream then carries exactly one canonical `traceparent`, no `b3`. When canonicalizing b3→W3C, **map the sampling/debug bit** (`X-B3-Sampled` / b3 debug flag → W3C `trace-flags` sampled bit) or downstream makes wrong sampling decisions — and handle b3's third state, **absent sampling header = *defer***: do NOT flatten defer to `trace-flags=00` (a `ParentBased` sampler would then drop every span as "remote-not-sampled"); on defer, make a fresh local sampling decision. Also **carry `tracestate`** through every hop — including the headerless-MQ JSON envelope (`_tracestate` alongside `_traceparent`), or vendor sampling/routing state is lost at the queue boundary. Rollout is ordered: deploy W3C+b3 **extraction** (or boundary bridges) everywhere **first**, then flip egress to W3C-only — a service that goes W3C-only before its b3-only consumers upgrade drops the trace for them.
+- **Normalize metadata/header keys per transport — don't blindly lowercase, and reject collisions.** Keys must canonicalize the same way across every stack/adapter, or a key set by one is missed by the other — but canonicalize by the **transport's own rule**: HTTP headers / CGI / gRPC metadata are case-insensitive (canonicalize, or a differently-cased key is silently dropped — this bit a gRPC-metainfo bridge and an MQ header dedup path), while some MQ properties are case-**sensitive** (blind lowercasing corrupts them). After normalization, **reject duplicate / colliding propagation keys** rather than last-wins them — a collision otherwise lets a bridge or attacker choose which `traceparent`/`b3` wins. Preserve carrier order where a transport allows multiple carriers.
+- **This is non-authoritative correlation, not identity — and untrusted at the external edge.** Accepting inbound trace context does not relax the edge trust boundary: never derive `lane` / routing / authz from inbound propagation, and validate/regenerate `log-id` at the external edge (SKILL trust-boundary rule). At an **untrusted external ingress do not blindly adopt inbound `traceparent`/`b3` as the parent** — an attacker can graft unrelated requests into one trace-id (trace poisoning / mis-join); start a new root span and *link* to the remote context (or gate parenting behind a trusted-peer check), and strip/validate external `tracestate`. b3/W3C trace context is correlation evidence, not a trust carrier.
+
+Verification: (a) send both `traceparent` and a conflicting `b3` → the trace uses the **W3C** id (pinned conflict test); (b) send `b3` only → downstream carries a canonical `traceparent`, no `b3` re-emitted, and `X-B3-Sampled=0/1/debug` maps to the right `trace-flags`; (c) cross-stack — a Go service injects, a Python service extracts, assert the same trace id **and** `tracestate` join (and the reverse), including over an MQ envelope; (d) send a key in non-canonical case → still extracted; send duplicate/colliding keys → rejected, not last-wins; (e) at an untrusted edge, a forged inbound `traceparent` starts a new root (linked), not a child of the attacker's trace.
+
+## Logger field contract
+
+Every log line, regardless of language, MUST include these fields in the structured JSON:
+
+```
+_ts          ISO8601 timestamp with timezone
+_level       OTel severity text (TRACE/DEBUG/INFO/WARN/ERROR/FATAL)
+_msg         Message body
+_logid       Log id (above)
+_trace_id    OTel trace id (hex)
+_span_id     OTel span id (hex)
+_trace_flags OTel trace flags
+_service     Service identity (PSM-style or service.name)
+_lane / _env Environment label
+_idc         Region / data center
+_cluster     Kubernetes cluster
+_pod         Pod name
+_stack       Stack trace (ERROR and above only)
+```
+
+Domain fields use `app_*` or `biz_*` prefix to avoid collisions.
+
+## Logger extraction recipe
+
+The logger MUST automatically pull `_logid`, `_trace_id`, `_span_id`, `_trace_flags` from `context.Context`. The developer interface looks like:
+
+```go
+logs.CtxInfo(ctx, "ordered %d items", n)
+```
+
+Internally:
+1. Pull `log-id` from ctx via platform ctx-key.
+2. Pull current OTel span from ctx; pull trace-id, span-id, trace-flags from span context.
+3. Attach all four as structured fields on the log record.
+4. Emit JSON.
+
+If a developer can write `logs.Info("ordered %d", n)` (without `ctx`) and the line still ships to prod without trace fields, the framework is broken. Make ctx mandatory or make the no-ctx form a lint failure.
+
+Inject correlation fields at the record factory or on the emitting handler — never via a filter attached to an ancestor/root *logger*: under stdlib logging semantics a record that propagates up from a child logger does NOT run an ancestor logger's own filters, so an ancestor-logger filter silently drops the fields on every propagating child (the common path). A filter on a *handler* is different — propagated records DO reach ancestor/root handlers and run their filters, so a handler-attached filter is a valid seam. Placement trade-off per key: a record-factory injection covers every path but a caller-supplied `extra` of the same key collides (use for ids that must always win); a handler filter/processor runs late, after record creation, so it survives propagation and yields to a caller override (use for overridable defaults) but must be attached to every handler path. Verify with a child logger that propagates to root.
+
+## Trace ↔ Log join
+
+In kibana / OpenSearch / equivalent:
+- Saved search: `_trace_id:"<id>"` → all log lines from one trace.
+- Saved search: `_logid:"<id>"` → human-friendly alternative.
+- Dashboard panel: log volume by service, drill-down to a specific trace.
+
+In Jaeger / Tempo / equivalent:
+- Span attribute `log_id` mirrored from the request → click span → see linked log lines in the log tool.
+
+The two tools MUST be cross-linked in the dashboard so on-call can pivot in one click.
+
+## Verification
+
+End-to-end smoke test (run after every major framework change):
+
+1. Inject a synthetic request with no log-id header.
+2. Confirm response carries a generated log-id.
+3. Search log index by that log-id → expect ≥ 1 hop. For a known multi-hop path, expect all hops.
+4. Pick any line's `_trace_id` → search trace tool → expect a trace with > 1 span.
+5. Each span's `service.name` should match a `_service` value in the log search.
+
+If any step fails, the correlation chain is broken; do not ship.

package/dist/assets/marketplace/plugins/ccl-skills/skills/platform-observability/references/log-schema-canonical.md

@@ -0,0 +1,208 @@
+# Canonical Log Schema (Concrete Field Set)
+
+The authoritative field list for structured platform logs, derived from a working production deployment. Use as a template; localize header names but preserve the field shape.
+
+## Required fields (every log line)
+
+| Field | Type | Purpose |
+|---|---|---|
+| `_ts` | keyword | ISO-8601 timestamp with timezone (e.g. `2026-05-20T14:23:01.123+08:00`) |
+| `_datetime` | date | Parsed timestamp for the search backend |
+| `_timestamp` | long | Unix ms (Asia/Shanghai or UTC; document the choice) |
+| `_level` | keyword | TRACE / DEBUG / INFO / WARN / ERROR / FATAL (OTel-spec severity text) |
+| `_msg` | text | Message body |
+| `_logid` | keyword | Platform log-id (correlation handle, generated at edge if absent) |
+| `_trace_id` | keyword | OTel trace id (hex) |
+| `_span_id` | keyword | OTel span id (hex) |
+| `_trace_flags` | keyword | OTel trace flags |
+| `_psm` | keyword | Service identity (PSM-style or `service.name` OTel attr) |
+| `_lane` / `_env` | text | Lane / environment label |
+| `_idc` | text | Region / data center |
+| `_cluster` | text | Kubernetes cluster name |
+| `_pod` | text | Pod name |
+| `_addr` | ip | Service IP (pod IP) |
+| `_caller` | text | Source file:line where the log was emitted (file.go:123 form) |
+| `_language` | text | Programming language (go, python, java, ...) |
+
+Fields prefixed `_` are platform-reserved. Application fields use `app_*` or `biz_*` prefix to prevent collision.
+
+`_stack` (type `text`) is set only for ERROR and above; contains the full stack trace.
+
+## Why each field
+
+- `_logid` — human-friendly correlation id; the only id a user reports in a ticket.
+- `_trace_id` + `_span_id` — cross-service trace join; OTel-managed.
+- `_caller` — source location; powerful for diff-based canary checks (see `platform-release-engineering/references/lane-orchestration-control-plane.md` § canary algorithm). Two services that emit "DB timeout" from different code lines are correctly distinguished.
+- `_language` — multi-language platforms need this to group / filter log queries; also useful for stack-format detection.
+- `_addr` / `_pod` — drill from "lane has problem" → "this specific pod" → "this specific replica".
+- `_idc` / `_cluster` — multi-region / multi-cluster drilldown.
+
+## Index template (Elasticsearch / OpenSearch)
+
+```yaml
+setup.template:
+  enabled: true
+  overwrite: true
+  pattern: "bizlogs-*"        # localize prefix
+  fields: fields.yml
+  order: 140
+  settings:
+    index.number_of_shards: 3
+    _source.enabled: true
+  mappings:
+    dynamic: true
+    dynamic_date_formats: "strict_date_optional_time"
+    date_detection: true
+```
+
+3 shards is a reasonable default for cluster sizing 10-50 nodes. Adjust based on ingest rate.
+
+## ILM policy
+
+Hot 7 days, then delete is a common defensible default for biz logs:
+
+```yaml
+policy:
+  phases:
+    hot:
+      min_age: "0ms"
+      actions:
+        rollover:
+          max_age: "7d"
+          max_size: "30gb"
+    delete:
+      min_age: "7d"
+      actions:
+        delete: {}
+```
+
+Adjust by:
+- **Compliance** — some industries require 90 days+ of audit logs (set a separate index/policy for audit-class logs).
+- **Cost** — warm tier (cheaper storage) for 30-90 days is a middle ground.
+- **Investigation cadence** — if incidents are typically detected and root-caused within 7d, 7d hot is sufficient.
+
+Separate ILM policies for `audit-*` (long retention), `bizlogs-*` (7d), `debug-*` (1-3d) prevents pinning all retention to the strictest requirement.
+
+## Filebeat configuration shape
+
+```yaml
+filebeat.inputs:
+  - type: log
+    id: bizlogs
+    enabled: true
+    paths:
+      - /opt/<org>/app/log/*.log
+      - /opt/<org>/app/log/*/*.log
+      - /opt/<org>/app/log/*/*/*.log         # per-service/per-lane subdirs
+    fields_under_root: true
+    json:
+      keys_under_root: true
+      overwrite_keys: false                  # CRITICAL: see "Platform-field protection" below
+      add_error_key: true
+
+processors:
+  - add_host_metadata:
+      when.not.contains.tags: forwarded
+  - script:
+      lang: javascript
+      id: extract_ts
+      file: ${path.config}/extract_ts.js   # custom timestamp parser if needed
+  - timestamp:
+      field: "_timestamp"
+      target_field: "_datetime"
+      layouts: ["UNIX_MS"]
+      timezone: "Asia/Shanghai"            # or UTC
+```
+
+`fields_under_root: true` + `json.keys_under_root: true` together mean: the JSON log line's fields become root-level fields in the indexed document. So `_logid` from the log → `_logid` in Elasticsearch. No nested `parsed.<field>` digging.
+
+### Platform-field protection
+
+**Do NOT set `overwrite_keys: true` with `keys_under_root: true` for reserved fields.** A compromised or malicious service could emit `{"_lane":"prod", "_level":"INFO", "_msg":"..."}` from a non-prod pod, and the log pipeline would index it as if it came from prod with INFO severity — corrupting canary queries, audit searches, and incident triage.
+
+Enforcement options (pick at least one):
+1. **Logstash filter strips/renames `_`-prefixed fields supplied by the app**, then re-injects authoritative `_lane`, `_psm`, `_idc`, `_cluster`, `_pod`, `_addr` from metadata the pipeline knows for sure (e.g. from k8s pod annotations the filebeat shipper adds).
+2. **Filebeat `processors.drop_fields`** removes a denylist of app-supplied platform fields before they hit logstash.
+3. **Application logger contract**: SDK is the only code path that writes `_`-prefixed fields; lint rule rejects app code that touches them directly.
+
+Layered defense is best: SDK contract (#3) for the common case + pipeline strip (#1) as the durable guardrail in case the SDK is bypassed.
+
+## Log path layout
+
+Services write structured JSON logs to disk (not stdout in this pattern). Layout:
+
+```
+/opt/<org>/app/log/
+  <psm-1>/
+    <lane-A>/
+      app.log
+      app-error.log
+    <lane-B>/
+      ...
+  <psm-2>/
+    ...
+```
+
+Why files not stdout (in this pattern):
+- Per-service per-lane separation: cleaner debugging.
+- File rotation (size + time) is well-tooled.
+- Filebeat reads efficiently with offset tracking.
+- App can keep writing during transient filebeat outages (file buffer absorbs).
+
+Why stdout in other patterns (mature alternative):
+- k8s-native; logs visible to `kubectl logs`.
+- No volume mount needed.
+- Cloud-native log shippers (Cloud Logging, etc.) usually pull from stdout.
+
+Pick one per platform; do not mix.
+
+## In-pod sidecar vs DaemonSet collection
+
+Two patterns:
+
+**DaemonSet** (one filebeat per node, hostPath mount):
+```
+Node
+├─ filebeat (DaemonSet pod)
+│   └─ reads /opt/<org>/app/log/* via hostPath
+└─ App pods write to /opt/<org>/app/log/<psm>/<lane>/
+```
+Pros: one filebeat process per node, low overhead.
+Cons: hostPath access required; not compatible with serverless k8s.
+
+**Sidecar** (one filebeat per app pod):
+```
+App Pod
+├─ app container       writes to emptyDir
+└─ filebeat container  reads emptyDir
+```
+Pros: works in serverless k8s (no hostPath); per-pod resource accounting.
+Cons: filebeat container per pod multiplies resource cost.
+
+Many platforms use **both**, selected by pod environment: serverless k8s (e.g. cloud-provider container instance) → sidecar; node-attached k8s → DaemonSet.
+
+## Output endpoints by environment
+
+Online (production) and offline (staging/dev) typically have different log backends:
+
+```yaml
+output:
+  elasticsearch:
+    hosts: [{{ online_es_host or offline_es_host }}]
+
+setup.kibana:
+  host: {{ online_kibana_host or offline_kibana_host }}
+```
+
+Reasons to split:
+- Cost: offline ES can be smaller, single-node, no replicas.
+- Isolation: a dev mistake bursting logs doesn't pressure prod ES.
+- Compliance: prod logs may need stricter access control.
+
+## Verification
+
+- Open a service log line in kibana → all required fields present, no missing fields.
+- Search `_trace_id:<id>` → returns all hops of one trace.
+- Search `_caller:<source-location>` → returns the same error pattern across the fleet.
+- ILM rollover triggers at 7d / 30gb (test by writing high volume to a non-critical index).
+- Cluster failure of online ES → filebeat buffers locally; resumes when ES returns.