@moon791017/neo-skills 1.1.12 → 1.1.13
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2 -0
- package/package.json +1 -1
- package/skills/neo-opentelemetry/SKILL.md +94 -0
- package/skills/neo-opentelemetry/evals/eval_queries.json +82 -0
- package/skills/neo-opentelemetry/evals/evals.json +64 -0
- package/skills/neo-opentelemetry/references/collector-architecture-and-configuration.md +138 -0
- package/skills/neo-opentelemetry/references/collector-operations.md +137 -0
- package/skills/neo-opentelemetry/references/compatibility-and-guidance.md +112 -0
- package/skills/neo-opentelemetry/references/foundations.md +98 -0
- package/skills/neo-opentelemetry/references/platforms-and-zero-code.md +108 -0
- package/skills/neo-opentelemetry/references/security.md +119 -0
- package/skills/neo-opentelemetry/references/signals-and-sampling.md +113 -0
- package/skills/neo-opentelemetry/references/sources-and-scope.md +60 -0
package/README.md
CHANGED
|
@@ -60,6 +60,7 @@
|
|
|
60
60
|
| Vue | `neo-vue` | 建置、除錯、重構或審查 Vue 3、SFC、Composition API、Pinia、Vue Router、Vite 與 Vue+TypeScript。 |
|
|
61
61
|
| Agent 架構 | `neo-sub-agent` | 設計、建立、審查或轉換 sub-agent、custom agent、worker/reviewer/planner agent 或 multi-agent workflow。 |
|
|
62
62
|
| Agent 架構 | `neo-agentic-design` | 設計、評估或實作 Agent 工作流、提示詞鏈、路由、規劃、反思、多 Agent 協作與記憶體管理等框架無關模式。 |
|
|
63
|
+
| 可觀測性 | `neo-opentelemetry` | 解釋 OpenTelemetry 概念、規劃或審查 Collector 架構與設定、處理維運排障、安全、零程式 instrumentation 及相容性遷移。 |
|
|
63
64
|
| 文字潤飾 | `neo-stop-slop` | 去除繁中或英文中的 AI 腔、贅詞、公式化句式,支援文件、註解、commit message 與 PR 說明。 |
|
|
64
65
|
|
|
65
66
|
## 安裝
|
|
@@ -155,6 +156,7 @@ npx -p @moon791017/neo-skills install-system-instructions \
|
|
|
155
156
|
| 改善 AI 開發流程 | `neo-agent-harness` | `評估這個專案讓 coding agent 協作的可靠度` |
|
|
156
157
|
| 建立 sub-agent | `neo-sub-agent` | `幫我新增一個 Codex code-reviewer sub agent` |
|
|
157
158
|
| 設計 Agent 編排架構 | `neo-agentic-design` | `幫我設計一個多 Agent 客服系統的拓撲結構與重試機制` |
|
|
159
|
+
| 規劃 OpenTelemetry | `neo-opentelemetry` | `幫我審查 Collector 架構、韌性、安全與零程式觀測方案` |
|
|
158
160
|
| 去掉 AI 腔 | `neo-stop-slop` | `把這段 PR 說明改得自然、直接一點` |
|
|
159
161
|
|
|
160
162
|
## 開發
|
package/package.json
CHANGED
|
@@ -0,0 +1,94 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: neo-opentelemetry
|
|
3
|
+
description: >
|
|
4
|
+
Use this skill when explaining OpenTelemetry concepts, choosing telemetry signals,
|
|
5
|
+
planning or reviewing Collector architectures and declarative configuration,
|
|
6
|
+
operating or troubleshooting telemetry pipelines, securing telemetry data,
|
|
7
|
+
evaluating zero-code instrumentation, or planning Prometheus, OpenCensus, or
|
|
8
|
+
OpenTracing interoperability and migration. Do not use it to write language SDK/API
|
|
9
|
+
instrumentation code or custom Collector components.
|
|
10
|
+
compatibility: No runtime required. Network access is required to verify current versions, component status, security advisories, and other time-sensitive claims.
|
|
11
|
+
metadata:
|
|
12
|
+
version: "1.0.0"
|
|
13
|
+
pattern: "tool-wrapper,pipeline"
|
|
14
|
+
domain: "observability-operations"
|
|
15
|
+
source-revision: "b18eb4d01fab9be2c5c22b1aaafdeb23ba97e3d4"
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
# Neo OpenTelemetry
|
|
19
|
+
|
|
20
|
+
Provide source-grounded OpenTelemetry guidance for concepts and operations without generating programming-language instrumentation code.
|
|
21
|
+
|
|
22
|
+
## Scope boundary
|
|
23
|
+
|
|
24
|
+
In scope:
|
|
25
|
+
|
|
26
|
+
- Observability concepts, telemetry signals, correlation, resources, semantic conventions, and sampling.
|
|
27
|
+
- Collector pipelines, components, deployment patterns, declarative configuration, scaling, resiliency, internal telemetry, transformation, and troubleshooting.
|
|
28
|
+
- Kubernetes, FaaS, OpenTelemetry Operator, OBI, and other zero-code operational approaches.
|
|
29
|
+
- Security, sensitive-data handling, Prometheus interoperability, and OpenCensus/OpenTracing migration planning.
|
|
30
|
+
|
|
31
|
+
Out of scope:
|
|
32
|
+
|
|
33
|
+
- Language-specific SDK/API usage, manual instrumentation, source-code examples, and library integration code.
|
|
34
|
+
- Building the Collector from source or developing custom receivers, processors, exporters, connectors, or extensions.
|
|
35
|
+
- Vendor pricing, backend-specific query languages, and backend product administration unless needed only as context for an OpenTelemetry decision.
|
|
36
|
+
|
|
37
|
+
If the request crosses the boundary, answer the in-scope operational portion and identify the excluded portion. Point to the appropriate official language or component-development documentation instead of inventing code.
|
|
38
|
+
|
|
39
|
+
## Workflow
|
|
40
|
+
|
|
41
|
+
1. Classify the request as explanation, architecture decision, configuration review, security review, migration, or troubleshooting.
|
|
42
|
+
2. Load `references/sources-and-scope.md` whenever the answer depends on source coverage, current status, or an excluded area.
|
|
43
|
+
3. Load only the references required by the request:
|
|
44
|
+
- Foundations and terminology: `references/foundations.md`
|
|
45
|
+
- Signals and sampling: `references/signals-and-sampling.md`
|
|
46
|
+
- Collector topology and configuration: `references/collector-architecture-and-configuration.md`
|
|
47
|
+
- Collector operations and troubleshooting: `references/collector-operations.md`
|
|
48
|
+
- Kubernetes, FaaS, Operator, and zero-code: `references/platforms-and-zero-code.md`
|
|
49
|
+
- Security and sensitive data: `references/security.md`
|
|
50
|
+
- Prometheus, migration, and deployment guidance: `references/compatibility-and-guidance.md`
|
|
51
|
+
4. Inspect user-provided configuration and environment evidence before recommending changes. Never assume topology, volume, failure tolerance, trust boundaries, or backend capabilities.
|
|
52
|
+
5. Verify time-sensitive facts against official OpenTelemetry sources. If verification is unavailable, identify the snapshot revision and state that current status cannot be confirmed.
|
|
53
|
+
6. Produce the smallest useful response and cite the official page supporting material claims.
|
|
54
|
+
|
|
55
|
+
## Decision rules
|
|
56
|
+
|
|
57
|
+
- Treat OpenTelemetry as a telemetry generation, collection, processing, and export framework, not as a storage or visualization backend.
|
|
58
|
+
- Default to the simplest deployment pattern that satisfies the stated requirements. Add gateways, stateful routing, persistent queues, or external message queues only for an identified need.
|
|
59
|
+
- Separate symptoms by pipeline stage: generation, reception, processing, queuing, export, network, or backend.
|
|
60
|
+
- Treat sampling, filtering, redaction, and transformation as data-loss or data-quality decisions. State what is intentionally discarded or changed.
|
|
61
|
+
- Never recommend scaling Collectors before checking whether the bottleneck is the Collector, network, or backend.
|
|
62
|
+
- Never place credentials, tokens, personal data, or other sensitive values in examples.
|
|
63
|
+
- Prefer least privilege, restricted listener addresses, authenticated and encrypted network paths, data minimization, and Collector self-observability.
|
|
64
|
+
|
|
65
|
+
## Response contracts
|
|
66
|
+
|
|
67
|
+
For explanations, provide:
|
|
68
|
+
|
|
69
|
+
1. Direct answer.
|
|
70
|
+
2. Relevant OpenTelemetry concepts and boundaries.
|
|
71
|
+
3. Official source links.
|
|
72
|
+
|
|
73
|
+
For architecture or configuration reviews, provide:
|
|
74
|
+
|
|
75
|
+
1. Observed requirements or configuration facts.
|
|
76
|
+
2. Findings ordered by impact.
|
|
77
|
+
3. Recommended topology or declarative changes with trade-offs.
|
|
78
|
+
4. Capacity, resiliency, security, and validation checks.
|
|
79
|
+
5. Official source links.
|
|
80
|
+
|
|
81
|
+
For troubleshooting, provide:
|
|
82
|
+
|
|
83
|
+
1. Observed symptom and missing evidence.
|
|
84
|
+
2. Most likely pipeline stage based on supplied evidence; label this as an inference.
|
|
85
|
+
3. Non-destructive checks ordered from source to destination.
|
|
86
|
+
4. Remediation only after the failing stage is isolated.
|
|
87
|
+
5. Verification signals and official source links.
|
|
88
|
+
|
|
89
|
+
## Fact-check gate
|
|
90
|
+
|
|
91
|
+
- Distinguish verified source facts from recommendations and inferences.
|
|
92
|
+
- Do not convert documentation examples into universal defaults.
|
|
93
|
+
- Do not state version numbers, maturity levels, component availability, default ports, default limits, or CVE status without checking the current official source when the answer depends on them.
|
|
94
|
+
- Say that data is insufficient when the request lacks evidence needed for a safe operational conclusion.
|
|
@@ -0,0 +1,82 @@
|
|
|
1
|
+
[
|
|
2
|
+
{
|
|
3
|
+
"query": "Explain when I should use traces, metrics, logs, and profiles together in OpenTelemetry.",
|
|
4
|
+
"should_trigger": true
|
|
5
|
+
},
|
|
6
|
+
{
|
|
7
|
+
"query": "Review our OpenTelemetry Collector YAML for missing queues, unsafe listeners, and processor ordering.",
|
|
8
|
+
"should_trigger": true
|
|
9
|
+
},
|
|
10
|
+
{
|
|
11
|
+
"query": "Should our Kubernetes telemetry pipeline use Collector agents, gateways, or both?",
|
|
12
|
+
"should_trigger": true
|
|
13
|
+
},
|
|
14
|
+
{
|
|
15
|
+
"query": "Our otel gateway queue keeps filling and spans are being dropped. Help me isolate the bottleneck.",
|
|
16
|
+
"should_trigger": true
|
|
17
|
+
},
|
|
18
|
+
{
|
|
19
|
+
"query": "Design a resilient OpenTelemetry Collector topology that can tolerate a backend outage.",
|
|
20
|
+
"should_trigger": true
|
|
21
|
+
},
|
|
22
|
+
{
|
|
23
|
+
"query": "Check whether our baggage and resource attributes could leak customer data.",
|
|
24
|
+
"should_trigger": true
|
|
25
|
+
},
|
|
26
|
+
{
|
|
27
|
+
"query": "Plan a zero-code observability rollout for Linux workloads using the Operator or OBI.",
|
|
28
|
+
"should_trigger": true
|
|
29
|
+
},
|
|
30
|
+
{
|
|
31
|
+
"query": "How should we migrate Prometheus metric collection into an OTLP and Collector architecture?",
|
|
32
|
+
"should_trigger": true
|
|
33
|
+
},
|
|
34
|
+
{
|
|
35
|
+
"query": "Create an operational migration plan from OpenTracing to OpenTelemetry without breaking dashboards.",
|
|
36
|
+
"should_trigger": true
|
|
37
|
+
},
|
|
38
|
+
{
|
|
39
|
+
"query": "Why are traces split after they cross a service boundary even though both services export spans?",
|
|
40
|
+
"should_trigger": true
|
|
41
|
+
},
|
|
42
|
+
{
|
|
43
|
+
"query": "Write C# ActivitySource instrumentation for my ASP.NET Core checkout endpoint.",
|
|
44
|
+
"should_trigger": false
|
|
45
|
+
},
|
|
46
|
+
{
|
|
47
|
+
"query": "Show me the Java OpenTelemetry SDK calls needed to create a custom histogram.",
|
|
48
|
+
"should_trigger": false
|
|
49
|
+
},
|
|
50
|
+
{
|
|
51
|
+
"query": "Implement a custom OpenTelemetry Collector receiver in Go.",
|
|
52
|
+
"should_trigger": false
|
|
53
|
+
},
|
|
54
|
+
{
|
|
55
|
+
"query": "Build the OpenTelemetry Collector from source with a custom exporter.",
|
|
56
|
+
"should_trigger": false
|
|
57
|
+
},
|
|
58
|
+
{
|
|
59
|
+
"query": "Write a PromQL query for calculating HTTP error rate in Grafana.",
|
|
60
|
+
"should_trigger": false
|
|
61
|
+
},
|
|
62
|
+
{
|
|
63
|
+
"query": "Troubleshoot why my Kubernetes Deployment cannot pull its application image.",
|
|
64
|
+
"should_trigger": false
|
|
65
|
+
},
|
|
66
|
+
{
|
|
67
|
+
"query": "Compare the current pricing of Datadog, New Relic, and Honeycomb.",
|
|
68
|
+
"should_trigger": false
|
|
69
|
+
},
|
|
70
|
+
{
|
|
71
|
+
"query": "Create OpenAPI documentation for my REST endpoints.",
|
|
72
|
+
"should_trigger": false
|
|
73
|
+
},
|
|
74
|
+
{
|
|
75
|
+
"query": "Help me submit a pull request to translate the OpenTelemetry website.",
|
|
76
|
+
"should_trigger": false
|
|
77
|
+
},
|
|
78
|
+
{
|
|
79
|
+
"query": "Design a generic application logging API that does not use OpenTelemetry.",
|
|
80
|
+
"should_trigger": false
|
|
81
|
+
}
|
|
82
|
+
]
|
|
@@ -0,0 +1,64 @@
|
|
|
1
|
+
{
|
|
2
|
+
"skill_name": "neo-opentelemetry",
|
|
3
|
+
"evals": [
|
|
4
|
+
{
|
|
5
|
+
"id": 1,
|
|
6
|
+
"prompt": "We operate a checkout service and only have CPU dashboards. Explain which OpenTelemetry signals would help us detect reliability problems and determine why one request is slow. Do not provide instrumentation code.",
|
|
7
|
+
"expected_output": "A source-grounded explanation that distinguishes metrics, traces, logs, and profiles, describes correlation, and stays outside language SDK/API implementation.",
|
|
8
|
+
"assertions": [
|
|
9
|
+
"The response explains a distinct operational question answered by metrics, traces, logs, and profiles",
|
|
10
|
+
"The response explains that resource identity or trace context is needed for correlation",
|
|
11
|
+
"The response contains at least one official opentelemetry.io documentation link",
|
|
12
|
+
"The response contains no programming-language instrumentation code"
|
|
13
|
+
]
|
|
14
|
+
},
|
|
15
|
+
{
|
|
16
|
+
"id": 2,
|
|
17
|
+
"prompt": "Plan an OpenTelemetry Collector deployment for 60 Kubernetes nodes. We need node logs and host metrics, centralized redaction, one controlled egress point, and tail sampling for traces. State assumptions where capacity data is missing.",
|
|
18
|
+
"expected_output": "An agent-to-gateway architecture with node-local agents, centralized gateway processing, trace-affine routing for tail sampling, explicit security and capacity caveats, and verification steps.",
|
|
19
|
+
"assertions": [
|
|
20
|
+
"The response assigns node logs and host metrics to node-local agents",
|
|
21
|
+
"The response assigns centralized redaction, credentials, and backend export to gateways",
|
|
22
|
+
"The response states that every span for a trace must reach the same tail-sampling instance",
|
|
23
|
+
"The response labels capacity recommendations as unresolved because telemetry volume is not provided",
|
|
24
|
+
"The response includes security, resiliency, and rollout verification checks"
|
|
25
|
+
]
|
|
26
|
+
},
|
|
27
|
+
{
|
|
28
|
+
"id": 3,
|
|
29
|
+
"prompt": "Review this Collector configuration:\n\nreceivers:\n otlp:\n protocols:\n grpc:\n endpoint: 0.0.0.0:4317\nprocessors:\n batch:\nexporters:\n otlp:\n endpoint: vendor.example:4317\n headers:\n api-key: hard-coded-example-secret\nservice:\n pipelines:\n traces:\n receivers: [otlp]\n processors: [batch]\n exporters: [otlp]\n\nIdentify operational and security risks, but do not assume whether remote clients require the unrestricted listener.",
|
|
30
|
+
"expected_output": "A configuration review that identifies the unrestricted listener, embedded secret, missing memory protection, queue/retry and TLS questions, while treating topology-dependent changes as conditional.",
|
|
31
|
+
"assertions": [
|
|
32
|
+
"The response identifies the embedded credential as a critical issue and recommends protected runtime secret injection",
|
|
33
|
+
"The response identifies 0.0.0.0 as an exposure requiring topology evidence rather than replacing it unconditionally",
|
|
34
|
+
"The response checks memory limiting, batching, queueing, retries, TLS, and Collector self-observability",
|
|
35
|
+
"Findings are ordered by impact and distinguish facts from conditional recommendations",
|
|
36
|
+
"The response includes official Collector configuration and security links"
|
|
37
|
+
]
|
|
38
|
+
},
|
|
39
|
+
{
|
|
40
|
+
"id": 4,
|
|
41
|
+
"prompt": "During a backend outage our Collector exporter queue reaches capacity, enqueue failures increase, and queued data disappears when gateway pods restart. Give us a diagnosis and a resilient remediation sequence.",
|
|
42
|
+
"expected_output": "A diagnosis grounded in queue saturation and in-memory loss, followed by bounded retry/queue tuning, persistent storage evaluation, monitoring, capacity checks, and an optional external queue only when stronger durability justifies it.",
|
|
43
|
+
"assertions": [
|
|
44
|
+
"The response identifies queue saturation and pod restart with in-memory buffering as observed loss mechanisms",
|
|
45
|
+
"The response recommends measuring arrival rate, outage duration, queue capacity, memory, disk, and backend recovery before sizing",
|
|
46
|
+
"The response discusses persistent sending queues and their disk and permission trade-offs",
|
|
47
|
+
"The response treats an external message queue as an optional higher-complexity durability layer",
|
|
48
|
+
"The response defines post-change verification using queue, failure, restart, and export evidence"
|
|
49
|
+
]
|
|
50
|
+
},
|
|
51
|
+
{
|
|
52
|
+
"id": 5,
|
|
53
|
+
"prompt": "We want zero-code telemetry for mixed workloads: Kubernetes services, Linux VMs, and AWS Lambda. Produce a decision framework that covers the Operator, OBI, Collector agents, and Lambda layers. Current versions and runtime details are not supplied.",
|
|
54
|
+
"expected_output": "A platform-specific decision framework that compares supported zero-code approaches, flags missing compatibility evidence, includes least-privilege and rollout checks, and avoids unsupported version claims or language code.",
|
|
55
|
+
"assertions": [
|
|
56
|
+
"The response maps the Operator to Kubernetes, OBI or Collector agents to compatible Linux environments, and published layers to supported Lambda runtimes",
|
|
57
|
+
"The response states that runtime, kernel, architecture, region, artifact, and current-version compatibility must be verified",
|
|
58
|
+
"The response includes least-privilege, network, secret, performance, and rollback checks",
|
|
59
|
+
"The response identifies that zero-code methods cannot guarantee custom business telemetry",
|
|
60
|
+
"The response contains no manual instrumentation code"
|
|
61
|
+
]
|
|
62
|
+
}
|
|
63
|
+
]
|
|
64
|
+
}
|
|
@@ -0,0 +1,138 @@
|
|
|
1
|
+
# Collector Architecture and Configuration
|
|
2
|
+
|
|
3
|
+
Use this reference to choose a Collector topology or review declarative Collector configuration. Component availability and configuration keys vary by distribution and version; verify them in the current component documentation.
|
|
4
|
+
|
|
5
|
+
## Pipeline model
|
|
6
|
+
|
|
7
|
+
A Collector pipeline moves one signal type through:
|
|
8
|
+
|
|
9
|
+
1. Receivers that accept or obtain telemetry.
|
|
10
|
+
2. Optional processors that transform, enrich, batch, sample, filter, or drop telemetry.
|
|
11
|
+
3. Exporters that send telemetry to a destination.
|
|
12
|
+
|
|
13
|
+
Connectors join two pipelines while acting as an exporter for one and a receiver for another. Extensions add operational capabilities such as health checks, storage, authentication, or service discovery and are enabled separately from telemetry pipelines.
|
|
14
|
+
|
|
15
|
+
A minimal declarative shape is:
|
|
16
|
+
|
|
17
|
+
```yaml
|
|
18
|
+
receivers:
|
|
19
|
+
otlp:
|
|
20
|
+
protocols:
|
|
21
|
+
grpc:
|
|
22
|
+
|
|
23
|
+
processors:
|
|
24
|
+
memory_limiter:
|
|
25
|
+
check_interval: 1s
|
|
26
|
+
limit_mib: 512
|
|
27
|
+
batch:
|
|
28
|
+
|
|
29
|
+
exporters:
|
|
30
|
+
otlp:
|
|
31
|
+
endpoint: telemetry-backend.example:4317
|
|
32
|
+
|
|
33
|
+
service:
|
|
34
|
+
pipelines:
|
|
35
|
+
traces:
|
|
36
|
+
receivers: [otlp]
|
|
37
|
+
processors: [memory_limiter, batch]
|
|
38
|
+
exporters: [otlp]
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
This is a structural example, not a production-ready universal default. Authentication, TLS, listener addresses, queueing, resource limits, and component support must be decided for the target environment.
|
|
42
|
+
|
|
43
|
+
## Configuration mechanics
|
|
44
|
+
|
|
45
|
+
- Define components at the top level, then activate them by referencing them from a pipeline or the service extension list.
|
|
46
|
+
- Use `type/name` identifiers when more than one instance of a component type is needed.
|
|
47
|
+
- A defined component that is not referenced by `service` is not enabled.
|
|
48
|
+
- Signal types in a pipeline must be supported by every referenced component.
|
|
49
|
+
- Processor order is behaviorally significant.
|
|
50
|
+
- Multiple configuration sources can be merged, but the merged result must still be complete.
|
|
51
|
+
- Use environment-variable substitution for deployment-supplied values; never embed secrets in the Skill, repository examples, or review output.
|
|
52
|
+
|
|
53
|
+
Validate before deployment:
|
|
54
|
+
|
|
55
|
+
```shell
|
|
56
|
+
otelcol validate --config=collector.yaml
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
Confirm the executable and subcommand for the selected Collector distribution rather than assuming every distribution uses the same binary name.
|
|
60
|
+
|
|
61
|
+
## Reuse and fan-out gotcha
|
|
62
|
+
|
|
63
|
+
A receiver can feed multiple pipelines and an exporter can receive from multiple pipelines. Receiver fan-out is synchronous: a blocking processor in one attached pipeline can block the other pipelines and the receiver. Do not reuse a receiver across pipelines without evaluating this failure coupling.
|
|
64
|
+
|
|
65
|
+
Processors referenced by multiple pipelines use the same configuration but each pipeline gets its own processor instance and state.
|
|
66
|
+
|
|
67
|
+
## Deployment patterns
|
|
68
|
+
|
|
69
|
+
### No Collector
|
|
70
|
+
|
|
71
|
+
Direct export has the fewest moving parts but couples telemetry delivery and backend changes to the instrumented workload. Use it for simple development or constrained cases only after accepting the limited processing, exporter, and operational flexibility.
|
|
72
|
+
|
|
73
|
+
### Agent
|
|
74
|
+
|
|
75
|
+
An agent runs close to workloads, commonly as a host process, sidecar, or Kubernetes DaemonSet.
|
|
76
|
+
|
|
77
|
+
Choose an agent when you need:
|
|
78
|
+
|
|
79
|
+
- Host-local metrics or logs.
|
|
80
|
+
- Local enrichment with host or Kubernetes resource attributes.
|
|
81
|
+
- A stable local telemetry destination.
|
|
82
|
+
- Reduced application responsibility for credentials and backend export.
|
|
83
|
+
|
|
84
|
+
Agents multiply configuration and resource overhead across hosts or workloads. Keep their processing focused.
|
|
85
|
+
|
|
86
|
+
### Gateway
|
|
87
|
+
|
|
88
|
+
A gateway is a standalone Collector service shared by multiple telemetry sources.
|
|
89
|
+
|
|
90
|
+
Choose a gateway when you need:
|
|
91
|
+
|
|
92
|
+
- Centralized credentials, filtering, transformation, or policy.
|
|
93
|
+
- Central processing that must observe traffic from multiple sources.
|
|
94
|
+
- Independent scaling of telemetry processing.
|
|
95
|
+
- A controlled egress point to external backends.
|
|
96
|
+
|
|
97
|
+
Gateways add a network hop and a shared failure domain. Load balance stateless gateways and monitor them as production services.
|
|
98
|
+
|
|
99
|
+
### Agent to gateway
|
|
100
|
+
|
|
101
|
+
Combine agents and gateways when both local collection and centralized processing are required. Typical separation:
|
|
102
|
+
|
|
103
|
+
- Agents: local reception, host collection, resource detection, memory protection, and efficient forwarding.
|
|
104
|
+
- Gateways: centralized filtering, tail sampling, batching, credentials, and resilient export.
|
|
105
|
+
|
|
106
|
+
This pattern is justified by identified needs such as network isolation, host telemetry, central policy, trace-aware sampling, or scale. It is not the default for small deployments.
|
|
107
|
+
|
|
108
|
+
## Stateful routing rules
|
|
109
|
+
|
|
110
|
+
- Tail sampling requires all spans for a trace to reach the same decision-making instance.
|
|
111
|
+
- Cumulative-to-delta conversion and some aggregation use cases also require data-aware routing.
|
|
112
|
+
- Prometheus scraping requires target allocation or sharding to avoid duplicate collection.
|
|
113
|
+
- Metric streams must preserve the single-writer principle and globally unique identities.
|
|
114
|
+
- Standard layer-4 load balancing can be ineffective for persistent gRPC connections; verify that the load-balancing method distributes the actual connection pattern.
|
|
115
|
+
|
|
116
|
+
## Configuration review checklist
|
|
117
|
+
|
|
118
|
+
1. Identify each enabled pipeline and its signal type.
|
|
119
|
+
2. Verify every referenced component exists in the selected distribution.
|
|
120
|
+
3. Trace data from each receiver through processors to every exporter.
|
|
121
|
+
4. Inspect processor order and intentional data drops.
|
|
122
|
+
5. Check listener scope, authentication, TLS, and secret injection.
|
|
123
|
+
6. Check memory limiting, batching, queueing, retry, and persistent storage requirements.
|
|
124
|
+
7. Check stateful processors and their routing assumptions.
|
|
125
|
+
8. Check resource identity and duplicate-writer risk.
|
|
126
|
+
9. Check Collector internal telemetry, health checks, and alerting.
|
|
127
|
+
10. Validate the merged configuration with the target executable.
|
|
128
|
+
|
|
129
|
+
## Official sources
|
|
130
|
+
|
|
131
|
+
- [Collector architecture](https://opentelemetry.io/docs/collector/architecture/)
|
|
132
|
+
- [Collector configuration](https://opentelemetry.io/docs/collector/configuration/)
|
|
133
|
+
- [Collector components](https://opentelemetry.io/docs/collector/components/)
|
|
134
|
+
- [Choose a deployment pattern](https://opentelemetry.io/docs/collector/deploy/choose/)
|
|
135
|
+
- [Agent deployment pattern](https://opentelemetry.io/docs/collector/deploy/agent/)
|
|
136
|
+
- [Gateway deployment pattern](https://opentelemetry.io/docs/collector/deploy/gateway/)
|
|
137
|
+
- [Agent-to-gateway deployment pattern](https://opentelemetry.io/docs/collector/deploy/other/agent-to-gateway/)
|
|
138
|
+
- [No Collector](https://opentelemetry.io/docs/collector/deploy/other/no-collector/)
|
|
@@ -0,0 +1,137 @@
|
|
|
1
|
+
# Collector Operations and Troubleshooting
|
|
2
|
+
|
|
3
|
+
Use this reference for installation planning, self-observability, capacity, resiliency, transformation, and incident diagnosis.
|
|
4
|
+
|
|
5
|
+
## Installation and distribution checks
|
|
6
|
+
|
|
7
|
+
Before using any component or command, confirm:
|
|
8
|
+
|
|
9
|
+
- Target operating system or Kubernetes environment.
|
|
10
|
+
- Collector distribution and version.
|
|
11
|
+
- Included receivers, processors, exporters, connectors, and extensions.
|
|
12
|
+
- Configuration file location and executable name.
|
|
13
|
+
- Required network access, filesystem access, capabilities, and credentials.
|
|
14
|
+
|
|
15
|
+
Do not assume the core and contrib distributions contain the same components. Registry listings and component maturity are time-sensitive and require current verification.
|
|
16
|
+
|
|
17
|
+
## Internal telemetry
|
|
18
|
+
|
|
19
|
+
The Collector must expose enough telemetry to distinguish its own saturation from upstream, network, or backend failures. Monitor at least:
|
|
20
|
+
|
|
21
|
+
- Accepted and refused telemetry by receiver and signal.
|
|
22
|
+
- Processor drops and refusals.
|
|
23
|
+
- Export successes, permanent failures, and enqueue failures.
|
|
24
|
+
- Exporter queue size and capacity.
|
|
25
|
+
- CPU, memory, and restart behavior.
|
|
26
|
+
- Request latency and errors for downstream destinations.
|
|
27
|
+
- Scrape duration relative to scrape interval for pull-based receivers.
|
|
28
|
+
|
|
29
|
+
Metric names can change across versions. Confirm the current names before creating alerts; use the names documented by the installed version.
|
|
30
|
+
|
|
31
|
+
## Scaling decision flow
|
|
32
|
+
|
|
33
|
+
1. Split demand by signal, protocol, and collection behavior.
|
|
34
|
+
2. Identify whether components are stateless, scrapers, or stateful.
|
|
35
|
+
3. Check refusal, queue, resource, and downstream latency evidence.
|
|
36
|
+
4. Determine whether the constraint is Collector CPU/memory, a scrape schedule, the network, or the backend.
|
|
37
|
+
5. Scale only the constrained tier and preserve stateful routing rules.
|
|
38
|
+
|
|
39
|
+
Scaling patterns:
|
|
40
|
+
|
|
41
|
+
- Stateless receivers and processors can usually scale horizontally behind a suitable load balancer.
|
|
42
|
+
- Scrapers require target sharding or allocation to prevent duplicate collection.
|
|
43
|
+
- Stateful processors require affinity and careful redistribution behavior.
|
|
44
|
+
- Signal-specific Collector tiers can isolate failures and scale independently.
|
|
45
|
+
|
|
46
|
+
Do not add Collectors when the exporter queue remains full because the backend or network cannot accept more traffic. That can increase pressure without increasing throughput.
|
|
47
|
+
|
|
48
|
+
## Resiliency layers
|
|
49
|
+
|
|
50
|
+
### In-memory sending queue
|
|
51
|
+
|
|
52
|
+
Buffers transient downstream failures and works with retries. Data can still be lost when the queue fills, retry limits expire, or the Collector terminates.
|
|
53
|
+
|
|
54
|
+
### Persistent sending queue
|
|
55
|
+
|
|
56
|
+
A storage extension can back the sending queue with a write-ahead log so queued data survives Collector restarts. It introduces disk capacity, permissions, corruption, and lifecycle requirements.
|
|
57
|
+
|
|
58
|
+
```yaml
|
|
59
|
+
extensions:
|
|
60
|
+
file_storage:
|
|
61
|
+
directory: /var/lib/otelcol/storage
|
|
62
|
+
|
|
63
|
+
exporters:
|
|
64
|
+
otlp:
|
|
65
|
+
endpoint: telemetry-backend.example:4317
|
|
66
|
+
sending_queue:
|
|
67
|
+
storage: file_storage
|
|
68
|
+
|
|
69
|
+
service:
|
|
70
|
+
extensions: [file_storage]
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
Treat the directory, retention, queue size, and retry duration as workload-specific settings.
|
|
74
|
+
|
|
75
|
+
### External message queue
|
|
76
|
+
|
|
77
|
+
An external queue can decouple Collector tiers and provide stronger durability across outages. It also creates another production system with its own availability, retention, security, ordering, and capacity requirements. Add it only when the required durability and failure isolation justify the operational cost.
|
|
78
|
+
|
|
79
|
+
## Data-loss checklist
|
|
80
|
+
|
|
81
|
+
Check for:
|
|
82
|
+
|
|
83
|
+
- Downstream unavailability exceeding retry duration.
|
|
84
|
+
- Full sending queues or persistent storage.
|
|
85
|
+
- Collector restart with only in-memory buffering.
|
|
86
|
+
- Disk failure or exhausted persistent storage.
|
|
87
|
+
- External queue outage or retention expiration.
|
|
88
|
+
- Receiver, processor, exporter, or routing misconfiguration.
|
|
89
|
+
- Sampling or filtering that intentionally discards data.
|
|
90
|
+
- Load-balancing that splits state required by a processor.
|
|
91
|
+
- Backend rejection, throttling, or duplicate metric writers.
|
|
92
|
+
|
|
93
|
+
## Transformation safety
|
|
94
|
+
|
|
95
|
+
Transformations can normalize, enrich, redact, filter, or route telemetry. Before applying one:
|
|
96
|
+
|
|
97
|
+
1. Define the exact operational purpose.
|
|
98
|
+
2. Identify affected signals and attributes.
|
|
99
|
+
3. Preserve resource identity and semantic meaning.
|
|
100
|
+
4. Quantify records that will be dropped or rewritten.
|
|
101
|
+
5. Test representative and malformed input.
|
|
102
|
+
6. Monitor transform errors and output volume after rollout.
|
|
103
|
+
|
|
104
|
+
Prefer allowlists for sensitive attributes and explicit filters for unwanted data. Avoid transformations that silently collapse distinct metric streams or break trace correlation.
|
|
105
|
+
|
|
106
|
+
## Troubleshooting sequence
|
|
107
|
+
|
|
108
|
+
Diagnose from source to destination:
|
|
109
|
+
|
|
110
|
+
1. Confirm the source is producing the expected signal.
|
|
111
|
+
2. Confirm network reachability and protocol compatibility to the receiver.
|
|
112
|
+
3. Confirm the receiver accepts records and attaches expected resource data.
|
|
113
|
+
4. Inspect processor refusals, errors, sampling, filtering, and transformations.
|
|
114
|
+
5. Inspect queue occupancy, retries, enqueue failures, and storage health.
|
|
115
|
+
6. Inspect exporter errors, TLS, authentication, endpoint, and timeouts.
|
|
116
|
+
7. Confirm the backend accepts, indexes, and exposes the data.
|
|
117
|
+
|
|
118
|
+
Useful non-destructive actions include validating the active configuration, raising Collector log detail temporarily, using diagnostic extensions or a debug exporter in a controlled environment, and comparing accepted versus exported counts. Do not enable verbose telemetry in production without considering sensitive data and volume.
|
|
119
|
+
|
|
120
|
+
## Change verification
|
|
121
|
+
|
|
122
|
+
- Validate configuration before restart.
|
|
123
|
+
- Roll out to a limited Collector subset when possible.
|
|
124
|
+
- Compare accepted, dropped, queued, retried, and exported telemetry before and after.
|
|
125
|
+
- Confirm resource identity, trace completeness, metric continuity, and log timestamps.
|
|
126
|
+
- Confirm rollback conditions and configuration are ready.
|
|
127
|
+
|
|
128
|
+
## Official sources
|
|
129
|
+
|
|
130
|
+
- [Collector installation](https://opentelemetry.io/docs/collector/install/)
|
|
131
|
+
- [Collector internal telemetry](https://opentelemetry.io/docs/collector/internal-telemetry/)
|
|
132
|
+
- [Collector management](https://opentelemetry.io/docs/collector/management/)
|
|
133
|
+
- [Scaling the Collector](https://opentelemetry.io/docs/collector/scaling/)
|
|
134
|
+
- [Collector resiliency](https://opentelemetry.io/docs/collector/resiliency/)
|
|
135
|
+
- [Transforming telemetry](https://opentelemetry.io/docs/collector/transforming-telemetry/)
|
|
136
|
+
- [Collector troubleshooting](https://opentelemetry.io/docs/collector/troubleshooting/)
|
|
137
|
+
- [Collector registry](https://opentelemetry.io/docs/collector/registry/)
|
|
@@ -0,0 +1,112 @@
|
|
|
1
|
+
# Compatibility, Migration, and Platform Guidance
|
|
2
|
+
|
|
3
|
+
Use this reference for Prometheus interoperability, OpenCensus/OpenTracing migration planning, and organization-level telemetry-platform decisions. Programming-language migration code remains outside this skill.
|
|
4
|
+
|
|
5
|
+
## Prometheus interoperability
|
|
6
|
+
|
|
7
|
+
Prometheus is built around discovering and scraping targets. OpenTelemetry also supports transporting metrics through OTLP. Newer Prometheus versions can receive OTLP metrics, but the official OpenTelemetry guidance recommends a Collector intermediary for production OpenTelemetry deployments.
|
|
8
|
+
|
|
9
|
+
Key differences to preserve during migration or translation:
|
|
10
|
+
|
|
11
|
+
- Prometheus traditionally uses pull-based scraping; OTLP commonly pushes telemetry.
|
|
12
|
+
- Metric naming, units, suffixes, and resource-to-label mapping can change during translation.
|
|
13
|
+
- Prometheus metrics are cumulative; OpenTelemetry supports more than one aggregation temporality, while Prometheus-facing export requires compatible cumulative behavior.
|
|
14
|
+
- OpenTelemetry resource identity maps into Prometheus target identity and related metadata.
|
|
15
|
+
- Instrumentation-scope labels can distinguish otherwise identical metric names.
|
|
16
|
+
- Duplicate writers or duplicate scrapes can create invalid or out-of-order series.
|
|
17
|
+
|
|
18
|
+
Operational patterns:
|
|
19
|
+
|
|
20
|
+
- Use Prometheus receivers when the Collector must scrape existing targets.
|
|
21
|
+
- Use the Target Allocator or another sharding mechanism when multiple Collectors scrape Kubernetes targets.
|
|
22
|
+
- Use a Collector between OTLP producers and Prometheus for central policy, conversion, buffering, and diagnostics.
|
|
23
|
+
- Verify the current Prometheus OTLP receiver requirements before recommending direct ingestion.
|
|
24
|
+
|
|
25
|
+
## Migration principles
|
|
26
|
+
|
|
27
|
+
Treat migration as a telemetry-contract change, not only a deployment change.
|
|
28
|
+
|
|
29
|
+
1. Inventory existing signals, protocols, resource identity, propagation, baggage, dashboards, alerts, SLOs, and retention rules.
|
|
30
|
+
2. Define the target OTLP and Collector topology.
|
|
31
|
+
3. Establish a compatibility or overlap period when supported.
|
|
32
|
+
4. Migrate one bounded telemetry path at a time.
|
|
33
|
+
5. Compare old and new volume, identity, attributes, trace continuity, metric continuity, dashboards, and alerts.
|
|
34
|
+
6. Update downstream consumers before removing the old feed.
|
|
35
|
+
7. Remove compatibility layers only after acceptance criteria pass.
|
|
36
|
+
|
|
37
|
+
### OpenTracing
|
|
38
|
+
|
|
39
|
+
OpenTelemetry documents an incremental migration path using compatibility shims, but shim and language details are outside this skill and must be checked in current language documentation.
|
|
40
|
+
|
|
41
|
+
Operational risks that remain in scope:
|
|
42
|
+
|
|
43
|
+
- Semantic-convention changes can break dashboards and alerts.
|
|
44
|
+
- Baggage behavior differs and can lose propagation when old and new mechanisms are mixed.
|
|
45
|
+
- Context-management differences can fragment traces.
|
|
46
|
+
- Dual export can duplicate telemetry and increase cost.
|
|
47
|
+
- A Collector transformation can provide a temporary compatibility view, but it must have a removal plan.
|
|
48
|
+
|
|
49
|
+
### OpenCensus
|
|
50
|
+
|
|
51
|
+
The local documentation entry redirects to the current OpenTelemetry migration guidance rather than containing an operational procedure. Verify the current official migration page and target-language support before planning implementation. Do not invent a generic shim or compatibility guarantee.
|
|
52
|
+
|
|
53
|
+
## Organization-level platform guidance
|
|
54
|
+
|
|
55
|
+
### Non-Kubernetes environments
|
|
56
|
+
|
|
57
|
+
Common challenges include fragmented agent deployment, inconsistent instrumentation, and siloed export paths. A managed approach can:
|
|
58
|
+
|
|
59
|
+
- Define baseline telemetry and resource-attribution standards.
|
|
60
|
+
- Centrally manage agent lifecycle with controlled workload overrides.
|
|
61
|
+
- Use a Collector gateway layer for shared policy and backend export.
|
|
62
|
+
- Separate global defaults, environment policy, and workload-specific configuration.
|
|
63
|
+
- Monitor configuration rollout and agent/Collector health.
|
|
64
|
+
|
|
65
|
+
### Kubernetes environments
|
|
66
|
+
|
|
67
|
+
A managed telemetry platform can:
|
|
68
|
+
|
|
69
|
+
- Distribute consistent defaults through the Operator, Helm, or internal platform automation.
|
|
70
|
+
- Give application teams controlled override points without duplicating complete configurations.
|
|
71
|
+
- Maintain shared gateway tiers for policy, credentials, and backend routing.
|
|
72
|
+
- Process and sample at the layer that has the required data scope.
|
|
73
|
+
- Establish joint ownership among platform, observability, security, and workload teams.
|
|
74
|
+
|
|
75
|
+
Avoid one unbounded shared configuration for every signal and team. Separate pipelines or tiers where load, failure domains, state, or ownership differ.
|
|
76
|
+
|
|
77
|
+
## Reference implementation lessons
|
|
78
|
+
|
|
79
|
+
Official reference implementations describe real organizations, not universal blueprints. Extract constraints and trade-offs rather than copying their topology.
|
|
80
|
+
|
|
81
|
+
Recurring lessons include:
|
|
82
|
+
|
|
83
|
+
- Start with a simple path and add tiers only when requirements justify them.
|
|
84
|
+
- Centralize lifecycle management and safe defaults.
|
|
85
|
+
- Keep workload-facing configuration minimal and extensible.
|
|
86
|
+
- Separate high-volume or stateful processing from simple collection.
|
|
87
|
+
- Plan for component deprecation and controlled upgrades.
|
|
88
|
+
- Monitor the telemetry platform itself.
|
|
89
|
+
|
|
90
|
+
When citing a reference implementation, label the design as that organization's experience rather than an OpenTelemetry requirement.
|
|
91
|
+
|
|
92
|
+
## Migration acceptance criteria
|
|
93
|
+
|
|
94
|
+
- Required signals arrive with stable service and resource identity.
|
|
95
|
+
- Trace context remains continuous across migrated boundaries.
|
|
96
|
+
- Metric names, units, temporality, and labels meet the target contract.
|
|
97
|
+
- Dashboards, alerts, and SLOs have equivalent or intentionally changed behavior.
|
|
98
|
+
- Expected telemetry volume, cardinality, and cost remain within limits.
|
|
99
|
+
- Sensitive-data and access controls remain effective.
|
|
100
|
+
- Rollback and overlap behavior are tested.
|
|
101
|
+
- Old exporters, shims, routes, and dashboards have an explicit retirement point.
|
|
102
|
+
|
|
103
|
+
## Official sources
|
|
104
|
+
|
|
105
|
+
- [OpenTelemetry compatibility](https://opentelemetry.io/docs/compatibility/)
|
|
106
|
+
- [Prometheus interoperability](https://opentelemetry.io/docs/compatibility/prometheus/)
|
|
107
|
+
- [OTLP metrics export to Prometheus](https://opentelemetry.io/docs/compatibility/prometheus/otlp-metrics-export/)
|
|
108
|
+
- [Migrating from OpenCensus](https://opentelemetry.io/docs/compatibility/migration/opencensus/)
|
|
109
|
+
- [Migrating from OpenTracing](https://opentelemetry.io/docs/compatibility/migration/opentracing/)
|
|
110
|
+
- [Infrastructure and processes in non-Kubernetes environments](https://opentelemetry.io/docs/guidance/blueprints/instrumenting-applications-and-processes-on-nonk8s-environments/)
|
|
111
|
+
- [Managed telemetry platforms for Kubernetes workloads](https://opentelemetry.io/docs/guidance/blueprints/managed-telemetry-platforms-for-k8s-workloads/)
|
|
112
|
+
- [Reference implementations](https://opentelemetry.io/docs/guidance/reference-implementations/)
|
|
@@ -0,0 +1,98 @@
|
|
|
1
|
+
# OpenTelemetry Foundations
|
|
2
|
+
|
|
3
|
+
Use this reference for definitions, system boundaries, correlation, resources, semantic conventions, and terminology. Do not use it for language-specific instrumentation.
|
|
4
|
+
|
|
5
|
+
## What OpenTelemetry is
|
|
6
|
+
|
|
7
|
+
OpenTelemetry is an open-source, vendor- and tool-agnostic framework and toolkit for generating, collecting, processing, and exporting telemetry such as traces, metrics, and logs. It is not an observability backend: storage, querying, and visualization are provided by other systems.
|
|
8
|
+
|
|
9
|
+
This distinction prevents two common category errors:
|
|
10
|
+
|
|
11
|
+
- Choosing OpenTelemetry does not choose the telemetry storage or user interface.
|
|
12
|
+
- Changing a backend does not necessarily require changing how telemetry is represented or transported when both sides support OpenTelemetry standards.
|
|
13
|
+
|
|
14
|
+
OpenTelemetry's operational building blocks include the OpenTelemetry Protocol (OTLP), semantic conventions, automatic instrumentation, the Collector, the Kubernetes Operator, Helm charts, and FaaS assets. Programming-language APIs, SDKs, and manual instrumentation are intentionally outside this skill.
|
|
15
|
+
|
|
16
|
+
## Observability and reliability
|
|
17
|
+
|
|
18
|
+
Observability is the ability to reason about a system's internal state from its outputs, including novel failure modes that were not predicted in advance. Telemetry is the emitted data used for that reasoning.
|
|
19
|
+
|
|
20
|
+
Reliability asks whether a service delivers the behavior its users expect. Availability alone is not sufficient. A Service Level Indicator (SLI) measures behavior, ideally from the user's perspective; a Service Level Objective (SLO) connects one or more SLIs to an organizational reliability target.
|
|
21
|
+
|
|
22
|
+
Use signals together:
|
|
23
|
+
|
|
24
|
+
- Metrics reveal aggregate behavior and trends.
|
|
25
|
+
- Traces explain the path and timing of individual requests.
|
|
26
|
+
- Logs record discrete events and become more useful when correlated with traces.
|
|
27
|
+
- Profiles identify where resources are consumed and can be correlated with other signals.
|
|
28
|
+
|
|
29
|
+
## Context and propagation
|
|
30
|
+
|
|
31
|
+
Context contains information used to correlate work across service or execution boundaries. Propagation moves that context between processes or services by serializing it into a carrier and recovering it downstream. The default OpenTelemetry propagator uses W3C Trace Context headers.
|
|
32
|
+
|
|
33
|
+
Operational consequences:
|
|
34
|
+
|
|
35
|
+
- Broken propagation produces disconnected traces even when every service emits spans.
|
|
36
|
+
- Trace and span identifiers can correlate logs with the request that produced them.
|
|
37
|
+
- Incoming context from untrusted sources can be forged; outgoing context can disclose internal information.
|
|
38
|
+
- Baggage travels across boundaries and must not contain credentials, tokens, personal data, or other sensitive values.
|
|
39
|
+
|
|
40
|
+
Manual propagation implementation is outside this skill. Diagnose propagation by verifying carrier presence, trust-boundary policy, and continuity of trace and parent identifiers.
|
|
41
|
+
|
|
42
|
+
## Resources
|
|
43
|
+
|
|
44
|
+
A resource describes the entity producing telemetry. Typical dimensions include service identity, host, operating system, process, container, Kubernetes workload, and cloud platform.
|
|
45
|
+
|
|
46
|
+
Resource guidance:
|
|
47
|
+
|
|
48
|
+
- Make service identity explicit and stable enough to distinguish logical services.
|
|
49
|
+
- Use semantic-convention attribute names where defined.
|
|
50
|
+
- Use resource detection for environmental facts, then verify the detected values.
|
|
51
|
+
- Avoid attributes whose values change per request; those belong on individual telemetry records, not the producing resource.
|
|
52
|
+
- Ensure identities remain globally distinguishable when multiple Collectors or environments report to the same backend.
|
|
53
|
+
|
|
54
|
+
## Instrumentation scope
|
|
55
|
+
|
|
56
|
+
Instrumentation scope identifies the logical source that emitted telemetry. Its identity includes a required name and optional version, schema URL, and attributes. Backends can use the scope to filter, group, or compare telemetry from different components or versions.
|
|
57
|
+
|
|
58
|
+
Keep resource and scope separate:
|
|
59
|
+
|
|
60
|
+
- Resource answers “what entity produced this telemetry?”
|
|
61
|
+
- Instrumentation scope answers “what logical instrumentation source produced this record?”
|
|
62
|
+
|
|
63
|
+
## Semantic conventions
|
|
64
|
+
|
|
65
|
+
Semantic conventions standardize names and meanings for common operations and resources. They make telemetry portable and comparable across sources and backends.
|
|
66
|
+
|
|
67
|
+
When reviewing telemetry:
|
|
68
|
+
|
|
69
|
+
1. Check whether an applicable convention exists.
|
|
70
|
+
2. Check the current convention version and stability before asserting a field is required or stable.
|
|
71
|
+
3. Preserve the semantic meaning during transformation; renaming without a mapping can break dashboards, alerts, and queries.
|
|
72
|
+
4. Treat custom attributes as additions, not replacements for applicable standard attributes.
|
|
73
|
+
|
|
74
|
+
## Terminology checkpoints
|
|
75
|
+
|
|
76
|
+
| Term | Operational meaning |
|
|
77
|
+
| --- | --- |
|
|
78
|
+
| Telemetry | Data emitted by a system about its behavior. |
|
|
79
|
+
| Signal | A category of telemetry, such as traces, metrics, logs, baggage, or profiles. |
|
|
80
|
+
| OTLP | The standard OpenTelemetry protocol for telemetry transport. |
|
|
81
|
+
| Resource | Metadata describing the entity that produced telemetry. |
|
|
82
|
+
| Attribute | Key-value metadata attached to telemetry. |
|
|
83
|
+
| Context | Correlation state associated with an execution. |
|
|
84
|
+
| Propagation | Movement of context across boundaries. |
|
|
85
|
+
| Baggage | User-defined contextual key-value data propagated alongside context. |
|
|
86
|
+
| Instrumentation scope | Identity of the logical source that emitted telemetry. |
|
|
87
|
+
| Collector | A vendor-agnostic service that receives, processes, and exports telemetry. |
|
|
88
|
+
|
|
89
|
+
## Official sources
|
|
90
|
+
|
|
91
|
+
- [What is OpenTelemetry?](https://opentelemetry.io/docs/what-is-opentelemetry/)
|
|
92
|
+
- [Observability primer](https://opentelemetry.io/docs/concepts/observability-primer/)
|
|
93
|
+
- [Components](https://opentelemetry.io/docs/concepts/components/)
|
|
94
|
+
- [Context propagation](https://opentelemetry.io/docs/concepts/context-propagation/)
|
|
95
|
+
- [Resources](https://opentelemetry.io/docs/concepts/resources/)
|
|
96
|
+
- [Instrumentation scope](https://opentelemetry.io/docs/concepts/instrumentation-scope/)
|
|
97
|
+
- [Semantic conventions](https://opentelemetry.io/docs/concepts/semantic-conventions/)
|
|
98
|
+
- [Glossary](https://opentelemetry.io/docs/concepts/glossary/)
|
|
@@ -0,0 +1,108 @@
|
|
|
1
|
+
# Platforms and Zero-Code Instrumentation
|
|
2
|
+
|
|
3
|
+
Use this reference when telemetry must be added or operated without editing application source. Do not turn it into language-specific integration guidance.
|
|
4
|
+
|
|
5
|
+
## Zero-code model
|
|
6
|
+
|
|
7
|
+
Zero-code instrumentation adds telemetry capabilities without application source changes. Depending on the environment, it can use runtime agents, process injection, platform hooks, bytecode techniques, monkey patching, or eBPF.
|
|
8
|
+
|
|
9
|
+
It commonly captures framework-visible activity such as inbound and outbound requests, database calls, and messaging operations. It generally cannot infer custom business events, application-specific attributes, or arbitrary internal operations that are not visible to the mechanism.
|
|
10
|
+
|
|
11
|
+
Operational configuration usually includes:
|
|
12
|
+
|
|
13
|
+
- Service and resource identity.
|
|
14
|
+
- Enabled instrumentations or observed protocols.
|
|
15
|
+
- Export destination and transport.
|
|
16
|
+
- Propagators and context behavior.
|
|
17
|
+
- Sampling and signal enablement.
|
|
18
|
+
- Resource limits and performance controls.
|
|
19
|
+
|
|
20
|
+
Verify current support for the target language, runtime, framework, operating system, and deployment platform. Do not infer support from a different runtime.
|
|
21
|
+
|
|
22
|
+
## Selection guide
|
|
23
|
+
|
|
24
|
+
| Approach | Best fit | Main trade-offs |
|
|
25
|
+
| --- | --- | --- |
|
|
26
|
+
| Runtime or language agent | Supported runtimes where library-level operations must be observed without source changes | Runtime-specific support, startup and performance impact, version compatibility |
|
|
27
|
+
| Kubernetes Operator injection | Centrally managed Kubernetes workloads with supported automatic-instrumentation images | Admission/injection policy, image lifecycle, resource overhead, rollout coordination |
|
|
28
|
+
| OBI/eBPF | Linux environments requiring protocol and network visibility without process-level source changes | Kernel, architecture, privileges, protocol visibility, and application-detail limitations |
|
|
29
|
+
| FaaS layer | Supported serverless runtimes where a published instrumentation or Collector layer is available | Cold-start impact, regional artifacts, layer lifecycle, limited runtime control |
|
|
30
|
+
| Collector-only infrastructure collection | Host, container, Kubernetes, network, and existing log or metric sources | Does not create application-internal spans that the Collector cannot observe |
|
|
31
|
+
|
|
32
|
+
## Kubernetes
|
|
33
|
+
|
|
34
|
+
Choose the Collector mode based on the data source and processing requirement:
|
|
35
|
+
|
|
36
|
+
- DaemonSet for node-local collection such as host metrics and node logs.
|
|
37
|
+
- Sidecar when a workload requires strong local isolation or per-workload collection behavior.
|
|
38
|
+
- Deployment or StatefulSet for shared gateways, central processing, or components with identity/state requirements.
|
|
39
|
+
|
|
40
|
+
The OpenTelemetry Operator can manage Collector custom resources and inject supported automatic instrumentation. Helm charts provide another declarative deployment mechanism. Treat Operator and chart versions as time-sensitive and verify their current custom-resource fields and defaults.
|
|
41
|
+
|
|
42
|
+
### Prometheus collection
|
|
43
|
+
|
|
44
|
+
Avoid having multiple Collectors scrape the same target. The Target Allocator separates target discovery from collection and distributes Prometheus targets across Collector instances. Verify the reconciled configuration because enabling allocation can replace existing service-discovery entries with allocator-provided discovery.
|
|
45
|
+
|
|
46
|
+
### Horizontal scaling
|
|
47
|
+
|
|
48
|
+
Before enabling autoscaling:
|
|
49
|
+
|
|
50
|
+
1. Set realistic CPU and memory requests and limits.
|
|
51
|
+
2. Confirm the metrics source used by the autoscaler.
|
|
52
|
+
3. Choose signals that reflect Collector pressure rather than only host utilization.
|
|
53
|
+
4. Preserve stateful-routing requirements.
|
|
54
|
+
5. Set minimum capacity for failure tolerance and maximum capacity for backend protection.
|
|
55
|
+
|
|
56
|
+
## OpenTelemetry eBPF Instrumentation (OBI)
|
|
57
|
+
|
|
58
|
+
OBI observes Linux application executables and networking with eBPF and can emit application, network, trace, metric, and correlation data without source changes. Its precise protocol coverage, feature status, artifacts, kernel requirements, and supported architectures change over time; verify the current OBI documentation.
|
|
59
|
+
|
|
60
|
+
Selection constraints:
|
|
61
|
+
|
|
62
|
+
- Linux kernel and BTF compatibility.
|
|
63
|
+
- Supported CPU architecture.
|
|
64
|
+
- Access to process, network, and kernel interfaces.
|
|
65
|
+
- Required Linux capabilities for the enabled features.
|
|
66
|
+
- Visibility limits for encrypted, asynchronous, runtime-specific, or application-specific behavior.
|
|
67
|
+
- Cardinality and telemetry-volume controls.
|
|
68
|
+
|
|
69
|
+
Use least privilege. Prefer the exact capabilities required by enabled features over unrestricted root access. Treat `CAP_SYS_ADMIN` and host-level access as high-risk and document why they are needed.
|
|
70
|
+
|
|
71
|
+
OBI is not a substitute for custom business telemetry. If the requirement depends on application semantics that OBI cannot observe, identify that limitation without generating manual instrumentation code.
|
|
72
|
+
|
|
73
|
+
## FaaS
|
|
74
|
+
|
|
75
|
+
Serverless environments have platform-specific lifecycle, startup, networking, and packaging constraints. For supported Lambda layers:
|
|
76
|
+
|
|
77
|
+
- Confirm the layer exists for the runtime and region.
|
|
78
|
+
- Account for initialization and cold-start overhead.
|
|
79
|
+
- Add a Collector layer or verified external Collector path when the instrumentation layer requires it.
|
|
80
|
+
- Configure secrets through the platform secret mechanism or protected environment injection.
|
|
81
|
+
- Verify the transport, endpoint, and layer compatibility before rollout.
|
|
82
|
+
- Publish and test a new function version with rollback available.
|
|
83
|
+
|
|
84
|
+
Do not copy version numbers or layer ARNs from a snapshot. Resolve the current official release for the target region at execution time.
|
|
85
|
+
|
|
86
|
+
## Rollout checklist
|
|
87
|
+
|
|
88
|
+
1. Inventory platform, runtime, kernel, architecture, and workload restrictions.
|
|
89
|
+
2. Confirm the zero-code method supports the target.
|
|
90
|
+
3. Define expected signals and known blind spots.
|
|
91
|
+
4. Establish service/resource identity and export path.
|
|
92
|
+
5. Review privileges, network exposure, and secrets.
|
|
93
|
+
6. Measure startup, CPU, memory, latency, and telemetry volume on a limited rollout.
|
|
94
|
+
7. Verify trace continuity, metric cardinality, log correlation, and backend ingestion.
|
|
95
|
+
8. Document rollback and upgrade ownership.
|
|
96
|
+
|
|
97
|
+
## Official sources
|
|
98
|
+
|
|
99
|
+
- [Zero-code instrumentation concept](https://opentelemetry.io/docs/concepts/instrumentation/zero-code/)
|
|
100
|
+
- [Zero-code instrumentation documentation](https://opentelemetry.io/docs/zero-code/)
|
|
101
|
+
- [OpenTelemetry eBPF Instrumentation](https://opentelemetry.io/docs/zero-code/obi/)
|
|
102
|
+
- [OBI security, permissions, and capabilities](https://opentelemetry.io/docs/zero-code/obi/security/)
|
|
103
|
+
- [OpenTelemetry with Kubernetes](https://opentelemetry.io/docs/platforms/kubernetes/)
|
|
104
|
+
- [OpenTelemetry Operator](https://opentelemetry.io/docs/platforms/kubernetes/operator/)
|
|
105
|
+
- [Target Allocator](https://opentelemetry.io/docs/platforms/kubernetes/operator/target-allocator/)
|
|
106
|
+
- [Functions as a Service](https://opentelemetry.io/docs/platforms/faas/)
|
|
107
|
+
- [Lambda auto-instrumentation](https://opentelemetry.io/docs/platforms/faas/lambda-auto-instrument/)
|
|
108
|
+
- [Lambda Collector configuration](https://opentelemetry.io/docs/platforms/faas/lambda-collector/)
|
|
@@ -0,0 +1,119 @@
|
|
|
1
|
+
# OpenTelemetry Security
|
|
2
|
+
|
|
3
|
+
Use this reference for Collector hardening, sensitive-data controls, context trust boundaries, and security review. OpenTelemetry cannot decide which data is sensitive for a particular organization; that responsibility remains with the operator.
|
|
4
|
+
|
|
5
|
+
## Security objectives
|
|
6
|
+
|
|
7
|
+
- Minimize telemetry collection to data with an explicit observability purpose.
|
|
8
|
+
- Protect telemetry in transit, in memory, on disk, and in the backend.
|
|
9
|
+
- Restrict Collector listeners and administrative endpoints.
|
|
10
|
+
- Authenticate peers and authorize access.
|
|
11
|
+
- Run with the least filesystem, network, Kubernetes, and kernel privilege.
|
|
12
|
+
- Prevent resource exhaustion and uncontrolled cardinality.
|
|
13
|
+
- Keep credentials out of configuration files and telemetry.
|
|
14
|
+
- Monitor current OpenTelemetry security advisories.
|
|
15
|
+
|
|
16
|
+
## Sensitive data
|
|
17
|
+
|
|
18
|
+
Potentially sensitive telemetry includes personal information, credentials, session tokens, financial or health data, user behavior, request and response content, database statements, URLs, headers, logs, and baggage.
|
|
19
|
+
|
|
20
|
+
Apply controls in this order:
|
|
21
|
+
|
|
22
|
+
1. Do not collect unnecessary data.
|
|
23
|
+
2. Prefer aggregation, truncation, or categorization over unique identifiers.
|
|
24
|
+
3. Use an allowlist when the acceptable attribute set is known.
|
|
25
|
+
4. Remove or mask sensitive values before they cross a trust boundary.
|
|
26
|
+
5. Restrict access and retention for telemetry that must remain identifiable.
|
|
27
|
+
6. Periodically inspect actual emitted telemetry because instrumentation and schemas change.
|
|
28
|
+
|
|
29
|
+
Hashing does not automatically anonymize predictable values. A small or enumerable input space can permit reversal by guessing. Document the threat model before treating hashing as a privacy control.
|
|
30
|
+
|
|
31
|
+
## Collector network hardening
|
|
32
|
+
|
|
33
|
+
- Bind server-like components to the narrowest required interface. Prefer loopback for local-only clients.
|
|
34
|
+
- Do not expose receivers, health endpoints, diagnostics, or management endpoints to the public internet without an explicit design.
|
|
35
|
+
- Use TLS for network hops that cross trust boundaries.
|
|
36
|
+
- Authenticate receivers and exporters where supported.
|
|
37
|
+
- Restrict inbound and outbound traffic with platform network controls.
|
|
38
|
+
- Review trust separately for application-to-agent, agent-to-gateway, and gateway-to-backend hops.
|
|
39
|
+
|
|
40
|
+
An illustrative local-only receiver is:
|
|
41
|
+
|
|
42
|
+
```yaml
|
|
43
|
+
receivers:
|
|
44
|
+
otlp:
|
|
45
|
+
protocols:
|
|
46
|
+
grpc:
|
|
47
|
+
endpoint: 127.0.0.1:4317
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
Container and Kubernetes networking can make loopback inappropriate for cross-container traffic. Select the interface from the actual topology rather than replacing it with unrestricted `0.0.0.0` by default.
|
|
51
|
+
|
|
52
|
+
## Secrets and configuration
|
|
53
|
+
|
|
54
|
+
- Store secrets in the deployment platform's protected secret mechanism.
|
|
55
|
+
- Inject secret values at runtime; do not commit them in Collector YAML.
|
|
56
|
+
- Restrict read access to configuration, certificates, queues, WAL directories, and raw telemetry.
|
|
57
|
+
- Avoid printing resolved configuration or verbose telemetry where secrets may appear.
|
|
58
|
+
- Rotate credentials and certificates without requiring broad unrelated configuration changes.
|
|
59
|
+
|
|
60
|
+
Use placeholders in examples:
|
|
61
|
+
|
|
62
|
+
```yaml
|
|
63
|
+
exporters:
|
|
64
|
+
otlp:
|
|
65
|
+
endpoint: telemetry-backend.example:4317
|
|
66
|
+
headers:
|
|
67
|
+
authorization: ${env:OTEL_EXPORTER_AUTHORIZATION}
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
## Least privilege
|
|
71
|
+
|
|
72
|
+
- Avoid running the Collector as root unless a required data source cannot be accessed otherwise.
|
|
73
|
+
- Grant read-only filesystem mounts when collection does not require writes.
|
|
74
|
+
- Scope Kubernetes RBAC to the resources and verbs required by enabled receivers, processors, observers, or the Operator.
|
|
75
|
+
- Review observer and service-discovery extensions because discovery can require broad environment visibility.
|
|
76
|
+
- For OBI/eBPF, grant only capabilities required by enabled modes and document any use of powerful capabilities.
|
|
77
|
+
|
|
78
|
+
## Resource-exhaustion controls
|
|
79
|
+
|
|
80
|
+
- Set workload CPU and memory requests and limits.
|
|
81
|
+
- Put the memory limiter early in pipelines.
|
|
82
|
+
- Bound exporter queues and retry duration.
|
|
83
|
+
- Filter telemetry that has no operational value.
|
|
84
|
+
- Control metric and log cardinality.
|
|
85
|
+
- Rate-limit or isolate untrusted senders.
|
|
86
|
+
- Alert on refusal, queue saturation, export failure, CPU, memory, disk, and restart signals.
|
|
87
|
+
|
|
88
|
+
Scaling does not protect a saturated backend and can amplify a denial-of-service condition. Verify the constrained component before scaling.
|
|
89
|
+
|
|
90
|
+
## Context and baggage boundaries
|
|
91
|
+
|
|
92
|
+
- Treat incoming trace context as untrusted unless the source is authenticated and authorized.
|
|
93
|
+
- Define whether external trace context is accepted, replaced, or sanitized.
|
|
94
|
+
- Prevent internal trace identifiers or baggage from being forwarded to untrusted external services when disclosure is unacceptable.
|
|
95
|
+
- Never store credentials, API keys, session tokens, or personal data in baggage.
|
|
96
|
+
- Remember that baggage has no built-in guarantee that a value originated from a trusted service.
|
|
97
|
+
|
|
98
|
+
## Security review checklist
|
|
99
|
+
|
|
100
|
+
1. Map data sources, Collector tiers, destinations, and trust boundaries.
|
|
101
|
+
2. Inventory listeners, protocols, addresses, ports, and administrative endpoints.
|
|
102
|
+
3. Verify TLS, authentication, and authorization for each network hop.
|
|
103
|
+
4. Inspect configuration and secret delivery.
|
|
104
|
+
5. Inventory attributes, log bodies, baggage, and transformed data for sensitivity.
|
|
105
|
+
6. Review filtering, redaction, and failure behavior.
|
|
106
|
+
7. Review filesystem, RBAC, container, and kernel privileges.
|
|
107
|
+
8. Review resource limits, queue bounds, and abuse controls.
|
|
108
|
+
9. Verify internal telemetry and security-alert ownership.
|
|
109
|
+
10. Check current CVEs and advisories for deployed components and versions.
|
|
110
|
+
|
|
111
|
+
## Official sources
|
|
112
|
+
|
|
113
|
+
- [OpenTelemetry security](https://opentelemetry.io/docs/security/)
|
|
114
|
+
- [Collector configuration security](https://opentelemetry.io/docs/security/config-best-practices/)
|
|
115
|
+
- [Collector hosting security](https://opentelemetry.io/docs/security/hosting-best-practices/)
|
|
116
|
+
- [Handling sensitive data](https://opentelemetry.io/docs/security/handling-sensitive-data/)
|
|
117
|
+
- [OpenTelemetry CVEs](https://opentelemetry.io/docs/security/cve/)
|
|
118
|
+
- [Context propagation security](https://opentelemetry.io/docs/concepts/context-propagation/#security-best-practices)
|
|
119
|
+
- [Baggage security considerations](https://opentelemetry.io/docs/concepts/signals/baggage/#baggage-security-considerations)
|
|
@@ -0,0 +1,113 @@
|
|
|
1
|
+
# Signals and Sampling
|
|
2
|
+
|
|
3
|
+
Use this reference to select signals, reason about correlation, and evaluate sampling. Signal maturity and feature support can change; verify current status before making version-dependent claims.
|
|
4
|
+
|
|
5
|
+
## Signal selection
|
|
6
|
+
|
|
7
|
+
| Signal | Primary question | Operational strengths | Important limitations |
|
|
8
|
+
| --- | --- | --- | --- |
|
|
9
|
+
| Traces | How did one request travel through the system? | Causality, dependency paths, per-request latency, error localization | Volume can be high; incomplete propagation fragments traces |
|
|
10
|
+
| Metrics | What is happening in aggregate over time? | Alerting, trends, capacity, SLI/SLO measurement | Aggregation loses individual-event detail; dimensions can create high cardinality |
|
|
11
|
+
| Logs | What discrete event occurred? | Detailed event records and existing operational workflows | Unstructured records are costly to normalize; correlation requires context |
|
|
12
|
+
| Baggage | What request-scoped context must be available downstream? | Carries selected context across service boundaries | Propagates broadly, is not automatically a signal attribute, and has security risks |
|
|
13
|
+
| Profiles | Where are CPU, memory, or other resources being consumed? | Resource bottlenecks and correlation with slow requests or resource spikes | Availability, status, and supported profile types depend on current tooling |
|
|
14
|
+
|
|
15
|
+
Do not replace one signal with another merely because they share attributes. Prefer correlation through stable resource identity and trace context.
|
|
16
|
+
|
|
17
|
+
## Traces
|
|
18
|
+
|
|
19
|
+
A trace represents the path of a request and contains one or more spans. A span represents one unit of work and can carry timing, status, attributes, events, and links. Parent-child relationships describe causal or nested work.
|
|
20
|
+
|
|
21
|
+
Review trace quality by checking:
|
|
22
|
+
|
|
23
|
+
- Root and child span boundaries reflect meaningful operations.
|
|
24
|
+
- Trace and parent identifiers remain continuous across boundaries.
|
|
25
|
+
- Attributes follow semantic conventions and avoid sensitive or unbounded values.
|
|
26
|
+
- Error state and latency are represented consistently enough for sampling and analysis.
|
|
27
|
+
- Resource identity distinguishes services and instances.
|
|
28
|
+
|
|
29
|
+
## Metrics
|
|
30
|
+
|
|
31
|
+
A metric represents measurements aggregated over time. Operationally, distinguish:
|
|
32
|
+
|
|
33
|
+
- Additive changes from current-state values.
|
|
34
|
+
- Monotonic values from values that may rise or fall.
|
|
35
|
+
- Cumulative from delta temporality.
|
|
36
|
+
- Metric identity from attribute dimensions.
|
|
37
|
+
|
|
38
|
+
Cardinality is the number of distinct time series produced by the combination of metric identity and attributes. User IDs, request IDs, raw URLs, and other unbounded values can make a metric operationally expensive or unusable. Prefer bounded dimensions that support an explicit operational question.
|
|
39
|
+
|
|
40
|
+
When multiple Collector instances handle the same metric stream, preserve a globally unique writer identity and avoid duplicate writers that can create gaps, jumps, or out-of-order samples.
|
|
41
|
+
|
|
42
|
+
## Logs
|
|
43
|
+
|
|
44
|
+
OpenTelemetry can receive, normalize, correlate, process, and export existing logs. Structured logs have a stable schema or typed fields; JSON encoding alone does not guarantee structure. Semistructured and unstructured logs usually need parsing and normalization before reliable analysis.
|
|
45
|
+
|
|
46
|
+
Operational priorities:
|
|
47
|
+
|
|
48
|
+
1. Preserve event time and observed time when both are available.
|
|
49
|
+
2. Add resource identity at collection time when the source does not provide it.
|
|
50
|
+
3. Correlate logs with trace and span identifiers where supported.
|
|
51
|
+
4. Normalize schemas before downstream rules depend on them.
|
|
52
|
+
5. Filter or redact sensitive fields before export.
|
|
53
|
+
|
|
54
|
+
Language logging bridges and application logging code are outside this skill.
|
|
55
|
+
|
|
56
|
+
## Baggage
|
|
57
|
+
|
|
58
|
+
Baggage is propagated contextual data, separate from span, metric, and log attributes. A baggage item is not automatically copied into telemetry attributes.
|
|
59
|
+
|
|
60
|
+
Use baggage only when downstream components genuinely need request-scoped context. Never place credentials, API keys, session tokens, personal information, or other sensitive data in baggage. Define trust-boundary rules for accepting and forwarding baggage because it can leave the controlled network through automatic propagation.
|
|
61
|
+
|
|
62
|
+
## Profiles
|
|
63
|
+
|
|
64
|
+
Profiles contain samples and metadata describing resource consumption over time. They can complement:
|
|
65
|
+
|
|
66
|
+
- Metrics by explaining a CPU or memory spike.
|
|
67
|
+
- Traces by locating resource consumption associated with a slow request.
|
|
68
|
+
- Logs by connecting a failure event to resource pressure.
|
|
69
|
+
|
|
70
|
+
Keep profile collection details out of recommendations unless current official support has been verified for the target environment.
|
|
71
|
+
|
|
72
|
+
## Sampling
|
|
73
|
+
|
|
74
|
+
Sampling reduces trace volume by retaining a representative subset. Filtering and aggregation may also reduce data, but they do not automatically preserve representativeness.
|
|
75
|
+
|
|
76
|
+
### Head sampling
|
|
77
|
+
|
|
78
|
+
Head sampling decides early without examining the complete trace. It is comparatively simple and efficient, but cannot reliably retain every trace that later becomes slow or erroneous.
|
|
79
|
+
|
|
80
|
+
Use head sampling when an early, consistent probability decision satisfies the observability goal and the lost late-trace information is acceptable.
|
|
81
|
+
|
|
82
|
+
### Tail sampling
|
|
83
|
+
|
|
84
|
+
Tail sampling decides after receiving all or most spans in a trace. It can retain traces based on whole-trace criteria such as errors, latency, attributes, or service-specific policy.
|
|
85
|
+
|
|
86
|
+
Tail sampling is stateful and operationally expensive. It requires:
|
|
87
|
+
|
|
88
|
+
- Enough memory and decision time to retain spans awaiting a decision.
|
|
89
|
+
- Routing that sends every span for a trace to the same sampling instance.
|
|
90
|
+
- Monitoring for dropped spans, late spans, decision-cache pressure, and uneven routing.
|
|
91
|
+
- Reassessment as traffic and service behavior change.
|
|
92
|
+
|
|
93
|
+
For multiple gateways, trace-ID-aware routing is necessary but does not remove every scaling caveat. Prefer a single adequately sized tail-sampling gateway until the need for a distributed design and its consistency strategy are demonstrated.
|
|
94
|
+
|
|
95
|
+
### Sampling decision checklist
|
|
96
|
+
|
|
97
|
+
1. Identify the cost or capacity problem sampling must solve.
|
|
98
|
+
2. State what information may be lost and whether regulation permits dropping it.
|
|
99
|
+
3. Measure trace volume and traffic variation by service.
|
|
100
|
+
4. Define required retention rules for errors, latency, or business-critical traffic.
|
|
101
|
+
5. Choose head sampling unless whole-trace evidence is required.
|
|
102
|
+
6. If using tail sampling, design trace-affine routing and capacity before enabling it.
|
|
103
|
+
7. Validate the retained population and monitor sampling infrastructure.
|
|
104
|
+
|
|
105
|
+
## Official sources
|
|
106
|
+
|
|
107
|
+
- [Signals](https://opentelemetry.io/docs/concepts/signals/)
|
|
108
|
+
- [Traces](https://opentelemetry.io/docs/concepts/signals/traces/)
|
|
109
|
+
- [Metrics](https://opentelemetry.io/docs/concepts/signals/metrics/)
|
|
110
|
+
- [Logs](https://opentelemetry.io/docs/concepts/signals/logs/)
|
|
111
|
+
- [Baggage](https://opentelemetry.io/docs/concepts/signals/baggage/)
|
|
112
|
+
- [Profiles](https://opentelemetry.io/docs/concepts/signals/profiles/)
|
|
113
|
+
- [Sampling](https://opentelemetry.io/docs/concepts/sampling/)
|
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
# Sources, Scope, and Currency
|
|
2
|
+
|
|
3
|
+
## Source snapshot
|
|
4
|
+
|
|
5
|
+
This skill was derived from the English OpenTelemetry documentation stored at:
|
|
6
|
+
|
|
7
|
+
- Repository: `https://github.com/open-telemetry/opentelemetry.io`
|
|
8
|
+
- Source subtree: `content/en/docs`
|
|
9
|
+
- Revision: `b18eb4d01fab9be2c5c22b1aaafdeb23ba97e3d4`
|
|
10
|
+
- Revision timestamp: `2026-07-17T02:20:14Z`
|
|
11
|
+
- Documentation root: `https://opentelemetry.io/docs/`
|
|
12
|
+
|
|
13
|
+
The source subtree contained 423 Markdown files at generation time. The material was curated and rewritten rather than copied wholesale.
|
|
14
|
+
|
|
15
|
+
## Included source areas
|
|
16
|
+
|
|
17
|
+
- `what-is-opentelemetry.md`
|
|
18
|
+
- Non-programming material from `concepts/`
|
|
19
|
+
- Collector architecture, components, configuration, installation, deployment, management, internal telemetry, resiliency, scaling, transformation, troubleshooting, distributions, benchmarks, and registry material
|
|
20
|
+
- Operational material from `getting-started/ops.md`
|
|
21
|
+
- Prometheus interoperability and migration concepts from `compatibility/`
|
|
22
|
+
- Non-programming platform blueprints and reference-implementation lessons from `guidance/`
|
|
23
|
+
- Kubernetes, FaaS automatic instrumentation, and operational platform material from `platforms/`
|
|
24
|
+
- Security and sensitive-data material from `security/`
|
|
25
|
+
- Zero-code and OBI operational material from `zero-code/`
|
|
26
|
+
- Non-programming demo architecture or operational lessons only when they clarify an included concept
|
|
27
|
+
|
|
28
|
+
## Excluded source areas
|
|
29
|
+
|
|
30
|
+
- All language API and SDK instructions under `languages/`
|
|
31
|
+
- Code-based instrumentation and language-library integration
|
|
32
|
+
- Developer getting-started and reference-application implementation material
|
|
33
|
+
- Collector source builds, Collector Builder, and custom component development under `collector/extend/`
|
|
34
|
+
- Manual FaaS instrumentation
|
|
35
|
+
- Demo service source, development, tests, and manual span examples
|
|
36
|
+
- Contribution workflow and website-development documentation
|
|
37
|
+
- API, SDK, data-model implementation, and programming-oriented specification pages
|
|
38
|
+
- Source-code snippets in Java, C#, Go, JavaScript, TypeScript, Python, PHP, Ruby, Rust, Swift, C++, and other programming languages
|
|
39
|
+
|
|
40
|
+
Mixed pages were reduced to concepts, declarative YAML or JSON configuration, environment variables, and operational shell, Docker, cloud, or Kubernetes commands. High-level references to an API or SDK may remain only when needed to state a system boundary or migration limitation; this skill never provides their programming interface.
|
|
41
|
+
|
|
42
|
+
## Currency policy
|
|
43
|
+
|
|
44
|
+
The snapshot is evidence for the content as of its revision. It is not evidence that a version, status, default, component, platform matrix, security advisory, image, package, or command remains current.
|
|
45
|
+
|
|
46
|
+
Verify against official sources before answering time-sensitive questions about:
|
|
47
|
+
|
|
48
|
+
- Collector, Operator, Helm, OBI, FaaS layer, or semantic-convention versions.
|
|
49
|
+
- Signal or component maturity.
|
|
50
|
+
- Component availability in a distribution.
|
|
51
|
+
- Runtime, platform, kernel, architecture, or protocol support.
|
|
52
|
+
- Default endpoints, limits, queue behavior, or feature gates.
|
|
53
|
+
- CVEs, security advisories, and mitigations.
|
|
54
|
+
- Migration shims and compatibility guarantees.
|
|
55
|
+
|
|
56
|
+
If current verification is unavailable, state:
|
|
57
|
+
|
|
58
|
+
1. The claim is based on revision `b18eb4d01fab9be2c5c22b1aaafdeb23ba97e3d4`.
|
|
59
|
+
2. Current status cannot be confirmed.
|
|
60
|
+
3. The official page the user should verify.
|