@moon791017/neo-skills 1.1.11 → 1.1.13
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +4 -0
- package/package.json +1 -1
- package/skills/neo-agentic-design/SKILL.md +89 -0
- package/skills/neo-agentic-design/evals/eval_queries.json +58 -0
- package/skills/neo-agentic-design/evals/evals.json +27 -0
- package/skills/neo-agentic-design/references/advanced-safety.md +158 -0
- package/skills/neo-agentic-design/references/base-workflows.md +219 -0
- package/skills/neo-agentic-design/references/resilience-hitl.md +105 -0
- package/skills/neo-agentic-design/references/system-components.md +93 -0
- package/skills/neo-opentelemetry/SKILL.md +94 -0
- package/skills/neo-opentelemetry/evals/eval_queries.json +82 -0
- package/skills/neo-opentelemetry/evals/evals.json +64 -0
- package/skills/neo-opentelemetry/references/collector-architecture-and-configuration.md +138 -0
- package/skills/neo-opentelemetry/references/collector-operations.md +137 -0
- package/skills/neo-opentelemetry/references/compatibility-and-guidance.md +112 -0
- package/skills/neo-opentelemetry/references/foundations.md +98 -0
- package/skills/neo-opentelemetry/references/platforms-and-zero-code.md +108 -0
- package/skills/neo-opentelemetry/references/security.md +119 -0
- package/skills/neo-opentelemetry/references/signals-and-sampling.md +113 -0
- package/skills/neo-opentelemetry/references/sources-and-scope.md +60 -0
|
@@ -0,0 +1,137 @@
|
|
|
1
|
+
# Collector Operations and Troubleshooting
|
|
2
|
+
|
|
3
|
+
Use this reference for installation planning, self-observability, capacity, resiliency, transformation, and incident diagnosis.
|
|
4
|
+
|
|
5
|
+
## Installation and distribution checks
|
|
6
|
+
|
|
7
|
+
Before using any component or command, confirm:
|
|
8
|
+
|
|
9
|
+
- Target operating system or Kubernetes environment.
|
|
10
|
+
- Collector distribution and version.
|
|
11
|
+
- Included receivers, processors, exporters, connectors, and extensions.
|
|
12
|
+
- Configuration file location and executable name.
|
|
13
|
+
- Required network access, filesystem access, capabilities, and credentials.
|
|
14
|
+
|
|
15
|
+
Do not assume the core and contrib distributions contain the same components. Registry listings and component maturity are time-sensitive and require current verification.
|
|
16
|
+
|
|
17
|
+
## Internal telemetry
|
|
18
|
+
|
|
19
|
+
The Collector must expose enough telemetry to distinguish its own saturation from upstream, network, or backend failures. Monitor at least:
|
|
20
|
+
|
|
21
|
+
- Accepted and refused telemetry by receiver and signal.
|
|
22
|
+
- Processor drops and refusals.
|
|
23
|
+
- Export successes, permanent failures, and enqueue failures.
|
|
24
|
+
- Exporter queue size and capacity.
|
|
25
|
+
- CPU, memory, and restart behavior.
|
|
26
|
+
- Request latency and errors for downstream destinations.
|
|
27
|
+
- Scrape duration relative to scrape interval for pull-based receivers.
|
|
28
|
+
|
|
29
|
+
Metric names can change across versions. Confirm the current names before creating alerts; use the names documented by the installed version.
|
|
30
|
+
|
|
31
|
+
## Scaling decision flow
|
|
32
|
+
|
|
33
|
+
1. Split demand by signal, protocol, and collection behavior.
|
|
34
|
+
2. Identify whether components are stateless, scrapers, or stateful.
|
|
35
|
+
3. Check refusal, queue, resource, and downstream latency evidence.
|
|
36
|
+
4. Determine whether the constraint is Collector CPU/memory, a scrape schedule, the network, or the backend.
|
|
37
|
+
5. Scale only the constrained tier and preserve stateful routing rules.
|
|
38
|
+
|
|
39
|
+
Scaling patterns:
|
|
40
|
+
|
|
41
|
+
- Stateless receivers and processors can usually scale horizontally behind a suitable load balancer.
|
|
42
|
+
- Scrapers require target sharding or allocation to prevent duplicate collection.
|
|
43
|
+
- Stateful processors require affinity and careful redistribution behavior.
|
|
44
|
+
- Signal-specific Collector tiers can isolate failures and scale independently.
|
|
45
|
+
|
|
46
|
+
Do not add Collectors when the exporter queue remains full because the backend or network cannot accept more traffic. That can increase pressure without increasing throughput.
|
|
47
|
+
|
|
48
|
+
## Resiliency layers
|
|
49
|
+
|
|
50
|
+
### In-memory sending queue
|
|
51
|
+
|
|
52
|
+
Buffers transient downstream failures and works with retries. Data can still be lost when the queue fills, retry limits expire, or the Collector terminates.
|
|
53
|
+
|
|
54
|
+
### Persistent sending queue
|
|
55
|
+
|
|
56
|
+
A storage extension can back the sending queue with a write-ahead log so queued data survives Collector restarts. It introduces disk capacity, permissions, corruption, and lifecycle requirements.
|
|
57
|
+
|
|
58
|
+
```yaml
|
|
59
|
+
extensions:
|
|
60
|
+
file_storage:
|
|
61
|
+
directory: /var/lib/otelcol/storage
|
|
62
|
+
|
|
63
|
+
exporters:
|
|
64
|
+
otlp:
|
|
65
|
+
endpoint: telemetry-backend.example:4317
|
|
66
|
+
sending_queue:
|
|
67
|
+
storage: file_storage
|
|
68
|
+
|
|
69
|
+
service:
|
|
70
|
+
extensions: [file_storage]
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
Treat the directory, retention, queue size, and retry duration as workload-specific settings.
|
|
74
|
+
|
|
75
|
+
### External message queue
|
|
76
|
+
|
|
77
|
+
An external queue can decouple Collector tiers and provide stronger durability across outages. It also creates another production system with its own availability, retention, security, ordering, and capacity requirements. Add it only when the required durability and failure isolation justify the operational cost.
|
|
78
|
+
|
|
79
|
+
## Data-loss checklist
|
|
80
|
+
|
|
81
|
+
Check for:
|
|
82
|
+
|
|
83
|
+
- Downstream unavailability exceeding retry duration.
|
|
84
|
+
- Full sending queues or persistent storage.
|
|
85
|
+
- Collector restart with only in-memory buffering.
|
|
86
|
+
- Disk failure or exhausted persistent storage.
|
|
87
|
+
- External queue outage or retention expiration.
|
|
88
|
+
- Receiver, processor, exporter, or routing misconfiguration.
|
|
89
|
+
- Sampling or filtering that intentionally discards data.
|
|
90
|
+
- Load-balancing that splits state required by a processor.
|
|
91
|
+
- Backend rejection, throttling, or duplicate metric writers.
|
|
92
|
+
|
|
93
|
+
## Transformation safety
|
|
94
|
+
|
|
95
|
+
Transformations can normalize, enrich, redact, filter, or route telemetry. Before applying one:
|
|
96
|
+
|
|
97
|
+
1. Define the exact operational purpose.
|
|
98
|
+
2. Identify affected signals and attributes.
|
|
99
|
+
3. Preserve resource identity and semantic meaning.
|
|
100
|
+
4. Quantify records that will be dropped or rewritten.
|
|
101
|
+
5. Test representative and malformed input.
|
|
102
|
+
6. Monitor transform errors and output volume after rollout.
|
|
103
|
+
|
|
104
|
+
Prefer allowlists for sensitive attributes and explicit filters for unwanted data. Avoid transformations that silently collapse distinct metric streams or break trace correlation.
|
|
105
|
+
|
|
106
|
+
## Troubleshooting sequence
|
|
107
|
+
|
|
108
|
+
Diagnose from source to destination:
|
|
109
|
+
|
|
110
|
+
1. Confirm the source is producing the expected signal.
|
|
111
|
+
2. Confirm network reachability and protocol compatibility to the receiver.
|
|
112
|
+
3. Confirm the receiver accepts records and attaches expected resource data.
|
|
113
|
+
4. Inspect processor refusals, errors, sampling, filtering, and transformations.
|
|
114
|
+
5. Inspect queue occupancy, retries, enqueue failures, and storage health.
|
|
115
|
+
6. Inspect exporter errors, TLS, authentication, endpoint, and timeouts.
|
|
116
|
+
7. Confirm the backend accepts, indexes, and exposes the data.
|
|
117
|
+
|
|
118
|
+
Useful non-destructive actions include validating the active configuration, raising Collector log detail temporarily, using diagnostic extensions or a debug exporter in a controlled environment, and comparing accepted versus exported counts. Do not enable verbose telemetry in production without considering sensitive data and volume.
|
|
119
|
+
|
|
120
|
+
## Change verification
|
|
121
|
+
|
|
122
|
+
- Validate configuration before restart.
|
|
123
|
+
- Roll out to a limited Collector subset when possible.
|
|
124
|
+
- Compare accepted, dropped, queued, retried, and exported telemetry before and after.
|
|
125
|
+
- Confirm resource identity, trace completeness, metric continuity, and log timestamps.
|
|
126
|
+
- Confirm rollback conditions and configuration are ready.
|
|
127
|
+
|
|
128
|
+
## Official sources
|
|
129
|
+
|
|
130
|
+
- [Collector installation](https://opentelemetry.io/docs/collector/install/)
|
|
131
|
+
- [Collector internal telemetry](https://opentelemetry.io/docs/collector/internal-telemetry/)
|
|
132
|
+
- [Collector management](https://opentelemetry.io/docs/collector/management/)
|
|
133
|
+
- [Scaling the Collector](https://opentelemetry.io/docs/collector/scaling/)
|
|
134
|
+
- [Collector resiliency](https://opentelemetry.io/docs/collector/resiliency/)
|
|
135
|
+
- [Transforming telemetry](https://opentelemetry.io/docs/collector/transforming-telemetry/)
|
|
136
|
+
- [Collector troubleshooting](https://opentelemetry.io/docs/collector/troubleshooting/)
|
|
137
|
+
- [Collector registry](https://opentelemetry.io/docs/collector/registry/)
|
|
@@ -0,0 +1,112 @@
|
|
|
1
|
+
# Compatibility, Migration, and Platform Guidance
|
|
2
|
+
|
|
3
|
+
Use this reference for Prometheus interoperability, OpenCensus/OpenTracing migration planning, and organization-level telemetry-platform decisions. Programming-language migration code remains outside this skill.
|
|
4
|
+
|
|
5
|
+
## Prometheus interoperability
|
|
6
|
+
|
|
7
|
+
Prometheus is built around discovering and scraping targets. OpenTelemetry also supports transporting metrics through OTLP. Newer Prometheus versions can receive OTLP metrics, but the official OpenTelemetry guidance recommends a Collector intermediary for production OpenTelemetry deployments.
|
|
8
|
+
|
|
9
|
+
Key differences to preserve during migration or translation:
|
|
10
|
+
|
|
11
|
+
- Prometheus traditionally uses pull-based scraping; OTLP commonly pushes telemetry.
|
|
12
|
+
- Metric naming, units, suffixes, and resource-to-label mapping can change during translation.
|
|
13
|
+
- Prometheus metrics are cumulative; OpenTelemetry supports more than one aggregation temporality, while Prometheus-facing export requires compatible cumulative behavior.
|
|
14
|
+
- OpenTelemetry resource identity maps into Prometheus target identity and related metadata.
|
|
15
|
+
- Instrumentation-scope labels can distinguish otherwise identical metric names.
|
|
16
|
+
- Duplicate writers or duplicate scrapes can create invalid or out-of-order series.
|
|
17
|
+
|
|
18
|
+
Operational patterns:
|
|
19
|
+
|
|
20
|
+
- Use Prometheus receivers when the Collector must scrape existing targets.
|
|
21
|
+
- Use the Target Allocator or another sharding mechanism when multiple Collectors scrape Kubernetes targets.
|
|
22
|
+
- Use a Collector between OTLP producers and Prometheus for central policy, conversion, buffering, and diagnostics.
|
|
23
|
+
- Verify the current Prometheus OTLP receiver requirements before recommending direct ingestion.
|
|
24
|
+
|
|
25
|
+
## Migration principles
|
|
26
|
+
|
|
27
|
+
Treat migration as a telemetry-contract change, not only a deployment change.
|
|
28
|
+
|
|
29
|
+
1. Inventory existing signals, protocols, resource identity, propagation, baggage, dashboards, alerts, SLOs, and retention rules.
|
|
30
|
+
2. Define the target OTLP and Collector topology.
|
|
31
|
+
3. Establish a compatibility or overlap period when supported.
|
|
32
|
+
4. Migrate one bounded telemetry path at a time.
|
|
33
|
+
5. Compare old and new volume, identity, attributes, trace continuity, metric continuity, dashboards, and alerts.
|
|
34
|
+
6. Update downstream consumers before removing the old feed.
|
|
35
|
+
7. Remove compatibility layers only after acceptance criteria pass.
|
|
36
|
+
|
|
37
|
+
### OpenTracing
|
|
38
|
+
|
|
39
|
+
OpenTelemetry documents an incremental migration path using compatibility shims, but shim and language details are outside this skill and must be checked in current language documentation.
|
|
40
|
+
|
|
41
|
+
Operational risks that remain in scope:
|
|
42
|
+
|
|
43
|
+
- Semantic-convention changes can break dashboards and alerts.
|
|
44
|
+
- Baggage behavior differs and can lose propagation when old and new mechanisms are mixed.
|
|
45
|
+
- Context-management differences can fragment traces.
|
|
46
|
+
- Dual export can duplicate telemetry and increase cost.
|
|
47
|
+
- A Collector transformation can provide a temporary compatibility view, but it must have a removal plan.
|
|
48
|
+
|
|
49
|
+
### OpenCensus
|
|
50
|
+
|
|
51
|
+
The local documentation entry redirects to the current OpenTelemetry migration guidance rather than containing an operational procedure. Verify the current official migration page and target-language support before planning implementation. Do not invent a generic shim or compatibility guarantee.
|
|
52
|
+
|
|
53
|
+
## Organization-level platform guidance
|
|
54
|
+
|
|
55
|
+
### Non-Kubernetes environments
|
|
56
|
+
|
|
57
|
+
Common challenges include fragmented agent deployment, inconsistent instrumentation, and siloed export paths. A managed approach can:
|
|
58
|
+
|
|
59
|
+
- Define baseline telemetry and resource-attribution standards.
|
|
60
|
+
- Centrally manage agent lifecycle with controlled workload overrides.
|
|
61
|
+
- Use a Collector gateway layer for shared policy and backend export.
|
|
62
|
+
- Separate global defaults, environment policy, and workload-specific configuration.
|
|
63
|
+
- Monitor configuration rollout and agent/Collector health.
|
|
64
|
+
|
|
65
|
+
### Kubernetes environments
|
|
66
|
+
|
|
67
|
+
A managed telemetry platform can:
|
|
68
|
+
|
|
69
|
+
- Distribute consistent defaults through the Operator, Helm, or internal platform automation.
|
|
70
|
+
- Give application teams controlled override points without duplicating complete configurations.
|
|
71
|
+
- Maintain shared gateway tiers for policy, credentials, and backend routing.
|
|
72
|
+
- Process and sample at the layer that has the required data scope.
|
|
73
|
+
- Establish joint ownership among platform, observability, security, and workload teams.
|
|
74
|
+
|
|
75
|
+
Avoid one unbounded shared configuration for every signal and team. Separate pipelines or tiers where load, failure domains, state, or ownership differ.
|
|
76
|
+
|
|
77
|
+
## Reference implementation lessons
|
|
78
|
+
|
|
79
|
+
Official reference implementations describe real organizations, not universal blueprints. Extract constraints and trade-offs rather than copying their topology.
|
|
80
|
+
|
|
81
|
+
Recurring lessons include:
|
|
82
|
+
|
|
83
|
+
- Start with a simple path and add tiers only when requirements justify them.
|
|
84
|
+
- Centralize lifecycle management and safe defaults.
|
|
85
|
+
- Keep workload-facing configuration minimal and extensible.
|
|
86
|
+
- Separate high-volume or stateful processing from simple collection.
|
|
87
|
+
- Plan for component deprecation and controlled upgrades.
|
|
88
|
+
- Monitor the telemetry platform itself.
|
|
89
|
+
|
|
90
|
+
When citing a reference implementation, label the design as that organization's experience rather than an OpenTelemetry requirement.
|
|
91
|
+
|
|
92
|
+
## Migration acceptance criteria
|
|
93
|
+
|
|
94
|
+
- Required signals arrive with stable service and resource identity.
|
|
95
|
+
- Trace context remains continuous across migrated boundaries.
|
|
96
|
+
- Metric names, units, temporality, and labels meet the target contract.
|
|
97
|
+
- Dashboards, alerts, and SLOs have equivalent or intentionally changed behavior.
|
|
98
|
+
- Expected telemetry volume, cardinality, and cost remain within limits.
|
|
99
|
+
- Sensitive-data and access controls remain effective.
|
|
100
|
+
- Rollback and overlap behavior are tested.
|
|
101
|
+
- Old exporters, shims, routes, and dashboards have an explicit retirement point.
|
|
102
|
+
|
|
103
|
+
## Official sources
|
|
104
|
+
|
|
105
|
+
- [OpenTelemetry compatibility](https://opentelemetry.io/docs/compatibility/)
|
|
106
|
+
- [Prometheus interoperability](https://opentelemetry.io/docs/compatibility/prometheus/)
|
|
107
|
+
- [OTLP metrics export to Prometheus](https://opentelemetry.io/docs/compatibility/prometheus/otlp-metrics-export/)
|
|
108
|
+
- [Migrating from OpenCensus](https://opentelemetry.io/docs/compatibility/migration/opencensus/)
|
|
109
|
+
- [Migrating from OpenTracing](https://opentelemetry.io/docs/compatibility/migration/opentracing/)
|
|
110
|
+
- [Infrastructure and processes in non-Kubernetes environments](https://opentelemetry.io/docs/guidance/blueprints/instrumenting-applications-and-processes-on-nonk8s-environments/)
|
|
111
|
+
- [Managed telemetry platforms for Kubernetes workloads](https://opentelemetry.io/docs/guidance/blueprints/managed-telemetry-platforms-for-k8s-workloads/)
|
|
112
|
+
- [Reference implementations](https://opentelemetry.io/docs/guidance/reference-implementations/)
|
|
@@ -0,0 +1,98 @@
|
|
|
1
|
+
# OpenTelemetry Foundations
|
|
2
|
+
|
|
3
|
+
Use this reference for definitions, system boundaries, correlation, resources, semantic conventions, and terminology. Do not use it for language-specific instrumentation.
|
|
4
|
+
|
|
5
|
+
## What OpenTelemetry is
|
|
6
|
+
|
|
7
|
+
OpenTelemetry is an open-source, vendor- and tool-agnostic framework and toolkit for generating, collecting, processing, and exporting telemetry such as traces, metrics, and logs. It is not an observability backend: storage, querying, and visualization are provided by other systems.
|
|
8
|
+
|
|
9
|
+
This distinction prevents two common category errors:
|
|
10
|
+
|
|
11
|
+
- Choosing OpenTelemetry does not choose the telemetry storage or user interface.
|
|
12
|
+
- Changing a backend does not necessarily require changing how telemetry is represented or transported when both sides support OpenTelemetry standards.
|
|
13
|
+
|
|
14
|
+
OpenTelemetry's operational building blocks include the OpenTelemetry Protocol (OTLP), semantic conventions, automatic instrumentation, the Collector, the Kubernetes Operator, Helm charts, and FaaS assets. Programming-language APIs, SDKs, and manual instrumentation are intentionally outside this skill.
|
|
15
|
+
|
|
16
|
+
## Observability and reliability
|
|
17
|
+
|
|
18
|
+
Observability is the ability to reason about a system's internal state from its outputs, including novel failure modes that were not predicted in advance. Telemetry is the emitted data used for that reasoning.
|
|
19
|
+
|
|
20
|
+
Reliability asks whether a service delivers the behavior its users expect. Availability alone is not sufficient. A Service Level Indicator (SLI) measures behavior, ideally from the user's perspective; a Service Level Objective (SLO) connects one or more SLIs to an organizational reliability target.
|
|
21
|
+
|
|
22
|
+
Use signals together:
|
|
23
|
+
|
|
24
|
+
- Metrics reveal aggregate behavior and trends.
|
|
25
|
+
- Traces explain the path and timing of individual requests.
|
|
26
|
+
- Logs record discrete events and become more useful when correlated with traces.
|
|
27
|
+
- Profiles identify where resources are consumed and can be correlated with other signals.
|
|
28
|
+
|
|
29
|
+
## Context and propagation
|
|
30
|
+
|
|
31
|
+
Context contains information used to correlate work across service or execution boundaries. Propagation moves that context between processes or services by serializing it into a carrier and recovering it downstream. The default OpenTelemetry propagator uses W3C Trace Context headers.
|
|
32
|
+
|
|
33
|
+
Operational consequences:
|
|
34
|
+
|
|
35
|
+
- Broken propagation produces disconnected traces even when every service emits spans.
|
|
36
|
+
- Trace and span identifiers can correlate logs with the request that produced them.
|
|
37
|
+
- Incoming context from untrusted sources can be forged; outgoing context can disclose internal information.
|
|
38
|
+
- Baggage travels across boundaries and must not contain credentials, tokens, personal data, or other sensitive values.
|
|
39
|
+
|
|
40
|
+
Manual propagation implementation is outside this skill. Diagnose propagation by verifying carrier presence, trust-boundary policy, and continuity of trace and parent identifiers.
|
|
41
|
+
|
|
42
|
+
## Resources
|
|
43
|
+
|
|
44
|
+
A resource describes the entity producing telemetry. Typical dimensions include service identity, host, operating system, process, container, Kubernetes workload, and cloud platform.
|
|
45
|
+
|
|
46
|
+
Resource guidance:
|
|
47
|
+
|
|
48
|
+
- Make service identity explicit and stable enough to distinguish logical services.
|
|
49
|
+
- Use semantic-convention attribute names where defined.
|
|
50
|
+
- Use resource detection for environmental facts, then verify the detected values.
|
|
51
|
+
- Avoid attributes whose values change per request; those belong on individual telemetry records, not the producing resource.
|
|
52
|
+
- Ensure identities remain globally distinguishable when multiple Collectors or environments report to the same backend.
|
|
53
|
+
|
|
54
|
+
## Instrumentation scope
|
|
55
|
+
|
|
56
|
+
Instrumentation scope identifies the logical source that emitted telemetry. Its identity includes a required name and optional version, schema URL, and attributes. Backends can use the scope to filter, group, or compare telemetry from different components or versions.
|
|
57
|
+
|
|
58
|
+
Keep resource and scope separate:
|
|
59
|
+
|
|
60
|
+
- Resource answers “what entity produced this telemetry?”
|
|
61
|
+
- Instrumentation scope answers “what logical instrumentation source produced this record?”
|
|
62
|
+
|
|
63
|
+
## Semantic conventions
|
|
64
|
+
|
|
65
|
+
Semantic conventions standardize names and meanings for common operations and resources. They make telemetry portable and comparable across sources and backends.
|
|
66
|
+
|
|
67
|
+
When reviewing telemetry:
|
|
68
|
+
|
|
69
|
+
1. Check whether an applicable convention exists.
|
|
70
|
+
2. Check the current convention version and stability before asserting a field is required or stable.
|
|
71
|
+
3. Preserve the semantic meaning during transformation; renaming without a mapping can break dashboards, alerts, and queries.
|
|
72
|
+
4. Treat custom attributes as additions, not replacements for applicable standard attributes.
|
|
73
|
+
|
|
74
|
+
## Terminology checkpoints
|
|
75
|
+
|
|
76
|
+
| Term | Operational meaning |
|
|
77
|
+
| --- | --- |
|
|
78
|
+
| Telemetry | Data emitted by a system about its behavior. |
|
|
79
|
+
| Signal | A category of telemetry, such as traces, metrics, logs, baggage, or profiles. |
|
|
80
|
+
| OTLP | The standard OpenTelemetry protocol for telemetry transport. |
|
|
81
|
+
| Resource | Metadata describing the entity that produced telemetry. |
|
|
82
|
+
| Attribute | Key-value metadata attached to telemetry. |
|
|
83
|
+
| Context | Correlation state associated with an execution. |
|
|
84
|
+
| Propagation | Movement of context across boundaries. |
|
|
85
|
+
| Baggage | User-defined contextual key-value data propagated alongside context. |
|
|
86
|
+
| Instrumentation scope | Identity of the logical source that emitted telemetry. |
|
|
87
|
+
| Collector | A vendor-agnostic service that receives, processes, and exports telemetry. |
|
|
88
|
+
|
|
89
|
+
## Official sources
|
|
90
|
+
|
|
91
|
+
- [What is OpenTelemetry?](https://opentelemetry.io/docs/what-is-opentelemetry/)
|
|
92
|
+
- [Observability primer](https://opentelemetry.io/docs/concepts/observability-primer/)
|
|
93
|
+
- [Components](https://opentelemetry.io/docs/concepts/components/)
|
|
94
|
+
- [Context propagation](https://opentelemetry.io/docs/concepts/context-propagation/)
|
|
95
|
+
- [Resources](https://opentelemetry.io/docs/concepts/resources/)
|
|
96
|
+
- [Instrumentation scope](https://opentelemetry.io/docs/concepts/instrumentation-scope/)
|
|
97
|
+
- [Semantic conventions](https://opentelemetry.io/docs/concepts/semantic-conventions/)
|
|
98
|
+
- [Glossary](https://opentelemetry.io/docs/concepts/glossary/)
|
|
@@ -0,0 +1,108 @@
|
|
|
1
|
+
# Platforms and Zero-Code Instrumentation
|
|
2
|
+
|
|
3
|
+
Use this reference when telemetry must be added or operated without editing application source. Do not turn it into language-specific integration guidance.
|
|
4
|
+
|
|
5
|
+
## Zero-code model
|
|
6
|
+
|
|
7
|
+
Zero-code instrumentation adds telemetry capabilities without application source changes. Depending on the environment, it can use runtime agents, process injection, platform hooks, bytecode techniques, monkey patching, or eBPF.
|
|
8
|
+
|
|
9
|
+
It commonly captures framework-visible activity such as inbound and outbound requests, database calls, and messaging operations. It generally cannot infer custom business events, application-specific attributes, or arbitrary internal operations that are not visible to the mechanism.
|
|
10
|
+
|
|
11
|
+
Operational configuration usually includes:
|
|
12
|
+
|
|
13
|
+
- Service and resource identity.
|
|
14
|
+
- Enabled instrumentations or observed protocols.
|
|
15
|
+
- Export destination and transport.
|
|
16
|
+
- Propagators and context behavior.
|
|
17
|
+
- Sampling and signal enablement.
|
|
18
|
+
- Resource limits and performance controls.
|
|
19
|
+
|
|
20
|
+
Verify current support for the target language, runtime, framework, operating system, and deployment platform. Do not infer support from a different runtime.
|
|
21
|
+
|
|
22
|
+
## Selection guide
|
|
23
|
+
|
|
24
|
+
| Approach | Best fit | Main trade-offs |
|
|
25
|
+
| --- | --- | --- |
|
|
26
|
+
| Runtime or language agent | Supported runtimes where library-level operations must be observed without source changes | Runtime-specific support, startup and performance impact, version compatibility |
|
|
27
|
+
| Kubernetes Operator injection | Centrally managed Kubernetes workloads with supported automatic-instrumentation images | Admission/injection policy, image lifecycle, resource overhead, rollout coordination |
|
|
28
|
+
| OBI/eBPF | Linux environments requiring protocol and network visibility without process-level source changes | Kernel, architecture, privileges, protocol visibility, and application-detail limitations |
|
|
29
|
+
| FaaS layer | Supported serverless runtimes where a published instrumentation or Collector layer is available | Cold-start impact, regional artifacts, layer lifecycle, limited runtime control |
|
|
30
|
+
| Collector-only infrastructure collection | Host, container, Kubernetes, network, and existing log or metric sources | Does not create application-internal spans that the Collector cannot observe |
|
|
31
|
+
|
|
32
|
+
## Kubernetes
|
|
33
|
+
|
|
34
|
+
Choose the Collector mode based on the data source and processing requirement:
|
|
35
|
+
|
|
36
|
+
- DaemonSet for node-local collection such as host metrics and node logs.
|
|
37
|
+
- Sidecar when a workload requires strong local isolation or per-workload collection behavior.
|
|
38
|
+
- Deployment or StatefulSet for shared gateways, central processing, or components with identity/state requirements.
|
|
39
|
+
|
|
40
|
+
The OpenTelemetry Operator can manage Collector custom resources and inject supported automatic instrumentation. Helm charts provide another declarative deployment mechanism. Treat Operator and chart versions as time-sensitive and verify their current custom-resource fields and defaults.
|
|
41
|
+
|
|
42
|
+
### Prometheus collection
|
|
43
|
+
|
|
44
|
+
Avoid having multiple Collectors scrape the same target. The Target Allocator separates target discovery from collection and distributes Prometheus targets across Collector instances. Verify the reconciled configuration because enabling allocation can replace existing service-discovery entries with allocator-provided discovery.
|
|
45
|
+
|
|
46
|
+
### Horizontal scaling
|
|
47
|
+
|
|
48
|
+
Before enabling autoscaling:
|
|
49
|
+
|
|
50
|
+
1. Set realistic CPU and memory requests and limits.
|
|
51
|
+
2. Confirm the metrics source used by the autoscaler.
|
|
52
|
+
3. Choose signals that reflect Collector pressure rather than only host utilization.
|
|
53
|
+
4. Preserve stateful-routing requirements.
|
|
54
|
+
5. Set minimum capacity for failure tolerance and maximum capacity for backend protection.
|
|
55
|
+
|
|
56
|
+
## OpenTelemetry eBPF Instrumentation (OBI)
|
|
57
|
+
|
|
58
|
+
OBI observes Linux application executables and networking with eBPF and can emit application, network, trace, metric, and correlation data without source changes. Its precise protocol coverage, feature status, artifacts, kernel requirements, and supported architectures change over time; verify the current OBI documentation.
|
|
59
|
+
|
|
60
|
+
Selection constraints:
|
|
61
|
+
|
|
62
|
+
- Linux kernel and BTF compatibility.
|
|
63
|
+
- Supported CPU architecture.
|
|
64
|
+
- Access to process, network, and kernel interfaces.
|
|
65
|
+
- Required Linux capabilities for the enabled features.
|
|
66
|
+
- Visibility limits for encrypted, asynchronous, runtime-specific, or application-specific behavior.
|
|
67
|
+
- Cardinality and telemetry-volume controls.
|
|
68
|
+
|
|
69
|
+
Use least privilege. Prefer the exact capabilities required by enabled features over unrestricted root access. Treat `CAP_SYS_ADMIN` and host-level access as high-risk and document why they are needed.
|
|
70
|
+
|
|
71
|
+
OBI is not a substitute for custom business telemetry. If the requirement depends on application semantics that OBI cannot observe, identify that limitation without generating manual instrumentation code.
|
|
72
|
+
|
|
73
|
+
## FaaS
|
|
74
|
+
|
|
75
|
+
Serverless environments have platform-specific lifecycle, startup, networking, and packaging constraints. For supported Lambda layers:
|
|
76
|
+
|
|
77
|
+
- Confirm the layer exists for the runtime and region.
|
|
78
|
+
- Account for initialization and cold-start overhead.
|
|
79
|
+
- Add a Collector layer or verified external Collector path when the instrumentation layer requires it.
|
|
80
|
+
- Configure secrets through the platform secret mechanism or protected environment injection.
|
|
81
|
+
- Verify the transport, endpoint, and layer compatibility before rollout.
|
|
82
|
+
- Publish and test a new function version with rollback available.
|
|
83
|
+
|
|
84
|
+
Do not copy version numbers or layer ARNs from a snapshot. Resolve the current official release for the target region at execution time.
|
|
85
|
+
|
|
86
|
+
## Rollout checklist
|
|
87
|
+
|
|
88
|
+
1. Inventory platform, runtime, kernel, architecture, and workload restrictions.
|
|
89
|
+
2. Confirm the zero-code method supports the target.
|
|
90
|
+
3. Define expected signals and known blind spots.
|
|
91
|
+
4. Establish service/resource identity and export path.
|
|
92
|
+
5. Review privileges, network exposure, and secrets.
|
|
93
|
+
6. Measure startup, CPU, memory, latency, and telemetry volume on a limited rollout.
|
|
94
|
+
7. Verify trace continuity, metric cardinality, log correlation, and backend ingestion.
|
|
95
|
+
8. Document rollback and upgrade ownership.
|
|
96
|
+
|
|
97
|
+
## Official sources
|
|
98
|
+
|
|
99
|
+
- [Zero-code instrumentation concept](https://opentelemetry.io/docs/concepts/instrumentation/zero-code/)
|
|
100
|
+
- [Zero-code instrumentation documentation](https://opentelemetry.io/docs/zero-code/)
|
|
101
|
+
- [OpenTelemetry eBPF Instrumentation](https://opentelemetry.io/docs/zero-code/obi/)
|
|
102
|
+
- [OBI security, permissions, and capabilities](https://opentelemetry.io/docs/zero-code/obi/security/)
|
|
103
|
+
- [OpenTelemetry with Kubernetes](https://opentelemetry.io/docs/platforms/kubernetes/)
|
|
104
|
+
- [OpenTelemetry Operator](https://opentelemetry.io/docs/platforms/kubernetes/operator/)
|
|
105
|
+
- [Target Allocator](https://opentelemetry.io/docs/platforms/kubernetes/operator/target-allocator/)
|
|
106
|
+
- [Functions as a Service](https://opentelemetry.io/docs/platforms/faas/)
|
|
107
|
+
- [Lambda auto-instrumentation](https://opentelemetry.io/docs/platforms/faas/lambda-auto-instrument/)
|
|
108
|
+
- [Lambda Collector configuration](https://opentelemetry.io/docs/platforms/faas/lambda-collector/)
|
|
@@ -0,0 +1,119 @@
|
|
|
1
|
+
# OpenTelemetry Security
|
|
2
|
+
|
|
3
|
+
Use this reference for Collector hardening, sensitive-data controls, context trust boundaries, and security review. OpenTelemetry cannot decide which data is sensitive for a particular organization; that responsibility remains with the operator.
|
|
4
|
+
|
|
5
|
+
## Security objectives
|
|
6
|
+
|
|
7
|
+
- Minimize telemetry collection to data with an explicit observability purpose.
|
|
8
|
+
- Protect telemetry in transit, in memory, on disk, and in the backend.
|
|
9
|
+
- Restrict Collector listeners and administrative endpoints.
|
|
10
|
+
- Authenticate peers and authorize access.
|
|
11
|
+
- Run with the least filesystem, network, Kubernetes, and kernel privilege.
|
|
12
|
+
- Prevent resource exhaustion and uncontrolled cardinality.
|
|
13
|
+
- Keep credentials out of configuration files and telemetry.
|
|
14
|
+
- Monitor current OpenTelemetry security advisories.
|
|
15
|
+
|
|
16
|
+
## Sensitive data
|
|
17
|
+
|
|
18
|
+
Potentially sensitive telemetry includes personal information, credentials, session tokens, financial or health data, user behavior, request and response content, database statements, URLs, headers, logs, and baggage.
|
|
19
|
+
|
|
20
|
+
Apply controls in this order:
|
|
21
|
+
|
|
22
|
+
1. Do not collect unnecessary data.
|
|
23
|
+
2. Prefer aggregation, truncation, or categorization over unique identifiers.
|
|
24
|
+
3. Use an allowlist when the acceptable attribute set is known.
|
|
25
|
+
4. Remove or mask sensitive values before they cross a trust boundary.
|
|
26
|
+
5. Restrict access and retention for telemetry that must remain identifiable.
|
|
27
|
+
6. Periodically inspect actual emitted telemetry because instrumentation and schemas change.
|
|
28
|
+
|
|
29
|
+
Hashing does not automatically anonymize predictable values. A small or enumerable input space can permit reversal by guessing. Document the threat model before treating hashing as a privacy control.
|
|
30
|
+
|
|
31
|
+
## Collector network hardening
|
|
32
|
+
|
|
33
|
+
- Bind server-like components to the narrowest required interface. Prefer loopback for local-only clients.
|
|
34
|
+
- Do not expose receivers, health endpoints, diagnostics, or management endpoints to the public internet without an explicit design.
|
|
35
|
+
- Use TLS for network hops that cross trust boundaries.
|
|
36
|
+
- Authenticate receivers and exporters where supported.
|
|
37
|
+
- Restrict inbound and outbound traffic with platform network controls.
|
|
38
|
+
- Review trust separately for application-to-agent, agent-to-gateway, and gateway-to-backend hops.
|
|
39
|
+
|
|
40
|
+
An illustrative local-only receiver is:
|
|
41
|
+
|
|
42
|
+
```yaml
|
|
43
|
+
receivers:
|
|
44
|
+
otlp:
|
|
45
|
+
protocols:
|
|
46
|
+
grpc:
|
|
47
|
+
endpoint: 127.0.0.1:4317
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
Container and Kubernetes networking can make loopback inappropriate for cross-container traffic. Select the interface from the actual topology rather than replacing it with unrestricted `0.0.0.0` by default.
|
|
51
|
+
|
|
52
|
+
## Secrets and configuration
|
|
53
|
+
|
|
54
|
+
- Store secrets in the deployment platform's protected secret mechanism.
|
|
55
|
+
- Inject secret values at runtime; do not commit them in Collector YAML.
|
|
56
|
+
- Restrict read access to configuration, certificates, queues, WAL directories, and raw telemetry.
|
|
57
|
+
- Avoid printing resolved configuration or verbose telemetry where secrets may appear.
|
|
58
|
+
- Rotate credentials and certificates without requiring broad unrelated configuration changes.
|
|
59
|
+
|
|
60
|
+
Use placeholders in examples:
|
|
61
|
+
|
|
62
|
+
```yaml
|
|
63
|
+
exporters:
|
|
64
|
+
otlp:
|
|
65
|
+
endpoint: telemetry-backend.example:4317
|
|
66
|
+
headers:
|
|
67
|
+
authorization: ${env:OTEL_EXPORTER_AUTHORIZATION}
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
## Least privilege
|
|
71
|
+
|
|
72
|
+
- Avoid running the Collector as root unless a required data source cannot be accessed otherwise.
|
|
73
|
+
- Grant read-only filesystem mounts when collection does not require writes.
|
|
74
|
+
- Scope Kubernetes RBAC to the resources and verbs required by enabled receivers, processors, observers, or the Operator.
|
|
75
|
+
- Review observer and service-discovery extensions because discovery can require broad environment visibility.
|
|
76
|
+
- For OBI/eBPF, grant only capabilities required by enabled modes and document any use of powerful capabilities.
|
|
77
|
+
|
|
78
|
+
## Resource-exhaustion controls
|
|
79
|
+
|
|
80
|
+
- Set workload CPU and memory requests and limits.
|
|
81
|
+
- Put the memory limiter early in pipelines.
|
|
82
|
+
- Bound exporter queues and retry duration.
|
|
83
|
+
- Filter telemetry that has no operational value.
|
|
84
|
+
- Control metric and log cardinality.
|
|
85
|
+
- Rate-limit or isolate untrusted senders.
|
|
86
|
+
- Alert on refusal, queue saturation, export failure, CPU, memory, disk, and restart signals.
|
|
87
|
+
|
|
88
|
+
Scaling does not protect a saturated backend and can amplify a denial-of-service condition. Verify the constrained component before scaling.
|
|
89
|
+
|
|
90
|
+
## Context and baggage boundaries
|
|
91
|
+
|
|
92
|
+
- Treat incoming trace context as untrusted unless the source is authenticated and authorized.
|
|
93
|
+
- Define whether external trace context is accepted, replaced, or sanitized.
|
|
94
|
+
- Prevent internal trace identifiers or baggage from being forwarded to untrusted external services when disclosure is unacceptable.
|
|
95
|
+
- Never store credentials, API keys, session tokens, or personal data in baggage.
|
|
96
|
+
- Remember that baggage has no built-in guarantee that a value originated from a trusted service.
|
|
97
|
+
|
|
98
|
+
## Security review checklist
|
|
99
|
+
|
|
100
|
+
1. Map data sources, Collector tiers, destinations, and trust boundaries.
|
|
101
|
+
2. Inventory listeners, protocols, addresses, ports, and administrative endpoints.
|
|
102
|
+
3. Verify TLS, authentication, and authorization for each network hop.
|
|
103
|
+
4. Inspect configuration and secret delivery.
|
|
104
|
+
5. Inventory attributes, log bodies, baggage, and transformed data for sensitivity.
|
|
105
|
+
6. Review filtering, redaction, and failure behavior.
|
|
106
|
+
7. Review filesystem, RBAC, container, and kernel privileges.
|
|
107
|
+
8. Review resource limits, queue bounds, and abuse controls.
|
|
108
|
+
9. Verify internal telemetry and security-alert ownership.
|
|
109
|
+
10. Check current CVEs and advisories for deployed components and versions.
|
|
110
|
+
|
|
111
|
+
## Official sources
|
|
112
|
+
|
|
113
|
+
- [OpenTelemetry security](https://opentelemetry.io/docs/security/)
|
|
114
|
+
- [Collector configuration security](https://opentelemetry.io/docs/security/config-best-practices/)
|
|
115
|
+
- [Collector hosting security](https://opentelemetry.io/docs/security/hosting-best-practices/)
|
|
116
|
+
- [Handling sensitive data](https://opentelemetry.io/docs/security/handling-sensitive-data/)
|
|
117
|
+
- [OpenTelemetry CVEs](https://opentelemetry.io/docs/security/cve/)
|
|
118
|
+
- [Context propagation security](https://opentelemetry.io/docs/concepts/context-propagation/#security-best-practices)
|
|
119
|
+
- [Baggage security considerations](https://opentelemetry.io/docs/concepts/signals/baggage/#baggage-security-considerations)
|