@moon791017/neo-skills 1.1.11 → 1.1.13
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +4 -0
- package/package.json +1 -1
- package/skills/neo-agentic-design/SKILL.md +89 -0
- package/skills/neo-agentic-design/evals/eval_queries.json +58 -0
- package/skills/neo-agentic-design/evals/evals.json +27 -0
- package/skills/neo-agentic-design/references/advanced-safety.md +158 -0
- package/skills/neo-agentic-design/references/base-workflows.md +219 -0
- package/skills/neo-agentic-design/references/resilience-hitl.md +105 -0
- package/skills/neo-agentic-design/references/system-components.md +93 -0
- package/skills/neo-opentelemetry/SKILL.md +94 -0
- package/skills/neo-opentelemetry/evals/eval_queries.json +82 -0
- package/skills/neo-opentelemetry/evals/evals.json +64 -0
- package/skills/neo-opentelemetry/references/collector-architecture-and-configuration.md +138 -0
- package/skills/neo-opentelemetry/references/collector-operations.md +137 -0
- package/skills/neo-opentelemetry/references/compatibility-and-guidance.md +112 -0
- package/skills/neo-opentelemetry/references/foundations.md +98 -0
- package/skills/neo-opentelemetry/references/platforms-and-zero-code.md +108 -0
- package/skills/neo-opentelemetry/references/security.md +119 -0
- package/skills/neo-opentelemetry/references/signals-and-sampling.md +113 -0
- package/skills/neo-opentelemetry/references/sources-and-scope.md +60 -0
|
@@ -0,0 +1,105 @@
|
|
|
1
|
+
# Resilience, Exceptions & HITL
|
|
2
|
+
|
|
3
|
+
This document provides conceptual designs for system resilience, human interaction, and knowledge grounding, covering exception handling, Human-in-the-Loop (HITL) gates, and Retrieval-Augmented Generation (RAG).
|
|
4
|
+
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
## Chapter 12: Exception Handling and Recovery
|
|
8
|
+
|
|
9
|
+
### 1. Definition
|
|
10
|
+
Designs automatic detection, retry, fallback, and state rollback mechanisms for exceptions that may occur during agent execution (such as API timeouts, network disconnections, LLM format errors, and invalid tool parameters).
|
|
11
|
+
|
|
12
|
+
### 2. Problems Addressed
|
|
13
|
+
* System fragility: Prevents long-cycle workflows from breaking due to transient network or API issues.
|
|
14
|
+
* Format pollution: Guides the LLM to self-heal when its output does not conform to the expected JSON schema.
|
|
15
|
+
|
|
16
|
+
### 3. Workflow
|
|
17
|
+
```mermaid
|
|
18
|
+
graph TD
|
|
19
|
+
Step[Execute Tool / Call LLM] --> Success{Successful?}
|
|
20
|
+
Success -->|Yes| Next[Proceed to Next Step]
|
|
21
|
+
Success -->|No: Exception| Detector[Exception Detector]
|
|
22
|
+
Detector --> RuleCheck{Evaluate Exception Type}
|
|
23
|
+
RuleCheck -->|Network/Timeout| Retry[Auto Retry with Backoff]
|
|
24
|
+
RuleCheck -->|Format Error| Refine[Guide LLM to Self-Correct]
|
|
25
|
+
RuleCheck -->|Tool Failure| Fallback[Route to Fallback/Alternative Tool]
|
|
26
|
+
RuleCheck -->|Critical Error| Rollback[Rollback State to Checkpoint]
|
|
27
|
+
Retry --> Step
|
|
28
|
+
Refine --> Step
|
|
29
|
+
Fallback --> Step
|
|
30
|
+
Rollback --> UserEscalation[Human Intervention]
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
### 4. Trade-offs
|
|
34
|
+
* **Pros**: Improves system robustness and reduces manual maintenance costs.
|
|
35
|
+
* **Cons**: Excessive retries or fallbacks can mask underlying bugs or quietly degrade output quality.
|
|
36
|
+
|
|
37
|
+
---
|
|
38
|
+
|
|
39
|
+
## Chapter 13: Human-in-the-Loop (HITL)
|
|
40
|
+
|
|
41
|
+
### 1. Definition
|
|
42
|
+
Strategically embeds human review, intervention, and authorization mechanisms into the agent's autonomous decision-making workflow, combining human common sense, ethics, and legal judgment with AI automation.
|
|
43
|
+
|
|
44
|
+
### 2. Problems Addressed
|
|
45
|
+
* High-risk operations: Prevents agent errors when performing large financial transactions, deleting sensitive data, or executing legally sensitive actions.
|
|
46
|
+
* Automation boundaries: Requests human guidance when decision confidence falls below a set threshold.
|
|
47
|
+
|
|
48
|
+
### 3. Three Core Interaction Modes
|
|
49
|
+
````carousel
|
|
50
|
+
### 1. Human-in-the-Loop (HITL)
|
|
51
|
+
* **Mechanism**: The agent pauses when reaching a high-risk step (e.g., large bank transfer), suspends the task, and sends it to a pending review queue.
|
|
52
|
+
* **Workflow**: Agent pauses -> Human reviews (Approve/Reject/Modify) -> Agent receives input and resumes execution.
|
|
53
|
+
* **Key Characteristic**: Human approval is a mandatory gate.
|
|
54
|
+
<!-- slide -->
|
|
55
|
+
### 2. Human-on-the-Loop (HOTL)
|
|
56
|
+
* **Mechanism**: The agent executes tasks autonomously while a human supervisor monitors and adjusts strategies.
|
|
57
|
+
* **Workflow**: Human sets macro rules (e.g., transaction limits) -> Agent trades automatically -> Human monitors metrics -> Human intervenes via a Kill Switch if necessary.
|
|
58
|
+
* **Key Characteristic**: Human does not intervene in individual decisions but maintains macro-level oversight.
|
|
59
|
+
<!-- slide -->
|
|
60
|
+
### 3. Decision Augmentation
|
|
61
|
+
* **Mechanism**: The agent acts as an analytical assistant, gathering data and presenting candidates. Decision-making and execution are performed entirely by a human.
|
|
62
|
+
* **Workflow**: Human asks query -> Agent collects and analyzes data -> Agent proposes options A, B, and C with pros/cons -> Human selects and executes.
|
|
63
|
+
* **Key Characteristic**: Agent provides cognitive augmentation without execution authority.
|
|
64
|
+
````
|
|
65
|
+
|
|
66
|
+
### 4. Trade-offs
|
|
67
|
+
* **Pros**: Provides a safety net and compliance guarantee for high-risk decisions; collects human feedback to optimize agent alignment.
|
|
68
|
+
* **Cons**: Human intervention limits system scalability and speed; designing human-in-the-loop review queues increases development costs.
|
|
69
|
+
|
|
70
|
+
---
|
|
71
|
+
|
|
72
|
+
## Chapter 14: Knowledge Retrieval / RAG
|
|
73
|
+
|
|
74
|
+
### 1. Definition
|
|
75
|
+
Retrieves relevant information from a knowledge base before the LLM generates a response, injecting the retrieved text chunks into the prompt context to guide the LLM toward producing factually grounded answers.
|
|
76
|
+
|
|
77
|
+
### 2. Advanced Agentic RAG Variants
|
|
78
|
+
```mermaid
|
|
79
|
+
graph TD
|
|
80
|
+
subgraph Traditional RAG
|
|
81
|
+
Query[User Query] --> VectorSearch[Vector Similarity Search]
|
|
82
|
+
VectorSearch --> Context[Concatenate Context Chunks]
|
|
83
|
+
Context --> LLMGen[LLM Generates Response]
|
|
84
|
+
end
|
|
85
|
+
subgraph Graph RAG
|
|
86
|
+
GQuery[User Query] --> GraphSearch[Navigate Knowledge Graph Nodes & Edges]
|
|
87
|
+
GraphSearch --> UnifiedContext[Cross-document Context Linkage]
|
|
88
|
+
end
|
|
89
|
+
subgraph Agentic RAG
|
|
90
|
+
AQuery[User Query] --> AgentLayer[Agent Decision Layer]
|
|
91
|
+
AgentLayer -->|1. Decompose Task| SubQueries[Multi-step Sub-retrieval Tasks]
|
|
92
|
+
AgentLayer -->|2. Self-Reflection| SourceVal[Source Timeliness & Quality Check]
|
|
93
|
+
AgentLayer -->|3. Resolve Conflicts| ConflictRecon[Active Conflict Reconciliation]
|
|
94
|
+
AgentLayer -->|4. Tool Call| WebSearch[Web Search for Knowledge Gaps]
|
|
95
|
+
end
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
### 3. Problems Addressed
|
|
99
|
+
* Outdated knowledge: Bypasses the temporal limits of static training data.
|
|
100
|
+
* Hallucination: Restricts the model within factual boundaries using verified document contexts.
|
|
101
|
+
* Fragmented information: Resolves vector search limitations that struggle to answer comprehensive questions spanning multiple documents.
|
|
102
|
+
|
|
103
|
+
### 4. Trade-offs
|
|
104
|
+
* **Pros**: Minimizes factual errors; supports precise citations; imports private knowledge without retraining models.
|
|
105
|
+
* **Cons**: Highly sensitive to the quality of text chunking and embeddings; multi-step reasoning in Agentic RAG increases response latency.
|
|
@@ -0,0 +1,93 @@
|
|
|
1
|
+
# System Components & Protocols
|
|
2
|
+
|
|
3
|
+
This document provides conceptual designs for system architecture components, resources, and protocols, covering memory management, learning and adaptation, Model Context Protocol (MCP), and goal setting and monitoring.
|
|
4
|
+
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
## Chapter 8: Memory Management
|
|
8
|
+
|
|
9
|
+
### 1. Definition
|
|
10
|
+
Provides agents with the ability to store and retrieve information across sessions and tasks through persistence mechanisms. The memory system is generally divided into short-term and long-term memory, managed by a unified Memory Service.
|
|
11
|
+
|
|
12
|
+
### 2. Memory Classification
|
|
13
|
+
| Memory Type | Medium | Function | Eviction & Retrieval Mechanism |
|
|
14
|
+
| :--- | :--- | :--- | :--- |
|
|
15
|
+
| **Short-term** | Current Context Window | Stores current conversation context and task execution trajectory | Sliding window, context pruning, and summarization |
|
|
16
|
+
| **Long-term Semantic** | Vector Database / Knowledge Base | Retains factual knowledge, concepts, and external rules | Vector semantic retrieval based on user input |
|
|
17
|
+
| **Long-term Episodic** | Structured Database / Log Store | Records past task execution experiences and outcomes | Used for few-shot learning or similar scenario matching |
|
|
18
|
+
| **Long-term Procedural**| Codebase / Tool Definitions / Prompt Templates | Records Standard Operating Procedures (SOPs) and toolbox definitions for specific tasks | Dynamically loaded based on task type |
|
|
19
|
+
|
|
20
|
+
### 3. Problems Addressed
|
|
21
|
+
* Amnesia (Context limits): Prevents long conversations from causing the LLM to lose critical history.
|
|
22
|
+
* Repeated errors: Ensures the agent learns from past executions to improve decision success rates.
|
|
23
|
+
|
|
24
|
+
---
|
|
25
|
+
|
|
26
|
+
## Chapter 9: Learning and Adaptation
|
|
27
|
+
|
|
28
|
+
### 1. Definition
|
|
29
|
+
Enables the agent to autonomously modify prompts or self-modify execution code in a code sandbox (SICA - Self-Improving Coding Agent) by collecting behavioral feedback and rewards from interactions with the environment, users, or other agents.
|
|
30
|
+
|
|
31
|
+
### 2. Problems Addressed
|
|
32
|
+
* Static configuration lag: Solves the issue of agents failing to adjust when environmental rules change.
|
|
33
|
+
* High development cost: Eliminates the manual process of fine-tuning prompts.
|
|
34
|
+
|
|
35
|
+
### 3. Workflow
|
|
36
|
+
```mermaid
|
|
37
|
+
graph TD
|
|
38
|
+
Interaction[Agent-Environment Interaction] --> Result[Execution Results & Metrics]
|
|
39
|
+
Result --> evaluator[Evaluator / Scoring System]
|
|
40
|
+
evaluator -->|Feedback/Score| Learner[Learning Engine]
|
|
41
|
+
Learner -->|Self-Optimize Prompts or Refactor Code| AgentUpgrade[Upgraded Agent]
|
|
42
|
+
AgentUpgrade -->|Next Task Turn| Interaction
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
### 4. Trade-offs
|
|
46
|
+
* **Pros**: High potential for long-term self-evolution; can discover high-quality logic not designed by humans in specific vertical disciplines (e.g., mathematical proofs, code generation).
|
|
47
|
+
* **Cons**: Unpredictable evolution paths, which may generate harmful mutations; self-modifying prompts can lead to privilege escalation or security vulnerabilities; extremely high overhead for training and testing iterations.
|
|
48
|
+
|
|
49
|
+
---
|
|
50
|
+
|
|
51
|
+
## Chapter 10: Model Context Protocol (MCP)
|
|
52
|
+
|
|
53
|
+
### 1. Definition
|
|
54
|
+
A standardized **Client-Server communication protocol** that establishes a plug-and-play integration standard between LLMs/Agents (Clients) and external data sources, development tools, and API services (Servers). MCP standardizes three core types of context exchange: **Resources**, **Prompts**, and **Tools**.
|
|
55
|
+
|
|
56
|
+
```mermaid
|
|
57
|
+
graph LR
|
|
58
|
+
subgraph Agentic Client
|
|
59
|
+
Agent[AI Agent / LLM]
|
|
60
|
+
end
|
|
61
|
+
subgraph MCP Server
|
|
62
|
+
Res[Resources: Files/Databases]
|
|
63
|
+
Pmt[Prompts: Templates]
|
|
64
|
+
Tls[Tools: APIs/Sandboxes]
|
|
65
|
+
end
|
|
66
|
+
Agent <-->|Standard JSON-RPC 2.0| MCP_Link[MCP Protocol Layer]
|
|
67
|
+
MCP_Link <--> Res
|
|
68
|
+
MCP_Link <--> Pmt
|
|
69
|
+
MCP_Link <--> Tls
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
### 2. Problems Addressed
|
|
73
|
+
* Tedious integration: Avoids repeatedly writing custom wrapper code when developing new agents or integrating new tools.
|
|
74
|
+
* Fragmented context acquisition: Provides external data and actions to the model in a unified interface format.
|
|
75
|
+
|
|
76
|
+
### 3. Trade-offs
|
|
77
|
+
* **Pros**: Reduces integration costs for multiple tools and data sources; decouples data sources from reasoning entities; supports dynamic discovery.
|
|
78
|
+
* **Cons**: Protocol serialization and JSON-RPC wrapping introduce minor performance overhead; requires tool providers to actively adopt the protocol.
|
|
79
|
+
|
|
80
|
+
---
|
|
81
|
+
|
|
82
|
+
## Chapter 11: Goal Setting and Monitoring
|
|
83
|
+
|
|
84
|
+
### 1. Definition
|
|
85
|
+
Sets structured and quantifiable goals (SMART principles) before agent initialization, and introduces an independent monitor during the execution phase to observe progress in real time (Progress Checkpoints), detect blocks, and trigger human-agent collaboration escalation when necessary.
|
|
86
|
+
|
|
87
|
+
### 2. Problems Addressed
|
|
88
|
+
* Blind execution: Prevents agents from entering infinite retry loops when encountering logical obstacles, wasting budget.
|
|
89
|
+
* Lack of observability: Solves the black-box execution problem, providing a clear progress path.
|
|
90
|
+
|
|
91
|
+
### 3. Use Cases
|
|
92
|
+
* Automated marketing campaign execution.
|
|
93
|
+
* Long-cycle autonomous codebase refactoring.
|
|
@@ -0,0 +1,94 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: neo-opentelemetry
|
|
3
|
+
description: >
|
|
4
|
+
Use this skill when explaining OpenTelemetry concepts, choosing telemetry signals,
|
|
5
|
+
planning or reviewing Collector architectures and declarative configuration,
|
|
6
|
+
operating or troubleshooting telemetry pipelines, securing telemetry data,
|
|
7
|
+
evaluating zero-code instrumentation, or planning Prometheus, OpenCensus, or
|
|
8
|
+
OpenTracing interoperability and migration. Do not use it to write language SDK/API
|
|
9
|
+
instrumentation code or custom Collector components.
|
|
10
|
+
compatibility: No runtime required. Network access is required to verify current versions, component status, security advisories, and other time-sensitive claims.
|
|
11
|
+
metadata:
|
|
12
|
+
version: "1.0.0"
|
|
13
|
+
pattern: "tool-wrapper,pipeline"
|
|
14
|
+
domain: "observability-operations"
|
|
15
|
+
source-revision: "b18eb4d01fab9be2c5c22b1aaafdeb23ba97e3d4"
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
# Neo OpenTelemetry
|
|
19
|
+
|
|
20
|
+
Provide source-grounded OpenTelemetry guidance for concepts and operations without generating programming-language instrumentation code.
|
|
21
|
+
|
|
22
|
+
## Scope boundary
|
|
23
|
+
|
|
24
|
+
In scope:
|
|
25
|
+
|
|
26
|
+
- Observability concepts, telemetry signals, correlation, resources, semantic conventions, and sampling.
|
|
27
|
+
- Collector pipelines, components, deployment patterns, declarative configuration, scaling, resiliency, internal telemetry, transformation, and troubleshooting.
|
|
28
|
+
- Kubernetes, FaaS, OpenTelemetry Operator, OBI, and other zero-code operational approaches.
|
|
29
|
+
- Security, sensitive-data handling, Prometheus interoperability, and OpenCensus/OpenTracing migration planning.
|
|
30
|
+
|
|
31
|
+
Out of scope:
|
|
32
|
+
|
|
33
|
+
- Language-specific SDK/API usage, manual instrumentation, source-code examples, and library integration code.
|
|
34
|
+
- Building the Collector from source or developing custom receivers, processors, exporters, connectors, or extensions.
|
|
35
|
+
- Vendor pricing, backend-specific query languages, and backend product administration unless needed only as context for an OpenTelemetry decision.
|
|
36
|
+
|
|
37
|
+
If the request crosses the boundary, answer the in-scope operational portion and identify the excluded portion. Point to the appropriate official language or component-development documentation instead of inventing code.
|
|
38
|
+
|
|
39
|
+
## Workflow
|
|
40
|
+
|
|
41
|
+
1. Classify the request as explanation, architecture decision, configuration review, security review, migration, or troubleshooting.
|
|
42
|
+
2. Load `references/sources-and-scope.md` whenever the answer depends on source coverage, current status, or an excluded area.
|
|
43
|
+
3. Load only the references required by the request:
|
|
44
|
+
- Foundations and terminology: `references/foundations.md`
|
|
45
|
+
- Signals and sampling: `references/signals-and-sampling.md`
|
|
46
|
+
- Collector topology and configuration: `references/collector-architecture-and-configuration.md`
|
|
47
|
+
- Collector operations and troubleshooting: `references/collector-operations.md`
|
|
48
|
+
- Kubernetes, FaaS, Operator, and zero-code: `references/platforms-and-zero-code.md`
|
|
49
|
+
- Security and sensitive data: `references/security.md`
|
|
50
|
+
- Prometheus, migration, and deployment guidance: `references/compatibility-and-guidance.md`
|
|
51
|
+
4. Inspect user-provided configuration and environment evidence before recommending changes. Never assume topology, volume, failure tolerance, trust boundaries, or backend capabilities.
|
|
52
|
+
5. Verify time-sensitive facts against official OpenTelemetry sources. If verification is unavailable, identify the snapshot revision and state that current status cannot be confirmed.
|
|
53
|
+
6. Produce the smallest useful response and cite the official page supporting material claims.
|
|
54
|
+
|
|
55
|
+
## Decision rules
|
|
56
|
+
|
|
57
|
+
- Treat OpenTelemetry as a telemetry generation, collection, processing, and export framework, not as a storage or visualization backend.
|
|
58
|
+
- Default to the simplest deployment pattern that satisfies the stated requirements. Add gateways, stateful routing, persistent queues, or external message queues only for an identified need.
|
|
59
|
+
- Separate symptoms by pipeline stage: generation, reception, processing, queuing, export, network, or backend.
|
|
60
|
+
- Treat sampling, filtering, redaction, and transformation as data-loss or data-quality decisions. State what is intentionally discarded or changed.
|
|
61
|
+
- Never recommend scaling Collectors before checking whether the bottleneck is the Collector, network, or backend.
|
|
62
|
+
- Never place credentials, tokens, personal data, or other sensitive values in examples.
|
|
63
|
+
- Prefer least privilege, restricted listener addresses, authenticated and encrypted network paths, data minimization, and Collector self-observability.
|
|
64
|
+
|
|
65
|
+
## Response contracts
|
|
66
|
+
|
|
67
|
+
For explanations, provide:
|
|
68
|
+
|
|
69
|
+
1. Direct answer.
|
|
70
|
+
2. Relevant OpenTelemetry concepts and boundaries.
|
|
71
|
+
3. Official source links.
|
|
72
|
+
|
|
73
|
+
For architecture or configuration reviews, provide:
|
|
74
|
+
|
|
75
|
+
1. Observed requirements or configuration facts.
|
|
76
|
+
2. Findings ordered by impact.
|
|
77
|
+
3. Recommended topology or declarative changes with trade-offs.
|
|
78
|
+
4. Capacity, resiliency, security, and validation checks.
|
|
79
|
+
5. Official source links.
|
|
80
|
+
|
|
81
|
+
For troubleshooting, provide:
|
|
82
|
+
|
|
83
|
+
1. Observed symptom and missing evidence.
|
|
84
|
+
2. Most likely pipeline stage based on supplied evidence; label this as an inference.
|
|
85
|
+
3. Non-destructive checks ordered from source to destination.
|
|
86
|
+
4. Remediation only after the failing stage is isolated.
|
|
87
|
+
5. Verification signals and official source links.
|
|
88
|
+
|
|
89
|
+
## Fact-check gate
|
|
90
|
+
|
|
91
|
+
- Distinguish verified source facts from recommendations and inferences.
|
|
92
|
+
- Do not convert documentation examples into universal defaults.
|
|
93
|
+
- Do not state version numbers, maturity levels, component availability, default ports, default limits, or CVE status without checking the current official source when the answer depends on them.
|
|
94
|
+
- Say that data is insufficient when the request lacks evidence needed for a safe operational conclusion.
|
|
@@ -0,0 +1,82 @@
|
|
|
1
|
+
[
|
|
2
|
+
{
|
|
3
|
+
"query": "Explain when I should use traces, metrics, logs, and profiles together in OpenTelemetry.",
|
|
4
|
+
"should_trigger": true
|
|
5
|
+
},
|
|
6
|
+
{
|
|
7
|
+
"query": "Review our OpenTelemetry Collector YAML for missing queues, unsafe listeners, and processor ordering.",
|
|
8
|
+
"should_trigger": true
|
|
9
|
+
},
|
|
10
|
+
{
|
|
11
|
+
"query": "Should our Kubernetes telemetry pipeline use Collector agents, gateways, or both?",
|
|
12
|
+
"should_trigger": true
|
|
13
|
+
},
|
|
14
|
+
{
|
|
15
|
+
"query": "Our otel gateway queue keeps filling and spans are being dropped. Help me isolate the bottleneck.",
|
|
16
|
+
"should_trigger": true
|
|
17
|
+
},
|
|
18
|
+
{
|
|
19
|
+
"query": "Design a resilient OpenTelemetry Collector topology that can tolerate a backend outage.",
|
|
20
|
+
"should_trigger": true
|
|
21
|
+
},
|
|
22
|
+
{
|
|
23
|
+
"query": "Check whether our baggage and resource attributes could leak customer data.",
|
|
24
|
+
"should_trigger": true
|
|
25
|
+
},
|
|
26
|
+
{
|
|
27
|
+
"query": "Plan a zero-code observability rollout for Linux workloads using the Operator or OBI.",
|
|
28
|
+
"should_trigger": true
|
|
29
|
+
},
|
|
30
|
+
{
|
|
31
|
+
"query": "How should we migrate Prometheus metric collection into an OTLP and Collector architecture?",
|
|
32
|
+
"should_trigger": true
|
|
33
|
+
},
|
|
34
|
+
{
|
|
35
|
+
"query": "Create an operational migration plan from OpenTracing to OpenTelemetry without breaking dashboards.",
|
|
36
|
+
"should_trigger": true
|
|
37
|
+
},
|
|
38
|
+
{
|
|
39
|
+
"query": "Why are traces split after they cross a service boundary even though both services export spans?",
|
|
40
|
+
"should_trigger": true
|
|
41
|
+
},
|
|
42
|
+
{
|
|
43
|
+
"query": "Write C# ActivitySource instrumentation for my ASP.NET Core checkout endpoint.",
|
|
44
|
+
"should_trigger": false
|
|
45
|
+
},
|
|
46
|
+
{
|
|
47
|
+
"query": "Show me the Java OpenTelemetry SDK calls needed to create a custom histogram.",
|
|
48
|
+
"should_trigger": false
|
|
49
|
+
},
|
|
50
|
+
{
|
|
51
|
+
"query": "Implement a custom OpenTelemetry Collector receiver in Go.",
|
|
52
|
+
"should_trigger": false
|
|
53
|
+
},
|
|
54
|
+
{
|
|
55
|
+
"query": "Build the OpenTelemetry Collector from source with a custom exporter.",
|
|
56
|
+
"should_trigger": false
|
|
57
|
+
},
|
|
58
|
+
{
|
|
59
|
+
"query": "Write a PromQL query for calculating HTTP error rate in Grafana.",
|
|
60
|
+
"should_trigger": false
|
|
61
|
+
},
|
|
62
|
+
{
|
|
63
|
+
"query": "Troubleshoot why my Kubernetes Deployment cannot pull its application image.",
|
|
64
|
+
"should_trigger": false
|
|
65
|
+
},
|
|
66
|
+
{
|
|
67
|
+
"query": "Compare the current pricing of Datadog, New Relic, and Honeycomb.",
|
|
68
|
+
"should_trigger": false
|
|
69
|
+
},
|
|
70
|
+
{
|
|
71
|
+
"query": "Create OpenAPI documentation for my REST endpoints.",
|
|
72
|
+
"should_trigger": false
|
|
73
|
+
},
|
|
74
|
+
{
|
|
75
|
+
"query": "Help me submit a pull request to translate the OpenTelemetry website.",
|
|
76
|
+
"should_trigger": false
|
|
77
|
+
},
|
|
78
|
+
{
|
|
79
|
+
"query": "Design a generic application logging API that does not use OpenTelemetry.",
|
|
80
|
+
"should_trigger": false
|
|
81
|
+
}
|
|
82
|
+
]
|
|
@@ -0,0 +1,64 @@
|
|
|
1
|
+
{
|
|
2
|
+
"skill_name": "neo-opentelemetry",
|
|
3
|
+
"evals": [
|
|
4
|
+
{
|
|
5
|
+
"id": 1,
|
|
6
|
+
"prompt": "We operate a checkout service and only have CPU dashboards. Explain which OpenTelemetry signals would help us detect reliability problems and determine why one request is slow. Do not provide instrumentation code.",
|
|
7
|
+
"expected_output": "A source-grounded explanation that distinguishes metrics, traces, logs, and profiles, describes correlation, and stays outside language SDK/API implementation.",
|
|
8
|
+
"assertions": [
|
|
9
|
+
"The response explains a distinct operational question answered by metrics, traces, logs, and profiles",
|
|
10
|
+
"The response explains that resource identity or trace context is needed for correlation",
|
|
11
|
+
"The response contains at least one official opentelemetry.io documentation link",
|
|
12
|
+
"The response contains no programming-language instrumentation code"
|
|
13
|
+
]
|
|
14
|
+
},
|
|
15
|
+
{
|
|
16
|
+
"id": 2,
|
|
17
|
+
"prompt": "Plan an OpenTelemetry Collector deployment for 60 Kubernetes nodes. We need node logs and host metrics, centralized redaction, one controlled egress point, and tail sampling for traces. State assumptions where capacity data is missing.",
|
|
18
|
+
"expected_output": "An agent-to-gateway architecture with node-local agents, centralized gateway processing, trace-affine routing for tail sampling, explicit security and capacity caveats, and verification steps.",
|
|
19
|
+
"assertions": [
|
|
20
|
+
"The response assigns node logs and host metrics to node-local agents",
|
|
21
|
+
"The response assigns centralized redaction, credentials, and backend export to gateways",
|
|
22
|
+
"The response states that every span for a trace must reach the same tail-sampling instance",
|
|
23
|
+
"The response labels capacity recommendations as unresolved because telemetry volume is not provided",
|
|
24
|
+
"The response includes security, resiliency, and rollout verification checks"
|
|
25
|
+
]
|
|
26
|
+
},
|
|
27
|
+
{
|
|
28
|
+
"id": 3,
|
|
29
|
+
"prompt": "Review this Collector configuration:\n\nreceivers:\n otlp:\n protocols:\n grpc:\n endpoint: 0.0.0.0:4317\nprocessors:\n batch:\nexporters:\n otlp:\n endpoint: vendor.example:4317\n headers:\n api-key: hard-coded-example-secret\nservice:\n pipelines:\n traces:\n receivers: [otlp]\n processors: [batch]\n exporters: [otlp]\n\nIdentify operational and security risks, but do not assume whether remote clients require the unrestricted listener.",
|
|
30
|
+
"expected_output": "A configuration review that identifies the unrestricted listener, embedded secret, missing memory protection, queue/retry and TLS questions, while treating topology-dependent changes as conditional.",
|
|
31
|
+
"assertions": [
|
|
32
|
+
"The response identifies the embedded credential as a critical issue and recommends protected runtime secret injection",
|
|
33
|
+
"The response identifies 0.0.0.0 as an exposure requiring topology evidence rather than replacing it unconditionally",
|
|
34
|
+
"The response checks memory limiting, batching, queueing, retries, TLS, and Collector self-observability",
|
|
35
|
+
"Findings are ordered by impact and distinguish facts from conditional recommendations",
|
|
36
|
+
"The response includes official Collector configuration and security links"
|
|
37
|
+
]
|
|
38
|
+
},
|
|
39
|
+
{
|
|
40
|
+
"id": 4,
|
|
41
|
+
"prompt": "During a backend outage our Collector exporter queue reaches capacity, enqueue failures increase, and queued data disappears when gateway pods restart. Give us a diagnosis and a resilient remediation sequence.",
|
|
42
|
+
"expected_output": "A diagnosis grounded in queue saturation and in-memory loss, followed by bounded retry/queue tuning, persistent storage evaluation, monitoring, capacity checks, and an optional external queue only when stronger durability justifies it.",
|
|
43
|
+
"assertions": [
|
|
44
|
+
"The response identifies queue saturation and pod restart with in-memory buffering as observed loss mechanisms",
|
|
45
|
+
"The response recommends measuring arrival rate, outage duration, queue capacity, memory, disk, and backend recovery before sizing",
|
|
46
|
+
"The response discusses persistent sending queues and their disk and permission trade-offs",
|
|
47
|
+
"The response treats an external message queue as an optional higher-complexity durability layer",
|
|
48
|
+
"The response defines post-change verification using queue, failure, restart, and export evidence"
|
|
49
|
+
]
|
|
50
|
+
},
|
|
51
|
+
{
|
|
52
|
+
"id": 5,
|
|
53
|
+
"prompt": "We want zero-code telemetry for mixed workloads: Kubernetes services, Linux VMs, and AWS Lambda. Produce a decision framework that covers the Operator, OBI, Collector agents, and Lambda layers. Current versions and runtime details are not supplied.",
|
|
54
|
+
"expected_output": "A platform-specific decision framework that compares supported zero-code approaches, flags missing compatibility evidence, includes least-privilege and rollout checks, and avoids unsupported version claims or language code.",
|
|
55
|
+
"assertions": [
|
|
56
|
+
"The response maps the Operator to Kubernetes, OBI or Collector agents to compatible Linux environments, and published layers to supported Lambda runtimes",
|
|
57
|
+
"The response states that runtime, kernel, architecture, region, artifact, and current-version compatibility must be verified",
|
|
58
|
+
"The response includes least-privilege, network, secret, performance, and rollback checks",
|
|
59
|
+
"The response identifies that zero-code methods cannot guarantee custom business telemetry",
|
|
60
|
+
"The response contains no manual instrumentation code"
|
|
61
|
+
]
|
|
62
|
+
}
|
|
63
|
+
]
|
|
64
|
+
}
|
|
@@ -0,0 +1,138 @@
|
|
|
1
|
+
# Collector Architecture and Configuration
|
|
2
|
+
|
|
3
|
+
Use this reference to choose a Collector topology or review declarative Collector configuration. Component availability and configuration keys vary by distribution and version; verify them in the current component documentation.
|
|
4
|
+
|
|
5
|
+
## Pipeline model
|
|
6
|
+
|
|
7
|
+
A Collector pipeline moves one signal type through:
|
|
8
|
+
|
|
9
|
+
1. Receivers that accept or obtain telemetry.
|
|
10
|
+
2. Optional processors that transform, enrich, batch, sample, filter, or drop telemetry.
|
|
11
|
+
3. Exporters that send telemetry to a destination.
|
|
12
|
+
|
|
13
|
+
Connectors join two pipelines while acting as an exporter for one and a receiver for another. Extensions add operational capabilities such as health checks, storage, authentication, or service discovery and are enabled separately from telemetry pipelines.
|
|
14
|
+
|
|
15
|
+
A minimal declarative shape is:
|
|
16
|
+
|
|
17
|
+
```yaml
|
|
18
|
+
receivers:
|
|
19
|
+
otlp:
|
|
20
|
+
protocols:
|
|
21
|
+
grpc:
|
|
22
|
+
|
|
23
|
+
processors:
|
|
24
|
+
memory_limiter:
|
|
25
|
+
check_interval: 1s
|
|
26
|
+
limit_mib: 512
|
|
27
|
+
batch:
|
|
28
|
+
|
|
29
|
+
exporters:
|
|
30
|
+
otlp:
|
|
31
|
+
endpoint: telemetry-backend.example:4317
|
|
32
|
+
|
|
33
|
+
service:
|
|
34
|
+
pipelines:
|
|
35
|
+
traces:
|
|
36
|
+
receivers: [otlp]
|
|
37
|
+
processors: [memory_limiter, batch]
|
|
38
|
+
exporters: [otlp]
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
This is a structural example, not a production-ready universal default. Authentication, TLS, listener addresses, queueing, resource limits, and component support must be decided for the target environment.
|
|
42
|
+
|
|
43
|
+
## Configuration mechanics
|
|
44
|
+
|
|
45
|
+
- Define components at the top level, then activate them by referencing them from a pipeline or the service extension list.
|
|
46
|
+
- Use `type/name` identifiers when more than one instance of a component type is needed.
|
|
47
|
+
- A defined component that is not referenced by `service` is not enabled.
|
|
48
|
+
- Signal types in a pipeline must be supported by every referenced component.
|
|
49
|
+
- Processor order is behaviorally significant.
|
|
50
|
+
- Multiple configuration sources can be merged, but the merged result must still be complete.
|
|
51
|
+
- Use environment-variable substitution for deployment-supplied values; never embed secrets in the Skill, repository examples, or review output.
|
|
52
|
+
|
|
53
|
+
Validate before deployment:
|
|
54
|
+
|
|
55
|
+
```shell
|
|
56
|
+
otelcol validate --config=collector.yaml
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
Confirm the executable and subcommand for the selected Collector distribution rather than assuming every distribution uses the same binary name.
|
|
60
|
+
|
|
61
|
+
## Reuse and fan-out gotcha
|
|
62
|
+
|
|
63
|
+
A receiver can feed multiple pipelines and an exporter can receive from multiple pipelines. Receiver fan-out is synchronous: a blocking processor in one attached pipeline can block the other pipelines and the receiver. Do not reuse a receiver across pipelines without evaluating this failure coupling.
|
|
64
|
+
|
|
65
|
+
Processors referenced by multiple pipelines use the same configuration but each pipeline gets its own processor instance and state.
|
|
66
|
+
|
|
67
|
+
## Deployment patterns
|
|
68
|
+
|
|
69
|
+
### No Collector
|
|
70
|
+
|
|
71
|
+
Direct export has the fewest moving parts but couples telemetry delivery and backend changes to the instrumented workload. Use it for simple development or constrained cases only after accepting the limited processing, exporter, and operational flexibility.
|
|
72
|
+
|
|
73
|
+
### Agent
|
|
74
|
+
|
|
75
|
+
An agent runs close to workloads, commonly as a host process, sidecar, or Kubernetes DaemonSet.
|
|
76
|
+
|
|
77
|
+
Choose an agent when you need:
|
|
78
|
+
|
|
79
|
+
- Host-local metrics or logs.
|
|
80
|
+
- Local enrichment with host or Kubernetes resource attributes.
|
|
81
|
+
- A stable local telemetry destination.
|
|
82
|
+
- Reduced application responsibility for credentials and backend export.
|
|
83
|
+
|
|
84
|
+
Agents multiply configuration and resource overhead across hosts or workloads. Keep their processing focused.
|
|
85
|
+
|
|
86
|
+
### Gateway
|
|
87
|
+
|
|
88
|
+
A gateway is a standalone Collector service shared by multiple telemetry sources.
|
|
89
|
+
|
|
90
|
+
Choose a gateway when you need:
|
|
91
|
+
|
|
92
|
+
- Centralized credentials, filtering, transformation, or policy.
|
|
93
|
+
- Central processing that must observe traffic from multiple sources.
|
|
94
|
+
- Independent scaling of telemetry processing.
|
|
95
|
+
- A controlled egress point to external backends.
|
|
96
|
+
|
|
97
|
+
Gateways add a network hop and a shared failure domain. Load balance stateless gateways and monitor them as production services.
|
|
98
|
+
|
|
99
|
+
### Agent to gateway
|
|
100
|
+
|
|
101
|
+
Combine agents and gateways when both local collection and centralized processing are required. Typical separation:
|
|
102
|
+
|
|
103
|
+
- Agents: local reception, host collection, resource detection, memory protection, and efficient forwarding.
|
|
104
|
+
- Gateways: centralized filtering, tail sampling, batching, credentials, and resilient export.
|
|
105
|
+
|
|
106
|
+
This pattern is justified by identified needs such as network isolation, host telemetry, central policy, trace-aware sampling, or scale. It is not the default for small deployments.
|
|
107
|
+
|
|
108
|
+
## Stateful routing rules
|
|
109
|
+
|
|
110
|
+
- Tail sampling requires all spans for a trace to reach the same decision-making instance.
|
|
111
|
+
- Cumulative-to-delta conversion and some aggregation use cases also require data-aware routing.
|
|
112
|
+
- Prometheus scraping requires target allocation or sharding to avoid duplicate collection.
|
|
113
|
+
- Metric streams must preserve the single-writer principle and globally unique identities.
|
|
114
|
+
- Standard layer-4 load balancing can be ineffective for persistent gRPC connections; verify that the load-balancing method distributes the actual connection pattern.
|
|
115
|
+
|
|
116
|
+
## Configuration review checklist
|
|
117
|
+
|
|
118
|
+
1. Identify each enabled pipeline and its signal type.
|
|
119
|
+
2. Verify every referenced component exists in the selected distribution.
|
|
120
|
+
3. Trace data from each receiver through processors to every exporter.
|
|
121
|
+
4. Inspect processor order and intentional data drops.
|
|
122
|
+
5. Check listener scope, authentication, TLS, and secret injection.
|
|
123
|
+
6. Check memory limiting, batching, queueing, retry, and persistent storage requirements.
|
|
124
|
+
7. Check stateful processors and their routing assumptions.
|
|
125
|
+
8. Check resource identity and duplicate-writer risk.
|
|
126
|
+
9. Check Collector internal telemetry, health checks, and alerting.
|
|
127
|
+
10. Validate the merged configuration with the target executable.
|
|
128
|
+
|
|
129
|
+
## Official sources
|
|
130
|
+
|
|
131
|
+
- [Collector architecture](https://opentelemetry.io/docs/collector/architecture/)
|
|
132
|
+
- [Collector configuration](https://opentelemetry.io/docs/collector/configuration/)
|
|
133
|
+
- [Collector components](https://opentelemetry.io/docs/collector/components/)
|
|
134
|
+
- [Choose a deployment pattern](https://opentelemetry.io/docs/collector/deploy/choose/)
|
|
135
|
+
- [Agent deployment pattern](https://opentelemetry.io/docs/collector/deploy/agent/)
|
|
136
|
+
- [Gateway deployment pattern](https://opentelemetry.io/docs/collector/deploy/gateway/)
|
|
137
|
+
- [Agent-to-gateway deployment pattern](https://opentelemetry.io/docs/collector/deploy/other/agent-to-gateway/)
|
|
138
|
+
- [No Collector](https://opentelemetry.io/docs/collector/deploy/other/no-collector/)
|