@mastra/mcp-docs-server 1.1.45 → 1.1.46-alpha.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.docs/docs/agents/code-mode.md +2 -0
- package/.docs/docs/agents/signal-providers.md +222 -0
- package/.docs/docs/agents/signals.md +64 -190
- package/.docs/docs/getting-started/build-with-ai.md +2 -2
- package/.docs/docs/mastra-platform/configuration.md +1 -1
- package/.docs/docs/mastra-platform/overview.md +1 -1
- package/.docs/docs/observability/config.md +139 -0
- package/.docs/docs/observability/{tracing → integrations}/bridges/datadog.md +7 -7
- package/.docs/docs/observability/{tracing → integrations}/bridges/otel.md +3 -3
- package/.docs/docs/observability/{tracing → integrations}/exporters/datadog.md +1 -1
- package/.docs/docs/observability/{tracing → integrations}/exporters/mastra-platform.md +1 -1
- package/.docs/docs/observability/{tracing → integrations}/exporters/mastra-storage.md +1 -1
- package/.docs/docs/observability/{tracing → integrations}/exporters/otel.md +4 -4
- package/.docs/docs/observability/{tracing → integrations}/exporters/posthog.md +16 -0
- package/.docs/docs/observability/integrations/overview.md +45 -0
- package/.docs/docs/observability/metrics/overview.md +1 -1
- package/.docs/docs/observability/metrics/querying.md +18 -0
- package/.docs/docs/observability/overview.md +8 -4
- package/.docs/docs/observability/storage.md +79 -0
- package/.docs/docs/observability/tracing/overview.md +98 -390
- package/.docs/guides/guide/signal-provider.md +217 -0
- package/.docs/guides/migrations/upgrade-to-v1/tracing.md +11 -11
- package/.docs/reference/agents/agent.md +150 -2
- package/.docs/reference/client-js/agents.md +4 -0
- package/.docs/reference/core/mastra-class.md +28 -0
- package/.docs/reference/harness/harness-class.md +1 -1
- package/.docs/reference/index.md +3 -0
- package/.docs/reference/observability/tracing/bridges/datadog.md +2 -2
- package/.docs/reference/observability/tracing/bridges/otel.md +2 -2
- package/.docs/reference/observability/tracing/configuration.md +1 -1
- package/.docs/reference/observability/tracing/exporters/arize.md +1 -1
- package/.docs/reference/observability/tracing/exporters/arthur.md +1 -1
- package/.docs/reference/observability/tracing/exporters/cloud-exporter.md +1 -1
- package/.docs/reference/observability/tracing/exporters/console-exporter.md +1 -1
- package/.docs/reference/observability/tracing/exporters/default-exporter.md +1 -1
- package/.docs/reference/observability/tracing/exporters/mastra-platform-exporter.md +1 -1
- package/.docs/reference/observability/tracing/exporters/mastra-storage-exporter.md +1 -1
- package/.docs/reference/observability/tracing/exporters/otel.md +2 -2
- package/.docs/reference/observability/tracing/span-filtering.md +1 -1
- package/.docs/reference/pubsub/base.md +24 -0
- package/.docs/reference/pubsub/caching-pubsub.md +6 -0
- package/.docs/reference/pubsub/event-emitter.md +36 -1
- package/.docs/reference/signals/create-notification-inbox-tool.md +67 -0
- package/.docs/reference/signals/signal-provider.md +406 -0
- package/.docs/reference/signals/webhook-signal-provider.md +137 -0
- package/.docs/reference/storage/clickhouse.md +3 -3
- package/.docs/reference/storage/composite.md +1 -1
- package/.docs/reference/storage/duckdb.md +1 -1
- package/.docs/reference/storage/libsql.md +1 -1
- package/.docs/reference/storage/postgresql.md +1 -1
- package/CHANGELOG.md +15 -0
- package/package.json +7 -7
- /package/.docs/docs/observability/{tracing → integrations}/exporters/arize.md +0 -0
- /package/.docs/docs/observability/{tracing → integrations}/exporters/arthur.md +0 -0
- /package/.docs/docs/observability/{tracing → integrations}/exporters/braintrust.md +0 -0
- /package/.docs/docs/observability/{tracing → integrations}/exporters/laminar.md +0 -0
- /package/.docs/docs/observability/{tracing → integrations}/exporters/langfuse.md +0 -0
- /package/.docs/docs/observability/{tracing → integrations}/exporters/langsmith.md +0 -0
- /package/.docs/docs/observability/{tracing → integrations}/exporters/sentry.md +0 -0
- /package/.docs/docs/observability/{tracing → integrations}/processors/sensitive-data-filter.md +0 -0
|
@@ -1,166 +1,29 @@
|
|
|
1
1
|
# Tracing
|
|
2
2
|
|
|
3
|
-
Tracing
|
|
3
|
+
Tracing is the observability signal that records how a request moves through agents, workflows, tools, and model calls. Mastra represents each operation as a span and groups related spans into a trace so you can inspect the full execution path.
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
This page focuses on trace-specific concepts: span hierarchy, sampling, metadata, filtering, trace IDs, and third-party trace context.
|
|
6
6
|
|
|
7
|
-
##
|
|
7
|
+
## When to use tracing
|
|
8
8
|
|
|
9
|
-
|
|
9
|
+
- Debug unexpected agent or workflow behavior by inspecting the full execution path.
|
|
10
|
+
- Follow model calls, tool calls, and workflow steps inside a single request.
|
|
11
|
+
- Attach trace-specific metadata and tags for filtering and investigation.
|
|
12
|
+
- Connect Mastra traces to a third-party tracing system.
|
|
10
13
|
|
|
11
|
-
|
|
12
|
-
- **Set sampling strategies** → control which traces are collected
|
|
13
|
-
- **Run agents and workflows** → Mastra auto-instruments them with Tracing
|
|
14
|
+
## Get started
|
|
14
15
|
|
|
15
|
-
|
|
16
|
+
To get started with tracing, configure observability in your Mastra instance and run an agent or workflow. You can configure behavior through the following features:
|
|
16
17
|
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
import { Mastra } from '@mastra/core'
|
|
21
|
-
import { LibSQLStore } from '@mastra/libsql'
|
|
22
|
-
import {
|
|
23
|
-
Observability,
|
|
24
|
-
MastraStorageExporter,
|
|
25
|
-
MastraPlatformExporter,
|
|
26
|
-
SensitiveDataFilter,
|
|
27
|
-
} from '@mastra/observability'
|
|
28
|
-
|
|
29
|
-
export const mastra = new Mastra({
|
|
30
|
-
observability: new Observability({
|
|
31
|
-
configs: {
|
|
32
|
-
default: {
|
|
33
|
-
serviceName: 'mastra',
|
|
34
|
-
exporters: [
|
|
35
|
-
new MastraStorageExporter(), // Persists observability events to Mastra Storage
|
|
36
|
-
new MastraPlatformExporter(), // Sends observability events to Mastra platform (if MASTRA_PLATFORM_ACCESS_TOKEN is set)
|
|
37
|
-
],
|
|
38
|
-
spanOutputProcessors: [
|
|
39
|
-
new SensitiveDataFilter(), // Redacts sensitive data like passwords, tokens, keys
|
|
40
|
-
],
|
|
41
|
-
},
|
|
42
|
-
},
|
|
43
|
-
}),
|
|
44
|
-
storage: new LibSQLStore({
|
|
45
|
-
id: 'mastra-storage',
|
|
46
|
-
url: 'file:./mastra.db', // Storage is required for tracing
|
|
47
|
-
}),
|
|
48
|
-
})
|
|
49
|
-
```
|
|
50
|
-
|
|
51
|
-
This configuration includes:
|
|
52
|
-
|
|
53
|
-
- **Service Name**: `"mastra"` - identifies your service in traces
|
|
54
|
-
|
|
55
|
-
- **Sampling**: `"always"` by default (100% of traces)
|
|
56
|
-
|
|
57
|
-
- **Exporters**:
|
|
58
|
-
|
|
59
|
-
- `MastraStorageExporter` - Persists observability events to Mastra Storage
|
|
60
|
-
- `MastraPlatformExporter` - Sends observability events to Mastra platform (requires `MASTRA_PLATFORM_ACCESS_TOKEN`)
|
|
61
|
-
|
|
62
|
-
- **Span Output Processors**: `SensitiveDataFilter` - Redacts sensitive fields
|
|
63
|
-
|
|
64
|
-
## Exporters
|
|
65
|
-
|
|
66
|
-
Exporters determine where your trace data is sent and how it's stored. They integrate with your existing observability stack, support data residency requirements, and can be optimized for cost and performance. You can use multiple exporters simultaneously to send the same trace data to different destinations — for example, storing detailed traces locally for debugging while sending sampled data to a cloud provider for production monitoring.
|
|
67
|
-
|
|
68
|
-
### Internal Exporters
|
|
69
|
-
|
|
70
|
-
Mastra provides two built-in exporters:
|
|
71
|
-
|
|
72
|
-
- **[Mastra Storage](https://mastra.ai/docs/observability/tracing/exporters/mastra-storage)**: Persists observability events to Mastra Storage for viewing in Studio
|
|
73
|
-
- **[Mastra platform](https://mastra.ai/docs/observability/tracing/exporters/mastra-platform)**: Sends observability events to Mastra platform for production monitoring and collaboration
|
|
74
|
-
|
|
75
|
-
### External Exporters
|
|
76
|
-
|
|
77
|
-
In addition to the internal exporters, Mastra supports integration with popular observability platforms. These exporters allow you to leverage your existing monitoring infrastructure and take advantage of platform-specific features like alerting, dashboards, and correlation with other application metrics.
|
|
78
|
-
|
|
79
|
-
- **[Arize](https://mastra.ai/docs/observability/tracing/exporters/arize)** - Exports traces to Arize Phoenix or Arize AX using OpenInference semantic conventions
|
|
80
|
-
- **[Braintrust](https://mastra.ai/docs/observability/tracing/exporters/braintrust)** - Exports traces to Braintrust's eval and observability platform
|
|
81
|
-
- **[Datadog](https://mastra.ai/docs/observability/tracing/exporters/datadog)** - Sends traces to Datadog APM via OTLP for full-stack observability with AI tracing
|
|
82
|
-
- **[Laminar](https://mastra.ai/docs/observability/tracing/exporters/laminar)** - Sends traces to Laminar via OTLP/HTTP (protobuf) with Laminar-native span attributes + scorer support
|
|
83
|
-
- **[Langfuse](https://mastra.ai/docs/observability/tracing/exporters/langfuse)** - Sends traces to the Langfuse open-source LLM engineering platform
|
|
84
|
-
- **[LangSmith](https://mastra.ai/docs/observability/tracing/exporters/langsmith)** - Pushes traces into LangSmith's observability and evaluation toolkit
|
|
85
|
-
- **[PostHog](https://mastra.ai/docs/observability/tracing/exporters/posthog)** - Sends traces to PostHog for AI analytics and product insights
|
|
86
|
-
- **[Sentry](https://mastra.ai/docs/observability/tracing/exporters/sentry)** - Sends traces to Sentry for AI tracing and monitoring using OpenTelemetry semantic conventions
|
|
87
|
-
- **[OpenTelemetry](https://mastra.ai/docs/observability/tracing/exporters/otel)** - Deliver traces to any OpenTelemetry-compatible observability system
|
|
88
|
-
- Supports: Dash0, MLflow, New Relic, SigNoz, Traceloop, Zipkin, and others!
|
|
89
|
-
|
|
90
|
-
## Bridges
|
|
91
|
-
|
|
92
|
-
Bridges provide bidirectional integration with external tracing systems. Unlike exporters that send trace data to external platforms, bridges create native spans in external systems and inherit context from them. This enables Mastra operations to participate in existing distributed traces.
|
|
93
|
-
|
|
94
|
-
- **[OpenTelemetry Bridge](https://mastra.ai/docs/observability/tracing/bridges/otel)** - Integrate with existing OpenTelemetry infrastructure
|
|
95
|
-
|
|
96
|
-
### Bridges vs Exporters
|
|
97
|
-
|
|
98
|
-
| Feature | Bridges | Exporters |
|
|
99
|
-
| ---------------------------------------- | ---------------------------- | ------------------------- |
|
|
100
|
-
| Creates native spans in external systems | Yes | No |
|
|
101
|
-
| Inherits context from external systems | Yes | No |
|
|
102
|
-
| Sends data to backends | Via external SDK | Directly |
|
|
103
|
-
| Use case | Existing distributed tracing | Standalone Mastra tracing |
|
|
104
|
-
|
|
105
|
-
You can use both together — a bridge for context propagation and exporters to send traces to additional destinations.
|
|
18
|
+
- [Configuration](https://mastra.ai/docs/observability/config): Base observability config, multiple configs, and serverless flushing
|
|
19
|
+
- [Storage](https://mastra.ai/docs/observability/storage): Storage routing for traces, logs, and metrics
|
|
20
|
+
- [Integrations overview](https://mastra.ai/docs/observability/integrations/overview): Exporters, bridges, and processors
|
|
106
21
|
|
|
107
22
|
## Sampling strategies
|
|
108
23
|
|
|
109
24
|
Sampling allows you to control which traces are collected, helping you balance between observability needs and resource costs. In production environments with high traffic, collecting every trace can be expensive and unnecessary. Sampling strategies let you capture a representative subset of traces while ensuring you don't miss critical information about errors or important operations.
|
|
110
25
|
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
### Always Sample
|
|
114
|
-
|
|
115
|
-
Collects 100% of traces. Best for development, debugging, or low-traffic scenarios where you need complete visibility.
|
|
116
|
-
|
|
117
|
-
```ts
|
|
118
|
-
sampling: {
|
|
119
|
-
type: 'always'
|
|
120
|
-
}
|
|
121
|
-
```
|
|
122
|
-
|
|
123
|
-
### Never Sample
|
|
124
|
-
|
|
125
|
-
Disables tracing entirely. Useful for specific environments where tracing adds no value or when you need to temporarily disable tracing without removing configuration.
|
|
126
|
-
|
|
127
|
-
```ts
|
|
128
|
-
sampling: {
|
|
129
|
-
type: 'never'
|
|
130
|
-
}
|
|
131
|
-
```
|
|
132
|
-
|
|
133
|
-
### Ratio-Based Sampling
|
|
134
|
-
|
|
135
|
-
Randomly samples a percentage of traces. Ideal for production environments where you want statistical insights without the cost of full tracing. The probability value ranges from 0 (no traces) to 1 (all traces).
|
|
136
|
-
|
|
137
|
-
```ts
|
|
138
|
-
sampling: {
|
|
139
|
-
type: 'ratio',
|
|
140
|
-
probability: 0.1 // Sample 10% of traces
|
|
141
|
-
}
|
|
142
|
-
```
|
|
143
|
-
|
|
144
|
-
### Custom Sampling
|
|
145
|
-
|
|
146
|
-
Implements your own sampling logic based on request context, metadata, or business rules. Perfect for complex scenarios like sampling based on user tier, request type, or error conditions.
|
|
147
|
-
|
|
148
|
-
```ts
|
|
149
|
-
sampling: {
|
|
150
|
-
type: 'custom',
|
|
151
|
-
sampler: (options) => {
|
|
152
|
-
// Sample premium users at higher rate
|
|
153
|
-
if (options?.metadata?.userTier === 'premium') {
|
|
154
|
-
return Math.random() < 0.5; // 50% sampling
|
|
155
|
-
}
|
|
156
|
-
|
|
157
|
-
// Default 1% sampling for others
|
|
158
|
-
return Math.random() < 0.01;
|
|
159
|
-
}
|
|
160
|
-
}
|
|
161
|
-
```
|
|
162
|
-
|
|
163
|
-
### Complete Example
|
|
26
|
+
You can configure sampling at the observability config level:
|
|
164
27
|
|
|
165
28
|
```ts
|
|
166
29
|
export const mastra = new Mastra({
|
|
@@ -180,139 +43,49 @@ export const mastra = new Mastra({
|
|
|
180
43
|
})
|
|
181
44
|
```
|
|
182
45
|
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
Complex applications often require different tracing configurations for different scenarios. You might want detailed traces with full sampling during development, sampled traces sent to external providers in production, and specialized configurations for specific features or customer segments. The `configSelector` function enables dynamic configuration selection at runtime, allowing you to route traces based on request context, environment variables, feature flags, or any custom logic.
|
|
186
|
-
|
|
187
|
-
This approach is particularly valuable when:
|
|
188
|
-
|
|
189
|
-
- Running A/B tests with different observability requirements
|
|
190
|
-
- Providing enhanced debugging for specific customers or support cases
|
|
191
|
-
- Gradually rolling out new tracing providers without affecting existing monitoring
|
|
192
|
-
- Optimizing costs by using different sampling rates for different request types
|
|
193
|
-
- Maintaining separate trace streams for compliance or data residency requirements
|
|
194
|
-
|
|
195
|
-
> **Info:** Note that only a single config can be used for a specific execution. But a single config can send data to multiple exporters simultaneously.
|
|
196
|
-
|
|
197
|
-
### Dynamic Configuration Selection
|
|
198
|
-
|
|
199
|
-
Use `configSelector` to choose the appropriate tracing configuration based on request context:
|
|
200
|
-
|
|
201
|
-
```ts
|
|
202
|
-
export const mastra = new Mastra({
|
|
203
|
-
observability: new Observability({
|
|
204
|
-
configs: {
|
|
205
|
-
langfuse: {
|
|
206
|
-
serviceName: 'langfuse-service',
|
|
207
|
-
exporters: [langfuseExporter],
|
|
208
|
-
},
|
|
209
|
-
braintrust: {
|
|
210
|
-
serviceName: 'braintrust-service',
|
|
211
|
-
exporters: [braintrustExporter],
|
|
212
|
-
},
|
|
213
|
-
debug: {
|
|
214
|
-
serviceName: 'debug-service',
|
|
215
|
-
sampling: { type: 'always' },
|
|
216
|
-
exporters: [new MastraStorageExporter()],
|
|
217
|
-
},
|
|
218
|
-
},
|
|
219
|
-
configSelector: (context, availableTracers) => {
|
|
220
|
-
// Use debug config for support requests
|
|
221
|
-
if (context.requestContext?.get('supportMode')) {
|
|
222
|
-
return 'debug'
|
|
223
|
-
}
|
|
224
|
-
|
|
225
|
-
// Route specific customers to different providers
|
|
226
|
-
const customerId = context.requestContext?.get('customerId')
|
|
227
|
-
if (customerId && premiumCustomers.includes(customerId)) {
|
|
228
|
-
return 'braintrust'
|
|
229
|
-
}
|
|
230
|
-
|
|
231
|
-
// Route specific requests to langfuse
|
|
232
|
-
if (context.requestContext?.get('useExternalTracing')) {
|
|
233
|
-
return 'langfuse'
|
|
234
|
-
}
|
|
235
|
-
|
|
236
|
-
throw new Error('no config found')
|
|
237
|
-
},
|
|
238
|
-
}),
|
|
239
|
-
})
|
|
240
|
-
```
|
|
241
|
-
|
|
242
|
-
### Environment-Based Configuration
|
|
46
|
+
The `sampling` option allows you to control which traces are collected, helping you balance between observability needs and resource costs. Mastra supports four sampling strategies:
|
|
243
47
|
|
|
244
|
-
|
|
48
|
+
1. **Always Sample**: Collects 100% of traces. Best for development, debugging, or low-traffic scenarios where you need complete visibility.
|
|
245
49
|
|
|
246
|
-
```ts
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
|
|
251
|
-
serviceName: 'my-service-dev',
|
|
252
|
-
sampling: { type: 'always' },
|
|
253
|
-
exporters: [new MastraStorageExporter()],
|
|
254
|
-
},
|
|
255
|
-
staging: {
|
|
256
|
-
serviceName: 'my-service-staging',
|
|
257
|
-
sampling: { type: 'ratio', probability: 0.5 },
|
|
258
|
-
exporters: [langfuseExporter],
|
|
259
|
-
},
|
|
260
|
-
production: {
|
|
261
|
-
serviceName: 'my-service-prod',
|
|
262
|
-
sampling: { type: 'ratio', probability: 0.01 },
|
|
263
|
-
exporters: [mastraObserveExporter, langfuseExporter],
|
|
264
|
-
},
|
|
265
|
-
},
|
|
266
|
-
configSelector: (context, availableTracers) => {
|
|
267
|
-
const env = process.env.NODE_ENV || 'development'
|
|
268
|
-
return env
|
|
269
|
-
},
|
|
270
|
-
}),
|
|
271
|
-
})
|
|
272
|
-
```
|
|
50
|
+
```ts
|
|
51
|
+
sampling: {
|
|
52
|
+
type: 'always'
|
|
53
|
+
}
|
|
54
|
+
```
|
|
273
55
|
|
|
274
|
-
|
|
56
|
+
2. **Never Sample**: Disables tracing entirely. Useful for specific environments where tracing adds no value or when you need to temporarily disable tracing without removing configuration.
|
|
275
57
|
|
|
276
|
-
|
|
58
|
+
```ts
|
|
59
|
+
sampling: {
|
|
60
|
+
type: 'never'
|
|
61
|
+
}
|
|
62
|
+
```
|
|
277
63
|
|
|
278
|
-
|
|
64
|
+
3. **Ratio-Based Sampling**: Randomly samples a percentage of traces. Ideal for production environments where you want statistical insights without the cost of full tracing. The probability value ranges from 0 (no traces) to 1 (all traces).
|
|
279
65
|
|
|
280
|
-
```ts
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
|
|
285
|
-
|
|
286
|
-
} from '@mastra/observability'
|
|
287
|
-
import { ArizeExporter } from '@mastra/arize'
|
|
66
|
+
```ts
|
|
67
|
+
sampling: {
|
|
68
|
+
type: 'ratio',
|
|
69
|
+
probability: 0.1 // Sample 10% of traces
|
|
70
|
+
}
|
|
71
|
+
```
|
|
288
72
|
|
|
289
|
-
|
|
290
|
-
observability: new Observability({
|
|
291
|
-
configs: {
|
|
292
|
-
production: {
|
|
293
|
-
serviceName: 'my-service',
|
|
294
|
-
exporters: [
|
|
295
|
-
new ArizeExporter({
|
|
296
|
-
endpoint: process.env.PHOENIX_COLLECTOR_ENDPOINT,
|
|
297
|
-
apiKey: process.env.PHOENIX_API_KEY,
|
|
298
|
-
}),
|
|
299
|
-
new MastraStorageExporter(), // Keep Studio access
|
|
300
|
-
new MastraPlatformExporter(), // Keep Cloud access
|
|
301
|
-
],
|
|
302
|
-
spanOutputProcessors: [new SensitiveDataFilter()],
|
|
303
|
-
},
|
|
304
|
-
},
|
|
305
|
-
}),
|
|
306
|
-
})
|
|
307
|
-
```
|
|
308
|
-
|
|
309
|
-
This configuration sends traces to all three destinations simultaneously:
|
|
73
|
+
4. **Custom Sampling**: Implements your own sampling logic based on request context, metadata, or business rules. Perfect for complex scenarios like sampling based on user tier, request type, or error conditions.
|
|
310
74
|
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
|
|
75
|
+
```ts
|
|
76
|
+
sampling: {
|
|
77
|
+
type: 'custom',
|
|
78
|
+
sampler: (options) => {
|
|
79
|
+
// Sample premium users at higher rate
|
|
80
|
+
if (options?.metadata?.userTier === 'premium') {
|
|
81
|
+
return Math.random() < 0.5; // 50% sampling
|
|
82
|
+
}
|
|
314
83
|
|
|
315
|
-
|
|
84
|
+
// Default 1% sampling for others
|
|
85
|
+
return Math.random() < 0.01;
|
|
86
|
+
}
|
|
87
|
+
}
|
|
88
|
+
```
|
|
316
89
|
|
|
317
90
|
## Adding custom metadata
|
|
318
91
|
|
|
@@ -362,13 +135,13 @@ export const mastra = new Mastra({
|
|
|
362
135
|
|
|
363
136
|
If `environment` is not set, Mastra falls back to `process.env.NODE_ENV`. If neither is set, the field is left undefined rather than guessed.
|
|
364
137
|
|
|
365
|
-
Per-call `tracingOptions.metadata.environment` always takes precedence, so individual calls can
|
|
138
|
+
Per-call `tracingOptions.metadata.environment` always takes precedence, so individual calls can override the value when needed.
|
|
366
139
|
|
|
367
|
-
### Automatic
|
|
140
|
+
### Automatic metadata from `RequestContext`
|
|
368
141
|
|
|
369
142
|
Instead of manually adding metadata to each span, you can configure Mastra to automatically extract values from RequestContext and attach them as metadata to all spans in a trace. This is useful for consistently tracking user identifiers, environment information, feature flags, or any request-scoped data across your entire trace.
|
|
370
143
|
|
|
371
|
-
#### Configuration-
|
|
144
|
+
#### Configuration-level extraction
|
|
372
145
|
|
|
373
146
|
Define which RequestContext keys to extract in your tracing configuration. These keys will be automatically included as metadata for all spans created with this configuration:
|
|
374
147
|
|
|
@@ -400,7 +173,7 @@ const result = await agent.generate('Hello', {
|
|
|
400
173
|
})
|
|
401
174
|
```
|
|
402
175
|
|
|
403
|
-
#### Per-
|
|
176
|
+
#### Per-request additions
|
|
404
177
|
|
|
405
178
|
You can add trace-specific keys using `tracingOptions.requestContextKeys`. These are merged with the configuration-level keys:
|
|
406
179
|
|
|
@@ -420,7 +193,7 @@ const result = await agent.generate('Hello', {
|
|
|
420
193
|
// All spans now have: userId, environment, AND experimentId
|
|
421
194
|
```
|
|
422
195
|
|
|
423
|
-
#### Nested
|
|
196
|
+
#### Nested value extraction
|
|
424
197
|
|
|
425
198
|
Use dot notation to extract nested values from RequestContext:
|
|
426
199
|
|
|
@@ -450,7 +223,7 @@ requestContext.set('session', { data: { experimentId: 'exp-999' } })
|
|
|
450
223
|
3. **Child Span Extraction**: Child spans can also extract metadata if you pass `requestContext` when creating them
|
|
451
224
|
4. **Metadata Precedence**: Explicit metadata passed to span options always takes precedence over extracted metadata
|
|
452
225
|
|
|
453
|
-
### Adding
|
|
226
|
+
### Adding tags to traces
|
|
454
227
|
|
|
455
228
|
Tags are string labels that help you categorize and filter traces. Unlike metadata (which contains structured key-value data), tags are plain strings designed for quick filtering and organization.
|
|
456
229
|
|
|
@@ -474,17 +247,17 @@ const result = await run.start({
|
|
|
474
247
|
})
|
|
475
248
|
```
|
|
476
249
|
|
|
477
|
-
#### How
|
|
250
|
+
#### How tags work
|
|
478
251
|
|
|
479
252
|
- **Root span only**: Tags are applied only to the root span of a trace (the agent run or workflow run span)
|
|
480
253
|
|
|
481
254
|
- **Widely supported**: Tags are supported by most exporters for filtering and searching traces:
|
|
482
255
|
|
|
483
|
-
- **Braintrust
|
|
484
|
-
- **Langfuse
|
|
485
|
-
- **ArizeExporter
|
|
486
|
-
- **OtelExporter
|
|
487
|
-
- **OtelBridge
|
|
256
|
+
- **Braintrust**: Native `tags` field
|
|
257
|
+
- **Langfuse**: Native `tags` field on traces
|
|
258
|
+
- **ArizeExporter**: `tag.tags` OpenInference attribute
|
|
259
|
+
- **OtelExporter**: `mastra.tags` span attribute
|
|
260
|
+
- **OtelBridge**: `mastra.tags` span attribute
|
|
488
261
|
|
|
489
262
|
- **Combinable with metadata**: You can use both `tags` and `metadata` in the same `tracingOptions`
|
|
490
263
|
|
|
@@ -497,7 +270,7 @@ const result = await agent.generate([{ role: 'user', content: 'Analyze this' }],
|
|
|
497
270
|
})
|
|
498
271
|
```
|
|
499
272
|
|
|
500
|
-
#### Common
|
|
273
|
+
#### Common tag patterns
|
|
501
274
|
|
|
502
275
|
- **Environment**: `"production"`, `"staging"`, `"development"`
|
|
503
276
|
- **Feature flags**: `"feature-x-enabled"`, `"beta-user"`
|
|
@@ -505,7 +278,7 @@ const result = await agent.generate([{ role: 'user', content: 'Analyze this' }],
|
|
|
505
278
|
- **Priority levels**: `"priority-high"`, `"priority-low"`
|
|
506
279
|
- **Experiments**: `"experiment-v1"`, `"control-group"`, `"treatment-a"`
|
|
507
280
|
|
|
508
|
-
### Hiding
|
|
281
|
+
### Hiding sensitive input/output
|
|
509
282
|
|
|
510
283
|
When processing sensitive data, you may want to prevent input and output values from being logged to your observability platforms. Use `hideInput` and `hideOutput` in `tracingOptions` to exclude this data from all spans in a trace:
|
|
511
284
|
|
|
@@ -536,7 +309,7 @@ const result = await agent.generate([{ role: 'user', content: 'Handle confidenti
|
|
|
536
309
|
#### How it works
|
|
537
310
|
|
|
538
311
|
- **Trace-wide effect**: When set on the root span, these options apply to all child spans in the trace (tool calls, model generations, etc.)
|
|
539
|
-
- **Export-time filtering**: The data
|
|
312
|
+
- **Export-time filtering**: The data remains available internally during execution but is excluded when spans are exported to observability platforms
|
|
540
313
|
- **Combinable with other options**: You can use `hideInput`/`hideOutput` alongside `tags`, `metadata`, and other `tracingOptions`
|
|
541
314
|
|
|
542
315
|
```ts
|
|
@@ -550,9 +323,9 @@ const result = await agent.generate([{ role: 'user', content: 'Sensitive operati
|
|
|
550
323
|
})
|
|
551
324
|
```
|
|
552
325
|
|
|
553
|
-
> **Tip:** For more granular control over sensitive data, consider using the [Sensitive Data Filter](https://mastra.ai/docs/observability/
|
|
326
|
+
> **Tip:** For more granular control over sensitive data, consider using the [Sensitive Data Filter](https://mastra.ai/docs/observability/integrations/processors/sensitive-data-filter) processor, which can redact specific fields (like passwords, tokens, and keys) while preserving the rest of the input/output.
|
|
554
327
|
|
|
555
|
-
#### Child
|
|
328
|
+
#### Child spans and metadata extraction
|
|
556
329
|
|
|
557
330
|
When creating child spans within tools or workflow steps, you can pass the `requestContext` parameter to enable metadata extraction:
|
|
558
331
|
|
|
@@ -634,15 +407,15 @@ Mastra provides two ways to transform span data before it reaches your observabi
|
|
|
634
407
|
|
|
635
408
|
Use **span processors** for synchronous transformations that should apply to all exporters (like redacting sensitive data). Use **custom span formatters** when different exporters need different representations of the same data (like plain text for one platform and structured data for another), or when you need to perform asynchronous operations like fetching data from external APIs.
|
|
636
409
|
|
|
637
|
-
### Span
|
|
410
|
+
### Span processors
|
|
638
411
|
|
|
639
412
|
Span processors transform, filter, or enrich trace data before it's exported. They act as a pipeline between span creation and export, enabling you to modify spans for security, compliance, or debugging purposes. Processors run once and affect all exporters.
|
|
640
413
|
|
|
641
|
-
#### Built-in
|
|
414
|
+
#### Built-in processors
|
|
642
415
|
|
|
643
|
-
- [Sensitive Data Filter](https://mastra.ai/docs/observability/
|
|
416
|
+
- [Sensitive Data Filter](https://mastra.ai/docs/observability/integrations/processors/sensitive-data-filter) redacts sensitive information. It's enabled in the default observability config.
|
|
644
417
|
|
|
645
|
-
#### Creating
|
|
418
|
+
#### Creating custom processors
|
|
646
419
|
|
|
647
420
|
You can create custom span processors by implementing the `SpanOutputProcessor` interface. Here's a basic example that converts all input text in spans to lowercase:
|
|
648
421
|
|
|
@@ -683,6 +456,8 @@ Processors are executed in the order they're defined, allowing you to chain mult
|
|
|
683
456
|
- Normalizing data formats
|
|
684
457
|
- Enriching spans with business context
|
|
685
458
|
|
|
459
|
+
For the broader exporter, bridge, and processor model, see [Integrations overview](https://mastra.ai/docs/observability/integrations/overview).
|
|
460
|
+
|
|
686
461
|
## Span filtering
|
|
687
462
|
|
|
688
463
|
Span filtering lets you reduce noise and per-span costs before data reaches your observability platform. Configure it per observability instance, so different exporters or environments can keep different levels of detail.
|
|
@@ -727,16 +502,16 @@ Filtering happens at export time in this order:
|
|
|
727
502
|
|
|
728
503
|
If `spanFilter` throws, Mastra keeps the span and logs the error to avoid silent data loss. For the full span type list and more examples, see the [Span filtering reference](https://mastra.ai/reference/observability/tracing/span-filtering).
|
|
729
504
|
|
|
730
|
-
### Custom
|
|
505
|
+
### Custom span formatters
|
|
731
506
|
|
|
732
507
|
Custom span formatters transform how spans appear in specific observability platforms. Unlike span processors, formatters are configured per-exporter, allowing different formatting for different destinations. Formatters support both synchronous and asynchronous operations.
|
|
733
508
|
|
|
734
509
|
#### Use cases
|
|
735
510
|
|
|
736
|
-
- **Extract plain text from AI SDK messages
|
|
737
|
-
- **Transform input/output formats
|
|
738
|
-
- **Platform-specific field mapping
|
|
739
|
-
- **Async data enrichment
|
|
511
|
+
- **Extract plain text from AI SDK messages**: Convert structured message arrays to readable text
|
|
512
|
+
- **Transform input/output formats**: Customize how data displays in specific platforms
|
|
513
|
+
- **Platform-specific field mapping**: Add or remove fields based on platform requirements
|
|
514
|
+
- **Async data enrichment**: Fetch additional context from external APIs or databases
|
|
740
515
|
|
|
741
516
|
#### Configuration
|
|
742
517
|
|
|
@@ -779,7 +554,7 @@ export const mastra = new Mastra({
|
|
|
779
554
|
})
|
|
780
555
|
```
|
|
781
556
|
|
|
782
|
-
#### Chaining
|
|
557
|
+
#### Chaining multiple formatters
|
|
783
558
|
|
|
784
559
|
Use `chainFormatters` to combine multiple formatters. Chains support both sync and async formatters:
|
|
785
560
|
|
|
@@ -801,7 +576,7 @@ const exporter = new BraintrustExporter({
|
|
|
801
576
|
})
|
|
802
577
|
```
|
|
803
578
|
|
|
804
|
-
#### Async
|
|
579
|
+
#### Async formatters
|
|
805
580
|
|
|
806
581
|
Custom span formatters support asynchronous operations, enabling use cases like fetching data from external APIs or databases to enrich your spans:
|
|
807
582
|
|
|
@@ -888,7 +663,7 @@ export const mastra = new Mastra({
|
|
|
888
663
|
})
|
|
889
664
|
```
|
|
890
665
|
|
|
891
|
-
### Available
|
|
666
|
+
### Available options
|
|
892
667
|
|
|
893
668
|
| Option | Default | Description |
|
|
894
669
|
| ----------------- | ------- | ---------------------------------------------------------------- |
|
|
@@ -926,7 +701,7 @@ All options are optional — if not specified, they fall back to the defaults sh
|
|
|
926
701
|
|
|
927
702
|
When you execute agents or workflows with tracing enabled, the response includes a `traceId` that you can use to look up the full trace in your observability platform. This is useful for debugging, customer support, or correlating traces with other events in your system.
|
|
928
703
|
|
|
929
|
-
### Agent
|
|
704
|
+
### Agent trace IDs
|
|
930
705
|
|
|
931
706
|
Both `generate` and `stream` methods return the trace ID in their response:
|
|
932
707
|
|
|
@@ -942,7 +717,7 @@ const streamResult = await agent.stream('Tell me a story')
|
|
|
942
717
|
console.log('Trace ID:', streamResult.traceId)
|
|
943
718
|
```
|
|
944
719
|
|
|
945
|
-
### Workflow
|
|
720
|
+
### Workflow trace IDs
|
|
946
721
|
|
|
947
722
|
Workflow executions also return trace IDs:
|
|
948
723
|
|
|
@@ -967,7 +742,7 @@ const finalState = await getWorkflowState()
|
|
|
967
742
|
console.log('Trace ID:', finalState.traceId)
|
|
968
743
|
```
|
|
969
744
|
|
|
970
|
-
### Using
|
|
745
|
+
### Using trace IDs
|
|
971
746
|
|
|
972
747
|
Once you have a trace ID, you can:
|
|
973
748
|
|
|
@@ -982,7 +757,7 @@ The trace ID is only available when tracing is enabled. If tracing is disabled o
|
|
|
982
757
|
|
|
983
758
|
When running Mastra agents or workflows within applications that have existing distributed tracing (OpenTelemetry, Datadog, etc.), you can connect Mastra traces to your parent trace context. This creates a unified view of your entire request flow, making it easier to understand how Mastra operations fit into the broader system.
|
|
984
759
|
|
|
985
|
-
### Passing
|
|
760
|
+
### Passing external trace IDs
|
|
986
761
|
|
|
987
762
|
Use the `tracingOptions` parameter to specify the trace context from your parent system:
|
|
988
763
|
|
|
@@ -1002,7 +777,7 @@ const result = await agent.generate('Analyze this data', {
|
|
|
1002
777
|
// The Mastra trace will now appear as a child in your distributed trace
|
|
1003
778
|
```
|
|
1004
779
|
|
|
1005
|
-
### OpenTelemetry
|
|
780
|
+
### OpenTelemetry integration
|
|
1006
781
|
|
|
1007
782
|
Integration with OpenTelemetry allows Mastra traces to appear seamlessly in your existing observability platform:
|
|
1008
783
|
|
|
@@ -1023,7 +798,7 @@ if (spanContext) {
|
|
|
1023
798
|
}
|
|
1024
799
|
```
|
|
1025
800
|
|
|
1026
|
-
### Workflow
|
|
801
|
+
### Workflow integration
|
|
1027
802
|
|
|
1028
803
|
Workflows support the same pattern for trace propagation:
|
|
1029
804
|
|
|
@@ -1040,7 +815,7 @@ const result = await run.start({
|
|
|
1040
815
|
})
|
|
1041
816
|
```
|
|
1042
817
|
|
|
1043
|
-
### ID
|
|
818
|
+
### ID format requirements
|
|
1044
819
|
|
|
1045
820
|
Mastra validates trace and span IDs to ensure compatibility:
|
|
1046
821
|
|
|
@@ -1054,7 +829,7 @@ Invalid IDs are handled gracefully — Mastra logs an error and continues:
|
|
|
1054
829
|
|
|
1055
830
|
This ensures tracing never crashes your application, even with malformed input.
|
|
1056
831
|
|
|
1057
|
-
### Example: Express
|
|
832
|
+
### Example: Express middleware
|
|
1058
833
|
|
|
1059
834
|
Here's a complete example showing trace propagation in an Express application:
|
|
1060
835
|
|
|
@@ -1084,97 +859,30 @@ app.post('/api/analyze', async (req, res) => {
|
|
|
1084
859
|
|
|
1085
860
|
This creates a single distributed trace that includes both the HTTP request handling and the Mastra agent execution, viewable in your observability platform of choice.
|
|
1086
861
|
|
|
1087
|
-
## Flushing traces in serverless environments
|
|
1088
|
-
|
|
1089
|
-
In serverless environments like Vercel's fluid compute, AWS Lambda, or Cloudflare Workers, runtime instances can be reused across multiple requests. The `flush()` method allows you to ensure all buffered spans are exported before the runtime terminates, without shutting down the exporter (which would prevent future exports).
|
|
1090
|
-
|
|
1091
|
-
> **Storage requirements:** Serverless environments have ephemeral filesystems. Use external storage instead of local file storage (`file:./mastra.db`). See the [Vercel deployment guide](https://mastra.ai/guides/deployment/vercel) for a complete setup example.
|
|
1092
|
-
|
|
1093
|
-
### Using `flush()`
|
|
1094
|
-
|
|
1095
|
-
Call `flush()` on the observability instance to flush all exporters:
|
|
1096
|
-
|
|
1097
|
-
```ts
|
|
1098
|
-
// Get the observability instance from Mastra
|
|
1099
|
-
const observability = mastra.getObservability()
|
|
1100
|
-
|
|
1101
|
-
// Flush all buffered spans to all exporters
|
|
1102
|
-
await observability.flush()
|
|
1103
|
-
```
|
|
1104
|
-
|
|
1105
|
-
### When to Use `flush()`
|
|
1106
|
-
|
|
1107
|
-
Use `flush()` in these scenarios:
|
|
1108
|
-
|
|
1109
|
-
- **End of serverless function execution**: Ensure spans are exported before the runtime is paused or terminated
|
|
1110
|
-
- **Before long-running operations**: Flush accumulated spans before a potentially slow operation
|
|
1111
|
-
- **Periodic flushing**: In long-running processes, periodically flush to ensure timely data availability
|
|
1112
|
-
|
|
1113
|
-
```ts
|
|
1114
|
-
// Example: Vercel serverless function
|
|
1115
|
-
export async function POST(req: Request) {
|
|
1116
|
-
const result = await agent.generate([{ role: 'user', content: await req.text() }])
|
|
1117
|
-
|
|
1118
|
-
// Ensure spans are exported before function completes
|
|
1119
|
-
const observability = mastra.getObservability()
|
|
1120
|
-
await observability.flush()
|
|
1121
|
-
|
|
1122
|
-
return Response.json(result)
|
|
1123
|
-
}
|
|
1124
|
-
```
|
|
1125
|
-
|
|
1126
|
-
### `flush()` vs `shutdown()`
|
|
1127
|
-
|
|
1128
|
-
| Method | Behavior | Use Case |
|
|
1129
|
-
| ------------ | --------------------------------------------- | ------------------------------------------ |
|
|
1130
|
-
| `flush()` | Exports buffered spans, keeps exporter active | Serverless environments, periodic flushing |
|
|
1131
|
-
| `shutdown()` | Exports buffered spans, releases resources | Application shutdown, graceful termination |
|
|
1132
|
-
|
|
1133
|
-
Use `flush()` when you need to ensure data is exported but want to keep the exporter ready for future requests. Use `shutdown()` only when the application is terminating.
|
|
1134
|
-
|
|
1135
862
|
## What gets traced
|
|
1136
863
|
|
|
1137
864
|
Mastra automatically creates spans for:
|
|
1138
865
|
|
|
1139
|
-
### Agent
|
|
866
|
+
### Agent operations
|
|
1140
867
|
|
|
1141
|
-
- **Agent runs
|
|
1142
|
-
- **LLM calls
|
|
1143
|
-
- **Tool executions
|
|
1144
|
-
- **Memory operations
|
|
868
|
+
- **Agent runs**: Complete execution with instructions and tools
|
|
869
|
+
- **LLM calls**: Model interactions with tokens and parameters
|
|
870
|
+
- **Tool executions**: Function calls with inputs and outputs
|
|
871
|
+
- **Memory operations**: Thread and semantic recall
|
|
1145
872
|
|
|
1146
|
-
### Workflow
|
|
873
|
+
### Workflow operations
|
|
1147
874
|
|
|
1148
|
-
- **Workflow runs
|
|
1149
|
-
- **Individual steps
|
|
1150
|
-
- **Control flow
|
|
1151
|
-
- **Wait operations
|
|
875
|
+
- **Workflow runs**: Full execution from start to finish
|
|
876
|
+
- **Individual steps**: Step processing with inputs/outputs
|
|
877
|
+
- **Control flow**: Conditionals, loops, parallel execution
|
|
878
|
+
- **Wait operations**: Delays and event waiting
|
|
1152
879
|
|
|
1153
880
|
## See also
|
|
1154
881
|
|
|
1155
|
-
### Reference
|
|
882
|
+
### Reference documentation
|
|
1156
883
|
|
|
1157
884
|
- [Configuration API](https://mastra.ai/reference/observability/tracing/configuration): ObservabilityConfig details
|
|
1158
885
|
- [Tracing Classes](https://mastra.ai/reference/observability/tracing/instances): Core classes and methods
|
|
1159
886
|
- [Span Interfaces](https://mastra.ai/reference/observability/tracing/spans): Span types and lifecycle
|
|
1160
887
|
- [Type Definitions](https://mastra.ai/reference/observability/tracing/interfaces): Complete interface reference
|
|
1161
|
-
- [Span filtering](https://mastra.ai/reference/observability/tracing/span-filtering): Filtering behavior, span types, and examples
|
|
1162
|
-
|
|
1163
|
-
### Exporters
|
|
1164
|
-
|
|
1165
|
-
- [MastraStorageExporter](https://mastra.ai/reference/observability/tracing/exporters/mastra-storage-exporter): Storage persistence
|
|
1166
|
-
- [MastraPlatformExporter](https://mastra.ai/reference/observability/tracing/exporters/mastra-platform-exporter): Mastra platform integration
|
|
1167
|
-
- [ConsoleExporter](https://mastra.ai/reference/observability/tracing/exporters/console-exporter): Debug output
|
|
1168
|
-
- [Arize](https://mastra.ai/reference/observability/tracing/exporters/arize): Arize Phoenix and Arize AX integration
|
|
1169
|
-
- [Braintrust](https://mastra.ai/reference/observability/tracing/exporters/braintrust): Braintrust integration
|
|
1170
|
-
- [Langfuse](https://mastra.ai/reference/observability/tracing/exporters/langfuse): Langfuse integration
|
|
1171
|
-
- [MLflow](https://mastra.ai/docs/observability/tracing/exporters/otel): MLflow OTLP endpoint setup
|
|
1172
|
-
- [OpenTelemetry](https://mastra.ai/reference/observability/tracing/exporters/otel): OTEL-compatible platforms
|
|
1173
|
-
|
|
1174
|
-
### Bridges
|
|
1175
|
-
|
|
1176
|
-
- [OpenTelemetry Bridge](https://mastra.ai/reference/observability/tracing/bridges/otel): OTEL context integration
|
|
1177
|
-
|
|
1178
|
-
### Processors
|
|
1179
|
-
|
|
1180
|
-
- [Sensitive Data Filter](https://mastra.ai/docs/observability/tracing/processors/sensitive-data-filter): Data redaction
|
|
888
|
+
- [Span filtering](https://mastra.ai/reference/observability/tracing/span-filtering): Filtering behavior, span types, and examples
|