@mastra/mcp-docs-server 1.1.45 → 1.1.46-alpha.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (60) hide show
  1. package/.docs/docs/agents/code-mode.md +2 -0
  2. package/.docs/docs/agents/signal-providers.md +222 -0
  3. package/.docs/docs/agents/signals.md +64 -190
  4. package/.docs/docs/getting-started/build-with-ai.md +2 -2
  5. package/.docs/docs/mastra-platform/configuration.md +1 -1
  6. package/.docs/docs/mastra-platform/overview.md +1 -1
  7. package/.docs/docs/observability/config.md +139 -0
  8. package/.docs/docs/observability/{tracing → integrations}/bridges/datadog.md +7 -7
  9. package/.docs/docs/observability/{tracing → integrations}/bridges/otel.md +3 -3
  10. package/.docs/docs/observability/{tracing → integrations}/exporters/datadog.md +1 -1
  11. package/.docs/docs/observability/{tracing → integrations}/exporters/mastra-platform.md +1 -1
  12. package/.docs/docs/observability/{tracing → integrations}/exporters/mastra-storage.md +1 -1
  13. package/.docs/docs/observability/{tracing → integrations}/exporters/otel.md +4 -4
  14. package/.docs/docs/observability/{tracing → integrations}/exporters/posthog.md +16 -0
  15. package/.docs/docs/observability/integrations/overview.md +45 -0
  16. package/.docs/docs/observability/metrics/overview.md +1 -1
  17. package/.docs/docs/observability/metrics/querying.md +18 -0
  18. package/.docs/docs/observability/overview.md +8 -4
  19. package/.docs/docs/observability/storage.md +79 -0
  20. package/.docs/docs/observability/tracing/overview.md +98 -390
  21. package/.docs/guides/guide/signal-provider.md +217 -0
  22. package/.docs/guides/migrations/upgrade-to-v1/tracing.md +11 -11
  23. package/.docs/reference/agents/agent.md +150 -2
  24. package/.docs/reference/client-js/agents.md +4 -0
  25. package/.docs/reference/core/mastra-class.md +28 -0
  26. package/.docs/reference/harness/harness-class.md +1 -1
  27. package/.docs/reference/index.md +3 -0
  28. package/.docs/reference/observability/tracing/bridges/datadog.md +2 -2
  29. package/.docs/reference/observability/tracing/bridges/otel.md +2 -2
  30. package/.docs/reference/observability/tracing/configuration.md +1 -1
  31. package/.docs/reference/observability/tracing/exporters/arize.md +1 -1
  32. package/.docs/reference/observability/tracing/exporters/arthur.md +1 -1
  33. package/.docs/reference/observability/tracing/exporters/cloud-exporter.md +1 -1
  34. package/.docs/reference/observability/tracing/exporters/console-exporter.md +1 -1
  35. package/.docs/reference/observability/tracing/exporters/default-exporter.md +1 -1
  36. package/.docs/reference/observability/tracing/exporters/mastra-platform-exporter.md +1 -1
  37. package/.docs/reference/observability/tracing/exporters/mastra-storage-exporter.md +1 -1
  38. package/.docs/reference/observability/tracing/exporters/otel.md +2 -2
  39. package/.docs/reference/observability/tracing/span-filtering.md +1 -1
  40. package/.docs/reference/pubsub/base.md +24 -0
  41. package/.docs/reference/pubsub/caching-pubsub.md +6 -0
  42. package/.docs/reference/pubsub/event-emitter.md +36 -1
  43. package/.docs/reference/signals/create-notification-inbox-tool.md +67 -0
  44. package/.docs/reference/signals/signal-provider.md +406 -0
  45. package/.docs/reference/signals/webhook-signal-provider.md +137 -0
  46. package/.docs/reference/storage/clickhouse.md +3 -3
  47. package/.docs/reference/storage/composite.md +1 -1
  48. package/.docs/reference/storage/duckdb.md +1 -1
  49. package/.docs/reference/storage/libsql.md +1 -1
  50. package/.docs/reference/storage/postgresql.md +1 -1
  51. package/CHANGELOG.md +15 -0
  52. package/package.json +7 -7
  53. /package/.docs/docs/observability/{tracing → integrations}/exporters/arize.md +0 -0
  54. /package/.docs/docs/observability/{tracing → integrations}/exporters/arthur.md +0 -0
  55. /package/.docs/docs/observability/{tracing → integrations}/exporters/braintrust.md +0 -0
  56. /package/.docs/docs/observability/{tracing → integrations}/exporters/laminar.md +0 -0
  57. /package/.docs/docs/observability/{tracing → integrations}/exporters/langfuse.md +0 -0
  58. /package/.docs/docs/observability/{tracing → integrations}/exporters/langsmith.md +0 -0
  59. /package/.docs/docs/observability/{tracing → integrations}/exporters/sentry.md +0 -0
  60. /package/.docs/docs/observability/{tracing → integrations}/processors/sensitive-data-filter.md +0 -0
@@ -1,166 +1,29 @@
1
1
  # Tracing
2
2
 
3
- Tracing provides specialized monitoring and debugging for the AI-related operations in your application. When enabled, Mastra automatically creates traces for agent runs, LLM generations, tool calls, and workflow steps with AI-specific context and metadata.
3
+ Tracing is the observability signal that records how a request moves through agents, workflows, tools, and model calls. Mastra represents each operation as a span and groups related spans into a trace so you can inspect the full execution path.
4
4
 
5
- Unlike traditional application tracing, Tracing focuses specifically on understanding your AI pipeline — capturing token usage, model parameters, tool execution details, and conversation flows. This makes it easier to debug issues, optimize performance, and understand how your AI systems behave in production.
5
+ This page focuses on trace-specific concepts: span hierarchy, sampling, metadata, filtering, trace IDs, and third-party trace context.
6
6
 
7
- ## How it works
7
+ ## When to use tracing
8
8
 
9
- Traces are created by:
9
+ - Debug unexpected agent or workflow behavior by inspecting the full execution path.
10
+ - Follow model calls, tool calls, and workflow steps inside a single request.
11
+ - Attach trace-specific metadata and tags for filtering and investigation.
12
+ - Connect Mastra traces to a third-party tracing system.
10
13
 
11
- - **Configure exporters** → send trace data to observability platforms
12
- - **Set sampling strategies** → control which traces are collected
13
- - **Run agents and workflows** → Mastra auto-instruments them with Tracing
14
+ ## Get started
14
15
 
15
- ## Configuration
16
+ To get started with tracing, configure observability in your Mastra instance and run an agent or workflow. You can configure behavior through the following features:
16
17
 
17
- ### Basic Config
18
-
19
- ```ts
20
- import { Mastra } from '@mastra/core'
21
- import { LibSQLStore } from '@mastra/libsql'
22
- import {
23
- Observability,
24
- MastraStorageExporter,
25
- MastraPlatformExporter,
26
- SensitiveDataFilter,
27
- } from '@mastra/observability'
28
-
29
- export const mastra = new Mastra({
30
- observability: new Observability({
31
- configs: {
32
- default: {
33
- serviceName: 'mastra',
34
- exporters: [
35
- new MastraStorageExporter(), // Persists observability events to Mastra Storage
36
- new MastraPlatformExporter(), // Sends observability events to Mastra platform (if MASTRA_PLATFORM_ACCESS_TOKEN is set)
37
- ],
38
- spanOutputProcessors: [
39
- new SensitiveDataFilter(), // Redacts sensitive data like passwords, tokens, keys
40
- ],
41
- },
42
- },
43
- }),
44
- storage: new LibSQLStore({
45
- id: 'mastra-storage',
46
- url: 'file:./mastra.db', // Storage is required for tracing
47
- }),
48
- })
49
- ```
50
-
51
- This configuration includes:
52
-
53
- - **Service Name**: `"mastra"` - identifies your service in traces
54
-
55
- - **Sampling**: `"always"` by default (100% of traces)
56
-
57
- - **Exporters**:
58
-
59
- - `MastraStorageExporter` - Persists observability events to Mastra Storage
60
- - `MastraPlatformExporter` - Sends observability events to Mastra platform (requires `MASTRA_PLATFORM_ACCESS_TOKEN`)
61
-
62
- - **Span Output Processors**: `SensitiveDataFilter` - Redacts sensitive fields
63
-
64
- ## Exporters
65
-
66
- Exporters determine where your trace data is sent and how it's stored. They integrate with your existing observability stack, support data residency requirements, and can be optimized for cost and performance. You can use multiple exporters simultaneously to send the same trace data to different destinations — for example, storing detailed traces locally for debugging while sending sampled data to a cloud provider for production monitoring.
67
-
68
- ### Internal Exporters
69
-
70
- Mastra provides two built-in exporters:
71
-
72
- - **[Mastra Storage](https://mastra.ai/docs/observability/tracing/exporters/mastra-storage)**: Persists observability events to Mastra Storage for viewing in Studio
73
- - **[Mastra platform](https://mastra.ai/docs/observability/tracing/exporters/mastra-platform)**: Sends observability events to Mastra platform for production monitoring and collaboration
74
-
75
- ### External Exporters
76
-
77
- In addition to the internal exporters, Mastra supports integration with popular observability platforms. These exporters allow you to leverage your existing monitoring infrastructure and take advantage of platform-specific features like alerting, dashboards, and correlation with other application metrics.
78
-
79
- - **[Arize](https://mastra.ai/docs/observability/tracing/exporters/arize)** - Exports traces to Arize Phoenix or Arize AX using OpenInference semantic conventions
80
- - **[Braintrust](https://mastra.ai/docs/observability/tracing/exporters/braintrust)** - Exports traces to Braintrust's eval and observability platform
81
- - **[Datadog](https://mastra.ai/docs/observability/tracing/exporters/datadog)** - Sends traces to Datadog APM via OTLP for full-stack observability with AI tracing
82
- - **[Laminar](https://mastra.ai/docs/observability/tracing/exporters/laminar)** - Sends traces to Laminar via OTLP/HTTP (protobuf) with Laminar-native span attributes + scorer support
83
- - **[Langfuse](https://mastra.ai/docs/observability/tracing/exporters/langfuse)** - Sends traces to the Langfuse open-source LLM engineering platform
84
- - **[LangSmith](https://mastra.ai/docs/observability/tracing/exporters/langsmith)** - Pushes traces into LangSmith's observability and evaluation toolkit
85
- - **[PostHog](https://mastra.ai/docs/observability/tracing/exporters/posthog)** - Sends traces to PostHog for AI analytics and product insights
86
- - **[Sentry](https://mastra.ai/docs/observability/tracing/exporters/sentry)** - Sends traces to Sentry for AI tracing and monitoring using OpenTelemetry semantic conventions
87
- - **[OpenTelemetry](https://mastra.ai/docs/observability/tracing/exporters/otel)** - Deliver traces to any OpenTelemetry-compatible observability system
88
- - Supports: Dash0, MLflow, New Relic, SigNoz, Traceloop, Zipkin, and others!
89
-
90
- ## Bridges
91
-
92
- Bridges provide bidirectional integration with external tracing systems. Unlike exporters that send trace data to external platforms, bridges create native spans in external systems and inherit context from them. This enables Mastra operations to participate in existing distributed traces.
93
-
94
- - **[OpenTelemetry Bridge](https://mastra.ai/docs/observability/tracing/bridges/otel)** - Integrate with existing OpenTelemetry infrastructure
95
-
96
- ### Bridges vs Exporters
97
-
98
- | Feature | Bridges | Exporters |
99
- | ---------------------------------------- | ---------------------------- | ------------------------- |
100
- | Creates native spans in external systems | Yes | No |
101
- | Inherits context from external systems | Yes | No |
102
- | Sends data to backends | Via external SDK | Directly |
103
- | Use case | Existing distributed tracing | Standalone Mastra tracing |
104
-
105
- You can use both together — a bridge for context propagation and exporters to send traces to additional destinations.
18
+ - [Configuration](https://mastra.ai/docs/observability/config): Base observability config, multiple configs, and serverless flushing
19
+ - [Storage](https://mastra.ai/docs/observability/storage): Storage routing for traces, logs, and metrics
20
+ - [Integrations overview](https://mastra.ai/docs/observability/integrations/overview): Exporters, bridges, and processors
106
21
 
107
22
  ## Sampling strategies
108
23
 
109
24
  Sampling allows you to control which traces are collected, helping you balance between observability needs and resource costs. In production environments with high traffic, collecting every trace can be expensive and unnecessary. Sampling strategies let you capture a representative subset of traces while ensuring you don't miss critical information about errors or important operations.
110
25
 
111
- Mastra supports four sampling strategies:
112
-
113
- ### Always Sample
114
-
115
- Collects 100% of traces. Best for development, debugging, or low-traffic scenarios where you need complete visibility.
116
-
117
- ```ts
118
- sampling: {
119
- type: 'always'
120
- }
121
- ```
122
-
123
- ### Never Sample
124
-
125
- Disables tracing entirely. Useful for specific environments where tracing adds no value or when you need to temporarily disable tracing without removing configuration.
126
-
127
- ```ts
128
- sampling: {
129
- type: 'never'
130
- }
131
- ```
132
-
133
- ### Ratio-Based Sampling
134
-
135
- Randomly samples a percentage of traces. Ideal for production environments where you want statistical insights without the cost of full tracing. The probability value ranges from 0 (no traces) to 1 (all traces).
136
-
137
- ```ts
138
- sampling: {
139
- type: 'ratio',
140
- probability: 0.1 // Sample 10% of traces
141
- }
142
- ```
143
-
144
- ### Custom Sampling
145
-
146
- Implements your own sampling logic based on request context, metadata, or business rules. Perfect for complex scenarios like sampling based on user tier, request type, or error conditions.
147
-
148
- ```ts
149
- sampling: {
150
- type: 'custom',
151
- sampler: (options) => {
152
- // Sample premium users at higher rate
153
- if (options?.metadata?.userTier === 'premium') {
154
- return Math.random() < 0.5; // 50% sampling
155
- }
156
-
157
- // Default 1% sampling for others
158
- return Math.random() < 0.01;
159
- }
160
- }
161
- ```
162
-
163
- ### Complete Example
26
+ You can configure sampling at the observability config level:
164
27
 
165
28
  ```ts
166
29
  export const mastra = new Mastra({
@@ -180,139 +43,49 @@ export const mastra = new Mastra({
180
43
  })
181
44
  ```
182
45
 
183
- ## Multi-config setup
184
-
185
- Complex applications often require different tracing configurations for different scenarios. You might want detailed traces with full sampling during development, sampled traces sent to external providers in production, and specialized configurations for specific features or customer segments. The `configSelector` function enables dynamic configuration selection at runtime, allowing you to route traces based on request context, environment variables, feature flags, or any custom logic.
186
-
187
- This approach is particularly valuable when:
188
-
189
- - Running A/B tests with different observability requirements
190
- - Providing enhanced debugging for specific customers or support cases
191
- - Gradually rolling out new tracing providers without affecting existing monitoring
192
- - Optimizing costs by using different sampling rates for different request types
193
- - Maintaining separate trace streams for compliance or data residency requirements
194
-
195
- > **Info:** Note that only a single config can be used for a specific execution. But a single config can send data to multiple exporters simultaneously.
196
-
197
- ### Dynamic Configuration Selection
198
-
199
- Use `configSelector` to choose the appropriate tracing configuration based on request context:
200
-
201
- ```ts
202
- export const mastra = new Mastra({
203
- observability: new Observability({
204
- configs: {
205
- langfuse: {
206
- serviceName: 'langfuse-service',
207
- exporters: [langfuseExporter],
208
- },
209
- braintrust: {
210
- serviceName: 'braintrust-service',
211
- exporters: [braintrustExporter],
212
- },
213
- debug: {
214
- serviceName: 'debug-service',
215
- sampling: { type: 'always' },
216
- exporters: [new MastraStorageExporter()],
217
- },
218
- },
219
- configSelector: (context, availableTracers) => {
220
- // Use debug config for support requests
221
- if (context.requestContext?.get('supportMode')) {
222
- return 'debug'
223
- }
224
-
225
- // Route specific customers to different providers
226
- const customerId = context.requestContext?.get('customerId')
227
- if (customerId && premiumCustomers.includes(customerId)) {
228
- return 'braintrust'
229
- }
230
-
231
- // Route specific requests to langfuse
232
- if (context.requestContext?.get('useExternalTracing')) {
233
- return 'langfuse'
234
- }
235
-
236
- throw new Error('no config found')
237
- },
238
- }),
239
- })
240
- ```
241
-
242
- ### Environment-Based Configuration
46
+ The `sampling` option allows you to control which traces are collected, helping you balance between observability needs and resource costs. Mastra supports four sampling strategies:
243
47
 
244
- A common pattern is to select configurations based on deployment environment:
48
+ 1. **Always Sample**: Collects 100% of traces. Best for development, debugging, or low-traffic scenarios where you need complete visibility.
245
49
 
246
- ```ts
247
- export const mastra = new Mastra({
248
- observability: new Observability({
249
- configs: {
250
- development: {
251
- serviceName: 'my-service-dev',
252
- sampling: { type: 'always' },
253
- exporters: [new MastraStorageExporter()],
254
- },
255
- staging: {
256
- serviceName: 'my-service-staging',
257
- sampling: { type: 'ratio', probability: 0.5 },
258
- exporters: [langfuseExporter],
259
- },
260
- production: {
261
- serviceName: 'my-service-prod',
262
- sampling: { type: 'ratio', probability: 0.01 },
263
- exporters: [mastraObserveExporter, langfuseExporter],
264
- },
265
- },
266
- configSelector: (context, availableTracers) => {
267
- const env = process.env.NODE_ENV || 'development'
268
- return env
269
- },
270
- }),
271
- })
272
- ```
50
+ ```ts
51
+ sampling: {
52
+ type: 'always'
53
+ }
54
+ ```
273
55
 
274
- ### Common Configuration Patterns & Troubleshooting
56
+ 2. **Never Sample**: Disables tracing entirely. Useful for specific environments where tracing adds no value or when you need to temporarily disable tracing without removing configuration.
275
57
 
276
- #### Maintaining Studio access
58
+ ```ts
59
+ sampling: {
60
+ type: 'never'
61
+ }
62
+ ```
277
63
 
278
- When adding external exporters, include `MastraStorageExporter` and `MastraPlatformExporter` to maintain access to Studio:
64
+ 3. **Ratio-Based Sampling**: Randomly samples a percentage of traces. Ideal for production environments where you want statistical insights without the cost of full tracing. The probability value ranges from 0 (no traces) to 1 (all traces).
279
65
 
280
- ```ts
281
- import {
282
- Observability,
283
- MastraStorageExporter,
284
- MastraPlatformExporter,
285
- SensitiveDataFilter,
286
- } from '@mastra/observability'
287
- import { ArizeExporter } from '@mastra/arize'
66
+ ```ts
67
+ sampling: {
68
+ type: 'ratio',
69
+ probability: 0.1 // Sample 10% of traces
70
+ }
71
+ ```
288
72
 
289
- export const mastra = new Mastra({
290
- observability: new Observability({
291
- configs: {
292
- production: {
293
- serviceName: 'my-service',
294
- exporters: [
295
- new ArizeExporter({
296
- endpoint: process.env.PHOENIX_COLLECTOR_ENDPOINT,
297
- apiKey: process.env.PHOENIX_API_KEY,
298
- }),
299
- new MastraStorageExporter(), // Keep Studio access
300
- new MastraPlatformExporter(), // Keep Cloud access
301
- ],
302
- spanOutputProcessors: [new SensitiveDataFilter()],
303
- },
304
- },
305
- }),
306
- })
307
- ```
308
-
309
- This configuration sends traces to all three destinations simultaneously:
73
+ 4. **Custom Sampling**: Implements your own sampling logic based on request context, metadata, or business rules. Perfect for complex scenarios like sampling based on user tier, request type, or error conditions.
310
74
 
311
- - **Arize Phoenix/AX** for external observability
312
- - **MastraStorageExporter** for Studio
313
- - **MastraPlatformExporter** for hosted Studio dashboard
75
+ ```ts
76
+ sampling: {
77
+ type: 'custom',
78
+ sampler: (options) => {
79
+ // Sample premium users at higher rate
80
+ if (options?.metadata?.userTier === 'premium') {
81
+ return Math.random() < 0.5; // 50% sampling
82
+ }
314
83
 
315
- > **Info:** Remember: A single trace can be sent to multiple exporters. You don't need separate configs for each exporter unless you want different sampling rates or processors.
84
+ // Default 1% sampling for others
85
+ return Math.random() < 0.01;
86
+ }
87
+ }
88
+ ```
316
89
 
317
90
  ## Adding custom metadata
318
91
 
@@ -362,13 +135,13 @@ export const mastra = new Mastra({
362
135
 
363
136
  If `environment` is not set, Mastra falls back to `process.env.NODE_ENV`. If neither is set, the field is left undefined rather than guessed.
364
137
 
365
- Per-call `tracingOptions.metadata.environment` always takes precedence, so individual calls can still override the value when needed.
138
+ Per-call `tracingOptions.metadata.environment` always takes precedence, so individual calls can override the value when needed.
366
139
 
367
- ### Automatic Metadata from `RequestContext`
140
+ ### Automatic metadata from `RequestContext`
368
141
 
369
142
  Instead of manually adding metadata to each span, you can configure Mastra to automatically extract values from RequestContext and attach them as metadata to all spans in a trace. This is useful for consistently tracking user identifiers, environment information, feature flags, or any request-scoped data across your entire trace.
370
143
 
371
- #### Configuration-Level Extraction
144
+ #### Configuration-level extraction
372
145
 
373
146
  Define which RequestContext keys to extract in your tracing configuration. These keys will be automatically included as metadata for all spans created with this configuration:
374
147
 
@@ -400,7 +173,7 @@ const result = await agent.generate('Hello', {
400
173
  })
401
174
  ```
402
175
 
403
- #### Per-Request Additions
176
+ #### Per-request additions
404
177
 
405
178
  You can add trace-specific keys using `tracingOptions.requestContextKeys`. These are merged with the configuration-level keys:
406
179
 
@@ -420,7 +193,7 @@ const result = await agent.generate('Hello', {
420
193
  // All spans now have: userId, environment, AND experimentId
421
194
  ```
422
195
 
423
- #### Nested Value Extraction
196
+ #### Nested value extraction
424
197
 
425
198
  Use dot notation to extract nested values from RequestContext:
426
199
 
@@ -450,7 +223,7 @@ requestContext.set('session', { data: { experimentId: 'exp-999' } })
450
223
  3. **Child Span Extraction**: Child spans can also extract metadata if you pass `requestContext` when creating them
451
224
  4. **Metadata Precedence**: Explicit metadata passed to span options always takes precedence over extracted metadata
452
225
 
453
- ### Adding Tags to Traces
226
+ ### Adding tags to traces
454
227
 
455
228
  Tags are string labels that help you categorize and filter traces. Unlike metadata (which contains structured key-value data), tags are plain strings designed for quick filtering and organization.
456
229
 
@@ -474,17 +247,17 @@ const result = await run.start({
474
247
  })
475
248
  ```
476
249
 
477
- #### How Tags Work
250
+ #### How tags work
478
251
 
479
252
  - **Root span only**: Tags are applied only to the root span of a trace (the agent run or workflow run span)
480
253
 
481
254
  - **Widely supported**: Tags are supported by most exporters for filtering and searching traces:
482
255
 
483
- - **Braintrust** - Native `tags` field
484
- - **Langfuse** - Native `tags` field on traces
485
- - **ArizeExporter** - `tag.tags` OpenInference attribute
486
- - **OtelExporter** - `mastra.tags` span attribute
487
- - **OtelBridge** - `mastra.tags` span attribute
256
+ - **Braintrust**: Native `tags` field
257
+ - **Langfuse**: Native `tags` field on traces
258
+ - **ArizeExporter**: `tag.tags` OpenInference attribute
259
+ - **OtelExporter**: `mastra.tags` span attribute
260
+ - **OtelBridge**: `mastra.tags` span attribute
488
261
 
489
262
  - **Combinable with metadata**: You can use both `tags` and `metadata` in the same `tracingOptions`
490
263
 
@@ -497,7 +270,7 @@ const result = await agent.generate([{ role: 'user', content: 'Analyze this' }],
497
270
  })
498
271
  ```
499
272
 
500
- #### Common Tag Patterns
273
+ #### Common tag patterns
501
274
 
502
275
  - **Environment**: `"production"`, `"staging"`, `"development"`
503
276
  - **Feature flags**: `"feature-x-enabled"`, `"beta-user"`
@@ -505,7 +278,7 @@ const result = await agent.generate([{ role: 'user', content: 'Analyze this' }],
505
278
  - **Priority levels**: `"priority-high"`, `"priority-low"`
506
279
  - **Experiments**: `"experiment-v1"`, `"control-group"`, `"treatment-a"`
507
280
 
508
- ### Hiding Sensitive Input/Output
281
+ ### Hiding sensitive input/output
509
282
 
510
283
  When processing sensitive data, you may want to prevent input and output values from being logged to your observability platforms. Use `hideInput` and `hideOutput` in `tracingOptions` to exclude this data from all spans in a trace:
511
284
 
@@ -536,7 +309,7 @@ const result = await agent.generate([{ role: 'user', content: 'Handle confidenti
536
309
  #### How it works
537
310
 
538
311
  - **Trace-wide effect**: When set on the root span, these options apply to all child spans in the trace (tool calls, model generations, etc.)
539
- - **Export-time filtering**: The data is still available internally during execution but is excluded when spans are exported to observability platforms
312
+ - **Export-time filtering**: The data remains available internally during execution but is excluded when spans are exported to observability platforms
540
313
  - **Combinable with other options**: You can use `hideInput`/`hideOutput` alongside `tags`, `metadata`, and other `tracingOptions`
541
314
 
542
315
  ```ts
@@ -550,9 +323,9 @@ const result = await agent.generate([{ role: 'user', content: 'Sensitive operati
550
323
  })
551
324
  ```
552
325
 
553
- > **Tip:** For more granular control over sensitive data, consider using the [Sensitive Data Filter](https://mastra.ai/docs/observability/tracing/processors/sensitive-data-filter) processor, which can redact specific fields (like passwords, tokens, and keys) while preserving the rest of the input/output.
326
+ > **Tip:** For more granular control over sensitive data, consider using the [Sensitive Data Filter](https://mastra.ai/docs/observability/integrations/processors/sensitive-data-filter) processor, which can redact specific fields (like passwords, tokens, and keys) while preserving the rest of the input/output.
554
327
 
555
- #### Child Spans and Metadata Extraction
328
+ #### Child spans and metadata extraction
556
329
 
557
330
  When creating child spans within tools or workflow steps, you can pass the `requestContext` parameter to enable metadata extraction:
558
331
 
@@ -634,15 +407,15 @@ Mastra provides two ways to transform span data before it reaches your observabi
634
407
 
635
408
  Use **span processors** for synchronous transformations that should apply to all exporters (like redacting sensitive data). Use **custom span formatters** when different exporters need different representations of the same data (like plain text for one platform and structured data for another), or when you need to perform asynchronous operations like fetching data from external APIs.
636
409
 
637
- ### Span Processors
410
+ ### Span processors
638
411
 
639
412
  Span processors transform, filter, or enrich trace data before it's exported. They act as a pipeline between span creation and export, enabling you to modify spans for security, compliance, or debugging purposes. Processors run once and affect all exporters.
640
413
 
641
- #### Built-in Processors
414
+ #### Built-in processors
642
415
 
643
- - [Sensitive Data Filter](https://mastra.ai/docs/observability/tracing/processors/sensitive-data-filter) redacts sensitive information. It's enabled in the default observability config.
416
+ - [Sensitive Data Filter](https://mastra.ai/docs/observability/integrations/processors/sensitive-data-filter) redacts sensitive information. It's enabled in the default observability config.
644
417
 
645
- #### Creating Custom Processors
418
+ #### Creating custom processors
646
419
 
647
420
  You can create custom span processors by implementing the `SpanOutputProcessor` interface. Here's a basic example that converts all input text in spans to lowercase:
648
421
 
@@ -683,6 +456,8 @@ Processors are executed in the order they're defined, allowing you to chain mult
683
456
  - Normalizing data formats
684
457
  - Enriching spans with business context
685
458
 
459
+ For the broader exporter, bridge, and processor model, see [Integrations overview](https://mastra.ai/docs/observability/integrations/overview).
460
+
686
461
  ## Span filtering
687
462
 
688
463
  Span filtering lets you reduce noise and per-span costs before data reaches your observability platform. Configure it per observability instance, so different exporters or environments can keep different levels of detail.
@@ -727,16 +502,16 @@ Filtering happens at export time in this order:
727
502
 
728
503
  If `spanFilter` throws, Mastra keeps the span and logs the error to avoid silent data loss. For the full span type list and more examples, see the [Span filtering reference](https://mastra.ai/reference/observability/tracing/span-filtering).
729
504
 
730
- ### Custom Span Formatters
505
+ ### Custom span formatters
731
506
 
732
507
  Custom span formatters transform how spans appear in specific observability platforms. Unlike span processors, formatters are configured per-exporter, allowing different formatting for different destinations. Formatters support both synchronous and asynchronous operations.
733
508
 
734
509
  #### Use cases
735
510
 
736
- - **Extract plain text from AI SDK messages** - Convert structured message arrays to readable text
737
- - **Transform input/output formats** - Customize how data displays in specific platforms
738
- - **Platform-specific field mapping** - Add or remove fields based on platform requirements
739
- - **Async data enrichment** - Fetch additional context from external APIs or databases
511
+ - **Extract plain text from AI SDK messages**: Convert structured message arrays to readable text
512
+ - **Transform input/output formats**: Customize how data displays in specific platforms
513
+ - **Platform-specific field mapping**: Add or remove fields based on platform requirements
514
+ - **Async data enrichment**: Fetch additional context from external APIs or databases
740
515
 
741
516
  #### Configuration
742
517
 
@@ -779,7 +554,7 @@ export const mastra = new Mastra({
779
554
  })
780
555
  ```
781
556
 
782
- #### Chaining Multiple Formatters
557
+ #### Chaining multiple formatters
783
558
 
784
559
  Use `chainFormatters` to combine multiple formatters. Chains support both sync and async formatters:
785
560
 
@@ -801,7 +576,7 @@ const exporter = new BraintrustExporter({
801
576
  })
802
577
  ```
803
578
 
804
- #### Async Formatters
579
+ #### Async formatters
805
580
 
806
581
  Custom span formatters support asynchronous operations, enabling use cases like fetching data from external APIs or databases to enrich your spans:
807
582
 
@@ -888,7 +663,7 @@ export const mastra = new Mastra({
888
663
  })
889
664
  ```
890
665
 
891
- ### Available Options
666
+ ### Available options
892
667
 
893
668
  | Option | Default | Description |
894
669
  | ----------------- | ------- | ---------------------------------------------------------------- |
@@ -926,7 +701,7 @@ All options are optional — if not specified, they fall back to the defaults sh
926
701
 
927
702
  When you execute agents or workflows with tracing enabled, the response includes a `traceId` that you can use to look up the full trace in your observability platform. This is useful for debugging, customer support, or correlating traces with other events in your system.
928
703
 
929
- ### Agent Trace IDs
704
+ ### Agent trace IDs
930
705
 
931
706
  Both `generate` and `stream` methods return the trace ID in their response:
932
707
 
@@ -942,7 +717,7 @@ const streamResult = await agent.stream('Tell me a story')
942
717
  console.log('Trace ID:', streamResult.traceId)
943
718
  ```
944
719
 
945
- ### Workflow Trace IDs
720
+ ### Workflow trace IDs
946
721
 
947
722
  Workflow executions also return trace IDs:
948
723
 
@@ -967,7 +742,7 @@ const finalState = await getWorkflowState()
967
742
  console.log('Trace ID:', finalState.traceId)
968
743
  ```
969
744
 
970
- ### Using Trace IDs
745
+ ### Using trace IDs
971
746
 
972
747
  Once you have a trace ID, you can:
973
748
 
@@ -982,7 +757,7 @@ The trace ID is only available when tracing is enabled. If tracing is disabled o
982
757
 
983
758
  When running Mastra agents or workflows within applications that have existing distributed tracing (OpenTelemetry, Datadog, etc.), you can connect Mastra traces to your parent trace context. This creates a unified view of your entire request flow, making it easier to understand how Mastra operations fit into the broader system.
984
759
 
985
- ### Passing External Trace IDs
760
+ ### Passing external trace IDs
986
761
 
987
762
  Use the `tracingOptions` parameter to specify the trace context from your parent system:
988
763
 
@@ -1002,7 +777,7 @@ const result = await agent.generate('Analyze this data', {
1002
777
  // The Mastra trace will now appear as a child in your distributed trace
1003
778
  ```
1004
779
 
1005
- ### OpenTelemetry Integration
780
+ ### OpenTelemetry integration
1006
781
 
1007
782
  Integration with OpenTelemetry allows Mastra traces to appear seamlessly in your existing observability platform:
1008
783
 
@@ -1023,7 +798,7 @@ if (spanContext) {
1023
798
  }
1024
799
  ```
1025
800
 
1026
- ### Workflow Integration
801
+ ### Workflow integration
1027
802
 
1028
803
  Workflows support the same pattern for trace propagation:
1029
804
 
@@ -1040,7 +815,7 @@ const result = await run.start({
1040
815
  })
1041
816
  ```
1042
817
 
1043
- ### ID Format Requirements
818
+ ### ID format requirements
1044
819
 
1045
820
  Mastra validates trace and span IDs to ensure compatibility:
1046
821
 
@@ -1054,7 +829,7 @@ Invalid IDs are handled gracefully — Mastra logs an error and continues:
1054
829
 
1055
830
  This ensures tracing never crashes your application, even with malformed input.
1056
831
 
1057
- ### Example: Express Middleware
832
+ ### Example: Express middleware
1058
833
 
1059
834
  Here's a complete example showing trace propagation in an Express application:
1060
835
 
@@ -1084,97 +859,30 @@ app.post('/api/analyze', async (req, res) => {
1084
859
 
1085
860
  This creates a single distributed trace that includes both the HTTP request handling and the Mastra agent execution, viewable in your observability platform of choice.
1086
861
 
1087
- ## Flushing traces in serverless environments
1088
-
1089
- In serverless environments like Vercel's fluid compute, AWS Lambda, or Cloudflare Workers, runtime instances can be reused across multiple requests. The `flush()` method allows you to ensure all buffered spans are exported before the runtime terminates, without shutting down the exporter (which would prevent future exports).
1090
-
1091
- > **Storage requirements:** Serverless environments have ephemeral filesystems. Use external storage instead of local file storage (`file:./mastra.db`). See the [Vercel deployment guide](https://mastra.ai/guides/deployment/vercel) for a complete setup example.
1092
-
1093
- ### Using `flush()`
1094
-
1095
- Call `flush()` on the observability instance to flush all exporters:
1096
-
1097
- ```ts
1098
- // Get the observability instance from Mastra
1099
- const observability = mastra.getObservability()
1100
-
1101
- // Flush all buffered spans to all exporters
1102
- await observability.flush()
1103
- ```
1104
-
1105
- ### When to Use `flush()`
1106
-
1107
- Use `flush()` in these scenarios:
1108
-
1109
- - **End of serverless function execution**: Ensure spans are exported before the runtime is paused or terminated
1110
- - **Before long-running operations**: Flush accumulated spans before a potentially slow operation
1111
- - **Periodic flushing**: In long-running processes, periodically flush to ensure timely data availability
1112
-
1113
- ```ts
1114
- // Example: Vercel serverless function
1115
- export async function POST(req: Request) {
1116
- const result = await agent.generate([{ role: 'user', content: await req.text() }])
1117
-
1118
- // Ensure spans are exported before function completes
1119
- const observability = mastra.getObservability()
1120
- await observability.flush()
1121
-
1122
- return Response.json(result)
1123
- }
1124
- ```
1125
-
1126
- ### `flush()` vs `shutdown()`
1127
-
1128
- | Method | Behavior | Use Case |
1129
- | ------------ | --------------------------------------------- | ------------------------------------------ |
1130
- | `flush()` | Exports buffered spans, keeps exporter active | Serverless environments, periodic flushing |
1131
- | `shutdown()` | Exports buffered spans, releases resources | Application shutdown, graceful termination |
1132
-
1133
- Use `flush()` when you need to ensure data is exported but want to keep the exporter ready for future requests. Use `shutdown()` only when the application is terminating.
1134
-
1135
862
  ## What gets traced
1136
863
 
1137
864
  Mastra automatically creates spans for:
1138
865
 
1139
- ### Agent Operations
866
+ ### Agent operations
1140
867
 
1141
- - **Agent runs** - Complete execution with instructions and tools
1142
- - **LLM calls** - Model interactions with tokens and parameters
1143
- - **Tool executions** - Function calls with inputs and outputs
1144
- - **Memory operations** - Thread and semantic recall
868
+ - **Agent runs**: Complete execution with instructions and tools
869
+ - **LLM calls**: Model interactions with tokens and parameters
870
+ - **Tool executions**: Function calls with inputs and outputs
871
+ - **Memory operations**: Thread and semantic recall
1145
872
 
1146
- ### Workflow Operations
873
+ ### Workflow operations
1147
874
 
1148
- - **Workflow runs** - Full execution from start to finish
1149
- - **Individual steps** - Step processing with inputs/outputs
1150
- - **Control flow** - Conditionals, loops, parallel execution
1151
- - **Wait operations** - Delays and event waiting
875
+ - **Workflow runs**: Full execution from start to finish
876
+ - **Individual steps**: Step processing with inputs/outputs
877
+ - **Control flow**: Conditionals, loops, parallel execution
878
+ - **Wait operations**: Delays and event waiting
1152
879
 
1153
880
  ## See also
1154
881
 
1155
- ### Reference Documentation
882
+ ### Reference documentation
1156
883
 
1157
884
  - [Configuration API](https://mastra.ai/reference/observability/tracing/configuration): ObservabilityConfig details
1158
885
  - [Tracing Classes](https://mastra.ai/reference/observability/tracing/instances): Core classes and methods
1159
886
  - [Span Interfaces](https://mastra.ai/reference/observability/tracing/spans): Span types and lifecycle
1160
887
  - [Type Definitions](https://mastra.ai/reference/observability/tracing/interfaces): Complete interface reference
1161
- - [Span filtering](https://mastra.ai/reference/observability/tracing/span-filtering): Filtering behavior, span types, and examples
1162
-
1163
- ### Exporters
1164
-
1165
- - [MastraStorageExporter](https://mastra.ai/reference/observability/tracing/exporters/mastra-storage-exporter): Storage persistence
1166
- - [MastraPlatformExporter](https://mastra.ai/reference/observability/tracing/exporters/mastra-platform-exporter): Mastra platform integration
1167
- - [ConsoleExporter](https://mastra.ai/reference/observability/tracing/exporters/console-exporter): Debug output
1168
- - [Arize](https://mastra.ai/reference/observability/tracing/exporters/arize): Arize Phoenix and Arize AX integration
1169
- - [Braintrust](https://mastra.ai/reference/observability/tracing/exporters/braintrust): Braintrust integration
1170
- - [Langfuse](https://mastra.ai/reference/observability/tracing/exporters/langfuse): Langfuse integration
1171
- - [MLflow](https://mastra.ai/docs/observability/tracing/exporters/otel): MLflow OTLP endpoint setup
1172
- - [OpenTelemetry](https://mastra.ai/reference/observability/tracing/exporters/otel): OTEL-compatible platforms
1173
-
1174
- ### Bridges
1175
-
1176
- - [OpenTelemetry Bridge](https://mastra.ai/reference/observability/tracing/bridges/otel): OTEL context integration
1177
-
1178
- ### Processors
1179
-
1180
- - [Sensitive Data Filter](https://mastra.ai/docs/observability/tracing/processors/sensitive-data-filter): Data redaction
888
+ - [Span filtering](https://mastra.ai/reference/observability/tracing/span-filtering): Filtering behavior, span types, and examples