@mastra/client-js 1.56.0-alpha.3 → 1.56.0-alpha.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/client.d.ts +2 -2
- package/dist/client.d.ts.map +1 -1
- package/dist/docs/SKILL.md +2 -2
- package/dist/docs/assets/SOURCE_MAP.json +1 -1
- package/dist/docs/references/docs-harness-agent-controller.md +1 -1
- package/dist/docs/references/reference-client-js-agent-controller.md +2 -0
- package/dist/docs/references/reference-client-js-observability.md +121 -6
- package/dist/docs/references/{reference-observability-tracing-trace-query.md → reference-observability-tracing-advanced-queries.md} +477 -158
- package/dist/index.cjs +81 -7
- package/dist/index.cjs.map +1 -1
- package/dist/index.js +81 -7
- package/dist/index.js.map +1 -1
- package/dist/resources/agent-controller.d.ts +11 -0
- package/dist/resources/agent-controller.d.ts.map +1 -1
- package/dist/resources/base.d.ts.map +1 -1
- package/dist/resources/observability.d.ts +9 -7
- package/dist/resources/observability.d.ts.map +1 -1
- package/dist/route-types.generated.d.ts +81 -16
- package/dist/route-types.generated.d.ts.map +1 -1
- package/dist/utils/ai-sdk-v4/process-data-stream.d.ts +1 -1
- package/dist/utils/ai-sdk-v4/process-data-stream.d.ts.map +1 -1
- package/dist/utils/index.d.ts +7 -0
- package/dist/utils/index.d.ts.map +1 -1
- package/package.json +8 -9
|
@@ -2,26 +2,51 @@
|
|
|
2
2
|
|
|
3
3
|
> Discover all available pages from the documentation index: https://mastra.ai/llms.txt
|
|
4
4
|
|
|
5
|
-
#
|
|
5
|
+
# Advanced trace queries
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
Advanced trace queries select completed traces with a structured time range and predicate, then either list or aggregate the matching traces:
|
|
8
8
|
|
|
9
|
-
|
|
9
|
+
| API | Use it to | Client SDK | HTTP endpoint |
|
|
10
|
+
| ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ------------------------------------------ |
|
|
11
|
+
| Query traces | List matching traces, paginate through them, and poll for new ones. | `queryTraces()` | `POST /api/observability/traces/query` |
|
|
12
|
+
| Aggregate traces | Count matching traces and compute measures, such as error rate, latency percentiles, tokens, and cost, optionally grouped by dimensions and time buckets. | `aggregateTraces()` | `POST /api/observability/traces/aggregate` |
|
|
10
13
|
|
|
11
|
-
|
|
14
|
+
Both APIs accept the same [trace selection](#select-traces). The filters, operators, and limits on this page apply to each. Use the aggregate API to find the group that needs attention, such as an agent with a high error rate, then send the same predicate plus the group's dimensions to the query API to list the traces behind it. Each query result represents a trace. Use its `traceId` to [retrieve the full trace](https://mastra.ai/reference/client-js/observability).
|
|
15
|
+
|
|
16
|
+
The endpoints use the same authentication as other observability routes and require the `observability:read` permission. Every request requires a time range.
|
|
12
17
|
|
|
13
18
|
For hosted traces, see [Query observability data on Mastra Platform](https://mastra.ai/docs/mastra-platform/observability) for project credentials and regional endpoints.
|
|
14
19
|
|
|
15
|
-
##
|
|
20
|
+
## Check capabilities
|
|
16
21
|
|
|
17
|
-
|
|
22
|
+
The configured observability store must support each API. Call `getObservabilityCapabilities()` or `GET /api/observability/capabilities` and check these flags before showing a feature:
|
|
18
23
|
|
|
19
|
-
|
|
20
|
-
|
|
24
|
+
| Capability | Store feature | Enables |
|
|
25
|
+
| ---------------------------- | --------------------------------- | -------------------------------------------------------------------------------------- |
|
|
26
|
+
| `traceQuery` | `trace-query` | `queryTraces()` |
|
|
27
|
+
| `traceAggregate` | `trace-aggregate` | `aggregateTraces()` |
|
|
28
|
+
| `traceQueryRootDuration` | `trace-query-root-duration` | Trace-scope `durationMs` predicates |
|
|
29
|
+
| `traceQueryTimeToFirstChunk` | `trace-query-time-to-first-chunk` | `timeToFirstChunkMs` predicates and results |
|
|
30
|
+
| `traceQueryContextIds` | `trace-query-context-ids` | `runId`, `sessionId`, `userId`, and `organizationId` predicates |
|
|
31
|
+
| `traceQueryRootSpanFields` | `trace-query-root-span-fields` | Trace-scope `name` and `spanType` predicates, dimensions, and `countDistinct` measures |
|
|
32
|
+
| `traceQueryDiscovery` | `trace-query-discovery` | [Field and value discovery](#discover-fields-and-values) |
|
|
33
|
+
| `traceQueryTenantScope` | `trace-query-tenant-scope` | [Trusted tenant scope](#trusted-tenant-scope) |
|
|
21
34
|
|
|
22
|
-
|
|
35
|
+
A missing `traceAggregate` value means the server doesn't support aggregation. Servers without the aggregate route return `404`, and stores without the feature return `501 TRACE_AGGREGATE_UNSUPPORTED`. Against a store without a sub-feature, a request that uses a gated field returns `501` before storage runs. Root span `name` and `spanType` are the exception and return `422` with a `field_not_allowed` issue.
|
|
23
36
|
|
|
24
|
-
|
|
37
|
+
## Select traces
|
|
38
|
+
|
|
39
|
+
Every request selects traces with two fields:
|
|
40
|
+
|
|
41
|
+
| Field | Required | Description |
|
|
42
|
+
| ----------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
43
|
+
| `timeRange` | Yes | Trace start-time boundary. `from` is inclusive and `to` is exclusive. Both values must be ISO timestamps, and `from` must be earlier than `to`. |
|
|
44
|
+
| `where` | No | Recursive trace predicate. Supports scalar conditions and `spans.some` or `spans.none` clauses. |
|
|
45
|
+
|
|
46
|
+
This selection matches traces from the first week of August that contain a tool call with an error:
|
|
47
|
+
|
|
48
|
+
```typescript
|
|
49
|
+
const selection = {
|
|
25
50
|
timeRange: {
|
|
26
51
|
from: '2026-08-01T00:00:00.000Z',
|
|
27
52
|
to: '2026-08-08T00:00:00.000Z',
|
|
@@ -37,140 +62,35 @@ const result = await mastraClient.queryTraces({
|
|
|
37
62
|
},
|
|
38
63
|
},
|
|
39
64
|
},
|
|
40
|
-
|
|
41
|
-
page: { limit: 25 },
|
|
42
|
-
})
|
|
65
|
+
}
|
|
43
66
|
```
|
|
44
67
|
|
|
68
|
+
The same selection returns the same traces from either API. Aggregation counts exactly the traces that a query would list.
|
|
69
|
+
|
|
45
70
|
A `spans.some` clause matches when one span satisfies all its conditions. In this example, the same span must be a tool call and have an error. Use `spans.none` to find traces with no spans matching the conditions.
|
|
46
71
|
|
|
47
72
|
Span clauses examine the current root span and current child spans. The root `timeRange` applies only to the selected current root's `startedAt`. Related spans can participate even when their own timestamps are outside that range. Related records correlate only through a matching non-null `traceId`.
|
|
48
73
|
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
Pass the query as one JSON argument to `mastra api trace query`. Filters and pagination options belong in the JSON input. This command runs the same failed-tool query as the SDK example:
|
|
52
|
-
|
|
53
|
-
```bash
|
|
54
|
-
mastra api --url http://localhost:4111 trace query '{
|
|
55
|
-
"timeRange": {
|
|
56
|
-
"from": "2026-08-01T00:00:00.000Z",
|
|
57
|
-
"to": "2026-08-08T00:00:00.000Z"
|
|
58
|
-
},
|
|
59
|
-
"where": {
|
|
60
|
-
"spans": {
|
|
61
|
-
"some": {
|
|
62
|
-
"op": "and",
|
|
63
|
-
"args": [
|
|
64
|
-
{ "op": "eq", "left": { "path": "spanType" }, "right": { "literal": "tool_call" } },
|
|
65
|
-
{ "op": "exists", "path": "error" }
|
|
66
|
-
]
|
|
67
|
-
}
|
|
68
|
-
}
|
|
69
|
-
},
|
|
70
|
-
"orderBy": [{ "field": "startedAt", "direction": "desc" }],
|
|
71
|
-
"page": { "limit": 25 }
|
|
72
|
-
}'
|
|
73
|
-
```
|
|
74
|
-
|
|
75
|
-
The explicit `--url` targets a local Mastra server. Without it, observability commands default to `https://observability.mastra.ai`, even when a local server is running. See [CLI target resolution](https://mastra.ai/reference/cli/mastra) for authentication and regional endpoints. The target server and configured store must support the requested query capabilities.
|
|
76
|
-
|
|
77
|
-
Inspect the command with:
|
|
78
|
-
|
|
79
|
-
```bash
|
|
80
|
-
mastra api trace query --help
|
|
81
|
-
```
|
|
82
|
-
|
|
83
|
-
For a local Mastra server, inspect its request and response schemas with:
|
|
84
|
-
|
|
85
|
-
```bash
|
|
86
|
-
mastra api --url http://localhost:4111 trace query --schema
|
|
87
|
-
```
|
|
88
|
-
|
|
89
|
-
Mastra Platform doesn't currently expose the schema endpoint used by `--schema`. Use [Request fields](#request-fields) and [Fields and operators](#fields-and-operators) to construct queries for either target. To find fields and values recorded in your project, use [field and value discovery](#discover-fields-and-values).
|
|
90
|
-
|
|
91
|
-
## Send an HTTP request
|
|
92
|
-
|
|
93
|
-
Send the same query as a JSON request body:
|
|
94
|
-
|
|
95
|
-
```bash
|
|
96
|
-
curl --request POST \
|
|
97
|
-
--url http://localhost:4111/api/observability/traces/query \
|
|
98
|
-
--header 'Authorization: Bearer <token>' \
|
|
99
|
-
--header 'Content-Type: application/json' \
|
|
100
|
-
--data '{
|
|
101
|
-
"timeRange": {
|
|
102
|
-
"from": "2026-08-01T00:00:00.000Z",
|
|
103
|
-
"to": "2026-08-08T00:00:00.000Z"
|
|
104
|
-
},
|
|
105
|
-
"where": {
|
|
106
|
-
"spans": {
|
|
107
|
-
"some": {
|
|
108
|
-
"op": "and",
|
|
109
|
-
"args": [
|
|
110
|
-
{ "op": "eq", "left": { "path": "spanType" }, "right": { "literal": "tool_call" } },
|
|
111
|
-
{ "op": "exists", "path": "error" }
|
|
112
|
-
]
|
|
113
|
-
}
|
|
114
|
-
}
|
|
115
|
-
},
|
|
116
|
-
"orderBy": [{ "field": "startedAt", "direction": "desc" }],
|
|
117
|
-
"page": { "limit": 25 }
|
|
118
|
-
}'
|
|
119
|
-
```
|
|
120
|
-
|
|
121
|
-
## Request fields
|
|
122
|
-
|
|
123
|
-
| Field | Required | Description |
|
|
124
|
-
| ------------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
125
|
-
| `timeRange` | Yes | Trace start-time boundary. `from` is inclusive and `to` is exclusive. Both values must be ISO timestamps, and `from` must be earlier than `to`. |
|
|
126
|
-
| `where` | No | Recursive trace predicate. Supports scalar conditions and `spans.some` or `spans.none` clauses. |
|
|
127
|
-
| `orderBy` | No | One item ordering results by `startedAt` or `endedAt`, in `asc` or `desc` order. Defaults to `startedAt desc`. Not accepted in delta mode. |
|
|
128
|
-
| `page` | No | `{ limit, after }`. `limit` defaults to 100 and has a maximum of 1000. Pass the opaque `page.next` value as `after`. |
|
|
129
|
-
| `pagination` | No | Numbered pages: `{ page, perPage }`. Defaults to page 0 and 10 results. `perPage` has a maximum of 100. |
|
|
130
|
-
| `mode` | No | Set to `'delta'` to poll using a delta cursor instead of `page` or `pagination`. |
|
|
131
|
-
| `after` | No | Delta mode only. Opaque `deltaCursor` from a numbered page or previous poll. Omit to establish the current watermark. |
|
|
132
|
-
| `limit` | No | Delta mode only. Maximum results per batch. Defaults to 10 and has a maximum of 100. |
|
|
133
|
-
|
|
134
|
-
Unknown fields are rejected. Requests can't select a projection, declare joins, request counts or measures, or control authorization. Request bodies are limited to 256 KiB.
|
|
135
|
-
|
|
136
|
-
## Query limits
|
|
137
|
-
|
|
138
|
-
The planner rejects a query before storage execution when it exceeds any of these limits:
|
|
139
|
-
|
|
140
|
-
| Input | Maximum |
|
|
141
|
-
| ----------------------------------------- | ----------------- |
|
|
142
|
-
| Predicate nesting depth | 12 levels |
|
|
143
|
-
| Predicate nodes | 100 |
|
|
144
|
-
| Related `spans` clauses | 8 total |
|
|
145
|
-
| Values in one `in` or `notIn` set | 100 |
|
|
146
|
-
| Comparison literals and membership values | 1,000 total |
|
|
147
|
-
| String literal | 4,096 UTF-8 bytes |
|
|
148
|
-
| Raw predicate path | 128 UTF-8 bytes |
|
|
149
|
-
| `page.limit` | 1,000 |
|
|
150
|
-
| HTTP request body | 256 KiB |
|
|
151
|
-
|
|
152
|
-
Each comparison literal counts as one literal. Each member of an `in` or `notIn` set also counts as one literal, even when the set is within its per-set limit.
|
|
153
|
-
|
|
154
|
-
Hono and Fastify enforce the request-body limit before JSON parsing. Express and Elysia reject an oversized body before the route handler or storage runs, but the host framework may already have parsed a request without a reliable `Content-Length` header. Configure the host application's JSON parser or body limit when you need a hard pre-parse memory ceiling.
|
|
155
|
-
|
|
156
|
-
## Fields and operators
|
|
74
|
+
### Fields and operators
|
|
157
75
|
|
|
158
76
|
| Predicate context | Fields | Operators |
|
|
159
77
|
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------- |
|
|
160
|
-
| Trace | `traceId`, `threadId`, `resourceId`, `runId`, `sessionId`, `userId`, `organizationId`, `entityName`, `entityType`, `environment`, `status`
|
|
78
|
+
| Trace | `traceId`, `threadId`, `resourceId`, `runId`, `sessionId`, `userId`, `organizationId`, `entityName`, `entityType`, `name`, `spanType`, `environment`, `status` | `eq`, `ne`, `in`, `notIn`, `exists`, `notExists` |
|
|
161
79
|
| Trace metadata | `metadata.<key>` | `eq`, `ne`, `in`, `notIn`, `exists`, `notExists` |
|
|
162
|
-
| Trace | `startedAt`, `endedAt`, `durationMs`
|
|
80
|
+
| Trace | `startedAt`, `endedAt`, `durationMs`, `timeToFirstChunkMs` | `eq`, `ne`, `in`, `notIn`, `lt`, `lte`, `gt`, `gte`, `exists`, `notExists` |
|
|
163
81
|
| Trace | `tags` | `includes`, `notIncludes`, `exists`, `notExists` |
|
|
164
82
|
| Span | `name` | `eq`, `ne`, `in`, `notIn`, `exists`, `notExists`, `matches`, `notMatches` |
|
|
165
83
|
| Span | `spanType`, `model`, `provider`, `status`, `entityType`, `entityId`, `entityName`, `entityVersionId`, `parentEntityVersionId`, `rootEntityVersionId`, `runId`, `sessionId`, `userId`, `organizationId` | `eq`, `ne`, `in`, `notIn`, `exists`, `notExists` |
|
|
166
|
-
| Span | `startedAt`, `endedAt`, `durationMs`
|
|
84
|
+
| Span | `startedAt`, `endedAt`, `durationMs`, `timeToFirstChunkMs` | `eq`, `ne`, `in`, `notIn`, `lt`, `lte`, `gt`, `gte`, `exists`, `notExists` |
|
|
167
85
|
| Span | `error` | `exists`, `notExists` |
|
|
168
86
|
|
|
87
|
+
Trace-scope `name` and `spanType` read the current root span, so `{ op: 'eq', left: { path: 'spanType' }, right: { literal: 'workflow_run' } }` selects traces started by a workflow run. Inside `spans.some` or `spans.none`, the same fields match any span in the trace.
|
|
88
|
+
|
|
169
89
|
Compose predicates with `{ op: 'and', args: [...] }`, `{ op: 'or', args: [...] }`, and `{ op: 'not', arg: ... }`. Comparison predicates place a field reference on the left and a literal on the right. Membership predicates use a field reference in `value` and a homogeneous literal array in `set`. Tag predicates name the stored tag collection in `path`. `includes` and `notIncludes` also take one tag in `value`, while `exists` and `notExists` take only `op` and `path`. See [Filter by tags](#filter-by-tags). Text predicates use the comparison shape with a string literal. See [Filter by text](#filter-by-text).
|
|
170
90
|
|
|
171
|
-
String comparisons are case-sensitive, and literals are never coerced. `matches` and `notMatches` are the exception: they compare words case-insensitively. Canonical string fields compare exact stored values. Metadata string values are trimmed before comparison, as described below. Duration predicates require numbers, while timestamp predicates require ISO timestamp strings. A missing value satisfies neither positive nor ordered predicates, although it does satisfy the negative operators `ne` and `notIn`. Combine a negative predicate with `exists` when the field must also be present.
|
|
91
|
+
String comparisons are case-sensitive, and literals are never coerced. `matches` and `notMatches` are the exception: they compare words case-insensitively. Canonical string fields compare exact stored values. Metadata string values are trimmed before comparison, as described below. Duration and time-to-first-chunk predicates require numbers, while timestamp predicates require ISO timestamp strings. A missing value satisfies neither positive nor ordered predicates, although it does satisfy the negative operators `ne` and `notIn`. Combine a negative predicate with `exists` when the field must also be present.
|
|
172
92
|
|
|
173
|
-
|
|
93
|
+
#### Filter by root duration
|
|
174
94
|
|
|
175
95
|
At trace scope, `durationMs` is the current completed root span's elapsed time in milliseconds. The value is derived from `endedAt - startedAt` when the query runs:
|
|
176
96
|
|
|
@@ -186,7 +106,31 @@ This top-level predicate doesn't inspect child spans. In contrast, `spans.some`
|
|
|
186
106
|
|
|
187
107
|
Trace queries exclude incomplete roots before evaluating predicates. As a result, `{ op: 'notExists', path: 'durationMs' }` doesn't find running traces or malformed roots without a usable `endedAt`.
|
|
188
108
|
|
|
189
|
-
|
|
109
|
+
#### Filter by time to first chunk
|
|
110
|
+
|
|
111
|
+
`timeToFirstChunkMs` is the time in milliseconds from a span's start to the first chunk of its model response. Agent run and model spans record it when the model streams a response. At trace scope, the field reads the current root span's value. Inside `spans.some` or `spans.none`, it reads each related span.
|
|
112
|
+
|
|
113
|
+
This predicate finds traces with a model call that took more than 2 seconds to stream its first chunk:
|
|
114
|
+
|
|
115
|
+
```typescript
|
|
116
|
+
const slowFirstChunk = {
|
|
117
|
+
spans: {
|
|
118
|
+
some: {
|
|
119
|
+
op: 'and',
|
|
120
|
+
args: [
|
|
121
|
+
{ op: 'eq', left: { path: 'spanType' }, right: { literal: 'model_generation' } },
|
|
122
|
+
{ op: 'gt', left: { path: 'timeToFirstChunkMs' }, right: { literal: 2000 } },
|
|
123
|
+
],
|
|
124
|
+
},
|
|
125
|
+
},
|
|
126
|
+
}
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
The value is `null` when the span didn't stream a chunk, for example a tool call or a workflow run. A `null` value counts as missing, so it matches `notExists` and never matches ordered predicates. A value of `0` isn't missing. It means the first chunk arrived in the same millisecond the span started.
|
|
130
|
+
|
|
131
|
+
Trace query results include the root span's `timeToFirstChunkMs`, and `querySpans()` results include each span's value. Sorting by this field isn't supported. Stores advertise support through the `trace-query-time-to-first-chunk` feature, and the server reports it as the `traceQueryTimeToFirstChunk` capability. Against an older store, results omit the field, a predicate on it returns `501` before storage runs, and field discovery omits it.
|
|
132
|
+
|
|
133
|
+
#### Filter by context identifiers
|
|
190
134
|
|
|
191
135
|
`runId`, `sessionId`, `userId`, and `organizationId` are available at trace scope and inside `spans.some` / `spans.none`. Trace-scope predicates read the current root span. Span-scope predicates bind to one related current span, so a run recorded only on a child span is found through `spans.some`:
|
|
192
136
|
|
|
@@ -207,7 +151,7 @@ const sessionTracesWithRun = {
|
|
|
207
151
|
|
|
208
152
|
These identifiers support `eq`, `ne`, `in`, `notIn`, `exists`, and `notExists`. Field discovery lists them, but value discovery doesn't return their recorded values. Stores advertise support through the `trace-query-context-ids` feature. A predicate on one of these fields against an older store returns `501` before storage runs, and field discovery omits them for that store.
|
|
209
153
|
|
|
210
|
-
|
|
154
|
+
#### Filter by span properties
|
|
211
155
|
|
|
212
156
|
Every condition inside one `spans.some` or `spans.none` clause applies to the same current span. For example, this predicate finds a failed tool span whose name is `medication_lookup`; a matching name on one span and an error on another don't satisfy it:
|
|
213
157
|
|
|
@@ -287,7 +231,7 @@ const versionedRetrieval = {
|
|
|
287
231
|
}
|
|
288
232
|
```
|
|
289
233
|
|
|
290
|
-
|
|
234
|
+
#### Filter by metadata
|
|
291
235
|
|
|
292
236
|
Use `metadata.<key>` paths for custom dimensions stored on the current root span. Metadata predicates compare trimmed, non-empty string values exactly and case-sensitively. Leading and trailing whitespace in a stored string value is ignored. Values empty after trimming count as missing. Null, numeric, boolean, object, and array values also count as missing.
|
|
293
237
|
|
|
@@ -314,7 +258,7 @@ The key must name one top-level property. Empty keys and nested paths are reject
|
|
|
314
258
|
|
|
315
259
|
Trace-query discovery returns executable top-level string metadata fields observed in the selected time range.
|
|
316
260
|
|
|
317
|
-
|
|
261
|
+
#### Filter by tags
|
|
318
262
|
|
|
319
263
|
Tags are a list of strings stored on the current root span, so they use collection operators instead of the scalar `in` and `notIn` operators. This query finds production traces tagged for manual review that don't carry the `archived` tag:
|
|
320
264
|
|
|
@@ -340,7 +284,7 @@ Tags compare exactly and case-sensitively as whole strings. `value` must be one
|
|
|
340
284
|
|
|
341
285
|
`tags` is available in trace predicates only. `in`, `notIn`, and the comparison operators are rejected for `tags`. Value discovery for `tags` returns each observed tag with the number of qualifying traces that carry it.
|
|
342
286
|
|
|
343
|
-
|
|
287
|
+
#### Filter by text
|
|
344
288
|
|
|
345
289
|
Span names support word matching in addition to exact comparison. `matches` succeeds when the literal's words appear in the field, in order, as whole words. `notMatches` succeeds when the field exists but doesn't contain those words. This query finds traces with a span whose name contains the words "document search":
|
|
346
290
|
|
|
@@ -365,7 +309,148 @@ Both the literal and the stored value are lowercased and split into words. A wor
|
|
|
365
309
|
|
|
366
310
|
The literal must contain at least one letter or digit; otherwise the request fails with `422 TRACE_QUERY_INVALID`. A missing field satisfies neither operator. Use `{ op: 'not', arg: { op: 'matches', ... } }` when records without the field should also match. Text operators are available on span `name`. Other string fields, numbers, timestamps, tags, and metadata reject them. Substring, wildcard, regular-expression, and fuzzy matching aren't supported, and there is no search across several fields.
|
|
367
311
|
|
|
368
|
-
|
|
312
|
+
### Query limits
|
|
313
|
+
|
|
314
|
+
The planner rejects a query before storage execution when it exceeds any of these limits:
|
|
315
|
+
|
|
316
|
+
| Input | Maximum |
|
|
317
|
+
| ----------------------------------------- | ----------------- |
|
|
318
|
+
| Predicate nesting depth | 12 levels |
|
|
319
|
+
| Predicate nodes | 100 |
|
|
320
|
+
| Related `spans` clauses | 8 total |
|
|
321
|
+
| Values in one `in` or `notIn` set | 100 |
|
|
322
|
+
| Comparison literals and membership values | 1,000 total |
|
|
323
|
+
| String literal | 4,096 UTF-8 bytes |
|
|
324
|
+
| Raw predicate path | 128 UTF-8 bytes |
|
|
325
|
+
| `page.limit` | 1,000 |
|
|
326
|
+
| HTTP request body | 256 KiB |
|
|
327
|
+
|
|
328
|
+
Each comparison literal counts as one literal. Each member of an `in` or `notIn` set also counts as one literal, even when the set is within its per-set limit.
|
|
329
|
+
|
|
330
|
+
Hono and Fastify enforce the request-body limit before JSON parsing. Express and Elysia reject an oversized body before the route handler or storage runs, but the host framework may already have parsed a request without a reliable `Content-Length` header. Configure the host application's JSON parser or body limit when you need a hard pre-parse memory ceiling.
|
|
331
|
+
|
|
332
|
+
## Query traces
|
|
333
|
+
|
|
334
|
+
Use `queryTraces()`, `mastra api trace query`, or `POST /api/observability/traces/query` to list the selected traces. All three accept the same request, which adds ordering and pagination to the [trace selection](#select-traces).
|
|
335
|
+
|
|
336
|
+
### Query traces with the client SDK
|
|
337
|
+
|
|
338
|
+
This query finds traces containing a tool call with an error:
|
|
339
|
+
|
|
340
|
+
```typescript
|
|
341
|
+
import { MastraClient } from '@mastra/client-js'
|
|
342
|
+
|
|
343
|
+
const mastraClient = new MastraClient({ baseUrl: 'http://localhost:4111' })
|
|
344
|
+
|
|
345
|
+
const result = await mastraClient.queryTraces({
|
|
346
|
+
timeRange: {
|
|
347
|
+
from: '2026-08-01T00:00:00.000Z',
|
|
348
|
+
to: '2026-08-08T00:00:00.000Z',
|
|
349
|
+
},
|
|
350
|
+
where: {
|
|
351
|
+
spans: {
|
|
352
|
+
some: {
|
|
353
|
+
op: 'and',
|
|
354
|
+
args: [
|
|
355
|
+
{ op: 'eq', left: { path: 'spanType' }, right: { literal: 'tool_call' } },
|
|
356
|
+
{ op: 'exists', path: 'error' },
|
|
357
|
+
],
|
|
358
|
+
},
|
|
359
|
+
},
|
|
360
|
+
},
|
|
361
|
+
orderBy: [{ field: 'startedAt', direction: 'desc' }],
|
|
362
|
+
page: { limit: 25 },
|
|
363
|
+
})
|
|
364
|
+
```
|
|
365
|
+
|
|
366
|
+
### Query traces with the CLI
|
|
367
|
+
|
|
368
|
+
Pass the query as one JSON argument to `mastra api trace query`. Filters and pagination options belong in the JSON input. This command runs the same failed-tool query as the SDK example:
|
|
369
|
+
|
|
370
|
+
```bash
|
|
371
|
+
mastra api --url http://localhost:4111 trace query '{
|
|
372
|
+
"timeRange": {
|
|
373
|
+
"from": "2026-08-01T00:00:00.000Z",
|
|
374
|
+
"to": "2026-08-08T00:00:00.000Z"
|
|
375
|
+
},
|
|
376
|
+
"where": {
|
|
377
|
+
"spans": {
|
|
378
|
+
"some": {
|
|
379
|
+
"op": "and",
|
|
380
|
+
"args": [
|
|
381
|
+
{ "op": "eq", "left": { "path": "spanType" }, "right": { "literal": "tool_call" } },
|
|
382
|
+
{ "op": "exists", "path": "error" }
|
|
383
|
+
]
|
|
384
|
+
}
|
|
385
|
+
}
|
|
386
|
+
},
|
|
387
|
+
"orderBy": [{ "field": "startedAt", "direction": "desc" }],
|
|
388
|
+
"page": { "limit": 25 }
|
|
389
|
+
}'
|
|
390
|
+
```
|
|
391
|
+
|
|
392
|
+
The explicit `--url` targets a local Mastra server. Without it, observability commands default to `https://observability.mastra.ai`, even when a local server is running. See [CLI target resolution](https://mastra.ai/reference/cli/mastra) for authentication and regional endpoints. The target server and configured store must support the requested query capabilities.
|
|
393
|
+
|
|
394
|
+
Inspect the command with:
|
|
395
|
+
|
|
396
|
+
```bash
|
|
397
|
+
mastra api trace query --help
|
|
398
|
+
```
|
|
399
|
+
|
|
400
|
+
For a local Mastra server, inspect its request and response schemas with:
|
|
401
|
+
|
|
402
|
+
```bash
|
|
403
|
+
mastra api --url http://localhost:4111 trace query --schema
|
|
404
|
+
```
|
|
405
|
+
|
|
406
|
+
Mastra Platform doesn't currently expose the schema endpoint used by `--schema`. Use [Request fields](#request-fields) and [Fields and operators](#fields-and-operators) to construct queries for either target. To find fields and values recorded in your project, use [field and value discovery](#discover-fields-and-values).
|
|
407
|
+
|
|
408
|
+
### Send an HTTP request
|
|
409
|
+
|
|
410
|
+
Send the same query as a JSON request body:
|
|
411
|
+
|
|
412
|
+
```bash
|
|
413
|
+
curl --request POST \
|
|
414
|
+
--url http://localhost:4111/api/observability/traces/query \
|
|
415
|
+
--header 'Authorization: Bearer <token>' \
|
|
416
|
+
--header 'Content-Type: application/json' \
|
|
417
|
+
--data '{
|
|
418
|
+
"timeRange": {
|
|
419
|
+
"from": "2026-08-01T00:00:00.000Z",
|
|
420
|
+
"to": "2026-08-08T00:00:00.000Z"
|
|
421
|
+
},
|
|
422
|
+
"where": {
|
|
423
|
+
"spans": {
|
|
424
|
+
"some": {
|
|
425
|
+
"op": "and",
|
|
426
|
+
"args": [
|
|
427
|
+
{ "op": "eq", "left": { "path": "spanType" }, "right": { "literal": "tool_call" } },
|
|
428
|
+
{ "op": "exists", "path": "error" }
|
|
429
|
+
]
|
|
430
|
+
}
|
|
431
|
+
}
|
|
432
|
+
},
|
|
433
|
+
"orderBy": [{ "field": "startedAt", "direction": "desc" }],
|
|
434
|
+
"page": { "limit": 25 }
|
|
435
|
+
}'
|
|
436
|
+
```
|
|
437
|
+
|
|
438
|
+
### Request fields
|
|
439
|
+
|
|
440
|
+
| Field | Required | Description |
|
|
441
|
+
| ------------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
442
|
+
| `timeRange` | Yes | Trace start-time boundary. `from` is inclusive and `to` is exclusive. Both values must be ISO timestamps, and `from` must be earlier than `to`. |
|
|
443
|
+
| `where` | No | Recursive trace predicate. Supports scalar conditions and `spans.some` or `spans.none` clauses. |
|
|
444
|
+
| `orderBy` | No | One item ordering results by `startedAt` or `endedAt`, in `asc` or `desc` order. Defaults to `startedAt desc`. Not accepted in delta mode. |
|
|
445
|
+
| `page` | No | `{ limit, after }`. `limit` defaults to 100 and has a maximum of 1000. Pass the opaque `page.next` value as `after`. |
|
|
446
|
+
| `pagination` | No | Numbered pages: `{ page, perPage }`. Defaults to page 0 and 10 results. `perPage` has a maximum of 100. |
|
|
447
|
+
| `mode` | No | Set to `'delta'` to poll using a delta cursor instead of `page` or `pagination`. |
|
|
448
|
+
| `after` | No | Delta mode only. Opaque `deltaCursor` from a numbered page or previous poll. Omit to establish the current watermark. |
|
|
449
|
+
| `limit` | No | Delta mode only. Maximum results per batch. Defaults to 10 and has a maximum of 100. |
|
|
450
|
+
|
|
451
|
+
Unknown fields are rejected. Requests can't select a projection, declare joins, request counts or measures, or control authorization. Use [`aggregateTraces()`](#aggregate-traces) for counts and measures. Request bodies are limited to 256 KiB.
|
|
452
|
+
|
|
453
|
+
### Responses
|
|
369
454
|
|
|
370
455
|
A query using keyset pagination returns lightweight completed traces and a cursor for the next page:
|
|
371
456
|
|
|
@@ -382,16 +467,19 @@ A query using keyset pagination returns lightweight completed traces and a curso
|
|
|
382
467
|
"entityName": "support-agent",
|
|
383
468
|
"entityType": "agent",
|
|
384
469
|
"environment": "production",
|
|
385
|
-
"status": "success"
|
|
470
|
+
"status": "success",
|
|
471
|
+
"timeToFirstChunkMs": 412
|
|
386
472
|
}
|
|
387
473
|
],
|
|
388
474
|
"page": { "next": null }
|
|
389
475
|
}
|
|
390
476
|
```
|
|
391
477
|
|
|
478
|
+
`timeToFirstChunkMs` is present only when the configured store supports it. See [Filter by time to first chunk](#filter-by-time-to-first-chunk).
|
|
479
|
+
|
|
392
480
|
The response doesn't include the related spans that matched the predicate. Use [`getTrace()`](https://mastra.ai/reference/client-js/observability) to load the full trace after selecting a result.
|
|
393
481
|
|
|
394
|
-
|
|
482
|
+
#### CLI responses
|
|
395
483
|
|
|
396
484
|
The CLI wraps the API response in `data`:
|
|
397
485
|
|
|
@@ -410,9 +498,9 @@ Each CLI invocation makes one query. To fetch another keyset page, pass `data.pa
|
|
|
410
498
|
|
|
411
499
|
To inspect a selected trace, use `mastra api --url http://localhost:4111 trace get <traceId> --verbose`. Use the same target and authentication as the query.
|
|
412
500
|
|
|
413
|
-
|
|
501
|
+
### Pagination and errors
|
|
414
502
|
|
|
415
|
-
|
|
503
|
+
#### Choose a pagination mode
|
|
416
504
|
|
|
417
505
|
Trace queries support three mutually exclusive modes:
|
|
418
506
|
|
|
@@ -424,7 +512,7 @@ Trace queries support three mutually exclusive modes:
|
|
|
424
512
|
|
|
425
513
|
Numbered pages are zero-based. Their total and rows are read consistently within each request. Don't combine `page`, `pagination`, or delta mode. Top-level `after` and `limit` are accepted only in delta mode.
|
|
426
514
|
|
|
427
|
-
|
|
515
|
+
#### Poll after loading a numbered page
|
|
428
516
|
|
|
429
517
|
Use the numbered response's `deltaCursor` to replace the numbered-page-to-delta-polling workflow of `listTracesLight()`. Results use `traces` instead of `spans` and contain completed traces only. The configured store must support delta polling.
|
|
430
518
|
|
|
@@ -474,7 +562,7 @@ ClickHouse polling is best effort. Its delta index and trace records use separat
|
|
|
474
562
|
|
|
475
563
|
ClickHouse's delta index retains two days of events and doesn't backfill historical rows. Reload numbered pages after a polling interruption longer than this retention window. Delta polling isn't a durable change feed and doesn't guarantee delivery of every matching trace.
|
|
476
564
|
|
|
477
|
-
|
|
565
|
+
#### Keyset ordering and errors
|
|
478
566
|
|
|
479
567
|
Ordering is deterministic. Trace ordering appends `traceId` ascending as a tie-breaker.
|
|
480
568
|
|
|
@@ -482,20 +570,6 @@ Keyset cursors are bound to the operation, accepted normalized query, and orderi
|
|
|
482
570
|
|
|
483
571
|
Numbered-page handoff cursors and delta cursors also bind the authorization state. Changes to the caller's roles or permissions invalidate these cursors and return `409`. If a delta poll returns `409`, reload the numbered pages and resume polling with the new `deltaCursor`. A malformed cursor returns `400`.
|
|
484
572
|
|
|
485
|
-
### Trusted tenant scope
|
|
486
|
-
|
|
487
|
-
Hosts that serve more than one tenant supply a trusted scope outside the query document. The server reads the reserved `organizationId` request-context key, which only server-side authentication can set, and passes `{ organizationId }` to the planner. The scope is carried on the trusted plan and ANDed into every scan: current roots, related spans and both discovery routes. A related record from another tenant that shares a `traceId` never qualifies a trace.
|
|
488
|
-
|
|
489
|
-
Hosts that call storage directly can pass a fuller scope to `planTraceQuery()`, `planTraceQueryObservedFields()`, and `planTraceQueryValues()`:
|
|
490
|
-
|
|
491
|
-
```typescript
|
|
492
|
-
const plan = planTraceQuery(parseTraceQueryRequest(request), {
|
|
493
|
-
scope: { organizationId: 'org_123', resourceId: 'project_456' },
|
|
494
|
-
})
|
|
495
|
-
```
|
|
496
|
-
|
|
497
|
-
`resourceId` is optional and narrows within the organization. Keyset and delta cursors are bound to the scope. A cursor created under one scope and reused under another returns `409` before storage runs. A scoped request against an observability store or `@mastra/core` that predates tenant scope returns `501` instead of running unscoped. Stores advertise support through the `trace-query-tenant-scope` feature. Callers can't name `projectId` in predicates. An `organizationId` predicate compares recorded data and is ANDed with the scope, so it can only narrow the tenant's own traces and never widens it. Without a scope the routes behave as before, so self-hosted installations keep their own tenant model.
|
|
498
|
-
|
|
499
573
|
Cursor pagination is deterministic, but it isn't a database snapshot. Traces or replacement signals written between page requests can change later pages.
|
|
500
574
|
|
|
501
575
|
PostgreSQL and ClickHouse stop trace queries after 15 seconds by default and return `504` when the database timeout is exceeded. Set `traceQueryTimeoutMs` in the store's vNext observability configuration to an integer from 1 through 300,000 milliseconds to change the timeout. DuckDB doesn't currently provide query-scoped timeout or cancellation through its driver wrapper, so this `504` guarantee doesn't apply to DuckDB.
|
|
@@ -510,6 +584,230 @@ PostgreSQL and ClickHouse stop trace queries after 15 seconds by default and ret
|
|
|
510
584
|
| `503` | Discovery exceeded a backend memory or resource budget and returned `TRACE_QUERY_RESOURCE_LIMIT`. |
|
|
511
585
|
| `504` | A PostgreSQL or ClickHouse query exceeded its configured database execution timeout. Discovery returns `TRACE_QUERY_EXECUTION_TIMEOUT`. |
|
|
512
586
|
|
|
587
|
+
## Aggregate traces
|
|
588
|
+
|
|
589
|
+
Use `aggregateTraces()` or `POST /api/observability/traces/aggregate` to count the selected traces and compute measures over them. Add `groupBy` to split the result by up to two dimensions, and `interval` to split each group into time buckets. The response contains one row per group, or one row per group and bucket.
|
|
590
|
+
|
|
591
|
+
### Aggregate traces with the client SDK
|
|
592
|
+
|
|
593
|
+
This request compares agents by volume, error rate, and latency over one week:
|
|
594
|
+
|
|
595
|
+
```typescript
|
|
596
|
+
import { MastraClient } from '@mastra/client-js'
|
|
597
|
+
|
|
598
|
+
const mastraClient = new MastraClient({ baseUrl: 'http://localhost:4111' })
|
|
599
|
+
|
|
600
|
+
const { capabilities } = await mastraClient.getObservabilityCapabilities()
|
|
601
|
+
|
|
602
|
+
if (capabilities.traceAggregate) {
|
|
603
|
+
const { rows, truncated } = await mastraClient.aggregateTraces({
|
|
604
|
+
timeRange: {
|
|
605
|
+
from: '2026-08-01T00:00:00.000Z',
|
|
606
|
+
to: '2026-08-08T00:00:00.000Z',
|
|
607
|
+
},
|
|
608
|
+
where: { op: 'eq', left: { path: 'entityType' }, right: { literal: 'agent' } },
|
|
609
|
+
groupBy: ['entityName'],
|
|
610
|
+
measures: ['count', 'errorRate', 'duration.p95'],
|
|
611
|
+
orderBy: { field: 'errorRate', direction: 'desc' },
|
|
612
|
+
limit: 10,
|
|
613
|
+
})
|
|
614
|
+
}
|
|
615
|
+
```
|
|
616
|
+
|
|
617
|
+
The client retries server errors like `queryTraces()`, but doesn't retry `422` validation errors.
|
|
618
|
+
|
|
619
|
+
### Send an aggregate HTTP request
|
|
620
|
+
|
|
621
|
+
Send the same request as a JSON body:
|
|
622
|
+
|
|
623
|
+
```bash
|
|
624
|
+
curl --request POST \
|
|
625
|
+
--url http://localhost:4111/api/observability/traces/aggregate \
|
|
626
|
+
--header 'Authorization: Bearer <token>' \
|
|
627
|
+
--header 'Content-Type: application/json' \
|
|
628
|
+
--data '{
|
|
629
|
+
"timeRange": {
|
|
630
|
+
"from": "2026-08-01T00:00:00.000Z",
|
|
631
|
+
"to": "2026-08-08T00:00:00.000Z"
|
|
632
|
+
},
|
|
633
|
+
"where": { "op": "eq", "left": { "path": "entityType" }, "right": { "literal": "agent" } },
|
|
634
|
+
"groupBy": ["entityName"],
|
|
635
|
+
"measures": ["count", "errorRate", "duration.p95"],
|
|
636
|
+
"orderBy": { "field": "errorRate", "direction": "desc" },
|
|
637
|
+
"limit": 10
|
|
638
|
+
}'
|
|
639
|
+
```
|
|
640
|
+
|
|
641
|
+
### Aggregate request fields
|
|
642
|
+
|
|
643
|
+
| Field | Required | Description |
|
|
644
|
+
| ----------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
645
|
+
| `timeRange` | Yes | Trace start-time boundary, as in [Select traces](#select-traces). The range can span up to 365 days. |
|
|
646
|
+
| `where` | No | Trace predicate, as in [Select traces](#select-traces). |
|
|
647
|
+
| `measures` | Yes | One or more distinct [measure names](#measures). |
|
|
648
|
+
| `groupBy` | No | Up to two distinct [dimensions](#dimensions). Defaults to `[]`, which returns one row for all selected traces. |
|
|
649
|
+
| `interval` | No | Bucket width: `'1m'`, `'5m'`, `'15m'`, `'1h'`, or `'1d'`. See [Time buckets](#time-buckets). |
|
|
650
|
+
| `having` | No | Scalar predicate over measures that removes groups after grouping. See [Filter and order groups](#filter-and-order-groups). |
|
|
651
|
+
| `orderBy` | No | `{ field, direction }`. `field` is a requested measure, `count`, or a `groupBy` dimension, and `direction` is `asc` or `desc`. Defaults to `count desc`. |
|
|
652
|
+
| `limit` | No | Maximum number of groups. Defaults to 100 and has a maximum of 1,000. |
|
|
653
|
+
|
|
654
|
+
Unknown fields are rejected. Aggregation isn't paginated. Narrow the selection, or rank by a different measure, when `truncated` is `true`.
|
|
655
|
+
|
|
656
|
+
### Dimensions
|
|
657
|
+
|
|
658
|
+
Dimensions read the current root span of each selected trace:
|
|
659
|
+
|
|
660
|
+
| Dimension | Notes |
|
|
661
|
+
| ------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
|
|
662
|
+
| `entityType`, `entityName`, `environment`, `status`, `serviceName`, `executionSource` | Root span attributes. |
|
|
663
|
+
| `threadId`, `resourceId`, `userId`, `sessionId`, `organizationId`, `experimentId` | Context identifiers recorded on the root span. |
|
|
664
|
+
| `name`, `spanType` | Root span name and type. Requires the `traceQueryRootSpanFields` capability. |
|
|
665
|
+
| `metadata.<key>` | One non-empty top-level metadata key. Values are trimmed. Missing, empty, and non-string values group as `null`. |
|
|
666
|
+
|
|
667
|
+
Every dimension except `serviceName`, `executionSource`, and `experimentId` is also a trace-scope `where` field, so you can drill into a group by adding its values to the predicate. `traceId`, `spanId`, `runId`, and `requestId` aren't groupable. Traces with no value for a dimension form their own group with a `null` value.
|
|
668
|
+
|
|
669
|
+
### Measures
|
|
670
|
+
|
|
671
|
+
| Measure | Value |
|
|
672
|
+
| -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
|
|
673
|
+
| `count` | Number of selected traces. |
|
|
674
|
+
| `countDistinct.<field>` | Number of distinct non-null values of a dimension or `traceId`, such as `countDistinct.userId`. |
|
|
675
|
+
| `duration.avg`, `duration.min`, `duration.max` | Root span duration in milliseconds. |
|
|
676
|
+
| `duration.p50`, `duration.p90`, `duration.p95`, `duration.p99` | Root span duration percentile in milliseconds. Percentiles can be approximate, depending on the store. |
|
|
677
|
+
| `errorCount` | Number of traces whose root span recorded an error. |
|
|
678
|
+
| `errorRate` | `errorCount` divided by `count`, from 0 to 1. |
|
|
679
|
+
| `tokens.input.sum`, `tokens.input.avg` | Input tokens. |
|
|
680
|
+
| `tokens.output.sum`, `tokens.output.avg` | Output tokens. |
|
|
681
|
+
| `tokens.total.sum`, `tokens.total.avg` | Input plus output tokens. |
|
|
682
|
+
| `tokens.reasoning.sum`, `tokens.reasoning.avg` | Reasoning output tokens. |
|
|
683
|
+
| `tokens.cached.sum`, `tokens.cached.avg` | Input tokens read from the provider's cache. |
|
|
684
|
+
| `cost.sum`, `cost.avg` | Estimated model cost. |
|
|
685
|
+
|
|
686
|
+
Numeric measures over metadata aren't supported.
|
|
687
|
+
|
|
688
|
+
#### Token and cost measures
|
|
689
|
+
|
|
690
|
+
Token and cost measures read the model usage metrics recorded for each trace, so they're only available when the store retains metrics for the selected time range. A metric belongs to a trace when it shares the trace's `traceId` and was recorded at or after `timeRange.from`. Every such metric counts, including usage recorded after `timeRange.to` and usage recorded before a suspend and resume. Retried metric exports are counted once.
|
|
691
|
+
|
|
692
|
+
A trace that recorded at least one usage metric is usage-bearing. `tokens.<kind>.sum` adds the tokens of a group's usage-bearing traces, and `tokens.<kind>.avg` divides that sum by the number of usage-bearing traces, not by `count`. Both are `null` when the group has no usage-bearing traces.
|
|
693
|
+
|
|
694
|
+
Cost comes from estimated cost recorded on total input and total output token metrics. `cost.sum` adds the cost of traces with at least one priced metric, and `cost.avg` divides by the number of those traces. A metric whose pricing failed, including a partial cost, isn't priced.
|
|
695
|
+
|
|
696
|
+
When you request any `cost.*` measure, every row also includes a `cost` object:
|
|
697
|
+
|
|
698
|
+
| Field | Description |
|
|
699
|
+
| ---------- | ------------------------------------------------------------------------------------------------------------------------------ |
|
|
700
|
+
| `coverage` | Fraction of usage-bearing traces whose usage was fully priced, from 0 to 1. `null` when the group has no usage-bearing traces. |
|
|
701
|
+
| `unit` | Cost unit of the priced metrics, such as `USD`. `'mixed'` when they use more than one unit, and `null` when none are priced. |
|
|
702
|
+
|
|
703
|
+
`cost.sum` and `cost.avg` are `null` when the group has no priced traces or `unit` is `'mixed'`. A coverage below 1 means the cost understates the group's spend. The `cost` object isn't a measure, so `having` and `orderBy` can't reference it.
|
|
704
|
+
|
|
705
|
+
### Filter and order groups
|
|
706
|
+
|
|
707
|
+
A group is one distinct combination of `groupBy` values. `having`, `orderBy`, and `limit` operate on groups and use each group's measures over the whole time range, even when `interval` is set. The server applies `having` first, then orders the remaining groups and keeps the first `limit`. `truncated` is `true` when more groups remained than `limit`.
|
|
708
|
+
|
|
709
|
+
`having` uses the scalar predicate shape with a measure name as the path, such as `{ op: 'gt', left: { path: 'duration.p95' }, right: { literal: 5000 } }`. It can reference requested measures and `count`, even when `count` isn't requested. It supports comparison operators, `in`, and `notIn` with number literals, combined with `and`, `or`, and `not`. Presence, tag, and text operators aren't supported. A `null` measure never satisfies a `having` comparison, so the group is removed.
|
|
710
|
+
|
|
711
|
+
This request keeps workflows with a 95th-percentile duration above five seconds:
|
|
712
|
+
|
|
713
|
+
```typescript
|
|
714
|
+
const slowWorkflows = await mastraClient.aggregateTraces({
|
|
715
|
+
timeRange: {
|
|
716
|
+
from: '2026-08-01T00:00:00.000Z',
|
|
717
|
+
to: '2026-08-08T00:00:00.000Z',
|
|
718
|
+
},
|
|
719
|
+
where: { op: 'eq', left: { path: 'entityType' }, right: { literal: 'workflow_run' } },
|
|
720
|
+
groupBy: ['entityName'],
|
|
721
|
+
measures: ['count', 'duration.p95', 'duration.p99'],
|
|
722
|
+
having: { op: 'gt', left: { path: 'duration.p95' }, right: { literal: 5000 } },
|
|
723
|
+
orderBy: { field: 'duration.p95', direction: 'desc' },
|
|
724
|
+
})
|
|
725
|
+
```
|
|
726
|
+
|
|
727
|
+
Ordering is deterministic. Ties are broken by dimension values in ascending order. `null` measure and dimension values sort last in both directions.
|
|
728
|
+
|
|
729
|
+
### Time buckets
|
|
730
|
+
|
|
731
|
+
With `interval`, each group that survives `having` and `limit` expands into one row per non-empty bucket. Buckets are aligned to UTC, so `1d` buckets start at midnight UTC, and each row's `bucket` is the ISO start time of its bucket. A trace belongs to the bucket containing its root `startedAt`. Rows are ordered by group, then by `bucket` ascending.
|
|
732
|
+
|
|
733
|
+
Buckets without traces are omitted rather than returned as zero rows. Fill gaps on the client when you draw a chart. Because groups are ranked over the whole time range first, every returned group includes its full series of non-empty buckets.
|
|
734
|
+
|
|
735
|
+
This request returns daily token usage and cost for each agent:
|
|
736
|
+
|
|
737
|
+
```typescript
|
|
738
|
+
const dailyCost = await mastraClient.aggregateTraces({
|
|
739
|
+
timeRange: {
|
|
740
|
+
from: '2026-08-01T00:00:00.000Z',
|
|
741
|
+
to: '2026-08-08T00:00:00.000Z',
|
|
742
|
+
},
|
|
743
|
+
where: { op: 'eq', left: { path: 'entityType' }, right: { literal: 'agent' } },
|
|
744
|
+
groupBy: ['entityName'],
|
|
745
|
+
interval: '1d',
|
|
746
|
+
measures: ['count', 'tokens.total.sum', 'cost.sum', 'cost.avg'],
|
|
747
|
+
orderBy: { field: 'cost.sum', direction: 'desc' },
|
|
748
|
+
limit: 5,
|
|
749
|
+
})
|
|
750
|
+
```
|
|
751
|
+
|
|
752
|
+
### Aggregate responses
|
|
753
|
+
|
|
754
|
+
```json
|
|
755
|
+
{
|
|
756
|
+
"rows": [
|
|
757
|
+
{
|
|
758
|
+
"dimensions": { "entityName": "supportAgent" },
|
|
759
|
+
"bucket": "2026-08-01T00:00:00.000Z",
|
|
760
|
+
"measures": {
|
|
761
|
+
"count": 412,
|
|
762
|
+
"tokens.total.sum": 1830442,
|
|
763
|
+
"cost.sum": 4.21,
|
|
764
|
+
"cost.avg": 0.0103
|
|
765
|
+
},
|
|
766
|
+
"cost": { "coverage": 0.99, "unit": "USD" }
|
|
767
|
+
}
|
|
768
|
+
],
|
|
769
|
+
"truncated": false
|
|
770
|
+
}
|
|
771
|
+
```
|
|
772
|
+
|
|
773
|
+
`dimensions` is keyed by `groupBy` field and is present when `groupBy` isn't empty. `bucket` is present when `interval` is set. `measures` is keyed by the requested measure names. When no traces match, or `having` removes every group, `rows` is empty.
|
|
774
|
+
|
|
775
|
+
### Aggregate limits and errors
|
|
776
|
+
|
|
777
|
+
| Limit | Maximum |
|
|
778
|
+
| ------------------------- | ------------ |
|
|
779
|
+
| Time range | 365 days |
|
|
780
|
+
| `groupBy` dimensions | 2 |
|
|
781
|
+
| `limit` | 1,000 groups |
|
|
782
|
+
| Buckets in the time range | 1,000 |
|
|
783
|
+
| `limit` × buckets | 10,000 rows |
|
|
784
|
+
|
|
785
|
+
Predicate limits from [Query limits](#query-limits) apply to both `where` and `having`, and the request body is limited to 256 KiB. A time range that doesn't start on a bucket boundary touches one more bucket than its length divided by the interval. If the range has too many buckets, choose a wider `interval`. If `limit` × buckets exceeds 10,000, reduce `limit` or widen the interval.
|
|
786
|
+
|
|
787
|
+
PostgreSQL and ClickHouse stop aggregations after the same `traceQueryTimeoutMs` timeout as trace queries, which defaults to 15 seconds. The timeout doesn't apply to DuckDB.
|
|
788
|
+
|
|
789
|
+
| Status | Meaning |
|
|
790
|
+
| ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
791
|
+
| `400` | Malformed JSON. |
|
|
792
|
+
| `413` | The request body exceeds 256 KiB. |
|
|
793
|
+
| `422` | The request is invalid and returns `TRACE_QUERY_INVALID` with issues. Issue codes include `time_range_too_large`, `too_many_buckets`, `too_many_rows`, `field_not_allowed`, `invalid_metadata_key`, and `operator_not_allowed`. |
|
|
794
|
+
| `501` | The installed Core or configured observability store doesn't support aggregation or a requested field. Returns `TRACE_AGGREGATE_UNSUPPORTED`. |
|
|
795
|
+
| `504` | A PostgreSQL or ClickHouse aggregation exceeded its execution timeout. Returns `TRACE_QUERY_EXECUTION_TIMEOUT` without partial results. |
|
|
796
|
+
|
|
797
|
+
## Trusted tenant scope
|
|
798
|
+
|
|
799
|
+
Hosts that serve more than one tenant supply a trusted scope outside the query document. The server reads the reserved `organizationId` request-context key, which only server-side authentication can set, and passes `{ organizationId }` to the planner. The scope is carried on the trusted plan and ANDed into every scan: current roots, related spans, the usage metrics read by token and cost measures, and both discovery routes. A related record from another tenant that shares a `traceId` never qualifies a trace.
|
|
800
|
+
|
|
801
|
+
Hosts that call storage directly can pass a fuller scope to `planTraceQuery()`, `planTraceAggregate()`, `planTraceQueryObservedFields()`, and `planTraceQueryValues()`:
|
|
802
|
+
|
|
803
|
+
```typescript
|
|
804
|
+
const plan = planTraceQuery(parseTraceQueryRequest(request), {
|
|
805
|
+
scope: { organizationId: 'org_123', resourceId: 'project_456' },
|
|
806
|
+
})
|
|
807
|
+
```
|
|
808
|
+
|
|
809
|
+
`resourceId` is optional and narrows within the organization. Keyset and delta cursors are bound to the scope. A cursor created under one scope and reused under another returns `409` before storage runs. A scoped request against an observability store or `@mastra/core` that predates tenant scope returns `501` instead of running unscoped. Stores advertise support through the `trace-query-tenant-scope` feature. Callers can't name `projectId` in predicates. An `organizationId` predicate compares recorded data and is ANDed with the scope, so it can only narrow the tenant's own traces and never widens it. Without a scope the routes behave as before, so self-hosted installations keep their own tenant model.
|
|
810
|
+
|
|
513
811
|
## Discover fields and values
|
|
514
812
|
|
|
515
813
|
Use trace-query discovery to build field and value autocomplete without scanning trace payloads or copying the query grammar into your client. The configured observability store must support `trace-query-discovery`.
|
|
@@ -561,6 +859,27 @@ Canonical fields come from the same registry used to validate trace queries. The
|
|
|
561
859
|
|
|
562
860
|
A field picker can request the `trace` and `spans` scopes in parallel and label each field with its scope. Every returned path is directly accepted by `queryTraces()` in its matching scope. `metadata` by itself isn't a field. A user interface can offer manual metadata-key entry separately.
|
|
563
861
|
|
|
862
|
+
For the `trace` scope, when the server and store support aggregation, the fields response also includes an `aggregate` object that describes what `aggregateTraces()` accepts:
|
|
863
|
+
|
|
864
|
+
```json
|
|
865
|
+
{
|
|
866
|
+
"aggregate": {
|
|
867
|
+
"dimensions": [{ "path": "entityName", "valueKind": "string" }],
|
|
868
|
+
"measures": [
|
|
869
|
+
{
|
|
870
|
+
"name": "duration.p95",
|
|
871
|
+
"kind": "duration",
|
|
872
|
+
"statistic": "p95",
|
|
873
|
+
"unit": "milliseconds",
|
|
874
|
+
"approximate": true
|
|
875
|
+
}
|
|
876
|
+
]
|
|
877
|
+
}
|
|
878
|
+
}
|
|
879
|
+
```
|
|
880
|
+
|
|
881
|
+
`dimensions` lists the canonical dimensions that the store supports. Every observed `metadata.<key>` field is also a dimension, and `countDistinct.<field>` accepts any dimension or `traceId`. `measures` lists each fixed measure with its `kind`, `unit`, and whether it's `approximate`.
|
|
882
|
+
|
|
564
883
|
After a user selects one field with `valueSuggestions: true`, call `getTraceQueryValues()`:
|
|
565
884
|
|
|
566
885
|
```typescript
|
|
@@ -591,12 +910,12 @@ Discovery returns string values only. Missing values, empty strings, and strings
|
|
|
591
910
|
|
|
592
911
|
Value suggestions are available for these canonical fields:
|
|
593
912
|
|
|
594
|
-
| Scope | Fields
|
|
595
|
-
| ----- |
|
|
596
|
-
| Trace | `entityName`, `entityType`, `environment`, `status`, `tags`
|
|
597
|
-
| Spans | `name`, `spanType`, `model`, `provider`, `status`, `entityType`, `entityName`
|
|
913
|
+
| Scope | Fields |
|
|
914
|
+
| ----- | ------------------------------------------------------------------------------- |
|
|
915
|
+
| Trace | `entityName`, `entityType`, `name`, `spanType`, `environment`, `status`, `tags` |
|
|
916
|
+
| Spans | `name`, `spanType`, `model`, `provider`, `status`, `entityType`, `entityName` |
|
|
598
917
|
|
|
599
|
-
Each executable top-level string `metadata.<key>` field also supports value suggestions in the `trace` scope. Identifiers, timestamps, durations, errors, and version IDs don't. The fields response still includes canonical fields with `valueSuggestions: false`. The values endpoint rejects those paths with `422 TRACE_QUERY_INVALID`.
|
|
918
|
+
Each executable top-level string `metadata.<key>` field also supports value suggestions in the `trace` scope. Identifiers, timestamps, durations, time to first chunk, errors, and version IDs don't. The fields response still includes canonical fields with `valueSuggestions: false`. The values endpoint rejects those paths with `422 TRACE_QUERY_INVALID`.
|
|
600
919
|
|
|
601
920
|
Both discovery requests require a half-open time range. They use the same current-record semantics as `queryTraces()`: qualifying completed roots satisfy `from <= startedAt < to`, and related values come only from current records joined to those roots by `traceId`. Discovery doesn't accept or apply a draft `where` predicate.
|
|
602
921
|
|
|
@@ -610,7 +929,7 @@ Both routes require `observability:read` and use the configured observability st
|
|
|
610
929
|
|
|
611
930
|
## Limitations
|
|
612
931
|
|
|
613
|
-
|
|
932
|
+
Both APIs consider completed traces only and don't include running traces. Trace queries can't return custom projections, embedded evidence, or summaries, and numbered trace pages include a total count. Aggregation can't group by span fields, `traceId`, or `attributes.*`, compute numeric measures over metadata, paginate groups, or zero-fill empty buckets. The CLI doesn't expose a trace aggregation command, so use the SDK or the HTTP endpoint.
|
|
614
933
|
|
|
615
934
|
## Related
|
|
616
935
|
|