@mastra/server 1.78.0-alpha.1 → 1.78.0-alpha.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/dist/docs/SKILL.md
CHANGED
|
@@ -3,7 +3,7 @@ name: mastra-server
|
|
|
3
3
|
description: Documentation for @mastra/server. Use when working with @mastra/server APIs, configuration, or implementation.
|
|
4
4
|
metadata:
|
|
5
5
|
package: "@mastra/server"
|
|
6
|
-
version: "1.78.0-alpha.
|
|
6
|
+
version: "1.78.0-alpha.2"
|
|
7
7
|
---
|
|
8
8
|
|
|
9
9
|
## When to use
|
|
@@ -24,7 +24,7 @@ Read the individual reference documents for detailed explanations and code examp
|
|
|
24
24
|
|
|
25
25
|
### Reference
|
|
26
26
|
|
|
27
|
-
- [
|
|
27
|
+
- [Trace queries](references/reference-observability-tracing-trace-query.md) - Find completed traces by trace fields and related spans. Filter results, paginate, poll for updates, and discover query fields.
|
|
28
28
|
- [Reference: createRoute()](references/reference-server-create-route.md) - Use createRoute() to define type-safe Mastra server routes with Zod validation for requests and responses, plus handler context and OpenAPI metadata.
|
|
29
29
|
- [Reference: MastraServer](references/reference-server-mastra-server.md) - The MastraServer abstract class is the base for all server adapters. Extend this class to create adapters for frameworks other than Hono or Express.
|
|
30
30
|
- [Reference: Server routes](references/reference-server-routes.md) - Server adapters register these routes when you call server.init(). All routes are prefixed with the prefix option if configured.
|
|
@@ -2,143 +2,37 @@
|
|
|
2
2
|
|
|
3
3
|
> Discover all available pages from the documentation index: https://mastra.ai/llms.txt
|
|
4
4
|
|
|
5
|
-
#
|
|
5
|
+
# Trace queries
|
|
6
6
|
|
|
7
|
-
Use `queryTraces()` or `POST /api/observability/traces/query` to find completed
|
|
7
|
+
Use `queryTraces()`, `mastra api trace query`, or `POST /api/observability/traces/query` to find completed traces by their fields and related spans. All three accept the same query structure. The filters, operators, sorting, and pagination options on this page apply to each interface.
|
|
8
8
|
|
|
9
|
-
|
|
9
|
+
For example, find traces with a failed tool call or a duration above a threshold. Each result represents a trace. Use its `traceId` to [retrieve the full trace](https://mastra.ai/reference/client-js/observability).
|
|
10
10
|
|
|
11
|
-
|
|
11
|
+
The endpoint uses the same authentication as other observability routes and requires the `observability:read` permission. The configured observability store must support trace queries. Every query requires a time range of at most 31 days.
|
|
12
12
|
|
|
13
|
-
|
|
13
|
+
For hosted traces, see [Query observability data on Mastra Platform](https://mastra.ai/docs/mastra-platform/observability) for project credentials and regional endpoints.
|
|
14
14
|
|
|
15
|
-
## Query
|
|
16
|
-
|
|
17
|
-
`querySpans()` returns one row per matching completed span across traces, including matching child spans. Use it to list the spans themselves, for example every failed tool call in the last day.
|
|
18
|
-
|
|
19
|
-
`queryTraces()` with a `spans.some` predicate answers a different question. It returns the traces that contain at least one matching span, and it doesn't identify which spans matched. Both use the same span fields and operators.
|
|
20
|
-
|
|
21
|
-
```typescript
|
|
22
|
-
const timeRange = {
|
|
23
|
-
from: '2026-10-01T00:00:00.000Z',
|
|
24
|
-
to: '2026-10-02T00:00:00.000Z',
|
|
25
|
-
}
|
|
26
|
-
const where = {
|
|
27
|
-
op: 'and',
|
|
28
|
-
args: [
|
|
29
|
-
{ op: 'eq', left: { path: 'spanType' }, right: { literal: 'tool_call' } },
|
|
30
|
-
{ op: 'eq', left: { path: 'status' }, right: { literal: 'error' } },
|
|
31
|
-
],
|
|
32
|
-
} as const
|
|
33
|
-
|
|
34
|
-
const firstPage = await mastraClient.querySpans({
|
|
35
|
-
timeRange,
|
|
36
|
-
where,
|
|
37
|
-
orderBy: [{ field: 'startedAt', direction: 'desc' }],
|
|
38
|
-
page: { limit: 50 },
|
|
39
|
-
})
|
|
40
|
-
|
|
41
|
-
if (firstPage.page.next) {
|
|
42
|
-
const secondPage = await mastraClient.querySpans({
|
|
43
|
-
timeRange,
|
|
44
|
-
where,
|
|
45
|
-
orderBy: [{ field: 'startedAt', direction: 'desc' }],
|
|
46
|
-
page: { limit: 50, after: firstPage.page.next },
|
|
47
|
-
})
|
|
48
|
-
}
|
|
49
|
-
```
|
|
50
|
-
|
|
51
|
-
The HTTP route accepts the same body:
|
|
52
|
-
|
|
53
|
-
```bash
|
|
54
|
-
curl --request POST \
|
|
55
|
-
--url http://localhost:4111/api/observability/spans/query \
|
|
56
|
-
--header 'Authorization: Bearer <token>' \
|
|
57
|
-
--header 'Content-Type: application/json' \
|
|
58
|
-
--data '{
|
|
59
|
-
"timeRange": {
|
|
60
|
-
"from": "2026-10-01T00:00:00.000Z",
|
|
61
|
-
"to": "2026-10-02T00:00:00.000Z"
|
|
62
|
-
},
|
|
63
|
-
"where": { "op": "eq", "left": { "path": "spanType" }, "right": { "literal": "tool_call" } },
|
|
64
|
-
"page": { "limit": 50 }
|
|
65
|
-
}'
|
|
66
|
-
```
|
|
67
|
-
|
|
68
|
-
Span queries are optional for observability stores. Call `mastraClient.getObservabilityCapabilities()` and check `capabilities.spanQuery` before showing a span list. Treat a missing `spanQuery` value as unsupported, because older servers don't report it. A server whose store or `@mastra/core` version doesn't support span queries returns `501` with code `SPAN_QUERY_UNSUPPORTED`.
|
|
69
|
-
|
|
70
|
-
The other statuses match trace queries: `400` for malformed JSON or a malformed cursor, `409` for a cursor that doesn't match the query, `413` for a body over 256 KiB, `422` with code `TRACE_QUERY_INVALID` and structured `issues` for an invalid query, `503` with code `TRACE_QUERY_RESOURCE_LIMIT` when a page exceeds a resource limit, and `504` with code `TRACE_QUERY_EXECUTION_TIMEOUT` when the database deadline is exceeded.
|
|
71
|
-
|
|
72
|
-
The server applies [trusted tenant scope](#trusted-tenant-scope) to span queries the same way it does for trace queries. A cursor created under one scope and reused under another returns `409`.
|
|
73
|
-
|
|
74
|
-
### Query individual spans through storage
|
|
75
|
-
|
|
76
|
-
The observability domain in DuckDB and the PostgreSQL and ClickHouse v-next stores supports `querySpans(plan)`. Check `storage.getFeatures()?.includes('span-query')` before using this optional storage method.
|
|
15
|
+
## Query traces with the client SDK
|
|
77
16
|
|
|
78
|
-
|
|
17
|
+
This query finds traces containing a tool call with an error:
|
|
79
18
|
|
|
80
19
|
```typescript
|
|
81
|
-
import {
|
|
82
|
-
|
|
83
|
-
const storage = await mastra.getStorage()?.getStore('observability')
|
|
84
|
-
if (!storage?.getFeatures()?.includes('span-query')) {
|
|
85
|
-
throw new Error('This store does not support span queries')
|
|
86
|
-
}
|
|
87
|
-
|
|
88
|
-
const plan = planSpanQuery({
|
|
89
|
-
timeRange: {
|
|
90
|
-
from: '2026-10-01T00:00:00.000Z',
|
|
91
|
-
to: '2026-10-02T00:00:00.000Z',
|
|
92
|
-
},
|
|
93
|
-
where: {
|
|
94
|
-
op: 'eq',
|
|
95
|
-
left: { path: 'spanType' },
|
|
96
|
-
right: { literal: 'tool_call' },
|
|
97
|
-
},
|
|
98
|
-
page: { limit: 50 },
|
|
99
|
-
})
|
|
100
|
-
|
|
101
|
-
const { spans, page } = await storage.querySpans(plan)
|
|
102
|
-
```
|
|
103
|
-
|
|
104
|
-
This example uses local, unscoped storage. For a shared database, pass `{ scope: { organizationId, resourceId } }` as the second argument to `planSpanQuery()`, using identifiers resolved by your backend's authorization. The request body can't supply trusted scope. Apply the same authorization when loading full span details with `getSpan({ traceId, spanId })`.
|
|
105
|
-
|
|
106
|
-
Span queries use the scalar fields and operators available inside `queryTraces()`'s `spans.some` predicate. They don't accept related-record clauses. Nested attributes require support in the shared predicate contract and each store.
|
|
107
|
-
|
|
108
|
-
The `timeRange` applies to each span's `startedAt`, includes `from`, excludes `to`, and can cover at most 31 days. Sort by `startedAt` or `endedAt`, in either direction. The default is `startedAt` descending. A page contains up to 1,000 rows, with a default of 100. Pass `page.next` back as `page.after` with the same filters, time range, scope, and ordering. Cursors use scope and span identifiers to break timestamp ties. They aren't a snapshot across requests. New completions can appear when the view is refreshed.
|
|
109
|
-
|
|
110
|
-
Each row contains span and trace identifiers, display fields, duration, status, and input/output previews of at most 256 Unicode characters. Truncation flags indicate when a preview is incomplete. Stores select the page before loading previews and cost. Full payloads remain available through the existing span detail method.
|
|
111
|
-
|
|
112
|
-
The `cost` field describes only that span's model-token cost:
|
|
113
|
-
|
|
114
|
-
- `available` includes `amount` and `currency`. Zero is a valid amount.
|
|
115
|
-
- `missing` means no matching model metrics were found.
|
|
116
|
-
- `unavailable` means metrics exist but a complete, unambiguous cost can't be established.
|
|
20
|
+
import { MastraClient } from '@mastra/client-js'
|
|
117
21
|
|
|
118
|
-
|
|
22
|
+
const mastraClient = new MastraClient({ baseUrl: 'http://localhost:4111' })
|
|
119
23
|
|
|
120
|
-
Current-record selection runs before mutable filters. PostgreSQL and ClickHouse select the greatest completed `endedAt`. PostgreSQL breaks ties with its ingestion cursor. DuckDB reconstructs the earliest start event and selects the latest ingested completed event, matching its existing span-predicate behavior. Time bounds are checked again after reconstruction. ClickHouse uses its existing completion-only, insert-only model, where equal-version retries must contain identical records. Querying spans doesn't add a table or materialized view.
|
|
121
|
-
|
|
122
|
-
Queries have a 15-second default execution deadline. PostgreSQL and ClickHouse use their existing `traceQuery.timeoutMs` setting. ClickHouse also caps each query at 512 MiB. A page with more than 50,000 matching model metrics fails with a resource-limit error instead of returning partial cost. A small page size limits returned rows, but doesn't limit the number of records the database must examine.
|
|
123
|
-
|
|
124
|
-
## Query traces with the client SDK
|
|
125
|
-
|
|
126
|
-
Pass the query to `queryTraces()`:
|
|
127
|
-
|
|
128
|
-
```typescript
|
|
129
24
|
const result = await mastraClient.queryTraces({
|
|
130
25
|
timeRange: {
|
|
131
26
|
from: '2026-08-01T00:00:00.000Z',
|
|
132
27
|
to: '2026-08-08T00:00:00.000Z',
|
|
133
28
|
},
|
|
134
29
|
where: {
|
|
135
|
-
|
|
30
|
+
spans: {
|
|
136
31
|
some: {
|
|
137
32
|
op: 'and',
|
|
138
33
|
args: [
|
|
139
|
-
{ op: 'eq', left: { path: '
|
|
140
|
-
{ op: '
|
|
141
|
-
{ op: 'lt', left: { path: 'score' }, right: { literal: 0.6 } },
|
|
34
|
+
{ op: 'eq', left: { path: 'spanType' }, right: { literal: 'tool_call' } },
|
|
35
|
+
{ op: 'exists', path: 'error' },
|
|
142
36
|
],
|
|
143
37
|
},
|
|
144
38
|
},
|
|
@@ -148,79 +42,51 @@ const result = await mastraClient.queryTraces({
|
|
|
148
42
|
})
|
|
149
43
|
```
|
|
150
44
|
|
|
151
|
-
A `some` clause matches when one
|
|
45
|
+
A `spans.some` clause matches when one span satisfies all its conditions. In this example, the same span must be a tool call and have an error. Use `spans.none` to find traces with no spans matching the conditions.
|
|
152
46
|
|
|
153
|
-
Span clauses examine the current root span and current child spans. The root `timeRange` applies only to the selected current root's `startedAt`. Related spans
|
|
47
|
+
Span clauses examine the current root span and current child spans. The root `timeRange` applies only to the selected current root's `startedAt`. Related spans can participate even when their own timestamps are outside that range. Related records correlate only through a matching non-null `traceId`.
|
|
154
48
|
|
|
155
|
-
## Query
|
|
49
|
+
## Query traces with the CLI
|
|
156
50
|
|
|
157
|
-
Pass
|
|
51
|
+
Pass the query as one JSON argument to `mastra api trace query`. Filters and pagination options belong in the JSON input. This command runs the same failed-tool query as the SDK example:
|
|
158
52
|
|
|
159
|
-
```
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
to: '2026-08-08T00:00:00.000Z',
|
|
165
|
-
},
|
|
166
|
-
where: {
|
|
167
|
-
op: 'eq',
|
|
168
|
-
left: { path: 'environment' },
|
|
169
|
-
right: { literal: 'production' },
|
|
170
|
-
},
|
|
53
|
+
```bash
|
|
54
|
+
mastra api --url http://localhost:4111 trace query '{
|
|
55
|
+
"timeRange": {
|
|
56
|
+
"from": "2026-08-01T00:00:00.000Z",
|
|
57
|
+
"to": "2026-08-08T00:00:00.000Z"
|
|
171
58
|
},
|
|
172
|
-
where: {
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
right: { literal: 0.6 },
|
|
183
|
-
},
|
|
184
|
-
},
|
|
185
|
-
},
|
|
186
|
-
},
|
|
187
|
-
},
|
|
188
|
-
{
|
|
189
|
-
traces: {
|
|
190
|
-
some: {
|
|
191
|
-
feedback: {
|
|
192
|
-
some: {
|
|
193
|
-
op: 'eq',
|
|
194
|
-
left: { path: 'feedbackType' },
|
|
195
|
-
right: { literal: 'clinician-correction' },
|
|
196
|
-
},
|
|
197
|
-
},
|
|
198
|
-
},
|
|
199
|
-
},
|
|
200
|
-
},
|
|
201
|
-
],
|
|
59
|
+
"where": {
|
|
60
|
+
"spans": {
|
|
61
|
+
"some": {
|
|
62
|
+
"op": "and",
|
|
63
|
+
"args": [
|
|
64
|
+
{ "op": "eq", "left": { "path": "spanType" }, "right": { "literal": "tool_call" } },
|
|
65
|
+
{ "op": "exists", "path": "error" }
|
|
66
|
+
]
|
|
67
|
+
}
|
|
68
|
+
}
|
|
202
69
|
},
|
|
203
|
-
|
|
204
|
-
}
|
|
70
|
+
"orderBy": [{ "field": "startedAt", "direction": "desc" }],
|
|
71
|
+
"page": { "limit": 25 }
|
|
72
|
+
}'
|
|
205
73
|
```
|
|
206
74
|
|
|
207
|
-
The
|
|
208
|
-
|
|
209
|
-
One `traces.some` clause requires one eligible trace to satisfy its complete nested trace predicate. Separate `traces.some` clauses are independent, so different traces in the same thread can satisfy them. Nested `spans.some`, `scores.some`, and `feedback.some` clauses still require one related record to satisfy every condition inside that clause.
|
|
75
|
+
The explicit `--url` targets a local Mastra server. Without it, observability commands default to `https://observability.mastra.ai`, even when a local server is running. See [CLI target resolution](https://mastra.ai/reference/cli/mastra) for authentication and regional endpoints. The target server and configured store must support the requested query capabilities.
|
|
210
76
|
|
|
211
|
-
|
|
77
|
+
Inspect the command and the target server's request and response schemas with:
|
|
212
78
|
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
{
|
|
217
|
-
"threads": [{ "threadId": "thread-123" }],
|
|
218
|
-
"page": { "next": null }
|
|
219
|
-
}
|
|
79
|
+
```bash
|
|
80
|
+
mastra api trace query --help
|
|
81
|
+
mastra api --url http://localhost:4111 trace query --schema
|
|
220
82
|
```
|
|
221
83
|
|
|
84
|
+
`--schema` describes the request structure. Use [Fields and operators](#fields-and-operators) for the supported predicate paths and operators.
|
|
85
|
+
|
|
222
86
|
## Send an HTTP request
|
|
223
87
|
|
|
88
|
+
Send the same query as a JSON request body:
|
|
89
|
+
|
|
224
90
|
```bash
|
|
225
91
|
curl --request POST \
|
|
226
92
|
--url http://localhost:4111/api/observability/traces/query \
|
|
@@ -241,152 +107,43 @@ curl --request POST \
|
|
|
241
107
|
]
|
|
242
108
|
}
|
|
243
109
|
}
|
|
244
|
-
}
|
|
110
|
+
},
|
|
111
|
+
"orderBy": [{ "field": "startedAt", "direction": "desc" }],
|
|
112
|
+
"page": { "limit": 25 }
|
|
245
113
|
}'
|
|
246
114
|
```
|
|
247
115
|
|
|
248
|
-
## Discover fields and values
|
|
249
|
-
|
|
250
|
-
Use trace-query discovery to build field and value autocomplete without scanning trace payloads or copying the query grammar into your client. The configured observability store must support `trace-query-discovery`.
|
|
251
|
-
|
|
252
|
-
Call `getTraceQueryFields()` for one predicate scope:
|
|
253
|
-
|
|
254
|
-
```typescript
|
|
255
|
-
const timeRange = {
|
|
256
|
-
from: '2026-08-01T00:00:00.000Z',
|
|
257
|
-
to: '2026-08-08T00:00:00.000Z',
|
|
258
|
-
}
|
|
259
|
-
|
|
260
|
-
const fields = await mastraClient.getTraceQueryFields({
|
|
261
|
-
timeRange,
|
|
262
|
-
predicateScope: 'trace',
|
|
263
|
-
search: 'region',
|
|
264
|
-
limit: 25,
|
|
265
|
-
})
|
|
266
|
-
```
|
|
267
|
-
|
|
268
|
-
The response separates canonical fields from observed metadata fields:
|
|
269
|
-
|
|
270
|
-
```json
|
|
271
|
-
{
|
|
272
|
-
"canonicalFields": [],
|
|
273
|
-
"observedFields": [
|
|
274
|
-
{
|
|
275
|
-
"path": "metadata.region",
|
|
276
|
-
"valueKind": "string",
|
|
277
|
-
"operators": ["eq", "ne", "in", "notIn", "exists", "notExists"],
|
|
278
|
-
"valueSuggestions": true,
|
|
279
|
-
"occurrences": 125
|
|
280
|
-
}
|
|
281
|
-
],
|
|
282
|
-
"observedFieldsTruncated": false
|
|
283
|
-
}
|
|
284
|
-
```
|
|
285
|
-
|
|
286
|
-
Canonical fields come from the same registry used to validate trace queries. They don't consume `limit`. The canonical order follows that registry. `observedFields` contains top-level string `metadata.<key>` paths found on qualifying current root spans. These fields are ordered by `occurrences` descending, then by path. Nested metadata, keys containing `.`, empty keys, non-string values, and oversized paths are omitted.
|
|
287
|
-
|
|
288
|
-
`predicateScope` selects the local predicate grammar. It isn't an authorization scope:
|
|
289
|
-
|
|
290
|
-
| `predicateScope` | Example returned path | Query usage |
|
|
291
|
-
| ---------------- | ---------------------------------- | ----------------------------------------- |
|
|
292
|
-
| `trace` | `environment` or `metadata.region` | Root `where` predicate |
|
|
293
|
-
| `spans` | `model` | Inside `spans.some` or `spans.none` |
|
|
294
|
-
| `scores` | `scorerId` | Inside `scores.some` or `scores.none` |
|
|
295
|
-
| `feedback` | `feedbackType` | Inside `feedback.some` or `feedback.none` |
|
|
296
|
-
|
|
297
|
-
A global field picker can request all four scopes in parallel and qualify display labels locally. Every returned path is directly accepted by `queryTraces()` in its matching scope. `metadata` by itself isn't a field. A user interface can offer manual metadata-key entry separately.
|
|
298
|
-
|
|
299
|
-
After a user selects one field with `valueSuggestions: true`, call `getTraceQueryValues()`:
|
|
300
|
-
|
|
301
|
-
```typescript
|
|
302
|
-
const abortController = new AbortController()
|
|
303
|
-
const suggestions = await mastraClient.getTraceQueryValues(
|
|
304
|
-
{
|
|
305
|
-
timeRange,
|
|
306
|
-
predicateScope: 'trace',
|
|
307
|
-
path: 'metadata.region',
|
|
308
|
-
search: 'west',
|
|
309
|
-
limit: 25,
|
|
310
|
-
},
|
|
311
|
-
{ signal: abortController.signal },
|
|
312
|
-
)
|
|
313
|
-
```
|
|
314
|
-
|
|
315
|
-
```json
|
|
316
|
-
{
|
|
317
|
-
"values": [
|
|
318
|
-
{ "value": "eu-west-1", "count": 82 },
|
|
319
|
-
{ "value": "us-west-2", "count": 43 }
|
|
320
|
-
],
|
|
321
|
-
"valuesTruncated": false
|
|
322
|
-
}
|
|
323
|
-
```
|
|
324
|
-
|
|
325
|
-
Discovery returns string values only. Missing values, empty strings, and strings larger than 4,096 UTF-8 bytes are omitted. Values are ordered by `count` descending, then by value. Suggestions are advisory: `queryTraces()` continues to accept valid manual literals that discovery doesn't return.
|
|
326
|
-
|
|
327
|
-
Value suggestions are available for these canonical fields:
|
|
328
|
-
|
|
329
|
-
| Scope | Fields |
|
|
330
|
-
| -------- | ----------------------------------------------------------------------------- |
|
|
331
|
-
| Trace | `entityName`, `entityType`, `environment`, `status`, `tags` |
|
|
332
|
-
| Spans | `name`, `spanType`, `model`, `provider`, `status`, `entityType`, `entityName` |
|
|
333
|
-
| Scores | `scorerId`, `scorerVersion`, `scoreSource` |
|
|
334
|
-
| Feedback | `feedbackType`, `feedbackSource` |
|
|
335
|
-
|
|
336
|
-
Each executable top-level string `metadata.<key>` field also supports value suggestions in the `trace` scope. Identifiers, timestamps, durations, numeric score or feedback values, comments, errors, and version IDs don't. The fields response still includes canonical fields with `valueSuggestions: false`. The values endpoint rejects those paths with `422 TRACE_QUERY_INVALID`.
|
|
337
|
-
|
|
338
|
-
Both discovery requests require a half-open time range of at most 31 days. They use the same current-record semantics as `queryTraces()`: qualifying completed roots satisfy `from <= startedAt < to`, and related values come only from current records joined to those roots by `traceId`. Discovery doesn't accept or apply a draft `where` predicate.
|
|
339
|
-
|
|
340
|
-
`search` defaults to the empty string, is trimmed, and matches case-insensitive literal substrings. Characters such as `%` and `_` have no wildcard meaning. An empty search returns the most frequent results. `limit` defaults to 25 and has a maximum of 100. Both endpoints fetch one extra result to set their truncation flag, but don't provide a cursor. When a flag is `true`, refine `search`. It doesn't imply that a next page is available. No matches return an empty array and `false` truncation.
|
|
341
|
-
|
|
342
|
-
If discovery exceeds its execution timeout, the route returns `504 TRACE_QUERY_EXECUTION_TIMEOUT`. If the backend exceeds a memory or resource budget, it returns `503 TRACE_QUERY_RESOURCE_LIMIT`. Neither failure returns partial suggestions or sets a truncation flag. Truncation only means a complete ranking was cut to the requested response limit.
|
|
343
|
-
|
|
344
|
-
ClickHouse discovery requests default to a 5-second timeout and a 256 MiB per-query memory limit. Configure these with `observability.traceQuery.discovery.timeoutMs` and `observability.traceQuery.discovery.memoryLimitBytes` on `ClickhouseStoreVNext`. Discovery falls back to `observability.traceQuery.timeoutMs` when its timeout isn't configured, without changing ordinary `queryTraces()` execution limits.
|
|
345
|
-
|
|
346
|
-
Both routes require `observability:read` and use the configured observability storage boundary. Don't send organization or project authorization fields in the request. Treat discovered paths, values, and counts as observability data.
|
|
347
|
-
|
|
348
116
|
## Request fields
|
|
349
117
|
|
|
350
|
-
### Trace queries
|
|
351
|
-
|
|
352
118
|
| Field | Required | Description |
|
|
353
119
|
| ------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
354
120
|
| `timeRange` | Yes | Trace start-time boundary. `from` is inclusive and `to` is exclusive. Both values must be ISO timestamps, `from` must be earlier than `to`, and the range can't exceed 31 days. |
|
|
355
|
-
| `where` | No | Recursive trace predicate. Supports scalar conditions and `spans
|
|
356
|
-
| `
|
|
357
|
-
| `orderBy` | No | One item ordering ungrouped results by `startedAt` or `endedAt`, in `asc` or `desc` order. Defaults to `startedAt desc`. Not accepted with `group` or in delta mode. |
|
|
121
|
+
| `where` | No | Recursive trace predicate. Supports scalar conditions and `spans.some` or `spans.none` clauses. |
|
|
122
|
+
| `orderBy` | No | One item ordering results by `startedAt` or `endedAt`, in `asc` or `desc` order. Defaults to `startedAt desc`. Not accepted in delta mode. |
|
|
358
123
|
| `page` | No | `{ limit, after }`. `limit` defaults to 100 and has a maximum of 1000. Pass the opaque `page.next` value as `after`. |
|
|
359
124
|
| `pagination` | No | Numbered pages: `{ page, perPage }`. Defaults to page 0 and 10 results. `perPage` has a maximum of 100. |
|
|
360
125
|
| `mode` | No | Set to `'delta'` to poll using a delta cursor instead of `page` or `pagination`. |
|
|
361
126
|
| `after` | No | Delta mode only. Opaque `deltaCursor` from a numbered page or previous poll. Omit to establish the current watermark. |
|
|
362
127
|
| `limit` | No | Delta mode only. Maximum results per batch. Defaults to 10 and has a maximum of 100. |
|
|
363
128
|
|
|
364
|
-
### Thread queries
|
|
365
|
-
|
|
366
|
-
| Field | Required | Description |
|
|
367
|
-
| -------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
368
|
-
| `traces` | Yes | Eligible trace selection containing `timeRange` and an optional trace `where` predicate. |
|
|
369
|
-
| `where` | No | Recursive thread predicate composed with boolean operators and `traces.some` or `traces.none`. Each quantifier contains a complete trace predicate. |
|
|
370
|
-
| `page` | No | `{ limit, after }`. Thread identities always use fixed ordinal `threadId` ascending order. Pass the opaque `page.next` value as `after`. |
|
|
371
|
-
|
|
372
129
|
Unknown fields are rejected. Requests can't select a projection, declare joins, request counts or measures, or control authorization. Request bodies are limited to 256 KiB.
|
|
373
130
|
|
|
374
131
|
## Query limits
|
|
375
132
|
|
|
376
133
|
The planner rejects a query before storage execution when it exceeds any of these limits:
|
|
377
134
|
|
|
378
|
-
| Input
|
|
379
|
-
|
|
|
380
|
-
| Trace selection `timeRange`
|
|
381
|
-
| Predicate nesting depth
|
|
382
|
-
| Predicate nodes
|
|
383
|
-
| Related `
|
|
384
|
-
| Values in one `in` or `notIn` set
|
|
385
|
-
| Comparison literals and membership values
|
|
386
|
-
| String literal
|
|
387
|
-
| Raw predicate path
|
|
388
|
-
| `page.limit`
|
|
389
|
-
| HTTP request body
|
|
135
|
+
| Input | Maximum |
|
|
136
|
+
| ----------------------------------------- | ----------------- |
|
|
137
|
+
| Trace selection `timeRange` | 31 days |
|
|
138
|
+
| Predicate nesting depth | 12 levels |
|
|
139
|
+
| Predicate nodes | 100 |
|
|
140
|
+
| Related `spans` clauses | 8 total |
|
|
141
|
+
| Values in one `in` or `notIn` set | 100 |
|
|
142
|
+
| Comparison literals and membership values | 1,000 total |
|
|
143
|
+
| String literal | 4,096 UTF-8 bytes |
|
|
144
|
+
| Raw predicate path | 128 UTF-8 bytes |
|
|
145
|
+
| `page.limit` | 1,000 |
|
|
146
|
+
| HTTP request body | 256 KiB |
|
|
390
147
|
|
|
391
148
|
Each comparison literal counts as one literal. Each member of an `in` or `notIn` set also counts as one literal, even when the set is within its per-set limit.
|
|
392
149
|
|
|
@@ -404,16 +161,10 @@ Hono and Fastify enforce the request-body limit before JSON parsing. Express and
|
|
|
404
161
|
| Span | `spanType`, `model`, `provider`, `status`, `entityType`, `entityId`, `entityName`, `entityVersionId`, `parentEntityVersionId`, `rootEntityVersionId`, `runId`, `sessionId`, `userId`, `organizationId` | `eq`, `ne`, `in`, `notIn`, `exists`, `notExists` |
|
|
405
162
|
| Span | `startedAt`, `endedAt`, `durationMs` | `eq`, `ne`, `in`, `notIn`, `lt`, `lte`, `gt`, `gte`, `exists`, `notExists` |
|
|
406
163
|
| Span | `error` | `exists`, `notExists` |
|
|
407
|
-
| Score | `scorerId`, `scorerVersion`, `scoreSource`, `entityVersionId`, `parentEntityVersionId`, `rootEntityVersionId` | `eq`, `ne`, `in`, `notIn`, `exists`, `notExists` |
|
|
408
|
-
| Score | `score`, `timestamp` | `eq`, `ne`, `in`, `notIn`, `lt`, `lte`, `gt`, `gte`, `exists`, `notExists` |
|
|
409
|
-
| Score | `spanId` | `exists`, `notExists` |
|
|
410
|
-
| Feedback | `feedbackType`, `feedbackSource`, `feedbackUserId`, `sourceId`, `entityVersionId`, `parentEntityVersionId`, `rootEntityVersionId` | `eq`, `ne`, `in`, `notIn`, `exists`, `notExists` |
|
|
411
|
-
| Feedback | `value`, `timestamp` | `eq`, `ne`, `in`, `notIn`, `lt`, `lte`, `gt`, `gte`, `exists`, `notExists` |
|
|
412
|
-
| Feedback | `comment` | `eq`, `ne`, `in`, `notIn`, `exists`, `notExists`, `matches`, `notMatches` |
|
|
413
164
|
|
|
414
165
|
Compose predicates with `{ op: 'and', args: [...] }`, `{ op: 'or', args: [...] }`, and `{ op: 'not', arg: ... }`. Comparison predicates place a field reference on the left and a literal on the right. Membership predicates use a field reference in `value` and a homogeneous literal array in `set`. Tag predicates name the stored tag collection in `path`. `includes` and `notIncludes` also take one tag in `value`, while `exists` and `notExists` take only `op` and `path`. See [Filter by tags](#filter-by-tags). Text predicates use the comparison shape with a string literal. See [Filter by text](#filter-by-text).
|
|
415
166
|
|
|
416
|
-
String comparisons are case-sensitive, and literals are never coerced. `matches` and `notMatches` are the exception: they compare words case-insensitively. Canonical string fields compare exact stored values. Metadata string values are trimmed before comparison, as described below.
|
|
167
|
+
String comparisons are case-sensitive, and literals are never coerced. `matches` and `notMatches` are the exception: they compare words case-insensitively. Canonical string fields compare exact stored values. Metadata string values are trimmed before comparison, as described below. Duration predicates require numbers, while timestamp predicates require ISO timestamp strings. A missing value satisfies neither positive nor ordered predicates, although it does satisfy the negative operators `ne` and `notIn`. Combine a negative predicate with `exists` when the field must also be present.
|
|
417
168
|
|
|
418
169
|
### Filter by root duration
|
|
419
170
|
|
|
@@ -532,53 +283,6 @@ const versionedRetrieval = {
|
|
|
532
283
|
}
|
|
533
284
|
```
|
|
534
285
|
|
|
535
|
-
### Filter by score time
|
|
536
|
-
|
|
537
|
-
Score timestamps are independent of the required top-level trace `timeRange`. The top-level range selects trace candidates by trace start time. A nested `timestamp` predicate limits related score records:
|
|
538
|
-
|
|
539
|
-
```typescript
|
|
540
|
-
const query = {
|
|
541
|
-
timeRange: {
|
|
542
|
-
from: '2026-08-01T00:00:00.000Z',
|
|
543
|
-
to: '2026-08-08T00:00:00.000Z',
|
|
544
|
-
},
|
|
545
|
-
where: {
|
|
546
|
-
scores: {
|
|
547
|
-
some: {
|
|
548
|
-
op: 'and',
|
|
549
|
-
args: [
|
|
550
|
-
{ op: 'eq', left: { path: 'scoreSource' }, right: { literal: 'automated' } },
|
|
551
|
-
{
|
|
552
|
-
op: 'gte',
|
|
553
|
-
left: { path: 'timestamp' },
|
|
554
|
-
right: { literal: '2026-07-15T00:00:00.000Z' },
|
|
555
|
-
},
|
|
556
|
-
{
|
|
557
|
-
op: 'lt',
|
|
558
|
-
left: { path: 'timestamp' },
|
|
559
|
-
right: { literal: '2026-08-01T00:00:00.000Z' },
|
|
560
|
-
},
|
|
561
|
-
],
|
|
562
|
-
},
|
|
563
|
-
},
|
|
564
|
-
},
|
|
565
|
-
}
|
|
566
|
-
```
|
|
567
|
-
|
|
568
|
-
### Filter by span anchoring
|
|
569
|
-
|
|
570
|
-
Use `spanId` only to test whether a score targets a span. The predicate doesn't join the score to a matching span predicate:
|
|
571
|
-
|
|
572
|
-
```typescript
|
|
573
|
-
// At least one score targets a span.
|
|
574
|
-
const anchoredToSpanWhere = { scores: { some: { op: 'exists', path: 'spanId' } } }
|
|
575
|
-
|
|
576
|
-
// At least one score applies to the trace rather than a span.
|
|
577
|
-
const traceLevelScoreWhere = { scores: { some: { op: 'notExists', path: 'spanId' } } }
|
|
578
|
-
```
|
|
579
|
-
|
|
580
|
-
The deprecated `source` field and `scorerName` aren't available as score predicate fields. Use the canonical `scoreSource` and `scorerId` fields.
|
|
581
|
-
|
|
582
286
|
### Filter by metadata
|
|
583
287
|
|
|
584
288
|
Use `metadata.<key>` paths for custom dimensions stored on the current root span. Metadata predicates compare trimmed, non-empty string values exactly and case-sensitively. Leading and trailing whitespace in a stored string value is ignored. Values empty after trimming count as missing. Null, numeric, boolean, object, and array values also count as missing.
|
|
@@ -604,7 +308,7 @@ const messageTrace = {
|
|
|
604
308
|
|
|
605
309
|
The key must name one top-level property. Empty keys and nested paths are rejected. Metadata keys aren't trimmed, so leading and trailing whitespace remains part of the exact key identity. When metadata duplicates a canonical trace field, such as `resourceId`, `threadId`, or `environment`, prefer the canonical field because it uses the dedicated storage column. The `metadata.<key>` form remains available when you specifically need the value from the metadata object. Metadata and canonical values may differ.
|
|
606
310
|
|
|
607
|
-
|
|
311
|
+
Trace-query discovery returns executable top-level string metadata fields observed in the selected time range.
|
|
608
312
|
|
|
609
313
|
### Filter by tags
|
|
610
314
|
|
|
@@ -630,19 +334,19 @@ const manualReview = {
|
|
|
630
334
|
|
|
631
335
|
Tags compare exactly and case-sensitively as whole strings. `value` must be one string with at least one non-whitespace character. Combine several `includes` predicates with `and` or `or` to require or allow multiple tags. A trace with no recorded tags and a trace with an empty tag list behave the same: both satisfy `notExists`, neither satisfies `includes`, `notIncludes`, or `exists`. Use `{ op: 'not', arg: { op: 'includes', ... } }` when untagged traces should also match.
|
|
632
336
|
|
|
633
|
-
`tags` is available in trace predicates only
|
|
337
|
+
`tags` is available in trace predicates only. `in`, `notIn`, and the comparison operators are rejected for `tags`. Value discovery for `tags` returns each observed tag with the number of qualifying traces that carry it.
|
|
634
338
|
|
|
635
339
|
### Filter by text
|
|
636
340
|
|
|
637
|
-
|
|
341
|
+
Span names support word matching in addition to exact comparison. `matches` succeeds when the literal's words appear in the field, in order, as whole words. `notMatches` succeeds when the field exists but doesn't contain those words. This query finds traces with a span whose name contains the words "document search":
|
|
638
342
|
|
|
639
343
|
```typescript
|
|
640
|
-
const
|
|
641
|
-
|
|
344
|
+
const documentSearch = {
|
|
345
|
+
spans: {
|
|
642
346
|
some: {
|
|
643
347
|
op: 'matches',
|
|
644
|
-
left: { path: '
|
|
645
|
-
right: { literal: '
|
|
348
|
+
left: { path: 'name' },
|
|
349
|
+
right: { literal: 'document search' },
|
|
646
350
|
},
|
|
647
351
|
},
|
|
648
352
|
}
|
|
@@ -655,50 +359,11 @@ Both the literal and the stored value are lowercased and split into words. A wor
|
|
|
655
359
|
| `matches` | The field contains the literal's words as a contiguous whole-word phrase |
|
|
656
360
|
| `notMatches` | The field is present and doesn't contain that phrase |
|
|
657
361
|
|
|
658
|
-
The literal must contain at least one letter or digit; otherwise the request fails with `422 TRACE_QUERY_INVALID`. A missing field satisfies neither operator. Use `{ op: 'not', arg: { op: 'matches', ... } }` when records without the field should also match. Text operators are available on span `name
|
|
659
|
-
|
|
660
|
-
### Filter by feedback
|
|
661
|
-
|
|
662
|
-
Every condition inside one `feedback.some` or `feedback.none` clause applies to the same current feedback record. `feedbackType` and `feedbackSource` are exact application-defined strings rather than built-in enums. This query finds traces with a numeric patient rating below zero:
|
|
663
|
-
|
|
664
|
-
```typescript
|
|
665
|
-
const negativePatientRating = {
|
|
666
|
-
feedback: {
|
|
667
|
-
some: {
|
|
668
|
-
op: 'and',
|
|
669
|
-
args: [
|
|
670
|
-
{ op: 'eq', left: { path: 'feedbackType' }, right: { literal: 'rating' } },
|
|
671
|
-
{ op: 'eq', left: { path: 'feedbackSource' }, right: { literal: 'patient' } },
|
|
672
|
-
{ op: 'lt', left: { path: 'value' }, right: { literal: 0 } },
|
|
673
|
-
],
|
|
674
|
-
},
|
|
675
|
-
},
|
|
676
|
-
}
|
|
677
|
-
```
|
|
678
|
-
|
|
679
|
-
Strict stored-value types are the portable contract for feedback predicates. PostgreSQL and ClickHouse distinguish numeric `3` from textual `'3'` for equality and ordered comparisons. DuckDB currently persists feedback values as `VARCHAR`, so numeric-looking strings may be coerced for equality and ordered numeric predicates. OBS-306 will remove this DuckDB exception through typed persistence. Ordered operators require a finite numeric literal. `eq` and `ne` accept one string or number, while `in` and `notIn` require a non-empty set containing only strings or only numbers. `exists` and `notExists` test for either value type.
|
|
680
|
-
|
|
681
|
-
Use `none` to select traces without a matching record. Traces with no feedback also match:
|
|
682
|
-
|
|
683
|
-
```typescript
|
|
684
|
-
const missingClinicianReview = {
|
|
685
|
-
feedback: {
|
|
686
|
-
none: {
|
|
687
|
-
op: 'and',
|
|
688
|
-
args: [
|
|
689
|
-
{ op: 'eq', left: { path: 'feedbackType' }, right: { literal: 'clinical-review' } },
|
|
690
|
-
{ op: 'eq', left: { path: 'feedbackSource' }, right: { literal: 'clinician' } },
|
|
691
|
-
],
|
|
692
|
-
},
|
|
693
|
-
},
|
|
694
|
-
}
|
|
695
|
-
```
|
|
696
|
-
|
|
697
|
-
Feedback `timestamp` predicates are independent of the root `timeRange`. `comment` supports exact string operators and word matching with `matches` and `notMatches`. See [Filter by text](#filter-by-text). The deprecated feedback fields `source` and `userId` aren't available. Use `feedbackSource` and `feedbackUserId`.
|
|
362
|
+
The literal must contain at least one letter or digit; otherwise the request fails with `422 TRACE_QUERY_INVALID`. A missing field satisfies neither operator. Use `{ op: 'not', arg: { op: 'matches', ... } }` when records without the field should also match. Text operators are available on span `name`. Other string fields, numbers, timestamps, tags, and metadata reject them. Substring, wildcard, regular-expression, and fuzzy matching aren't supported, and there is no search across several fields.
|
|
698
363
|
|
|
699
364
|
## Responses
|
|
700
365
|
|
|
701
|
-
|
|
366
|
+
A query using keyset pagination returns lightweight completed traces and a cursor for the next page:
|
|
702
367
|
|
|
703
368
|
```json
|
|
704
369
|
{
|
|
@@ -720,38 +385,26 @@ An ungrouped query returns only lightweight completed traces:
|
|
|
720
385
|
}
|
|
721
386
|
```
|
|
722
387
|
|
|
723
|
-
|
|
724
|
-
|
|
725
|
-
```typescript
|
|
726
|
-
const timeRange = {
|
|
727
|
-
from: '2026-08-01T00:00:00.000Z',
|
|
728
|
-
to: '2026-08-08T00:00:00.000Z',
|
|
729
|
-
}
|
|
388
|
+
The response doesn't include the related spans that matched the predicate. Use [`getTrace()`](https://mastra.ai/reference/client-js/observability) to load the full trace after selecting a result.
|
|
730
389
|
|
|
731
|
-
|
|
732
|
-
const legacyResult = await mastraClient.queryTraces({
|
|
733
|
-
timeRange,
|
|
734
|
-
group: { by: ['threadId'] },
|
|
735
|
-
})
|
|
736
|
-
// { groups: [{ threadId: 'thread-123' }], page: { next: null } }
|
|
737
|
-
|
|
738
|
-
// Replacement
|
|
739
|
-
const result = await mastraClient.queryTraceThreads({
|
|
740
|
-
traces: { timeRange },
|
|
741
|
-
})
|
|
742
|
-
// { threads: [{ threadId: 'thread-123' }], page: { next: null } }
|
|
743
|
-
```
|
|
390
|
+
### CLI responses
|
|
744
391
|
|
|
745
|
-
|
|
392
|
+
The CLI wraps the API response in `data`:
|
|
746
393
|
|
|
747
394
|
```json
|
|
748
395
|
{
|
|
749
|
-
"
|
|
750
|
-
|
|
396
|
+
"data": {
|
|
397
|
+
"traces": [],
|
|
398
|
+
"page": { "next": null }
|
|
399
|
+
}
|
|
751
400
|
}
|
|
752
401
|
```
|
|
753
402
|
|
|
754
|
-
|
|
403
|
+
Read results from `data.traces`. Pagination metadata is available at `data.page.next` for keyset queries, `data.pagination` for numbered pages, and `data.deltaCursor` and `data.delta.hasMore` for delta polling.
|
|
404
|
+
|
|
405
|
+
Each CLI invocation makes one query. To fetch another keyset page, pass `data.page.next` as `page.after` in the next request. For delta polling, repeat the command with `data.deltaCursor` as the top-level `after` value. Keep the same time range and predicate. See [Pagination and errors](#pagination-and-errors) for mode-specific behavior and polling limitations.
|
|
406
|
+
|
|
407
|
+
To inspect a selected trace, use `mastra api --url http://localhost:4111 trace get <traceId> --verbose`. Use the same target and authentication as the query.
|
|
755
408
|
|
|
756
409
|
## Pagination and errors
|
|
757
410
|
|
|
@@ -765,7 +418,7 @@ Trace queries support three mutually exclusive modes:
|
|
|
765
418
|
| Numbered pages | `pagination: { page, perPage }` | `pagination: { total, page, perPage, hasMore }` and, when delta polling is supported, `deltaCursor` |
|
|
766
419
|
| Delta polling | `mode: 'delta'`, optional top-level `after` and `limit` | `delta: { limit, hasMore }` and `deltaCursor` |
|
|
767
420
|
|
|
768
|
-
Numbered pages are zero-based. Their total and rows are read consistently within each request. Don't combine `page`, `pagination`, or delta mode. Top-level `after` and `limit` are accepted only in delta mode.
|
|
421
|
+
Numbered pages are zero-based. Their total and rows are read consistently within each request. Don't combine `page`, `pagination`, or delta mode. Top-level `after` and `limit` are accepted only in delta mode.
|
|
769
422
|
|
|
770
423
|
### Poll after loading a numbered page
|
|
771
424
|
|
|
@@ -811,7 +464,7 @@ Delta ordering uses the storage ingestion watermark and adapter-specific tie-bre
|
|
|
811
464
|
|
|
812
465
|
Keep the same normalized `where` predicate and exact `timeRange` bounds for every poll. The range continues to select root `startedAt`, even when a trace completes later. Changing `timeRange.to` to the current time invalidates the cursor. Reload a numbered page and use its cursor whenever the selection or authorization scope changes. You can change the batch limit without restarting.
|
|
813
466
|
|
|
814
|
-
New completed roots and roots completing after the cursor can be returned. Related spans
|
|
467
|
+
New completed roots and roots completing after the cursor can be returned. Related spans are evaluated when a root is selected, but later related-record writes don't guarantee that the trace is emitted again. Deleted traces and traces that stop matching aren't returned as removals or tombstones. Refresh numbered pages when you need to reconcile those changes.
|
|
815
468
|
|
|
816
469
|
ClickHouse polling is best effort. Its delta index and trace records use separate materialized-view tables. Concurrent queries can observe an insert in one table before another, as described in [ClickHouse's materialized-view visibility rules](https://github.com/ClickHouse/clickhouse-docs/blob/main/knowledgebase/are_materialized_views_inserted_asynchronously.mdx). A poll can advance past an index entry before the matching trace becomes visible and miss that trace in later polls. Reload numbered pages periodically to reconcile these gaps.
|
|
817
470
|
|
|
@@ -819,17 +472,17 @@ ClickHouse's delta index retains two days of events and doesn't backfill histori
|
|
|
819
472
|
|
|
820
473
|
### Keyset ordering and errors
|
|
821
474
|
|
|
822
|
-
Ordering is deterministic. Trace ordering appends `traceId` ascending as a tie-breaker.
|
|
475
|
+
Ordering is deterministic. Trace ordering appends `traceId` ascending as a tie-breaker.
|
|
823
476
|
|
|
824
|
-
Keyset cursors are bound to the operation, accepted normalized query, and ordering. Reusing a keyset cursor after changing the trace selection, predicates, or ordering returns `409`.
|
|
477
|
+
Keyset cursors are bound to the operation, accepted normalized query, and ordering. Reusing a keyset cursor after changing the trace selection, predicates, or ordering returns `409`.
|
|
825
478
|
|
|
826
479
|
Numbered-page handoff cursors and delta cursors also bind the authorization state. Changes to the caller's roles or permissions invalidate these cursors and return `409`. If a delta poll returns `409`, reload the numbered pages and resume polling with the new `deltaCursor`. A malformed cursor returns `400`.
|
|
827
480
|
|
|
828
481
|
### Trusted tenant scope
|
|
829
482
|
|
|
830
|
-
Hosts that serve more than one tenant supply a trusted scope outside the query document. The server reads the reserved `organizationId` request-context key, which only server-side authentication can set, and passes `{ organizationId }` to the planner. The scope is carried on the trusted plan and ANDed into every scan: current roots, related spans
|
|
483
|
+
Hosts that serve more than one tenant supply a trusted scope outside the query document. The server reads the reserved `organizationId` request-context key, which only server-side authentication can set, and passes `{ organizationId }` to the planner. The scope is carried on the trusted plan and ANDed into every scan: current roots, related spans and both discovery routes. A related record from another tenant that shares a `traceId` never qualifies a trace.
|
|
831
484
|
|
|
832
|
-
Hosts that call storage directly can pass a fuller scope to `planTraceQuery()`, `
|
|
485
|
+
Hosts that call storage directly can pass a fuller scope to `planTraceQuery()`, `planTraceQueryObservedFields()`, and `planTraceQueryValues()`:
|
|
833
486
|
|
|
834
487
|
```typescript
|
|
835
488
|
const plan = planTraceQuery(parseTraceQueryRequest(request), {
|
|
@@ -841,7 +494,7 @@ const plan = planTraceQuery(parseTraceQueryRequest(request), {
|
|
|
841
494
|
|
|
842
495
|
Cursor pagination is deterministic, but it isn't a database snapshot. Traces or replacement signals written between page requests can change later pages.
|
|
843
496
|
|
|
844
|
-
PostgreSQL and ClickHouse stop
|
|
497
|
+
PostgreSQL and ClickHouse stop trace queries after 15 seconds by default and return `504` when the database timeout is exceeded. Set `traceQueryTimeoutMs` in the store's vNext observability configuration to an integer from 1 through 300,000 milliseconds to change the timeout. DuckDB doesn't currently provide query-scoped timeout or cancellation through its driver wrapper, so this `504` guarantee doesn't apply to DuckDB.
|
|
845
498
|
|
|
846
499
|
| Status | Meaning |
|
|
847
500
|
| ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
@@ -853,9 +506,107 @@ PostgreSQL and ClickHouse stop advanced trace and thread queries after 15 second
|
|
|
853
506
|
| `503` | Discovery exceeded a backend memory or resource budget and returned `TRACE_QUERY_RESOURCE_LIMIT`. |
|
|
854
507
|
| `504` | A PostgreSQL or ClickHouse query exceeded its configured database execution timeout. Discovery returns `TRACE_QUERY_EXECUTION_TIMEOUT`. |
|
|
855
508
|
|
|
509
|
+
## Discover fields and values
|
|
510
|
+
|
|
511
|
+
Use trace-query discovery to build field and value autocomplete without scanning trace payloads or copying the query grammar into your client. The configured observability store must support `trace-query-discovery`.
|
|
512
|
+
|
|
513
|
+
The CLI doesn't expose dedicated discovery commands, so use the SDK methods below or the HTTP endpoints `POST /api/observability/traces/query/fields` and `POST /api/observability/traces/query/values`.
|
|
514
|
+
|
|
515
|
+
Call `getTraceQueryFields()` for one predicate scope:
|
|
516
|
+
|
|
517
|
+
```typescript
|
|
518
|
+
const timeRange = {
|
|
519
|
+
from: '2026-08-01T00:00:00.000Z',
|
|
520
|
+
to: '2026-08-08T00:00:00.000Z',
|
|
521
|
+
}
|
|
522
|
+
|
|
523
|
+
const fields = await mastraClient.getTraceQueryFields({
|
|
524
|
+
timeRange,
|
|
525
|
+
predicateScope: 'trace',
|
|
526
|
+
search: 'region',
|
|
527
|
+
limit: 25,
|
|
528
|
+
})
|
|
529
|
+
```
|
|
530
|
+
|
|
531
|
+
The response separates canonical fields from observed metadata fields:
|
|
532
|
+
|
|
533
|
+
```json
|
|
534
|
+
{
|
|
535
|
+
"canonicalFields": [],
|
|
536
|
+
"observedFields": [
|
|
537
|
+
{
|
|
538
|
+
"path": "metadata.region",
|
|
539
|
+
"valueKind": "string",
|
|
540
|
+
"operators": ["eq", "ne", "in", "notIn", "exists", "notExists"],
|
|
541
|
+
"valueSuggestions": true,
|
|
542
|
+
"occurrences": 125
|
|
543
|
+
}
|
|
544
|
+
],
|
|
545
|
+
"observedFieldsTruncated": false
|
|
546
|
+
}
|
|
547
|
+
```
|
|
548
|
+
|
|
549
|
+
Canonical fields come from the same registry used to validate trace queries. They don't consume `limit`. The canonical order follows that registry. `observedFields` contains top-level string `metadata.<key>` paths found on qualifying current root spans. These fields are ordered by `occurrences` descending, then by path. Nested metadata, keys containing `.`, empty keys, non-string values, and oversized paths are omitted.
|
|
550
|
+
|
|
551
|
+
`predicateScope` selects the local predicate grammar. It isn't an authorization scope:
|
|
552
|
+
|
|
553
|
+
| `predicateScope` | Example returned path | Query usage |
|
|
554
|
+
| ---------------- | ---------------------------------- | ----------------------------------- |
|
|
555
|
+
| `trace` | `environment` or `metadata.region` | Root `where` predicate |
|
|
556
|
+
| `spans` | `model` | Inside `spans.some` or `spans.none` |
|
|
557
|
+
|
|
558
|
+
A field picker can request the `trace` and `spans` scopes in parallel and label each field with its scope. Every returned path is directly accepted by `queryTraces()` in its matching scope. `metadata` by itself isn't a field. A user interface can offer manual metadata-key entry separately.
|
|
559
|
+
|
|
560
|
+
After a user selects one field with `valueSuggestions: true`, call `getTraceQueryValues()`:
|
|
561
|
+
|
|
562
|
+
```typescript
|
|
563
|
+
const abortController = new AbortController()
|
|
564
|
+
const suggestions = await mastraClient.getTraceQueryValues(
|
|
565
|
+
{
|
|
566
|
+
timeRange,
|
|
567
|
+
predicateScope: 'trace',
|
|
568
|
+
path: 'metadata.region',
|
|
569
|
+
search: 'west',
|
|
570
|
+
limit: 25,
|
|
571
|
+
},
|
|
572
|
+
{ signal: abortController.signal },
|
|
573
|
+
)
|
|
574
|
+
```
|
|
575
|
+
|
|
576
|
+
```json
|
|
577
|
+
{
|
|
578
|
+
"values": [
|
|
579
|
+
{ "value": "eu-west-1", "count": 82 },
|
|
580
|
+
{ "value": "us-west-2", "count": 43 }
|
|
581
|
+
],
|
|
582
|
+
"valuesTruncated": false
|
|
583
|
+
}
|
|
584
|
+
```
|
|
585
|
+
|
|
586
|
+
Discovery returns string values only. Missing values, empty strings, and strings larger than 4,096 UTF-8 bytes are omitted. Values are ordered by `count` descending, then by value. Suggestions are advisory: `queryTraces()` continues to accept valid manual literals that discovery doesn't return.
|
|
587
|
+
|
|
588
|
+
Value suggestions are available for these canonical fields:
|
|
589
|
+
|
|
590
|
+
| Scope | Fields |
|
|
591
|
+
| ----- | ----------------------------------------------------------------------------- |
|
|
592
|
+
| Trace | `entityName`, `entityType`, `environment`, `status`, `tags` |
|
|
593
|
+
| Spans | `name`, `spanType`, `model`, `provider`, `status`, `entityType`, `entityName` |
|
|
594
|
+
|
|
595
|
+
Each executable top-level string `metadata.<key>` field also supports value suggestions in the `trace` scope. Identifiers, timestamps, durations, errors, and version IDs don't. The fields response still includes canonical fields with `valueSuggestions: false`. The values endpoint rejects those paths with `422 TRACE_QUERY_INVALID`.
|
|
596
|
+
|
|
597
|
+
Both discovery requests require a half-open time range of at most 31 days. They use the same current-record semantics as `queryTraces()`: qualifying completed roots satisfy `from <= startedAt < to`, and related values come only from current records joined to those roots by `traceId`. Discovery doesn't accept or apply a draft `where` predicate.
|
|
598
|
+
|
|
599
|
+
`search` defaults to the empty string, is trimmed, and matches case-insensitive literal substrings. Characters such as `%` and `_` have no wildcard meaning. An empty search returns the most frequent results. `limit` defaults to 25 and has a maximum of 100. Both endpoints fetch one extra result to set their truncation flag, but don't provide a cursor. When a flag is `true`, refine `search`. It doesn't imply that a next page is available. No matches return an empty array and `false` truncation.
|
|
600
|
+
|
|
601
|
+
If discovery exceeds its execution timeout, the route returns `504 TRACE_QUERY_EXECUTION_TIMEOUT`. If the backend exceeds a memory or resource budget, it returns `503 TRACE_QUERY_RESOURCE_LIMIT`. Neither failure returns partial suggestions or sets a truncation flag. Truncation only means a complete ranking was cut to the requested response limit.
|
|
602
|
+
|
|
603
|
+
ClickHouse discovery requests default to a 5-second timeout and a 256 MiB per-query memory limit. Configure these with `observability.traceQuery.discovery.timeoutMs` and `observability.traceQuery.discovery.memoryLimitBytes` on `ClickhouseStoreVNext`. Discovery falls back to `observability.traceQuery.timeoutMs` when its timeout isn't configured, without changing ordinary `queryTraces()` execution limits.
|
|
604
|
+
|
|
605
|
+
Both routes require `observability:read` and use the configured observability storage boundary. Don't send organization or project authorization fields in the request. Treat discovered paths, values, and counts as observability data.
|
|
606
|
+
|
|
856
607
|
## Limitations
|
|
857
608
|
|
|
858
|
-
Trace
|
|
609
|
+
Trace queries consider completed traces only. They don't support running traces, custom projections, embedded evidence, summaries, aggregations, measures, or custom grouping. Numbered trace pages include a total count.
|
|
859
610
|
|
|
860
611
|
## Related
|
|
861
612
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@mastra/server",
|
|
3
|
-
"version": "1.78.0-alpha.
|
|
3
|
+
"version": "1.78.0-alpha.2",
|
|
4
4
|
"description": "",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"files": [
|
|
@@ -116,15 +116,15 @@
|
|
|
116
116
|
"vitest": "4.1.11",
|
|
117
117
|
"zod": "^4.6.4",
|
|
118
118
|
"zod-to-ts": "^2.1.0",
|
|
119
|
-
"@internal/core": "0.1.2",
|
|
120
119
|
"@internal/lint": "0.0.143",
|
|
121
120
|
"@internal/storage-test-utils": "0.0.139",
|
|
122
|
-
"@
|
|
121
|
+
"@internal/core": "0.1.2",
|
|
123
122
|
"@internal/test-utils": "0.0.79",
|
|
124
|
-
"@internal/voice": "0.0.41",
|
|
125
123
|
"@internal/types-builder": "0.0.118",
|
|
124
|
+
"@mastra/agent-builder": "1.1.28",
|
|
125
|
+
"@internal/voice": "0.0.41",
|
|
126
126
|
"@mastra/schema-compat": "1.3.14",
|
|
127
|
-
"@mastra/core": "1.78.0-alpha.
|
|
127
|
+
"@mastra/core": "1.78.0-alpha.2"
|
|
128
128
|
},
|
|
129
129
|
"homepage": "https://mastra.ai",
|
|
130
130
|
"repository": {
|