@mastra/server 1.78.0-alpha.1 → 1.78.0-alpha.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/_types/@mastra_schema-compat/dist/provider-compats/openai-reasoning.d.ts +0 -1
- package/dist/_types/@mastra_schema-compat/dist/provider-compats/openai.d.ts +4 -0
- package/dist/{dist-aAfueiQI.js → dist-0eJzATDs.js} +104 -3
- package/dist/dist-0eJzATDs.js.map +1 -0
- package/dist/{dist-Cfv3ewR4.cjs → dist-KzbPuRgb.cjs} +104 -3
- package/dist/dist-KzbPuRgb.cjs.map +1 -0
- package/dist/docs/SKILL.md +2 -2
- package/dist/docs/assets/SOURCE_MAP.json +1 -1
- package/dist/docs/references/reference-observability-tracing-trace-query.md +198 -443
- package/dist/route-builder-2yrD15TL.cjs.map +1 -1
- package/dist/route-builder-lBil8Cvo.js.map +1 -1
- package/dist/schema-e0VHpMt0-BS9KHnuT.cjs.map +1 -1
- package/dist/schema-e0VHpMt0-CvPvLS06.js.map +1 -1
- package/dist/server/handlers/agent-builder.cjs +1 -1
- package/dist/server/handlers/agent-builder.js +1 -1
- package/dist/server/handlers/observability-new-endpoints.d.ts +32 -0
- package/dist/server/handlers/observability-new-endpoints.d.ts.map +1 -1
- package/dist/server/server-adapter/routes/observability.d.ts +16 -0
- package/dist/server/server-adapter/routes/observability.d.ts.map +1 -1
- package/package.json +5 -5
- package/dist/dist-Cfv3ewR4.cjs.map +0 -1
- package/dist/dist-aAfueiQI.js.map +0 -1
|
@@ -2,143 +2,37 @@
|
|
|
2
2
|
|
|
3
3
|
> Discover all available pages from the documentation index: https://mastra.ai/llms.txt
|
|
4
4
|
|
|
5
|
-
#
|
|
5
|
+
# Trace queries
|
|
6
6
|
|
|
7
|
-
Use `queryTraces()` or `POST /api/observability/traces/query` to find completed
|
|
7
|
+
Use `queryTraces()`, `mastra api trace query`, or `POST /api/observability/traces/query` to find completed traces by their fields and related spans. All three accept the same query structure. The filters, operators, sorting, and pagination options on this page apply to each interface.
|
|
8
8
|
|
|
9
|
-
|
|
9
|
+
For example, find traces with a failed tool call or a duration above a threshold. Each result represents a trace. Use its `traceId` to [retrieve the full trace](https://mastra.ai/reference/client-js/observability).
|
|
10
10
|
|
|
11
|
-
|
|
11
|
+
The endpoint uses the same authentication as other observability routes and requires the `observability:read` permission. The configured observability store must support trace queries. Every query requires a time range.
|
|
12
12
|
|
|
13
|
-
|
|
13
|
+
For hosted traces, see [Query observability data on Mastra Platform](https://mastra.ai/docs/mastra-platform/observability) for project credentials and regional endpoints.
|
|
14
14
|
|
|
15
|
-
## Query
|
|
16
|
-
|
|
17
|
-
`querySpans()` returns one row per matching completed span across traces, including matching child spans. Use it to list the spans themselves, for example every failed tool call in the last day.
|
|
18
|
-
|
|
19
|
-
`queryTraces()` with a `spans.some` predicate answers a different question. It returns the traces that contain at least one matching span, and it doesn't identify which spans matched. Both use the same span fields and operators.
|
|
20
|
-
|
|
21
|
-
```typescript
|
|
22
|
-
const timeRange = {
|
|
23
|
-
from: '2026-10-01T00:00:00.000Z',
|
|
24
|
-
to: '2026-10-02T00:00:00.000Z',
|
|
25
|
-
}
|
|
26
|
-
const where = {
|
|
27
|
-
op: 'and',
|
|
28
|
-
args: [
|
|
29
|
-
{ op: 'eq', left: { path: 'spanType' }, right: { literal: 'tool_call' } },
|
|
30
|
-
{ op: 'eq', left: { path: 'status' }, right: { literal: 'error' } },
|
|
31
|
-
],
|
|
32
|
-
} as const
|
|
33
|
-
|
|
34
|
-
const firstPage = await mastraClient.querySpans({
|
|
35
|
-
timeRange,
|
|
36
|
-
where,
|
|
37
|
-
orderBy: [{ field: 'startedAt', direction: 'desc' }],
|
|
38
|
-
page: { limit: 50 },
|
|
39
|
-
})
|
|
40
|
-
|
|
41
|
-
if (firstPage.page.next) {
|
|
42
|
-
const secondPage = await mastraClient.querySpans({
|
|
43
|
-
timeRange,
|
|
44
|
-
where,
|
|
45
|
-
orderBy: [{ field: 'startedAt', direction: 'desc' }],
|
|
46
|
-
page: { limit: 50, after: firstPage.page.next },
|
|
47
|
-
})
|
|
48
|
-
}
|
|
49
|
-
```
|
|
50
|
-
|
|
51
|
-
The HTTP route accepts the same body:
|
|
52
|
-
|
|
53
|
-
```bash
|
|
54
|
-
curl --request POST \
|
|
55
|
-
--url http://localhost:4111/api/observability/spans/query \
|
|
56
|
-
--header 'Authorization: Bearer <token>' \
|
|
57
|
-
--header 'Content-Type: application/json' \
|
|
58
|
-
--data '{
|
|
59
|
-
"timeRange": {
|
|
60
|
-
"from": "2026-10-01T00:00:00.000Z",
|
|
61
|
-
"to": "2026-10-02T00:00:00.000Z"
|
|
62
|
-
},
|
|
63
|
-
"where": { "op": "eq", "left": { "path": "spanType" }, "right": { "literal": "tool_call" } },
|
|
64
|
-
"page": { "limit": 50 }
|
|
65
|
-
}'
|
|
66
|
-
```
|
|
67
|
-
|
|
68
|
-
Span queries are optional for observability stores. Call `mastraClient.getObservabilityCapabilities()` and check `capabilities.spanQuery` before showing a span list. Treat a missing `spanQuery` value as unsupported, because older servers don't report it. A server whose store or `@mastra/core` version doesn't support span queries returns `501` with code `SPAN_QUERY_UNSUPPORTED`.
|
|
69
|
-
|
|
70
|
-
The other statuses match trace queries: `400` for malformed JSON or a malformed cursor, `409` for a cursor that doesn't match the query, `413` for a body over 256 KiB, `422` with code `TRACE_QUERY_INVALID` and structured `issues` for an invalid query, `503` with code `TRACE_QUERY_RESOURCE_LIMIT` when a page exceeds a resource limit, and `504` with code `TRACE_QUERY_EXECUTION_TIMEOUT` when the database deadline is exceeded.
|
|
71
|
-
|
|
72
|
-
The server applies [trusted tenant scope](#trusted-tenant-scope) to span queries the same way it does for trace queries. A cursor created under one scope and reused under another returns `409`.
|
|
73
|
-
|
|
74
|
-
### Query individual spans through storage
|
|
75
|
-
|
|
76
|
-
The observability domain in DuckDB and the PostgreSQL and ClickHouse v-next stores supports `querySpans(plan)`. Check `storage.getFeatures()?.includes('span-query')` before using this optional storage method.
|
|
15
|
+
## Query traces with the client SDK
|
|
77
16
|
|
|
78
|
-
|
|
17
|
+
This query finds traces containing a tool call with an error:
|
|
79
18
|
|
|
80
19
|
```typescript
|
|
81
|
-
import {
|
|
82
|
-
|
|
83
|
-
const storage = await mastra.getStorage()?.getStore('observability')
|
|
84
|
-
if (!storage?.getFeatures()?.includes('span-query')) {
|
|
85
|
-
throw new Error('This store does not support span queries')
|
|
86
|
-
}
|
|
87
|
-
|
|
88
|
-
const plan = planSpanQuery({
|
|
89
|
-
timeRange: {
|
|
90
|
-
from: '2026-10-01T00:00:00.000Z',
|
|
91
|
-
to: '2026-10-02T00:00:00.000Z',
|
|
92
|
-
},
|
|
93
|
-
where: {
|
|
94
|
-
op: 'eq',
|
|
95
|
-
left: { path: 'spanType' },
|
|
96
|
-
right: { literal: 'tool_call' },
|
|
97
|
-
},
|
|
98
|
-
page: { limit: 50 },
|
|
99
|
-
})
|
|
100
|
-
|
|
101
|
-
const { spans, page } = await storage.querySpans(plan)
|
|
102
|
-
```
|
|
103
|
-
|
|
104
|
-
This example uses local, unscoped storage. For a shared database, pass `{ scope: { organizationId, resourceId } }` as the second argument to `planSpanQuery()`, using identifiers resolved by your backend's authorization. The request body can't supply trusted scope. Apply the same authorization when loading full span details with `getSpan({ traceId, spanId })`.
|
|
105
|
-
|
|
106
|
-
Span queries use the scalar fields and operators available inside `queryTraces()`'s `spans.some` predicate. They don't accept related-record clauses. Nested attributes require support in the shared predicate contract and each store.
|
|
107
|
-
|
|
108
|
-
The `timeRange` applies to each span's `startedAt`, includes `from`, excludes `to`, and can cover at most 31 days. Sort by `startedAt` or `endedAt`, in either direction. The default is `startedAt` descending. A page contains up to 1,000 rows, with a default of 100. Pass `page.next` back as `page.after` with the same filters, time range, scope, and ordering. Cursors use scope and span identifiers to break timestamp ties. They aren't a snapshot across requests. New completions can appear when the view is refreshed.
|
|
109
|
-
|
|
110
|
-
Each row contains span and trace identifiers, display fields, duration, status, and input/output previews of at most 256 Unicode characters. Truncation flags indicate when a preview is incomplete. Stores select the page before loading previews and cost. Full payloads remain available through the existing span detail method.
|
|
111
|
-
|
|
112
|
-
The `cost` field describes only that span's model-token cost:
|
|
113
|
-
|
|
114
|
-
- `available` includes `amount` and `currency`. Zero is a valid amount.
|
|
115
|
-
- `missing` means no matching model metrics were found.
|
|
116
|
-
- `unavailable` means metrics exist but a complete, unambiguous cost can't be established.
|
|
20
|
+
import { MastraClient } from '@mastra/client-js'
|
|
117
21
|
|
|
118
|
-
|
|
22
|
+
const mastraClient = new MastraClient({ baseUrl: 'http://localhost:4111' })
|
|
119
23
|
|
|
120
|
-
Current-record selection runs before mutable filters. PostgreSQL and ClickHouse select the greatest completed `endedAt`. PostgreSQL breaks ties with its ingestion cursor. DuckDB reconstructs the earliest start event and selects the latest ingested completed event, matching its existing span-predicate behavior. Time bounds are checked again after reconstruction. ClickHouse uses its existing completion-only, insert-only model, where equal-version retries must contain identical records. Querying spans doesn't add a table or materialized view.
|
|
121
|
-
|
|
122
|
-
Queries have a 15-second default execution deadline. PostgreSQL and ClickHouse use their existing `traceQuery.timeoutMs` setting. ClickHouse also caps each query at 512 MiB. A page with more than 50,000 matching model metrics fails with a resource-limit error instead of returning partial cost. A small page size limits returned rows, but doesn't limit the number of records the database must examine.
|
|
123
|
-
|
|
124
|
-
## Query traces with the client SDK
|
|
125
|
-
|
|
126
|
-
Pass the query to `queryTraces()`:
|
|
127
|
-
|
|
128
|
-
```typescript
|
|
129
24
|
const result = await mastraClient.queryTraces({
|
|
130
25
|
timeRange: {
|
|
131
26
|
from: '2026-08-01T00:00:00.000Z',
|
|
132
27
|
to: '2026-08-08T00:00:00.000Z',
|
|
133
28
|
},
|
|
134
29
|
where: {
|
|
135
|
-
|
|
30
|
+
spans: {
|
|
136
31
|
some: {
|
|
137
32
|
op: 'and',
|
|
138
33
|
args: [
|
|
139
|
-
{ op: 'eq', left: { path: '
|
|
140
|
-
{ op: '
|
|
141
|
-
{ op: 'lt', left: { path: 'score' }, right: { literal: 0.6 } },
|
|
34
|
+
{ op: 'eq', left: { path: 'spanType' }, right: { literal: 'tool_call' } },
|
|
35
|
+
{ op: 'exists', path: 'error' },
|
|
142
36
|
],
|
|
143
37
|
},
|
|
144
38
|
},
|
|
@@ -148,79 +42,56 @@ const result = await mastraClient.queryTraces({
|
|
|
148
42
|
})
|
|
149
43
|
```
|
|
150
44
|
|
|
151
|
-
A `some` clause matches when one
|
|
45
|
+
A `spans.some` clause matches when one span satisfies all its conditions. In this example, the same span must be a tool call and have an error. Use `spans.none` to find traces with no spans matching the conditions.
|
|
152
46
|
|
|
153
|
-
Span clauses examine the current root span and current child spans. The root `timeRange` applies only to the selected current root's `startedAt`. Related spans
|
|
47
|
+
Span clauses examine the current root span and current child spans. The root `timeRange` applies only to the selected current root's `startedAt`. Related spans can participate even when their own timestamps are outside that range. Related records correlate only through a matching non-null `traceId`.
|
|
154
48
|
|
|
155
|
-
## Query
|
|
49
|
+
## Query traces with the CLI
|
|
156
50
|
|
|
157
|
-
Pass
|
|
51
|
+
Pass the query as one JSON argument to `mastra api trace query`. Filters and pagination options belong in the JSON input. This command runs the same failed-tool query as the SDK example:
|
|
158
52
|
|
|
159
|
-
```
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
to: '2026-08-08T00:00:00.000Z',
|
|
165
|
-
},
|
|
166
|
-
where: {
|
|
167
|
-
op: 'eq',
|
|
168
|
-
left: { path: 'environment' },
|
|
169
|
-
right: { literal: 'production' },
|
|
170
|
-
},
|
|
53
|
+
```bash
|
|
54
|
+
mastra api --url http://localhost:4111 trace query '{
|
|
55
|
+
"timeRange": {
|
|
56
|
+
"from": "2026-08-01T00:00:00.000Z",
|
|
57
|
+
"to": "2026-08-08T00:00:00.000Z"
|
|
171
58
|
},
|
|
172
|
-
where: {
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
right: { literal: 0.6 },
|
|
183
|
-
},
|
|
184
|
-
},
|
|
185
|
-
},
|
|
186
|
-
},
|
|
187
|
-
},
|
|
188
|
-
{
|
|
189
|
-
traces: {
|
|
190
|
-
some: {
|
|
191
|
-
feedback: {
|
|
192
|
-
some: {
|
|
193
|
-
op: 'eq',
|
|
194
|
-
left: { path: 'feedbackType' },
|
|
195
|
-
right: { literal: 'clinician-correction' },
|
|
196
|
-
},
|
|
197
|
-
},
|
|
198
|
-
},
|
|
199
|
-
},
|
|
200
|
-
},
|
|
201
|
-
],
|
|
59
|
+
"where": {
|
|
60
|
+
"spans": {
|
|
61
|
+
"some": {
|
|
62
|
+
"op": "and",
|
|
63
|
+
"args": [
|
|
64
|
+
{ "op": "eq", "left": { "path": "spanType" }, "right": { "literal": "tool_call" } },
|
|
65
|
+
{ "op": "exists", "path": "error" }
|
|
66
|
+
]
|
|
67
|
+
}
|
|
68
|
+
}
|
|
202
69
|
},
|
|
203
|
-
|
|
204
|
-
}
|
|
70
|
+
"orderBy": [{ "field": "startedAt", "direction": "desc" }],
|
|
71
|
+
"page": { "limit": 25 }
|
|
72
|
+
}'
|
|
205
73
|
```
|
|
206
74
|
|
|
207
|
-
The
|
|
75
|
+
The explicit `--url` targets a local Mastra server. Without it, observability commands default to `https://observability.mastra.ai`, even when a local server is running. See [CLI target resolution](https://mastra.ai/reference/cli/mastra) for authentication and regional endpoints. The target server and configured store must support the requested query capabilities.
|
|
208
76
|
|
|
209
|
-
|
|
77
|
+
Inspect the command with:
|
|
210
78
|
|
|
211
|
-
|
|
79
|
+
```bash
|
|
80
|
+
mastra api trace query --help
|
|
81
|
+
```
|
|
212
82
|
|
|
213
|
-
|
|
83
|
+
For a local Mastra server, inspect its request and response schemas with:
|
|
214
84
|
|
|
215
|
-
```
|
|
216
|
-
|
|
217
|
-
"threads": [{ "threadId": "thread-123" }],
|
|
218
|
-
"page": { "next": null }
|
|
219
|
-
}
|
|
85
|
+
```bash
|
|
86
|
+
mastra api --url http://localhost:4111 trace query --schema
|
|
220
87
|
```
|
|
221
88
|
|
|
89
|
+
Mastra Platform doesn't currently expose the schema endpoint used by `--schema`. Use [Request fields](#request-fields) and [Fields and operators](#fields-and-operators) to construct queries for either target. To find fields and values recorded in your project, use [field and value discovery](#discover-fields-and-values).
|
|
90
|
+
|
|
222
91
|
## Send an HTTP request
|
|
223
92
|
|
|
93
|
+
Send the same query as a JSON request body:
|
|
94
|
+
|
|
224
95
|
```bash
|
|
225
96
|
curl --request POST \
|
|
226
97
|
--url http://localhost:4111/api/observability/traces/query \
|
|
@@ -241,133 +112,24 @@ curl --request POST \
|
|
|
241
112
|
]
|
|
242
113
|
}
|
|
243
114
|
}
|
|
244
|
-
}
|
|
115
|
+
},
|
|
116
|
+
"orderBy": [{ "field": "startedAt", "direction": "desc" }],
|
|
117
|
+
"page": { "limit": 25 }
|
|
245
118
|
}'
|
|
246
119
|
```
|
|
247
120
|
|
|
248
|
-
## Discover fields and values
|
|
249
|
-
|
|
250
|
-
Use trace-query discovery to build field and value autocomplete without scanning trace payloads or copying the query grammar into your client. The configured observability store must support `trace-query-discovery`.
|
|
251
|
-
|
|
252
|
-
Call `getTraceQueryFields()` for one predicate scope:
|
|
253
|
-
|
|
254
|
-
```typescript
|
|
255
|
-
const timeRange = {
|
|
256
|
-
from: '2026-08-01T00:00:00.000Z',
|
|
257
|
-
to: '2026-08-08T00:00:00.000Z',
|
|
258
|
-
}
|
|
259
|
-
|
|
260
|
-
const fields = await mastraClient.getTraceQueryFields({
|
|
261
|
-
timeRange,
|
|
262
|
-
predicateScope: 'trace',
|
|
263
|
-
search: 'region',
|
|
264
|
-
limit: 25,
|
|
265
|
-
})
|
|
266
|
-
```
|
|
267
|
-
|
|
268
|
-
The response separates canonical fields from observed metadata fields:
|
|
269
|
-
|
|
270
|
-
```json
|
|
271
|
-
{
|
|
272
|
-
"canonicalFields": [],
|
|
273
|
-
"observedFields": [
|
|
274
|
-
{
|
|
275
|
-
"path": "metadata.region",
|
|
276
|
-
"valueKind": "string",
|
|
277
|
-
"operators": ["eq", "ne", "in", "notIn", "exists", "notExists"],
|
|
278
|
-
"valueSuggestions": true,
|
|
279
|
-
"occurrences": 125
|
|
280
|
-
}
|
|
281
|
-
],
|
|
282
|
-
"observedFieldsTruncated": false
|
|
283
|
-
}
|
|
284
|
-
```
|
|
285
|
-
|
|
286
|
-
Canonical fields come from the same registry used to validate trace queries. They don't consume `limit`. The canonical order follows that registry. `observedFields` contains top-level string `metadata.<key>` paths found on qualifying current root spans. These fields are ordered by `occurrences` descending, then by path. Nested metadata, keys containing `.`, empty keys, non-string values, and oversized paths are omitted.
|
|
287
|
-
|
|
288
|
-
`predicateScope` selects the local predicate grammar. It isn't an authorization scope:
|
|
289
|
-
|
|
290
|
-
| `predicateScope` | Example returned path | Query usage |
|
|
291
|
-
| ---------------- | ---------------------------------- | ----------------------------------------- |
|
|
292
|
-
| `trace` | `environment` or `metadata.region` | Root `where` predicate |
|
|
293
|
-
| `spans` | `model` | Inside `spans.some` or `spans.none` |
|
|
294
|
-
| `scores` | `scorerId` | Inside `scores.some` or `scores.none` |
|
|
295
|
-
| `feedback` | `feedbackType` | Inside `feedback.some` or `feedback.none` |
|
|
296
|
-
|
|
297
|
-
A global field picker can request all four scopes in parallel and qualify display labels locally. Every returned path is directly accepted by `queryTraces()` in its matching scope. `metadata` by itself isn't a field. A user interface can offer manual metadata-key entry separately.
|
|
298
|
-
|
|
299
|
-
After a user selects one field with `valueSuggestions: true`, call `getTraceQueryValues()`:
|
|
300
|
-
|
|
301
|
-
```typescript
|
|
302
|
-
const abortController = new AbortController()
|
|
303
|
-
const suggestions = await mastraClient.getTraceQueryValues(
|
|
304
|
-
{
|
|
305
|
-
timeRange,
|
|
306
|
-
predicateScope: 'trace',
|
|
307
|
-
path: 'metadata.region',
|
|
308
|
-
search: 'west',
|
|
309
|
-
limit: 25,
|
|
310
|
-
},
|
|
311
|
-
{ signal: abortController.signal },
|
|
312
|
-
)
|
|
313
|
-
```
|
|
314
|
-
|
|
315
|
-
```json
|
|
316
|
-
{
|
|
317
|
-
"values": [
|
|
318
|
-
{ "value": "eu-west-1", "count": 82 },
|
|
319
|
-
{ "value": "us-west-2", "count": 43 }
|
|
320
|
-
],
|
|
321
|
-
"valuesTruncated": false
|
|
322
|
-
}
|
|
323
|
-
```
|
|
324
|
-
|
|
325
|
-
Discovery returns string values only. Missing values, empty strings, and strings larger than 4,096 UTF-8 bytes are omitted. Values are ordered by `count` descending, then by value. Suggestions are advisory: `queryTraces()` continues to accept valid manual literals that discovery doesn't return.
|
|
326
|
-
|
|
327
|
-
Value suggestions are available for these canonical fields:
|
|
328
|
-
|
|
329
|
-
| Scope | Fields |
|
|
330
|
-
| -------- | ----------------------------------------------------------------------------- |
|
|
331
|
-
| Trace | `entityName`, `entityType`, `environment`, `status`, `tags` |
|
|
332
|
-
| Spans | `name`, `spanType`, `model`, `provider`, `status`, `entityType`, `entityName` |
|
|
333
|
-
| Scores | `scorerId`, `scorerVersion`, `scoreSource` |
|
|
334
|
-
| Feedback | `feedbackType`, `feedbackSource` |
|
|
335
|
-
|
|
336
|
-
Each executable top-level string `metadata.<key>` field also supports value suggestions in the `trace` scope. Identifiers, timestamps, durations, numeric score or feedback values, comments, errors, and version IDs don't. The fields response still includes canonical fields with `valueSuggestions: false`. The values endpoint rejects those paths with `422 TRACE_QUERY_INVALID`.
|
|
337
|
-
|
|
338
|
-
Both discovery requests require a half-open time range of at most 31 days. They use the same current-record semantics as `queryTraces()`: qualifying completed roots satisfy `from <= startedAt < to`, and related values come only from current records joined to those roots by `traceId`. Discovery doesn't accept or apply a draft `where` predicate.
|
|
339
|
-
|
|
340
|
-
`search` defaults to the empty string, is trimmed, and matches case-insensitive literal substrings. Characters such as `%` and `_` have no wildcard meaning. An empty search returns the most frequent results. `limit` defaults to 25 and has a maximum of 100. Both endpoints fetch one extra result to set their truncation flag, but don't provide a cursor. When a flag is `true`, refine `search`. It doesn't imply that a next page is available. No matches return an empty array and `false` truncation.
|
|
341
|
-
|
|
342
|
-
If discovery exceeds its execution timeout, the route returns `504 TRACE_QUERY_EXECUTION_TIMEOUT`. If the backend exceeds a memory or resource budget, it returns `503 TRACE_QUERY_RESOURCE_LIMIT`. Neither failure returns partial suggestions or sets a truncation flag. Truncation only means a complete ranking was cut to the requested response limit.
|
|
343
|
-
|
|
344
|
-
ClickHouse discovery requests default to a 5-second timeout and a 256 MiB per-query memory limit. Configure these with `observability.traceQuery.discovery.timeoutMs` and `observability.traceQuery.discovery.memoryLimitBytes` on `ClickhouseStoreVNext`. Discovery falls back to `observability.traceQuery.timeoutMs` when its timeout isn't configured, without changing ordinary `queryTraces()` execution limits.
|
|
345
|
-
|
|
346
|
-
Both routes require `observability:read` and use the configured observability storage boundary. Don't send organization or project authorization fields in the request. Treat discovered paths, values, and counts as observability data.
|
|
347
|
-
|
|
348
121
|
## Request fields
|
|
349
122
|
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
|
|
|
353
|
-
|
|
|
354
|
-
| `
|
|
355
|
-
| `
|
|
356
|
-
| `
|
|
357
|
-
| `
|
|
358
|
-
| `
|
|
359
|
-
| `
|
|
360
|
-
| `mode` | No | Set to `'delta'` to poll using a delta cursor instead of `page` or `pagination`. |
|
|
361
|
-
| `after` | No | Delta mode only. Opaque `deltaCursor` from a numbered page or previous poll. Omit to establish the current watermark. |
|
|
362
|
-
| `limit` | No | Delta mode only. Maximum results per batch. Defaults to 10 and has a maximum of 100. |
|
|
363
|
-
|
|
364
|
-
### Thread queries
|
|
365
|
-
|
|
366
|
-
| Field | Required | Description |
|
|
367
|
-
| -------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
368
|
-
| `traces` | Yes | Eligible trace selection containing `timeRange` and an optional trace `where` predicate. |
|
|
369
|
-
| `where` | No | Recursive thread predicate composed with boolean operators and `traces.some` or `traces.none`. Each quantifier contains a complete trace predicate. |
|
|
370
|
-
| `page` | No | `{ limit, after }`. Thread identities always use fixed ordinal `threadId` ascending order. Pass the opaque `page.next` value as `after`. |
|
|
123
|
+
| Field | Required | Description |
|
|
124
|
+
| ------------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
125
|
+
| `timeRange` | Yes | Trace start-time boundary. `from` is inclusive and `to` is exclusive. Both values must be ISO timestamps, and `from` must be earlier than `to`. |
|
|
126
|
+
| `where` | No | Recursive trace predicate. Supports scalar conditions and `spans.some` or `spans.none` clauses. |
|
|
127
|
+
| `orderBy` | No | One item ordering results by `startedAt` or `endedAt`, in `asc` or `desc` order. Defaults to `startedAt desc`. Not accepted in delta mode. |
|
|
128
|
+
| `page` | No | `{ limit, after }`. `limit` defaults to 100 and has a maximum of 1000. Pass the opaque `page.next` value as `after`. |
|
|
129
|
+
| `pagination` | No | Numbered pages: `{ page, perPage }`. Defaults to page 0 and 10 results. `perPage` has a maximum of 100. |
|
|
130
|
+
| `mode` | No | Set to `'delta'` to poll using a delta cursor instead of `page` or `pagination`. |
|
|
131
|
+
| `after` | No | Delta mode only. Opaque `deltaCursor` from a numbered page or previous poll. Omit to establish the current watermark. |
|
|
132
|
+
| `limit` | No | Delta mode only. Maximum results per batch. Defaults to 10 and has a maximum of 100. |
|
|
371
133
|
|
|
372
134
|
Unknown fields are rejected. Requests can't select a projection, declare joins, request counts or measures, or control authorization. Request bodies are limited to 256 KiB.
|
|
373
135
|
|
|
@@ -375,18 +137,17 @@ Unknown fields are rejected. Requests can't select a projection, declare joins,
|
|
|
375
137
|
|
|
376
138
|
The planner rejects a query before storage execution when it exceeds any of these limits:
|
|
377
139
|
|
|
378
|
-
| Input
|
|
379
|
-
|
|
|
380
|
-
|
|
|
381
|
-
| Predicate
|
|
382
|
-
|
|
|
383
|
-
|
|
|
384
|
-
|
|
|
385
|
-
|
|
|
386
|
-
|
|
|
387
|
-
|
|
|
388
|
-
|
|
|
389
|
-
| HTTP request body | 256 KiB |
|
|
140
|
+
| Input | Maximum |
|
|
141
|
+
| ----------------------------------------- | ----------------- |
|
|
142
|
+
| Predicate nesting depth | 12 levels |
|
|
143
|
+
| Predicate nodes | 100 |
|
|
144
|
+
| Related `spans` clauses | 8 total |
|
|
145
|
+
| Values in one `in` or `notIn` set | 100 |
|
|
146
|
+
| Comparison literals and membership values | 1,000 total |
|
|
147
|
+
| String literal | 4,096 UTF-8 bytes |
|
|
148
|
+
| Raw predicate path | 128 UTF-8 bytes |
|
|
149
|
+
| `page.limit` | 1,000 |
|
|
150
|
+
| HTTP request body | 256 KiB |
|
|
390
151
|
|
|
391
152
|
Each comparison literal counts as one literal. Each member of an `in` or `notIn` set also counts as one literal, even when the set is within its per-set limit.
|
|
392
153
|
|
|
@@ -404,16 +165,10 @@ Hono and Fastify enforce the request-body limit before JSON parsing. Express and
|
|
|
404
165
|
| Span | `spanType`, `model`, `provider`, `status`, `entityType`, `entityId`, `entityName`, `entityVersionId`, `parentEntityVersionId`, `rootEntityVersionId`, `runId`, `sessionId`, `userId`, `organizationId` | `eq`, `ne`, `in`, `notIn`, `exists`, `notExists` |
|
|
405
166
|
| Span | `startedAt`, `endedAt`, `durationMs` | `eq`, `ne`, `in`, `notIn`, `lt`, `lte`, `gt`, `gte`, `exists`, `notExists` |
|
|
406
167
|
| Span | `error` | `exists`, `notExists` |
|
|
407
|
-
| Score | `scorerId`, `scorerVersion`, `scoreSource`, `entityVersionId`, `parentEntityVersionId`, `rootEntityVersionId` | `eq`, `ne`, `in`, `notIn`, `exists`, `notExists` |
|
|
408
|
-
| Score | `score`, `timestamp` | `eq`, `ne`, `in`, `notIn`, `lt`, `lte`, `gt`, `gte`, `exists`, `notExists` |
|
|
409
|
-
| Score | `spanId` | `exists`, `notExists` |
|
|
410
|
-
| Feedback | `feedbackType`, `feedbackSource`, `feedbackUserId`, `sourceId`, `entityVersionId`, `parentEntityVersionId`, `rootEntityVersionId` | `eq`, `ne`, `in`, `notIn`, `exists`, `notExists` |
|
|
411
|
-
| Feedback | `value`, `timestamp` | `eq`, `ne`, `in`, `notIn`, `lt`, `lte`, `gt`, `gte`, `exists`, `notExists` |
|
|
412
|
-
| Feedback | `comment` | `eq`, `ne`, `in`, `notIn`, `exists`, `notExists`, `matches`, `notMatches` |
|
|
413
168
|
|
|
414
169
|
Compose predicates with `{ op: 'and', args: [...] }`, `{ op: 'or', args: [...] }`, and `{ op: 'not', arg: ... }`. Comparison predicates place a field reference on the left and a literal on the right. Membership predicates use a field reference in `value` and a homogeneous literal array in `set`. Tag predicates name the stored tag collection in `path`. `includes` and `notIncludes` also take one tag in `value`, while `exists` and `notExists` take only `op` and `path`. See [Filter by tags](#filter-by-tags). Text predicates use the comparison shape with a string literal. See [Filter by text](#filter-by-text).
|
|
415
170
|
|
|
416
|
-
String comparisons are case-sensitive, and literals are never coerced. `matches` and `notMatches` are the exception: they compare words case-insensitively. Canonical string fields compare exact stored values. Metadata string values are trimmed before comparison, as described below.
|
|
171
|
+
String comparisons are case-sensitive, and literals are never coerced. `matches` and `notMatches` are the exception: they compare words case-insensitively. Canonical string fields compare exact stored values. Metadata string values are trimmed before comparison, as described below. Duration predicates require numbers, while timestamp predicates require ISO timestamp strings. A missing value satisfies neither positive nor ordered predicates, although it does satisfy the negative operators `ne` and `notIn`. Combine a negative predicate with `exists` when the field must also be present.
|
|
417
172
|
|
|
418
173
|
### Filter by root duration
|
|
419
174
|
|
|
@@ -532,53 +287,6 @@ const versionedRetrieval = {
|
|
|
532
287
|
}
|
|
533
288
|
```
|
|
534
289
|
|
|
535
|
-
### Filter by score time
|
|
536
|
-
|
|
537
|
-
Score timestamps are independent of the required top-level trace `timeRange`. The top-level range selects trace candidates by trace start time. A nested `timestamp` predicate limits related score records:
|
|
538
|
-
|
|
539
|
-
```typescript
|
|
540
|
-
const query = {
|
|
541
|
-
timeRange: {
|
|
542
|
-
from: '2026-08-01T00:00:00.000Z',
|
|
543
|
-
to: '2026-08-08T00:00:00.000Z',
|
|
544
|
-
},
|
|
545
|
-
where: {
|
|
546
|
-
scores: {
|
|
547
|
-
some: {
|
|
548
|
-
op: 'and',
|
|
549
|
-
args: [
|
|
550
|
-
{ op: 'eq', left: { path: 'scoreSource' }, right: { literal: 'automated' } },
|
|
551
|
-
{
|
|
552
|
-
op: 'gte',
|
|
553
|
-
left: { path: 'timestamp' },
|
|
554
|
-
right: { literal: '2026-07-15T00:00:00.000Z' },
|
|
555
|
-
},
|
|
556
|
-
{
|
|
557
|
-
op: 'lt',
|
|
558
|
-
left: { path: 'timestamp' },
|
|
559
|
-
right: { literal: '2026-08-01T00:00:00.000Z' },
|
|
560
|
-
},
|
|
561
|
-
],
|
|
562
|
-
},
|
|
563
|
-
},
|
|
564
|
-
},
|
|
565
|
-
}
|
|
566
|
-
```
|
|
567
|
-
|
|
568
|
-
### Filter by span anchoring
|
|
569
|
-
|
|
570
|
-
Use `spanId` only to test whether a score targets a span. The predicate doesn't join the score to a matching span predicate:
|
|
571
|
-
|
|
572
|
-
```typescript
|
|
573
|
-
// At least one score targets a span.
|
|
574
|
-
const anchoredToSpanWhere = { scores: { some: { op: 'exists', path: 'spanId' } } }
|
|
575
|
-
|
|
576
|
-
// At least one score applies to the trace rather than a span.
|
|
577
|
-
const traceLevelScoreWhere = { scores: { some: { op: 'notExists', path: 'spanId' } } }
|
|
578
|
-
```
|
|
579
|
-
|
|
580
|
-
The deprecated `source` field and `scorerName` aren't available as score predicate fields. Use the canonical `scoreSource` and `scorerId` fields.
|
|
581
|
-
|
|
582
290
|
### Filter by metadata
|
|
583
291
|
|
|
584
292
|
Use `metadata.<key>` paths for custom dimensions stored on the current root span. Metadata predicates compare trimmed, non-empty string values exactly and case-sensitively. Leading and trailing whitespace in a stored string value is ignored. Values empty after trimming count as missing. Null, numeric, boolean, object, and array values also count as missing.
|
|
@@ -604,7 +312,7 @@ const messageTrace = {
|
|
|
604
312
|
|
|
605
313
|
The key must name one top-level property. Empty keys and nested paths are rejected. Metadata keys aren't trimmed, so leading and trailing whitespace remains part of the exact key identity. When metadata duplicates a canonical trace field, such as `resourceId`, `threadId`, or `environment`, prefer the canonical field because it uses the dedicated storage column. The `metadata.<key>` form remains available when you specifically need the value from the metadata object. Metadata and canonical values may differ.
|
|
606
314
|
|
|
607
|
-
|
|
315
|
+
Trace-query discovery returns executable top-level string metadata fields observed in the selected time range.
|
|
608
316
|
|
|
609
317
|
### Filter by tags
|
|
610
318
|
|
|
@@ -630,19 +338,19 @@ const manualReview = {
|
|
|
630
338
|
|
|
631
339
|
Tags compare exactly and case-sensitively as whole strings. `value` must be one string with at least one non-whitespace character. Combine several `includes` predicates with `and` or `or` to require or allow multiple tags. A trace with no recorded tags and a trace with an empty tag list behave the same: both satisfy `notExists`, neither satisfies `includes`, `notIncludes`, or `exists`. Use `{ op: 'not', arg: { op: 'includes', ... } }` when untagged traces should also match.
|
|
632
340
|
|
|
633
|
-
`tags` is available in trace predicates only
|
|
341
|
+
`tags` is available in trace predicates only. `in`, `notIn`, and the comparison operators are rejected for `tags`. Value discovery for `tags` returns each observed tag with the number of qualifying traces that carry it.
|
|
634
342
|
|
|
635
343
|
### Filter by text
|
|
636
344
|
|
|
637
|
-
|
|
345
|
+
Span names support word matching in addition to exact comparison. `matches` succeeds when the literal's words appear in the field, in order, as whole words. `notMatches` succeeds when the field exists but doesn't contain those words. This query finds traces with a span whose name contains the words "document search":
|
|
638
346
|
|
|
639
347
|
```typescript
|
|
640
|
-
const
|
|
641
|
-
|
|
348
|
+
const documentSearch = {
|
|
349
|
+
spans: {
|
|
642
350
|
some: {
|
|
643
351
|
op: 'matches',
|
|
644
|
-
left: { path: '
|
|
645
|
-
right: { literal: '
|
|
352
|
+
left: { path: 'name' },
|
|
353
|
+
right: { literal: 'document search' },
|
|
646
354
|
},
|
|
647
355
|
},
|
|
648
356
|
}
|
|
@@ -655,50 +363,11 @@ Both the literal and the stored value are lowercased and split into words. A wor
|
|
|
655
363
|
| `matches` | The field contains the literal's words as a contiguous whole-word phrase |
|
|
656
364
|
| `notMatches` | The field is present and doesn't contain that phrase |
|
|
657
365
|
|
|
658
|
-
The literal must contain at least one letter or digit; otherwise the request fails with `422 TRACE_QUERY_INVALID`. A missing field satisfies neither operator. Use `{ op: 'not', arg: { op: 'matches', ... } }` when records without the field should also match. Text operators are available on span `name
|
|
659
|
-
|
|
660
|
-
### Filter by feedback
|
|
661
|
-
|
|
662
|
-
Every condition inside one `feedback.some` or `feedback.none` clause applies to the same current feedback record. `feedbackType` and `feedbackSource` are exact application-defined strings rather than built-in enums. This query finds traces with a numeric patient rating below zero:
|
|
663
|
-
|
|
664
|
-
```typescript
|
|
665
|
-
const negativePatientRating = {
|
|
666
|
-
feedback: {
|
|
667
|
-
some: {
|
|
668
|
-
op: 'and',
|
|
669
|
-
args: [
|
|
670
|
-
{ op: 'eq', left: { path: 'feedbackType' }, right: { literal: 'rating' } },
|
|
671
|
-
{ op: 'eq', left: { path: 'feedbackSource' }, right: { literal: 'patient' } },
|
|
672
|
-
{ op: 'lt', left: { path: 'value' }, right: { literal: 0 } },
|
|
673
|
-
],
|
|
674
|
-
},
|
|
675
|
-
},
|
|
676
|
-
}
|
|
677
|
-
```
|
|
678
|
-
|
|
679
|
-
Strict stored-value types are the portable contract for feedback predicates. PostgreSQL and ClickHouse distinguish numeric `3` from textual `'3'` for equality and ordered comparisons. DuckDB currently persists feedback values as `VARCHAR`, so numeric-looking strings may be coerced for equality and ordered numeric predicates. OBS-306 will remove this DuckDB exception through typed persistence. Ordered operators require a finite numeric literal. `eq` and `ne` accept one string or number, while `in` and `notIn` require a non-empty set containing only strings or only numbers. `exists` and `notExists` test for either value type.
|
|
680
|
-
|
|
681
|
-
Use `none` to select traces without a matching record. Traces with no feedback also match:
|
|
682
|
-
|
|
683
|
-
```typescript
|
|
684
|
-
const missingClinicianReview = {
|
|
685
|
-
feedback: {
|
|
686
|
-
none: {
|
|
687
|
-
op: 'and',
|
|
688
|
-
args: [
|
|
689
|
-
{ op: 'eq', left: { path: 'feedbackType' }, right: { literal: 'clinical-review' } },
|
|
690
|
-
{ op: 'eq', left: { path: 'feedbackSource' }, right: { literal: 'clinician' } },
|
|
691
|
-
],
|
|
692
|
-
},
|
|
693
|
-
},
|
|
694
|
-
}
|
|
695
|
-
```
|
|
696
|
-
|
|
697
|
-
Feedback `timestamp` predicates are independent of the root `timeRange`. `comment` supports exact string operators and word matching with `matches` and `notMatches`. See [Filter by text](#filter-by-text). The deprecated feedback fields `source` and `userId` aren't available. Use `feedbackSource` and `feedbackUserId`.
|
|
366
|
+
The literal must contain at least one letter or digit; otherwise the request fails with `422 TRACE_QUERY_INVALID`. A missing field satisfies neither operator. Use `{ op: 'not', arg: { op: 'matches', ... } }` when records without the field should also match. Text operators are available on span `name`. Other string fields, numbers, timestamps, tags, and metadata reject them. Substring, wildcard, regular-expression, and fuzzy matching aren't supported, and there is no search across several fields.
|
|
698
367
|
|
|
699
368
|
## Responses
|
|
700
369
|
|
|
701
|
-
|
|
370
|
+
A query using keyset pagination returns lightweight completed traces and a cursor for the next page:
|
|
702
371
|
|
|
703
372
|
```json
|
|
704
373
|
{
|
|
@@ -720,38 +389,26 @@ An ungrouped query returns only lightweight completed traces:
|
|
|
720
389
|
}
|
|
721
390
|
```
|
|
722
391
|
|
|
723
|
-
|
|
724
|
-
|
|
725
|
-
```typescript
|
|
726
|
-
const timeRange = {
|
|
727
|
-
from: '2026-08-01T00:00:00.000Z',
|
|
728
|
-
to: '2026-08-08T00:00:00.000Z',
|
|
729
|
-
}
|
|
392
|
+
The response doesn't include the related spans that matched the predicate. Use [`getTrace()`](https://mastra.ai/reference/client-js/observability) to load the full trace after selecting a result.
|
|
730
393
|
|
|
731
|
-
|
|
732
|
-
const legacyResult = await mastraClient.queryTraces({
|
|
733
|
-
timeRange,
|
|
734
|
-
group: { by: ['threadId'] },
|
|
735
|
-
})
|
|
736
|
-
// { groups: [{ threadId: 'thread-123' }], page: { next: null } }
|
|
737
|
-
|
|
738
|
-
// Replacement
|
|
739
|
-
const result = await mastraClient.queryTraceThreads({
|
|
740
|
-
traces: { timeRange },
|
|
741
|
-
})
|
|
742
|
-
// { threads: [{ threadId: 'thread-123' }], page: { next: null } }
|
|
743
|
-
```
|
|
394
|
+
### CLI responses
|
|
744
395
|
|
|
745
|
-
|
|
396
|
+
The CLI wraps the API response in `data`:
|
|
746
397
|
|
|
747
398
|
```json
|
|
748
399
|
{
|
|
749
|
-
"
|
|
750
|
-
|
|
400
|
+
"data": {
|
|
401
|
+
"traces": [],
|
|
402
|
+
"page": { "next": null }
|
|
403
|
+
}
|
|
751
404
|
}
|
|
752
405
|
```
|
|
753
406
|
|
|
754
|
-
|
|
407
|
+
Read results from `data.traces`. Pagination metadata is available at `data.page.next` for keyset queries, `data.pagination` for numbered pages, and `data.deltaCursor` and `data.delta.hasMore` for delta polling.
|
|
408
|
+
|
|
409
|
+
Each CLI invocation makes one query. To fetch another keyset page, pass `data.page.next` as `page.after` in the next request. For delta polling, repeat the command with `data.deltaCursor` as the top-level `after` value. Keep the same time range and predicate. See [Pagination and errors](#pagination-and-errors) for mode-specific behavior and polling limitations.
|
|
410
|
+
|
|
411
|
+
To inspect a selected trace, use `mastra api --url http://localhost:4111 trace get <traceId> --verbose`. Use the same target and authentication as the query.
|
|
755
412
|
|
|
756
413
|
## Pagination and errors
|
|
757
414
|
|
|
@@ -765,7 +422,7 @@ Trace queries support three mutually exclusive modes:
|
|
|
765
422
|
| Numbered pages | `pagination: { page, perPage }` | `pagination: { total, page, perPage, hasMore }` and, when delta polling is supported, `deltaCursor` |
|
|
766
423
|
| Delta polling | `mode: 'delta'`, optional top-level `after` and `limit` | `delta: { limit, hasMore }` and `deltaCursor` |
|
|
767
424
|
|
|
768
|
-
Numbered pages are zero-based. Their total and rows are read consistently within each request. Don't combine `page`, `pagination`, or delta mode. Top-level `after` and `limit` are accepted only in delta mode.
|
|
425
|
+
Numbered pages are zero-based. Their total and rows are read consistently within each request. Don't combine `page`, `pagination`, or delta mode. Top-level `after` and `limit` are accepted only in delta mode.
|
|
769
426
|
|
|
770
427
|
### Poll after loading a numbered page
|
|
771
428
|
|
|
@@ -811,7 +468,7 @@ Delta ordering uses the storage ingestion watermark and adapter-specific tie-bre
|
|
|
811
468
|
|
|
812
469
|
Keep the same normalized `where` predicate and exact `timeRange` bounds for every poll. The range continues to select root `startedAt`, even when a trace completes later. Changing `timeRange.to` to the current time invalidates the cursor. Reload a numbered page and use its cursor whenever the selection or authorization scope changes. You can change the batch limit without restarting.
|
|
813
470
|
|
|
814
|
-
New completed roots and roots completing after the cursor can be returned. Related spans
|
|
471
|
+
New completed roots and roots completing after the cursor can be returned. Related spans are evaluated when a root is selected, but later related-record writes don't guarantee that the trace is emitted again. Deleted traces and traces that stop matching aren't returned as removals or tombstones. Refresh numbered pages when you need to reconcile those changes.
|
|
815
472
|
|
|
816
473
|
ClickHouse polling is best effort. Its delta index and trace records use separate materialized-view tables. Concurrent queries can observe an insert in one table before another, as described in [ClickHouse's materialized-view visibility rules](https://github.com/ClickHouse/clickhouse-docs/blob/main/knowledgebase/are_materialized_views_inserted_asynchronously.mdx). A poll can advance past an index entry before the matching trace becomes visible and miss that trace in later polls. Reload numbered pages periodically to reconcile these gaps.
|
|
817
474
|
|
|
@@ -819,17 +476,17 @@ ClickHouse's delta index retains two days of events and doesn't backfill histori
|
|
|
819
476
|
|
|
820
477
|
### Keyset ordering and errors
|
|
821
478
|
|
|
822
|
-
Ordering is deterministic. Trace ordering appends `traceId` ascending as a tie-breaker.
|
|
479
|
+
Ordering is deterministic. Trace ordering appends `traceId` ascending as a tie-breaker.
|
|
823
480
|
|
|
824
|
-
Keyset cursors are bound to the operation, accepted normalized query, and ordering. Reusing a keyset cursor after changing the trace selection, predicates, or ordering returns `409`.
|
|
481
|
+
Keyset cursors are bound to the operation, accepted normalized query, and ordering. Reusing a keyset cursor after changing the trace selection, predicates, or ordering returns `409`.
|
|
825
482
|
|
|
826
483
|
Numbered-page handoff cursors and delta cursors also bind the authorization state. Changes to the caller's roles or permissions invalidate these cursors and return `409`. If a delta poll returns `409`, reload the numbered pages and resume polling with the new `deltaCursor`. A malformed cursor returns `400`.
|
|
827
484
|
|
|
828
485
|
### Trusted tenant scope
|
|
829
486
|
|
|
830
|
-
Hosts that serve more than one tenant supply a trusted scope outside the query document. The server reads the reserved `organizationId` request-context key, which only server-side authentication can set, and passes `{ organizationId }` to the planner. The scope is carried on the trusted plan and ANDed into every scan: current roots, related spans
|
|
487
|
+
Hosts that serve more than one tenant supply a trusted scope outside the query document. The server reads the reserved `organizationId` request-context key, which only server-side authentication can set, and passes `{ organizationId }` to the planner. The scope is carried on the trusted plan and ANDed into every scan: current roots, related spans and both discovery routes. A related record from another tenant that shares a `traceId` never qualifies a trace.
|
|
831
488
|
|
|
832
|
-
Hosts that call storage directly can pass a fuller scope to `planTraceQuery()`, `
|
|
489
|
+
Hosts that call storage directly can pass a fuller scope to `planTraceQuery()`, `planTraceQueryObservedFields()`, and `planTraceQueryValues()`:
|
|
833
490
|
|
|
834
491
|
```typescript
|
|
835
492
|
const plan = planTraceQuery(parseTraceQueryRequest(request), {
|
|
@@ -841,7 +498,7 @@ const plan = planTraceQuery(parseTraceQueryRequest(request), {
|
|
|
841
498
|
|
|
842
499
|
Cursor pagination is deterministic, but it isn't a database snapshot. Traces or replacement signals written between page requests can change later pages.
|
|
843
500
|
|
|
844
|
-
PostgreSQL and ClickHouse stop
|
|
501
|
+
PostgreSQL and ClickHouse stop trace queries after 15 seconds by default and return `504` when the database timeout is exceeded. Set `traceQueryTimeoutMs` in the store's vNext observability configuration to an integer from 1 through 300,000 milliseconds to change the timeout. DuckDB doesn't currently provide query-scoped timeout or cancellation through its driver wrapper, so this `504` guarantee doesn't apply to DuckDB.
|
|
845
502
|
|
|
846
503
|
| Status | Meaning |
|
|
847
504
|
| ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
@@ -853,9 +510,107 @@ PostgreSQL and ClickHouse stop advanced trace and thread queries after 15 second
|
|
|
853
510
|
| `503` | Discovery exceeded a backend memory or resource budget and returned `TRACE_QUERY_RESOURCE_LIMIT`. |
|
|
854
511
|
| `504` | A PostgreSQL or ClickHouse query exceeded its configured database execution timeout. Discovery returns `TRACE_QUERY_EXECUTION_TIMEOUT`. |
|
|
855
512
|
|
|
513
|
+
## Discover fields and values
|
|
514
|
+
|
|
515
|
+
Use trace-query discovery to build field and value autocomplete without scanning trace payloads or copying the query grammar into your client. The configured observability store must support `trace-query-discovery`.
|
|
516
|
+
|
|
517
|
+
The CLI doesn't expose dedicated discovery commands, so use the SDK methods below or the HTTP endpoints `POST /api/observability/traces/query/fields` and `POST /api/observability/traces/query/values`.
|
|
518
|
+
|
|
519
|
+
Call `getTraceQueryFields()` for one predicate scope:
|
|
520
|
+
|
|
521
|
+
```typescript
|
|
522
|
+
const timeRange = {
|
|
523
|
+
from: '2026-08-01T00:00:00.000Z',
|
|
524
|
+
to: '2026-08-08T00:00:00.000Z',
|
|
525
|
+
}
|
|
526
|
+
|
|
527
|
+
const fields = await mastraClient.getTraceQueryFields({
|
|
528
|
+
timeRange,
|
|
529
|
+
predicateScope: 'trace',
|
|
530
|
+
search: 'region',
|
|
531
|
+
limit: 25,
|
|
532
|
+
})
|
|
533
|
+
```
|
|
534
|
+
|
|
535
|
+
The response separates canonical fields from observed metadata fields:
|
|
536
|
+
|
|
537
|
+
```json
|
|
538
|
+
{
|
|
539
|
+
"canonicalFields": [],
|
|
540
|
+
"observedFields": [
|
|
541
|
+
{
|
|
542
|
+
"path": "metadata.region",
|
|
543
|
+
"valueKind": "string",
|
|
544
|
+
"operators": ["eq", "ne", "in", "notIn", "exists", "notExists"],
|
|
545
|
+
"valueSuggestions": true,
|
|
546
|
+
"occurrences": 125
|
|
547
|
+
}
|
|
548
|
+
],
|
|
549
|
+
"observedFieldsTruncated": false
|
|
550
|
+
}
|
|
551
|
+
```
|
|
552
|
+
|
|
553
|
+
Canonical fields come from the same registry used to validate trace queries. They don't consume `limit`. The canonical order follows that registry. `observedFields` contains top-level string `metadata.<key>` paths found on qualifying current root spans. These fields are ordered by `occurrences` descending, then by path. Nested metadata, keys containing `.`, empty keys, non-string values, and oversized paths are omitted.
|
|
554
|
+
|
|
555
|
+
`predicateScope` selects the local predicate grammar. It isn't an authorization scope:
|
|
556
|
+
|
|
557
|
+
| `predicateScope` | Example returned path | Query usage |
|
|
558
|
+
| ---------------- | ---------------------------------- | ----------------------------------- |
|
|
559
|
+
| `trace` | `environment` or `metadata.region` | Root `where` predicate |
|
|
560
|
+
| `spans` | `model` | Inside `spans.some` or `spans.none` |
|
|
561
|
+
|
|
562
|
+
A field picker can request the `trace` and `spans` scopes in parallel and label each field with its scope. Every returned path is directly accepted by `queryTraces()` in its matching scope. `metadata` by itself isn't a field. A user interface can offer manual metadata-key entry separately.
|
|
563
|
+
|
|
564
|
+
After a user selects one field with `valueSuggestions: true`, call `getTraceQueryValues()`:
|
|
565
|
+
|
|
566
|
+
```typescript
|
|
567
|
+
const abortController = new AbortController()
|
|
568
|
+
const suggestions = await mastraClient.getTraceQueryValues(
|
|
569
|
+
{
|
|
570
|
+
timeRange,
|
|
571
|
+
predicateScope: 'trace',
|
|
572
|
+
path: 'metadata.region',
|
|
573
|
+
search: 'west',
|
|
574
|
+
limit: 25,
|
|
575
|
+
},
|
|
576
|
+
{ signal: abortController.signal },
|
|
577
|
+
)
|
|
578
|
+
```
|
|
579
|
+
|
|
580
|
+
```json
|
|
581
|
+
{
|
|
582
|
+
"values": [
|
|
583
|
+
{ "value": "eu-west-1", "count": 82 },
|
|
584
|
+
{ "value": "us-west-2", "count": 43 }
|
|
585
|
+
],
|
|
586
|
+
"valuesTruncated": false
|
|
587
|
+
}
|
|
588
|
+
```
|
|
589
|
+
|
|
590
|
+
Discovery returns string values only. Missing values, empty strings, and strings larger than 4,096 UTF-8 bytes are omitted. Values are ordered by `count` descending, then by value. Suggestions are advisory: `queryTraces()` continues to accept valid manual literals that discovery doesn't return.
|
|
591
|
+
|
|
592
|
+
Value suggestions are available for these canonical fields:
|
|
593
|
+
|
|
594
|
+
| Scope | Fields |
|
|
595
|
+
| ----- | ----------------------------------------------------------------------------- |
|
|
596
|
+
| Trace | `entityName`, `entityType`, `environment`, `status`, `tags` |
|
|
597
|
+
| Spans | `name`, `spanType`, `model`, `provider`, `status`, `entityType`, `entityName` |
|
|
598
|
+
|
|
599
|
+
Each executable top-level string `metadata.<key>` field also supports value suggestions in the `trace` scope. Identifiers, timestamps, durations, errors, and version IDs don't. The fields response still includes canonical fields with `valueSuggestions: false`. The values endpoint rejects those paths with `422 TRACE_QUERY_INVALID`.
|
|
600
|
+
|
|
601
|
+
Both discovery requests require a half-open time range. They use the same current-record semantics as `queryTraces()`: qualifying completed roots satisfy `from <= startedAt < to`, and related values come only from current records joined to those roots by `traceId`. Discovery doesn't accept or apply a draft `where` predicate.
|
|
602
|
+
|
|
603
|
+
`search` defaults to the empty string, is trimmed, and matches case-insensitive literal substrings. Characters such as `%` and `_` have no wildcard meaning. An empty search returns the most frequent results. `limit` defaults to 25 and has a maximum of 100. Both endpoints fetch one extra result to set their truncation flag, but don't provide a cursor. When a flag is `true`, refine `search`. It doesn't imply that a next page is available. No matches return an empty array and `false` truncation.
|
|
604
|
+
|
|
605
|
+
If discovery exceeds its execution timeout, the route returns `504 TRACE_QUERY_EXECUTION_TIMEOUT`. If the backend exceeds a memory or resource budget, it returns `503 TRACE_QUERY_RESOURCE_LIMIT`. Neither failure returns partial suggestions or sets a truncation flag. Truncation only means a complete ranking was cut to the requested response limit.
|
|
606
|
+
|
|
607
|
+
ClickHouse discovery requests default to a 5-second timeout and a 256 MiB per-query memory limit. Configure these with `observability.traceQuery.discovery.timeoutMs` and `observability.traceQuery.discovery.memoryLimitBytes` on `ClickhouseStoreVNext`. Discovery falls back to `observability.traceQuery.timeoutMs` when its timeout isn't configured, without changing ordinary `queryTraces()` execution limits.
|
|
608
|
+
|
|
609
|
+
Both routes require `observability:read` and use the configured observability storage boundary. Don't send organization or project authorization fields in the request. Treat discovered paths, values, and counts as observability data.
|
|
610
|
+
|
|
856
611
|
## Limitations
|
|
857
612
|
|
|
858
|
-
Trace
|
|
613
|
+
Trace queries consider completed traces only. They don't support running traces, custom projections, embedded evidence, summaries, aggregations, measures, or custom grouping. Numbered trace pages include a total count.
|
|
859
614
|
|
|
860
615
|
## Related
|
|
861
616
|
|