@mastra/mcp-docs-server 1.3.0 → 1.3.1-alpha.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.docs/docs/evals/running-in-ci.md +7 -5
- package/.docs/docs/evals/vitest-integration.md +94 -31
- package/.docs/integrations/observability/opentelemetry.md +1 -0
- package/.docs/reference/evals/run-evals.md +24 -18
- package/.docs/reference/evals/trajectory-accuracy.md +20 -16
- package/.docs/reference/observability/tracing/bridges/otel.md +12 -3
- package/.docs/reference/rag/vector-databases.md +23 -0
- package/.docs/reference/vectors/mongodb.md +90 -5
- package/dist/index.js +1 -1
- package/dist/{src-CGZ6-uLS.js → src-Baf8l9Sp.js} +17 -7
- package/dist/src-Baf8l9Sp.js.map +1 -0
- package/dist/stdio.js +1 -1
- package/dist/tools/embedded-docs.d.ts.map +1 -1
- package/package.json +3 -3
- package/dist/src-CGZ6-uLS.js.map +0 -1
|
@@ -14,11 +14,13 @@ You can use any testing framework that supports ESM modules, such as [Vitest](ht
|
|
|
14
14
|
|
|
15
15
|
Use `runEvals` to evaluate your agent against multiple test cases. The function accepts an array of data items, each containing an `input` and optional `groundTruth` for scorer validation.
|
|
16
16
|
|
|
17
|
+
Resolve the `target` with `mastra.getAgent()` or `mastra.getWorkflow()` rather than importing it directly. A directly imported agent or workflow has no Mastra instance attached, so registry lookups inside steps or tools (such as `mastra.getAgent('weatherAgent')`) fail, scores aren't persisted, and trace-based trajectory scoring is unavailable.
|
|
18
|
+
|
|
17
19
|
```typescript
|
|
18
20
|
import { describe, it, expect } from 'vitest'
|
|
19
|
-
import {
|
|
20
|
-
import {
|
|
21
|
-
import { locationScorer } from '../scorers/location-scorer'
|
|
21
|
+
import { runEvals } from '@mastra/core/evals'
|
|
22
|
+
import { mastra } from '../src/mastra'
|
|
23
|
+
import { locationScorer } from '../src/mastra/scorers/location-scorer'
|
|
22
24
|
|
|
23
25
|
describe('Weather Agent Tests', () => {
|
|
24
26
|
it('should correctly extract locations from queries', async () => {
|
|
@@ -37,7 +39,7 @@ describe('Weather Agent Tests', () => {
|
|
|
37
39
|
groundTruth: { expectedLocation: 'Berlin', expectedCountry: 'RU' },
|
|
38
40
|
},
|
|
39
41
|
],
|
|
40
|
-
target: weatherAgent,
|
|
42
|
+
target: mastra.getAgent('weatherAgent'),
|
|
41
43
|
scorers: [locationScorer],
|
|
42
44
|
})
|
|
43
45
|
|
|
@@ -73,7 +75,7 @@ Create separate test cases for different evaluation scenarios:
|
|
|
73
75
|
|
|
74
76
|
```typescript
|
|
75
77
|
describe('Weather Agent Tests', () => {
|
|
76
|
-
const
|
|
78
|
+
const weatherAgent = mastra.getAgent('weatherAgent')
|
|
77
79
|
|
|
78
80
|
it('should handle location disambiguation', async () => {
|
|
79
81
|
const result = await runEvals({
|
|
@@ -30,6 +30,29 @@ export default defineConfig({
|
|
|
30
30
|
|
|
31
31
|
The setup file registers the custom matchers on `expect`. Alternatively, call `registerEvalMatchers()` from `@mastra/evals/vitest` in your own setup file.
|
|
32
32
|
|
|
33
|
+
## Resolving targets from the Mastra instance
|
|
34
|
+
|
|
35
|
+
Get the `target` (and any registered scorers) from your `Mastra` instance with `mastra.getAgent()`, `mastra.getWorkflow()`, and `mastra.getScorer()` instead of importing the agent or workflow file directly. A directly imported agent or workflow has no Mastra instance attached, so anything that reaches into the registry at runtime fails. For example, a workflow step that calls `mastra.getAgent('weatherAgent')` throws because `mastra` is `undefined`, and the same applies to child workflows, `.agent('agent-id')` references by string, and tools that call `mastra.getWorkflow()`. Registered targets also attach the configured `storage`, which is what lets scores persist and trace-based trajectory scoring work. `runEvals` logs a warning when the target isn't registered.
|
|
36
|
+
|
|
37
|
+
The examples below look up `answerRelevancyScorer` and `trajectoryAccuracyScorer` with `mastra.getScorer()`, so register them on the `Mastra` instance first:
|
|
38
|
+
|
|
39
|
+
```typescript
|
|
40
|
+
import { Mastra } from '@mastra/core'
|
|
41
|
+
import {
|
|
42
|
+
createAnswerRelevancyScorer,
|
|
43
|
+
createTrajectoryAccuracyScorerCode,
|
|
44
|
+
} from '@mastra/evals/scorers/prebuilt'
|
|
45
|
+
|
|
46
|
+
export const mastra = new Mastra({
|
|
47
|
+
agents: { weatherAgent },
|
|
48
|
+
workflows: { weatherWorkflow },
|
|
49
|
+
scorers: {
|
|
50
|
+
answerRelevancyScorer: createAnswerRelevancyScorer({ model: 'openai/gpt-5-mini' }),
|
|
51
|
+
trajectoryAccuracyScorer: createTrajectoryAccuracyScorerCode(),
|
|
52
|
+
},
|
|
53
|
+
})
|
|
54
|
+
```
|
|
55
|
+
|
|
33
56
|
## Asserting on a dataset with `expectEvals`
|
|
34
57
|
|
|
35
58
|
`expectEvals` runs a `runEvals` evaluation inside a regular `test()` and asserts a minimum pass rate. It accepts the same configuration as `runEvals`: a `target` agent or workflow, `data` items, and `scorers`, `gates`, or thresholds. Gates score each item pass/fail, so `toPass(0.8)` requires at least 80% of items to pass every gate. Scorer thresholds still compare the average score across items and must pass regardless of the rate:
|
|
@@ -37,20 +60,15 @@ The setup file registers the custom matchers on `expect`. Alternatively, call `r
|
|
|
37
60
|
```typescript
|
|
38
61
|
import { test } from 'vitest'
|
|
39
62
|
import { expectEvals } from '@mastra/evals/vitest'
|
|
40
|
-
import {
|
|
41
|
-
import {
|
|
42
|
-
import { createKeywordCoverageScorer } from '@mastra/evals/scorers/prebuilt'
|
|
63
|
+
import { checks } from '@mastra/evals/checks'
|
|
64
|
+
import { mastra } from '../src/mastra'
|
|
43
65
|
|
|
44
|
-
test('
|
|
66
|
+
test('weather agent calls the tool and answers relevantly', { timeout: 60_000 }, async () => {
|
|
45
67
|
await expectEvals({
|
|
46
|
-
target:
|
|
47
|
-
data: [
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
{ input: 'What is the capital of Australia?', groundTruth: 'Canberra' },
|
|
51
|
-
],
|
|
52
|
-
gates: [containsGroundTruth],
|
|
53
|
-
scorers: [{ scorer: createKeywordCoverageScorer(), threshold: 0.4 }],
|
|
68
|
+
target: mastra.getAgent('weatherAgent'),
|
|
69
|
+
data: [{ input: "What's the weather in London?" }],
|
|
70
|
+
gates: [checks.calledTool('weatherTool'), checks.noToolErrors(), checks.includes('London')],
|
|
71
|
+
scorers: [{ scorer: mastra.getScorer('answerRelevancyScorer'), threshold: 0.7 }],
|
|
54
72
|
}).toPass(0.8)
|
|
55
73
|
})
|
|
56
74
|
```
|
|
@@ -66,18 +84,18 @@ LLM-backed evals are far slower than Vitest's default 5-second timeout, so pass
|
|
|
66
84
|
```typescript
|
|
67
85
|
import { test } from 'vitest'
|
|
68
86
|
import { expectEval } from '@mastra/evals/vitest'
|
|
69
|
-
import {
|
|
70
|
-
import {
|
|
87
|
+
import { checks } from '@mastra/evals/checks'
|
|
88
|
+
import { mastra } from '../src/mastra'
|
|
71
89
|
|
|
72
90
|
test.for([
|
|
73
|
-
{ input: '
|
|
74
|
-
{ input: '
|
|
75
|
-
{ input: '
|
|
76
|
-
])('
|
|
91
|
+
{ input: "What's the weather in London?", groundTruth: 'London' },
|
|
92
|
+
{ input: "What's the weather in Tokyo?", groundTruth: 'Tokyo' },
|
|
93
|
+
{ input: "What's the weather in Sydney?", groundTruth: 'Sydney' },
|
|
94
|
+
])('weather agent: $input', { timeout: 60_000 }, async item => {
|
|
77
95
|
await expectEval({
|
|
78
|
-
target:
|
|
96
|
+
target: mastra.getAgent('weatherAgent'),
|
|
79
97
|
data: item,
|
|
80
|
-
gates: [
|
|
98
|
+
gates: [checks.calledTool('weatherTool'), checks.includes(item.groundTruth)],
|
|
81
99
|
}).toPass()
|
|
82
100
|
})
|
|
83
101
|
```
|
|
@@ -91,20 +109,20 @@ For finer-grained control, call `runEvals` directly inside a regular `test()` an
|
|
|
91
109
|
```typescript
|
|
92
110
|
import { test, expect } from 'vitest'
|
|
93
111
|
import { runEvals } from '@mastra/core/evals'
|
|
94
|
-
import {
|
|
95
|
-
import {
|
|
112
|
+
import { checks } from '@mastra/evals/checks'
|
|
113
|
+
import { mastra } from '../src/mastra'
|
|
96
114
|
|
|
97
|
-
test('
|
|
115
|
+
test('weather agent quality', { timeout: 60_000 }, async () => {
|
|
98
116
|
const result = await runEvals({
|
|
99
|
-
target:
|
|
100
|
-
data: [{ input: '
|
|
101
|
-
scorers: [
|
|
102
|
-
gates: [
|
|
117
|
+
target: mastra.getAgent('weatherAgent'),
|
|
118
|
+
data: [{ input: "What's the weather in London?" }],
|
|
119
|
+
scorers: [mastra.getScorer('answerRelevancyScorer')],
|
|
120
|
+
gates: [checks.noToolErrors()],
|
|
103
121
|
})
|
|
104
122
|
|
|
105
123
|
expect(result).toHaveVerdict('passed')
|
|
106
124
|
expect(result).toPassGates()
|
|
107
|
-
expect(result).toHaveScoreAbove('relevancy', 0.7)
|
|
125
|
+
expect(result).toHaveScoreAbove('answer-relevancy-scorer', 0.7)
|
|
108
126
|
})
|
|
109
127
|
```
|
|
110
128
|
|
|
@@ -115,6 +133,49 @@ Available matchers:
|
|
|
115
133
|
- `toPassGates()`: asserts all gates passed. Fails when no gates were configured.
|
|
116
134
|
- `toPassThresholds()`: asserts all scorer thresholds passed. Fails when no thresholds were configured.
|
|
117
135
|
|
|
136
|
+
## Evaluating a workflow
|
|
137
|
+
|
|
138
|
+
Workflows are valid targets too. Use a `WorkflowScorerConfig` object for `scorers` to score the workflow output, individual steps, or the step execution trajectory, and reference nested scores with dot-paths in the matchers:
|
|
139
|
+
|
|
140
|
+
```typescript
|
|
141
|
+
import { test, expect } from 'vitest'
|
|
142
|
+
import { runEvals } from '@mastra/core/evals'
|
|
143
|
+
import { mastra } from '../src/mastra'
|
|
144
|
+
|
|
145
|
+
test('weather workflow runs fetch-weather then plan-activities', { timeout: 60_000 }, async () => {
|
|
146
|
+
const result = await runEvals({
|
|
147
|
+
target: mastra.getWorkflow('weatherWorkflow'),
|
|
148
|
+
data: [
|
|
149
|
+
{
|
|
150
|
+
input: { city: 'London' },
|
|
151
|
+
expectedTrajectory: {
|
|
152
|
+
steps: [
|
|
153
|
+
{ stepType: 'workflow_step', name: 'fetch-weather' },
|
|
154
|
+
{ stepType: 'workflow_step', name: 'plan-activities' },
|
|
155
|
+
],
|
|
156
|
+
},
|
|
157
|
+
},
|
|
158
|
+
{
|
|
159
|
+
input: { city: 'Tokyo' },
|
|
160
|
+
expectedTrajectory: {
|
|
161
|
+
steps: [
|
|
162
|
+
{ stepType: 'workflow_step', name: 'fetch-weather' },
|
|
163
|
+
{ stepType: 'workflow_step', name: 'plan-activities' },
|
|
164
|
+
],
|
|
165
|
+
},
|
|
166
|
+
},
|
|
167
|
+
],
|
|
168
|
+
scorers: {
|
|
169
|
+
trajectory: [mastra.getScorer('trajectoryAccuracyScorer')],
|
|
170
|
+
},
|
|
171
|
+
})
|
|
172
|
+
|
|
173
|
+
expect(result).toHaveScoreAbove('trajectory.code-trajectory-accuracy-scorer', 0.9)
|
|
174
|
+
})
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
The `plan-activities` step in this workflow calls `mastra.getAgent('weatherAgent')` at runtime, which only works because the target was resolved with `mastra.getWorkflow()`. See the [`runEvals` reference](https://mastra.ai/reference/evals/run-evals) for the full `WorkflowScorerConfig` shape.
|
|
178
|
+
|
|
118
179
|
## Reading the reporter output
|
|
119
180
|
|
|
120
181
|
`MastraEvalsReporter` prints a score table for every eval test after the run completes:
|
|
@@ -122,9 +183,11 @@ Available matchers:
|
|
|
122
183
|
```text
|
|
123
184
|
Mastra Evals
|
|
124
185
|
|
|
125
|
-
✓
|
|
126
|
-
|
|
127
|
-
|
|
186
|
+
✓ weather agent calls the tool and answers relevantly (1 item)
|
|
187
|
+
check-called-tool (gate) 1.0 ✓
|
|
188
|
+
check-no-tool-errors (gate) 1.0 ✓
|
|
189
|
+
check-includes (gate) 1.0 ✓
|
|
190
|
+
answer-relevancy-scorer (threshold: min 0.7) 0.9 ✓
|
|
128
191
|
|
|
129
192
|
Eval runs: 1 (1 passed)
|
|
130
193
|
```
|
|
@@ -861,6 +861,7 @@ Tags are exported as a JSON string in the `mastra.tags` span attribute for broad
|
|
|
861
861
|
If traces aren't displaying or connecting as expected:
|
|
862
862
|
|
|
863
863
|
- Verify OTEL SDK is initialized before Mastra (use the `--import` flag or import at the top of the entry point)
|
|
864
|
+
- If you see `[OtelBridge] No OpenTelemetry tracer provider is registered globally`, the bridge can't find a tracer provider. Some frameworks create a provider without registering it globally. Pass that provider directly with `new OtelBridge({ tracerProvider })`. See [setup requirements](https://mastra.ai/reference/observability/tracing/bridges/otel)
|
|
864
865
|
- Ensure the `OtelBridge` is added to your observability config
|
|
865
866
|
- Check that your OTEL backend is running and accessible
|
|
866
867
|
|
|
@@ -10,17 +10,16 @@ The `runEvals` function enables batch evaluation of agents and workflows by runn
|
|
|
10
10
|
|
|
11
11
|
```typescript
|
|
12
12
|
import { runEvals } from '@mastra/core/evals'
|
|
13
|
-
import {
|
|
14
|
-
import { myScorer1, myScorer2 } from './scorers'
|
|
13
|
+
import { mastra } from './mastra'
|
|
15
14
|
|
|
16
15
|
const result = await runEvals({
|
|
17
|
-
target: myAgent,
|
|
16
|
+
target: mastra.getAgent('myAgent'),
|
|
18
17
|
data: [
|
|
19
18
|
{ input: 'What is machine learning?' },
|
|
20
19
|
{ input: 'Explain neural networks' },
|
|
21
20
|
{ input: 'How does AI work?' },
|
|
22
21
|
],
|
|
23
|
-
scorers: [myScorer1, myScorer2],
|
|
22
|
+
scorers: [mastra.getScorer('myScorer1'), mastra.getScorer('myScorer2')],
|
|
24
23
|
targetOptions: { maxSteps: 5 },
|
|
25
24
|
concurrency: 2,
|
|
26
25
|
onItemComplete: ({ item, targetResult, scorerResults }) => {
|
|
@@ -33,15 +32,17 @@ console.log(`Average scores:`, result.scores)
|
|
|
33
32
|
console.log(`Processed ${result.summary.totalItems} items`)
|
|
34
33
|
```
|
|
35
34
|
|
|
35
|
+
Resolve the `target` from the `Mastra` instance with `mastra.getAgent()` or `mastra.getWorkflow()` rather than importing the agent or workflow directly. A directly imported target has no Mastra instance attached, so registry lookups made during execution (a step calling `mastra.getAgent()`, a child workflow, an `.agent('agent-id')` reference, a tool calling `mastra.getWorkflow()`) fail, scores aren't persisted, and trace-based trajectory extraction is unavailable. `runEvals` logs a warning when the target isn't registered.
|
|
36
|
+
|
|
36
37
|
### Multi-turn evaluation
|
|
37
38
|
|
|
38
39
|
```typescript
|
|
39
40
|
import { runEvals } from '@mastra/core/evals'
|
|
40
41
|
import { checks } from '@mastra/evals/checks'
|
|
41
|
-
import {
|
|
42
|
+
import { mastra } from './mastra'
|
|
42
43
|
|
|
43
44
|
const result = await runEvals({
|
|
44
|
-
target: weatherAgent,
|
|
45
|
+
target: mastra.getAgent('weatherAgent'),
|
|
45
46
|
data: [
|
|
46
47
|
{
|
|
47
48
|
inputs: [
|
|
@@ -60,13 +61,16 @@ const result = await runEvals({
|
|
|
60
61
|
```typescript
|
|
61
62
|
import { runEvals } from '@mastra/core/evals'
|
|
62
63
|
import { checks } from '@mastra/evals/checks'
|
|
63
|
-
import {
|
|
64
|
+
import { mastra } from './mastra'
|
|
64
65
|
|
|
65
66
|
const result = await runEvals({
|
|
66
|
-
target:
|
|
67
|
+
target: mastra.getAgent('weatherAgent'),
|
|
67
68
|
data: [{ input: 'What is the weather in Brooklyn?' }],
|
|
68
69
|
gates: [checks.calledTool('get_weather'), checks.noToolErrors()],
|
|
69
|
-
scorers: [
|
|
70
|
+
scorers: [
|
|
71
|
+
{ scorer: mastra.getScorer('faithfulnessScorer'), threshold: 0.7 },
|
|
72
|
+
checks.includes('Brooklyn'),
|
|
73
|
+
],
|
|
70
74
|
})
|
|
71
75
|
|
|
72
76
|
result.verdict // 'passed' | 'scored' | 'failed'
|
|
@@ -173,7 +177,7 @@ import { runEvals } from '@mastra/core/evals'
|
|
|
173
177
|
import { checks } from '@mastra/evals/checks'
|
|
174
178
|
|
|
175
179
|
const result = await runEvals({
|
|
176
|
-
target: weatherAgent,
|
|
180
|
+
target: mastra.getAgent('weatherAgent'),
|
|
177
181
|
data: [{ input: 'What is the weather in Brooklyn?' }],
|
|
178
182
|
gates: [checks.calledTool('get_weather'), checks.noToolErrors()],
|
|
179
183
|
scorers: [
|
|
@@ -213,7 +217,7 @@ const myScorer = createScorer({
|
|
|
213
217
|
})
|
|
214
218
|
|
|
215
219
|
const result = await runEvals({
|
|
216
|
-
target: chatAgent,
|
|
220
|
+
target: mastra.getAgent('chatAgent'),
|
|
217
221
|
data: [
|
|
218
222
|
{
|
|
219
223
|
input: 'What is AI?',
|
|
@@ -240,7 +244,7 @@ import { createTrajectoryAccuracyScorerCode } from '@mastra/evals/scorers/code/t
|
|
|
240
244
|
const trajectoryScorer = createTrajectoryAccuracyScorerCode()
|
|
241
245
|
|
|
242
246
|
const result = await runEvals({
|
|
243
|
-
target: chatAgent,
|
|
247
|
+
target: mastra.getAgent('chatAgent'),
|
|
244
248
|
data: [
|
|
245
249
|
{
|
|
246
250
|
input: 'What is the weather in London?',
|
|
@@ -265,7 +269,7 @@ Pass execution options like `maxSteps` or `modelSettings` to customize agent beh
|
|
|
265
269
|
|
|
266
270
|
```typescript
|
|
267
271
|
const result = await runEvals({
|
|
268
|
-
target: chatAgent,
|
|
272
|
+
target: mastra.getAgent('chatAgent'),
|
|
269
273
|
data: [{ input: 'Summarize this article' }, { input: 'Translate to French' }],
|
|
270
274
|
scorers: [relevancyScorer],
|
|
271
275
|
targetOptions: {
|
|
@@ -277,9 +281,11 @@ const result = await runEvals({
|
|
|
277
281
|
|
|
278
282
|
### Workflow Evaluation
|
|
279
283
|
|
|
284
|
+
Resolve the workflow with `mastra.getWorkflow()` so steps that call `mastra.getAgent()` or run child workflows can reach the registry:
|
|
285
|
+
|
|
280
286
|
```typescript
|
|
281
287
|
const workflowResult = await runEvals({
|
|
282
|
-
target: myWorkflow,
|
|
288
|
+
target: mastra.getWorkflow('myWorkflow'),
|
|
283
289
|
data: [
|
|
284
290
|
{ input: { query: 'Process this data', priority: 'high' } },
|
|
285
291
|
{ input: { query: 'Another task', priority: 'low' } },
|
|
@@ -309,7 +315,7 @@ Add trajectory scoring to workflow evaluations to validate step execution order:
|
|
|
309
315
|
|
|
310
316
|
```typescript
|
|
311
317
|
const workflowResult = await runEvals({
|
|
312
|
-
target: myWorkflow,
|
|
318
|
+
target: mastra.getWorkflow('myWorkflow'),
|
|
313
319
|
data: [
|
|
314
320
|
{
|
|
315
321
|
input: { query: 'Process this data' },
|
|
@@ -340,7 +346,7 @@ Use `startOptions` on individual data items to customize each workflow run. Per-
|
|
|
340
346
|
|
|
341
347
|
```typescript
|
|
342
348
|
const result = await runEvals({
|
|
343
|
-
target: myWorkflow,
|
|
349
|
+
target: mastra.getWorkflow('myWorkflow'),
|
|
344
350
|
data: [
|
|
345
351
|
{
|
|
346
352
|
input: { query: 'hello' },
|
|
@@ -362,7 +368,7 @@ Use `inputs` to send sequential turns on a shared thread. Scorers see the accumu
|
|
|
362
368
|
|
|
363
369
|
```typescript
|
|
364
370
|
const result = await runEvals({
|
|
365
|
-
target: chatAgent,
|
|
371
|
+
target: mastra.getAgent('chatAgent'),
|
|
366
372
|
data: [
|
|
367
373
|
{
|
|
368
374
|
inputs: ['My favorite city is Brooklyn.', 'What is the weather in my favorite city?'],
|
|
@@ -385,7 +391,7 @@ Use `turns` to attach `gates`/`scorers` to individual turns. Each per-turn asser
|
|
|
385
391
|
|
|
386
392
|
```typescript
|
|
387
393
|
const result = await runEvals({
|
|
388
|
-
target: chatAgent,
|
|
394
|
+
target: mastra.getAgent('chatAgent'),
|
|
389
395
|
data: [
|
|
390
396
|
{
|
|
391
397
|
turns: [
|
|
@@ -221,7 +221,7 @@ const scorer = createTrajectoryAccuracyScorerCode({
|
|
|
221
221
|
const scorer = createTrajectoryAccuracyScorerCode()
|
|
222
222
|
|
|
223
223
|
await runEvals({
|
|
224
|
-
target: myAgent,
|
|
224
|
+
target: mastra.getAgent('myAgent'),
|
|
225
225
|
scorers: { trajectory: [scorer] },
|
|
226
226
|
data: [
|
|
227
227
|
{
|
|
@@ -245,16 +245,20 @@ await runEvals({
|
|
|
245
245
|
|
|
246
246
|
### Evaluation modes
|
|
247
247
|
|
|
248
|
-
The code-based scorer operates in
|
|
248
|
+
The code-based scorer operates in one of three modes based on `ordering`:
|
|
249
249
|
|
|
250
|
-
#### Strict mode (`
|
|
250
|
+
#### Strict mode (`ordering: 'strict'`)
|
|
251
251
|
|
|
252
|
-
|
|
252
|
+
Only an exact match (the same steps in the same order, with nothing extra or missing) scores `1.0`. Anything else gets partial credit for the expected steps that matched in position, with a penalty deducted for each extra step. For example, the two expected steps followed by one extra step scores `0.75`.
|
|
253
253
|
|
|
254
|
-
#### Relaxed mode (`
|
|
254
|
+
#### Relaxed mode (`ordering: 'relaxed'`, default)
|
|
255
255
|
|
|
256
256
|
Allows extra steps. Expected steps must appear in the correct relative order. The score is calculated based on how many expected steps were matched, with optional penalties for extra or repeated steps.
|
|
257
257
|
|
|
258
|
+
#### Unordered mode (`ordering: 'unordered'`)
|
|
259
|
+
|
|
260
|
+
Only checks that each expected step is present. Order is ignored, and extra steps are reported in the result but not penalized.
|
|
261
|
+
|
|
258
262
|
## Code-based scoring details
|
|
259
263
|
|
|
260
264
|
- **Continuous scores**: Returns values between 0.0 and 1.0 in relaxed mode; binary (0 or 1) in strict mode
|
|
@@ -303,16 +307,16 @@ const scorer = createTrajectoryAccuracyScorerCode({
|
|
|
303
307
|
{ stepType: 'tool_call', name: 'fetch-tool' },
|
|
304
308
|
],
|
|
305
309
|
},
|
|
306
|
-
comparisonOptions: {
|
|
310
|
+
comparisonOptions: { ordering: 'strict' },
|
|
307
311
|
})
|
|
308
312
|
|
|
309
313
|
const result = await runEvals({
|
|
310
|
-
target: myAgent,
|
|
314
|
+
target: mastra.getAgent('myAgent'),
|
|
311
315
|
scorers: { trajectory: [scorer] },
|
|
312
316
|
data: [{ input: 'Get my data' }],
|
|
313
317
|
})
|
|
314
318
|
|
|
315
|
-
console.log(result.scores.trajectory['trajectory-accuracy']) // 1.0
|
|
319
|
+
console.log(result.scores.trajectory['code-trajectory-accuracy-scorer']) // 1.0
|
|
316
320
|
```
|
|
317
321
|
|
|
318
322
|
### Agent trajectory with relaxed ordering
|
|
@@ -327,7 +331,7 @@ const scorer = createTrajectoryAccuracyScorerCode({
|
|
|
327
331
|
{ stepType: 'tool_call', name: 'summarize-tool' },
|
|
328
332
|
],
|
|
329
333
|
},
|
|
330
|
-
comparisonOptions: {
|
|
334
|
+
comparisonOptions: { ordering: 'relaxed' },
|
|
331
335
|
})
|
|
332
336
|
|
|
333
337
|
// Agent called search-tool → log-tool → summarize-tool
|
|
@@ -354,12 +358,12 @@ const scorer = createTrajectoryAccuracyScorerCode({
|
|
|
354
358
|
})
|
|
355
359
|
|
|
356
360
|
const result = await runEvals({
|
|
357
|
-
target: myWorkflow,
|
|
361
|
+
target: mastra.getWorkflow('myWorkflow'),
|
|
358
362
|
scorers: { trajectory: [scorer] },
|
|
359
363
|
data: [{ input: { data: 'test' } }],
|
|
360
364
|
})
|
|
361
365
|
|
|
362
|
-
console.log(result.scores.trajectory['trajectory-accuracy'])
|
|
366
|
+
console.log(result.scores.trajectory['code-trajectory-accuracy-scorer'])
|
|
363
367
|
```
|
|
364
368
|
|
|
365
369
|
### Comparing step data
|
|
@@ -517,7 +521,7 @@ const scorer = createTrajectoryScorerCode({
|
|
|
517
521
|
})
|
|
518
522
|
|
|
519
523
|
const result = await runEvals({
|
|
520
|
-
target: myAgent,
|
|
524
|
+
target: mastra.getAgent('myAgent'),
|
|
521
525
|
scorers: { trajectory: [scorer] },
|
|
522
526
|
data: [
|
|
523
527
|
{
|
|
@@ -583,7 +587,7 @@ const trajectoryScorer = createTrajectoryAccuracyScorerCode({
|
|
|
583
587
|
})
|
|
584
588
|
|
|
585
589
|
const result = await runEvals({
|
|
586
|
-
target: myAgent,
|
|
590
|
+
target: mastra.getAgent('myAgent'),
|
|
587
591
|
scorers: {
|
|
588
592
|
agent: [qualityScorer], // receives raw MastraDBMessage[] output
|
|
589
593
|
trajectory: [trajectoryScorer], // receives pre-extracted Trajectory
|
|
@@ -592,7 +596,7 @@ const result = await runEvals({
|
|
|
592
596
|
})
|
|
593
597
|
|
|
594
598
|
// result.scores.agent['quality'] — agent-level score
|
|
595
|
-
// result.scores.trajectory['trajectory-accuracy'] — trajectory score
|
|
599
|
+
// result.scores.trajectory['code-trajectory-accuracy-scorer'] — trajectory score
|
|
596
600
|
```
|
|
597
601
|
|
|
598
602
|
### Workflow trajectory evaluation
|
|
@@ -612,7 +616,7 @@ const workflowTrajectoryScorer = createTrajectoryAccuracyScorerCode({
|
|
|
612
616
|
})
|
|
613
617
|
|
|
614
618
|
const result = await runEvals({
|
|
615
|
-
target: myWorkflow,
|
|
619
|
+
target: mastra.getWorkflow('myWorkflow'),
|
|
616
620
|
scorers: {
|
|
617
621
|
workflow: [outputScorer], // receives workflow output
|
|
618
622
|
trajectory: [workflowTrajectoryScorer], // receives pre-extracted Trajectory from step results
|
|
@@ -621,7 +625,7 @@ const result = await runEvals({
|
|
|
621
625
|
})
|
|
622
626
|
|
|
623
627
|
// result.scores.workflow['output-quality'] — workflow-level score
|
|
624
|
-
// result.scores.trajectory['trajectory-accuracy'] — trajectory score
|
|
628
|
+
// result.scores.trajectory['code-trajectory-accuracy-scorer'] — trajectory score
|
|
625
629
|
```
|
|
626
630
|
|
|
627
631
|
## Related
|
|
@@ -11,7 +11,7 @@ Enables bidirectional integration between Mastra tracing and OpenTelemetry infra
|
|
|
11
11
|
## Constructor
|
|
12
12
|
|
|
13
13
|
```typescript
|
|
14
|
-
new OtelBridge()
|
|
14
|
+
new OtelBridge(config?: OtelBridgeConfig)
|
|
15
15
|
```
|
|
16
16
|
|
|
17
17
|
## Methods
|
|
@@ -118,9 +118,18 @@ const mastra = new Mastra({
|
|
|
118
118
|
})
|
|
119
119
|
```
|
|
120
120
|
|
|
121
|
-
##
|
|
121
|
+
## Setup requirements
|
|
122
122
|
|
|
123
|
-
The
|
|
123
|
+
The bridge creates spans through an OpenTelemetry tracer provider. For Mastra spans to be exported:
|
|
124
|
+
|
|
125
|
+
- **A tracer provider must be available.** Register one globally before Mastra runs, for example by calling `sdk.start()` on `NodeSDK` from `@opentelemetry/sdk-node`, or by calling `trace.setGlobalTracerProvider(provider)`. If the provider isn't registered globally, pass it with `new OtelBridge({ tracerProvider })`.
|
|
126
|
+
- **A context manager must be installed for Mastra spans to nest under outer spans.** `NodeSDK` installs one automatically. Without a context manager, spans are still exported, but Mastra root spans can't read the active span, so they start new traces instead of joining the outer trace.
|
|
127
|
+
|
|
128
|
+
When no tracer provider is available, agents and workflows still run, but no Mastra spans are exported through OpenTelemetry. The bridge logs this warning once:
|
|
129
|
+
|
|
130
|
+
```text
|
|
131
|
+
[OtelBridge] No OpenTelemetry tracer provider is registered globally, so Mastra spans will not be exported through OpenTelemetry. ...
|
|
132
|
+
```
|
|
124
133
|
|
|
125
134
|
See the [OtelBridge Guide](https://mastra.ai/integrations/observability/opentelemetry) for complete setup instructions, including how to configure OTEL instrumentation and run your application.
|
|
126
135
|
|
|
@@ -54,6 +54,29 @@ const results = await store.hybridQuery({
|
|
|
54
54
|
|
|
55
55
|
See the [MongoDB vector reference](https://mastra.ai/reference/vectors/mongodb) for details on `createSearchIndex()`, `textQuery()`, and `hybridQuery()`.
|
|
56
56
|
|
|
57
|
+
### Automated Embedding
|
|
58
|
+
|
|
59
|
+
MongoDB can generate the embeddings itself, so you don't need an embedding provider in your application. Create the index with `autoEmbed` and a Voyage AI model, write plain text, and search with a query string:
|
|
60
|
+
|
|
61
|
+
```ts
|
|
62
|
+
await store.createIndex({
|
|
63
|
+
indexName: 'myCollection',
|
|
64
|
+
autoEmbed: { model: 'voyage-4' },
|
|
65
|
+
})
|
|
66
|
+
await store.upsert({
|
|
67
|
+
indexName: 'myCollection',
|
|
68
|
+
documents: chunks.map(chunk => chunk.text),
|
|
69
|
+
metadata: chunks.map(chunk => ({ text: chunk.text })),
|
|
70
|
+
})
|
|
71
|
+
const results = await store.query({
|
|
72
|
+
indexName: 'myCollection',
|
|
73
|
+
queryText: 'search terms',
|
|
74
|
+
topK: 10,
|
|
75
|
+
})
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
Automated Embedding is a MongoDB Preview feature and needs a deployment where it's available. See the [MongoDB vector reference](https://mastra.ai/reference/vectors/mongodb) for the supported options and requirements.
|
|
79
|
+
|
|
57
80
|
**PgVector**:
|
|
58
81
|
|
|
59
82
|
```ts
|