@mastra/mcp-docs-server 1.0.0-beta.0 → 1.0.0-beta.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.docs/organized/changelogs/%40mastra%2Fai-sdk.md +10 -10
- package/.docs/organized/changelogs/%40mastra%2Fclient-js.md +17 -17
- package/.docs/organized/changelogs/%40mastra%2Fcodemod.md +8 -0
- package/.docs/organized/changelogs/%40mastra%2Fcore.md +18 -18
- package/.docs/organized/changelogs/%40mastra%2Fdeployer-cloud.md +19 -19
- package/.docs/organized/changelogs/%40mastra%2Fdeployer.md +19 -19
- package/.docs/organized/changelogs/%40mastra%2Fmcp-docs-server.md +18 -18
- package/.docs/organized/changelogs/%40mastra%2Fmcp.md +12 -12
- package/.docs/organized/changelogs/%40mastra%2Fmssql.md +16 -16
- package/.docs/organized/changelogs/%40mastra%2Fplayground-ui.md +32 -32
- package/.docs/organized/changelogs/%40mastra%2Freact.md +15 -15
- package/.docs/organized/changelogs/%40mastra%2Fserver.md +15 -15
- package/.docs/organized/changelogs/create-mastra.md +7 -7
- package/.docs/organized/changelogs/mastra.md +13 -13
- package/.docs/raw/agents/agent-memory.mdx +1 -1
- package/.docs/raw/agents/overview.mdx +2 -2
- package/.docs/raw/auth/auth0.mdx +1 -1
- package/.docs/raw/auth/clerk.mdx +1 -1
- package/.docs/raw/auth/firebase.mdx +1 -1
- package/.docs/raw/auth/jwt.mdx +1 -1
- package/.docs/raw/auth/supabase.mdx +1 -1
- package/.docs/raw/auth/workos.mdx +1 -1
- package/.docs/raw/course/01-first-agent/03-verifying-installation.md +1 -1
- package/.docs/raw/course/02-agent-tools-mcp/02-installing-mcp.md +1 -1
- package/.docs/raw/course/03-agent-memory/03-installing-memory.md +1 -1
- package/.docs/raw/deployment/cloud-providers/amazon-ec2.mdx +1 -1
- package/.docs/raw/deployment/cloud-providers/aws-lambda.mdx +1 -1
- package/.docs/raw/deployment/cloud-providers/azure-app-services.mdx +2 -2
- package/.docs/raw/deployment/cloud-providers/digital-ocean.mdx +1 -1
- package/.docs/raw/deployment/serverless-platforms/cloudflare-deployer.mdx +1 -1
- package/.docs/raw/deployment/serverless-platforms/netlify-deployer.mdx +1 -1
- package/.docs/raw/deployment/serverless-platforms/vercel-deployer.mdx +1 -1
- package/.docs/raw/evals/overview.mdx +1 -1
- package/.docs/raw/frameworks/agentic-uis/ai-sdk.mdx +8 -4
- package/.docs/raw/frameworks/agentic-uis/assistant-ui.mdx +1 -1
- package/.docs/raw/frameworks/agentic-uis/copilotkit.mdx +9 -9
- package/.docs/raw/frameworks/agentic-uis/openrouter.mdx +1 -1
- package/.docs/raw/frameworks/servers/express.mdx +1 -1
- package/.docs/raw/frameworks/web-frameworks/astro.mdx +12 -12
- package/.docs/raw/frameworks/web-frameworks/next-js.mdx +4 -4
- package/.docs/raw/frameworks/web-frameworks/sveltekit.mdx +12 -12
- package/.docs/raw/frameworks/web-frameworks/vite-react.mdx +5 -5
- package/.docs/raw/getting-started/installation.mdx +13 -13
- package/.docs/raw/getting-started/templates.mdx +5 -5
- package/.docs/raw/guides/guide/notes-mcp-server.mdx +1 -1
- package/.docs/raw/guides/guide/research-assistant.mdx +1 -10
- package/.docs/raw/guides/migrations/upgrade-to-v1/agent.mdx +16 -6
- package/.docs/raw/guides/migrations/upgrade-to-v1/client.mdx +42 -2
- package/.docs/raw/guides/migrations/upgrade-to-v1/evals.mdx +31 -1
- package/.docs/raw/guides/migrations/upgrade-to-v1/mastra.mdx +28 -4
- package/.docs/raw/guides/migrations/upgrade-to-v1/mcp.mdx +3 -3
- package/.docs/raw/guides/migrations/upgrade-to-v1/memory.mdx +28 -0
- package/.docs/raw/guides/migrations/upgrade-to-v1/overview.mdx +50 -36
- package/.docs/raw/guides/migrations/upgrade-to-v1/storage.mdx +80 -0
- package/.docs/raw/guides/migrations/upgrade-to-v1/tools.mdx +2 -2
- package/.docs/raw/guides/migrations/upgrade-to-v1/tracing.mdx +13 -13
- package/.docs/raw/guides/migrations/upgrade-to-v1/vectors.mdx +25 -1
- package/.docs/raw/guides/migrations/upgrade-to-v1/voice.mdx +1 -1
- package/.docs/raw/guides/migrations/upgrade-to-v1/workflows.mdx +32 -2
- package/.docs/raw/guides/quickstarts/nextjs.mdx +275 -0
- package/.docs/raw/index.mdx +1 -1
- package/.docs/raw/logging.mdx +3 -3
- package/.docs/raw/mastra-cloud/observability.mdx +6 -6
- package/.docs/raw/memory/overview.mdx +2 -2
- package/.docs/raw/memory/semantic-recall.mdx +2 -2
- package/.docs/raw/memory/storage/memory-with-libsql.mdx +2 -2
- package/.docs/raw/memory/storage/memory-with-mongodb.mdx +152 -0
- package/.docs/raw/memory/storage/memory-with-pg.mdx +2 -2
- package/.docs/raw/memory/storage/memory-with-upstash.mdx +2 -2
- package/.docs/raw/observability/overview.mdx +5 -5
- package/.docs/raw/observability/tracing/exporters/arize.mdx +2 -2
- package/.docs/raw/observability/tracing/exporters/braintrust.mdx +2 -2
- package/.docs/raw/observability/tracing/exporters/cloud.mdx +2 -2
- package/.docs/raw/observability/tracing/exporters/default.mdx +5 -5
- package/.docs/raw/observability/tracing/exporters/langfuse.mdx +2 -2
- package/.docs/raw/observability/tracing/exporters/langsmith.mdx +2 -2
- package/.docs/raw/observability/tracing/exporters/otel.mdx +5 -5
- package/.docs/raw/observability/tracing/overview.mdx +21 -21
- package/.docs/raw/observability/tracing/processors/sensitive-data-filter.mdx +2 -2
- package/.docs/raw/rag/overview.mdx +1 -1
- package/.docs/raw/reference/agents/agent.mdx +2 -2
- package/.docs/raw/reference/cli/create-mastra.mdx +10 -10
- package/.docs/raw/reference/cli/mastra.mdx +7 -7
- package/.docs/raw/reference/core/getLogger.mdx +2 -2
- package/.docs/raw/reference/core/getMCPServer.mdx +2 -2
- package/.docs/raw/reference/core/getMCPServerById.mdx +4 -4
- package/.docs/raw/reference/core/getTelemetry.mdx +2 -2
- package/.docs/raw/reference/core/listLogs.mdx +2 -2
- package/.docs/raw/reference/core/listLogsByRunId.mdx +2 -2
- package/.docs/raw/reference/core/listMCPServers.mdx +4 -4
- package/.docs/raw/reference/core/setLogger.mdx +2 -2
- package/.docs/raw/reference/core/setTelemetry.mdx +2 -2
- package/.docs/raw/reference/evals/create-scorer.mdx +58 -49
- package/.docs/raw/reference/evals/mastra-scorer.mdx +2 -0
- package/.docs/raw/reference/observability/tracing/configuration.mdx +12 -12
- package/.docs/raw/reference/observability/tracing/exporters/arize.mdx +1 -1
- package/.docs/raw/reference/observability/tracing/exporters/cloud-exporter.mdx +8 -8
- package/.docs/raw/reference/observability/tracing/exporters/console-exporter.mdx +8 -8
- package/.docs/raw/reference/observability/tracing/exporters/default-exporter.mdx +8 -8
- package/.docs/raw/reference/observability/tracing/instances.mdx +8 -8
- package/.docs/raw/reference/observability/tracing/interfaces.mdx +7 -7
- package/.docs/raw/reference/observability/tracing/spans.mdx +7 -7
- package/.docs/raw/reference/processors/batch-parts-processor.mdx +8 -2
- package/.docs/raw/reference/processors/language-detector.mdx +8 -2
- package/.docs/raw/reference/processors/moderation-processor.mdx +8 -2
- package/.docs/raw/reference/processors/pii-detector.mdx +8 -2
- package/.docs/raw/reference/processors/prompt-injection-detector.mdx +8 -2
- package/.docs/raw/reference/processors/system-prompt-scrubber.mdx +8 -2
- package/.docs/raw/reference/processors/token-limiter-processor.mdx +8 -2
- package/.docs/raw/reference/processors/unicode-normalizer.mdx +8 -2
- package/.docs/raw/reference/storage/cloudflare-d1.mdx +1 -1
- package/.docs/raw/reference/storage/cloudflare.mdx +1 -1
- package/.docs/raw/reference/storage/dynamodb.mdx +3 -3
- package/.docs/raw/reference/storage/lance.mdx +1 -1
- package/.docs/raw/reference/storage/libsql.mdx +1 -1
- package/.docs/raw/reference/storage/mongodb.mdx +1 -1
- package/.docs/raw/reference/storage/mssql.mdx +1 -1
- package/.docs/raw/reference/storage/postgresql.mdx +1 -1
- package/.docs/raw/reference/storage/upstash.mdx +1 -1
- package/.docs/raw/reference/templates/overview.mdx +1 -1
- package/.docs/raw/reference/tools/mcp-server.mdx +49 -0
- package/.docs/raw/reference/vectors/couchbase.mdx +8 -2
- package/.docs/raw/reference/vectors/libsql.mdx +1 -1
- package/.docs/raw/reference/vectors/mongodb.mdx +7 -1
- package/.docs/raw/reference/vectors/s3vectors.mdx +1 -1
- package/.docs/raw/server-db/mastra-client.mdx +4 -4
- package/.docs/raw/tools-mcp/mcp-overview.mdx +1 -1
- package/.docs/raw/voice/speech-to-text.mdx +1 -1
- package/.docs/raw/voice/text-to-speech.mdx +1 -1
- package/.docs/raw/workflows/inngest-workflow.mdx +1 -1
- package/CHANGELOG.md +17 -0
- package/dist/stdio.js +11 -15
- package/package.json +4 -4
|
@@ -56,10 +56,10 @@ This method does not accept any parameters.
|
|
|
56
56
|
|
|
57
57
|
## Related Methods
|
|
58
58
|
|
|
59
|
-
- [Mastra.getMCPServer()]/reference/v1/core/getMCPServer) - Retrieve an MCP server by its registry key
|
|
60
|
-
- [Mastra.getMCPServerById()]/reference/v1/core/getMCPServerById) - Retrieve an MCP server by its intrinsic `id` property
|
|
59
|
+
- [Mastra.getMCPServer()](/reference/v1/core/getMCPServer) - Retrieve an MCP server by its registry key
|
|
60
|
+
- [Mastra.getMCPServerById()](/reference/v1/core/getMCPServerById) - Retrieve an MCP server by its intrinsic `id` property
|
|
61
61
|
|
|
62
62
|
## See Also
|
|
63
63
|
|
|
64
|
-
- [MCP overview]/docs/v1/tools-mcp/mcp-overview)
|
|
65
|
-
- [MCP server reference]/reference/v1/tools/mcp-server)
|
|
64
|
+
- [MCP overview](/docs/v1/tools-mcp/mcp-overview)
|
|
65
|
+
- [MCP server reference](/reference/v1/tools/mcp-server)
|
|
@@ -12,8 +12,11 @@ Mastra provides a unified `createScorer` factory that allows you to define custo
|
|
|
12
12
|
Use the `createScorer` factory to define your scorer with a name, description, and optional judge configuration. Then chain step methods to build your evaluation pipeline. You must provide at least a `generateScore` step.
|
|
13
13
|
|
|
14
14
|
```typescript
|
|
15
|
+
import { createScorer } from "@mastra/core/evals";
|
|
16
|
+
|
|
15
17
|
const scorer = createScorer({
|
|
16
|
-
|
|
18
|
+
id: "my-custom-scorer",
|
|
19
|
+
name: "My Custom Scorer", // Optional, defaults to id
|
|
17
20
|
description: "Evaluates responses based on custom criteria",
|
|
18
21
|
type: "agent", // Optional: for agent evaluation with automatic typing
|
|
19
22
|
judge: {
|
|
@@ -39,29 +42,35 @@ const scorer = createScorer({
|
|
|
39
42
|
|
|
40
43
|
<PropertiesTable
|
|
41
44
|
content={[
|
|
45
|
+
{
|
|
46
|
+
name: "id",
|
|
47
|
+
type: "string",
|
|
48
|
+
isOptional: false,
|
|
49
|
+
description: "Unique identifier for the scorer. Used as the name if `name` is not provided.",
|
|
50
|
+
},
|
|
42
51
|
{
|
|
43
52
|
name: "name",
|
|
44
53
|
type: "string",
|
|
45
|
-
|
|
46
|
-
description: "Name of the scorer.",
|
|
54
|
+
isOptional: true,
|
|
55
|
+
description: "Name of the scorer. Defaults to `id` if not provided.",
|
|
47
56
|
},
|
|
48
57
|
{
|
|
49
58
|
name: "description",
|
|
50
59
|
type: "string",
|
|
51
|
-
|
|
60
|
+
isOptional: false,
|
|
52
61
|
description: "Description of what the scorer does.",
|
|
53
62
|
},
|
|
54
63
|
{
|
|
55
64
|
name: "judge",
|
|
56
65
|
type: "object",
|
|
57
|
-
|
|
66
|
+
isOptional: true,
|
|
58
67
|
description:
|
|
59
68
|
"Optional judge configuration for LLM-based steps. See Judge Object section below.",
|
|
60
69
|
},
|
|
61
70
|
{
|
|
62
71
|
name: "type",
|
|
63
72
|
type: "string",
|
|
64
|
-
|
|
73
|
+
isOptional: true,
|
|
65
74
|
description:
|
|
66
75
|
"Type specification for input/output. Use 'agent' for automatic agent types. For custom types, use the generic approach instead.",
|
|
67
76
|
},
|
|
@@ -77,13 +86,13 @@ This function returns a scorer builder that you can chain step methods onto. See
|
|
|
77
86
|
{
|
|
78
87
|
name: "model",
|
|
79
88
|
type: "LanguageModel",
|
|
80
|
-
|
|
89
|
+
isOptional: false,
|
|
81
90
|
description: "The LLM model instance to use for evaluation.",
|
|
82
91
|
},
|
|
83
92
|
{
|
|
84
93
|
name: "instructions",
|
|
85
94
|
type: "string",
|
|
86
|
-
|
|
95
|
+
isOptional: false,
|
|
87
96
|
description: "System prompt/instructions for the LLM.",
|
|
88
97
|
},
|
|
89
98
|
]}
|
|
@@ -98,7 +107,7 @@ You can specify input/output types when creating scorers for better type inferen
|
|
|
98
107
|
For evaluating agents, use `type: 'agent'` to automatically get the correct types for agent input/output:
|
|
99
108
|
|
|
100
109
|
```typescript
|
|
101
|
-
import { createScorer } from "@mastra/core/
|
|
110
|
+
import { createScorer } from "@mastra/core/evals";
|
|
102
111
|
|
|
103
112
|
// Agent scorer with automatic typing
|
|
104
113
|
const agentScorer = createScorer({
|
|
@@ -123,7 +132,7 @@ const agentScorer = createScorer({
|
|
|
123
132
|
For custom input/output types, use the generic approach:
|
|
124
133
|
|
|
125
134
|
```typescript
|
|
126
|
-
import { createScorer } from "@mastra/core/
|
|
135
|
+
import { createScorer } from "@mastra/core/evals";
|
|
127
136
|
|
|
128
137
|
type CustomInput = { query: string; context: string[] };
|
|
129
138
|
type CustomOutput = { answer: string; confidence: number };
|
|
@@ -185,34 +194,34 @@ Function: `({ run, results }) => any`
|
|
|
185
194
|
{
|
|
186
195
|
name: "run.input",
|
|
187
196
|
type: "any",
|
|
188
|
-
|
|
197
|
+
isOptional: false,
|
|
189
198
|
description:
|
|
190
199
|
"Input records provided to the scorer. If the scorer is added to an agent, this will be an array of user messages, e.g. `[{ role: 'user', content: 'hello world' }]`. If the scorer is used in a workflow, this will be the input of the workflow.",
|
|
191
200
|
},
|
|
192
201
|
{
|
|
193
202
|
name: "run.output",
|
|
194
203
|
type: "any",
|
|
195
|
-
|
|
204
|
+
isOptional: false,
|
|
196
205
|
description:
|
|
197
206
|
"Output record provided to the scorer. For agents, this is usually the agent's response. For workflows, this is the workflow's output.",
|
|
198
207
|
},
|
|
199
208
|
{
|
|
200
209
|
name: "run.runId",
|
|
201
210
|
type: "string",
|
|
202
|
-
|
|
211
|
+
isOptional: false,
|
|
203
212
|
description: "Unique identifier for this scoring run.",
|
|
204
213
|
},
|
|
205
214
|
{
|
|
206
215
|
name: "run.requestContext",
|
|
207
216
|
type: "object",
|
|
208
|
-
|
|
217
|
+
isOptional: true,
|
|
209
218
|
description:
|
|
210
219
|
"Request Context from the agent or workflow step being evaluated (optional).",
|
|
211
220
|
},
|
|
212
221
|
{
|
|
213
222
|
name: "results",
|
|
214
223
|
type: "object",
|
|
215
|
-
|
|
224
|
+
isOptional: false,
|
|
216
225
|
description: "Empty object (no previous steps).",
|
|
217
226
|
},
|
|
218
227
|
]}
|
|
@@ -228,26 +237,26 @@ The method can return any value. The returned value will be available to subsequ
|
|
|
228
237
|
{
|
|
229
238
|
name: "description",
|
|
230
239
|
type: "string",
|
|
231
|
-
|
|
240
|
+
isOptional: false,
|
|
232
241
|
description: "Description of what this preprocessing step does.",
|
|
233
242
|
},
|
|
234
243
|
{
|
|
235
244
|
name: "outputSchema",
|
|
236
245
|
type: "ZodSchema",
|
|
237
|
-
|
|
246
|
+
isOptional: false,
|
|
238
247
|
description: "Zod schema for the expected output of the preprocess step.",
|
|
239
248
|
},
|
|
240
249
|
{
|
|
241
250
|
name: "createPrompt",
|
|
242
251
|
type: "function",
|
|
243
|
-
|
|
252
|
+
isOptional: false,
|
|
244
253
|
description:
|
|
245
254
|
"Function: ({ run, results }) => string. Returns the prompt for the LLM.",
|
|
246
255
|
},
|
|
247
256
|
{
|
|
248
257
|
name: "judge",
|
|
249
258
|
type: "object",
|
|
250
|
-
|
|
259
|
+
isOptional: true,
|
|
251
260
|
description:
|
|
252
261
|
"(Optional) LLM judge for this step (can override main judge). See Judge Object section.",
|
|
253
262
|
},
|
|
@@ -266,34 +275,34 @@ Function: `({ run, results }) => any`
|
|
|
266
275
|
{
|
|
267
276
|
name: "run.input",
|
|
268
277
|
type: "any",
|
|
269
|
-
|
|
278
|
+
isOptional: false,
|
|
270
279
|
description:
|
|
271
280
|
"Input records provided to the scorer. If the scorer is added to an agent, this will be an array of user messages, e.g. `[{ role: 'user', content: 'hello world' }]`. If the scorer is used in a workflow, this will be the input of the workflow.",
|
|
272
281
|
},
|
|
273
282
|
{
|
|
274
283
|
name: "run.output",
|
|
275
284
|
type: "any",
|
|
276
|
-
|
|
285
|
+
isOptional: false,
|
|
277
286
|
description:
|
|
278
287
|
"Output record provided to the scorer. For agents, this is usually the agent's response. For workflows, this is the workflow's output.",
|
|
279
288
|
},
|
|
280
289
|
{
|
|
281
290
|
name: "run.runId",
|
|
282
291
|
type: "string",
|
|
283
|
-
|
|
292
|
+
isOptional: false,
|
|
284
293
|
description: "Unique identifier for this scoring run.",
|
|
285
294
|
},
|
|
286
295
|
{
|
|
287
296
|
name: "run.requestContext",
|
|
288
297
|
type: "object",
|
|
289
|
-
|
|
298
|
+
isOptional: true,
|
|
290
299
|
description:
|
|
291
300
|
"Request Context from the agent or workflow step being evaluated (optional).",
|
|
292
301
|
},
|
|
293
302
|
{
|
|
294
303
|
name: "results.preprocessStepResult",
|
|
295
304
|
type: "any",
|
|
296
|
-
|
|
305
|
+
isOptional: true,
|
|
297
306
|
description: "Result from preprocess step, if defined (optional).",
|
|
298
307
|
},
|
|
299
308
|
]}
|
|
@@ -309,26 +318,26 @@ The method can return any value. The returned value will be available to subsequ
|
|
|
309
318
|
{
|
|
310
319
|
name: "description",
|
|
311
320
|
type: "string",
|
|
312
|
-
|
|
321
|
+
isOptional: false,
|
|
313
322
|
description: "Description of what this analysis step does.",
|
|
314
323
|
},
|
|
315
324
|
{
|
|
316
325
|
name: "outputSchema",
|
|
317
326
|
type: "ZodSchema",
|
|
318
|
-
|
|
327
|
+
isOptional: false,
|
|
319
328
|
description: "Zod schema for the expected output of the analyze step.",
|
|
320
329
|
},
|
|
321
330
|
{
|
|
322
331
|
name: "createPrompt",
|
|
323
332
|
type: "function",
|
|
324
|
-
|
|
333
|
+
isOptional: false,
|
|
325
334
|
description:
|
|
326
335
|
"Function: ({ run, results }) => string. Returns the prompt for the LLM.",
|
|
327
336
|
},
|
|
328
337
|
{
|
|
329
338
|
name: "judge",
|
|
330
339
|
type: "object",
|
|
331
|
-
|
|
340
|
+
isOptional: true,
|
|
332
341
|
description:
|
|
333
342
|
"(Optional) LLM judge for this step (can override main judge). See Judge Object section.",
|
|
334
343
|
},
|
|
@@ -347,40 +356,40 @@ Function: `({ run, results }) => number`
|
|
|
347
356
|
{
|
|
348
357
|
name: "run.input",
|
|
349
358
|
type: "any",
|
|
350
|
-
|
|
359
|
+
isOptional: false,
|
|
351
360
|
description:
|
|
352
361
|
"Input records provided to the scorer. If the scorer is added to an agent, this will be an array of user messages, e.g. `[{ role: 'user', content: 'hello world' }]`. If the scorer is used in a workflow, this will be the input of the workflow.",
|
|
353
362
|
},
|
|
354
363
|
{
|
|
355
364
|
name: "run.output",
|
|
356
365
|
type: "any",
|
|
357
|
-
|
|
366
|
+
isOptional: false,
|
|
358
367
|
description:
|
|
359
368
|
"Output record provided to the scorer. For agents, this is usually the agent's response. For workflows, this is the workflow's output.",
|
|
360
369
|
},
|
|
361
370
|
{
|
|
362
371
|
name: "run.runId",
|
|
363
372
|
type: "string",
|
|
364
|
-
|
|
373
|
+
isOptional: false,
|
|
365
374
|
description: "Unique identifier for this scoring run.",
|
|
366
375
|
},
|
|
367
376
|
{
|
|
368
377
|
name: "run.requestContext",
|
|
369
378
|
type: "object",
|
|
370
|
-
|
|
379
|
+
isOptional: true,
|
|
371
380
|
description:
|
|
372
381
|
"Request Context from the agent or workflow step being evaluated (optional).",
|
|
373
382
|
},
|
|
374
383
|
{
|
|
375
384
|
name: "results.preprocessStepResult",
|
|
376
385
|
type: "any",
|
|
377
|
-
|
|
386
|
+
isOptional: true,
|
|
378
387
|
description: "Result from preprocess step, if defined (optional).",
|
|
379
388
|
},
|
|
380
389
|
{
|
|
381
390
|
name: "results.analyzeStepResult",
|
|
382
391
|
type: "any",
|
|
383
|
-
|
|
392
|
+
isOptional: true,
|
|
384
393
|
description: "Result from analyze step, if defined (optional).",
|
|
385
394
|
},
|
|
386
395
|
]}
|
|
@@ -396,27 +405,27 @@ The method must return a numerical score.
|
|
|
396
405
|
{
|
|
397
406
|
name: "description",
|
|
398
407
|
type: "string",
|
|
399
|
-
|
|
408
|
+
isOptional: false,
|
|
400
409
|
description: "Description of what this scoring step does.",
|
|
401
410
|
},
|
|
402
411
|
{
|
|
403
412
|
name: "outputSchema",
|
|
404
413
|
type: "ZodSchema",
|
|
405
|
-
|
|
414
|
+
isOptional: false,
|
|
406
415
|
description:
|
|
407
416
|
"Zod schema for the expected output of the generateScore step.",
|
|
408
417
|
},
|
|
409
418
|
{
|
|
410
419
|
name: "createPrompt",
|
|
411
420
|
type: "function",
|
|
412
|
-
|
|
421
|
+
isOptional: false,
|
|
413
422
|
description:
|
|
414
423
|
"Function: ({ run, results }) => string. Returns the prompt for the LLM.",
|
|
415
424
|
},
|
|
416
425
|
{
|
|
417
426
|
name: "judge",
|
|
418
427
|
type: "object",
|
|
419
|
-
|
|
428
|
+
isOptional: true,
|
|
420
429
|
description:
|
|
421
430
|
"(Optional) LLM judge for this step (can override main judge). See Judge Object section.",
|
|
422
431
|
},
|
|
@@ -430,7 +439,7 @@ When using prompt object mode, you must also provide a `calculateScore` function
|
|
|
430
439
|
{
|
|
431
440
|
name: "calculateScore",
|
|
432
441
|
type: "function",
|
|
433
|
-
|
|
442
|
+
isOptional: false,
|
|
434
443
|
description:
|
|
435
444
|
"Function: ({ run, results, analyzeStepResult }) => number. Converts the LLM's structured output into a numerical score.",
|
|
436
445
|
},
|
|
@@ -449,46 +458,46 @@ Function: `({ run, results, score }) => string`
|
|
|
449
458
|
{
|
|
450
459
|
name: "run.input",
|
|
451
460
|
type: "any",
|
|
452
|
-
|
|
461
|
+
isOptional: false,
|
|
453
462
|
description:
|
|
454
463
|
"Input records provided to the scorer. If the scorer is added to an agent, this will be an array of user messages, e.g. `[{ role: 'user', content: 'hello world' }]`. If the scorer is used in a workflow, this will be the input of the workflow.",
|
|
455
464
|
},
|
|
456
465
|
{
|
|
457
466
|
name: "run.output",
|
|
458
467
|
type: "any",
|
|
459
|
-
|
|
468
|
+
isOptional: false,
|
|
460
469
|
description:
|
|
461
470
|
"Output record provided to the scorer. For agents, this is usually the agent's response. For workflows, this is the workflow's output.",
|
|
462
471
|
},
|
|
463
472
|
{
|
|
464
473
|
name: "run.runId",
|
|
465
474
|
type: "string",
|
|
466
|
-
|
|
475
|
+
isOptional: false,
|
|
467
476
|
description: "Unique identifier for this scoring run.",
|
|
468
477
|
},
|
|
469
478
|
{
|
|
470
479
|
name: "run.requestContext",
|
|
471
480
|
type: "object",
|
|
472
|
-
|
|
481
|
+
isOptional: true,
|
|
473
482
|
description:
|
|
474
483
|
"Request Context from the agent or workflow step being evaluated (optional).",
|
|
475
484
|
},
|
|
476
485
|
{
|
|
477
486
|
name: "results.preprocessStepResult",
|
|
478
487
|
type: "any",
|
|
479
|
-
|
|
488
|
+
isOptional: true,
|
|
480
489
|
description: "Result from preprocess step, if defined (optional).",
|
|
481
490
|
},
|
|
482
491
|
{
|
|
483
492
|
name: "results.analyzeStepResult",
|
|
484
493
|
type: "any",
|
|
485
|
-
|
|
494
|
+
isOptional: true,
|
|
486
495
|
description: "Result from analyze step, if defined (optional).",
|
|
487
496
|
},
|
|
488
497
|
{
|
|
489
498
|
name: "score",
|
|
490
499
|
type: "number",
|
|
491
|
-
|
|
500
|
+
isOptional: false,
|
|
492
501
|
description: "Score computed by the generateScore step.",
|
|
493
502
|
},
|
|
494
503
|
]}
|
|
@@ -504,20 +513,20 @@ The method must return a string explaining the score.
|
|
|
504
513
|
{
|
|
505
514
|
name: "description",
|
|
506
515
|
type: "string",
|
|
507
|
-
|
|
516
|
+
isOptional: false,
|
|
508
517
|
description: "Description of what this reasoning step does.",
|
|
509
518
|
},
|
|
510
519
|
{
|
|
511
520
|
name: "createPrompt",
|
|
512
521
|
type: "function",
|
|
513
|
-
|
|
522
|
+
isOptional: false,
|
|
514
523
|
description:
|
|
515
524
|
"Function: ({ run, results, score }) => string. Returns the prompt for the LLM.",
|
|
516
525
|
},
|
|
517
526
|
{
|
|
518
527
|
name: "judge",
|
|
519
528
|
type: "object",
|
|
520
|
-
|
|
529
|
+
isOptional: true,
|
|
521
530
|
description:
|
|
522
531
|
"(Optional) LLM judge for this step (can override main judge). See Judge Object section.",
|
|
523
532
|
},
|
|
@@ -14,6 +14,8 @@ The `MastraScorer` class is the base class for all scorers in Mastra. It provide
|
|
|
14
14
|
Use the `createScorer` factory function, which returns a `MastraScorer` instance:
|
|
15
15
|
|
|
16
16
|
```typescript
|
|
17
|
+
import { createScorer } from "@mastra/core/evals";
|
|
18
|
+
|
|
17
19
|
const scorer = createScorer({
|
|
18
20
|
name: "My Custom Scorer",
|
|
19
21
|
description: "Evaluates responses based on custom criteria",
|
|
@@ -222,24 +222,24 @@ Shuts down all observability instances and clears the registry.
|
|
|
222
222
|
|
|
223
223
|
### Documentation
|
|
224
224
|
|
|
225
|
-
- [Tracing Overview]/docs/v1/observability/tracing/overview) - Concepts and usage guide
|
|
226
|
-
- [Sampling Strategies]/docs/v1/observability/tracing/overview#sampling-strategies) - Sampling configuration details
|
|
227
|
-
- [Multi-Config Setup]/docs/v1/observability/tracing/overview#multi-config-setup) - Using multiple configurations
|
|
225
|
+
- [Tracing Overview](/docs/v1/observability/tracing/overview) - Concepts and usage guide
|
|
226
|
+
- [Sampling Strategies](/docs/v1/observability/tracing/overview#sampling-strategies) - Sampling configuration details
|
|
227
|
+
- [Multi-Config Setup](/docs/v1/observability/tracing/overview#multi-config-setup) - Using multiple configurations
|
|
228
228
|
|
|
229
229
|
### Reference
|
|
230
230
|
|
|
231
|
-
- [Tracing Classes]/reference/v1/observability/tracing/instances) - Core tracing classes
|
|
232
|
-
- [Interfaces]/reference/v1/observability/tracing/interfaces) - Type definitions
|
|
233
|
-
- [Spans Reference]/reference/v1/observability/tracing/spans) - Span lifecycle
|
|
231
|
+
- [Tracing Classes](/reference/v1/observability/tracing/instances) - Core tracing classes
|
|
232
|
+
- [Interfaces](/reference/v1/observability/tracing/interfaces) - Type definitions
|
|
233
|
+
- [Spans Reference](/reference/v1/observability/tracing/spans) - Span lifecycle
|
|
234
234
|
|
|
235
235
|
### Examples
|
|
236
236
|
|
|
237
|
-
- [Basic Tracing]/examples/v1/observability/basic-tracing) - Getting started
|
|
237
|
+
- [Basic Tracing](/examples/v1/observability/basic-tracing) - Getting started
|
|
238
238
|
|
|
239
239
|
### Exporters
|
|
240
240
|
|
|
241
|
-
- [DefaultExporter]/reference/v1/observability/tracing/exporters/default-exporter) - Storage configuration
|
|
242
|
-
- [CloudExporter]/reference/v1/observability/tracing/exporters/cloud-exporter) - Cloud setup
|
|
243
|
-
- [Braintrust]/reference/v1/observability/tracing/exporters/braintrust) - Braintrust integration
|
|
244
|
-
- [Langfuse]/reference/v1/observability/tracing/exporters/langfuse) - Langfuse integration
|
|
245
|
-
- [LangSmith]/reference/v1/observability/tracing/exporters/langsmith) - LangSmith integration
|
|
241
|
+
- [DefaultExporter](/reference/v1/observability/tracing/exporters/default-exporter) - Storage configuration
|
|
242
|
+
- [CloudExporter](/reference/v1/observability/tracing/exporters/cloud-exporter) - Cloud setup
|
|
243
|
+
- [Braintrust](/reference/v1/observability/tracing/exporters/braintrust) - Braintrust integration
|
|
244
|
+
- [Langfuse](/reference/v1/observability/tracing/exporters/langfuse) - Langfuse integration
|
|
245
|
+
- [LangSmith](/reference/v1/observability/tracing/exporters/langsmith) - LangSmith integration
|
|
@@ -159,6 +159,6 @@ The ArizeExporter implements [OpenInference Semantic Conventions](https://github
|
|
|
159
159
|
|
|
160
160
|
## Related
|
|
161
161
|
|
|
162
|
-
- [ArizeExporter Documentation]/docs/v1/observability/tracing/exporters/arize)
|
|
162
|
+
- [ArizeExporter Documentation](/docs/v1/observability/tracing/exporters/arize)
|
|
163
163
|
- [Phoenix Documentation](https://docs.arize.com/phoenix)
|
|
164
164
|
- [Arize AX Documentation](https://docs.arize.com/)
|
|
@@ -167,17 +167,17 @@ const customExporter = new CloudExporter({
|
|
|
167
167
|
|
|
168
168
|
### Documentation
|
|
169
169
|
|
|
170
|
-
- [Tracing Overview]/docs/v1/observability/tracing/overview) - Complete guide
|
|
171
|
-
- [Exporters]/docs/v1/observability/tracing/overview#exporters) - Exporter concepts
|
|
170
|
+
- [Tracing Overview](/docs/v1/observability/tracing/overview) - Complete guide
|
|
171
|
+
- [Exporters](/docs/v1/observability/tracing/overview#exporters) - Exporter concepts
|
|
172
172
|
|
|
173
173
|
### Other Exporters
|
|
174
174
|
|
|
175
|
-
- [DefaultExporter]/reference/v1/observability/tracing/exporters/default-exporter) - Storage persistence
|
|
176
|
-
- [ConsoleExporter]/reference/v1/observability/tracing/exporters/console-exporter) - Debug output
|
|
177
|
-
- [Langfuse]/reference/v1/observability/tracing/exporters/langfuse) - Langfuse integration
|
|
178
|
-
- [Braintrust]/reference/v1/observability/tracing/exporters/braintrust) - Braintrust integration
|
|
175
|
+
- [DefaultExporter](/reference/v1/observability/tracing/exporters/default-exporter) - Storage persistence
|
|
176
|
+
- [ConsoleExporter](/reference/v1/observability/tracing/exporters/console-exporter) - Debug output
|
|
177
|
+
- [Langfuse](/reference/v1/observability/tracing/exporters/langfuse) - Langfuse integration
|
|
178
|
+
- [Braintrust](/reference/v1/observability/tracing/exporters/braintrust) - Braintrust integration
|
|
179
179
|
|
|
180
180
|
### Reference
|
|
181
181
|
|
|
182
|
-
- [Configuration]/reference/v1/observability/tracing/configuration) - Configuration options
|
|
183
|
-
- [Interfaces]/reference/v1/observability/tracing/interfaces) - Type definitions
|
|
182
|
+
- [Configuration](/reference/v1/observability/tracing/configuration) - Configuration options
|
|
183
|
+
- [Interfaces](/reference/v1/observability/tracing/interfaces) - Type definitions
|
|
@@ -168,17 +168,17 @@ const exporterWithLogger = new ConsoleExporter({
|
|
|
168
168
|
|
|
169
169
|
### Documentation
|
|
170
170
|
|
|
171
|
-
- [Tracing Overview]/docs/v1/observability/tracing/overview) - Complete guide
|
|
172
|
-
- [Exporters]/docs/v1/observability/tracing/overview#exporters) - Exporter concepts
|
|
171
|
+
- [Tracing Overview](/docs/v1/observability/tracing/overview) - Complete guide
|
|
172
|
+
- [Exporters](/docs/v1/observability/tracing/overview#exporters) - Exporter concepts
|
|
173
173
|
|
|
174
174
|
### Other Exporters
|
|
175
175
|
|
|
176
|
-
- [DefaultExporter]/reference/v1/observability/tracing/exporters/default-exporter) - Storage persistence
|
|
177
|
-
- [CloudExporter]/reference/v1/observability/tracing/exporters/cloud-exporter) - Mastra Cloud
|
|
178
|
-
- [Langfuse]/reference/v1/observability/tracing/exporters/langfuse) - Langfuse integration
|
|
179
|
-
- [Braintrust]/reference/v1/observability/tracing/exporters/braintrust) - Braintrust integration
|
|
176
|
+
- [DefaultExporter](/reference/v1/observability/tracing/exporters/default-exporter) - Storage persistence
|
|
177
|
+
- [CloudExporter](/reference/v1/observability/tracing/exporters/cloud-exporter) - Mastra Cloud
|
|
178
|
+
- [Langfuse](/reference/v1/observability/tracing/exporters/langfuse) - Langfuse integration
|
|
179
|
+
- [Braintrust](/reference/v1/observability/tracing/exporters/braintrust) - Braintrust integration
|
|
180
180
|
|
|
181
181
|
### Reference
|
|
182
182
|
|
|
183
|
-
- [Configuration]/reference/v1/observability/tracing/configuration) - Configuration options
|
|
184
|
-
- [Interfaces]/reference/v1/observability/tracing/interfaces) - Type definitions
|
|
183
|
+
- [Configuration](/reference/v1/observability/tracing/configuration) - Configuration options
|
|
184
|
+
- [Interfaces](/reference/v1/observability/tracing/interfaces) - Type definitions
|
|
@@ -189,17 +189,17 @@ const customExporter = new DefaultExporter({
|
|
|
189
189
|
|
|
190
190
|
### Documentation
|
|
191
191
|
|
|
192
|
-
- [Tracing Overview]/docs/v1/observability/tracing/overview) - Complete guide
|
|
193
|
-
- [Exporters]/docs/v1/observability/tracing/overview#exporters) - Exporter concepts
|
|
192
|
+
- [Tracing Overview](/docs/v1/observability/tracing/overview) - Complete guide
|
|
193
|
+
- [Exporters](/docs/v1/observability/tracing/overview#exporters) - Exporter concepts
|
|
194
194
|
|
|
195
195
|
### Other Exporters
|
|
196
196
|
|
|
197
|
-
- [CloudExporter]/reference/v1/observability/tracing/exporters/cloud-exporter) - Mastra Cloud
|
|
198
|
-
- [ConsoleExporter]/reference/v1/observability/tracing/exporters/console-exporter) - Debug output
|
|
199
|
-
- [Langfuse]/reference/v1/observability/tracing/exporters/langfuse) - Langfuse integration
|
|
200
|
-
- [Braintrust]/reference/v1/observability/tracing/exporters/braintrust) - Braintrust integration
|
|
197
|
+
- [CloudExporter](/reference/v1/observability/tracing/exporters/cloud-exporter) - Mastra Cloud
|
|
198
|
+
- [ConsoleExporter](/reference/v1/observability/tracing/exporters/console-exporter) - Debug output
|
|
199
|
+
- [Langfuse](/reference/v1/observability/tracing/exporters/langfuse) - Langfuse integration
|
|
200
|
+
- [Braintrust](/reference/v1/observability/tracing/exporters/braintrust) - Braintrust integration
|
|
201
201
|
|
|
202
202
|
### Reference
|
|
203
203
|
|
|
204
|
-
- [Configuration]/reference/v1/observability/tracing/configuration) - Configuration options
|
|
205
|
-
- [Interfaces]/reference/v1/observability/tracing/interfaces) - Type definitions
|
|
204
|
+
- [Configuration](/reference/v1/observability/tracing/configuration) - Configuration options
|
|
205
|
+
- [Interfaces](/reference/v1/observability/tracing/interfaces) - Type definitions
|
|
@@ -152,17 +152,17 @@ class CustomObservabilityInstance extends BaseObservabilityInstance {
|
|
|
152
152
|
|
|
153
153
|
### Documentation
|
|
154
154
|
|
|
155
|
-
- [Tracing Overview]/docs/v1/observability/tracing/overview) - Concepts and usage guide
|
|
156
|
-
- [Configuration Reference]/reference/v1/observability/tracing/configuration) - Configuration options
|
|
157
|
-
- [Interfaces Reference]/reference/v1/observability/tracing/interfaces) - Type definitions
|
|
158
|
-
- [Spans Reference]/reference/v1/observability/tracing/spans) - Span lifecycle and methods
|
|
155
|
+
- [Tracing Overview](/docs/v1/observability/tracing/overview) - Concepts and usage guide
|
|
156
|
+
- [Configuration Reference](/reference/v1/observability/tracing/configuration) - Configuration options
|
|
157
|
+
- [Interfaces Reference](/reference/v1/observability/tracing/interfaces) - Type definitions
|
|
158
|
+
- [Spans Reference](/reference/v1/observability/tracing/spans) - Span lifecycle and methods
|
|
159
159
|
|
|
160
160
|
### Examples
|
|
161
161
|
|
|
162
|
-
- [Basic Tracing]/examples/v1/observability/basic-tracing) - Getting started example
|
|
162
|
+
- [Basic Tracing](/examples/v1/observability/basic-tracing) - Getting started example
|
|
163
163
|
|
|
164
164
|
### Exporters
|
|
165
165
|
|
|
166
|
-
- [DefaultExporter]/reference/v1/observability/tracing/exporters/default-exporter) - Storage persistence
|
|
167
|
-
- [CloudExporter]/reference/v1/observability/tracing/exporters/cloud-exporter) - Mastra Cloud integration
|
|
168
|
-
- [ConsoleExporter]/reference/v1/observability/tracing/exporters/console-exporter) - Debug output
|
|
166
|
+
- [DefaultExporter](/reference/v1/observability/tracing/exporters/default-exporter) - Storage persistence
|
|
167
|
+
- [CloudExporter](/reference/v1/observability/tracing/exporters/cloud-exporter) - Mastra Cloud integration
|
|
168
|
+
- [ConsoleExporter](/reference/v1/observability/tracing/exporters/console-exporter) - Debug output
|
|
@@ -698,16 +698,16 @@ enum InternalSpans {
|
|
|
698
698
|
|
|
699
699
|
### Documentation
|
|
700
700
|
|
|
701
|
-
- [Tracing Overview]/docs/v1/observability/tracing/overview) - Complete guide to Tracing
|
|
702
|
-
- [Creating Child Spans]/docs/v1/observability/tracing/overview#creating-child-spans) - Working with span hierarchies
|
|
703
|
-
- [Adding Custom Metadata]/docs/v1/observability/tracing/overview#adding-custom-metadata) - Enriching traces
|
|
701
|
+
- [Tracing Overview](/docs/v1/observability/tracing/overview) - Complete guide to Tracing
|
|
702
|
+
- [Creating Child Spans](/docs/v1/observability/tracing/overview#creating-child-spans) - Working with span hierarchies
|
|
703
|
+
- [Adding Custom Metadata](/docs/v1/observability/tracing/overview#adding-custom-metadata) - Enriching traces
|
|
704
704
|
|
|
705
705
|
### Reference
|
|
706
706
|
|
|
707
|
-
- [Configuration]/reference/v1/observability/tracing/configuration) - Registry and configuration
|
|
708
|
-
- [Tracing Classes]/reference/v1/observability/tracing/instances) - Core implementations
|
|
709
|
-
- [Spans Reference]/reference/v1/observability/tracing/spans) - Span lifecycle methods
|
|
707
|
+
- [Configuration](/reference/v1/observability/tracing/configuration) - Registry and configuration
|
|
708
|
+
- [Tracing Classes](/reference/v1/observability/tracing/instances) - Core implementations
|
|
709
|
+
- [Spans Reference](/reference/v1/observability/tracing/spans) - Span lifecycle methods
|
|
710
710
|
|
|
711
711
|
### Examples
|
|
712
712
|
|
|
713
|
-
- [Basic Tracing]/examples/v1/observability/basic-tracing) - Implementation example
|
|
713
|
+
- [Basic Tracing](/examples/v1/observability/basic-tracing) - Implementation example
|
|
@@ -377,16 +377,16 @@ A span that performs no operations. All methods are no-ops:
|
|
|
377
377
|
|
|
378
378
|
### Documentation
|
|
379
379
|
|
|
380
|
-
- [Tracing Overview]/docs/v1/observability/tracing/overview) - Concepts and usage
|
|
381
|
-
- [Creating Child Spans]/docs/v1/observability/tracing/overview#creating-child-spans) - Practical examples
|
|
382
|
-
- [Retrieving Trace IDs]/docs/v1/observability/tracing/overview#retrieving-trace-ids) - Using trace IDs
|
|
380
|
+
- [Tracing Overview](/docs/v1/observability/tracing/overview) - Concepts and usage
|
|
381
|
+
- [Creating Child Spans](/docs/v1/observability/tracing/overview#creating-child-spans) - Practical examples
|
|
382
|
+
- [Retrieving Trace IDs](/docs/v1/observability/tracing/overview#retrieving-trace-ids) - Using trace IDs
|
|
383
383
|
|
|
384
384
|
### Reference
|
|
385
385
|
|
|
386
|
-
- [Tracing Classes]/reference/v1/observability/tracing/instances) - Core tracing classes
|
|
387
|
-
- [Interfaces]/reference/v1/observability/tracing/interfaces) - Complete type reference
|
|
388
|
-
- [Configuration]/reference/v1/observability/tracing/configuration) - Configuration options
|
|
386
|
+
- [Tracing Classes](/reference/v1/observability/tracing/instances) - Core tracing classes
|
|
387
|
+
- [Interfaces](/reference/v1/observability/tracing/interfaces) - Complete type reference
|
|
388
|
+
- [Configuration](/reference/v1/observability/tracing/configuration) - Configuration options
|
|
389
389
|
|
|
390
390
|
### Examples
|
|
391
391
|
|
|
392
|
-
- [Basic Tracing]/examples/v1/observability/basic-tracing) - Working with spans
|
|
392
|
+
- [Basic Tracing](/examples/v1/observability/basic-tracing) - Working with spans
|