@mastra/mcp-docs-server 0.0.0-commonjs-20250414101718
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.docs/organized/changelogs/%40mastra%2Fastra.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fchroma.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fclickhouse.md +161 -0
- package/.docs/organized/changelogs/%40mastra%2Fclient-js.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fcloudflare.md +110 -0
- package/.docs/organized/changelogs/%40mastra%2Fcore.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fdeployer-cloudflare.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fdeployer-netlify.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fdeployer-vercel.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fdeployer.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fevals.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Ffirecrawl.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fgithub.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Floggers.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fmcp-docs-server.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fmcp-registry-registry.md +26 -0
- package/.docs/organized/changelogs/%40mastra%2Fmcp.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fmem0.md +196 -0
- package/.docs/organized/changelogs/%40mastra%2Fmemory.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fpg.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fpinecone.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fplayground-ui.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fqdrant.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Frag.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fragie.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fserver.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fspeech-azure.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fspeech-deepgram.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fspeech-elevenlabs.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fspeech-google.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fspeech-ibm.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fspeech-murf.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fspeech-openai.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fspeech-playai.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fspeech-replicate.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fspeech-speechify.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fturbopuffer.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fupstash.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fvectorize.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fvoice-azure.md +250 -0
- package/.docs/organized/changelogs/%40mastra%2Fvoice-cloudflare.md +250 -0
- package/.docs/organized/changelogs/%40mastra%2Fvoice-deepgram.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fvoice-elevenlabs.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fvoice-google.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fvoice-murf.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fvoice-openai-realtime.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fvoice-openai.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fvoice-playai.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fvoice-sarvam.md +302 -0
- package/.docs/organized/changelogs/%40mastra%2Fvoice-speechify.md +302 -0
- package/.docs/organized/changelogs/create-mastra.md +302 -0
- package/.docs/organized/changelogs/mastra.md +302 -0
- package/.docs/organized/code-examples/agent-network.md +282 -0
- package/.docs/organized/code-examples/agent.md +388 -0
- package/.docs/organized/code-examples/ai-sdk-useChat.md +378 -0
- package/.docs/organized/code-examples/assistant-ui.md +37 -0
- package/.docs/organized/code-examples/bird-checker-with-express.md +235 -0
- package/.docs/organized/code-examples/bird-checker-with-nextjs-and-eval.md +360 -0
- package/.docs/organized/code-examples/bird-checker-with-nextjs.md +250 -0
- package/.docs/organized/code-examples/client-side-tools.md +69 -0
- package/.docs/organized/code-examples/crypto-chatbot.md +96 -0
- package/.docs/organized/code-examples/fireworks-r1.md +159 -0
- package/.docs/organized/code-examples/mcp-registry-registry.md +63 -0
- package/.docs/organized/code-examples/memory-todo-agent.md +164 -0
- package/.docs/organized/code-examples/memory-with-context.md +167 -0
- package/.docs/organized/code-examples/memory-with-libsql.md +204 -0
- package/.docs/organized/code-examples/memory-with-mem0.md +121 -0
- package/.docs/organized/code-examples/memory-with-pg.md +224 -0
- package/.docs/organized/code-examples/memory-with-upstash.md +268 -0
- package/.docs/organized/code-examples/quick-start.md +129 -0
- package/.docs/organized/code-examples/stock-price-tool.md +124 -0
- package/.docs/organized/code-examples/weather-agent.md +353 -0
- package/.docs/organized/code-examples/workflow-ai-recruiter.md +159 -0
- package/.docs/organized/code-examples/workflow-with-inline-steps.md +111 -0
- package/.docs/organized/code-examples/workflow-with-memory.md +393 -0
- package/.docs/organized/code-examples/workflow-with-separate-steps.md +131 -0
- package/.docs/raw/agents/adding-tools.mdx +317 -0
- package/.docs/raw/agents/adding-voice.mdx +175 -0
- package/.docs/raw/agents/agent-memory.mdx +62 -0
- package/.docs/raw/agents/mcp-guide.mdx +215 -0
- package/.docs/raw/agents/overview.mdx +303 -0
- package/.docs/raw/community/discord.mdx +12 -0
- package/.docs/raw/community/licensing.mdx +63 -0
- package/.docs/raw/deployment/client.mdx +120 -0
- package/.docs/raw/deployment/deployment.mdx +127 -0
- package/.docs/raw/deployment/server.mdx +282 -0
- package/.docs/raw/evals/custom-eval.mdx +22 -0
- package/.docs/raw/evals/overview.mdx +95 -0
- package/.docs/raw/evals/running-in-ci.mdx +81 -0
- package/.docs/raw/evals/textual-evals.mdx +54 -0
- package/.docs/raw/faq/index.mdx +63 -0
- package/.docs/raw/frameworks/ai-sdk.mdx +296 -0
- package/.docs/raw/frameworks/next-js.mdx +238 -0
- package/.docs/raw/getting-started/installation.mdx +442 -0
- package/.docs/raw/getting-started/mcp-docs-server.mdx +141 -0
- package/.docs/raw/getting-started/project-structure.mdx +80 -0
- package/.docs/raw/index.mdx +22 -0
- package/.docs/raw/integrations/index.mdx +213 -0
- package/.docs/raw/local-dev/add-to-existing-project.mdx +48 -0
- package/.docs/raw/local-dev/creating-a-new-project.mdx +54 -0
- package/.docs/raw/local-dev/mastra-dev.mdx +108 -0
- package/.docs/raw/memory/memory-processors.mdx +131 -0
- package/.docs/raw/memory/overview.mdx +119 -0
- package/.docs/raw/memory/semantic-recall.mdx +122 -0
- package/.docs/raw/memory/working-memory.mdx +87 -0
- package/.docs/raw/observability/logging.mdx +38 -0
- package/.docs/raw/observability/nextjs-tracing.mdx +108 -0
- package/.docs/raw/observability/tracing.mdx +115 -0
- package/.docs/raw/rag/chunking-and-embedding.mdx +156 -0
- package/.docs/raw/rag/overview.mdx +85 -0
- package/.docs/raw/rag/retrieval.mdx +365 -0
- package/.docs/raw/rag/vector-databases.mdx +340 -0
- package/.docs/raw/reference/agents/createTool.mdx +229 -0
- package/.docs/raw/reference/agents/generate.mdx +334 -0
- package/.docs/raw/reference/agents/getAgent.mdx +54 -0
- package/.docs/raw/reference/agents/stream.mdx +369 -0
- package/.docs/raw/reference/cli/build.mdx +55 -0
- package/.docs/raw/reference/cli/dev.mdx +134 -0
- package/.docs/raw/reference/cli/init.mdx +43 -0
- package/.docs/raw/reference/client-js/agents.mdx +107 -0
- package/.docs/raw/reference/client-js/error-handling.mdx +38 -0
- package/.docs/raw/reference/client-js/logs.mdx +24 -0
- package/.docs/raw/reference/client-js/memory.mdx +97 -0
- package/.docs/raw/reference/client-js/telemetry.mdx +20 -0
- package/.docs/raw/reference/client-js/tools.mdx +44 -0
- package/.docs/raw/reference/client-js/vectors.mdx +79 -0
- package/.docs/raw/reference/client-js/workflows.mdx +136 -0
- package/.docs/raw/reference/core/mastra-class.mdx +232 -0
- package/.docs/raw/reference/deployer/cloudflare.mdx +207 -0
- package/.docs/raw/reference/deployer/deployer.mdx +159 -0
- package/.docs/raw/reference/deployer/netlify.mdx +109 -0
- package/.docs/raw/reference/deployer/vercel.mdx +117 -0
- package/.docs/raw/reference/evals/answer-relevancy.mdx +186 -0
- package/.docs/raw/reference/evals/bias.mdx +186 -0
- package/.docs/raw/reference/evals/completeness.mdx +174 -0
- package/.docs/raw/reference/evals/content-similarity.mdx +183 -0
- package/.docs/raw/reference/evals/context-position.mdx +190 -0
- package/.docs/raw/reference/evals/context-precision.mdx +189 -0
- package/.docs/raw/reference/evals/context-relevancy.mdx +188 -0
- package/.docs/raw/reference/evals/contextual-recall.mdx +191 -0
- package/.docs/raw/reference/evals/faithfulness.mdx +193 -0
- package/.docs/raw/reference/evals/hallucination.mdx +219 -0
- package/.docs/raw/reference/evals/keyword-coverage.mdx +176 -0
- package/.docs/raw/reference/evals/prompt-alignment.mdx +238 -0
- package/.docs/raw/reference/evals/summarization.mdx +205 -0
- package/.docs/raw/reference/evals/textual-difference.mdx +161 -0
- package/.docs/raw/reference/evals/tone-consistency.mdx +181 -0
- package/.docs/raw/reference/evals/toxicity.mdx +165 -0
- package/.docs/raw/reference/index.mdx +12 -0
- package/.docs/raw/reference/memory/Memory.mdx +212 -0
- package/.docs/raw/reference/memory/createThread.mdx +95 -0
- package/.docs/raw/reference/memory/getThreadById.mdx +46 -0
- package/.docs/raw/reference/memory/getThreadsByResourceId.mdx +48 -0
- package/.docs/raw/reference/memory/query.mdx +167 -0
- package/.docs/raw/reference/networks/agent-network.mdx +159 -0
- package/.docs/raw/reference/observability/create-logger.mdx +106 -0
- package/.docs/raw/reference/observability/logger.mdx +55 -0
- package/.docs/raw/reference/observability/otel-config.mdx +120 -0
- package/.docs/raw/reference/observability/providers/braintrust.mdx +40 -0
- package/.docs/raw/reference/observability/providers/dash0.mdx +40 -0
- package/.docs/raw/reference/observability/providers/index.mdx +16 -0
- package/.docs/raw/reference/observability/providers/laminar.mdx +41 -0
- package/.docs/raw/reference/observability/providers/langfuse.mdx +51 -0
- package/.docs/raw/reference/observability/providers/langsmith.mdx +48 -0
- package/.docs/raw/reference/observability/providers/langwatch.mdx +45 -0
- package/.docs/raw/reference/observability/providers/new-relic.mdx +40 -0
- package/.docs/raw/reference/observability/providers/signoz.mdx +40 -0
- package/.docs/raw/reference/observability/providers/traceloop.mdx +40 -0
- package/.docs/raw/reference/rag/astra.mdx +258 -0
- package/.docs/raw/reference/rag/chroma.mdx +281 -0
- package/.docs/raw/reference/rag/chunk.mdx +235 -0
- package/.docs/raw/reference/rag/document.mdx +127 -0
- package/.docs/raw/reference/rag/embeddings.mdx +160 -0
- package/.docs/raw/reference/rag/extract-params.mdx +226 -0
- package/.docs/raw/reference/rag/graph-rag.mdx +182 -0
- package/.docs/raw/reference/rag/libsql.mdx +357 -0
- package/.docs/raw/reference/rag/metadata-filters.mdx +298 -0
- package/.docs/raw/reference/rag/pg.mdx +477 -0
- package/.docs/raw/reference/rag/pinecone.mdx +281 -0
- package/.docs/raw/reference/rag/qdrant.mdx +236 -0
- package/.docs/raw/reference/rag/rerank.mdx +212 -0
- package/.docs/raw/reference/rag/turbopuffer.mdx +249 -0
- package/.docs/raw/reference/rag/upstash.mdx +247 -0
- package/.docs/raw/reference/rag/vectorize.mdx +298 -0
- package/.docs/raw/reference/storage/libsql.mdx +74 -0
- package/.docs/raw/reference/storage/postgresql.mdx +48 -0
- package/.docs/raw/reference/storage/upstash.mdx +86 -0
- package/.docs/raw/reference/tools/client.mdx +207 -0
- package/.docs/raw/reference/tools/document-chunker-tool.mdx +141 -0
- package/.docs/raw/reference/tools/graph-rag-tool.mdx +154 -0
- package/.docs/raw/reference/tools/mcp-configuration.mdx +206 -0
- package/.docs/raw/reference/tools/vector-query-tool.mdx +212 -0
- package/.docs/raw/reference/voice/composite-voice.mdx +140 -0
- package/.docs/raw/reference/voice/deepgram.mdx +164 -0
- package/.docs/raw/reference/voice/elevenlabs.mdx +216 -0
- package/.docs/raw/reference/voice/google.mdx +198 -0
- package/.docs/raw/reference/voice/mastra-voice.mdx +394 -0
- package/.docs/raw/reference/voice/murf.mdx +251 -0
- package/.docs/raw/reference/voice/openai-realtime.mdx +431 -0
- package/.docs/raw/reference/voice/openai.mdx +168 -0
- package/.docs/raw/reference/voice/playai.mdx +159 -0
- package/.docs/raw/reference/voice/sarvam.mdx +260 -0
- package/.docs/raw/reference/voice/speechify.mdx +145 -0
- package/.docs/raw/reference/voice/voice.answer.mdx +122 -0
- package/.docs/raw/reference/voice/voice.connect.mdx +124 -0
- package/.docs/raw/reference/voice/voice.listen.mdx +195 -0
- package/.docs/raw/reference/voice/voice.on.mdx +189 -0
- package/.docs/raw/reference/voice/voice.send.mdx +118 -0
- package/.docs/raw/reference/voice/voice.speak.mdx +203 -0
- package/.docs/raw/reference/workflows/after.mdx +88 -0
- package/.docs/raw/reference/workflows/afterEvent.mdx +76 -0
- package/.docs/raw/reference/workflows/commit.mdx +37 -0
- package/.docs/raw/reference/workflows/createRun.mdx +77 -0
- package/.docs/raw/reference/workflows/else.mdx +72 -0
- package/.docs/raw/reference/workflows/events.mdx +305 -0
- package/.docs/raw/reference/workflows/execute.mdx +110 -0
- package/.docs/raw/reference/workflows/if.mdx +107 -0
- package/.docs/raw/reference/workflows/resume.mdx +155 -0
- package/.docs/raw/reference/workflows/resumeWithEvent.mdx +133 -0
- package/.docs/raw/reference/workflows/snapshots.mdx +207 -0
- package/.docs/raw/reference/workflows/start.mdx +84 -0
- package/.docs/raw/reference/workflows/step-class.mdx +100 -0
- package/.docs/raw/reference/workflows/step-condition.mdx +134 -0
- package/.docs/raw/reference/workflows/step-function.mdx +92 -0
- package/.docs/raw/reference/workflows/step-options.mdx +69 -0
- package/.docs/raw/reference/workflows/step-retries.mdx +203 -0
- package/.docs/raw/reference/workflows/suspend.mdx +70 -0
- package/.docs/raw/reference/workflows/then.mdx +74 -0
- package/.docs/raw/reference/workflows/until.mdx +165 -0
- package/.docs/raw/reference/workflows/watch.mdx +118 -0
- package/.docs/raw/reference/workflows/while.mdx +168 -0
- package/.docs/raw/reference/workflows/workflow.mdx +233 -0
- package/.docs/raw/storage/overview.mdx +378 -0
- package/.docs/raw/voice/overview.mdx +135 -0
- package/.docs/raw/voice/speech-to-text.mdx +45 -0
- package/.docs/raw/voice/text-to-speech.mdx +52 -0
- package/.docs/raw/voice/voice-to-voice.mdx +310 -0
- package/.docs/raw/workflows/control-flow.mdx +778 -0
- package/.docs/raw/workflows/dynamic-workflows.mdx +236 -0
- package/.docs/raw/workflows/error-handling.mdx +183 -0
- package/.docs/raw/workflows/nested-workflows.mdx +352 -0
- package/.docs/raw/workflows/overview.mdx +203 -0
- package/.docs/raw/workflows/steps.mdx +108 -0
- package/.docs/raw/workflows/suspend-and-resume.mdx +404 -0
- package/.docs/raw/workflows/variables.mdx +313 -0
- package/LICENSE.md +46 -0
- package/README.md +129 -0
- package/dist/_tsup-dts-rollup.d.ts +149 -0
- package/dist/chunk-QWYMT5LP.js +194 -0
- package/dist/prepare-docs/prepare.d.ts +1 -0
- package/dist/prepare-docs/prepare.js +1 -0
- package/dist/stdio.d.ts +1 -0
- package/dist/stdio.js +518 -0
- package/package.json +60 -0
|
@@ -0,0 +1,120 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "MastraClient"
|
|
3
|
+
description: "Learn how to set up and use the Mastra Client SDK"
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Mastra Client SDK
|
|
7
|
+
|
|
8
|
+
The Mastra Client SDK provides a simple and type-safe interface for interacting with your [Mastra Server](/docs/deployment/server) from your client environment.
|
|
9
|
+
|
|
10
|
+
## Development Requirements
|
|
11
|
+
|
|
12
|
+
To ensure smooth local development, make sure you have:
|
|
13
|
+
|
|
14
|
+
- Node.js 18.x or later installed
|
|
15
|
+
- TypeScript 4.7+ (if using TypeScript)
|
|
16
|
+
- A modern browser environment with Fetch API support
|
|
17
|
+
- Your local Mastra server running (typically on port 4111)
|
|
18
|
+
|
|
19
|
+
## Installation
|
|
20
|
+
|
|
21
|
+
import { Tabs } from "nextra/components";
|
|
22
|
+
|
|
23
|
+
<Tabs items={["npm", "yarn", "pnpm"]}>
|
|
24
|
+
<Tabs.Tab>
|
|
25
|
+
```bash copy
|
|
26
|
+
npm install @mastra/client-js
|
|
27
|
+
```
|
|
28
|
+
</Tabs.Tab>
|
|
29
|
+
<Tabs.Tab>
|
|
30
|
+
```bash copy
|
|
31
|
+
yarn add @mastra/client-js
|
|
32
|
+
```
|
|
33
|
+
</Tabs.Tab>
|
|
34
|
+
<Tabs.Tab>
|
|
35
|
+
```bash copy
|
|
36
|
+
pnpm add @mastra/client-js
|
|
37
|
+
```
|
|
38
|
+
</Tabs.Tab>
|
|
39
|
+
</Tabs>
|
|
40
|
+
|
|
41
|
+
## Initialize Mastra Client
|
|
42
|
+
|
|
43
|
+
To get started you'll need to initialize your MastraClient with necessary parameters:
|
|
44
|
+
|
|
45
|
+
```typescript
|
|
46
|
+
import { MastraClient } from "@mastra/client-js";
|
|
47
|
+
|
|
48
|
+
const client = new MastraClient({
|
|
49
|
+
baseUrl: "http://localhost:4111", // Default Mastra development server port
|
|
50
|
+
});
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
### Configuration Options
|
|
54
|
+
|
|
55
|
+
You can customize the client with various options:
|
|
56
|
+
|
|
57
|
+
```typescript
|
|
58
|
+
const client = new MastraClient({
|
|
59
|
+
// Required
|
|
60
|
+
baseUrl: "http://localhost:4111",
|
|
61
|
+
|
|
62
|
+
// Optional configurations for development
|
|
63
|
+
retries: 3, // Number of retry attempts
|
|
64
|
+
backoffMs: 300, // Initial retry backoff time
|
|
65
|
+
maxBackoffMs: 5000, // Maximum retry backoff time
|
|
66
|
+
headers: { // Custom headers for development
|
|
67
|
+
"X-Development": "true"
|
|
68
|
+
}
|
|
69
|
+
});
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
## Example
|
|
73
|
+
|
|
74
|
+
Once your MastraClient is initialized you can start making client calls via the type-safe
|
|
75
|
+
interface
|
|
76
|
+
|
|
77
|
+
```typescript
|
|
78
|
+
// Get a reference to your local agent
|
|
79
|
+
const agent = client.getAgent("dev-agent-id");
|
|
80
|
+
|
|
81
|
+
// Generate responses
|
|
82
|
+
const response = await agent.generate({
|
|
83
|
+
messages: [
|
|
84
|
+
{
|
|
85
|
+
role: "user",
|
|
86
|
+
content: "Hello, I'm testing the local development setup!"
|
|
87
|
+
}
|
|
88
|
+
]
|
|
89
|
+
});
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
## Available Features
|
|
93
|
+
|
|
94
|
+
Mastra client exposes all resources served by the Mastra Server
|
|
95
|
+
|
|
96
|
+
- [**Agents**](/reference/client-js/agents): Create and manage AI agents, generate responses, and handle streaming interactions
|
|
97
|
+
- [**Memory**](/reference/client-js/memory): Manage conversation threads and message history
|
|
98
|
+
- [**Tools**](/reference/client-js/tools): Access and execute tools available to agents
|
|
99
|
+
- [**Workflows**](/reference/client-js/workflows): Create and manage automated workflows
|
|
100
|
+
- [**Vectors**](/reference/client-js/vectors): Handle vector operations for semantic search and similarity matching
|
|
101
|
+
|
|
102
|
+
|
|
103
|
+
## Best Practices
|
|
104
|
+
1. **Error Handling**: Implement proper error handling for development scenarios
|
|
105
|
+
2. **Environment Variables**: Use environment variables for configuration
|
|
106
|
+
3. **Debugging**: Enable detailed logging when needed
|
|
107
|
+
|
|
108
|
+
```typescript
|
|
109
|
+
// Example with error handling and logging
|
|
110
|
+
try {
|
|
111
|
+
const agent = client.getAgent("dev-agent-id");
|
|
112
|
+
const response = await agent.generate({
|
|
113
|
+
messages: [{ role: "user", content: "Test message" }]
|
|
114
|
+
});
|
|
115
|
+
console.log("Response:", response);
|
|
116
|
+
} catch (error) {
|
|
117
|
+
console.error("Development error:", error);
|
|
118
|
+
}
|
|
119
|
+
```
|
|
120
|
+
|
|
@@ -0,0 +1,127 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Serverless Deployment"
|
|
3
|
+
description: "Build and deploy Mastra applications using platform-specific deployers or standard HTTP servers"
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Serverless Deployment
|
|
7
|
+
|
|
8
|
+
This guide covers deploying Mastra to Cloudflare Workers, Vercel, and Netlify using platform-specific deployers
|
|
9
|
+
|
|
10
|
+
For self-hosted Node.js server deployment, see the [Creating A Mastra Server](/docs/deployment/server) guide.
|
|
11
|
+
|
|
12
|
+
## Prerequisites
|
|
13
|
+
|
|
14
|
+
Before you begin, ensure you have:
|
|
15
|
+
|
|
16
|
+
- **Node.js** installed (version 18 or higher is recommended)
|
|
17
|
+
- If using a platform-specific deployer:
|
|
18
|
+
- An account with your chosen platform
|
|
19
|
+
- Required API keys or credentials
|
|
20
|
+
|
|
21
|
+
## Serverless Platform Deployers
|
|
22
|
+
|
|
23
|
+
Platform-specific deployers handle configuration and deployment for:
|
|
24
|
+
- **[Cloudflare Workers](/reference/deployer/cloudflare)**
|
|
25
|
+
- **[Vercel](/reference/deployer/vercel)**
|
|
26
|
+
- **[Netlify](/reference/deployer/netlify)**
|
|
27
|
+
|
|
28
|
+
As of April 2025, Mastra also offers [Mastra Cloud](https://mastra.ai/cloud-beta), a serverless agent environment with atomic deployments. You can sign up for the waitlist [here](https://mastra.ai/cloud-beta).
|
|
29
|
+
|
|
30
|
+
### Installing Deployers
|
|
31
|
+
|
|
32
|
+
```bash copy
|
|
33
|
+
# For Cloudflare
|
|
34
|
+
npm install @mastra/deployer-cloudflare
|
|
35
|
+
|
|
36
|
+
# For Vercel
|
|
37
|
+
npm install @mastra/deployer-vercel
|
|
38
|
+
|
|
39
|
+
# For Netlify
|
|
40
|
+
npm install @mastra/deployer-netlify
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
### Configuring Deployers
|
|
44
|
+
|
|
45
|
+
Configure the deployer in your entry file:
|
|
46
|
+
|
|
47
|
+
```typescript copy showLineNumbers
|
|
48
|
+
import { Mastra, createLogger } from '@mastra/core';
|
|
49
|
+
import { CloudflareDeployer } from '@mastra/deployer-cloudflare';
|
|
50
|
+
|
|
51
|
+
export const mastra = new Mastra({
|
|
52
|
+
agents: { /* your agents here */ },
|
|
53
|
+
logger: createLogger({ name: 'MyApp', level: 'debug' }),
|
|
54
|
+
deployer: new CloudflareDeployer({
|
|
55
|
+
scope: 'your-cloudflare-scope',
|
|
56
|
+
projectName: 'your-project-name',
|
|
57
|
+
// See complete configuration options in the reference docs
|
|
58
|
+
}),
|
|
59
|
+
});
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
### Deployer Configuration
|
|
63
|
+
|
|
64
|
+
Each deployer has specific configuration options. Below are basic examples, but refer to the reference documentation for complete details.
|
|
65
|
+
|
|
66
|
+
#### Cloudflare Deployer
|
|
67
|
+
|
|
68
|
+
```typescript copy showLineNumbers
|
|
69
|
+
new CloudflareDeployer({
|
|
70
|
+
scope: 'your-cloudflare-account-id',
|
|
71
|
+
projectName: 'your-project-name',
|
|
72
|
+
// For complete configuration options, see the reference documentation
|
|
73
|
+
})
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
[View Cloudflare Deployer Reference →](/reference/deployer/cloudflare)
|
|
77
|
+
|
|
78
|
+
#### Vercel Deployer
|
|
79
|
+
|
|
80
|
+
```typescript copy showLineNumbers
|
|
81
|
+
new VercelDeployer({
|
|
82
|
+
teamSlug: 'your-vercel-team-slug',
|
|
83
|
+
projectName: 'your-project-name',
|
|
84
|
+
token: 'your-vercel-token'
|
|
85
|
+
// For complete configuration options, see the reference documentation
|
|
86
|
+
})
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
[View Vercel Deployer Reference →](/reference/deployer/vercel)
|
|
90
|
+
|
|
91
|
+
#### Netlify Deployer
|
|
92
|
+
|
|
93
|
+
```typescript copy showLineNumbers
|
|
94
|
+
new NetlifyDeployer({
|
|
95
|
+
scope: 'your-netlify-team-slug',
|
|
96
|
+
projectName: 'your-project-name',
|
|
97
|
+
token: 'your-netlify-token'
|
|
98
|
+
})
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
[View Netlify Deployer Reference →](/reference/deployer/netlify)
|
|
102
|
+
|
|
103
|
+
## Environment Variables
|
|
104
|
+
|
|
105
|
+
Required variables:
|
|
106
|
+
|
|
107
|
+
1. Platform deployer variables (if using platform deployers):
|
|
108
|
+
- Platform credentials
|
|
109
|
+
2. Agent API keys:
|
|
110
|
+
- `OPENAI_API_KEY`
|
|
111
|
+
- `ANTHROPIC_API_KEY`
|
|
112
|
+
3. Server configuration (for universal deployment):
|
|
113
|
+
- `PORT`: HTTP server port (default: 3000)
|
|
114
|
+
- `HOST`: Server host (default: 0.0.0.0)
|
|
115
|
+
|
|
116
|
+
## Build Mastra Project
|
|
117
|
+
|
|
118
|
+
To build your Mastra project for your target platform run:
|
|
119
|
+
|
|
120
|
+
```bash
|
|
121
|
+
npx mastra build
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
When a Deployer is used, the build output is automatically prepared for the target platform.
|
|
125
|
+
You can then deploy the build output `.mastra/output` via your platform's (Vercel, netlify, cloudfare e.t.c)
|
|
126
|
+
CLI/UI.
|
|
127
|
+
|
|
@@ -0,0 +1,282 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Creating A Mastra Server"
|
|
3
|
+
description: "Configure and customize the Mastra server with middleware and other options"
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Creating A Mastra Server
|
|
7
|
+
|
|
8
|
+
While developing or when you deploy a Mastra application, it runs as an HTTP server that exposes your agents, workflows, and other functionality as API endpoints. This page explains how to configure and customize the server behavior.
|
|
9
|
+
|
|
10
|
+
## Server Architecture
|
|
11
|
+
|
|
12
|
+
Mastra uses [Hono](https://hono.dev) as its underlying HTTP server framework. When you build a Mastra application using `mastra build`, it generates a Hono-based HTTP server in the `.mastra` directory.
|
|
13
|
+
|
|
14
|
+
The server provides:
|
|
15
|
+
|
|
16
|
+
- API endpoints for all registered agents
|
|
17
|
+
- API endpoints for all registered workflows
|
|
18
|
+
- Custom api route supports
|
|
19
|
+
- Custom middleware support
|
|
20
|
+
- Configuration of timeout
|
|
21
|
+
- Configuration of port
|
|
22
|
+
|
|
23
|
+
## Server configuration
|
|
24
|
+
|
|
25
|
+
You can configure server `port` and `timeout` in the Mastra instance.
|
|
26
|
+
|
|
27
|
+
```typescript copy showLineNumbers
|
|
28
|
+
import { Mastra } from "@mastra/core";
|
|
29
|
+
|
|
30
|
+
export const mastra = new Mastra({
|
|
31
|
+
server: {
|
|
32
|
+
port: 3000, // Defaults to 4111
|
|
33
|
+
timeout: 10000, // Defaults to 30000 (30s)
|
|
34
|
+
},
|
|
35
|
+
});
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
## Custom API Routes
|
|
39
|
+
|
|
40
|
+
Mastra provides a list of api routes that are automatically generated based on the registered agents and workflows. You can also define custom api routes on the Mastra instance.
|
|
41
|
+
|
|
42
|
+
These routes can live in the same file as the Mastra instance or in a separate file. We recommend keeping them in a separate file to keep the Mastra instance clean.
|
|
43
|
+
|
|
44
|
+
```typescript copy showLineNumbers
|
|
45
|
+
import { Mastra } from "@mastra/core";
|
|
46
|
+
import { registerApiRoute } from "@mastra/core/server";
|
|
47
|
+
|
|
48
|
+
export const mastra = new Mastra({
|
|
49
|
+
server: {
|
|
50
|
+
apiRoutes: [
|
|
51
|
+
registerApiRoute("/my-custom-route", {
|
|
52
|
+
method: "GET",
|
|
53
|
+
handler: async (c) => {
|
|
54
|
+
// you have access to mastra instance here
|
|
55
|
+
const mastra = c.get("mastra");
|
|
56
|
+
|
|
57
|
+
// you can use the mastra instance to get agents, workflows, etc.
|
|
58
|
+
const agents = await mastra.getAgent("my-agent");
|
|
59
|
+
|
|
60
|
+
return c.json({ message: "Hello, world!" });
|
|
61
|
+
},
|
|
62
|
+
}),
|
|
63
|
+
],
|
|
64
|
+
},
|
|
65
|
+
// Other configuration options
|
|
66
|
+
});
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
## Custom CORS Config
|
|
70
|
+
|
|
71
|
+
Mastra allows you to configure CORS (Cross-Origin Resource Sharing) settings for your server.
|
|
72
|
+
|
|
73
|
+
```typescript copy showLineNumbers
|
|
74
|
+
import { Mastra } from '@mastra/core';
|
|
75
|
+
|
|
76
|
+
export const mastra = new Mastra({
|
|
77
|
+
server: {
|
|
78
|
+
cors: {
|
|
79
|
+
origin: ['https://example.com'], // Allow specific origins or '*' for all
|
|
80
|
+
allowMethods: ['GET', 'POST', 'PUT', 'DELETE', 'OPTIONS'],
|
|
81
|
+
allowHeaders: ['Content-Type', 'Authorization'],
|
|
82
|
+
credentials: false,
|
|
83
|
+
}
|
|
84
|
+
}
|
|
85
|
+
});
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
## Middleware
|
|
89
|
+
|
|
90
|
+
Mastra allows you to configure custom middleware functions that will be applied to API routes. This is useful for adding authentication, logging, CORS, or other HTTP-level functionality to your API endpoints.
|
|
91
|
+
|
|
92
|
+
```typescript copy showLineNumbers
|
|
93
|
+
import { Mastra } from '@mastra/core';
|
|
94
|
+
|
|
95
|
+
export const mastra = new Mastra({
|
|
96
|
+
// Other configuration options
|
|
97
|
+
server: {
|
|
98
|
+
middleware: [
|
|
99
|
+
{
|
|
100
|
+
handler: async (c, next) => {
|
|
101
|
+
// Example: Add authentication check
|
|
102
|
+
const authHeader = c.req.header('Authorization');
|
|
103
|
+
if (!authHeader) {
|
|
104
|
+
return new Response('Unauthorized', { status: 401 });
|
|
105
|
+
}
|
|
106
|
+
|
|
107
|
+
// Continue to the next middleware or route handler
|
|
108
|
+
await next();
|
|
109
|
+
},
|
|
110
|
+
path: '/api/*'
|
|
111
|
+
},
|
|
112
|
+
// add middleware to all routes
|
|
113
|
+
async (c, next) => {
|
|
114
|
+
// Example: Add request logging
|
|
115
|
+
console.log(`${c.req.method} ${c.req.url}`);
|
|
116
|
+
await next();
|
|
117
|
+
},
|
|
118
|
+
]
|
|
119
|
+
});
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
if you want to add a middleware to a single route, you can also specify that in the registerApiRoute using `registerApiRoute`.
|
|
123
|
+
|
|
124
|
+
```typescript copy showLineNumbers
|
|
125
|
+
registerApiRoute("/my-custom-route", {
|
|
126
|
+
method: "GET",
|
|
127
|
+
middleware: [
|
|
128
|
+
async (c, next) => {
|
|
129
|
+
// Example: Add request logging
|
|
130
|
+
console.log(`${c.req.method} ${c.req.url}`);
|
|
131
|
+
await next();
|
|
132
|
+
},
|
|
133
|
+
],
|
|
134
|
+
handler: async (c) => {
|
|
135
|
+
// you have access to mastra instance here
|
|
136
|
+
const mastra = c.get("mastra");
|
|
137
|
+
|
|
138
|
+
// you can use the mastra instance to get agents, workflows, etc.
|
|
139
|
+
const agents = await mastra.getAgent("my-agent");
|
|
140
|
+
|
|
141
|
+
return c.json({ message: "Hello, world!" });
|
|
142
|
+
},
|
|
143
|
+
});
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
### Middleware Behavior
|
|
147
|
+
|
|
148
|
+
Each middleware function:
|
|
149
|
+
|
|
150
|
+
- Receives a Hono context object (`c`) and a `next` function
|
|
151
|
+
- Can return a `Response` to short-circuit the request handling
|
|
152
|
+
- Can call `next()` to continue to the next middleware or route handler
|
|
153
|
+
- Can optionally specify a path pattern (defaults to '/api/\*')
|
|
154
|
+
- Inject request specific data for agent tool calling or workflows
|
|
155
|
+
|
|
156
|
+
### Common Middleware Use Cases
|
|
157
|
+
|
|
158
|
+
#### Authentication
|
|
159
|
+
|
|
160
|
+
```typescript copy
|
|
161
|
+
{
|
|
162
|
+
handler: async (c, next) => {
|
|
163
|
+
const authHeader = c.req.header('Authorization');
|
|
164
|
+
if (!authHeader || !authHeader.startsWith('Bearer ')) {
|
|
165
|
+
return new Response('Unauthorized', { status: 401 });
|
|
166
|
+
}
|
|
167
|
+
|
|
168
|
+
const token = authHeader.split(' ')[1];
|
|
169
|
+
// Validate token here
|
|
170
|
+
|
|
171
|
+
await next();
|
|
172
|
+
},
|
|
173
|
+
path: '/api/*',
|
|
174
|
+
}
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
#### CORS Support
|
|
178
|
+
|
|
179
|
+
```typescript copy
|
|
180
|
+
{
|
|
181
|
+
handler: async (c, next) => {
|
|
182
|
+
// Add CORS headers
|
|
183
|
+
c.header("Access-Control-Allow-Origin", "*");
|
|
184
|
+
c.header("Access-Control-Allow-Methods", "GET, POST, PUT, DELETE, OPTIONS");
|
|
185
|
+
c.header("Access-Control-Allow-Headers", "Content-Type, Authorization");
|
|
186
|
+
|
|
187
|
+
// Handle preflight requests
|
|
188
|
+
if (c.req.method === "OPTIONS") {
|
|
189
|
+
return new Response(null, { status: 204 });
|
|
190
|
+
}
|
|
191
|
+
|
|
192
|
+
await next();
|
|
193
|
+
};
|
|
194
|
+
}
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
#### Request Logging
|
|
198
|
+
|
|
199
|
+
```typescript copy
|
|
200
|
+
{
|
|
201
|
+
handler: async (c, next) => {
|
|
202
|
+
const start = Date.now();
|
|
203
|
+
await next();
|
|
204
|
+
const duration = Date.now() - start;
|
|
205
|
+
console.log(`${c.req.method} ${c.req.url} - ${duration}ms`);
|
|
206
|
+
};
|
|
207
|
+
}
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
### Special Mastra Headers
|
|
211
|
+
|
|
212
|
+
When integrating with Mastra Cloud or building custom clients, there are special headers that clients send to identify themselves and enable specific features. Your server middleware can check for these headers to customize behavior:
|
|
213
|
+
|
|
214
|
+
```typescript copy
|
|
215
|
+
{
|
|
216
|
+
handler: async (c, next) => {
|
|
217
|
+
// Check for Mastra-specific headers in incoming requests
|
|
218
|
+
const isFromMastraCloud = c.req.header("x-mastra-cloud") === "true";
|
|
219
|
+
const clientType = c.req.header("x-mastra-client-type"); // e.g., 'js', 'python'
|
|
220
|
+
const isDevPlayground = c.req.header("x-mastra-dev-playground") === "true";
|
|
221
|
+
|
|
222
|
+
// Customize behavior based on client information
|
|
223
|
+
if (isFromMastraCloud) {
|
|
224
|
+
// Special handling for Mastra Cloud requests
|
|
225
|
+
}
|
|
226
|
+
|
|
227
|
+
await next();
|
|
228
|
+
};
|
|
229
|
+
}
|
|
230
|
+
```
|
|
231
|
+
|
|
232
|
+
These headers have the following purposes:
|
|
233
|
+
|
|
234
|
+
- `x-mastra-cloud`: Indicates that the request is coming from Mastra Cloud
|
|
235
|
+
- `x-mastra-client-type`: Specifies the client SDK type (e.g., 'js', 'python')
|
|
236
|
+
- `x-mastra-dev-playground`: Indicates that the request is from the development playground
|
|
237
|
+
|
|
238
|
+
You can use these headers in your middleware to implement client-specific logic or enable features only for certain environments.
|
|
239
|
+
|
|
240
|
+
## Deployment
|
|
241
|
+
|
|
242
|
+
Since Mastra builds to a standard Node.js server, you can deploy to any platform that runs Node.js applications:
|
|
243
|
+
|
|
244
|
+
- Cloud VMs (AWS EC2, DigitalOcean Droplets, GCP Compute Engine)
|
|
245
|
+
- Container platforms (Docker, Kubernetes)
|
|
246
|
+
- Platform as a Service (Heroku, Railway)
|
|
247
|
+
- Self-hosted servers
|
|
248
|
+
|
|
249
|
+
### Building
|
|
250
|
+
|
|
251
|
+
Build the application:
|
|
252
|
+
|
|
253
|
+
```bash copy
|
|
254
|
+
# Build from current directory
|
|
255
|
+
mastra build
|
|
256
|
+
|
|
257
|
+
# Or specify a directory
|
|
258
|
+
mastra build --dir ./my-project
|
|
259
|
+
```
|
|
260
|
+
|
|
261
|
+
The build process:
|
|
262
|
+
|
|
263
|
+
1. Locates entry file (`src/mastra/index.ts` or `src/mastra/index.js`)
|
|
264
|
+
2. Creates `.mastra` output directory
|
|
265
|
+
3. Bundles code using Rollup with tree shaking and source maps
|
|
266
|
+
4. Generates [Hono](https://hono.dev) HTTP server
|
|
267
|
+
|
|
268
|
+
See [`mastra build`](/reference/cli/build) for all options.
|
|
269
|
+
|
|
270
|
+
### Running the Server
|
|
271
|
+
|
|
272
|
+
Start the HTTP server:
|
|
273
|
+
|
|
274
|
+
```bash copy
|
|
275
|
+
node .mastra/output/index.mjs
|
|
276
|
+
```
|
|
277
|
+
|
|
278
|
+
## Serverless Deployment
|
|
279
|
+
|
|
280
|
+
Mastra also supports serverless deployment on Cloudflare Workers, Vercel, and Netlify.
|
|
281
|
+
|
|
282
|
+
See our [Serverless Deployment](/docs/deployment/deployment) guide for setup instructions.
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Create your own Eval"
|
|
3
|
+
description: "Mastra allows so create your own evals, here is how."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Create your own Eval
|
|
7
|
+
|
|
8
|
+
Creating your own eval is as easy as creating a new function. You simply create a class that extends the `Metric` class and implement the `measure` method.
|
|
9
|
+
|
|
10
|
+
## Basic example
|
|
11
|
+
|
|
12
|
+
For a simple example of creating a custom metric that checks if the output contains certain words, see our [Word Inclusion example](/examples/evals/word-inclusion).
|
|
13
|
+
|
|
14
|
+
## Creating a custom LLM-Judge
|
|
15
|
+
|
|
16
|
+
A custom LLM judge helps evaluate specific aspects of your AI's responses. Think of it like having an expert reviewer for your particular use case:
|
|
17
|
+
|
|
18
|
+
- Medical Q&A → Judge checks for medical accuracy and safety
|
|
19
|
+
- Customer Service → Judge evaluates tone and helpfulness
|
|
20
|
+
- Code Generation → Judge verifies code correctness and style
|
|
21
|
+
|
|
22
|
+
For a practical example, see how we evaluate [Chef Michel's](/docs/guides/chef-michel) recipes for gluten content in our [Gluten Checker example](/examples/evals/custom-eval).
|
|
@@ -0,0 +1,95 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "Overview"
|
|
3
|
+
description: "Understanding how to evaluate and measure AI agent quality using Mastra evals."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Testing your agents with evals
|
|
7
|
+
|
|
8
|
+
While traditional software tests have clear pass/fail conditions, AI outputs are non-deterministic — they can vary with the same input. Evals help bridge this gap by providing quantifiable metrics for measuring agent quality.
|
|
9
|
+
|
|
10
|
+
Evals are automated tests that evaluate Agents outputs using model-graded, rule-based, and statistical methods. Each eval returns a normalized score between 0-1 that can be logged and compared. Evals can be customized with your own prompts and scoring functions.
|
|
11
|
+
|
|
12
|
+
Evals can be run in the cloud, capturing real-time results. But evals can also be part of your CI/CD pipeline, allowing you to test and monitor your agents over time.
|
|
13
|
+
|
|
14
|
+
## Types of Evals
|
|
15
|
+
|
|
16
|
+
There are different kinds of evals, each serving a specific purpose. Here are some common types:
|
|
17
|
+
|
|
18
|
+
1. **Textual Evals**: Evaluate accuracy, reliability, and context understanding of agent responses
|
|
19
|
+
2. **Classification Evals**: Measure accuracy in categorizing data based on predefined categories
|
|
20
|
+
3. **Tool Usage Evals**: Assess how effectively an agent uses external tools or APIs
|
|
21
|
+
4. **Prompt Engineering Evals**: Explore impact of different instructions and input formats
|
|
22
|
+
|
|
23
|
+
## Getting Started
|
|
24
|
+
|
|
25
|
+
Evals need to be added to an agent. Here's an example using the summarization, content similarity, and tone consistency metrics:
|
|
26
|
+
|
|
27
|
+
```typescript copy showLineNumbers filename="src/mastra/agents/index.ts"
|
|
28
|
+
import { Agent } from "@mastra/core/agent";
|
|
29
|
+
import { openai } from "@ai-sdk/openai";
|
|
30
|
+
import { SummarizationMetric } from "@mastra/evals/llm";
|
|
31
|
+
import {
|
|
32
|
+
ContentSimilarityMetric,
|
|
33
|
+
ToneConsistencyMetric,
|
|
34
|
+
} from "@mastra/evals/nlp";
|
|
35
|
+
|
|
36
|
+
const model = openai("gpt-4o");
|
|
37
|
+
|
|
38
|
+
export const myAgent = new Agent({
|
|
39
|
+
name: "ContentWriter",
|
|
40
|
+
instructions: "You are a content writer that creates accurate summaries",
|
|
41
|
+
model,
|
|
42
|
+
evals: {
|
|
43
|
+
summarization: new SummarizationMetric(model),
|
|
44
|
+
contentSimilarity: new ContentSimilarityMetric(),
|
|
45
|
+
tone: new ToneConsistencyMetric(),
|
|
46
|
+
},
|
|
47
|
+
});
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
You can view eval results in the Mastra dashboard when using `mastra dev`.
|
|
51
|
+
|
|
52
|
+
## Beyond Automated Testing
|
|
53
|
+
|
|
54
|
+
While automated evals are valuable, high-performing AI teams often combine them with:
|
|
55
|
+
|
|
56
|
+
1. **A/B Testing**: Compare different versions with real users
|
|
57
|
+
2. **Human Review**: Regular review of production data and traces
|
|
58
|
+
3. **Continuous Monitoring**: Track eval metrics over time to detect regressions
|
|
59
|
+
|
|
60
|
+
## Understanding Eval Results
|
|
61
|
+
|
|
62
|
+
Each eval metric measures a specific aspect of your agent's output. Here's how to interpret and improve your results:
|
|
63
|
+
|
|
64
|
+
### Understanding Scores
|
|
65
|
+
|
|
66
|
+
For any metric:
|
|
67
|
+
|
|
68
|
+
1. Check the metric documentation to understand the scoring process
|
|
69
|
+
2. Look for patterns in when scores change
|
|
70
|
+
3. Compare scores across different inputs and contexts
|
|
71
|
+
4. Track changes over time to spot trends
|
|
72
|
+
|
|
73
|
+
### Improving Results
|
|
74
|
+
|
|
75
|
+
When scores aren't meeting your targets:
|
|
76
|
+
|
|
77
|
+
1. Check your instructions - Are they clear? Try making them more specific
|
|
78
|
+
2. Look at your context - Is it giving the agent what it needs?
|
|
79
|
+
3. Simplify your prompts - Break complex tasks into smaller steps
|
|
80
|
+
4. Add guardrails - Include specific rules for tricky cases
|
|
81
|
+
|
|
82
|
+
### Maintaining Quality
|
|
83
|
+
|
|
84
|
+
Once you're hitting your targets:
|
|
85
|
+
|
|
86
|
+
1. Monitor stability - Do scores remain consistent?
|
|
87
|
+
2. Document what works - Keep notes on successful approaches
|
|
88
|
+
3. Test edge cases - Add examples that cover unusual scenarios
|
|
89
|
+
4. Fine-tune - Look for ways to improve efficiency
|
|
90
|
+
|
|
91
|
+
See [Textual Evals](/docs/evals/textual-evals) for more info on what evals can do.
|
|
92
|
+
|
|
93
|
+
For more info on how to create your own evals, see the [Custom Evals](/docs/evals/custom-eval) guide.
|
|
94
|
+
|
|
95
|
+
For running evals in your CI pipeline, see the [Running in CI](/docs/evals/running-in-ci) guide.
|