@mastra/mcp-docs-server 1.2.13-alpha.0 → 1.2.13-alpha.11
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.docs/docs/agents/agent-approval.md +72 -0
- package/.docs/docs/agents/overview.md +1 -1
- package/.docs/docs/agents/supervisor-agents.md +2 -2
- package/.docs/docs/agents/using-tools.md +61 -33
- package/.docs/docs/{evals/datasets → datasets}/overview.md +3 -3
- package/.docs/docs/{evals/datasets → datasets}/running-experiments.md +57 -9
- package/.docs/docs/deployment/workers.md +14 -14
- package/.docs/docs/editor/overview.md +201 -206
- package/.docs/docs/evals/evals-with-memory.md +1 -1
- package/.docs/docs/getting-started/develop.md +196 -0
- package/.docs/docs/index.md +78 -1
- package/.docs/docs/long-running-agents/durable-agents.md +2 -2
- package/.docs/docs/mastra-platform/overview.md +1 -1
- package/.docs/docs/mastra-platform/workspaces.md +152 -0
- package/.docs/docs/memory/observational-memory.md +53 -13
- package/.docs/docs/memory/overview.md +14 -0
- package/.docs/docs/server/auth/workers.md +7 -5
- package/.docs/docs/server/mastra-client.md +60 -0
- package/.docs/docs/server/pubsub.md +2 -2
- package/.docs/docs/studio/observability.md +4 -0
- package/.docs/docs/studio/overview.md +2 -2
- package/.docs/docs/workflows/overview.md +1 -1
- package/.docs/docs/workflows/scheduled-workflows.md +1 -0
- package/.docs/docs/workspace/search.md +1 -1
- package/.docs/guides/deployment/kubernetes.md +2 -0
- package/.docs/guides/deployment/mastra-workers.md +350 -6
- package/.docs/guides/deployment/vercel.md +2 -0
- package/.docs/guides/getting-started/quickstart.md +3 -3
- package/.docs/{docs → guides}/rag/chunking-and-embedding.md +1 -1
- package/.docs/{docs → guides}/rag/graph-rag.md +1 -1
- package/.docs/{docs → guides}/rag/overview.md +2 -2
- package/.docs/{docs → guides}/rag/retrieval.md +1 -1
- package/.docs/{docs → guides}/voice/overview.md +4 -4
- package/.docs/{docs → guides}/voice/realtime-voice.md +2 -2
- package/.docs/{docs → guides}/voice/speech-to-speech.md +1 -1
- package/.docs/{docs → guides}/voice/speech-to-text.md +1 -1
- package/.docs/{docs → guides}/voice/text-to-speech.md +2 -2
- package/.docs/models/embeddings.md +63 -65
- package/.docs/models/environment-variables.md +2 -1
- package/.docs/models/gateways/openrouter.md +3 -6
- package/.docs/models/gateways/vercel.md +4 -1
- package/.docs/models/index.md +1 -1
- package/.docs/models/providers/cortecs.md +2 -1
- package/.docs/models/providers/crof.md +2 -1
- package/.docs/models/providers/crossmodel.md +4 -3
- package/.docs/models/providers/empiriolabs.md +2 -1
- package/.docs/models/providers/greenpt.md +98 -0
- package/.docs/models/providers/huggingface.md +3 -1
- package/.docs/models/providers/hyper.md +10 -7
- package/.docs/models/providers/llmgateway.md +5 -4
- package/.docs/models/providers/minimax.md +1 -1
- package/.docs/models/providers/neuralwatt.md +3 -1
- package/.docs/models/providers/openai.md +2 -2
- package/.docs/models/providers/opencode-go.md +2 -1
- package/.docs/models/providers/opencode.md +1 -1
- package/.docs/models/providers/perplexity-agent.md +3 -1
- package/.docs/models/providers/scaleway.md +1 -3
- package/.docs/models/providers/the-grid-ai.md +3 -3
- package/.docs/models/providers/zai-coding-plan.md +11 -13
- package/.docs/models/providers/zhipuai-coding-plan.md +12 -12
- package/.docs/models/providers.md +1 -0
- package/.docs/reference/agents/agent.md +1 -1
- package/.docs/reference/agents/durable-agent.md +12 -1
- package/.docs/reference/agents/generate.md +1 -1
- package/.docs/reference/agents/getVoice.md +1 -1
- package/.docs/{docs/getting-started → reference}/build-with-ai.md +5 -110
- package/.docs/reference/cli/create-mastra.md +2 -0
- package/.docs/reference/cli/mastra.md +36 -16
- package/.docs/reference/client-js/agents.md +1 -1
- package/.docs/reference/configuration.md +2 -2
- package/.docs/reference/core/getEditor.md +1 -1
- package/.docs/reference/core/getVector.md +2 -2
- package/.docs/reference/core/listVectors.md +2 -2
- package/.docs/reference/core/mastra-class.md +2 -2
- package/.docs/reference/datasets/startExperiment.md +8 -0
- package/.docs/reference/datasets/startExperimentAsync.md +4 -0
- package/.docs/reference/editor/browser-provider.md +2 -2
- package/.docs/reference/editor/filesystem-provider.md +1 -1
- package/.docs/reference/editor/mastra-editor.md +31 -18
- package/.docs/reference/editor/prompt-blocks.md +137 -0
- package/.docs/reference/editor/sandbox-provider.md +1 -1
- package/.docs/reference/editor/storage-browser-ref.md +3 -3
- package/.docs/reference/editor/storage-workspace-ref.md +3 -3
- package/.docs/reference/editor/tool-provider.md +44 -10
- package/.docs/reference/editor/tools.md +100 -0
- package/.docs/reference/editor/versioning.md +130 -0
- package/.docs/reference/evals/mastra-scorer.md +68 -8
- package/.docs/reference/evals/summarization.md +203 -0
- package/.docs/reference/file-based-agents/config.md +39 -1
- package/.docs/reference/index.md +4 -4
- package/.docs/reference/memory/observational-memory.md +76 -24
- package/.docs/reference/memory/serialized-memory-config.md +3 -3
- package/.docs/reference/observability/tracing/interfaces.md +6 -1
- package/.docs/reference/processors/processor-interface.md +14 -2
- package/.docs/reference/processors/regex-filter-processor.md +81 -9
- package/.docs/reference/processors/token-limiter-processor.md +15 -1
- package/.docs/reference/project-structure.md +1 -1
- package/.docs/reference/rag/database-config.md +1 -1
- package/.docs/reference/storage/composite.md +2 -0
- package/.docs/reference/streaming/agents/stream.md +1 -1
- package/.docs/reference/tools/isolated-vm-transport.md +1 -1
- package/.docs/reference/vectors/mongodb.md +13 -13
- package/.docs/reference/voice/livekit.md +3 -3
- package/.docs/reference/workers/overview.md +10 -8
- package/.docs/reference/workspace/platform-filesystem.md +8 -5
- package/.docs/reference/workspace/platform-sandbox.md +83 -7
- package/CHANGELOG.md +51 -0
- package/README.md +1 -1
- package/package.json +5 -5
- package/.docs/docs/agent-builder/access-control.md +0 -99
- package/.docs/docs/agent-builder/browser.md +0 -63
- package/.docs/docs/agent-builder/channels.md +0 -78
- package/.docs/docs/agent-builder/configuration.md +0 -132
- package/.docs/docs/agent-builder/deploying.md +0 -124
- package/.docs/docs/agent-builder/integrations.md +0 -115
- package/.docs/docs/agent-builder/memory.md +0 -69
- package/.docs/docs/agent-builder/model-policy.md +0 -50
- package/.docs/docs/agent-builder/overview.md +0 -114
- package/.docs/docs/agent-builder/skill-registries.md +0 -31
- package/.docs/docs/agent-builder/workspace.md +0 -62
- package/.docs/docs/editor/prompts.md +0 -140
- package/.docs/docs/editor/tools.md +0 -148
- package/.docs/docs/getting-started/file-based-agents.md +0 -121
- package/.docs/docs/mastra-platform/workspace.md +0 -111
- package/.docs/docs/what-is-mastra.md +0 -161
- package/.docs/reference/client-js/agent-builder.md +0 -163
- package/.docs/reference/editor/agent-builder/agent-builder-options.md +0 -76
- package/.docs/reference/editor/agent-builder/builder-agent-defaults.md +0 -79
- package/.docs/reference/editor/agent-builder/builder-models.md +0 -66
- /package/.docs/{docs → guides}/getting-started/manual-install.md +0 -0
- /package/.docs/{docs → guides}/rag/vector-databases.md +0 -0
|
@@ -104,6 +104,78 @@ A tool's own `requireApproval` setting takes precedence over the function above.
|
|
|
104
104
|
|
|
105
105
|
> **Note:** Function-based `requireToolApproval` is only available on regular `stream()` / `generate()` calls. Durable agents and stored agents persist their options, and a function can't be serialized, so they accept only a boolean. If you pass a function in those contexts it falls back to requiring approval for every tool call.
|
|
106
106
|
|
|
107
|
+
#### Bind approval to the exact tool arguments
|
|
108
|
+
|
|
109
|
+
For sensitive tools, bind the approval to the exact tool name and arguments that were shown to the reviewer. If those arguments drift before execution, the tool shouldn't run under the old approval.
|
|
110
|
+
|
|
111
|
+
The `tool-call-approval` chunk already includes `toolName`, `toolCallId`, and `args`. You can fingerprint those fields when the approval request is shown. The example below uses a JSON string as the fingerprint, but in production you should use a stable hash of the tool name and arguments:
|
|
112
|
+
|
|
113
|
+
```typescript
|
|
114
|
+
import { Agent } from '@mastra/core/agent'
|
|
115
|
+
|
|
116
|
+
// For your production usecase, build a stable hash of the tool name and args
|
|
117
|
+
function actionFingerprint(toolName: string, args: unknown) {
|
|
118
|
+
const payload = JSON.stringify({ toolName, args })
|
|
119
|
+
return `fingerprint-${payload}`
|
|
120
|
+
}
|
|
121
|
+
|
|
122
|
+
const sensitiveTools = new Set(['issue_refund', 'delete_record'])
|
|
123
|
+
const approvedFingerprints = new Set<string>()
|
|
124
|
+
|
|
125
|
+
export const approvalBoundAgent = new Agent({
|
|
126
|
+
id: 'approval-bound-agent',
|
|
127
|
+
name: 'Approval Bound Agent',
|
|
128
|
+
model: 'openai/gpt-5.6-sol',
|
|
129
|
+
tools: { issueRefundTool, deleteRecordTool },
|
|
130
|
+
hooks: {
|
|
131
|
+
beforeToolCall: ({ toolName, input }) => {
|
|
132
|
+
if (!sensitiveTools.has(toolName)) return
|
|
133
|
+
|
|
134
|
+
const fingerprint = actionFingerprint(toolName, input)
|
|
135
|
+
if (!approvedFingerprints.delete(fingerprint)) {
|
|
136
|
+
return {
|
|
137
|
+
proceed: false,
|
|
138
|
+
output: `Tool call blocked: approval did not match ${toolName} arguments.`,
|
|
139
|
+
}
|
|
140
|
+
}
|
|
141
|
+
},
|
|
142
|
+
},
|
|
143
|
+
})
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
```typescript
|
|
147
|
+
const stream = await approvalBoundAgent.stream('Refund order ord-1042', {
|
|
148
|
+
requireToolApproval: ({ toolName }) => sensitiveTools.has(toolName),
|
|
149
|
+
})
|
|
150
|
+
|
|
151
|
+
async function consumeApprovalStream(currentStream: typeof stream) {
|
|
152
|
+
for await (const chunk of currentStream.fullStream) {
|
|
153
|
+
if (chunk.type === 'tool-call-approval') {
|
|
154
|
+
const { toolName, toolCallId, args } = chunk.payload
|
|
155
|
+
const fingerprint = actionFingerprint(toolName, args)
|
|
156
|
+
|
|
157
|
+
// Present toolName, args, and fingerprint to your approval UI.
|
|
158
|
+
const approved = await showApprovalDialog({ toolName, args, fingerprint })
|
|
159
|
+
|
|
160
|
+
const nextStream = approved
|
|
161
|
+
? await approveReviewedToolCall(currentStream.runId, toolCallId, fingerprint)
|
|
162
|
+
: await approvalBoundAgent.declineToolCall({ runId: currentStream.runId, toolCallId })
|
|
163
|
+
|
|
164
|
+
await consumeApprovalStream(nextStream)
|
|
165
|
+
}
|
|
166
|
+
}
|
|
167
|
+
}
|
|
168
|
+
|
|
169
|
+
async function approveReviewedToolCall(runId: string, toolCallId: string, fingerprint: string) {
|
|
170
|
+
approvedFingerprints.add(fingerprint)
|
|
171
|
+
return approvalBoundAgent.approveToolCall({ runId, toolCallId })
|
|
172
|
+
}
|
|
173
|
+
|
|
174
|
+
await consumeApprovalStream(stream)
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
In production, store the approved fingerprint in durable storage scoped to the user, run, tool call, and policy version. The `Set` above is intentionally small so the boundary is clear: the approval is consumed once, and only for the same canonical tool arguments that were reviewed.
|
|
178
|
+
|
|
107
179
|
### Runtime suspension with `suspend()`
|
|
108
180
|
|
|
109
181
|
A tool can also pause _during_ its `execute` function by calling `suspend()`. This is useful when the tool starts running and then discovers it needs additional user input or confirmation before it can finish.
|
|
@@ -210,7 +210,7 @@ Once your agent is running, use this table to find the right page for what you w
|
|
|
210
210
|
| Keep your agent safe | [Guardrails](https://mastra.ai/docs/agents/guardrails) |
|
|
211
211
|
| Build agents that correct their work | [Rubric scorer](https://mastra.ai/docs/agents/supervisor-agents) |
|
|
212
212
|
| Swap instructions or models based on request context | [Dynamic configuration](https://mastra.ai/docs/server/request-context) |
|
|
213
|
-
| Add speech-to-text or text-to-speech | [Voice](https://mastra.ai/
|
|
213
|
+
| Add speech-to-text or text-to-speech | [Voice](https://mastra.ai/guides/voice/overview) |
|
|
214
214
|
| Connect to Slack, Discord, or Telegram | [Channels](https://mastra.ai/docs/capabilities/channels/overview) |
|
|
215
215
|
|
|
216
216
|
## Multi-agent systems
|
|
@@ -420,12 +420,12 @@ const result = await supervisor.generate('Research and write about AI safety', {
|
|
|
420
420
|
})
|
|
421
421
|
```
|
|
422
422
|
|
|
423
|
-
Version overrides propagate automatically through delegation. See [Subagent versioning](https://mastra.ai/
|
|
423
|
+
Version overrides propagate automatically through delegation. See [Subagent versioning](https://mastra.ai/reference/editor/versioning) for details on resolution order and server API usage.
|
|
424
424
|
|
|
425
425
|
## Related
|
|
426
426
|
|
|
427
427
|
- [Background tasks](https://mastra.ai/docs/long-running-agents/background-tasks)
|
|
428
|
-
- [Subagent versioning](https://mastra.ai/
|
|
428
|
+
- [Subagent versioning](https://mastra.ai/reference/editor/versioning)
|
|
429
429
|
- [Guide: Research coordinator](https://mastra.ai/guides/guide/research-coordinator)
|
|
430
430
|
- [Agent.stream() reference](https://mastra.ai/reference/streaming/agents/stream)
|
|
431
431
|
- [Agent.streamUntilIdle() reference](https://mastra.ai/reference/streaming/agents/streamUntilIdle)
|
|
@@ -65,31 +65,6 @@ export const weatherAgent = new Agent({
|
|
|
65
65
|
})
|
|
66
66
|
```
|
|
67
67
|
|
|
68
|
-
## Use provider web search
|
|
69
|
-
|
|
70
|
-
Import `webSearchTool` from `@mastra/core/tools` when you want the model provider to run its native web search tool. The tool has no local `execute` function. Mastra resolves it at run time from the active model, then passes the provider-managed tool to the model.
|
|
71
|
-
|
|
72
|
-
```typescript
|
|
73
|
-
import { Agent } from '@mastra/core/agent'
|
|
74
|
-
import { webSearchTool } from '@mastra/core/tools'
|
|
75
|
-
|
|
76
|
-
export const researchAgent = new Agent({
|
|
77
|
-
id: 'research-agent',
|
|
78
|
-
name: 'Research Agent',
|
|
79
|
-
instructions: `
|
|
80
|
-
You are a helpful research assistant.
|
|
81
|
-
Use web search when you need current information.`,
|
|
82
|
-
model: 'openai/gpt-5.6-sol',
|
|
83
|
-
tools: {
|
|
84
|
-
webSearch: webSearchTool,
|
|
85
|
-
},
|
|
86
|
-
})
|
|
87
|
-
```
|
|
88
|
-
|
|
89
|
-
`webSearchTool` supports OpenAI, Anthropic, Google Gemini, and xAI models. If Mastra can't infer one of those providers from the active model, the agent run fails with a `MastraError`.
|
|
90
|
-
|
|
91
|
-
Only the `webSearchTool` value triggers provider web search. Custom tools with the names `webSearch` or `web_search` stay unchanged.
|
|
92
|
-
|
|
93
68
|
## Define schemas
|
|
94
69
|
|
|
95
70
|
You can define the tool's `inputSchema` and `outputSchema` with any library that supports [Standard JSON Schema](https://standardschema.dev/json-schema). This includes libraries like [Zod](https://zod.dev/), [Valibot](https://valibot.dev/), and [ArkType](https://arktype.io/).
|
|
@@ -494,14 +469,67 @@ Note that for subagents, you'll see two different identifiers in stream response
|
|
|
494
469
|
|
|
495
470
|
Mastra includes agent-agnostic built-in tools in `@mastra/core/tools` that add interactive and organizational capabilities to any agent.
|
|
496
471
|
|
|
497
|
-
| Tool | Purpose
|
|
498
|
-
| --------------- |
|
|
499
|
-
| `ask_user` | Ask the user a question and wait for their answer
|
|
500
|
-
| `submit_plan` | Submit a plan file for user approval
|
|
501
|
-
| `task_write` | Create or replace a structured task list
|
|
502
|
-
| `task_update` | Update one tracked task by ID
|
|
503
|
-
| `task_complete` | Mark one tracked task completed
|
|
504
|
-
| `task_check` | Check task list completion status
|
|
472
|
+
| Tool | Purpose |
|
|
473
|
+
| --------------- | ---------------------------------------------------- |
|
|
474
|
+
| `ask_user` | Ask the user a question and wait for their answer |
|
|
475
|
+
| `submit_plan` | Submit a plan file for user approval |
|
|
476
|
+
| `task_write` | Create or replace a structured task list |
|
|
477
|
+
| `task_update` | Update one tracked task by ID |
|
|
478
|
+
| `task_complete` | Mark one tracked task completed |
|
|
479
|
+
| `task_check` | Check task list completion status |
|
|
480
|
+
| `webSearchTool` | Run provider-native web search with the active model |
|
|
481
|
+
| `webFetchTool` | Fetch a web page by URL and return its text content |
|
|
482
|
+
|
|
483
|
+
### Use provider web search
|
|
484
|
+
|
|
485
|
+
Import `webSearchTool` from `@mastra/core/tools` when you want the model provider to run its native web search tool. Mastra resolves it at run time from the active model, then passes the provider-managed tool to the model.
|
|
486
|
+
|
|
487
|
+
```typescript
|
|
488
|
+
import { Agent } from '@mastra/core/agent'
|
|
489
|
+
import { webSearchTool } from '@mastra/core/tools'
|
|
490
|
+
|
|
491
|
+
export const researchAgent = new Agent({
|
|
492
|
+
id: 'research-agent',
|
|
493
|
+
name: 'Research Agent',
|
|
494
|
+
instructions: 'Use web search when you need current information.',
|
|
495
|
+
model: 'openai/gpt-5.6-sol',
|
|
496
|
+
tools: {
|
|
497
|
+
search: webSearchTool,
|
|
498
|
+
},
|
|
499
|
+
})
|
|
500
|
+
```
|
|
501
|
+
|
|
502
|
+
`webSearchTool` supports OpenAI, Anthropic, Google Gemini, and xAI models. If Mastra can't infer one of those providers from the active model, the agent run fails with a `MastraError`.
|
|
503
|
+
|
|
504
|
+
The `search` key is only the agent-local tool name. Use any key. The `webSearchTool` value tells Mastra to use provider web search.
|
|
505
|
+
|
|
506
|
+
### Fetch a web page
|
|
507
|
+
|
|
508
|
+
Import `webFetchTool` from `@mastra/core/tools` when the agent needs to read a specific URL. The tool requests the page over HTTP or HTTPS and returns its text content plus response metadata.
|
|
509
|
+
|
|
510
|
+
```typescript
|
|
511
|
+
import { Agent } from '@mastra/core/agent'
|
|
512
|
+
import { webFetchTool } from '@mastra/core/tools'
|
|
513
|
+
|
|
514
|
+
export const readerAgent = new Agent({
|
|
515
|
+
id: 'reader-agent',
|
|
516
|
+
name: 'Reader Agent',
|
|
517
|
+
instructions: 'Fetch the page the user links to before answering.',
|
|
518
|
+
model: 'openai/gpt-5.6-sol',
|
|
519
|
+
tools: {
|
|
520
|
+
fetch: webFetchTool,
|
|
521
|
+
},
|
|
522
|
+
})
|
|
523
|
+
```
|
|
524
|
+
|
|
525
|
+
The tool takes a single `url` input and returns `content`, `truncated`, `status`, `statusText`, `contentType`, `url`, and `ok`. It applies these limits:
|
|
526
|
+
|
|
527
|
+
- Only `http:` and `https:` URLs are allowed.
|
|
528
|
+
- Requests to `localhost` and to private or reserved IP addresses are blocked, including addresses returned by DNS resolution.
|
|
529
|
+
- Responses are truncated at 100,000 characters, with `truncated: true` in the result.
|
|
530
|
+
- Requests follow at most 5 redirects and time out after 15 seconds.
|
|
531
|
+
|
|
532
|
+
Failures don't throw. The tool returns `isError: true` with the reason in `content`, so the agent can retry or explain the problem.
|
|
505
533
|
|
|
506
534
|
### Ask the user a question
|
|
507
535
|
|
|
@@ -49,7 +49,7 @@ Visit the [`DatasetsManager` reference](https://mastra.ai/reference/datasets/dat
|
|
|
49
49
|
|
|
50
50
|
You can also manage datasets in [Studio](https://mastra.ai/docs/studio/overview). After opening Studio, select **Datasets** from the sidebar to see all your available datasets or create a new one.
|
|
51
51
|
|
|
52
|
-
To get started, select **Create Dataset** and set a name, description, and optional schemas. After confirming, you'll see the dataset details page with two tabs: **Items** and [**Experiments**](https://mastra.ai/docs/
|
|
52
|
+
To get started, select **Create Dataset** and set a name, description, and optional schemas. After confirming, you'll see the dataset details page with two tabs: **Items** and [**Experiments**](https://mastra.ai/docs/datasets/running-experiments).
|
|
53
53
|
|
|
54
54
|
In the **Items** view you can add, update, and delete items, and view version history. Select **Add Item** to insert a new item with JSON editors for input and ground truth. From this view you can also import items in bulk from a CSV or JSON file. When importing, map each column to the corresponding dataset field.
|
|
55
55
|
|
|
@@ -198,11 +198,11 @@ Fetch the exact items that existed at a past version:
|
|
|
198
198
|
const items = await dataset.listItems({ version: 2 })
|
|
199
199
|
```
|
|
200
200
|
|
|
201
|
-
You can also pin experiments to a version, see [running experiments](https://mastra.ai/docs/
|
|
201
|
+
You can also pin experiments to a version, see [running experiments](https://mastra.ai/docs/datasets/running-experiments). Visit the [`Dataset` reference](https://mastra.ai/reference/datasets/dataset) for the full list of methods and parameters.
|
|
202
202
|
|
|
203
203
|
## Related
|
|
204
204
|
|
|
205
|
-
- [Running experiments](https://mastra.ai/docs/
|
|
205
|
+
- [Running experiments](https://mastra.ai/docs/datasets/running-experiments)
|
|
206
206
|
- [Scorers overview](https://mastra.ai/docs/evals/overview)
|
|
207
207
|
- [DatasetsManager reference](https://mastra.ai/reference/datasets/datasets-manager)
|
|
208
208
|
- [Dataset reference](https://mastra.ai/reference/datasets/dataset)
|
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
|
|
5
5
|
**Added in:** `@mastra/core@1.4.0`
|
|
6
6
|
|
|
7
|
-
An experiment runs every item in a dataset through a target (an agent, a workflow, or a scorer) and then optionally scores the outputs. Use a scorer as the target when you want to evaluate an LLM judge itself.
|
|
7
|
+
An experiment runs every item in a dataset through a target (an agent, a workflow, or a scorer) and then optionally scores the outputs. Use a scorer as the target when you want to evaluate an LLM judge itself. By default, results are persisted to storage so you can compare runs across different prompts, models, or code changes.
|
|
8
8
|
|
|
9
9
|
**For AI agents:** Run `npx mastra api experiment run dataset_123 '{"name":"translation-baseline"}'` to start an experiment directly instead of opening Studio or writing a temporary script. Use a dataset ID returned by `npx mastra api dataset list` in place of the sample ID. The command requires a running Mastra server with dataset storage and registered experiment targets; start the local server with `npx mastra dev`, or pass the reachable server's base URL with `--url`. Run `npx mastra api experiment run --schema` before constructing different input, and get user approval before starting an experiment because it can make model calls. Install Mastra's skill with `npx skills add mastra-ai/skills --skill mastra` for complete API CLI discovery, targeting, schema, authentication, and error-handling guidance.
|
|
10
10
|
|
|
@@ -131,9 +131,33 @@ for (const item of summary.results) {
|
|
|
131
131
|
|
|
132
132
|
Visit the [Scorers overview](https://mastra.ai/docs/evals/overview) for details on available and custom scorers.
|
|
133
133
|
|
|
134
|
+
## Control persistence per run
|
|
135
|
+
|
|
136
|
+
Use `persistence` to skip storage writes for a specific run. Experiment records and score records can be disabled independently:
|
|
137
|
+
|
|
138
|
+
```typescript
|
|
139
|
+
const summary = await dataset.startExperiment({
|
|
140
|
+
targetType: 'agent',
|
|
141
|
+
targetId: 'translation-agent',
|
|
142
|
+
scorers: ['accuracy'],
|
|
143
|
+
persistence: {
|
|
144
|
+
experiments: 'none',
|
|
145
|
+
scores: 'none',
|
|
146
|
+
},
|
|
147
|
+
})
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
The target and scorers still run, and `startExperiment()` still returns the item results and scores in `summary`. The settings are independent. For example, set only `scores: 'none'` to persist the experiment and its item results without creating score records.
|
|
151
|
+
|
|
152
|
+
Omitted settings default to `'default'`, which preserves the standard storage behavior. This policy only controls experiment and score records created by the run. It doesn't disable storage used by the target, such as agent memory, vectors, observability, or custom tool storage.
|
|
153
|
+
|
|
154
|
+
When `startExperimentAsync()` runs with `experiments: 'none'`, it doesn't persist an experiment record, progress updates, or item results. Score persistence remains controlled separately by `persistence.scores`. Without an experiment event observer, the run is fire-and-forget, and the experiment API can't report whether it completed or failed.
|
|
155
|
+
|
|
156
|
+
Use synchronous `startExperiment()` when the caller needs the returned summary. An experiment event observer can receive lifecycle events and the terminal summary.
|
|
157
|
+
|
|
134
158
|
## Tool mocks
|
|
135
159
|
|
|
136
|
-
When an experiment runs an agent that calls side-effecting tools,
|
|
160
|
+
When an experiment runs an agent that calls side-effecting tools, attach static tool mocks to individual dataset items to make the run deterministic. During the experiment, a mocked tool returns its declared output instead of executing. Tools without a mock on the item run live by default.
|
|
137
161
|
|
|
138
162
|
Mocks live on the dataset item, so they version with the row and travel with the test case. Each mock declares a tool name, the arguments it expects, and the output to return:
|
|
139
163
|
|
|
@@ -152,6 +176,29 @@ await dataset.addItem({
|
|
|
152
176
|
|
|
153
177
|
Tool mocks are supported for `agent` targets only.
|
|
154
178
|
|
|
179
|
+
### Block undeclared tools
|
|
180
|
+
|
|
181
|
+
Set `unmockedToolPolicy: 'deny'` on an experiment to block every tool call that doesn't have a mock. This is useful when a live call could cause side effects:
|
|
182
|
+
|
|
183
|
+
```typescript
|
|
184
|
+
const summary = await dataset.startExperiment({
|
|
185
|
+
targetType: 'agent',
|
|
186
|
+
targetId: 'weather-agent',
|
|
187
|
+
unmockedToolPolicy: 'deny',
|
|
188
|
+
})
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
The default policy is `'allow'`. You can override the experiment policy on an individual stored or inline item:
|
|
192
|
+
|
|
193
|
+
```typescript
|
|
194
|
+
await dataset.addItem({
|
|
195
|
+
input: 'What is the weather in Seattle?',
|
|
196
|
+
unmockedToolPolicy: 'allow',
|
|
197
|
+
})
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
The item value takes precedence over the experiment value. A denied call fails with `TOOL_MOCK_NOT_DECLARED` before the tool executes. The failure isn't retried or added to `liveCalls`.
|
|
201
|
+
|
|
155
202
|
### Matching and consumption
|
|
156
203
|
|
|
157
204
|
Arguments are matched strictly: object key order is ignored and array order is substantial, plus there is no type coercion. A mock is served only when the agent calls the tool with arguments that deep-equal the mock's `args`.
|
|
@@ -175,14 +222,15 @@ This is useful when a tool's arguments are noisy or generated by the model. The
|
|
|
175
222
|
|
|
176
223
|
### Failures
|
|
177
224
|
|
|
178
|
-
A
|
|
225
|
+
A tool call fails the item when it violates the mock configuration:
|
|
179
226
|
|
|
180
227
|
- `TOOL_MOCK_MISMATCH`: the tool was called with arguments that no mock matches.
|
|
181
228
|
- `TOOL_MOCK_EXHAUSTED`: every matching mock has already been consumed.
|
|
229
|
+
- `TOOL_MOCK_NOT_DECLARED`: the tool has no mock and the effective `unmockedToolPolicy` is `'deny'`.
|
|
182
230
|
|
|
183
|
-
|
|
231
|
+
On any of these failures, the agent run is aborted immediately, so the model can't go on to call any further tools, including unmocked, side-effecting tools that would otherwise run live. These failures are deterministic, so they're not retried. Mocks that are declared but never used don't fail the item, they're reported as unconsumed.
|
|
184
232
|
|
|
185
|
-
While
|
|
233
|
+
While mock interception is active, the agent's tools execute sequentially so repeated `(toolName, args)` mocks are consumed in the provider's call order. Interception is active when the item declares mocks or its effective `unmockedToolPolicy` is `'deny'`.
|
|
186
234
|
|
|
187
235
|
### Diagnostics
|
|
188
236
|
|
|
@@ -195,8 +243,8 @@ for (const item of summary.results) {
|
|
|
195
243
|
|
|
196
244
|
console.log(report.served) // mocks matched and returned
|
|
197
245
|
console.log(report.unconsumed) // mocks declared but never used
|
|
198
|
-
console.log(report.liveCalls) //
|
|
199
|
-
console.log(report.failure) // the
|
|
246
|
+
console.log(report.liveCalls) // undeclared tools allowed to run live
|
|
247
|
+
console.log(report.failure) // the first deterministic mock failure, if any
|
|
200
248
|
}
|
|
201
249
|
```
|
|
202
250
|
|
|
@@ -205,7 +253,7 @@ In [Studio](https://mastra.ai/docs/studio/overview), edit a dataset item to auth
|
|
|
205
253
|
### Limitations
|
|
206
254
|
|
|
207
255
|
- **No tool span for mocked calls.** A mocked call returns its output before the tool executes, so it doesn't create a tool span. Trajectory scorers backed by stored traces may therefore not see mocked tool calls. Trajectory extraction that falls back to the agent's message output still sees them, so trajectory scoring can differ depending on your observability configuration.
|
|
208
|
-
- **Storage support.** Tool mocks and tool mock reports are persisted by the LibSQL, PostgreSQL, MongoDB, and Spanner adapters. The MySQL adapter doesn't support them and rejects writes that carry tool mocks or a tool mock report
|
|
256
|
+
- **Storage support.** Tool mocks and tool mock reports are persisted by the LibSQL, PostgreSQL, MongoDB, and Spanner adapters. The MySQL adapter doesn't support them and rejects writes that carry tool mocks or a tool mock report. All dataset storage adapters persist `unmockedToolPolicy`.
|
|
209
257
|
|
|
210
258
|
## Async experiments
|
|
211
259
|
|
|
@@ -350,7 +398,7 @@ Visit the [`startExperiment` reference](https://mastra.ai/reference/datasets/sta
|
|
|
350
398
|
|
|
351
399
|
## Related
|
|
352
400
|
|
|
353
|
-
- [Datasets overview](https://mastra.ai/docs/
|
|
401
|
+
- [Datasets overview](https://mastra.ai/docs/datasets/overview)
|
|
354
402
|
- [Scorers overview](https://mastra.ai/docs/evals/overview)
|
|
355
403
|
- [`startExperiment` reference](https://mastra.ai/reference/datasets/startExperiment)
|
|
356
404
|
- [`listExperimentResults` reference](https://mastra.ai/reference/datasets/listExperimentResults)
|
|
@@ -17,7 +17,7 @@ Workers matter when any of these apply:
|
|
|
17
17
|
- Different parts of the system need to scale independently (e.g., more orchestration capacity without more API instances)
|
|
18
18
|
- Background tool calls should run on dedicated compute
|
|
19
19
|
|
|
20
|
-
If your application handles light traffic and workflows complete
|
|
20
|
+
If your application handles light traffic and workflows complete fast, the default in-process setup works fine. Skip the worker infrastructure until you need it.
|
|
21
21
|
|
|
22
22
|
## Worker types
|
|
23
23
|
|
|
@@ -33,11 +33,11 @@ The orchestration worker requires a PubSub backend that supports pull mode (e.g.
|
|
|
33
33
|
|
|
34
34
|
### Scheduler worker
|
|
35
35
|
|
|
36
|
-
Polls storage for due cron schedules and publishes `workflow.start` events. It
|
|
36
|
+
Polls storage for due cron schedules and publishes `workflow.start` events. It's a producer only, meaning it creates work for the orchestration worker to pick up.
|
|
37
37
|
|
|
38
38
|
The scheduler reads declarative `schedule` fields from your workflow definitions automatically. See [Scheduled workflows](https://mastra.ai/docs/workflows/scheduled-workflows) for how to declare schedules.
|
|
39
39
|
|
|
40
|
-
**
|
|
40
|
+
**Don't run more than one scheduler instance.** Multiple schedulers polling the same storage would fire duplicate events for the same schedule.
|
|
41
41
|
|
|
42
42
|
### Background task worker
|
|
43
43
|
|
|
@@ -47,7 +47,7 @@ The background task worker manages concurrency limits, task lifecycle, and resul
|
|
|
47
47
|
|
|
48
48
|
## How workers run
|
|
49
49
|
|
|
50
|
-
### In-process (default)
|
|
50
|
+
### In-process mode (default)
|
|
51
51
|
|
|
52
52
|
With no configuration, Mastra creates and starts workers inside the API process. Events flow through an in-memory PubSub, and everything shares a single Node.js runtime.
|
|
53
53
|
|
|
@@ -64,7 +64,7 @@ This setup needs no external infrastructure beyond your storage adapter. It does
|
|
|
64
64
|
|
|
65
65
|
### Split processes
|
|
66
66
|
|
|
67
|
-
To run workers
|
|
67
|
+
To run workers in their own processes, configure a distributed [PubSub](https://mastra.ai/docs/server/pubsub) backend and use the `MASTRA_WORKERS` environment variable to control which workers start in each process.
|
|
68
68
|
|
|
69
69
|
**Redis Streams + PostgreSQL**:
|
|
70
70
|
|
|
@@ -100,38 +100,38 @@ export const mastra = new Mastra({
|
|
|
100
100
|
})
|
|
101
101
|
```
|
|
102
102
|
|
|
103
|
-
Any [supported storage backend](https://mastra.ai/reference/workers/overview) works
|
|
103
|
+
Any [supported storage backend](https://mastra.ai/reference/workers/overview) works. Swap the storage adapter for your preferred database.
|
|
104
104
|
|
|
105
105
|
Run the same build artifact in multiple containers, each with a different [`MASTRA_WORKERS`](https://mastra.ai/reference/workers/overview) value to control which worker starts in each process.
|
|
106
106
|
|
|
107
107
|
Split deployments require a distributed PubSub backend ([`RedisStreamsPubSub`](https://mastra.ai/reference/pubsub/redis-streams) or [`GoogleCloudPubSub`](https://mastra.ai/reference/pubsub/google-cloud-pubsub)), a shared [storage backend](https://mastra.ai/reference/workers/overview), and network connectivity between the orchestration worker and the API.
|
|
108
108
|
|
|
109
|
-
The [worker deployment guide](https://mastra.ai/guides/deployment/mastra-workers) walks through this setup with
|
|
109
|
+
The [worker deployment guide](https://mastra.ai/guides/deployment/mastra-workers) walks through this setup with Docker Compose and Kubernetes examples.
|
|
110
110
|
|
|
111
111
|
## Network architecture
|
|
112
112
|
|
|
113
|
-
Workers are internal infrastructure. They
|
|
113
|
+
Workers are internal infrastructure. They're not exposed to end users and don't need their own subdomain, public URL, or inbound HTTP route.
|
|
114
114
|
|
|
115
115
|
In a split deployment:
|
|
116
116
|
|
|
117
|
-
- **The API server is the only public-facing process
|
|
118
|
-
- **Workers connect outbound only
|
|
119
|
-
- **The orchestration worker calls the API internally
|
|
117
|
+
- **The API server is the only public-facing process**: It serves all client HTTP requests, including REST endpoints, agent interactions, workflow triggers, and any custom routes.
|
|
118
|
+
- **Workers connect outbound only**: They pull events from the distributed PubSub backend and read/write to the shared storage database. They don't accept inbound traffic from clients.
|
|
119
|
+
- **The orchestration worker calls the API internally**: It sends step execution requests to the API over the container network using `MASTRA_STEP_EXECUTION_URL`. This is internal service-to-service communication, not a public endpoint.
|
|
120
120
|
|
|
121
121
|
All three worker types (orchestration, scheduler, background task) sit behind the API on a private network. They share access to the PubSub backend and storage database but never receive traffic directly from clients. If a worker-related feature needs an HTTP route (for example, token minting for a voice integration), that route runs on the API server, not on the worker process.
|
|
122
122
|
|
|
123
123
|
## Known limitations
|
|
124
124
|
|
|
125
|
-
- **No dead-letter queue**: Failed events are nacked and retried, but there
|
|
125
|
+
- **No dead-letter queue**: Failed events are nacked and retried, but there's no DLQ for events that fail after all retries.
|
|
126
126
|
- **No built-in health endpoint**: Workers don't expose an HTTP health check. Use container-level liveness probes or process monitoring.
|
|
127
127
|
- **Scheduler is single-instance**: Running multiple scheduler processes causes duplicate schedule fires.
|
|
128
128
|
- **Runs stuck in "running" after API crash**: If the API process crashes while executing a workflow step, the run remains in `running` status with no automatic retry. For [durable agents](https://mastra.ai/docs/long-running-agents/durable-agents), set `recovery.durableAgents` to `'auto'` in the Mastra config to automatically re-drive orphaned runs on server restart. See [Crash recovery](https://mastra.ai/docs/long-running-agents/durable-agents) for details.
|
|
129
129
|
|
|
130
130
|
## Related
|
|
131
131
|
|
|
132
|
-
- [Worker deployment guide](https://mastra.ai/guides/deployment/mastra-workers): Docker Compose
|
|
132
|
+
- [Worker deployment guide](https://mastra.ai/guides/deployment/mastra-workers): Docker Compose and Kubernetes examples
|
|
133
133
|
- [Worker authentication](https://mastra.ai/docs/server/auth/workers): Secure worker-to-API communication
|
|
134
|
-
- [Workers reference](https://mastra.ai/reference/workers/overview):
|
|
134
|
+
- [Workers reference](https://mastra.ai/reference/workers/overview): Details about worker environment variables and types, with a list of supported storage backends
|
|
135
135
|
- [CLI reference](https://mastra.ai/reference/cli/mastra): `mastra worker build` and `mastra worker start`
|
|
136
136
|
- [PubSub](https://mastra.ai/docs/server/pubsub): Event delivery backends
|
|
137
137
|
- [Scheduled workflows](https://mastra.ai/docs/workflows/scheduled-workflows): Declare cron schedules on workflows
|