@mastra/mcp-docs-server 1.2.13-alpha.8 → 1.2.13
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.docs/docs/agents/overview.md +1 -1
- package/.docs/docs/agents/supervisor-agents.md +2 -2
- package/.docs/docs/agents/using-tools.md +29 -0
- package/.docs/docs/capabilities/channels/imessage.md +181 -0
- package/.docs/docs/capabilities/channels/overview.md +1 -0
- package/.docs/docs/{evals/datasets → datasets}/overview.md +3 -3
- package/.docs/docs/{evals/datasets → datasets}/running-experiments.md +59 -2
- package/.docs/docs/editor/overview.md +201 -206
- package/.docs/docs/evals/evals-with-memory.md +1 -1
- package/.docs/docs/getting-started/develop.md +196 -0
- package/.docs/docs/index.md +78 -1
- package/.docs/docs/mastra-platform/workspaces.md +13 -13
- package/.docs/docs/memory/observational-memory.md +23 -0
- package/.docs/docs/studio/observability.md +4 -0
- package/.docs/docs/studio/overview.md +2 -2
- package/.docs/docs/workspace/search.md +1 -1
- package/.docs/guides/getting-started/quickstart.md +3 -3
- package/.docs/{docs → guides}/rag/chunking-and-embedding.md +1 -1
- package/.docs/{docs → guides}/rag/graph-rag.md +1 -1
- package/.docs/{docs → guides}/rag/overview.md +2 -2
- package/.docs/{docs → guides}/rag/retrieval.md +1 -1
- package/.docs/{docs → guides}/voice/overview.md +4 -4
- package/.docs/{docs → guides}/voice/realtime-voice.md +2 -2
- package/.docs/{docs → guides}/voice/speech-to-speech.md +1 -1
- package/.docs/{docs → guides}/voice/speech-to-text.md +1 -1
- package/.docs/{docs → guides}/voice/text-to-speech.md +2 -2
- package/.docs/models/embeddings.md +63 -65
- package/.docs/models/environment-variables.md +5 -1
- package/.docs/models/gateways/netlify.md +1 -2
- package/.docs/models/gateways/openrouter.md +5 -3
- package/.docs/models/gateways/vercel.md +8 -1
- package/.docs/models/index.md +1 -1
- package/.docs/models/providers/alibaba-cn.md +3 -1
- package/.docs/models/providers/alibaba-token-plan-cn.md +27 -25
- package/.docs/models/providers/alibaba-token-plan.md +27 -25
- package/.docs/models/providers/alibaba.md +2 -1
- package/.docs/models/providers/ambient.md +13 -12
- package/.docs/models/providers/baseten.md +3 -1
- package/.docs/models/providers/chutes.md +6 -8
- package/.docs/models/providers/crof.md +2 -1
- package/.docs/models/providers/crossmodel.md +7 -3
- package/.docs/models/providers/deepinfra.md +9 -2
- package/.docs/models/providers/digitalocean.md +34 -27
- package/.docs/models/providers/empiriolabs.md +4 -1
- package/.docs/models/providers/fireworks-ai.md +20 -19
- package/.docs/models/providers/greenpt.md +17 -8
- package/.docs/models/providers/groq.md +20 -20
- package/.docs/models/providers/huggingface.md +3 -1
- package/.docs/models/providers/hyper.md +9 -7
- package/.docs/models/providers/impossibl.md +148 -0
- package/.docs/models/providers/infomaniak.md +83 -0
- package/.docs/models/providers/kilo.md +349 -350
- package/.docs/models/providers/llmgateway.md +6 -3
- package/.docs/models/providers/mistral.md +5 -2
- package/.docs/models/providers/modal.md +5 -4
- package/.docs/models/providers/modelis.md +81 -0
- package/.docs/models/providers/nano-gpt.md +146 -148
- package/.docs/models/providers/neuralwatt.md +2 -1
- package/.docs/models/providers/ofox.md +2 -2
- package/.docs/models/providers/ollama-cloud.md +23 -22
- package/.docs/models/providers/opencode-go.md +3 -2
- package/.docs/models/providers/requesty.md +54 -45
- package/.docs/models/providers/scx.md +74 -0
- package/.docs/models/providers/tensorx.md +97 -0
- package/.docs/models/providers/togetherai.md +2 -1
- package/.docs/models/providers/umans-ai-coding-plan.md +11 -9
- package/.docs/models/providers/umans-ai.md +11 -9
- package/.docs/models/providers/wandb.md +2 -2
- package/.docs/models/providers/zai-coding-plan.md +11 -13
- package/.docs/models/providers/zhipuai-coding-plan.md +12 -12
- package/.docs/models/providers.md +5 -1
- package/.docs/reference/agents/agent.md +1 -1
- package/.docs/reference/agents/generate.md +1 -1
- package/.docs/reference/agents/getVoice.md +1 -1
- package/.docs/{docs/getting-started → reference}/build-with-ai.md +5 -110
- package/.docs/reference/cli/create-mastra.md +2 -0
- package/.docs/reference/cli/mastra.md +66 -1
- package/.docs/reference/client-js/agents.md +1 -1
- package/.docs/reference/configuration.md +2 -2
- package/.docs/reference/core/getEditor.md +1 -1
- package/.docs/reference/core/getVector.md +2 -2
- package/.docs/reference/core/listVectors.md +2 -2
- package/.docs/reference/core/mastra-class.md +1 -1
- package/.docs/reference/datasets/startExperiment.md +6 -0
- package/.docs/reference/datasets/startExperimentAsync.md +4 -0
- package/.docs/reference/editor/browser-provider.md +2 -2
- package/.docs/reference/editor/filesystem-provider.md +1 -1
- package/.docs/reference/editor/mastra-editor.md +31 -18
- package/.docs/reference/editor/prompt-blocks.md +137 -0
- package/.docs/reference/editor/sandbox-provider.md +1 -1
- package/.docs/reference/editor/storage-browser-ref.md +3 -3
- package/.docs/reference/editor/storage-workspace-ref.md +3 -3
- package/.docs/reference/editor/tool-provider.md +44 -10
- package/.docs/reference/editor/tools.md +100 -0
- package/.docs/reference/editor/versioning.md +130 -0
- package/.docs/reference/evals/mastra-scorer.md +68 -8
- package/.docs/reference/file-based-agents/config.md +39 -1
- package/.docs/reference/index.md +3 -4
- package/.docs/reference/memory/observational-memory.md +3 -1
- package/.docs/reference/memory/serialized-memory-config.md +3 -3
- package/.docs/reference/observability/tracing/interfaces.md +3 -1
- package/.docs/reference/processors/token-limiter-processor.md +15 -1
- package/.docs/reference/project-structure.md +1 -1
- package/.docs/reference/rag/database-config.md +1 -1
- package/.docs/reference/storage/composite.md +2 -0
- package/.docs/reference/streaming/agents/stream.md +1 -1
- package/.docs/reference/voice/livekit.md +3 -3
- package/.docs/reference/workspace/platform-filesystem.md +3 -3
- package/.docs/reference/workspace/platform-sandbox.md +5 -5
- package/CHANGELOG.md +30 -0
- package/README.md +1 -1
- package/package.json +6 -6
- package/.docs/docs/agent-builder/access-control.md +0 -99
- package/.docs/docs/agent-builder/browser.md +0 -63
- package/.docs/docs/agent-builder/channels.md +0 -78
- package/.docs/docs/agent-builder/configuration.md +0 -132
- package/.docs/docs/agent-builder/deploying.md +0 -124
- package/.docs/docs/agent-builder/integrations.md +0 -115
- package/.docs/docs/agent-builder/memory.md +0 -69
- package/.docs/docs/agent-builder/model-policy.md +0 -50
- package/.docs/docs/agent-builder/overview.md +0 -114
- package/.docs/docs/agent-builder/skill-registries.md +0 -31
- package/.docs/docs/agent-builder/workspace.md +0 -62
- package/.docs/docs/editor/prompts.md +0 -140
- package/.docs/docs/editor/tools.md +0 -148
- package/.docs/docs/getting-started/file-based-agents.md +0 -121
- package/.docs/docs/what-is-mastra.md +0 -161
- package/.docs/reference/client-js/agent-builder.md +0 -163
- package/.docs/reference/editor/agent-builder/agent-builder-options.md +0 -76
- package/.docs/reference/editor/agent-builder/builder-agent-defaults.md +0 -79
- package/.docs/reference/editor/agent-builder/builder-models.md +0 -66
- /package/.docs/{docs → guides}/getting-started/manual-install.md +0 -0
- /package/.docs/{docs → guides}/rag/vector-databases.md +0 -0
|
@@ -210,7 +210,7 @@ Once your agent is running, use this table to find the right page for what you w
|
|
|
210
210
|
| Keep your agent safe | [Guardrails](https://mastra.ai/docs/agents/guardrails) |
|
|
211
211
|
| Build agents that correct their work | [Rubric scorer](https://mastra.ai/docs/agents/supervisor-agents) |
|
|
212
212
|
| Swap instructions or models based on request context | [Dynamic configuration](https://mastra.ai/docs/server/request-context) |
|
|
213
|
-
| Add speech-to-text or text-to-speech | [Voice](https://mastra.ai/
|
|
213
|
+
| Add speech-to-text or text-to-speech | [Voice](https://mastra.ai/guides/voice/overview) |
|
|
214
214
|
| Connect to Slack, Discord, or Telegram | [Channels](https://mastra.ai/docs/capabilities/channels/overview) |
|
|
215
215
|
|
|
216
216
|
## Multi-agent systems
|
|
@@ -420,12 +420,12 @@ const result = await supervisor.generate('Research and write about AI safety', {
|
|
|
420
420
|
})
|
|
421
421
|
```
|
|
422
422
|
|
|
423
|
-
Version overrides propagate automatically through delegation. See [Subagent versioning](https://mastra.ai/
|
|
423
|
+
Version overrides propagate automatically through delegation. See [Subagent versioning](https://mastra.ai/reference/editor/versioning) for details on resolution order and server API usage.
|
|
424
424
|
|
|
425
425
|
## Related
|
|
426
426
|
|
|
427
427
|
- [Background tasks](https://mastra.ai/docs/long-running-agents/background-tasks)
|
|
428
|
-
- [Subagent versioning](https://mastra.ai/
|
|
428
|
+
- [Subagent versioning](https://mastra.ai/reference/editor/versioning)
|
|
429
429
|
- [Guide: Research coordinator](https://mastra.ai/guides/guide/research-coordinator)
|
|
430
430
|
- [Agent.stream() reference](https://mastra.ai/reference/streaming/agents/stream)
|
|
431
431
|
- [Agent.streamUntilIdle() reference](https://mastra.ai/reference/streaming/agents/streamUntilIdle)
|
|
@@ -478,6 +478,7 @@ Mastra includes agent-agnostic built-in tools in `@mastra/core/tools` that add i
|
|
|
478
478
|
| `task_complete` | Mark one tracked task completed |
|
|
479
479
|
| `task_check` | Check task list completion status |
|
|
480
480
|
| `webSearchTool` | Run provider-native web search with the active model |
|
|
481
|
+
| `webFetchTool` | Fetch a web page by URL and return its text content |
|
|
481
482
|
|
|
482
483
|
### Use provider web search
|
|
483
484
|
|
|
@@ -502,6 +503,34 @@ export const researchAgent = new Agent({
|
|
|
502
503
|
|
|
503
504
|
The `search` key is only the agent-local tool name. Use any key. The `webSearchTool` value tells Mastra to use provider web search.
|
|
504
505
|
|
|
506
|
+
### Fetch a web page
|
|
507
|
+
|
|
508
|
+
Import `webFetchTool` from `@mastra/core/tools` when the agent needs to read a specific URL. The tool requests the page over HTTP or HTTPS and returns its text content plus response metadata.
|
|
509
|
+
|
|
510
|
+
```typescript
|
|
511
|
+
import { Agent } from '@mastra/core/agent'
|
|
512
|
+
import { webFetchTool } from '@mastra/core/tools'
|
|
513
|
+
|
|
514
|
+
export const readerAgent = new Agent({
|
|
515
|
+
id: 'reader-agent',
|
|
516
|
+
name: 'Reader Agent',
|
|
517
|
+
instructions: 'Fetch the page the user links to before answering.',
|
|
518
|
+
model: 'openai/gpt-5.6-sol',
|
|
519
|
+
tools: {
|
|
520
|
+
fetch: webFetchTool,
|
|
521
|
+
},
|
|
522
|
+
})
|
|
523
|
+
```
|
|
524
|
+
|
|
525
|
+
The tool takes a single `url` input and returns `content`, `truncated`, `status`, `statusText`, `contentType`, `url`, and `ok`. It applies these limits:
|
|
526
|
+
|
|
527
|
+
- Only `http:` and `https:` URLs are allowed.
|
|
528
|
+
- Requests to `localhost` and to private or reserved IP addresses are blocked, including addresses returned by DNS resolution.
|
|
529
|
+
- Responses are truncated at 100,000 characters, with `truncated: true` in the result.
|
|
530
|
+
- Requests follow at most 5 redirects and time out after 15 seconds.
|
|
531
|
+
|
|
532
|
+
Failures don't throw. The tool returns `isError: true` with the reason in `content`, so the agent can retry or explain the problem.
|
|
533
|
+
|
|
505
534
|
### Ask the user a question
|
|
506
535
|
|
|
507
536
|
Import [`askUserTool`](https://mastra.ai/reference/tools/ask-user-tool) and add it to the agent's toolset.
|
|
@@ -0,0 +1,181 @@
|
|
|
1
|
+
> Discover all available pages from the documentation index: https://mastra.ai/llms.txt
|
|
2
|
+
|
|
3
|
+
# iMessage
|
|
4
|
+
|
|
5
|
+
iMessage channels let a Mastra agent receive direct messages and group messages from iMessage. Mastra handles the agent wiring, the webhook route, and the gateway listener; the Photon iMessage adapter docs cover number provisioning, credentials, and webhook registration.
|
|
6
|
+
|
|
7
|
+
## Install the adapter
|
|
8
|
+
|
|
9
|
+
Install the Photon iMessage adapter:
|
|
10
|
+
|
|
11
|
+
**npm**:
|
|
12
|
+
|
|
13
|
+
```bash
|
|
14
|
+
npm install @photon-ai/chat-adapter-imessage
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
**pnpm**:
|
|
18
|
+
|
|
19
|
+
```bash
|
|
20
|
+
pnpm add @photon-ai/chat-adapter-imessage
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
**Yarn**:
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
yarn add @photon-ai/chat-adapter-imessage
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
**Bun**:
|
|
30
|
+
|
|
31
|
+
```bash
|
|
32
|
+
bun add @photon-ai/chat-adapter-imessage
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
## Agent configuration
|
|
36
|
+
|
|
37
|
+
Add `createiMessageAdapter()` to the agent's `channels.adapters` object:
|
|
38
|
+
|
|
39
|
+
```typescript
|
|
40
|
+
import { Agent } from '@mastra/core/agent'
|
|
41
|
+
import { createiMessageAdapter } from '@photon-ai/chat-adapter-imessage'
|
|
42
|
+
|
|
43
|
+
export const imessageAgent = new Agent({
|
|
44
|
+
id: 'imessage-agent',
|
|
45
|
+
name: 'iMessage Agent',
|
|
46
|
+
instructions: 'Answer questions and help with tasks over iMessage.',
|
|
47
|
+
model: 'openai/gpt-5.6-sol',
|
|
48
|
+
channels: {
|
|
49
|
+
adapters: {
|
|
50
|
+
imessage: {
|
|
51
|
+
adapter: createiMessageAdapter(),
|
|
52
|
+
toolDisplay: 'text',
|
|
53
|
+
},
|
|
54
|
+
},
|
|
55
|
+
threadContext: { maxMessages: 0 },
|
|
56
|
+
},
|
|
57
|
+
})
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
Register the agent on the Mastra instance:
|
|
61
|
+
|
|
62
|
+
```typescript
|
|
63
|
+
import { Mastra } from '@mastra/core'
|
|
64
|
+
import { imessageAgent } from './agents/imessage-agent'
|
|
65
|
+
|
|
66
|
+
export const mastra = new Mastra({
|
|
67
|
+
agents: { imessageAgent },
|
|
68
|
+
})
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
Use `imessage` as the adapter key. Mastra derives the webhook path and the `platform` value on `requestContext` from this key.
|
|
72
|
+
|
|
73
|
+
`toolDisplay: 'text'` describes tool calls in the message, since iMessage has no interactive cards for Approve and Deny actions. `threadContext: { maxMessages: 0 }` skips the platform history lookup Mastra runs when an agent is first mentioned in a group chat, which the adapter can't perform. Both override defaults that assume platform features iMessage lacks.
|
|
74
|
+
|
|
75
|
+
## Adapter setup
|
|
76
|
+
|
|
77
|
+
Follow the [Photon iMessage adapter docs](https://github.com/photon-hq/vercel-chat-adapter-imessage) for iMessage-specific setup, including number provisioning, hosted and self-hosted modes, and webhook registration. The adapter picks its mode from the environment variables you set.
|
|
78
|
+
|
|
79
|
+
For the hosted service, create a project at [app.photon.codes](https://app.photon.codes) and use the project credentials:
|
|
80
|
+
|
|
81
|
+
```bash
|
|
82
|
+
IMESSAGE_PROJECT_ID=your-project-id
|
|
83
|
+
IMESSAGE_PROJECT_SECRET=your-project-secret
|
|
84
|
+
IMESSAGE_WEBHOOK_SECRET=your-webhook-signing-secret
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
For a self-hosted server, point the adapter at its gRPC address, written as `host:port`. The adapter strips any URL scheme and appends `:443` to a bare host:
|
|
88
|
+
|
|
89
|
+
```bash
|
|
90
|
+
IMESSAGE_SERVER_URL=imessage.example.com:443
|
|
91
|
+
IMESSAGE_API_KEY=your-server-token
|
|
92
|
+
IMESSAGE_PHONE=+15551234567
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
`IMESSAGE_PHONE` is optional and routes messages when a self-hosted server has several numbers. You can also pass these values to `createiMessageAdapter()` directly, including a `credentials` function that resolves the project ID and secret at first use from a secret store.
|
|
96
|
+
|
|
97
|
+
## Webhook URL
|
|
98
|
+
|
|
99
|
+
Mastra generates the iMessage webhook route from the agent ID and adapter key:
|
|
100
|
+
|
|
101
|
+
```text
|
|
102
|
+
/api/agents/imessage-agent/channels/imessage/webhook
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
Use your public Mastra server URL as the base URL:
|
|
106
|
+
|
|
107
|
+
```text
|
|
108
|
+
https://your-app.example.com/api/agents/imessage-agent/channels/imessage/webhook
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
Register this URL in the [Photon dashboard](https://app.photon.codes), then set the signing secret it returns as `IMESSAGE_WEBHOOK_SECRET`. The secret is shown once at registration. The adapter verifies the signature on every delivery and rejects requests that don't match. Webhooks are available in hosted mode only.
|
|
112
|
+
|
|
113
|
+
Photon retries failed deliveries with backoff and delivers at least once, so the same message can arrive twice. Chat SDK drops the repeat using the channel state adapter, and Mastra's default keeps those dedup keys in memory. That covers a single long-running server.
|
|
114
|
+
|
|
115
|
+
A repeat can still reach the agent after a restart, or on serverless where the retry is routed to a different instance. Pass a shared state adapter on `channels.state` so dedup keys are visible everywhere. Install one alongside the adapter:
|
|
116
|
+
|
|
117
|
+
**npm**:
|
|
118
|
+
|
|
119
|
+
```bash
|
|
120
|
+
npm install @chat-adapter/state-redis
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
**pnpm**:
|
|
124
|
+
|
|
125
|
+
```bash
|
|
126
|
+
pnpm add @chat-adapter/state-redis
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
**Yarn**:
|
|
130
|
+
|
|
131
|
+
```bash
|
|
132
|
+
yarn add @chat-adapter/state-redis
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
**Bun**:
|
|
136
|
+
|
|
137
|
+
```bash
|
|
138
|
+
bun add @chat-adapter/state-redis
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
`createRedisState()` reads the `REDIS_URL` environment variable:
|
|
142
|
+
|
|
143
|
+
```typescript
|
|
144
|
+
import { createRedisState } from '@chat-adapter/state-redis'
|
|
145
|
+
|
|
146
|
+
channels: {
|
|
147
|
+
adapters: {
|
|
148
|
+
imessage: {
|
|
149
|
+
adapter: createiMessageAdapter(),
|
|
150
|
+
toolDisplay: 'text',
|
|
151
|
+
},
|
|
152
|
+
},
|
|
153
|
+
threadContext: { maxMessages: 0 },
|
|
154
|
+
state: createRedisState(),
|
|
155
|
+
},
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
This matters most for tools with side effects, where handling the same message twice is visible to the user.
|
|
159
|
+
|
|
160
|
+
> **Note:** Photon delivers to public HTTPS endpoints only. It won't deliver to `http://`, to private addresses like `localhost`, or through a redirect. For local development, use a tunnel as described in the [Channels overview](https://mastra.ai/docs/capabilities/channels/overview).
|
|
161
|
+
|
|
162
|
+
## Gateway listener
|
|
163
|
+
|
|
164
|
+
The adapter can hold an open connection and stream messages in real time instead of receiving webhooks. This works in both hosted and self-hosted modes.
|
|
165
|
+
|
|
166
|
+
Mastra starts this listener during initialization and reconnects it if it drops, so no cron job or extra route is needed on a long-running server. Set `gateway: false` on the adapter config to turn it off when you use webhooks:
|
|
167
|
+
|
|
168
|
+
```typescript
|
|
169
|
+
imessage: {
|
|
170
|
+
adapter: createiMessageAdapter(),
|
|
171
|
+
toolDisplay: 'text',
|
|
172
|
+
gateway: false,
|
|
173
|
+
},
|
|
174
|
+
```
|
|
175
|
+
|
|
176
|
+
On serverless platforms, prefer webhooks. A gateway listener needs a process that stays alive. See [Serverless deployment](https://mastra.ai/docs/capabilities/channels/overview).
|
|
177
|
+
|
|
178
|
+
## Related
|
|
179
|
+
|
|
180
|
+
- [Channels overview](https://mastra.ai/docs/capabilities/channels/overview)
|
|
181
|
+
- [More](https://mastra.ai/docs/capabilities/channels/other-adapters)
|
|
@@ -13,6 +13,7 @@ Start with the page for your platform:
|
|
|
13
13
|
- [Discord](https://mastra.ai/docs/capabilities/channels/discord)
|
|
14
14
|
- [Telegram](https://mastra.ai/docs/capabilities/channels/telegram)
|
|
15
15
|
- [WhatsApp](https://mastra.ai/docs/capabilities/channels/whatsapp)
|
|
16
|
+
- [iMessage](https://mastra.ai/docs/capabilities/channels/imessage)
|
|
16
17
|
|
|
17
18
|
[More](https://mastra.ai/docs/capabilities/channels/other-adapters) lists additional platforms. Mastra channels work with compatible [Chat SDK adapters](https://chat-sdk.dev/adapters) beyond the platforms listed here, and the same Mastra configuration pattern applies across adapters.
|
|
18
19
|
|
|
@@ -49,7 +49,7 @@ Visit the [`DatasetsManager` reference](https://mastra.ai/reference/datasets/dat
|
|
|
49
49
|
|
|
50
50
|
You can also manage datasets in [Studio](https://mastra.ai/docs/studio/overview). After opening Studio, select **Datasets** from the sidebar to see all your available datasets or create a new one.
|
|
51
51
|
|
|
52
|
-
To get started, select **Create Dataset** and set a name, description, and optional schemas. After confirming, you'll see the dataset details page with two tabs: **Items** and [**Experiments**](https://mastra.ai/docs/
|
|
52
|
+
To get started, select **Create Dataset** and set a name, description, and optional schemas. After confirming, you'll see the dataset details page with two tabs: **Items** and [**Experiments**](https://mastra.ai/docs/datasets/running-experiments).
|
|
53
53
|
|
|
54
54
|
In the **Items** view you can add, update, and delete items, and view version history. Select **Add Item** to insert a new item with JSON editors for input and ground truth. From this view you can also import items in bulk from a CSV or JSON file. When importing, map each column to the corresponding dataset field.
|
|
55
55
|
|
|
@@ -198,11 +198,11 @@ Fetch the exact items that existed at a past version:
|
|
|
198
198
|
const items = await dataset.listItems({ version: 2 })
|
|
199
199
|
```
|
|
200
200
|
|
|
201
|
-
You can also pin experiments to a version, see [running experiments](https://mastra.ai/docs/
|
|
201
|
+
You can also pin experiments to a version, see [running experiments](https://mastra.ai/docs/datasets/running-experiments). Visit the [`Dataset` reference](https://mastra.ai/reference/datasets/dataset) for the full list of methods and parameters.
|
|
202
202
|
|
|
203
203
|
## Related
|
|
204
204
|
|
|
205
|
-
- [Running experiments](https://mastra.ai/docs/
|
|
205
|
+
- [Running experiments](https://mastra.ai/docs/datasets/running-experiments)
|
|
206
206
|
- [Scorers overview](https://mastra.ai/docs/evals/overview)
|
|
207
207
|
- [DatasetsManager reference](https://mastra.ai/reference/datasets/datasets-manager)
|
|
208
208
|
- [Dataset reference](https://mastra.ai/reference/datasets/dataset)
|
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
|
|
5
5
|
**Added in:** `@mastra/core@1.4.0`
|
|
6
6
|
|
|
7
|
-
An experiment runs every item in a dataset through a target (an agent, a workflow, or a scorer) and then optionally scores the outputs. Use a scorer as the target when you want to evaluate an LLM judge itself.
|
|
7
|
+
An experiment runs every item in a dataset through a target (an agent, a workflow, or a scorer) and then optionally scores the outputs. Use a scorer as the target when you want to evaluate an LLM judge itself. By default, results are persisted to storage so you can compare runs across different prompts, models, or code changes.
|
|
8
8
|
|
|
9
9
|
**For AI agents:** Run `npx mastra api experiment run dataset_123 '{"name":"translation-baseline"}'` to start an experiment directly instead of opening Studio or writing a temporary script. Use a dataset ID returned by `npx mastra api dataset list` in place of the sample ID. The command requires a running Mastra server with dataset storage and registered experiment targets; start the local server with `npx mastra dev`, or pass the reachable server's base URL with `--url`. Run `npx mastra api experiment run --schema` before constructing different input, and get user approval before starting an experiment because it can make model calls. Install Mastra's skill with `npx skills add mastra-ai/skills --skill mastra` for complete API CLI discovery, targeting, schema, authentication, and error-handling guidance.
|
|
10
10
|
|
|
@@ -131,6 +131,63 @@ for (const item of summary.results) {
|
|
|
131
131
|
|
|
132
132
|
Visit the [Scorers overview](https://mastra.ai/docs/evals/overview) for details on available and custom scorers.
|
|
133
133
|
|
|
134
|
+
## Control persistence per run
|
|
135
|
+
|
|
136
|
+
Use `persistence` to skip storage writes for a specific run. Experiment records and score records can be disabled independently:
|
|
137
|
+
|
|
138
|
+
```typescript
|
|
139
|
+
const summary = await dataset.startExperiment({
|
|
140
|
+
targetType: 'agent',
|
|
141
|
+
targetId: 'translation-agent',
|
|
142
|
+
scorers: ['accuracy'],
|
|
143
|
+
persistence: {
|
|
144
|
+
experiments: 'none',
|
|
145
|
+
scores: 'none',
|
|
146
|
+
},
|
|
147
|
+
})
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
The target and scorers still run, and `startExperiment()` still returns the item results and scores in `summary`. The settings are independent. For example, set only `scores: 'none'` to persist the experiment and its item results without creating score records.
|
|
151
|
+
|
|
152
|
+
Omitted settings default to `'default'`, which preserves the standard storage behavior. This policy only controls experiment and score records created by the run. It doesn't disable storage used by the target, such as agent memory, vectors, observability, or custom tool storage.
|
|
153
|
+
|
|
154
|
+
When `startExperimentAsync()` runs with `experiments: 'none'`, it doesn't persist an experiment record, progress updates, or item results. Score persistence remains controlled separately by `persistence.scores`. Without an experiment event observer, the run is fire-and-forget, and the experiment API can't report whether it completed or failed.
|
|
155
|
+
|
|
156
|
+
Use synchronous `startExperiment()` when the caller needs the returned summary. An experiment event observer can receive lifecycle events and the terminal summary.
|
|
157
|
+
|
|
158
|
+
## Observe experiment events
|
|
159
|
+
|
|
160
|
+
Use `onEvent` to receive versioned, JSON-safe lifecycle events while an experiment runs. This works with `startExperiment()`, `startExperimentAsync()`, and `runExperiment()`.
|
|
161
|
+
|
|
162
|
+
```typescript
|
|
163
|
+
import type { ExperimentEvent } from '@mastra/core/datasets'
|
|
164
|
+
|
|
165
|
+
const events: ExperimentEvent[] = []
|
|
166
|
+
|
|
167
|
+
await dataset.startExperimentAsync({
|
|
168
|
+
task: async ({ input }) => processItem(input),
|
|
169
|
+
persistence: { experiments: 'none' },
|
|
170
|
+
onEvent: async event => {
|
|
171
|
+
events.push(event)
|
|
172
|
+
await publishEvent(event)
|
|
173
|
+
},
|
|
174
|
+
})
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
The observer receives these event types:
|
|
178
|
+
|
|
179
|
+
- `experiment.run.started`: Identifies the run, target, resolved dataset version, and item count.
|
|
180
|
+
- `experiment.item.completed`: Reports a committed item result after scoring, including scores, errors, retry count, tool mock details, and stable item identity.
|
|
181
|
+
- `experiment.run.finished`: Reports the terminal outcome and summary counters.
|
|
182
|
+
|
|
183
|
+
Mastra awaits each observer call before delivering the next event. This serialized delivery applies backpressure and ensures event `sequence` values match delivery order, while item execution can remain concurrent.
|
|
184
|
+
|
|
185
|
+
If the observer throws or rejects, Mastra aborts the remaining run and rejects `runExperiment()` with a `MastraError` whose `id` is `EXPERIMENT_EVENT_OBSERVER_FAILED`. It doesn't send a terminal event through the failed observer. For `startExperimentAsync()`, the method has already returned when a detached observer fails, so handle delivery failures inside the observer when the caller needs direct error reporting.
|
|
186
|
+
|
|
187
|
+
The `experiment.run.finished` event is awaited before Mastra persists the final experiment status. Treat the event as the authoritative terminal signal when experiment persistence is disabled, but don't use it as a read-after-write signal for storage.
|
|
188
|
+
|
|
189
|
+
The exported event types are `ExperimentEvent`, `ExperimentRunStartedEvent`, `ExperimentItemCompletedEvent`, and `ExperimentRunFinishedEvent`. Use the discriminated `type` field to narrow an event before reading event-specific properties.
|
|
190
|
+
|
|
134
191
|
## Tool mocks
|
|
135
192
|
|
|
136
193
|
When an experiment runs an agent that calls side-effecting tools, attach static tool mocks to individual dataset items to make the run deterministic. During the experiment, a mocked tool returns its declared output instead of executing. Tools without a mock on the item run live by default.
|
|
@@ -374,7 +431,7 @@ Visit the [`startExperiment` reference](https://mastra.ai/reference/datasets/sta
|
|
|
374
431
|
|
|
375
432
|
## Related
|
|
376
433
|
|
|
377
|
-
- [Datasets overview](https://mastra.ai/docs/
|
|
434
|
+
- [Datasets overview](https://mastra.ai/docs/datasets/overview)
|
|
378
435
|
- [Scorers overview](https://mastra.ai/docs/evals/overview)
|
|
379
436
|
- [`startExperiment` reference](https://mastra.ai/reference/datasets/startExperiment)
|
|
380
437
|
- [`listExperimentResults` reference](https://mastra.ai/reference/datasets/listExperimentResults)
|