@mastra/mcp-docs-server 1.2.1 → 1.2.2-alpha.10

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (51) hide show
  1. package/.docs/docs/agents/background-tasks.md +5 -5
  2. package/.docs/docs/agents/code-mode.md +1 -1
  3. package/.docs/docs/agents/goals.md +1 -1
  4. package/.docs/docs/agents/processors.md +1 -1
  5. package/.docs/docs/agents/signal-providers.md +1 -1
  6. package/.docs/docs/agents/signals.md +1 -1
  7. package/.docs/docs/agents/supervisor-agents.md +15 -0
  8. package/.docs/docs/evals/built-in-scorers.md +17 -1
  9. package/.docs/docs/evals/gates-and-verdicts.md +140 -0
  10. package/.docs/docs/evals/overview.md +2 -0
  11. package/.docs/docs/evals/quick-checks.md +136 -0
  12. package/.docs/docs/harness/modes.md +21 -0
  13. package/.docs/docs/harness/overview.md +1 -1
  14. package/.docs/docs/harness/session.md +7 -3
  15. package/.docs/docs/harness/threads-and-state.md +2 -0
  16. package/.docs/docs/memory/observational-memory.md +1 -3
  17. package/.docs/guides/guide/signal-provider.md +1 -1
  18. package/.docs/models/gateways/netlify.md +1 -3
  19. package/.docs/models/gateways/openrouter.md +2 -1
  20. package/.docs/models/gateways/vercel.md +3 -1
  21. package/.docs/models/index.md +1 -1
  22. package/.docs/models/providers/baseten.md +2 -2
  23. package/.docs/models/providers/berget.md +4 -3
  24. package/.docs/models/providers/evroc.md +19 -17
  25. package/.docs/models/providers/huggingface.md +4 -1
  26. package/.docs/models/providers/llmgateway.md +184 -199
  27. package/.docs/models/providers/minimax-cn-coding-plan.md +1 -1
  28. package/.docs/models/providers/minimax-cn.md +1 -1
  29. package/.docs/models/providers/minimax-coding-plan.md +1 -1
  30. package/.docs/models/providers/minimax.md +1 -1
  31. package/.docs/models/providers/neuralwatt.md +2 -1
  32. package/.docs/models/providers/opencode-go.md +1 -1
  33. package/.docs/models/providers/ovhcloud.md +1 -2
  34. package/.docs/models/providers/stepfun.md +3 -2
  35. package/.docs/models/providers/togetherai.md +3 -2
  36. package/.docs/reference/agents/channels.md +4 -0
  37. package/.docs/reference/agents/durable-agent.md +30 -3
  38. package/.docs/reference/agents/generate.md +1 -1
  39. package/.docs/reference/ai-sdk/workflow-snapshot-to-stream.md +51 -0
  40. package/.docs/reference/cli/mastra.md +8 -0
  41. package/.docs/reference/evals/checks.md +210 -0
  42. package/.docs/reference/evals/run-evals.md +71 -1
  43. package/.docs/reference/harness/harness-class.md +90 -50
  44. package/.docs/reference/harness/session.md +35 -4
  45. package/.docs/reference/index.md +2 -0
  46. package/.docs/reference/memory/observational-memory.md +1 -1
  47. package/.docs/reference/memory/serialized-memory-config.md +1 -1
  48. package/.docs/reference/processors/processor-interface.md +2 -0
  49. package/.docs/reference/streaming/agents/stream.md +1 -1
  50. package/CHANGELOG.md +49 -0
  51. package/package.json +3 -3
@@ -49,7 +49,7 @@ Once a tool has opted in, the LLM can optionally include a `_background` field i
49
49
 
50
50
  ### Tool-level
51
51
 
52
- Set `backgroundTasks.enabled: true` on the tool definition. Tools opted in at this layer run in the background whenever called by an agent that has the manager enabled.
52
+ Set `background.enabled: true` on the tool definition. Tools opted in at this layer run in the background whenever called by an agent that has the manager enabled.
53
53
 
54
54
  ```typescript
55
55
  import { createTool } from '@mastra/core/tools'
@@ -59,7 +59,7 @@ export const researchTool = createTool({
59
59
  id: 'research',
60
60
  description: 'Run a long research job',
61
61
  inputSchema: z.object({ topic: z.string() }),
62
- backgroundTasks: {
62
+ background: {
63
63
  enabled: true,
64
64
  timeoutMs: 600_000,
65
65
  maxRetries: 1,
@@ -111,7 +111,7 @@ The `_background` override is a _modifier_ on tools the developer has already op
111
111
  When a tool call is dispatched, the resolved background config is computed in this priority order:
112
112
 
113
113
  1. Agent-level `backgroundTasks.tools` entry for the tool.
114
- 2. Tool-level `backgroundTasks` config.
114
+ 2. Tool-level `background` config.
115
115
  3. LLM `_background.enabled` override (only used to enable background dispatch when the tool was opted in at one of the layers above).
116
116
  4. Manager defaults (`defaultTimeoutMs`, `defaultRetries`).
117
117
 
@@ -200,7 +200,7 @@ const stream = await supervisor.stream('Research AI in education and write an ar
200
200
 
201
201
  ### Inheriting from the subagent
202
202
 
203
- If a subagent isn't listed under the supervisor's `backgroundTasks.tools` but has background-eligible tools of its own (either via tool-level `backgroundTasks.enabled: true` or its own `backgroundTasks.tools` entry) the framework still dispatches the entire subagent invocation as a background task. The supervisor inherits the subagent's intent: the subagent itself becomes the background task, and its inner tools run in the foreground inside the subagent's loop.
203
+ If a subagent isn't listed under the supervisor's `backgroundTasks.tools` but has background-eligible tools of its own (either via tool-level `background.enabled: true` or its own `backgroundTasks.tools` entry) the framework still dispatches the entire subagent invocation as a background task. The supervisor inherits the subagent's intent: the subagent itself becomes the background task, and its inner tools run in the foreground inside the subagent's loop.
204
204
 
205
205
  The background config used for the inherited dispatch (for example `waitTimeoutMs`) is derived from the subagent's own `backgroundTasks` config.
206
206
 
@@ -285,7 +285,7 @@ The manager keeps tool executors in process memory. If the process restarts whil
285
285
 
286
286
  Each layer can register terminal-state callbacks. They don't replace one another, and success/failure hooks fire for their respective outcomes:
287
287
 
288
- - Tool-level `backgroundTasks.onComplete` / `onFailed`: scoped to one tool.
288
+ - Tool-level `background.onComplete` / `onFailed`: scoped to one tool.
289
289
  - Agent-level `backgroundTasks.onTaskComplete` / `onTaskFailed`: scoped to all tasks dispatched by this agent.
290
290
  - Manager-level `onTaskComplete` / `onTaskFailed`: scoped globally.
291
291
 
@@ -2,7 +2,7 @@
2
2
 
3
3
  **Added in:** `@mastra/core@1.38.0`
4
4
 
5
- > **Alpha:** This feature is in alpha. Breaking changes may occur without a major version bump until the API is stable.
5
+ > **Beta:** This feature is in beta. Breaking changes may occur without a major version bump until the API is stable.
6
6
 
7
7
  Code mode lets an agent run multi-tool computations in an isolated sandbox and return the result as a single, more accurate response. Instead of calling tools one turn at a time, the model writes a tailored function for the user query that orchestrates your existing tools as `external_*` functions, reduces or aggregates their results, and returns one structured answer.
8
8
 
@@ -2,7 +2,7 @@
2
2
 
3
3
  **Added in:** `@mastra/core@1.42.0`
4
4
 
5
- > **Alpha:** The Goals feature is in alpha stage and subject to breaking changes in minor versions until it graduates from its alpha status.
5
+ > **Beta:** The Goals feature is in beta stage and subject to breaking changes in minor versions until it graduates from its beta status.
6
6
 
7
7
  A goal is a durable, thread-scoped objective: a standing instruction the agent keeps working toward across loop iterations until a judge model decides it is satisfied or a run budget is exhausted. The objective is persisted in thread state, so it survives reloads and is evaluated in-loop — even when a new message arrives in the middle of an already-running turn.
8
8
 
@@ -427,7 +427,7 @@ See the [`ProviderHistoryCompat` reference](https://mastra.ai/reference/processo
427
427
 
428
428
  ## Response caching
429
429
 
430
- > **Alpha:** This feature is in alpha. Breaking changes may occur without a major version bump until the API is stable.
430
+ > **Beta:** This feature is in beta. Breaking changes may occur without a major version bump until the API is stable.
431
431
 
432
432
  Response caching skips the LLM call and replays a previously cached response when an agent receives an identical request. Use it to reduce latency and avoid paying for repeated calls.
433
433
 
@@ -2,7 +2,7 @@
2
2
 
3
3
  **Added in:** `@mastra/core@1.39.0`
4
4
 
5
- > **Alpha:** This feature is in alpha. Breaking changes may occur without a major version bump until the API is stable.
5
+ > **Beta:** This feature is in beta. Breaking changes may occur without a major version bump until the API is stable.
6
6
 
7
7
  A signal provider monitors an external source, such as GitHub, Slack, continuous integration (CI), or your own API, and pushes [notification signals](https://mastra.ai/docs/agents/signals) into subscribed agent threads.
8
8
 
@@ -2,7 +2,7 @@
2
2
 
3
3
  **Added in:** `@mastra/core@1.39.0`
4
4
 
5
- > **Alpha:** This feature is in alpha. Breaking changes may occur without a major version bump until the API is stable.
5
+ > **Beta:** This feature is in beta. Breaking changes may occur without a major version bump until the API is stable.
6
6
 
7
7
  Signals are a way to interact with an agent through a thread. Instead of starting every interaction with `agent.stream()`, subscribe to a thread and send messages or signals. Mastra either wakes the agent when the thread is idle, drops input into the running agent loop, or queues input for the next turn.
8
8
 
@@ -261,6 +261,21 @@ for await (const chunk of stream.fullStream) {
261
261
  }
262
262
  ```
263
263
 
264
+ ## Cancellation
265
+
266
+ When you pass an `abortSignal` to the supervisor's [`stream()`](https://mastra.ai/reference/streaming/agents/stream) or [`generate()`](https://mastra.ai/reference/agents/generate) call, Mastra forwards that same signal to delegated subagents. Calling `AbortController.abort()` cancels in-flight subagent runs at their next step instead of letting them run to completion.
267
+
268
+ ```typescript
269
+ const controller = new AbortController()
270
+
271
+ const stream = await supervisor.stream('Research AI trends', {
272
+ abortSignal: controller.signal,
273
+ })
274
+
275
+ // Cancel the supervisor and any in-flight subagents
276
+ controller.abort()
277
+ ```
278
+
264
279
  ## Task completion scoring
265
280
 
266
281
  Agents don't always produce a complete, correct output on the first try. Task completion scorers can help with that by validating whether the task is complete after each iteration. If validation fails, the supervisor continues iterating. Feedback from failed scorers is included in the conversation context so subagents can see what was missing.
@@ -45,4 +45,20 @@ These scorers evaluate adherence to format, style, and safety requirements:
45
45
  - [`tone-consistency`](https://mastra.ai/reference/evals/tone-consistency): Measures consistency in formality, complexity, and style (`0-1`, higher is better)
46
46
  - [`toxicity`](https://mastra.ai/reference/evals/toxicity): Detects harmful or inappropriate content (`0-1`, lower is better)
47
47
  - [`bias`](https://mastra.ai/reference/evals/bias): Detects potential biases in the output (`0-1`, lower is better)
48
- - [`keyword-coverage`](https://mastra.ai/reference/evals/keyword-coverage): Assesses technical terminology usage (`0-1`, higher is better)
48
+ - [`keyword-coverage`](https://mastra.ai/reference/evals/keyword-coverage): Assesses technical terminology usage (`0-1`, higher is better)
49
+
50
+ ### Quick Checks
51
+
52
+ Zero-LLM micro-scorers for fast, deterministic assertions. See the [Quick Checks](https://mastra.ai/docs/evals/quick-checks) guide for usage details.
53
+
54
+ - [`checks.includes`](https://mastra.ai/reference/evals/checks): Scores 1 if output contains a substring (`0` or `1`)
55
+ - [`checks.excludes`](https://mastra.ai/reference/evals/checks): Scores 1 if output does not contain a substring (`0` or `1`)
56
+ - [`checks.equals`](https://mastra.ai/reference/evals/checks): Scores 1 if output exactly matches a string (`0` or `1`)
57
+ - [`checks.matches`](https://mastra.ai/reference/evals/checks): Scores 1 if output matches a regex (`0` or `1`)
58
+ - [`checks.similarity`](https://mastra.ai/reference/evals/checks): String similarity via Dice coefficient (`0-1`, or binary with threshold)
59
+ - [`checks.calledTool`](https://mastra.ai/reference/evals/checks): Scores 1 if a tool was called at least N times (`0` or `1`)
60
+ - [`checks.didNotCall`](https://mastra.ai/reference/evals/checks): Scores 1 if a tool was not called (`0` or `1`)
61
+ - [`checks.toolOrder`](https://mastra.ai/reference/evals/checks): Scores 1 if tools were called in order (`0` or `1`)
62
+ - [`checks.maxToolCalls`](https://mastra.ai/reference/evals/checks): Scores 1 if tool call count is within limit (`0` or `1`)
63
+ - [`checks.usedNoTools`](https://mastra.ai/reference/evals/checks): Scores 1 if no tools were called (`0` or `1`)
64
+ - [`checks.noToolErrors`](https://mastra.ai/reference/evals/checks): Scores 1 if no tool calls had errors (`0` or `1`)
@@ -0,0 +1,140 @@
1
+ # Gates and verdicts
2
+
3
+ Gates and verdicts add severity semantics to `runEvals`. Gates are scorers that must score 1.0 — hard requirements that block a run. Thresholds are minimum acceptable scores on tracked metrics. The verdict summarizes the outcome as `passed`, `scored`, or `failed`.
4
+
5
+ ## When to use gates and verdicts
6
+
7
+ - Enforce hard requirements in CI (e.g., "agent must call the right tool")
8
+ - Track quality metrics with minimum thresholds (e.g., "faithfulness above 0.7")
9
+ - Get a single verdict signal (`passed`, `scored`, or `failed`) from an eval run without writing custom assertion logic
10
+ - Separate "must pass" gates from "nice to have" tracked metrics
11
+
12
+ ## Quickstart
13
+
14
+ ```typescript
15
+ import { runEvals } from '@mastra/core/evals'
16
+ import { checks } from '@mastra/evals/checks'
17
+ import { weatherAgent } from '../agents'
18
+ import { faithfulnessScorer } from '../scorers'
19
+
20
+ const result = await runEvals({
21
+ data: [{ input: 'What is the weather in Brooklyn?' }],
22
+ target: weatherAgent,
23
+
24
+ // Gates: must all score 1.0 or the run fails
25
+ gates: [checks.calledTool('get_weather'), checks.noToolErrors()],
26
+
27
+ // Scorers: tracked with optional thresholds
28
+ scorers: [
29
+ { scorer: faithfulnessScorer, threshold: 0.7 },
30
+ checks.includes('Brooklyn'), // no threshold = tracked only
31
+ ],
32
+ })
33
+
34
+ console.log(result.verdict) // 'passed' | 'scored' | 'failed'
35
+ ```
36
+
37
+ ## How verdicts work
38
+
39
+ The verdict is computed from gates and thresholds after all data items are processed:
40
+
41
+ - `failed`: At least one gate averaged below 1.0 across data items
42
+ - `scored`: All gates passed, but at least one threshold scorer missed its threshold
43
+ - `passed`: All gates scored 1.0 and all thresholds were met
44
+
45
+ When no gates or threshold-bearing scorers are provided, the verdict field is omitted and `runEvals` behaves exactly as before.
46
+
47
+ ## Gates
48
+
49
+ Gates are scorers passed via the `gates` field. They run before regular scorers on each data item. A gate must average a score of 1.0 across all data items to pass.
50
+
51
+ ```typescript
52
+ import { runEvals } from '@mastra/core/evals'
53
+ import { checks } from '@mastra/evals/checks'
54
+
55
+ const result = await runEvals({
56
+ data: [{ input: 'What is the weather?' }],
57
+ target: weatherAgent,
58
+ gates: [checks.calledTool('get_weather')],
59
+ scorers: [qualityScorer],
60
+ })
61
+
62
+ // result.gateResults: [{ id: 'check-called-tool', passed: true, score: 1 }]
63
+ ```
64
+
65
+ Any scorer works as a gate. Quick Checks are a natural fit because they return binary 1/0 scores.
66
+
67
+ > **Note:** Visit [runEvals() reference](https://mastra.ai/reference/evals/run-evals) for the full parameter and return type documentation.
68
+
69
+ ## Thresholds
70
+
71
+ Wrap a scorer in `{ scorer, threshold }` to set pass/fail bounds. The threshold is compared against the scorer's average score across all data items.
72
+
73
+ A `threshold` can be:
74
+
75
+ - **A number** — implies minimum (score at or above passes): `{ scorer, threshold: 0.7 }`
76
+ - **An object with `min` and/or `max`** — for range-based checks: `{ scorer, threshold: { max: 0.3 } }`
77
+
78
+ Use `max` for scorers where a high score is bad (e.g., hallucination, toxicity). Use `{ min, max }` when the score should fall within a specific band.
79
+
80
+ ```typescript
81
+ import { runEvals } from '@mastra/core/evals'
82
+
83
+ const result = await runEvals({
84
+ data: [{ input: 'Explain quantum computing' }],
85
+ target: myAgent,
86
+ scorers: [
87
+ { scorer: faithfulnessScorer, threshold: 0.7 }, // min threshold (number shorthand)
88
+ { scorer: hallucinationScorer, threshold: { max: 0.3 } }, // max threshold — high score = bad
89
+ { scorer: verbosityScorer, threshold: { min: 0.3, max: 0.8 } }, // range threshold
90
+ toneScorer, // bare scorer, no threshold — tracked only
91
+ ],
92
+ })
93
+
94
+ // result.thresholdResults:
95
+ // [
96
+ // { id: 'faithfulness', passed: true, averageScore: 0.85, threshold: 0.7 },
97
+ // { id: 'hallucination', passed: true, averageScore: 0.1, threshold: { max: 0.3 } },
98
+ // { id: 'verbosity', passed: false, averageScore: 0.9, threshold: { min: 0.3, max: 0.8 } },
99
+ // ]
100
+ ```
101
+
102
+ A bare scorer (no threshold) still appears in `result.scores` but does not affect the verdict.
103
+
104
+ ## Using verdicts in CI
105
+
106
+ The verdict gives a single signal for CI pipelines:
107
+
108
+ ```typescript
109
+ import { runEvals } from '@mastra/core/evals'
110
+ import { checks } from '@mastra/evals/checks'
111
+
112
+ const result = await runEvals({
113
+ data: testDataset,
114
+ target: myAgent,
115
+ gates: [checks.calledTool('search'), checks.noToolErrors()],
116
+ scorers: [{ scorer: faithfulnessScorer, threshold: 0.7 }],
117
+ })
118
+
119
+ if (result.verdict === 'failed') {
120
+ console.error(
121
+ 'Gate failures:',
122
+ result.gateResults?.filter(g => !g.passed),
123
+ )
124
+ process.exit(1)
125
+ }
126
+
127
+ if (result.verdict === 'scored') {
128
+ console.warn(
129
+ 'Threshold misses:',
130
+ result.thresholdResults?.filter(t => !t.passed),
131
+ )
132
+ }
133
+ ```
134
+
135
+ ## Related
136
+
137
+ - [Quick Checks](https://mastra.ai/docs/evals/quick-checks): Zero-LLM micro-scorers that work well as gates
138
+ - [runEvals() reference](https://mastra.ai/reference/evals/run-evals): Full API documentation
139
+ - [Built-in scorers](https://mastra.ai/docs/evals/built-in-scorers): LLM-based and code-based scorers
140
+ - [Running evals in CI](https://mastra.ai/docs/evals/running-in-ci): CI integration patterns
@@ -135,6 +135,8 @@ Once registered, you can score traces interactively within Studio under the **Ob
135
135
 
136
136
  ## Next steps
137
137
 
138
+ - Use [Quick Checks](https://mastra.ai/docs/evals/quick-checks) for fast, deterministic assertions on text and tool usage
139
+ - Add [gates and verdicts](https://mastra.ai/docs/evals/gates-and-verdicts) to enforce hard requirements and quality thresholds
138
140
  - Learn how to create your own scorers in the [Creating Custom Scorers](https://mastra.ai/docs/evals/custom-scorers) guide
139
141
  - Explore built-in scorers in the [Built-in Scorers](https://mastra.ai/docs/evals/built-in-scorers) section
140
142
  - Test scorers with [Studio](https://mastra.ai/docs/studio/overview)
@@ -0,0 +1,136 @@
1
+ # Quick Checks
2
+
3
+ Quick Checks are composable micro-scorers for common assertions like "output contains X" or "agent called tool Y." They require no LLM, run instantly, and plug into the same `scorers: [...]` array as any other scorer.
4
+
5
+ ## When to use Quick Checks
6
+
7
+ Use Quick Checks when you need fast, deterministic assertions:
8
+
9
+ - Verify output text contains or excludes specific strings
10
+ - Confirm an agent called (or avoided) specific tools
11
+ - Validate tool call ordering and count limits
12
+ - Gate CI pipelines with zero-cost binary checks
13
+ - Combine with LLM-based scorers for layered evaluation
14
+
15
+ For subjective or semantic evaluation, use [LLM-based scorers](https://mastra.ai/docs/evals/built-in-scorers) instead.
16
+
17
+ ## Quickstart
18
+
19
+ ```typescript
20
+ import { checks } from '@mastra/evals/checks'
21
+ import { runEvals } from '@mastra/core/evals'
22
+ import { weatherAgent } from '../agents'
23
+
24
+ const result = await runEvals({
25
+ data: [{ input: 'What is the weather in Brooklyn?' }],
26
+ target: weatherAgent,
27
+ scorers: [checks.includes('Brooklyn'), checks.calledTool('get_weather'), checks.noToolErrors()],
28
+ })
29
+
30
+ console.log(result.scores)
31
+ // { 'check-includes': 1, 'check-called-tool': 1, 'check-no-tool-errors': 1 }
32
+ ```
33
+
34
+ ## Available checks
35
+
36
+ Quick Checks fall into two categories:
37
+
38
+ ### Text checks
39
+
40
+ These scorers evaluate the agent's text output:
41
+
42
+ | Check | What it does | Score |
43
+ | ------------------------ | ------------------------------------- | -------------------------------- |
44
+ | `checks.includes(str)` | Output contains substring | 1 or 0 |
45
+ | `checks.excludes(str)` | Output does not contain substring | 1 or 0 |
46
+ | `checks.equals(str)` | Output exactly equals string | 1 or 0 |
47
+ | `checks.matches(regex)` | Output matches regular expression | 1 or 0 |
48
+ | `checks.similarity(str)` | Dice coefficient similarity to string | 0-1 (or binary with `threshold`) |
49
+
50
+ ### Tool call checks
51
+
52
+ These scorers evaluate tool usage from the agent's run:
53
+
54
+ | Check | What it does | Score |
55
+ | ------------------------- | -------------------------------- | ------ |
56
+ | `checks.calledTool(name)` | Tool was called at least N times | 1 or 0 |
57
+ | `checks.didNotCall(name)` | Tool was not called | 1 or 0 |
58
+ | `checks.toolOrder([...])` | Tools called in expected order | 1 or 0 |
59
+ | `checks.maxToolCalls(n)` | No more than N tool calls total | 1 or 0 |
60
+ | `checks.usedNoTools()` | No tools called at all | 1 or 0 |
61
+ | `checks.noToolErrors()` | No tool invocations had errors | 1 or 0 |
62
+
63
+ ## Combining checks with LLM scorers
64
+
65
+ Checks compose with LLM-based scorers in a single `runEvals` call. Use checks for deterministic gates and LLM scorers for qualitative evaluation:
66
+
67
+ ```typescript
68
+ import { checks } from '@mastra/evals/checks'
69
+ import { createFaithfulnessScorer } from '@mastra/evals/scorers/prebuilt'
70
+ import { runEvals } from '@mastra/core/evals'
71
+ import { myAgent } from '../agents'
72
+
73
+ const result = await runEvals({
74
+ data: [
75
+ {
76
+ input: 'What is the weather in Brooklyn?',
77
+ context: ['Brooklyn weather data: sunny, 72°F'],
78
+ },
79
+ ],
80
+ target: myAgent,
81
+ scorers: [
82
+ // Deterministic checks (instant, free)
83
+ checks.includes('Brooklyn'),
84
+ checks.calledTool('get_weather'),
85
+ checks.excludes('error'),
86
+ checks.noToolErrors(),
87
+
88
+ // LLM-based scorer (semantic, costs tokens)
89
+ createFaithfulnessScorer({ model: 'openai/gpt-5-mini' }),
90
+ ],
91
+ })
92
+ ```
93
+
94
+ ## Using checks in live scoring
95
+
96
+ Attach checks to agents for continuous monitoring:
97
+
98
+ ```typescript
99
+ import { Agent } from '@mastra/core/agent'
100
+ import { checks } from '@mastra/evals/checks'
101
+
102
+ export const weatherAgent = new Agent({
103
+ name: 'Weather Agent',
104
+ instructions: 'Answer weather questions using the get_weather tool.',
105
+ model: 'openai/gpt-5.5',
106
+ tools: { get_weather: weatherTool },
107
+ scorers: {
108
+ noErrors: {
109
+ scorer: checks.noToolErrors(),
110
+ sampling: { type: 'ratio', rate: 1 },
111
+ },
112
+ mentionCity: {
113
+ scorer: checks.includes('Brooklyn'),
114
+ sampling: { type: 'ratio', rate: 0.5 },
115
+ },
116
+ },
117
+ })
118
+ ```
119
+
120
+ ## How checks work
121
+
122
+ Each check is a standard `createScorer()` instance with a `preprocess` step and a `generateScore` step. They follow the same [four-step pipeline](https://mastra.ai/docs/evals/custom-scorers) as any other scorer:
123
+
124
+ 1. **preprocess**: Extracts and normalizes relevant data from the agent run (text content, tool calls)
125
+ 2. **generateScore**: Converts the preprocessed result into a score (typically binary 1 or 0)
126
+
127
+ Because checks skip the `analyze` and `generateReason` steps and make no LLM calls, they run in microseconds.
128
+
129
+ > **Note:** Visit the [Quick Checks reference](https://mastra.ai/reference/evals/checks) for the full API, including all parameters and options for each check.
130
+
131
+ ## Related
132
+
133
+ - [Quick Checks reference](https://mastra.ai/reference/evals/checks)
134
+ - [Built-in Scorers](https://mastra.ai/docs/evals/built-in-scorers)
135
+ - [Custom Scorers](https://mastra.ai/docs/evals/custom-scorers)
136
+ - [`runEvals()` reference](https://mastra.ai/reference/evals/run-evals)
@@ -79,6 +79,27 @@ const buildMode = { id: 'build', additionalTools: { deployTool } }
79
79
 
80
80
  You can't set both `tools` and `additionalTools` on the same mode.
81
81
 
82
+ ### Restricting tool visibility
83
+
84
+ `tools` and `additionalTools` control which tools are **added** to a mode's run — they don't hide the backing agent's own tools. To restrict which of those tools the model can actually see and call, set `availableTools`:
85
+
86
+ ```typescript
87
+ const reviewMode = {
88
+ id: 'review',
89
+ name: 'Review',
90
+ // Only these tools are visible to the model in this mode.
91
+ availableTools: ['view', 'find_files', 'search_content'],
92
+ }
93
+ ```
94
+
95
+ `availableTools` is a per-mode visibility allowlist that matches each tool by its final exposed name:
96
+
97
+ - **`undefined`** (default): no mode-level restriction — every tool is visible.
98
+ - **`[]`**: no tools are available for this mode.
99
+ - A denied tool stays hidden even if it appears in the list. Per-tool and per-category `deny` rules in your permission config always take precedence.
100
+
101
+ Workspace tools use the same list as every other tool — reference them by their exposed names (`view`, `write_file`, `find_files`, etc.). Visibility is enforced at LLM-call time, so the model never sees — and can't attempt to call — a tool outside the allowlist.
102
+
82
103
  ### Mode transitions
83
104
 
84
105
  A mode can declare a `transitionsTo` target. When the `submit_plan` built-in tool runs in that mode, the Harness transitions to the target mode on approval:
@@ -1,6 +1,6 @@
1
1
  # Harness overview
2
2
 
3
- > **Alpha:** The `Harness` feature is in alpha stage and subject to breaking changes in minor versions until it graduates from its alpha status.
3
+ > **Beta:** The `Harness` feature is in beta stage and subject to breaking changes in minor versions until it graduates from its beta status.
4
4
 
5
5
  The Harness is a session controller for building interactive agent applications. It handles the runtime concerns that sit between your UI and the agent loop: managing conversation threads, switching between agent modes, persisting state, gating tool execution with approvals, and coordinating subagents. You can focus on what your agent does rather than how to wire it together.
6
6
 
@@ -10,7 +10,7 @@ Because each session has its own identity and state, one Harness can serve many
10
10
 
11
11
  The rest of this page walks through the Session's state. Each part is a focused sub-object on `harness.session`:
12
12
 
13
- - **Who and where**: [Identity](#identity) sets the resource the session belongs to, and [Thread](#thread) tracks the currently bound thread.
13
+ - **Who and where**: [Identity](#identity) sets the stable session `id`, `ownerId`, and resource the session belongs to, and [Thread](#thread) tracks the currently bound thread.
14
14
  - **How the agent behaves**: [Mode and model](#mode-and-model) controls which agent profile and model are active, and [Permission grants](#permission-grants) records what the user has approved to run without prompting.
15
15
  - **What's happening right now**: [Run state](#run-state) reflects the in-flight generation, and [Follow-ups](#follow-ups) holds messages queued to send when it finishes.
16
16
  - **What you store and render**: [State](#state) holds your application's structured data, and [Display state](#display-state) is the single snapshot your UI renders from.
@@ -19,14 +19,18 @@ The Harness performs actions that change this state — switching threads, modes
19
19
 
20
20
  ## Identity
21
21
 
22
- `session.identity` holds the resource ID the conversation belongs to and the currently bound thread ID. Threads are scoped to a resource ID, so this is what groups a conversation's threads together:
22
+ `session.identity` holds three stable identifiers for the conversation: a session `id`, an `ownerId`, and the resource ID. Threads are scoped to a resource ID, so this is what groups a conversation's threads together:
23
23
 
24
24
  ```typescript
25
+ const sessionId = harness.session.identity.getId()
26
+ const ownerId = harness.session.identity.getOwnerId()
25
27
  const resourceId = harness.session.identity.getResourceId()
26
28
  const threadId = harness.session.thread.getId()
27
29
  ```
28
30
 
29
- The resource ID is set on the Harness constructor and defaults to the harness `id`. See [Resource IDs](https://mastra.ai/docs/harness/threads-and-state) for how it scopes threads.
31
+ The session `id` and `ownerId` are stable for the life of the session they don't change when the resource ID is switched. They mirror the `id` and `ownerId` fields on `SessionRecord` in storage, so storage layers can key sessions by a stable identifier rather than the mutable resource ID.
32
+
33
+ The resource ID is set when creating a session via [`harness.createSession()`](https://mastra.ai/reference/harness/harness-class) and defaults to the harness `id`. See [Resource IDs](https://mastra.ai/docs/harness/threads-and-state) for how it scopes threads.
30
34
 
31
35
  ## Thread
32
36
 
@@ -131,6 +131,8 @@ The resource ID is part of a conversation's identity, so you read it from the Se
131
131
  const resourceId = harness.session.identity.getResourceId()
132
132
  ```
133
133
 
134
+ The session also has a stable `id` and `ownerId` (read with `session.identity.getId()` and `session.identity.getOwnerId()`). Unlike the resource ID, these don't change when you switch resources — see [Session identity](https://mastra.ai/docs/harness/session) for details.
135
+
134
136
  ## Related
135
137
 
136
138
  - [Harness overview](https://mastra.ai/docs/harness/overview)
@@ -210,9 +210,7 @@ The result is a three-tier system:
210
210
  2. **Observations**: A log of what the Observer has seen
211
211
  3. **Reflections**: Condensed observations when memory becomes too long
212
212
 
213
- ### Retrieval mode (experimental)
214
-
215
- > **Note:** Retrieval mode is experimental. The API may change in future releases.
213
+ ### Retrieval mode
216
214
 
217
215
  Normal OM compresses messages into observations, which is great for staying on task, but the original wording is gone. Retrieval mode fixes this by keeping each observation group linked to the raw messages that produced it. When the agent needs exact wording, tool output, or chronology that the summary compressed away, it can call a `recall` tool to page through the source messages.
218
216
 
@@ -4,7 +4,7 @@ In this guide, you'll build a signal provider that polls an external service on
4
4
 
5
5
  The example watches build pipelines in a fake CI service, but the pattern applies to any pull-based source: an issue tracker, a status API, a queue, or your own backend.
6
6
 
7
- > **Alpha:** Signal providers are in alpha. Breaking changes may occur without a major version bump until the API is stable.
7
+ > **Beta:** Signal providers are in beta. Breaking changes may occur without a major version bump until the API is stable.
8
8
 
9
9
  ## Prerequisites
10
10
 
@@ -1,6 +1,6 @@
1
1
  # Netlify
2
2
 
3
- Netlify AI Gateway provides unified access to multiple providers with built-in caching and observability. Access 66 models through Mastra's model router.
3
+ Netlify AI Gateway provides unified access to multiple providers with built-in caching and observability. Access 64 models through Mastra's model router.
4
4
 
5
5
  Learn more in the [Netlify documentation](https://docs.netlify.com/build/ai-gateway/overview/).
6
6
 
@@ -52,9 +52,7 @@ ANTHROPIC_API_KEY=ant-...
52
52
  | `gemini/gemini-2.5-pro` |
53
53
  | `gemini/gemini-3-flash-preview` |
54
54
  | `gemini/gemini-3-pro-image` |
55
- | `gemini/gemini-3-pro-image-preview` |
56
55
  | `gemini/gemini-3.1-flash-image` |
57
- | `gemini/gemini-3.1-flash-image-preview` |
58
56
  | `gemini/gemini-3.1-flash-lite` |
59
57
  | `gemini/gemini-3.1-pro-preview` |
60
58
  | `gemini/gemini-3.1-pro-preview-customtools` |
@@ -1,6 +1,6 @@
1
1
  # ![OpenRouter logo](https://models.dev/logos/openrouter.svg)OpenRouter
2
2
 
3
- OpenRouter aggregates models from multiple providers with enhanced features like rate limiting and failover. Access 336 models through Mastra's model router.
3
+ OpenRouter aggregates models from multiple providers with enhanced features like rate limiting and failover. Access 337 models through Mastra's model router.
4
4
 
5
5
  Learn more in the [OpenRouter documentation](https://openrouter.ai/models).
6
6
 
@@ -336,6 +336,7 @@ ANTHROPIC_API_KEY=ant-...
336
336
  | `rekaai/reka-flash-3` |
337
337
  | `relace/relace-apply-3` |
338
338
  | `relace/relace-search` |
339
+ | `sakana/fugu-ultra` |
339
340
  | `sao10k/l3-lunaris-8b` |
340
341
  | `sao10k/l3.1-70b-hanami-x1` |
341
342
  | `sao10k/l3.1-euryale-70b` |
@@ -1,6 +1,6 @@
1
1
  # ![Vercel logo](https://models.dev/logos/vercel.svg)Vercel
2
2
 
3
- Vercel aggregates models from multiple providers with enhanced features like rate limiting and failover. Access 290 models through Mastra's model router.
3
+ Vercel aggregates models from multiple providers with enhanced features like rate limiting and failover. Access 292 models through Mastra's model router.
4
4
 
5
5
  Learn more in the [Vercel documentation](https://ai-sdk.dev/providers/ai-sdk-providers).
6
6
 
@@ -302,6 +302,7 @@ ANTHROPIC_API_KEY=ant-...
302
302
  | `xai/grok-build-0.1` |
303
303
  | `xai/grok-imagine-image` |
304
304
  | `xai/grok-imagine-video` |
305
+ | `xai/grok-imagine-video-1.5` |
305
306
  | `xai/grok-imagine-video-1.5-preview` |
306
307
  | `xai/grok-stt` |
307
308
  | `xai/grok-tts` |
@@ -323,4 +324,5 @@ ANTHROPIC_API_KEY=ant-...
323
324
  | `zai/glm-5-turbo` |
324
325
  | `zai/glm-5.1` |
325
326
  | `zai/glm-5.2` |
327
+ | `zai/glm-5.2-fast` |
326
328
  | `zai/glm-5v-turbo` |
@@ -1,6 +1,6 @@
1
1
  # Model Providers
2
2
 
3
- Mastra provides a unified interface for working with LLMs across multiple providers, giving you access to 4550 models from 133 providers through a single API.
3
+ Mastra provides a unified interface for working with LLMs across multiple providers, giving you access to 4544 models from 133 providers through a single API.
4
4
 
5
5
  ## Features
6
6
 
@@ -34,7 +34,7 @@ for await (const chunk of stream) {
34
34
 
35
35
  | Model | Context | Tools | Reasoning | Image | Audio | Video | Input $/1M | Output $/1M |
36
36
  | -------------------------------------------------- | ------- | ----- | --------- | ----- | ----- | ----- | ---------- | ----------- |
37
- | `baseten/deepseek-ai/DeepSeek-V4-Pro` | 131K | | | | | | $2 | $3 |
37
+ | `baseten/deepseek-ai/DeepSeek-V4-Pro` | 262K | | | | | | $2 | $3 |
38
38
  | `baseten/moonshotai/Kimi-K2.5` | 262K | | | | | | $0.60 | $3 |
39
39
  | `baseten/moonshotai/Kimi-K2.6` | 262K | | | | | | $0.95 | $4 |
40
40
  | `baseten/moonshotai/Kimi-K2.7-Code` | 262K | | | | | | $0.95 | $4 |
@@ -44,7 +44,7 @@ for await (const chunk of stream) {
44
44
  | `baseten/zai-org/GLM-4.7` | 200K | | | | | | $0.60 | $2 |
45
45
  | `baseten/zai-org/GLM-5` | 203K | | | | | | $0.95 | $3 |
46
46
  | `baseten/zai-org/GLM-5.1` | 203K | | | | | | $1 | $4 |
47
- | `baseten/zai-org/GLM-5.2` | 262K | | | | | | $1 | $4 |
47
+ | `baseten/zai-org/GLM-5.2` | 203K | | | | | | $1 | $4 |
48
48
 
49
49
  ## Advanced configuration
50
50