@mastra/mcp-docs-server 1.2.2-alpha.1 → 1.2.2-alpha.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.docs/docs/agents/code-mode.md +1 -1
- package/.docs/docs/agents/goals.md +1 -1
- package/.docs/docs/agents/processors.md +1 -1
- package/.docs/docs/agents/signal-providers.md +1 -1
- package/.docs/docs/agents/signals.md +1 -1
- package/.docs/docs/agents/supervisor-agents.md +15 -0
- package/.docs/docs/evals/built-in-scorers.md +17 -1
- package/.docs/docs/evals/gates-and-verdicts.md +140 -0
- package/.docs/docs/evals/overview.md +2 -0
- package/.docs/docs/evals/quick-checks.md +136 -0
- package/.docs/docs/harness/overview.md +1 -1
- package/.docs/docs/harness/session.md +7 -3
- package/.docs/docs/harness/threads-and-state.md +2 -0
- package/.docs/docs/memory/observational-memory.md +1 -3
- package/.docs/guides/guide/signal-provider.md +1 -1
- package/.docs/models/gateways/netlify.md +1 -3
- package/.docs/models/index.md +1 -1
- package/.docs/models/providers/baseten.md +1 -1
- package/.docs/models/providers/huggingface.md +4 -1
- package/.docs/models/providers/llmgateway.md +184 -199
- package/.docs/reference/agents/channels.md +4 -0
- package/.docs/reference/agents/generate.md +1 -1
- package/.docs/reference/evals/checks.md +210 -0
- package/.docs/reference/evals/run-evals.md +71 -1
- package/.docs/reference/harness/harness-class.md +28 -1
- package/.docs/reference/harness/session.md +20 -4
- package/.docs/reference/index.md +1 -0
- package/.docs/reference/memory/observational-memory.md +1 -1
- package/.docs/reference/memory/serialized-memory-config.md +1 -1
- package/.docs/reference/streaming/agents/stream.md +1 -1
- package/CHANGELOG.md +14 -0
- package/package.json +4 -4
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
**Added in:** `@mastra/core@1.38.0`
|
|
4
4
|
|
|
5
|
-
> **
|
|
5
|
+
> **Beta:** This feature is in beta. Breaking changes may occur without a major version bump until the API is stable.
|
|
6
6
|
|
|
7
7
|
Code mode lets an agent run multi-tool computations in an isolated sandbox and return the result as a single, more accurate response. Instead of calling tools one turn at a time, the model writes a tailored function for the user query that orchestrates your existing tools as `external_*` functions, reduces or aggregates their results, and returns one structured answer.
|
|
8
8
|
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
**Added in:** `@mastra/core@1.42.0`
|
|
4
4
|
|
|
5
|
-
> **
|
|
5
|
+
> **Beta:** The Goals feature is in beta stage and subject to breaking changes in minor versions until it graduates from its beta status.
|
|
6
6
|
|
|
7
7
|
A goal is a durable, thread-scoped objective: a standing instruction the agent keeps working toward across loop iterations until a judge model decides it is satisfied or a run budget is exhausted. The objective is persisted in thread state, so it survives reloads and is evaluated in-loop — even when a new message arrives in the middle of an already-running turn.
|
|
8
8
|
|
|
@@ -427,7 +427,7 @@ See the [`ProviderHistoryCompat` reference](https://mastra.ai/reference/processo
|
|
|
427
427
|
|
|
428
428
|
## Response caching
|
|
429
429
|
|
|
430
|
-
> **
|
|
430
|
+
> **Beta:** This feature is in beta. Breaking changes may occur without a major version bump until the API is stable.
|
|
431
431
|
|
|
432
432
|
Response caching skips the LLM call and replays a previously cached response when an agent receives an identical request. Use it to reduce latency and avoid paying for repeated calls.
|
|
433
433
|
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
**Added in:** `@mastra/core@1.39.0`
|
|
4
4
|
|
|
5
|
-
> **
|
|
5
|
+
> **Beta:** This feature is in beta. Breaking changes may occur without a major version bump until the API is stable.
|
|
6
6
|
|
|
7
7
|
A signal provider monitors an external source, such as GitHub, Slack, continuous integration (CI), or your own API, and pushes [notification signals](https://mastra.ai/docs/agents/signals) into subscribed agent threads.
|
|
8
8
|
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
**Added in:** `@mastra/core@1.39.0`
|
|
4
4
|
|
|
5
|
-
> **
|
|
5
|
+
> **Beta:** This feature is in beta. Breaking changes may occur without a major version bump until the API is stable.
|
|
6
6
|
|
|
7
7
|
Signals are a way to interact with an agent through a thread. Instead of starting every interaction with `agent.stream()`, subscribe to a thread and send messages or signals. Mastra either wakes the agent when the thread is idle, drops input into the running agent loop, or queues input for the next turn.
|
|
8
8
|
|
|
@@ -261,6 +261,21 @@ for await (const chunk of stream.fullStream) {
|
|
|
261
261
|
}
|
|
262
262
|
```
|
|
263
263
|
|
|
264
|
+
## Cancellation
|
|
265
|
+
|
|
266
|
+
When you pass an `abortSignal` to the supervisor's [`stream()`](https://mastra.ai/reference/streaming/agents/stream) or [`generate()`](https://mastra.ai/reference/agents/generate) call, Mastra forwards that same signal to delegated subagents. Calling `AbortController.abort()` cancels in-flight subagent runs at their next step instead of letting them run to completion.
|
|
267
|
+
|
|
268
|
+
```typescript
|
|
269
|
+
const controller = new AbortController()
|
|
270
|
+
|
|
271
|
+
const stream = await supervisor.stream('Research AI trends', {
|
|
272
|
+
abortSignal: controller.signal,
|
|
273
|
+
})
|
|
274
|
+
|
|
275
|
+
// Cancel the supervisor and any in-flight subagents
|
|
276
|
+
controller.abort()
|
|
277
|
+
```
|
|
278
|
+
|
|
264
279
|
## Task completion scoring
|
|
265
280
|
|
|
266
281
|
Agents don't always produce a complete, correct output on the first try. Task completion scorers can help with that by validating whether the task is complete after each iteration. If validation fails, the supervisor continues iterating. Feedback from failed scorers is included in the conversation context so subagents can see what was missing.
|
|
@@ -45,4 +45,20 @@ These scorers evaluate adherence to format, style, and safety requirements:
|
|
|
45
45
|
- [`tone-consistency`](https://mastra.ai/reference/evals/tone-consistency): Measures consistency in formality, complexity, and style (`0-1`, higher is better)
|
|
46
46
|
- [`toxicity`](https://mastra.ai/reference/evals/toxicity): Detects harmful or inappropriate content (`0-1`, lower is better)
|
|
47
47
|
- [`bias`](https://mastra.ai/reference/evals/bias): Detects potential biases in the output (`0-1`, lower is better)
|
|
48
|
-
- [`keyword-coverage`](https://mastra.ai/reference/evals/keyword-coverage): Assesses technical terminology usage (`0-1`, higher is better)
|
|
48
|
+
- [`keyword-coverage`](https://mastra.ai/reference/evals/keyword-coverage): Assesses technical terminology usage (`0-1`, higher is better)
|
|
49
|
+
|
|
50
|
+
### Quick Checks
|
|
51
|
+
|
|
52
|
+
Zero-LLM micro-scorers for fast, deterministic assertions. See the [Quick Checks](https://mastra.ai/docs/evals/quick-checks) guide for usage details.
|
|
53
|
+
|
|
54
|
+
- [`checks.includes`](https://mastra.ai/reference/evals/checks): Scores 1 if output contains a substring (`0` or `1`)
|
|
55
|
+
- [`checks.excludes`](https://mastra.ai/reference/evals/checks): Scores 1 if output does not contain a substring (`0` or `1`)
|
|
56
|
+
- [`checks.equals`](https://mastra.ai/reference/evals/checks): Scores 1 if output exactly matches a string (`0` or `1`)
|
|
57
|
+
- [`checks.matches`](https://mastra.ai/reference/evals/checks): Scores 1 if output matches a regex (`0` or `1`)
|
|
58
|
+
- [`checks.similarity`](https://mastra.ai/reference/evals/checks): String similarity via Dice coefficient (`0-1`, or binary with threshold)
|
|
59
|
+
- [`checks.calledTool`](https://mastra.ai/reference/evals/checks): Scores 1 if a tool was called at least N times (`0` or `1`)
|
|
60
|
+
- [`checks.didNotCall`](https://mastra.ai/reference/evals/checks): Scores 1 if a tool was not called (`0` or `1`)
|
|
61
|
+
- [`checks.toolOrder`](https://mastra.ai/reference/evals/checks): Scores 1 if tools were called in order (`0` or `1`)
|
|
62
|
+
- [`checks.maxToolCalls`](https://mastra.ai/reference/evals/checks): Scores 1 if tool call count is within limit (`0` or `1`)
|
|
63
|
+
- [`checks.usedNoTools`](https://mastra.ai/reference/evals/checks): Scores 1 if no tools were called (`0` or `1`)
|
|
64
|
+
- [`checks.noToolErrors`](https://mastra.ai/reference/evals/checks): Scores 1 if no tool calls had errors (`0` or `1`)
|
|
@@ -0,0 +1,140 @@
|
|
|
1
|
+
# Gates and verdicts
|
|
2
|
+
|
|
3
|
+
Gates and verdicts add severity semantics to `runEvals`. Gates are scorers that must score 1.0 — hard requirements that block a run. Thresholds are minimum acceptable scores on tracked metrics. The verdict summarizes the outcome as `passed`, `scored`, or `failed`.
|
|
4
|
+
|
|
5
|
+
## When to use gates and verdicts
|
|
6
|
+
|
|
7
|
+
- Enforce hard requirements in CI (e.g., "agent must call the right tool")
|
|
8
|
+
- Track quality metrics with minimum thresholds (e.g., "faithfulness above 0.7")
|
|
9
|
+
- Get a single verdict signal (`passed`, `scored`, or `failed`) from an eval run without writing custom assertion logic
|
|
10
|
+
- Separate "must pass" gates from "nice to have" tracked metrics
|
|
11
|
+
|
|
12
|
+
## Quickstart
|
|
13
|
+
|
|
14
|
+
```typescript
|
|
15
|
+
import { runEvals } from '@mastra/core/evals'
|
|
16
|
+
import { checks } from '@mastra/evals/checks'
|
|
17
|
+
import { weatherAgent } from '../agents'
|
|
18
|
+
import { faithfulnessScorer } from '../scorers'
|
|
19
|
+
|
|
20
|
+
const result = await runEvals({
|
|
21
|
+
data: [{ input: 'What is the weather in Brooklyn?' }],
|
|
22
|
+
target: weatherAgent,
|
|
23
|
+
|
|
24
|
+
// Gates: must all score 1.0 or the run fails
|
|
25
|
+
gates: [checks.calledTool('get_weather'), checks.noToolErrors()],
|
|
26
|
+
|
|
27
|
+
// Scorers: tracked with optional thresholds
|
|
28
|
+
scorers: [
|
|
29
|
+
{ scorer: faithfulnessScorer, threshold: 0.7 },
|
|
30
|
+
checks.includes('Brooklyn'), // no threshold = tracked only
|
|
31
|
+
],
|
|
32
|
+
})
|
|
33
|
+
|
|
34
|
+
console.log(result.verdict) // 'passed' | 'scored' | 'failed'
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
## How verdicts work
|
|
38
|
+
|
|
39
|
+
The verdict is computed from gates and thresholds after all data items are processed:
|
|
40
|
+
|
|
41
|
+
- `failed`: At least one gate averaged below 1.0 across data items
|
|
42
|
+
- `scored`: All gates passed, but at least one threshold scorer missed its threshold
|
|
43
|
+
- `passed`: All gates scored 1.0 and all thresholds were met
|
|
44
|
+
|
|
45
|
+
When no gates or threshold-bearing scorers are provided, the verdict field is omitted and `runEvals` behaves exactly as before.
|
|
46
|
+
|
|
47
|
+
## Gates
|
|
48
|
+
|
|
49
|
+
Gates are scorers passed via the `gates` field. They run before regular scorers on each data item. A gate must average a score of 1.0 across all data items to pass.
|
|
50
|
+
|
|
51
|
+
```typescript
|
|
52
|
+
import { runEvals } from '@mastra/core/evals'
|
|
53
|
+
import { checks } from '@mastra/evals/checks'
|
|
54
|
+
|
|
55
|
+
const result = await runEvals({
|
|
56
|
+
data: [{ input: 'What is the weather?' }],
|
|
57
|
+
target: weatherAgent,
|
|
58
|
+
gates: [checks.calledTool('get_weather')],
|
|
59
|
+
scorers: [qualityScorer],
|
|
60
|
+
})
|
|
61
|
+
|
|
62
|
+
// result.gateResults: [{ id: 'check-called-tool', passed: true, score: 1 }]
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
Any scorer works as a gate. Quick Checks are a natural fit because they return binary 1/0 scores.
|
|
66
|
+
|
|
67
|
+
> **Note:** Visit [runEvals() reference](https://mastra.ai/reference/evals/run-evals) for the full parameter and return type documentation.
|
|
68
|
+
|
|
69
|
+
## Thresholds
|
|
70
|
+
|
|
71
|
+
Wrap a scorer in `{ scorer, threshold }` to set pass/fail bounds. The threshold is compared against the scorer's average score across all data items.
|
|
72
|
+
|
|
73
|
+
A `threshold` can be:
|
|
74
|
+
|
|
75
|
+
- **A number** — implies minimum (score at or above passes): `{ scorer, threshold: 0.7 }`
|
|
76
|
+
- **An object with `min` and/or `max`** — for range-based checks: `{ scorer, threshold: { max: 0.3 } }`
|
|
77
|
+
|
|
78
|
+
Use `max` for scorers where a high score is bad (e.g., hallucination, toxicity). Use `{ min, max }` when the score should fall within a specific band.
|
|
79
|
+
|
|
80
|
+
```typescript
|
|
81
|
+
import { runEvals } from '@mastra/core/evals'
|
|
82
|
+
|
|
83
|
+
const result = await runEvals({
|
|
84
|
+
data: [{ input: 'Explain quantum computing' }],
|
|
85
|
+
target: myAgent,
|
|
86
|
+
scorers: [
|
|
87
|
+
{ scorer: faithfulnessScorer, threshold: 0.7 }, // min threshold (number shorthand)
|
|
88
|
+
{ scorer: hallucinationScorer, threshold: { max: 0.3 } }, // max threshold — high score = bad
|
|
89
|
+
{ scorer: verbosityScorer, threshold: { min: 0.3, max: 0.8 } }, // range threshold
|
|
90
|
+
toneScorer, // bare scorer, no threshold — tracked only
|
|
91
|
+
],
|
|
92
|
+
})
|
|
93
|
+
|
|
94
|
+
// result.thresholdResults:
|
|
95
|
+
// [
|
|
96
|
+
// { id: 'faithfulness', passed: true, averageScore: 0.85, threshold: 0.7 },
|
|
97
|
+
// { id: 'hallucination', passed: true, averageScore: 0.1, threshold: { max: 0.3 } },
|
|
98
|
+
// { id: 'verbosity', passed: false, averageScore: 0.9, threshold: { min: 0.3, max: 0.8 } },
|
|
99
|
+
// ]
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
A bare scorer (no threshold) still appears in `result.scores` but does not affect the verdict.
|
|
103
|
+
|
|
104
|
+
## Using verdicts in CI
|
|
105
|
+
|
|
106
|
+
The verdict gives a single signal for CI pipelines:
|
|
107
|
+
|
|
108
|
+
```typescript
|
|
109
|
+
import { runEvals } from '@mastra/core/evals'
|
|
110
|
+
import { checks } from '@mastra/evals/checks'
|
|
111
|
+
|
|
112
|
+
const result = await runEvals({
|
|
113
|
+
data: testDataset,
|
|
114
|
+
target: myAgent,
|
|
115
|
+
gates: [checks.calledTool('search'), checks.noToolErrors()],
|
|
116
|
+
scorers: [{ scorer: faithfulnessScorer, threshold: 0.7 }],
|
|
117
|
+
})
|
|
118
|
+
|
|
119
|
+
if (result.verdict === 'failed') {
|
|
120
|
+
console.error(
|
|
121
|
+
'Gate failures:',
|
|
122
|
+
result.gateResults?.filter(g => !g.passed),
|
|
123
|
+
)
|
|
124
|
+
process.exit(1)
|
|
125
|
+
}
|
|
126
|
+
|
|
127
|
+
if (result.verdict === 'scored') {
|
|
128
|
+
console.warn(
|
|
129
|
+
'Threshold misses:',
|
|
130
|
+
result.thresholdResults?.filter(t => !t.passed),
|
|
131
|
+
)
|
|
132
|
+
}
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
## Related
|
|
136
|
+
|
|
137
|
+
- [Quick Checks](https://mastra.ai/docs/evals/quick-checks): Zero-LLM micro-scorers that work well as gates
|
|
138
|
+
- [runEvals() reference](https://mastra.ai/reference/evals/run-evals): Full API documentation
|
|
139
|
+
- [Built-in scorers](https://mastra.ai/docs/evals/built-in-scorers): LLM-based and code-based scorers
|
|
140
|
+
- [Running evals in CI](https://mastra.ai/docs/evals/running-in-ci): CI integration patterns
|
|
@@ -135,6 +135,8 @@ Once registered, you can score traces interactively within Studio under the **Ob
|
|
|
135
135
|
|
|
136
136
|
## Next steps
|
|
137
137
|
|
|
138
|
+
- Use [Quick Checks](https://mastra.ai/docs/evals/quick-checks) for fast, deterministic assertions on text and tool usage
|
|
139
|
+
- Add [gates and verdicts](https://mastra.ai/docs/evals/gates-and-verdicts) to enforce hard requirements and quality thresholds
|
|
138
140
|
- Learn how to create your own scorers in the [Creating Custom Scorers](https://mastra.ai/docs/evals/custom-scorers) guide
|
|
139
141
|
- Explore built-in scorers in the [Built-in Scorers](https://mastra.ai/docs/evals/built-in-scorers) section
|
|
140
142
|
- Test scorers with [Studio](https://mastra.ai/docs/studio/overview)
|
|
@@ -0,0 +1,136 @@
|
|
|
1
|
+
# Quick Checks
|
|
2
|
+
|
|
3
|
+
Quick Checks are composable micro-scorers for common assertions like "output contains X" or "agent called tool Y." They require no LLM, run instantly, and plug into the same `scorers: [...]` array as any other scorer.
|
|
4
|
+
|
|
5
|
+
## When to use Quick Checks
|
|
6
|
+
|
|
7
|
+
Use Quick Checks when you need fast, deterministic assertions:
|
|
8
|
+
|
|
9
|
+
- Verify output text contains or excludes specific strings
|
|
10
|
+
- Confirm an agent called (or avoided) specific tools
|
|
11
|
+
- Validate tool call ordering and count limits
|
|
12
|
+
- Gate CI pipelines with zero-cost binary checks
|
|
13
|
+
- Combine with LLM-based scorers for layered evaluation
|
|
14
|
+
|
|
15
|
+
For subjective or semantic evaluation, use [LLM-based scorers](https://mastra.ai/docs/evals/built-in-scorers) instead.
|
|
16
|
+
|
|
17
|
+
## Quickstart
|
|
18
|
+
|
|
19
|
+
```typescript
|
|
20
|
+
import { checks } from '@mastra/evals/checks'
|
|
21
|
+
import { runEvals } from '@mastra/core/evals'
|
|
22
|
+
import { weatherAgent } from '../agents'
|
|
23
|
+
|
|
24
|
+
const result = await runEvals({
|
|
25
|
+
data: [{ input: 'What is the weather in Brooklyn?' }],
|
|
26
|
+
target: weatherAgent,
|
|
27
|
+
scorers: [checks.includes('Brooklyn'), checks.calledTool('get_weather'), checks.noToolErrors()],
|
|
28
|
+
})
|
|
29
|
+
|
|
30
|
+
console.log(result.scores)
|
|
31
|
+
// { 'check-includes': 1, 'check-called-tool': 1, 'check-no-tool-errors': 1 }
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
## Available checks
|
|
35
|
+
|
|
36
|
+
Quick Checks fall into two categories:
|
|
37
|
+
|
|
38
|
+
### Text checks
|
|
39
|
+
|
|
40
|
+
These scorers evaluate the agent's text output:
|
|
41
|
+
|
|
42
|
+
| Check | What it does | Score |
|
|
43
|
+
| ------------------------ | ------------------------------------- | -------------------------------- |
|
|
44
|
+
| `checks.includes(str)` | Output contains substring | 1 or 0 |
|
|
45
|
+
| `checks.excludes(str)` | Output does not contain substring | 1 or 0 |
|
|
46
|
+
| `checks.equals(str)` | Output exactly equals string | 1 or 0 |
|
|
47
|
+
| `checks.matches(regex)` | Output matches regular expression | 1 or 0 |
|
|
48
|
+
| `checks.similarity(str)` | Dice coefficient similarity to string | 0-1 (or binary with `threshold`) |
|
|
49
|
+
|
|
50
|
+
### Tool call checks
|
|
51
|
+
|
|
52
|
+
These scorers evaluate tool usage from the agent's run:
|
|
53
|
+
|
|
54
|
+
| Check | What it does | Score |
|
|
55
|
+
| ------------------------- | -------------------------------- | ------ |
|
|
56
|
+
| `checks.calledTool(name)` | Tool was called at least N times | 1 or 0 |
|
|
57
|
+
| `checks.didNotCall(name)` | Tool was not called | 1 or 0 |
|
|
58
|
+
| `checks.toolOrder([...])` | Tools called in expected order | 1 or 0 |
|
|
59
|
+
| `checks.maxToolCalls(n)` | No more than N tool calls total | 1 or 0 |
|
|
60
|
+
| `checks.usedNoTools()` | No tools called at all | 1 or 0 |
|
|
61
|
+
| `checks.noToolErrors()` | No tool invocations had errors | 1 or 0 |
|
|
62
|
+
|
|
63
|
+
## Combining checks with LLM scorers
|
|
64
|
+
|
|
65
|
+
Checks compose with LLM-based scorers in a single `runEvals` call. Use checks for deterministic gates and LLM scorers for qualitative evaluation:
|
|
66
|
+
|
|
67
|
+
```typescript
|
|
68
|
+
import { checks } from '@mastra/evals/checks'
|
|
69
|
+
import { createFaithfulnessScorer } from '@mastra/evals/scorers/prebuilt'
|
|
70
|
+
import { runEvals } from '@mastra/core/evals'
|
|
71
|
+
import { myAgent } from '../agents'
|
|
72
|
+
|
|
73
|
+
const result = await runEvals({
|
|
74
|
+
data: [
|
|
75
|
+
{
|
|
76
|
+
input: 'What is the weather in Brooklyn?',
|
|
77
|
+
context: ['Brooklyn weather data: sunny, 72°F'],
|
|
78
|
+
},
|
|
79
|
+
],
|
|
80
|
+
target: myAgent,
|
|
81
|
+
scorers: [
|
|
82
|
+
// Deterministic checks (instant, free)
|
|
83
|
+
checks.includes('Brooklyn'),
|
|
84
|
+
checks.calledTool('get_weather'),
|
|
85
|
+
checks.excludes('error'),
|
|
86
|
+
checks.noToolErrors(),
|
|
87
|
+
|
|
88
|
+
// LLM-based scorer (semantic, costs tokens)
|
|
89
|
+
createFaithfulnessScorer({ model: 'openai/gpt-5-mini' }),
|
|
90
|
+
],
|
|
91
|
+
})
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
## Using checks in live scoring
|
|
95
|
+
|
|
96
|
+
Attach checks to agents for continuous monitoring:
|
|
97
|
+
|
|
98
|
+
```typescript
|
|
99
|
+
import { Agent } from '@mastra/core/agent'
|
|
100
|
+
import { checks } from '@mastra/evals/checks'
|
|
101
|
+
|
|
102
|
+
export const weatherAgent = new Agent({
|
|
103
|
+
name: 'Weather Agent',
|
|
104
|
+
instructions: 'Answer weather questions using the get_weather tool.',
|
|
105
|
+
model: 'openai/gpt-5.5',
|
|
106
|
+
tools: { get_weather: weatherTool },
|
|
107
|
+
scorers: {
|
|
108
|
+
noErrors: {
|
|
109
|
+
scorer: checks.noToolErrors(),
|
|
110
|
+
sampling: { type: 'ratio', rate: 1 },
|
|
111
|
+
},
|
|
112
|
+
mentionCity: {
|
|
113
|
+
scorer: checks.includes('Brooklyn'),
|
|
114
|
+
sampling: { type: 'ratio', rate: 0.5 },
|
|
115
|
+
},
|
|
116
|
+
},
|
|
117
|
+
})
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
## How checks work
|
|
121
|
+
|
|
122
|
+
Each check is a standard `createScorer()` instance with a `preprocess` step and a `generateScore` step. They follow the same [four-step pipeline](https://mastra.ai/docs/evals/custom-scorers) as any other scorer:
|
|
123
|
+
|
|
124
|
+
1. **preprocess**: Extracts and normalizes relevant data from the agent run (text content, tool calls)
|
|
125
|
+
2. **generateScore**: Converts the preprocessed result into a score (typically binary 1 or 0)
|
|
126
|
+
|
|
127
|
+
Because checks skip the `analyze` and `generateReason` steps and make no LLM calls, they run in microseconds.
|
|
128
|
+
|
|
129
|
+
> **Note:** Visit the [Quick Checks reference](https://mastra.ai/reference/evals/checks) for the full API, including all parameters and options for each check.
|
|
130
|
+
|
|
131
|
+
## Related
|
|
132
|
+
|
|
133
|
+
- [Quick Checks reference](https://mastra.ai/reference/evals/checks)
|
|
134
|
+
- [Built-in Scorers](https://mastra.ai/docs/evals/built-in-scorers)
|
|
135
|
+
- [Custom Scorers](https://mastra.ai/docs/evals/custom-scorers)
|
|
136
|
+
- [`runEvals()` reference](https://mastra.ai/reference/evals/run-evals)
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Harness overview
|
|
2
2
|
|
|
3
|
-
> **
|
|
3
|
+
> **Beta:** The `Harness` feature is in beta stage and subject to breaking changes in minor versions until it graduates from its beta status.
|
|
4
4
|
|
|
5
5
|
The Harness is a session controller for building interactive agent applications. It handles the runtime concerns that sit between your UI and the agent loop: managing conversation threads, switching between agent modes, persisting state, gating tool execution with approvals, and coordinating subagents. You can focus on what your agent does rather than how to wire it together.
|
|
6
6
|
|
|
@@ -10,7 +10,7 @@ Because each session has its own identity and state, one Harness can serve many
|
|
|
10
10
|
|
|
11
11
|
The rest of this page walks through the Session's state. Each part is a focused sub-object on `harness.session`:
|
|
12
12
|
|
|
13
|
-
- **Who and where**: [Identity](#identity) sets the resource the session belongs to, and [Thread](#thread) tracks the currently bound thread.
|
|
13
|
+
- **Who and where**: [Identity](#identity) sets the stable session `id`, `ownerId`, and resource the session belongs to, and [Thread](#thread) tracks the currently bound thread.
|
|
14
14
|
- **How the agent behaves**: [Mode and model](#mode-and-model) controls which agent profile and model are active, and [Permission grants](#permission-grants) records what the user has approved to run without prompting.
|
|
15
15
|
- **What's happening right now**: [Run state](#run-state) reflects the in-flight generation, and [Follow-ups](#follow-ups) holds messages queued to send when it finishes.
|
|
16
16
|
- **What you store and render**: [State](#state) holds your application's structured data, and [Display state](#display-state) is the single snapshot your UI renders from.
|
|
@@ -19,14 +19,18 @@ The Harness performs actions that change this state — switching threads, modes
|
|
|
19
19
|
|
|
20
20
|
## Identity
|
|
21
21
|
|
|
22
|
-
`session.identity` holds
|
|
22
|
+
`session.identity` holds three stable identifiers for the conversation: a session `id`, an `ownerId`, and the resource ID. Threads are scoped to a resource ID, so this is what groups a conversation's threads together:
|
|
23
23
|
|
|
24
24
|
```typescript
|
|
25
|
+
const sessionId = harness.session.identity.getId()
|
|
26
|
+
const ownerId = harness.session.identity.getOwnerId()
|
|
25
27
|
const resourceId = harness.session.identity.getResourceId()
|
|
26
28
|
const threadId = harness.session.thread.getId()
|
|
27
29
|
```
|
|
28
30
|
|
|
29
|
-
The
|
|
31
|
+
The session `id` and `ownerId` are stable for the life of the session — they don't change when the resource ID is switched. They mirror the `id` and `ownerId` fields on `SessionRecord` in storage, so storage layers can key sessions by a stable identifier rather than the mutable resource ID.
|
|
32
|
+
|
|
33
|
+
The resource ID is set when creating a session via [`harness.createSession()`](https://mastra.ai/reference/harness/harness-class) and defaults to the harness `id`. See [Resource IDs](https://mastra.ai/docs/harness/threads-and-state) for how it scopes threads.
|
|
30
34
|
|
|
31
35
|
## Thread
|
|
32
36
|
|
|
@@ -131,6 +131,8 @@ The resource ID is part of a conversation's identity, so you read it from the Se
|
|
|
131
131
|
const resourceId = harness.session.identity.getResourceId()
|
|
132
132
|
```
|
|
133
133
|
|
|
134
|
+
The session also has a stable `id` and `ownerId` (read with `session.identity.getId()` and `session.identity.getOwnerId()`). Unlike the resource ID, these don't change when you switch resources — see [Session identity](https://mastra.ai/docs/harness/session) for details.
|
|
135
|
+
|
|
134
136
|
## Related
|
|
135
137
|
|
|
136
138
|
- [Harness overview](https://mastra.ai/docs/harness/overview)
|
|
@@ -210,9 +210,7 @@ The result is a three-tier system:
|
|
|
210
210
|
2. **Observations**: A log of what the Observer has seen
|
|
211
211
|
3. **Reflections**: Condensed observations when memory becomes too long
|
|
212
212
|
|
|
213
|
-
### Retrieval mode
|
|
214
|
-
|
|
215
|
-
> **Note:** Retrieval mode is experimental. The API may change in future releases.
|
|
213
|
+
### Retrieval mode
|
|
216
214
|
|
|
217
215
|
Normal OM compresses messages into observations, which is great for staying on task, but the original wording is gone. Retrieval mode fixes this by keeping each observation group linked to the raw messages that produced it. When the agent needs exact wording, tool output, or chronology that the summary compressed away, it can call a `recall` tool to page through the source messages.
|
|
218
216
|
|
|
@@ -4,7 +4,7 @@ In this guide, you'll build a signal provider that polls an external service on
|
|
|
4
4
|
|
|
5
5
|
The example watches build pipelines in a fake CI service, but the pattern applies to any pull-based source: an issue tracker, a status API, a queue, or your own backend.
|
|
6
6
|
|
|
7
|
-
> **
|
|
7
|
+
> **Beta:** Signal providers are in beta. Breaking changes may occur without a major version bump until the API is stable.
|
|
8
8
|
|
|
9
9
|
## Prerequisites
|
|
10
10
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Netlify
|
|
2
2
|
|
|
3
|
-
Netlify AI Gateway provides unified access to multiple providers with built-in caching and observability. Access
|
|
3
|
+
Netlify AI Gateway provides unified access to multiple providers with built-in caching and observability. Access 64 models through Mastra's model router.
|
|
4
4
|
|
|
5
5
|
Learn more in the [Netlify documentation](https://docs.netlify.com/build/ai-gateway/overview/).
|
|
6
6
|
|
|
@@ -52,9 +52,7 @@ ANTHROPIC_API_KEY=ant-...
|
|
|
52
52
|
| `gemini/gemini-2.5-pro` |
|
|
53
53
|
| `gemini/gemini-3-flash-preview` |
|
|
54
54
|
| `gemini/gemini-3-pro-image` |
|
|
55
|
-
| `gemini/gemini-3-pro-image-preview` |
|
|
56
55
|
| `gemini/gemini-3.1-flash-image` |
|
|
57
|
-
| `gemini/gemini-3.1-flash-image-preview` |
|
|
58
56
|
| `gemini/gemini-3.1-flash-lite` |
|
|
59
57
|
| `gemini/gemini-3.1-pro-preview` |
|
|
60
58
|
| `gemini/gemini-3.1-pro-preview-customtools` |
|
package/.docs/models/index.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Model Providers
|
|
2
2
|
|
|
3
|
-
Mastra provides a unified interface for working with LLMs across multiple providers, giving you access to
|
|
3
|
+
Mastra provides a unified interface for working with LLMs across multiple providers, giving you access to 4536 models from 133 providers through a single API.
|
|
4
4
|
|
|
5
5
|
## Features
|
|
6
6
|
|
|
@@ -44,7 +44,7 @@ for await (const chunk of stream) {
|
|
|
44
44
|
| `baseten/zai-org/GLM-4.7` | 200K | | | | | | $0.60 | $2 |
|
|
45
45
|
| `baseten/zai-org/GLM-5` | 203K | | | | | | $0.95 | $3 |
|
|
46
46
|
| `baseten/zai-org/GLM-5.1` | 203K | | | | | | $1 | $4 |
|
|
47
|
-
| `baseten/zai-org/GLM-5.2` |
|
|
47
|
+
| `baseten/zai-org/GLM-5.2` | 203K | | | | | | $1 | $4 |
|
|
48
48
|
|
|
49
49
|
## Advanced configuration
|
|
50
50
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Hugging Face
|
|
2
2
|
|
|
3
|
-
Access
|
|
3
|
+
Access 49 Hugging Face models through Mastra's model router. Authentication is handled automatically using the `HF_TOKEN` environment variable.
|
|
4
4
|
|
|
5
5
|
Learn more in the [Hugging Face documentation](https://huggingface.co).
|
|
6
6
|
|
|
@@ -68,9 +68,12 @@ for await (const chunk of stream) {
|
|
|
68
68
|
| `huggingface/Qwen/Qwen3.5-35B-A3B` | 262K | | | | | | $0.25 | $2 |
|
|
69
69
|
| `huggingface/Qwen/Qwen3.5-397B-A17B` | 262K | | | | | | $0.60 | $4 |
|
|
70
70
|
| `huggingface/Qwen/Qwen3.5-9B` | 262K | | | | | | $0.17 | $0.25 |
|
|
71
|
+
| `huggingface/Qwen/Qwen3.6-27B` | 262K | | | | | | $0.47 | $3 |
|
|
71
72
|
| `huggingface/Qwen/Qwen3.6-35B-A3B` | 262K | | | | | | $0.15 | $0.95 |
|
|
72
73
|
| `huggingface/stepfun-ai/Step-3.5-Flash` | 262K | | | | | | $0.10 | $0.30 |
|
|
74
|
+
| `huggingface/stepfun-ai/Step-3.7-Flash` | 262K | | | | | | $0.20 | $1 |
|
|
73
75
|
| `huggingface/XiaomiMiMo/MiMo-V2-Flash` | 262K | | | | | | $0.10 | $0.30 |
|
|
76
|
+
| `huggingface/XiaomiMiMo/MiMo-V2.5-Pro` | 1.0M | | | | | | $1 | $3 |
|
|
74
77
|
| `huggingface/zai-org/GLM-4.5` | 131K | | | | | | $0.60 | $2 |
|
|
75
78
|
| `huggingface/zai-org/GLM-4.5-Air` | 131K | | | | | | $0.13 | $0.85 |
|
|
76
79
|
| `huggingface/zai-org/GLM-4.5V` | 66K | | | | | | $0.60 | $2 |
|