@mastra/mcp-docs-server 1.2.2-alpha.2 → 1.2.2-alpha.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.docs/docs/evals/gates-and-verdicts.md +140 -0
- package/.docs/docs/evals/overview.md +1 -0
- package/.docs/models/gateways/netlify.md +1 -3
- package/.docs/models/index.md +1 -1
- package/.docs/models/providers/baseten.md +1 -1
- package/.docs/models/providers/huggingface.md +4 -1
- package/.docs/models/providers/llmgateway.md +184 -199
- package/.docs/reference/agents/channels.md +4 -0
- package/.docs/reference/evals/run-evals.md +71 -1
- package/CHANGELOG.md +7 -0
- package/package.json +5 -5
|
@@ -0,0 +1,140 @@
|
|
|
1
|
+
# Gates and verdicts
|
|
2
|
+
|
|
3
|
+
Gates and verdicts add severity semantics to `runEvals`. Gates are scorers that must score 1.0 — hard requirements that block a run. Thresholds are minimum acceptable scores on tracked metrics. The verdict summarizes the outcome as `passed`, `scored`, or `failed`.
|
|
4
|
+
|
|
5
|
+
## When to use gates and verdicts
|
|
6
|
+
|
|
7
|
+
- Enforce hard requirements in CI (e.g., "agent must call the right tool")
|
|
8
|
+
- Track quality metrics with minimum thresholds (e.g., "faithfulness above 0.7")
|
|
9
|
+
- Get a single verdict signal (`passed`, `scored`, or `failed`) from an eval run without writing custom assertion logic
|
|
10
|
+
- Separate "must pass" gates from "nice to have" tracked metrics
|
|
11
|
+
|
|
12
|
+
## Quickstart
|
|
13
|
+
|
|
14
|
+
```typescript
|
|
15
|
+
import { runEvals } from '@mastra/core/evals'
|
|
16
|
+
import { checks } from '@mastra/evals/checks'
|
|
17
|
+
import { weatherAgent } from '../agents'
|
|
18
|
+
import { faithfulnessScorer } from '../scorers'
|
|
19
|
+
|
|
20
|
+
const result = await runEvals({
|
|
21
|
+
data: [{ input: 'What is the weather in Brooklyn?' }],
|
|
22
|
+
target: weatherAgent,
|
|
23
|
+
|
|
24
|
+
// Gates: must all score 1.0 or the run fails
|
|
25
|
+
gates: [checks.calledTool('get_weather'), checks.noToolErrors()],
|
|
26
|
+
|
|
27
|
+
// Scorers: tracked with optional thresholds
|
|
28
|
+
scorers: [
|
|
29
|
+
{ scorer: faithfulnessScorer, threshold: 0.7 },
|
|
30
|
+
checks.includes('Brooklyn'), // no threshold = tracked only
|
|
31
|
+
],
|
|
32
|
+
})
|
|
33
|
+
|
|
34
|
+
console.log(result.verdict) // 'passed' | 'scored' | 'failed'
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
## How verdicts work
|
|
38
|
+
|
|
39
|
+
The verdict is computed from gates and thresholds after all data items are processed:
|
|
40
|
+
|
|
41
|
+
- `failed`: At least one gate averaged below 1.0 across data items
|
|
42
|
+
- `scored`: All gates passed, but at least one threshold scorer missed its threshold
|
|
43
|
+
- `passed`: All gates scored 1.0 and all thresholds were met
|
|
44
|
+
|
|
45
|
+
When no gates or threshold-bearing scorers are provided, the verdict field is omitted and `runEvals` behaves exactly as before.
|
|
46
|
+
|
|
47
|
+
## Gates
|
|
48
|
+
|
|
49
|
+
Gates are scorers passed via the `gates` field. They run before regular scorers on each data item. A gate must average a score of 1.0 across all data items to pass.
|
|
50
|
+
|
|
51
|
+
```typescript
|
|
52
|
+
import { runEvals } from '@mastra/core/evals'
|
|
53
|
+
import { checks } from '@mastra/evals/checks'
|
|
54
|
+
|
|
55
|
+
const result = await runEvals({
|
|
56
|
+
data: [{ input: 'What is the weather?' }],
|
|
57
|
+
target: weatherAgent,
|
|
58
|
+
gates: [checks.calledTool('get_weather')],
|
|
59
|
+
scorers: [qualityScorer],
|
|
60
|
+
})
|
|
61
|
+
|
|
62
|
+
// result.gateResults: [{ id: 'check-called-tool', passed: true, score: 1 }]
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
Any scorer works as a gate. Quick Checks are a natural fit because they return binary 1/0 scores.
|
|
66
|
+
|
|
67
|
+
> **Note:** Visit [runEvals() reference](https://mastra.ai/reference/evals/run-evals) for the full parameter and return type documentation.
|
|
68
|
+
|
|
69
|
+
## Thresholds
|
|
70
|
+
|
|
71
|
+
Wrap a scorer in `{ scorer, threshold }` to set pass/fail bounds. The threshold is compared against the scorer's average score across all data items.
|
|
72
|
+
|
|
73
|
+
A `threshold` can be:
|
|
74
|
+
|
|
75
|
+
- **A number** — implies minimum (score at or above passes): `{ scorer, threshold: 0.7 }`
|
|
76
|
+
- **An object with `min` and/or `max`** — for range-based checks: `{ scorer, threshold: { max: 0.3 } }`
|
|
77
|
+
|
|
78
|
+
Use `max` for scorers where a high score is bad (e.g., hallucination, toxicity). Use `{ min, max }` when the score should fall within a specific band.
|
|
79
|
+
|
|
80
|
+
```typescript
|
|
81
|
+
import { runEvals } from '@mastra/core/evals'
|
|
82
|
+
|
|
83
|
+
const result = await runEvals({
|
|
84
|
+
data: [{ input: 'Explain quantum computing' }],
|
|
85
|
+
target: myAgent,
|
|
86
|
+
scorers: [
|
|
87
|
+
{ scorer: faithfulnessScorer, threshold: 0.7 }, // min threshold (number shorthand)
|
|
88
|
+
{ scorer: hallucinationScorer, threshold: { max: 0.3 } }, // max threshold — high score = bad
|
|
89
|
+
{ scorer: verbosityScorer, threshold: { min: 0.3, max: 0.8 } }, // range threshold
|
|
90
|
+
toneScorer, // bare scorer, no threshold — tracked only
|
|
91
|
+
],
|
|
92
|
+
})
|
|
93
|
+
|
|
94
|
+
// result.thresholdResults:
|
|
95
|
+
// [
|
|
96
|
+
// { id: 'faithfulness', passed: true, averageScore: 0.85, threshold: 0.7 },
|
|
97
|
+
// { id: 'hallucination', passed: true, averageScore: 0.1, threshold: { max: 0.3 } },
|
|
98
|
+
// { id: 'verbosity', passed: false, averageScore: 0.9, threshold: { min: 0.3, max: 0.8 } },
|
|
99
|
+
// ]
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
A bare scorer (no threshold) still appears in `result.scores` but does not affect the verdict.
|
|
103
|
+
|
|
104
|
+
## Using verdicts in CI
|
|
105
|
+
|
|
106
|
+
The verdict gives a single signal for CI pipelines:
|
|
107
|
+
|
|
108
|
+
```typescript
|
|
109
|
+
import { runEvals } from '@mastra/core/evals'
|
|
110
|
+
import { checks } from '@mastra/evals/checks'
|
|
111
|
+
|
|
112
|
+
const result = await runEvals({
|
|
113
|
+
data: testDataset,
|
|
114
|
+
target: myAgent,
|
|
115
|
+
gates: [checks.calledTool('search'), checks.noToolErrors()],
|
|
116
|
+
scorers: [{ scorer: faithfulnessScorer, threshold: 0.7 }],
|
|
117
|
+
})
|
|
118
|
+
|
|
119
|
+
if (result.verdict === 'failed') {
|
|
120
|
+
console.error(
|
|
121
|
+
'Gate failures:',
|
|
122
|
+
result.gateResults?.filter(g => !g.passed),
|
|
123
|
+
)
|
|
124
|
+
process.exit(1)
|
|
125
|
+
}
|
|
126
|
+
|
|
127
|
+
if (result.verdict === 'scored') {
|
|
128
|
+
console.warn(
|
|
129
|
+
'Threshold misses:',
|
|
130
|
+
result.thresholdResults?.filter(t => !t.passed),
|
|
131
|
+
)
|
|
132
|
+
}
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
## Related
|
|
136
|
+
|
|
137
|
+
- [Quick Checks](https://mastra.ai/docs/evals/quick-checks): Zero-LLM micro-scorers that work well as gates
|
|
138
|
+
- [runEvals() reference](https://mastra.ai/reference/evals/run-evals): Full API documentation
|
|
139
|
+
- [Built-in scorers](https://mastra.ai/docs/evals/built-in-scorers): LLM-based and code-based scorers
|
|
140
|
+
- [Running evals in CI](https://mastra.ai/docs/evals/running-in-ci): CI integration patterns
|
|
@@ -136,6 +136,7 @@ Once registered, you can score traces interactively within Studio under the **Ob
|
|
|
136
136
|
## Next steps
|
|
137
137
|
|
|
138
138
|
- Use [Quick Checks](https://mastra.ai/docs/evals/quick-checks) for fast, deterministic assertions on text and tool usage
|
|
139
|
+
- Add [gates and verdicts](https://mastra.ai/docs/evals/gates-and-verdicts) to enforce hard requirements and quality thresholds
|
|
139
140
|
- Learn how to create your own scorers in the [Creating Custom Scorers](https://mastra.ai/docs/evals/custom-scorers) guide
|
|
140
141
|
- Explore built-in scorers in the [Built-in Scorers](https://mastra.ai/docs/evals/built-in-scorers) section
|
|
141
142
|
- Test scorers with [Studio](https://mastra.ai/docs/studio/overview)
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Netlify
|
|
2
2
|
|
|
3
|
-
Netlify AI Gateway provides unified access to multiple providers with built-in caching and observability. Access
|
|
3
|
+
Netlify AI Gateway provides unified access to multiple providers with built-in caching and observability. Access 64 models through Mastra's model router.
|
|
4
4
|
|
|
5
5
|
Learn more in the [Netlify documentation](https://docs.netlify.com/build/ai-gateway/overview/).
|
|
6
6
|
|
|
@@ -52,9 +52,7 @@ ANTHROPIC_API_KEY=ant-...
|
|
|
52
52
|
| `gemini/gemini-2.5-pro` |
|
|
53
53
|
| `gemini/gemini-3-flash-preview` |
|
|
54
54
|
| `gemini/gemini-3-pro-image` |
|
|
55
|
-
| `gemini/gemini-3-pro-image-preview` |
|
|
56
55
|
| `gemini/gemini-3.1-flash-image` |
|
|
57
|
-
| `gemini/gemini-3.1-flash-image-preview` |
|
|
58
56
|
| `gemini/gemini-3.1-flash-lite` |
|
|
59
57
|
| `gemini/gemini-3.1-pro-preview` |
|
|
60
58
|
| `gemini/gemini-3.1-pro-preview-customtools` |
|
package/.docs/models/index.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Model Providers
|
|
2
2
|
|
|
3
|
-
Mastra provides a unified interface for working with LLMs across multiple providers, giving you access to
|
|
3
|
+
Mastra provides a unified interface for working with LLMs across multiple providers, giving you access to 4536 models from 133 providers through a single API.
|
|
4
4
|
|
|
5
5
|
## Features
|
|
6
6
|
|
|
@@ -44,7 +44,7 @@ for await (const chunk of stream) {
|
|
|
44
44
|
| `baseten/zai-org/GLM-4.7` | 200K | | | | | | $0.60 | $2 |
|
|
45
45
|
| `baseten/zai-org/GLM-5` | 203K | | | | | | $0.95 | $3 |
|
|
46
46
|
| `baseten/zai-org/GLM-5.1` | 203K | | | | | | $1 | $4 |
|
|
47
|
-
| `baseten/zai-org/GLM-5.2` |
|
|
47
|
+
| `baseten/zai-org/GLM-5.2` | 203K | | | | | | $1 | $4 |
|
|
48
48
|
|
|
49
49
|
## Advanced configuration
|
|
50
50
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Hugging Face
|
|
2
2
|
|
|
3
|
-
Access
|
|
3
|
+
Access 49 Hugging Face models through Mastra's model router. Authentication is handled automatically using the `HF_TOKEN` environment variable.
|
|
4
4
|
|
|
5
5
|
Learn more in the [Hugging Face documentation](https://huggingface.co).
|
|
6
6
|
|
|
@@ -68,9 +68,12 @@ for await (const chunk of stream) {
|
|
|
68
68
|
| `huggingface/Qwen/Qwen3.5-35B-A3B` | 262K | | | | | | $0.25 | $2 |
|
|
69
69
|
| `huggingface/Qwen/Qwen3.5-397B-A17B` | 262K | | | | | | $0.60 | $4 |
|
|
70
70
|
| `huggingface/Qwen/Qwen3.5-9B` | 262K | | | | | | $0.17 | $0.25 |
|
|
71
|
+
| `huggingface/Qwen/Qwen3.6-27B` | 262K | | | | | | $0.47 | $3 |
|
|
71
72
|
| `huggingface/Qwen/Qwen3.6-35B-A3B` | 262K | | | | | | $0.15 | $0.95 |
|
|
72
73
|
| `huggingface/stepfun-ai/Step-3.5-Flash` | 262K | | | | | | $0.10 | $0.30 |
|
|
74
|
+
| `huggingface/stepfun-ai/Step-3.7-Flash` | 262K | | | | | | $0.20 | $1 |
|
|
73
75
|
| `huggingface/XiaomiMiMo/MiMo-V2-Flash` | 262K | | | | | | $0.10 | $0.30 |
|
|
76
|
+
| `huggingface/XiaomiMiMo/MiMo-V2.5-Pro` | 1.0M | | | | | | $1 | $3 |
|
|
74
77
|
| `huggingface/zai-org/GLM-4.5` | 131K | | | | | | $0.60 | $2 |
|
|
75
78
|
| `huggingface/zai-org/GLM-4.5-Air` | 131K | | | | | | $0.13 | $0.85 |
|
|
76
79
|
| `huggingface/zai-org/GLM-4.5V` | 66K | | | | | | $0.60 | $2 |
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# LLM Gateway
|
|
2
2
|
|
|
3
|
-
Access
|
|
3
|
+
Access 181 LLM Gateway models through Mastra's model router. Authentication is handled automatically using the `LLMGATEWAY_API_KEY` environment variable.
|
|
4
4
|
|
|
5
5
|
Learn more in the [LLM Gateway documentation](https://llmgateway.io/docs).
|
|
6
6
|
|
|
@@ -32,204 +32,189 @@ for await (const chunk of stream) {
|
|
|
32
32
|
|
|
33
33
|
## Models
|
|
34
34
|
|
|
35
|
-
| Model
|
|
36
|
-
|
|
|
37
|
-
| `llmgateway/auto`
|
|
38
|
-
| `llmgateway/claude-3-
|
|
39
|
-
| `llmgateway/claude-3-7-sonnet`
|
|
40
|
-
| `llmgateway/claude-3-
|
|
41
|
-
| `llmgateway/claude-
|
|
42
|
-
| `llmgateway/claude-haiku-4-5`
|
|
43
|
-
| `llmgateway/claude-
|
|
44
|
-
| `llmgateway/claude-opus-4-
|
|
45
|
-
| `llmgateway/claude-opus-4-
|
|
46
|
-
| `llmgateway/claude-opus-4-
|
|
47
|
-
| `llmgateway/claude-opus-4-
|
|
48
|
-
| `llmgateway/claude-
|
|
49
|
-
| `llmgateway/claude-
|
|
50
|
-
| `llmgateway/claude-sonnet-4-
|
|
51
|
-
| `llmgateway/
|
|
52
|
-
| `llmgateway/
|
|
53
|
-
| `llmgateway/
|
|
54
|
-
| `llmgateway/
|
|
55
|
-
| `llmgateway/
|
|
56
|
-
| `llmgateway/deepseek-
|
|
57
|
-
| `llmgateway/
|
|
58
|
-
| `llmgateway/
|
|
59
|
-
| `llmgateway/
|
|
60
|
-
| `llmgateway/
|
|
61
|
-
| `llmgateway/gemini-2.
|
|
62
|
-
| `llmgateway/gemini-
|
|
63
|
-
| `llmgateway/gemini-
|
|
64
|
-
| `llmgateway/gemini-
|
|
65
|
-
| `llmgateway/gemini-
|
|
66
|
-
| `llmgateway/gemini-
|
|
67
|
-
| `llmgateway/
|
|
68
|
-
| `llmgateway/
|
|
69
|
-
| `llmgateway/
|
|
70
|
-
| `llmgateway/
|
|
71
|
-
| `llmgateway/
|
|
72
|
-
| `llmgateway/
|
|
73
|
-
| `llmgateway/
|
|
74
|
-
| `llmgateway/
|
|
75
|
-
| `llmgateway/
|
|
76
|
-
| `llmgateway/
|
|
77
|
-
| `llmgateway/glm-4
|
|
78
|
-
| `llmgateway/glm-4.
|
|
79
|
-
| `llmgateway/glm-4.
|
|
80
|
-
| `llmgateway/glm-4.
|
|
81
|
-
| `llmgateway/glm-4.
|
|
82
|
-
| `llmgateway/glm-4.
|
|
83
|
-
| `llmgateway/glm-4.
|
|
84
|
-
| `llmgateway/glm-
|
|
85
|
-
| `llmgateway/glm-
|
|
86
|
-
| `llmgateway/glm-
|
|
87
|
-
| `llmgateway/
|
|
88
|
-
| `llmgateway/
|
|
89
|
-
| `llmgateway/
|
|
90
|
-
| `llmgateway/
|
|
91
|
-
| `llmgateway/
|
|
92
|
-
| `llmgateway/
|
|
93
|
-
| `llmgateway/
|
|
94
|
-
| `llmgateway/gpt-
|
|
95
|
-
| `llmgateway/gpt-
|
|
96
|
-
| `llmgateway/gpt-
|
|
97
|
-
| `llmgateway/gpt-
|
|
98
|
-
| `llmgateway/gpt-
|
|
99
|
-
| `llmgateway/gpt-
|
|
100
|
-
| `llmgateway/gpt-
|
|
101
|
-
| `llmgateway/gpt-
|
|
102
|
-
| `llmgateway/gpt-
|
|
103
|
-
| `llmgateway/gpt-
|
|
104
|
-
| `llmgateway/gpt-5`
|
|
105
|
-
| `llmgateway/gpt-5
|
|
106
|
-
| `llmgateway/gpt-5-
|
|
107
|
-
| `llmgateway/gpt-5-
|
|
108
|
-
| `llmgateway/gpt-5-pro`
|
|
109
|
-
| `llmgateway/gpt-5.
|
|
110
|
-
| `llmgateway/gpt-5.
|
|
111
|
-
| `llmgateway/gpt-5.
|
|
112
|
-
| `llmgateway/gpt-5.
|
|
113
|
-
| `llmgateway/gpt-5.
|
|
114
|
-
| `llmgateway/gpt-5.
|
|
115
|
-
| `llmgateway/gpt-5.
|
|
116
|
-
| `llmgateway/gpt-5.
|
|
117
|
-
| `llmgateway/gpt-
|
|
118
|
-
| `llmgateway/gpt-
|
|
119
|
-
| `llmgateway/
|
|
120
|
-
| `llmgateway/
|
|
121
|
-
| `llmgateway/
|
|
122
|
-
| `llmgateway/
|
|
123
|
-
| `llmgateway/
|
|
124
|
-
| `llmgateway/
|
|
125
|
-
| `llmgateway/
|
|
126
|
-
| `llmgateway/grok-4-
|
|
127
|
-
| `llmgateway/grok-
|
|
128
|
-
| `llmgateway/
|
|
129
|
-
| `llmgateway/
|
|
130
|
-
| `llmgateway/
|
|
131
|
-
| `llmgateway/
|
|
132
|
-
| `llmgateway/
|
|
133
|
-
| `llmgateway/
|
|
134
|
-
| `llmgateway/
|
|
135
|
-
| `llmgateway/
|
|
136
|
-
| `llmgateway/
|
|
137
|
-
| `llmgateway/
|
|
138
|
-
| `llmgateway/
|
|
139
|
-
| `llmgateway/
|
|
140
|
-
| `llmgateway/
|
|
141
|
-
| `llmgateway/
|
|
142
|
-
| `llmgateway/
|
|
143
|
-
| `llmgateway/llama-
|
|
144
|
-
| `llmgateway/
|
|
145
|
-
| `llmgateway/
|
|
146
|
-
| `llmgateway/
|
|
147
|
-
| `llmgateway/
|
|
148
|
-
| `llmgateway/
|
|
149
|
-
| `llmgateway/
|
|
150
|
-
| `llmgateway/
|
|
151
|
-
| `llmgateway/
|
|
152
|
-
| `llmgateway/
|
|
153
|
-
| `llmgateway/
|
|
154
|
-
| `llmgateway/
|
|
155
|
-
| `llmgateway/
|
|
156
|
-
| `llmgateway/
|
|
157
|
-
| `llmgateway/
|
|
158
|
-
| `llmgateway/
|
|
159
|
-
| `llmgateway/
|
|
160
|
-
| `llmgateway/
|
|
161
|
-
| `llmgateway/
|
|
162
|
-
| `llmgateway/
|
|
163
|
-
| `llmgateway/
|
|
164
|
-
| `llmgateway/
|
|
165
|
-
| `llmgateway/
|
|
166
|
-
| `llmgateway/
|
|
167
|
-
| `llmgateway/
|
|
168
|
-
| `llmgateway/
|
|
169
|
-
| `llmgateway/
|
|
170
|
-
| `llmgateway/
|
|
171
|
-
| `llmgateway/
|
|
172
|
-
| `llmgateway/
|
|
173
|
-
| `llmgateway/
|
|
174
|
-
| `llmgateway/
|
|
175
|
-
| `llmgateway/
|
|
176
|
-
| `llmgateway/
|
|
177
|
-
| `llmgateway/
|
|
178
|
-
| `llmgateway/
|
|
179
|
-
| `llmgateway/
|
|
180
|
-
| `llmgateway/
|
|
181
|
-
| `llmgateway/
|
|
182
|
-
| `llmgateway/
|
|
183
|
-
| `llmgateway/
|
|
184
|
-
| `llmgateway/
|
|
185
|
-
| `llmgateway/
|
|
186
|
-
| `llmgateway/
|
|
187
|
-
| `llmgateway/
|
|
188
|
-
| `llmgateway/
|
|
189
|
-
| `llmgateway/
|
|
190
|
-
| `llmgateway/
|
|
191
|
-
| `llmgateway/
|
|
192
|
-
| `llmgateway/
|
|
193
|
-
| `llmgateway/qwen3-
|
|
194
|
-
| `llmgateway/qwen3-
|
|
195
|
-
| `llmgateway/qwen3-
|
|
196
|
-
| `llmgateway/qwen3-
|
|
197
|
-
| `llmgateway/qwen3-
|
|
198
|
-
| `llmgateway/qwen3-30b-a3b-
|
|
199
|
-
| `llmgateway/qwen3-
|
|
200
|
-
| `llmgateway/qwen3-
|
|
201
|
-
| `llmgateway/qwen3-
|
|
202
|
-
| `llmgateway/qwen3-
|
|
203
|
-
| `llmgateway/qwen3-
|
|
204
|
-
| `llmgateway/qwen3-
|
|
205
|
-
| `llmgateway/qwen3-
|
|
206
|
-
| `llmgateway/qwen3-
|
|
207
|
-
| `llmgateway/qwen3-max`
|
|
208
|
-
| `llmgateway/qwen3-
|
|
209
|
-
| `llmgateway/
|
|
210
|
-
| `llmgateway/
|
|
211
|
-
| `llmgateway/
|
|
212
|
-
| `llmgateway/
|
|
213
|
-
| `llmgateway/
|
|
214
|
-
| `llmgateway/
|
|
215
|
-
| `llmgateway/
|
|
216
|
-
| `llmgateway/
|
|
217
|
-
| `llmgateway/
|
|
218
|
-
| `llmgateway/qwen3.5-9b` | 262K | | | | | | $0.10 | $0.15 |
|
|
219
|
-
| `llmgateway/qwen3.6-35b-a3b` | 262K | | | | | | $0.25 | $1 |
|
|
220
|
-
| `llmgateway/qwen3.6-max-preview` | 262K | | | | | | $1 | $8 |
|
|
221
|
-
| `llmgateway/qwen3.6-plus` | 1.0M | | | | | | $0.50 | $3 |
|
|
222
|
-
| `llmgateway/qwen3.7-max` | 1.0M | | | | | | $3 | $8 |
|
|
223
|
-
| `llmgateway/qwen3.7-plus` | 1.0M | | | | | | $0.40 | $2 |
|
|
224
|
-
| `llmgateway/qwen35-397b-a17b` | 262K | | | | | | $0.60 | $4 |
|
|
225
|
-
| `llmgateway/qwq-plus` | 131K | | | | | | $0.80 | $2 |
|
|
226
|
-
| `llmgateway/seed-1-6-250615` | 256K | | | | | | $0.25 | $2 |
|
|
227
|
-
| `llmgateway/seed-1-6-250915` | 256K | | | | | | $0.25 | $2 |
|
|
228
|
-
| `llmgateway/seed-1-6-flash-250715` | 256K | | | | | | $0.07 | $0.30 |
|
|
229
|
-
| `llmgateway/seed-1-8-251228` | 256K | | | | | | $0.25 | $2 |
|
|
230
|
-
| `llmgateway/sonar` | 128K | | | | | | $1 | $1 |
|
|
231
|
-
| `llmgateway/sonar-pro` | 200K | | | | | | $3 | $15 |
|
|
232
|
-
| `llmgateway/sonar-reasoning-pro` | 128K | | | | | | $2 | $8 |
|
|
35
|
+
| Model | Context | Tools | Reasoning | Image | Audio | Video | Input $/1M | Output $/1M |
|
|
36
|
+
| -------------------------------------------------- | ------- | ----- | --------- | ----- | ----- | ----- | ---------- | ----------- |
|
|
37
|
+
| `llmgateway/auto` | 128K | | | | | | — | — |
|
|
38
|
+
| `llmgateway/claude-3-7-sonnet` | 200K | | | | | | $3 | $15 |
|
|
39
|
+
| `llmgateway/claude-3-7-sonnet-20250219` | 200K | | | | | | $3 | $15 |
|
|
40
|
+
| `llmgateway/claude-3-opus` | 200K | | | | | | $15 | $75 |
|
|
41
|
+
| `llmgateway/claude-haiku-4-5` | 200K | | | | | | $1 | $5 |
|
|
42
|
+
| `llmgateway/claude-haiku-4-5-20251001` | 200K | | | | | | $1 | $5 |
|
|
43
|
+
| `llmgateway/claude-opus-4-1-20250805` | 200K | | | | | | $15 | $75 |
|
|
44
|
+
| `llmgateway/claude-opus-4-5-20251101` | 200K | | | | | | $5 | $25 |
|
|
45
|
+
| `llmgateway/claude-opus-4-6` | 1.0M | | | | | | $5 | $25 |
|
|
46
|
+
| `llmgateway/claude-opus-4-7` | 1.0M | | | | | | $5 | $25 |
|
|
47
|
+
| `llmgateway/claude-opus-4-8` | 1.0M | | | | | | $5 | $25 |
|
|
48
|
+
| `llmgateway/claude-sonnet-4-5` | 200K | | | | | | $3 | $15 |
|
|
49
|
+
| `llmgateway/claude-sonnet-4-5-20250929` | 200K | | | | | | $3 | $15 |
|
|
50
|
+
| `llmgateway/claude-sonnet-4-6` | 1.0M | | | | | | $3 | $15 |
|
|
51
|
+
| `llmgateway/codestral-2508` | 256K | | | | | | $0.30 | $0.90 |
|
|
52
|
+
| `llmgateway/custom` | 128K | | | | | | — | — |
|
|
53
|
+
| `llmgateway/deepseek-v3.1` | 128K | | | | | | $0.56 | $2 |
|
|
54
|
+
| `llmgateway/deepseek-v3.2` | 164K | | | | | | $0.28 | $0.42 |
|
|
55
|
+
| `llmgateway/deepseek-v4-flash` | 1.1M | | | | | | $0.14 | $0.28 |
|
|
56
|
+
| `llmgateway/deepseek-v4-pro` | 1.1M | | | | | | $0.43 | $0.87 |
|
|
57
|
+
| `llmgateway/fugu-ultra` | 1.0M | | | | | | $5 | $30 |
|
|
58
|
+
| `llmgateway/gemini-2.5-flash` | 1.0M | | | | | | $0.30 | $3 |
|
|
59
|
+
| `llmgateway/gemini-2.5-flash-lite` | 1.0M | | | | | | $0.10 | $0.40 |
|
|
60
|
+
| `llmgateway/gemini-2.5-flash-lite-preview-09-2025` | 1.0M | | | | | | $0.10 | $0.40 |
|
|
61
|
+
| `llmgateway/gemini-2.5-pro` | 1.0M | | | | | | $1 | $10 |
|
|
62
|
+
| `llmgateway/gemini-3-flash-preview` | 1.0M | | | | | | $0.50 | $3 |
|
|
63
|
+
| `llmgateway/gemini-3.1-flash-lite` | 1.0M | | | | | | $0.25 | $2 |
|
|
64
|
+
| `llmgateway/gemini-3.1-pro-preview` | 1.0M | | | | | | $2 | $12 |
|
|
65
|
+
| `llmgateway/gemini-3.5-flash` | 1.0M | | | | | | $2 | $9 |
|
|
66
|
+
| `llmgateway/gemini-pro-latest` | 1.0M | | | | | | $2 | $12 |
|
|
67
|
+
| `llmgateway/gemma-4-26b-a4b-it` | 262K | | | | | | $0.07 | $0.34 |
|
|
68
|
+
| `llmgateway/gemma-4-31b-it` | 262K | | | | | | $0.13 | $0.38 |
|
|
69
|
+
| `llmgateway/glm-4-32b-0414-128k` | 128K | | | | | | $0.10 | $0.10 |
|
|
70
|
+
| `llmgateway/glm-4.5` | 131K | | | | | | $0.60 | $2 |
|
|
71
|
+
| `llmgateway/glm-4.5-air` | 131K | | | | | | $0.20 | $1 |
|
|
72
|
+
| `llmgateway/glm-4.5-airx` | 128K | | | | | | $1 | $5 |
|
|
73
|
+
| `llmgateway/glm-4.5-flash` | 128K | | | | | | — | — |
|
|
74
|
+
| `llmgateway/glm-4.5-x` | 128K | | | | | | $2 | $9 |
|
|
75
|
+
| `llmgateway/glm-4.5v` | 128K | | | | | | $0.60 | $2 |
|
|
76
|
+
| `llmgateway/glm-4.6` | 205K | | | | | | $0.60 | $2 |
|
|
77
|
+
| `llmgateway/glm-4.6v` | 131K | | | | | | $0.30 | $0.90 |
|
|
78
|
+
| `llmgateway/glm-4.6v-flash` | 128K | | | | | | — | — |
|
|
79
|
+
| `llmgateway/glm-4.6v-flashx` | 128K | | | | | | $0.04 | $0.40 |
|
|
80
|
+
| `llmgateway/glm-4.7` | 205K | | | | | | $0.60 | $2 |
|
|
81
|
+
| `llmgateway/glm-4.7-flash` | 200K | | | | | | $0.06 | $0.40 |
|
|
82
|
+
| `llmgateway/glm-4.7-flash-free` | 200K | | | | | | — | — |
|
|
83
|
+
| `llmgateway/glm-4.7-flashx` | 200K | | | | | | $0.07 | $0.40 |
|
|
84
|
+
| `llmgateway/glm-5` | 203K | | | | | | $1 | $3 |
|
|
85
|
+
| `llmgateway/glm-5.1` | 205K | | | | | | $1 | $4 |
|
|
86
|
+
| `llmgateway/glm-5.2` | 1.0M | | | | | | $1 | $4 |
|
|
87
|
+
| `llmgateway/gpt-3.5-turbo` | 16K | | | | | | $0.50 | $2 |
|
|
88
|
+
| `llmgateway/gpt-4` | 8K | | | | | | $30 | $60 |
|
|
89
|
+
| `llmgateway/gpt-4-turbo` | 128K | | | | | | $10 | $30 |
|
|
90
|
+
| `llmgateway/gpt-4.1` | 1.0M | | | | | | $2 | $8 |
|
|
91
|
+
| `llmgateway/gpt-4.1-mini` | 1.0M | | | | | | $0.40 | $2 |
|
|
92
|
+
| `llmgateway/gpt-4.1-nano` | 1.0M | | | | | | $0.10 | $0.40 |
|
|
93
|
+
| `llmgateway/gpt-4o` | 128K | | | | | | $3 | $10 |
|
|
94
|
+
| `llmgateway/gpt-4o-mini` | 128K | | | | | | $0.15 | $0.60 |
|
|
95
|
+
| `llmgateway/gpt-4o-mini-search-preview` | 128K | | | | | | $0.15 | $0.60 |
|
|
96
|
+
| `llmgateway/gpt-4o-search-preview` | 128K | | | | | | $3 | $10 |
|
|
97
|
+
| `llmgateway/gpt-5` | 400K | | | | | | $1 | $10 |
|
|
98
|
+
| `llmgateway/gpt-5-chat-latest` | 400K | | | | | | $1 | $10 |
|
|
99
|
+
| `llmgateway/gpt-5-mini` | 400K | | | | | | $0.25 | $2 |
|
|
100
|
+
| `llmgateway/gpt-5-nano` | 400K | | | | | | $0.05 | $0.40 |
|
|
101
|
+
| `llmgateway/gpt-5-pro` | 400K | | | | | | $15 | $120 |
|
|
102
|
+
| `llmgateway/gpt-5.1` | 400K | | | | | | $1 | $10 |
|
|
103
|
+
| `llmgateway/gpt-5.1-codex` | 400K | | | | | | $1 | $10 |
|
|
104
|
+
| `llmgateway/gpt-5.1-codex-mini` | 400K | | | | | | $0.25 | $2 |
|
|
105
|
+
| `llmgateway/gpt-5.2` | 400K | | | | | | $2 | $14 |
|
|
106
|
+
| `llmgateway/gpt-5.2-chat-latest` | 128K | | | | | | $2 | $14 |
|
|
107
|
+
| `llmgateway/gpt-5.2-codex` | 400K | | | | | | $2 | $14 |
|
|
108
|
+
| `llmgateway/gpt-5.2-pro` | 400K | | | | | | $21 | $168 |
|
|
109
|
+
| `llmgateway/gpt-5.3-chat-latest` | 128K | | | | | | $2 | $14 |
|
|
110
|
+
| `llmgateway/gpt-5.3-codex` | 400K | | | | | | $2 | $14 |
|
|
111
|
+
| `llmgateway/gpt-5.4` | 1.1M | | | | | | $3 | $15 |
|
|
112
|
+
| `llmgateway/gpt-5.4-mini` | 400K | | | | | | $0.75 | $5 |
|
|
113
|
+
| `llmgateway/gpt-5.4-nano` | 400K | | | | | | $0.20 | $1 |
|
|
114
|
+
| `llmgateway/gpt-5.4-pro` | 1.1M | | | | | | $30 | $180 |
|
|
115
|
+
| `llmgateway/gpt-5.5` | 1.1M | | | | | | $5 | $30 |
|
|
116
|
+
| `llmgateway/gpt-5.5-pro` | 1.1M | | | | | | $30 | $180 |
|
|
117
|
+
| `llmgateway/gpt-oss-120b` | 131K | | | | | | $0.15 | $0.75 |
|
|
118
|
+
| `llmgateway/gpt-oss-20b` | 131K | | | | | | $0.10 | $0.50 |
|
|
119
|
+
| `llmgateway/grok-4` | 256K | | | | | | $3 | $15 |
|
|
120
|
+
| `llmgateway/grok-4-1-fast-non-reasoning` | 2.0M | | | | | | $0.20 | $0.50 |
|
|
121
|
+
| `llmgateway/grok-4-1-fast-reasoning` | 2.0M | | | | | | $0.20 | $0.50 |
|
|
122
|
+
| `llmgateway/grok-4-20-beta-0309-non-reasoning` | 2.0M | | | | | | $2 | $6 |
|
|
123
|
+
| `llmgateway/grok-4-20-beta-0309-reasoning` | 2.0M | | | | | | $2 | $6 |
|
|
124
|
+
| `llmgateway/grok-4-20-non-reasoning` | 2.0M | | | | | | $2 | $6 |
|
|
125
|
+
| `llmgateway/grok-4-20-reasoning` | 2.0M | | | | | | $2 | $6 |
|
|
126
|
+
| `llmgateway/grok-4-3` | 1.0M | | | | | | $1 | $3 |
|
|
127
|
+
| `llmgateway/grok-build-0-1` | 256K | | | | | | $1 | $2 |
|
|
128
|
+
| `llmgateway/kimi-k2` | 256K | | | | | | $1 | $3 |
|
|
129
|
+
| `llmgateway/kimi-k2-thinking` | 262K | | | | | | $0.60 | $3 |
|
|
130
|
+
| `llmgateway/kimi-k2-thinking-turbo` | 262K | | | | | | $1 | $8 |
|
|
131
|
+
| `llmgateway/kimi-k2.5` | 262K | | | | | | $0.60 | $3 |
|
|
132
|
+
| `llmgateway/kimi-k2.6` | 262K | | | | | | $0.95 | $4 |
|
|
133
|
+
| `llmgateway/kimi-k2.7-code` | 262K | | | | | | $0.95 | $4 |
|
|
134
|
+
| `llmgateway/kimi-k2.7-code-highspeed` | 262K | | | | | | $2 | $8 |
|
|
135
|
+
| `llmgateway/llama-3-70b-instruct` | 8K | | | | | | $0.51 | $0.74 |
|
|
136
|
+
| `llmgateway/llama-3-8b-instruct` | 8K | | | | | | $0.04 | $0.04 |
|
|
137
|
+
| `llmgateway/llama-3.1-70b-instruct` | 128K | | | | | | $0.72 | $0.72 |
|
|
138
|
+
| `llmgateway/llama-3.1-nemotron-ultra-253b` | 128K | | | | | | $0.60 | $2 |
|
|
139
|
+
| `llmgateway/llama-3.2-11b-instruct` | 128K | | | | | | $0.07 | $0.33 |
|
|
140
|
+
| `llmgateway/llama-3.2-3b-instruct` | 33K | | | | | | $0.03 | $0.05 |
|
|
141
|
+
| `llmgateway/llama-3.3-70b-instruct` | 131K | | | | | | $0.13 | $0.40 |
|
|
142
|
+
| `llmgateway/llama-4-maverick-17b-instruct` | 1.0M | | | | | | $0.24 | $0.97 |
|
|
143
|
+
| `llmgateway/llama-4-scout-17b-instruct` | 131K | | | | | | $0.17 | $0.66 |
|
|
144
|
+
| `llmgateway/mimo-v2-omni` | 256K | | | | | | $0.40 | $2 |
|
|
145
|
+
| `llmgateway/mimo-v2-pro` | 1.0M | | | | | | $1 | $3 |
|
|
146
|
+
| `llmgateway/mimo-v2.5` | 1.0M | | | | | | $0.14 | $0.28 |
|
|
147
|
+
| `llmgateway/mimo-v2.5-pro` | 1.0M | | | | | | $0.43 | $0.87 |
|
|
148
|
+
| `llmgateway/minimax-m2` | 197K | | | | | | $0.20 | $1 |
|
|
149
|
+
| `llmgateway/minimax-m2.1` | 205K | | | | | | $0.27 | $1 |
|
|
150
|
+
| `llmgateway/minimax-m2.1-lightning` | 197K | | | | | | $0.12 | $0.48 |
|
|
151
|
+
| `llmgateway/minimax-m2.5` | 229K | | | | | | $0.30 | $1 |
|
|
152
|
+
| `llmgateway/minimax-m2.5-highspeed` | 205K | | | | | | $0.60 | $2 |
|
|
153
|
+
| `llmgateway/minimax-m2.7` | 205K | | | | | | $0.30 | $1 |
|
|
154
|
+
| `llmgateway/minimax-m2.7-highspeed` | 205K | | | | | | $0.60 | $2 |
|
|
155
|
+
| `llmgateway/minimax-m3` | 512K | | | | | | $0.60 | $2 |
|
|
156
|
+
| `llmgateway/minimax-text-01` | 1.0M | | | | | | $0.20 | $1 |
|
|
157
|
+
| `llmgateway/ministral-14b-2512` | 262K | | | | | | $0.20 | $0.20 |
|
|
158
|
+
| `llmgateway/ministral-3b-2512` | 131K | | | | | | $0.10 | $0.10 |
|
|
159
|
+
| `llmgateway/ministral-8b-2512` | 262K | | | | | | $0.15 | $0.15 |
|
|
160
|
+
| `llmgateway/mistral-large-2512` | 262K | | | | | | $0.50 | $2 |
|
|
161
|
+
| `llmgateway/mistral-large-latest` | 128K | | | | | | $4 | $12 |
|
|
162
|
+
| `llmgateway/mistral-small-2506` | 128K | | | | | | $0.10 | $0.30 |
|
|
163
|
+
| `llmgateway/nemotron-3-ultra-550b` | 262K | | | | | | $0.50 | $3 |
|
|
164
|
+
| `llmgateway/o1` | 200K | | | | | | $15 | $60 |
|
|
165
|
+
| `llmgateway/o3` | 200K | | | | | | $2 | $8 |
|
|
166
|
+
| `llmgateway/o3-mini` | 200K | | | | | | $1 | $4 |
|
|
167
|
+
| `llmgateway/o4-mini` | 200K | | | | | | $1 | $4 |
|
|
168
|
+
| `llmgateway/pixtral-large-latest` | 128K | | | | | | $4 | $12 |
|
|
169
|
+
| `llmgateway/qwen-coder-plus` | 131K | | | | | | $0.50 | $1 |
|
|
170
|
+
| `llmgateway/qwen-flash` | 1.0M | | | | | | $0.05 | $0.40 |
|
|
171
|
+
| `llmgateway/qwen-max` | 33K | | | | | | $2 | $6 |
|
|
172
|
+
| `llmgateway/qwen-max-latest` | 33K | | | | | | $2 | $6 |
|
|
173
|
+
| `llmgateway/qwen-omni-turbo` | 33K | | | | | | $0.20 | $0.80 |
|
|
174
|
+
| `llmgateway/qwen-plus` | 131K | | | | | | $0.40 | $1 |
|
|
175
|
+
| `llmgateway/qwen-plus-latest` | 1.0M | | | | | | $0.40 | $1 |
|
|
176
|
+
| `llmgateway/qwen-turbo` | 1.0M | | | | | | $0.05 | $0.20 |
|
|
177
|
+
| `llmgateway/qwen-vl-max` | 131K | | | | | | $0.80 | $3 |
|
|
178
|
+
| `llmgateway/qwen-vl-plus` | 131K | | | | | | $0.21 | $0.64 |
|
|
179
|
+
| `llmgateway/qwen2-5-vl-32b-instruct` | 131K | | | | | | $1 | $4 |
|
|
180
|
+
| `llmgateway/qwen2-5-vl-72b-instruct` | 33K | | | | | | $0.13 | $0.40 |
|
|
181
|
+
| `llmgateway/qwen3-235b-a22b-fp8` | 41K | | | | | | $0.20 | $0.80 |
|
|
182
|
+
| `llmgateway/qwen3-235b-a22b-instruct-2507` | 262K | | | | | | $0.20 | $0.60 |
|
|
183
|
+
| `llmgateway/qwen3-235b-a22b-thinking-2507` | 262K | | | | | | $0.20 | $0.60 |
|
|
184
|
+
| `llmgateway/qwen3-30b-a3b-instruct-2507` | 262K | | | | | | $0.10 | $0.30 |
|
|
185
|
+
| `llmgateway/qwen3-32b` | 33K | | | | | | $0.10 | $0.30 |
|
|
186
|
+
| `llmgateway/qwen3-4b-fp8` | 128K | | | | | | $0.03 | $0.03 |
|
|
187
|
+
| `llmgateway/qwen3-coder-30b-a3b-instruct` | 262K | | | | | | $0.10 | $0.30 |
|
|
188
|
+
| `llmgateway/qwen3-coder-480b-a35b-instruct` | 262K | | | | | | $0.40 | $2 |
|
|
189
|
+
| `llmgateway/qwen3-coder-flash` | 1.0M | | | | | | $0.30 | $2 |
|
|
190
|
+
| `llmgateway/qwen3-coder-next` | 262K | | | | | | $0.11 | $0.68 |
|
|
191
|
+
| `llmgateway/qwen3-coder-plus` | 1.0M | | | | | | $6 | $60 |
|
|
192
|
+
| `llmgateway/qwen3-max` | 262K | | | | | | $3 | $15 |
|
|
193
|
+
| `llmgateway/qwen3-max-2026-01-23` | 262K | | | | | | $1 | $6 |
|
|
194
|
+
| `llmgateway/qwen3-next-80b-a3b-instruct` | 131K | | | | | | $0.50 | $2 |
|
|
195
|
+
| `llmgateway/qwen3-next-80b-a3b-thinking` | 131K | | | | | | $0.50 | $6 |
|
|
196
|
+
| `llmgateway/qwen3-vl-235b-a22b-instruct` | 131K | | | | | | $0.50 | $2 |
|
|
197
|
+
| `llmgateway/qwen3-vl-235b-a22b-thinking` | 131K | | | | | | $0.50 | $2 |
|
|
198
|
+
| `llmgateway/qwen3-vl-30b-a3b-instruct` | 131K | | | | | | $0.20 | $0.70 |
|
|
199
|
+
| `llmgateway/qwen3-vl-30b-a3b-thinking` | 131K | | | | | | $0.20 | $1 |
|
|
200
|
+
| `llmgateway/qwen3-vl-8b-instruct` | 131K | | | | | | $0.08 | $0.50 |
|
|
201
|
+
| `llmgateway/qwen3-vl-flash` | 262K | | | | | | $0.05 | $0.40 |
|
|
202
|
+
| `llmgateway/qwen3-vl-plus` | 262K | | | | | | $0.20 | $2 |
|
|
203
|
+
| `llmgateway/qwen3.5-9b` | 262K | | | | | | $0.10 | $0.15 |
|
|
204
|
+
| `llmgateway/qwen3.6-35b-a3b` | 262K | | | | | | $0.25 | $1 |
|
|
205
|
+
| `llmgateway/qwen3.6-max-preview` | 262K | | | | | | $1 | $8 |
|
|
206
|
+
| `llmgateway/qwen3.6-plus` | 262K | | | | | | $0.50 | $3 |
|
|
207
|
+
| `llmgateway/qwen3.7-max` | 1.0M | | | | | | $3 | $8 |
|
|
208
|
+
| `llmgateway/qwen3.7-plus` | 1.0M | | | | | | $0.40 | $2 |
|
|
209
|
+
| `llmgateway/qwen35-397b-a17b` | 262K | | | | | | $0.60 | $4 |
|
|
210
|
+
| `llmgateway/qwq-plus` | 131K | | | | | | $0.80 | $2 |
|
|
211
|
+
| `llmgateway/seed-1-6-250615` | 256K | | | | | | $0.25 | $2 |
|
|
212
|
+
| `llmgateway/seed-1-6-250915` | 256K | | | | | | $0.25 | $2 |
|
|
213
|
+
| `llmgateway/seed-1-6-flash-250715` | 256K | | | | | | $0.07 | $0.30 |
|
|
214
|
+
| `llmgateway/seed-1-8-251228` | 256K | | | | | | $0.25 | $2 |
|
|
215
|
+
| `llmgateway/sonar` | 130K | | | | | | $1 | $1 |
|
|
216
|
+
| `llmgateway/sonar-pro` | 200K | | | | | | $3 | $15 |
|
|
217
|
+
| `llmgateway/sonar-reasoning-pro` | 128K | | | | | | $2 | $8 |
|
|
233
218
|
|
|
234
219
|
## Advanced configuration
|
|
235
220
|
|
|
@@ -47,6 +47,10 @@ export const supportAgent = new Agent({
|
|
|
47
47
|
|
|
48
48
|
**resolveResourceId** (`(ctx: ResolveResourceIdContext) => string | Promise<string>`): Decide which \`resourceId\` owns resource-level memory for a channel thread, separately from who sent the message. Runs only when a new thread is created; reused threads keep their stored owner and never call the hook. Return \`ctx.defaultResourceId\` (\`${platform}:${message.author.userId}\`) to keep the built-in behavior.
|
|
49
49
|
|
|
50
|
+
**waitUntil** (`(promise: Promise<unknown>) => void`): Platform \`waitUntil\` function. Required on Vercel so background agent runs survive after the webhook returns 200. On Vercel pass \`waitUntil\` from \`@vercel/functions\`. Cloudflare Workers and Netlify Functions are detected automatically from the request context. AWS Lambda does not need \`waitUntil\` because it waits for the event loop to drain naturally.
|
|
51
|
+
|
|
52
|
+
**resolveWaitUntil** (`(c: Context) => ((promise: Promise<unknown>) => void) | undefined`): Resolver for runtimes where \`waitUntil\` lives on the Hono request context but is not covered by the built-in helper. Resolution order: bare \`waitUntil\` → \`resolveWaitUntil(c)\` → default (Cloudflare Workers, Netlify).
|
|
53
|
+
|
|
50
54
|
## Per-adapter options
|
|
51
55
|
|
|
52
56
|
Wrap an adapter in a `ChannelAdapterConfig` object to set per-adapter options:
|
|
@@ -29,13 +29,34 @@ console.log(`Average scores:`, result.scores)
|
|
|
29
29
|
console.log(`Processed ${result.summary.totalItems} items`)
|
|
30
30
|
```
|
|
31
31
|
|
|
32
|
+
### With gates and thresholds
|
|
33
|
+
|
|
34
|
+
```typescript
|
|
35
|
+
import { runEvals } from '@mastra/core/evals'
|
|
36
|
+
import { checks } from '@mastra/evals/checks'
|
|
37
|
+
import { faithfulnessScorer } from './scorers'
|
|
38
|
+
|
|
39
|
+
const result = await runEvals({
|
|
40
|
+
target: myAgent,
|
|
41
|
+
data: [{ input: 'What is the weather in Brooklyn?' }],
|
|
42
|
+
gates: [checks.calledTool('get_weather'), checks.noToolErrors()],
|
|
43
|
+
scorers: [{ scorer: faithfulnessScorer, threshold: 0.7 }, checks.includes('Brooklyn')],
|
|
44
|
+
})
|
|
45
|
+
|
|
46
|
+
result.verdict // 'passed' | 'scored' | 'failed'
|
|
47
|
+
result.gateResults // [{ id, passed, score }]
|
|
48
|
+
result.thresholdResults // [{ id, passed, averageScore, threshold }]
|
|
49
|
+
```
|
|
50
|
+
|
|
32
51
|
## Parameters
|
|
33
52
|
|
|
34
53
|
**target** (`Agent | Workflow`): The agent or workflow to evaluate.
|
|
35
54
|
|
|
36
55
|
**data** (`RunEvalsDataItem[]`): Array of test cases with input data and optional ground truth.
|
|
37
56
|
|
|
38
|
-
**scorers** (`
|
|
57
|
+
**scorers** (`ScorerEntry[] | AgentScorerConfig | WorkflowScorerConfig`): Scorers to use. Each entry is either a bare \`MastraScorer\` or \`{ scorer, threshold }\` for threshold tracking. An \`AgentScorerConfig\` object separates agent-level and trajectory scorers. A \`WorkflowScorerConfig\` object specifies scorers for the workflow, individual steps, and trajectory.
|
|
58
|
+
|
|
59
|
+
**gates** (`MastraScorer[]`): Scorers that must score 1.0 for the run to pass. If any gate averages below 1.0 across data items, the verdict is \`failed\`. Gates run before regular scorers on each data item.
|
|
39
60
|
|
|
40
61
|
**targetOptions** (`AgentExecutionOptions | WorkflowRunOptions`): Options forwarded to the target during execution. For agents: options passed to agent.generate() (e.g. maxSteps, modelSettings, instructions). For workflows: options passed to run.start() (e.g. perStep, outputOptions, initialState).
|
|
41
62
|
|
|
@@ -83,8 +104,55 @@ For workflows, use `WorkflowScorerConfig` to specify scorers at different levels
|
|
|
83
104
|
|
|
84
105
|
**summary.totalItems** (`number`): Total number of test cases processed.
|
|
85
106
|
|
|
107
|
+
**verdict** (`'passed' | 'scored' | 'failed'`): Present when \`gates\` or threshold-bearing scorers are provided. \`passed\` = all gates and thresholds met. \`scored\` = gates passed but a threshold was missed. \`failed\` = at least one gate did not score 1.0.
|
|
108
|
+
|
|
109
|
+
**gateResults** (`GateResult[]`): Per-gate results averaged across all data items. Each entry has \`id\`, \`passed\` (boolean), and \`score\` (0–1).
|
|
110
|
+
|
|
111
|
+
**thresholdResults** (`ThresholdResult[]`): Per-threshold-scorer results averaged across all data items. Each entry has \`id\`, \`passed\`, \`averageScore\`, and \`threshold\`.
|
|
112
|
+
|
|
113
|
+
## ScorerEntry
|
|
114
|
+
|
|
115
|
+
A scorer entry in the `scorers` array can be either a bare scorer or a scorer with a threshold:
|
|
116
|
+
|
|
117
|
+
**scorer** (`MastraScorer`): The scorer instance.
|
|
118
|
+
|
|
119
|
+
**threshold** (`number | { min?: number; max?: number }`): A number implies minimum threshold (score at or above passes). Use \`{ min, max }\` for range-based checks — e.g. \`{ max: 0.3 }\` for scorers like hallucination where a high score is bad. Both \`min\` and \`max\` must be between 0 and 1.
|
|
120
|
+
|
|
86
121
|
## Examples
|
|
87
122
|
|
|
123
|
+
### Gates and verdict
|
|
124
|
+
|
|
125
|
+
Use `gates` for hard pass/fail requirements and `{ scorer, threshold }` for tracked quality metrics:
|
|
126
|
+
|
|
127
|
+
```typescript
|
|
128
|
+
import { runEvals } from '@mastra/core/evals'
|
|
129
|
+
import { checks } from '@mastra/evals/checks'
|
|
130
|
+
|
|
131
|
+
const result = await runEvals({
|
|
132
|
+
target: weatherAgent,
|
|
133
|
+
data: [{ input: 'What is the weather in Brooklyn?' }],
|
|
134
|
+
gates: [checks.calledTool('get_weather'), checks.noToolErrors()],
|
|
135
|
+
scorers: [
|
|
136
|
+
{ scorer: faithfulnessScorer, threshold: 0.7 }, // min threshold (number shorthand)
|
|
137
|
+
{ scorer: hallucinationScorer, threshold: { max: 0.3 } }, // max threshold (high = bad)
|
|
138
|
+
{ scorer: toneScorer, threshold: { min: 0.5, max: 0.9 } }, // range threshold
|
|
139
|
+
checks.includes('Brooklyn'), // bare scorer, no threshold
|
|
140
|
+
],
|
|
141
|
+
})
|
|
142
|
+
|
|
143
|
+
if (result.verdict === 'failed') {
|
|
144
|
+
console.log(
|
|
145
|
+
'Gate failures:',
|
|
146
|
+
result.gateResults?.filter(g => !g.passed),
|
|
147
|
+
)
|
|
148
|
+
} else if (result.verdict === 'scored') {
|
|
149
|
+
console.log(
|
|
150
|
+
'Threshold misses:',
|
|
151
|
+
result.thresholdResults?.filter(t => !t.passed),
|
|
152
|
+
)
|
|
153
|
+
}
|
|
154
|
+
```
|
|
155
|
+
|
|
88
156
|
### Agent Evaluation
|
|
89
157
|
|
|
90
158
|
```typescript
|
|
@@ -246,6 +314,8 @@ const result = await runEvals({
|
|
|
246
314
|
|
|
247
315
|
## Related
|
|
248
316
|
|
|
317
|
+
- [Gates and Verdicts](https://mastra.ai/docs/evals/gates-and-verdicts): Conceptual guide to severity semantics
|
|
318
|
+
- [Quick Checks](https://mastra.ai/reference/evals/checks): Zero-LLM composable micro-scorers
|
|
249
319
|
- [createScorer()](https://mastra.ai/reference/evals/create-scorer): Create custom scorers for experiments
|
|
250
320
|
- [MastraScorer](https://mastra.ai/reference/evals/mastra-scorer): Learn about scorer structure and methods
|
|
251
321
|
- [Trajectory Accuracy](https://mastra.ai/reference/evals/trajectory-accuracy): Built-in trajectory evaluation scorers
|
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,12 @@
|
|
|
1
1
|
# @mastra/mcp-docs-server
|
|
2
2
|
|
|
3
|
+
## 1.2.2-alpha.3
|
|
4
|
+
|
|
5
|
+
### Patch Changes
|
|
6
|
+
|
|
7
|
+
- Updated dependencies [[`86623c1`](https://github.com/mastra-ai/mastra/commit/86623c1adf7d22de32cc916dda17f4155184db36), [`7c9dd77`](https://github.com/mastra-ai/mastra/commit/7c9dd77bd18cb8dc72797e25f1a0fbdc71a11347), [`9990965`](https://github.com/mastra-ai/mastra/commit/999096571635a83b42ef40841fd7028cfa630779), [`c0ffa3c`](https://github.com/mastra-ai/mastra/commit/c0ffa3c897ccd326de880df734740a7f0681a18f), [`0504bf5`](https://github.com/mastra-ai/mastra/commit/0504bf5e8cffc571a4b343326178de371e6f859b), [`5afe423`](https://github.com/mastra-ai/mastra/commit/5afe423e4badf040f1b0d4525183a856fcb8146e), [`86623c1`](https://github.com/mastra-ai/mastra/commit/86623c1adf7d22de32cc916dda17f4155184db36), [`8c9f1c0`](https://github.com/mastra-ai/mastra/commit/8c9f1c0361d89066f9bcd14a2f69e761b01766c8)]:
|
|
8
|
+
- @mastra/core@1.47.0-alpha.2
|
|
9
|
+
|
|
3
10
|
## 1.2.2-alpha.2
|
|
4
11
|
|
|
5
12
|
### Patch Changes
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@mastra/mcp-docs-server",
|
|
3
|
-
"version": "1.2.2-alpha.
|
|
3
|
+
"version": "1.2.2-alpha.4",
|
|
4
4
|
"description": "MCP server for accessing Mastra.ai documentation, changelogs, and news.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "dist/index.js",
|
|
@@ -28,8 +28,8 @@
|
|
|
28
28
|
"jsdom": "^26.1.0",
|
|
29
29
|
"local-pkg": "^1.1.2",
|
|
30
30
|
"zod": "^4.4.3",
|
|
31
|
-
"@mastra/
|
|
32
|
-
"@mastra/
|
|
31
|
+
"@mastra/core": "1.47.0-alpha.2",
|
|
32
|
+
"@mastra/mcp": "^1.12.0"
|
|
33
33
|
},
|
|
34
34
|
"devDependencies": {
|
|
35
35
|
"@hono/node-server": "^1.19.11",
|
|
@@ -45,9 +45,9 @@
|
|
|
45
45
|
"tsx": "^4.22.4",
|
|
46
46
|
"typescript": "^6.0.3",
|
|
47
47
|
"vitest": "4.1.8",
|
|
48
|
-
"@
|
|
48
|
+
"@mastra/core": "1.47.0-alpha.2",
|
|
49
49
|
"@internal/types-builder": "0.0.83",
|
|
50
|
-
"@
|
|
50
|
+
"@internal/lint": "0.0.108"
|
|
51
51
|
},
|
|
52
52
|
"homepage": "https://mastra.ai",
|
|
53
53
|
"repository": {
|