@mastra/mcp-docs-server 1.2.1 → 1.2.2-alpha.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.docs/docs/agents/code-mode.md +1 -1
- package/.docs/docs/agents/goals.md +1 -1
- package/.docs/docs/agents/processors.md +1 -1
- package/.docs/docs/agents/signal-providers.md +1 -1
- package/.docs/docs/agents/signals.md +1 -1
- package/.docs/docs/agents/supervisor-agents.md +15 -0
- package/.docs/docs/evals/built-in-scorers.md +17 -1
- package/.docs/docs/evals/overview.md +1 -0
- package/.docs/docs/evals/quick-checks.md +136 -0
- package/.docs/docs/harness/overview.md +1 -1
- package/.docs/docs/harness/session.md +7 -3
- package/.docs/docs/harness/threads-and-state.md +2 -0
- package/.docs/docs/memory/observational-memory.md +1 -3
- package/.docs/guides/guide/signal-provider.md +1 -1
- package/.docs/reference/agents/generate.md +1 -1
- package/.docs/reference/evals/checks.md +210 -0
- package/.docs/reference/harness/harness-class.md +28 -1
- package/.docs/reference/harness/session.md +20 -4
- package/.docs/reference/index.md +1 -0
- package/.docs/reference/memory/observational-memory.md +1 -1
- package/.docs/reference/memory/serialized-memory-config.md +1 -1
- package/.docs/reference/streaming/agents/stream.md +1 -1
- package/CHANGELOG.md +14 -0
- package/package.json +4 -4
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
**Added in:** `@mastra/core@1.38.0`
|
|
4
4
|
|
|
5
|
-
> **
|
|
5
|
+
> **Beta:** This feature is in beta. Breaking changes may occur without a major version bump until the API is stable.
|
|
6
6
|
|
|
7
7
|
Code mode lets an agent run multi-tool computations in an isolated sandbox and return the result as a single, more accurate response. Instead of calling tools one turn at a time, the model writes a tailored function for the user query that orchestrates your existing tools as `external_*` functions, reduces or aggregates their results, and returns one structured answer.
|
|
8
8
|
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
**Added in:** `@mastra/core@1.42.0`
|
|
4
4
|
|
|
5
|
-
> **
|
|
5
|
+
> **Beta:** The Goals feature is in beta stage and subject to breaking changes in minor versions until it graduates from its beta status.
|
|
6
6
|
|
|
7
7
|
A goal is a durable, thread-scoped objective: a standing instruction the agent keeps working toward across loop iterations until a judge model decides it is satisfied or a run budget is exhausted. The objective is persisted in thread state, so it survives reloads and is evaluated in-loop — even when a new message arrives in the middle of an already-running turn.
|
|
8
8
|
|
|
@@ -427,7 +427,7 @@ See the [`ProviderHistoryCompat` reference](https://mastra.ai/reference/processo
|
|
|
427
427
|
|
|
428
428
|
## Response caching
|
|
429
429
|
|
|
430
|
-
> **
|
|
430
|
+
> **Beta:** This feature is in beta. Breaking changes may occur without a major version bump until the API is stable.
|
|
431
431
|
|
|
432
432
|
Response caching skips the LLM call and replays a previously cached response when an agent receives an identical request. Use it to reduce latency and avoid paying for repeated calls.
|
|
433
433
|
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
**Added in:** `@mastra/core@1.39.0`
|
|
4
4
|
|
|
5
|
-
> **
|
|
5
|
+
> **Beta:** This feature is in beta. Breaking changes may occur without a major version bump until the API is stable.
|
|
6
6
|
|
|
7
7
|
A signal provider monitors an external source, such as GitHub, Slack, continuous integration (CI), or your own API, and pushes [notification signals](https://mastra.ai/docs/agents/signals) into subscribed agent threads.
|
|
8
8
|
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
**Added in:** `@mastra/core@1.39.0`
|
|
4
4
|
|
|
5
|
-
> **
|
|
5
|
+
> **Beta:** This feature is in beta. Breaking changes may occur without a major version bump until the API is stable.
|
|
6
6
|
|
|
7
7
|
Signals are a way to interact with an agent through a thread. Instead of starting every interaction with `agent.stream()`, subscribe to a thread and send messages or signals. Mastra either wakes the agent when the thread is idle, drops input into the running agent loop, or queues input for the next turn.
|
|
8
8
|
|
|
@@ -261,6 +261,21 @@ for await (const chunk of stream.fullStream) {
|
|
|
261
261
|
}
|
|
262
262
|
```
|
|
263
263
|
|
|
264
|
+
## Cancellation
|
|
265
|
+
|
|
266
|
+
When you pass an `abortSignal` to the supervisor's [`stream()`](https://mastra.ai/reference/streaming/agents/stream) or [`generate()`](https://mastra.ai/reference/agents/generate) call, Mastra forwards that same signal to delegated subagents. Calling `AbortController.abort()` cancels in-flight subagent runs at their next step instead of letting them run to completion.
|
|
267
|
+
|
|
268
|
+
```typescript
|
|
269
|
+
const controller = new AbortController()
|
|
270
|
+
|
|
271
|
+
const stream = await supervisor.stream('Research AI trends', {
|
|
272
|
+
abortSignal: controller.signal,
|
|
273
|
+
})
|
|
274
|
+
|
|
275
|
+
// Cancel the supervisor and any in-flight subagents
|
|
276
|
+
controller.abort()
|
|
277
|
+
```
|
|
278
|
+
|
|
264
279
|
## Task completion scoring
|
|
265
280
|
|
|
266
281
|
Agents don't always produce a complete, correct output on the first try. Task completion scorers can help with that by validating whether the task is complete after each iteration. If validation fails, the supervisor continues iterating. Feedback from failed scorers is included in the conversation context so subagents can see what was missing.
|
|
@@ -45,4 +45,20 @@ These scorers evaluate adherence to format, style, and safety requirements:
|
|
|
45
45
|
- [`tone-consistency`](https://mastra.ai/reference/evals/tone-consistency): Measures consistency in formality, complexity, and style (`0-1`, higher is better)
|
|
46
46
|
- [`toxicity`](https://mastra.ai/reference/evals/toxicity): Detects harmful or inappropriate content (`0-1`, lower is better)
|
|
47
47
|
- [`bias`](https://mastra.ai/reference/evals/bias): Detects potential biases in the output (`0-1`, lower is better)
|
|
48
|
-
- [`keyword-coverage`](https://mastra.ai/reference/evals/keyword-coverage): Assesses technical terminology usage (`0-1`, higher is better)
|
|
48
|
+
- [`keyword-coverage`](https://mastra.ai/reference/evals/keyword-coverage): Assesses technical terminology usage (`0-1`, higher is better)
|
|
49
|
+
|
|
50
|
+
### Quick Checks
|
|
51
|
+
|
|
52
|
+
Zero-LLM micro-scorers for fast, deterministic assertions. See the [Quick Checks](https://mastra.ai/docs/evals/quick-checks) guide for usage details.
|
|
53
|
+
|
|
54
|
+
- [`checks.includes`](https://mastra.ai/reference/evals/checks): Scores 1 if output contains a substring (`0` or `1`)
|
|
55
|
+
- [`checks.excludes`](https://mastra.ai/reference/evals/checks): Scores 1 if output does not contain a substring (`0` or `1`)
|
|
56
|
+
- [`checks.equals`](https://mastra.ai/reference/evals/checks): Scores 1 if output exactly matches a string (`0` or `1`)
|
|
57
|
+
- [`checks.matches`](https://mastra.ai/reference/evals/checks): Scores 1 if output matches a regex (`0` or `1`)
|
|
58
|
+
- [`checks.similarity`](https://mastra.ai/reference/evals/checks): String similarity via Dice coefficient (`0-1`, or binary with threshold)
|
|
59
|
+
- [`checks.calledTool`](https://mastra.ai/reference/evals/checks): Scores 1 if a tool was called at least N times (`0` or `1`)
|
|
60
|
+
- [`checks.didNotCall`](https://mastra.ai/reference/evals/checks): Scores 1 if a tool was not called (`0` or `1`)
|
|
61
|
+
- [`checks.toolOrder`](https://mastra.ai/reference/evals/checks): Scores 1 if tools were called in order (`0` or `1`)
|
|
62
|
+
- [`checks.maxToolCalls`](https://mastra.ai/reference/evals/checks): Scores 1 if tool call count is within limit (`0` or `1`)
|
|
63
|
+
- [`checks.usedNoTools`](https://mastra.ai/reference/evals/checks): Scores 1 if no tools were called (`0` or `1`)
|
|
64
|
+
- [`checks.noToolErrors`](https://mastra.ai/reference/evals/checks): Scores 1 if no tool calls had errors (`0` or `1`)
|
|
@@ -135,6 +135,7 @@ Once registered, you can score traces interactively within Studio under the **Ob
|
|
|
135
135
|
|
|
136
136
|
## Next steps
|
|
137
137
|
|
|
138
|
+
- Use [Quick Checks](https://mastra.ai/docs/evals/quick-checks) for fast, deterministic assertions on text and tool usage
|
|
138
139
|
- Learn how to create your own scorers in the [Creating Custom Scorers](https://mastra.ai/docs/evals/custom-scorers) guide
|
|
139
140
|
- Explore built-in scorers in the [Built-in Scorers](https://mastra.ai/docs/evals/built-in-scorers) section
|
|
140
141
|
- Test scorers with [Studio](https://mastra.ai/docs/studio/overview)
|
|
@@ -0,0 +1,136 @@
|
|
|
1
|
+
# Quick Checks
|
|
2
|
+
|
|
3
|
+
Quick Checks are composable micro-scorers for common assertions like "output contains X" or "agent called tool Y." They require no LLM, run instantly, and plug into the same `scorers: [...]` array as any other scorer.
|
|
4
|
+
|
|
5
|
+
## When to use Quick Checks
|
|
6
|
+
|
|
7
|
+
Use Quick Checks when you need fast, deterministic assertions:
|
|
8
|
+
|
|
9
|
+
- Verify output text contains or excludes specific strings
|
|
10
|
+
- Confirm an agent called (or avoided) specific tools
|
|
11
|
+
- Validate tool call ordering and count limits
|
|
12
|
+
- Gate CI pipelines with zero-cost binary checks
|
|
13
|
+
- Combine with LLM-based scorers for layered evaluation
|
|
14
|
+
|
|
15
|
+
For subjective or semantic evaluation, use [LLM-based scorers](https://mastra.ai/docs/evals/built-in-scorers) instead.
|
|
16
|
+
|
|
17
|
+
## Quickstart
|
|
18
|
+
|
|
19
|
+
```typescript
|
|
20
|
+
import { checks } from '@mastra/evals/checks'
|
|
21
|
+
import { runEvals } from '@mastra/core/evals'
|
|
22
|
+
import { weatherAgent } from '../agents'
|
|
23
|
+
|
|
24
|
+
const result = await runEvals({
|
|
25
|
+
data: [{ input: 'What is the weather in Brooklyn?' }],
|
|
26
|
+
target: weatherAgent,
|
|
27
|
+
scorers: [checks.includes('Brooklyn'), checks.calledTool('get_weather'), checks.noToolErrors()],
|
|
28
|
+
})
|
|
29
|
+
|
|
30
|
+
console.log(result.scores)
|
|
31
|
+
// { 'check-includes': 1, 'check-called-tool': 1, 'check-no-tool-errors': 1 }
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
## Available checks
|
|
35
|
+
|
|
36
|
+
Quick Checks fall into two categories:
|
|
37
|
+
|
|
38
|
+
### Text checks
|
|
39
|
+
|
|
40
|
+
These scorers evaluate the agent's text output:
|
|
41
|
+
|
|
42
|
+
| Check | What it does | Score |
|
|
43
|
+
| ------------------------ | ------------------------------------- | -------------------------------- |
|
|
44
|
+
| `checks.includes(str)` | Output contains substring | 1 or 0 |
|
|
45
|
+
| `checks.excludes(str)` | Output does not contain substring | 1 or 0 |
|
|
46
|
+
| `checks.equals(str)` | Output exactly equals string | 1 or 0 |
|
|
47
|
+
| `checks.matches(regex)` | Output matches regular expression | 1 or 0 |
|
|
48
|
+
| `checks.similarity(str)` | Dice coefficient similarity to string | 0-1 (or binary with `threshold`) |
|
|
49
|
+
|
|
50
|
+
### Tool call checks
|
|
51
|
+
|
|
52
|
+
These scorers evaluate tool usage from the agent's run:
|
|
53
|
+
|
|
54
|
+
| Check | What it does | Score |
|
|
55
|
+
| ------------------------- | -------------------------------- | ------ |
|
|
56
|
+
| `checks.calledTool(name)` | Tool was called at least N times | 1 or 0 |
|
|
57
|
+
| `checks.didNotCall(name)` | Tool was not called | 1 or 0 |
|
|
58
|
+
| `checks.toolOrder([...])` | Tools called in expected order | 1 or 0 |
|
|
59
|
+
| `checks.maxToolCalls(n)` | No more than N tool calls total | 1 or 0 |
|
|
60
|
+
| `checks.usedNoTools()` | No tools called at all | 1 or 0 |
|
|
61
|
+
| `checks.noToolErrors()` | No tool invocations had errors | 1 or 0 |
|
|
62
|
+
|
|
63
|
+
## Combining checks with LLM scorers
|
|
64
|
+
|
|
65
|
+
Checks compose with LLM-based scorers in a single `runEvals` call. Use checks for deterministic gates and LLM scorers for qualitative evaluation:
|
|
66
|
+
|
|
67
|
+
```typescript
|
|
68
|
+
import { checks } from '@mastra/evals/checks'
|
|
69
|
+
import { createFaithfulnessScorer } from '@mastra/evals/scorers/prebuilt'
|
|
70
|
+
import { runEvals } from '@mastra/core/evals'
|
|
71
|
+
import { myAgent } from '../agents'
|
|
72
|
+
|
|
73
|
+
const result = await runEvals({
|
|
74
|
+
data: [
|
|
75
|
+
{
|
|
76
|
+
input: 'What is the weather in Brooklyn?',
|
|
77
|
+
context: ['Brooklyn weather data: sunny, 72°F'],
|
|
78
|
+
},
|
|
79
|
+
],
|
|
80
|
+
target: myAgent,
|
|
81
|
+
scorers: [
|
|
82
|
+
// Deterministic checks (instant, free)
|
|
83
|
+
checks.includes('Brooklyn'),
|
|
84
|
+
checks.calledTool('get_weather'),
|
|
85
|
+
checks.excludes('error'),
|
|
86
|
+
checks.noToolErrors(),
|
|
87
|
+
|
|
88
|
+
// LLM-based scorer (semantic, costs tokens)
|
|
89
|
+
createFaithfulnessScorer({ model: 'openai/gpt-5-mini' }),
|
|
90
|
+
],
|
|
91
|
+
})
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
## Using checks in live scoring
|
|
95
|
+
|
|
96
|
+
Attach checks to agents for continuous monitoring:
|
|
97
|
+
|
|
98
|
+
```typescript
|
|
99
|
+
import { Agent } from '@mastra/core/agent'
|
|
100
|
+
import { checks } from '@mastra/evals/checks'
|
|
101
|
+
|
|
102
|
+
export const weatherAgent = new Agent({
|
|
103
|
+
name: 'Weather Agent',
|
|
104
|
+
instructions: 'Answer weather questions using the get_weather tool.',
|
|
105
|
+
model: 'openai/gpt-5.5',
|
|
106
|
+
tools: { get_weather: weatherTool },
|
|
107
|
+
scorers: {
|
|
108
|
+
noErrors: {
|
|
109
|
+
scorer: checks.noToolErrors(),
|
|
110
|
+
sampling: { type: 'ratio', rate: 1 },
|
|
111
|
+
},
|
|
112
|
+
mentionCity: {
|
|
113
|
+
scorer: checks.includes('Brooklyn'),
|
|
114
|
+
sampling: { type: 'ratio', rate: 0.5 },
|
|
115
|
+
},
|
|
116
|
+
},
|
|
117
|
+
})
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
## How checks work
|
|
121
|
+
|
|
122
|
+
Each check is a standard `createScorer()` instance with a `preprocess` step and a `generateScore` step. They follow the same [four-step pipeline](https://mastra.ai/docs/evals/custom-scorers) as any other scorer:
|
|
123
|
+
|
|
124
|
+
1. **preprocess**: Extracts and normalizes relevant data from the agent run (text content, tool calls)
|
|
125
|
+
2. **generateScore**: Converts the preprocessed result into a score (typically binary 1 or 0)
|
|
126
|
+
|
|
127
|
+
Because checks skip the `analyze` and `generateReason` steps and make no LLM calls, they run in microseconds.
|
|
128
|
+
|
|
129
|
+
> **Note:** Visit the [Quick Checks reference](https://mastra.ai/reference/evals/checks) for the full API, including all parameters and options for each check.
|
|
130
|
+
|
|
131
|
+
## Related
|
|
132
|
+
|
|
133
|
+
- [Quick Checks reference](https://mastra.ai/reference/evals/checks)
|
|
134
|
+
- [Built-in Scorers](https://mastra.ai/docs/evals/built-in-scorers)
|
|
135
|
+
- [Custom Scorers](https://mastra.ai/docs/evals/custom-scorers)
|
|
136
|
+
- [`runEvals()` reference](https://mastra.ai/reference/evals/run-evals)
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Harness overview
|
|
2
2
|
|
|
3
|
-
> **
|
|
3
|
+
> **Beta:** The `Harness` feature is in beta stage and subject to breaking changes in minor versions until it graduates from its beta status.
|
|
4
4
|
|
|
5
5
|
The Harness is a session controller for building interactive agent applications. It handles the runtime concerns that sit between your UI and the agent loop: managing conversation threads, switching between agent modes, persisting state, gating tool execution with approvals, and coordinating subagents. You can focus on what your agent does rather than how to wire it together.
|
|
6
6
|
|
|
@@ -10,7 +10,7 @@ Because each session has its own identity and state, one Harness can serve many
|
|
|
10
10
|
|
|
11
11
|
The rest of this page walks through the Session's state. Each part is a focused sub-object on `harness.session`:
|
|
12
12
|
|
|
13
|
-
- **Who and where**: [Identity](#identity) sets the resource the session belongs to, and [Thread](#thread) tracks the currently bound thread.
|
|
13
|
+
- **Who and where**: [Identity](#identity) sets the stable session `id`, `ownerId`, and resource the session belongs to, and [Thread](#thread) tracks the currently bound thread.
|
|
14
14
|
- **How the agent behaves**: [Mode and model](#mode-and-model) controls which agent profile and model are active, and [Permission grants](#permission-grants) records what the user has approved to run without prompting.
|
|
15
15
|
- **What's happening right now**: [Run state](#run-state) reflects the in-flight generation, and [Follow-ups](#follow-ups) holds messages queued to send when it finishes.
|
|
16
16
|
- **What you store and render**: [State](#state) holds your application's structured data, and [Display state](#display-state) is the single snapshot your UI renders from.
|
|
@@ -19,14 +19,18 @@ The Harness performs actions that change this state — switching threads, modes
|
|
|
19
19
|
|
|
20
20
|
## Identity
|
|
21
21
|
|
|
22
|
-
`session.identity` holds
|
|
22
|
+
`session.identity` holds three stable identifiers for the conversation: a session `id`, an `ownerId`, and the resource ID. Threads are scoped to a resource ID, so this is what groups a conversation's threads together:
|
|
23
23
|
|
|
24
24
|
```typescript
|
|
25
|
+
const sessionId = harness.session.identity.getId()
|
|
26
|
+
const ownerId = harness.session.identity.getOwnerId()
|
|
25
27
|
const resourceId = harness.session.identity.getResourceId()
|
|
26
28
|
const threadId = harness.session.thread.getId()
|
|
27
29
|
```
|
|
28
30
|
|
|
29
|
-
The
|
|
31
|
+
The session `id` and `ownerId` are stable for the life of the session — they don't change when the resource ID is switched. They mirror the `id` and `ownerId` fields on `SessionRecord` in storage, so storage layers can key sessions by a stable identifier rather than the mutable resource ID.
|
|
32
|
+
|
|
33
|
+
The resource ID is set when creating a session via [`harness.createSession()`](https://mastra.ai/reference/harness/harness-class) and defaults to the harness `id`. See [Resource IDs](https://mastra.ai/docs/harness/threads-and-state) for how it scopes threads.
|
|
30
34
|
|
|
31
35
|
## Thread
|
|
32
36
|
|
|
@@ -131,6 +131,8 @@ The resource ID is part of a conversation's identity, so you read it from the Se
|
|
|
131
131
|
const resourceId = harness.session.identity.getResourceId()
|
|
132
132
|
```
|
|
133
133
|
|
|
134
|
+
The session also has a stable `id` and `ownerId` (read with `session.identity.getId()` and `session.identity.getOwnerId()`). Unlike the resource ID, these don't change when you switch resources — see [Session identity](https://mastra.ai/docs/harness/session) for details.
|
|
135
|
+
|
|
134
136
|
## Related
|
|
135
137
|
|
|
136
138
|
- [Harness overview](https://mastra.ai/docs/harness/overview)
|
|
@@ -210,9 +210,7 @@ The result is a three-tier system:
|
|
|
210
210
|
2. **Observations**: A log of what the Observer has seen
|
|
211
211
|
3. **Reflections**: Condensed observations when memory becomes too long
|
|
212
212
|
|
|
213
|
-
### Retrieval mode
|
|
214
|
-
|
|
215
|
-
> **Note:** Retrieval mode is experimental. The API may change in future releases.
|
|
213
|
+
### Retrieval mode
|
|
216
214
|
|
|
217
215
|
Normal OM compresses messages into observations, which is great for staying on task, but the original wording is gone. Retrieval mode fixes this by keeping each observation group linked to the raw messages that produced it. When the agent needs exact wording, tool output, or chronology that the summary compressed away, it can call a `recall` tool to page through the source messages.
|
|
218
216
|
|
|
@@ -4,7 +4,7 @@ In this guide, you'll build a signal provider that polls an external service on
|
|
|
4
4
|
|
|
5
5
|
The example watches build pipelines in a fake CI service, but the pattern applies to any pull-based source: an issue tracker, a status API, a queue, or your own backend.
|
|
6
6
|
|
|
7
|
-
> **
|
|
7
|
+
> **Beta:** Signal providers are in beta. Breaking changes may occur without a major version bump until the API is stable.
|
|
8
8
|
|
|
9
9
|
## Prerequisites
|
|
10
10
|
|
|
@@ -170,7 +170,7 @@ const response = await agent.generate('Help me organize my day', {
|
|
|
170
170
|
|
|
171
171
|
**options.activeTools** (`Array<keyof ToolSet> | undefined`): Array of tool names that should be active during execution. If undefined, all available tools are active.
|
|
172
172
|
|
|
173
|
-
**options.abortSignal** (`AbortSignal`): Signal object that allows you to abort the agent's execution. When the signal is aborted, all ongoing operations will be terminated.
|
|
173
|
+
**options.abortSignal** (`AbortSignal`): Signal object that allows you to abort the agent's execution. When the signal is aborted, all ongoing operations will be terminated, including any in-flight subagent runs the agent delegated to.
|
|
174
174
|
|
|
175
175
|
**options.prepareStep** (`PrepareStepFunction`): Callback function called before each step of multi-step execution.
|
|
176
176
|
|
|
@@ -0,0 +1,210 @@
|
|
|
1
|
+
# Quick Checks
|
|
2
|
+
|
|
3
|
+
Quick Checks are zero-LLM, composable micro-scorers for common assertions. They plug into the existing `scorers: [...]` array anywhere scorers are used — in `runEvals`, live scoring, experiments, and Studio.
|
|
4
|
+
|
|
5
|
+
Internally they are standard `createScorer()` instances, so they have the same observability, storage, and pipeline integration as any other scorer.
|
|
6
|
+
|
|
7
|
+
## Usage example
|
|
8
|
+
|
|
9
|
+
```typescript
|
|
10
|
+
import { checks } from '@mastra/evals/checks'
|
|
11
|
+
import { runEvals } from '@mastra/core/evals'
|
|
12
|
+
import { myAgent } from '../agents'
|
|
13
|
+
|
|
14
|
+
const result = await runEvals({
|
|
15
|
+
data: [{ input: 'What is the weather in Brooklyn?' }],
|
|
16
|
+
target: myAgent,
|
|
17
|
+
scorers: [
|
|
18
|
+
checks.includes('sunny'),
|
|
19
|
+
checks.calledTool('get_weather'),
|
|
20
|
+
checks.toolOrder(['get_weather', 'summarize']),
|
|
21
|
+
checks.noToolErrors(),
|
|
22
|
+
],
|
|
23
|
+
})
|
|
24
|
+
|
|
25
|
+
console.log(result.scores)
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
## Text checks
|
|
29
|
+
|
|
30
|
+
### `checks.includes(expected, options?)`
|
|
31
|
+
|
|
32
|
+
Scores 1 if the agent's output text contains the expected substring, 0 otherwise.
|
|
33
|
+
|
|
34
|
+
```typescript
|
|
35
|
+
checks.includes('sunny')
|
|
36
|
+
checks.includes('Sunny', { ignoreCase: false })
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
**expected** (`string`): Substring to search for in the output.
|
|
40
|
+
|
|
41
|
+
**options.ignoreCase** (`boolean`): Case-insensitive matching. (Default: `true`)
|
|
42
|
+
|
|
43
|
+
Returns: `1` if found, `0` otherwise.
|
|
44
|
+
|
|
45
|
+
### `checks.excludes(unwanted, options?)`
|
|
46
|
+
|
|
47
|
+
Scores 1 if the agent's output text does NOT contain the substring, 0 otherwise.
|
|
48
|
+
|
|
49
|
+
```typescript
|
|
50
|
+
checks.excludes('error')
|
|
51
|
+
checks.excludes('Error', { ignoreCase: false })
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
**unwanted** (`string`): Substring that must not appear in the output.
|
|
55
|
+
|
|
56
|
+
**options.ignoreCase** (`boolean`): Case-insensitive matching. (Default: `true`)
|
|
57
|
+
|
|
58
|
+
Returns: `1` if absent, `0` otherwise.
|
|
59
|
+
|
|
60
|
+
### `checks.equals(expected, options?)`
|
|
61
|
+
|
|
62
|
+
Scores 1 if the output text exactly equals the expected string after optional normalization.
|
|
63
|
+
|
|
64
|
+
```typescript
|
|
65
|
+
checks.equals('Hello, world!')
|
|
66
|
+
checks.equals('Hello', { ignoreCase: false })
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
**expected** (`string`): Exact string the output must match.
|
|
70
|
+
|
|
71
|
+
**options.ignoreCase** (`boolean`): Case-insensitive matching. (Default: `true`)
|
|
72
|
+
|
|
73
|
+
Returns: `1` if equal, `0` otherwise.
|
|
74
|
+
|
|
75
|
+
### `checks.matches(pattern, options?)`
|
|
76
|
+
|
|
77
|
+
Scores 1 if the output matches the given regular expression.
|
|
78
|
+
|
|
79
|
+
```typescript
|
|
80
|
+
checks.matches(/\d+°[FC]/)
|
|
81
|
+
checks.matches(/^hello$/, { exact: true })
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
**pattern** (`RegExp`): Regular expression to test against the output.
|
|
85
|
+
|
|
86
|
+
**options.exact** (`boolean`): Anchor the pattern to match the entire output (adds ^ and $). (Default: `false`)
|
|
87
|
+
|
|
88
|
+
Returns: `1` if matched, `0` otherwise.
|
|
89
|
+
|
|
90
|
+
### `checks.similarity(expected, options?)`
|
|
91
|
+
|
|
92
|
+
Returns the string similarity score (0-1) between the output and an expected string using the Dice coefficient. When a `threshold` is set, returns binary 1/0 instead.
|
|
93
|
+
|
|
94
|
+
```typescript
|
|
95
|
+
checks.similarity('Sunny, 72°F')
|
|
96
|
+
checks.similarity('Sunny, 72°F', { threshold: 0.7 })
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
**expected** (`string`): Reference string to compare against.
|
|
100
|
+
|
|
101
|
+
**options.threshold** (`number`): Minimum similarity score (0-1) to return 1. When omitted, returns the raw similarity score.
|
|
102
|
+
|
|
103
|
+
**options.ignoreCase** (`boolean`): Case-insensitive comparison. (Default: `true`)
|
|
104
|
+
|
|
105
|
+
Returns: Raw similarity score (0-1), or binary `1`/`0` when `threshold` is set.
|
|
106
|
+
|
|
107
|
+
## Tool call checks
|
|
108
|
+
|
|
109
|
+
### `checks.calledTool(toolName, options?)`
|
|
110
|
+
|
|
111
|
+
Scores 1 if the agent called the specified tool at least the required number of times.
|
|
112
|
+
|
|
113
|
+
```typescript
|
|
114
|
+
checks.calledTool('get_weather')
|
|
115
|
+
checks.calledTool('search', { times: 2 })
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
**toolName** (`string`): Name of the tool to look for.
|
|
119
|
+
|
|
120
|
+
**options.times** (`number`): Minimum number of times the tool must be called. (Default: `1`)
|
|
121
|
+
|
|
122
|
+
Returns: `1` if called at least `times` times, `0` otherwise.
|
|
123
|
+
|
|
124
|
+
### `checks.didNotCall(toolName)`
|
|
125
|
+
|
|
126
|
+
Scores 1 if the agent did NOT call the specified tool.
|
|
127
|
+
|
|
128
|
+
```typescript
|
|
129
|
+
checks.didNotCall('delete_user')
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
**toolName** (`string`): Name of the tool that must not appear.
|
|
133
|
+
|
|
134
|
+
Returns: `1` if the tool was not called, `0` otherwise.
|
|
135
|
+
|
|
136
|
+
### `checks.toolOrder(expectedOrder)`
|
|
137
|
+
|
|
138
|
+
Scores 1 if the tools were called in the specified order. Uses relaxed matching — other tool calls between the expected tools are allowed.
|
|
139
|
+
|
|
140
|
+
```typescript
|
|
141
|
+
checks.toolOrder(['search', 'summarize', 'respond'])
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
**expectedOrder** (`string[]`): Tool names in the expected calling sequence. Must appear as a subsequence of the actual tool calls.
|
|
145
|
+
|
|
146
|
+
Returns: `1` if the expected order is satisfied, `0` otherwise.
|
|
147
|
+
|
|
148
|
+
### `checks.maxToolCalls(max)`
|
|
149
|
+
|
|
150
|
+
Scores 1 if the agent used no more than `max` tool calls.
|
|
151
|
+
|
|
152
|
+
```typescript
|
|
153
|
+
checks.maxToolCalls(5)
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
**max** (`number`): Maximum number of tool calls allowed.
|
|
157
|
+
|
|
158
|
+
Returns: `1` if within the limit, `0` otherwise.
|
|
159
|
+
|
|
160
|
+
### `checks.usedNoTools()`
|
|
161
|
+
|
|
162
|
+
Scores 1 if the agent made no tool calls at all.
|
|
163
|
+
|
|
164
|
+
```typescript
|
|
165
|
+
checks.usedNoTools()
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
Returns: `1` if no tools were called, `0` otherwise.
|
|
169
|
+
|
|
170
|
+
### `checks.noToolErrors()`
|
|
171
|
+
|
|
172
|
+
Scores 1 if none of the tool invocations resulted in an error state. Detects both error results (`result.error` present) and incomplete tool calls (`state === 'call'`).
|
|
173
|
+
|
|
174
|
+
```typescript
|
|
175
|
+
checks.noToolErrors()
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
Returns: `1` if all tool calls succeeded, `0` otherwise.
|
|
179
|
+
|
|
180
|
+
## Combining checks with other scorers
|
|
181
|
+
|
|
182
|
+
Checks compose with LLM-based and code-based scorers in the same `scorers` array:
|
|
183
|
+
|
|
184
|
+
```typescript
|
|
185
|
+
import { checks } from '@mastra/evals/checks'
|
|
186
|
+
import { createAnswerRelevancyScorer } from '@mastra/evals/scorers/prebuilt'
|
|
187
|
+
import { runEvals } from '@mastra/core/evals'
|
|
188
|
+
import { myAgent } from '../agents'
|
|
189
|
+
|
|
190
|
+
const result = await runEvals({
|
|
191
|
+
data: [{ input: 'What is the weather in Brooklyn?' }],
|
|
192
|
+
target: myAgent,
|
|
193
|
+
scorers: [
|
|
194
|
+
// Zero-LLM checks
|
|
195
|
+
checks.includes('Brooklyn'),
|
|
196
|
+
checks.calledTool('get_weather'),
|
|
197
|
+
checks.noToolErrors(),
|
|
198
|
+
// LLM-based scorer
|
|
199
|
+
createAnswerRelevancyScorer({ model: 'openai/gpt-5-mini' }),
|
|
200
|
+
],
|
|
201
|
+
})
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
## Related
|
|
205
|
+
|
|
206
|
+
- [Quick Checks overview](https://mastra.ai/docs/evals/quick-checks)
|
|
207
|
+
- [Built-in Scorers](https://mastra.ai/docs/evals/built-in-scorers)
|
|
208
|
+
- [`createScorer()` reference](https://mastra.ai/reference/evals/create-scorer)
|
|
209
|
+
- [`runEvals()` reference](https://mastra.ai/reference/evals/run-evals)
|
|
210
|
+
- [Custom Scorers](https://mastra.ai/docs/evals/custom-scorers)
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Harness class
|
|
2
2
|
|
|
3
|
-
> **
|
|
3
|
+
> **Beta:** The `Harness` feature is in beta stage and subject to breaking changes in minor versions until it graduates from its beta status.
|
|
4
4
|
|
|
5
5
|
The `Harness` class orchestrates multiple agent modes, shared state, memory, and storage. It provides a control layer that a TUI or other UI can drive to manage threads, switch models and modes, send messages, handle tool approvals, and track events.
|
|
6
6
|
|
|
@@ -166,6 +166,33 @@ Initialize the harness. Loads storage, initializes the workspace, propagates mem
|
|
|
166
166
|
await harness.init()
|
|
167
167
|
```
|
|
168
168
|
|
|
169
|
+
#### `createSession({ id, ownerId, resourceId? })`
|
|
170
|
+
|
|
171
|
+
Create a new, fully-wired `Session` and bring it online. The session starts in the default mode with the seeded model, connects to the Harness's shared machinery (agent, storage/lock, config catalog), and has a current thread (the most recent thread for the resource, or a freshly created one). Call `init()` once before creating sessions so shared storage and workspace are ready.
|
|
172
|
+
|
|
173
|
+
The Harness owns no session of its own — every consumer creates its own session and drives all work through it. In a server or multiplayer setting, each request, thread, or user gets its own session, isolated from every other: independent event bus, mode, model, state, and current thread.
|
|
174
|
+
|
|
175
|
+
`id` and `ownerId` are required — they mirror `SessionRecord.id` and `SessionRecord.ownerId` and are stable for the life of the session. `resourceId` is optional and defaults to `config.resourceId` then `config.id`.
|
|
176
|
+
|
|
177
|
+
```typescript
|
|
178
|
+
const session = await harness.createSession({
|
|
179
|
+
id: 'session-xyz',
|
|
180
|
+
ownerId: 'user-123',
|
|
181
|
+
})
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
You can also override the resource:
|
|
185
|
+
|
|
186
|
+
```typescript
|
|
187
|
+
const session = await harness.createSession({
|
|
188
|
+
id: 'session-xyz',
|
|
189
|
+
ownerId: 'user-123',
|
|
190
|
+
resourceId: 'project-abc',
|
|
191
|
+
})
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
Switching the resource ID via `harness.setResourceId()` changes only the resource ID, not `id` or `ownerId`. Read them through `session.identity.getId()` and `session.identity.getOwnerId()`.
|
|
195
|
+
|
|
169
196
|
#### `selectOrCreateThread()`
|
|
170
197
|
|
|
171
198
|
Select the most recent thread for the current resource, or create one if none exist. Loads thread metadata and acquires a thread lock.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Session class
|
|
2
2
|
|
|
3
|
-
> **
|
|
3
|
+
> **Beta:** The `Harness` feature is in beta stage and subject to breaking changes in minor versions until it graduates from its beta status.
|
|
4
4
|
|
|
5
5
|
A `Session` owns all the state tied to a single conversation. The [`Harness`](https://mastra.ai/reference/harness/harness-class) is the shared host — agents, storage, config, the thread lock, and the event bus — while the `Session` holds everything that is per-conversation: identity, the active thread binding and reads, mode and model selection, run and abort state, the live agent stream, tool suspensions, follow-ups, approvals, permission grants, token usage, and the display-state snapshot.
|
|
6
6
|
|
|
@@ -29,7 +29,7 @@ harness.subscribe(event => {
|
|
|
29
29
|
|
|
30
30
|
The session is organized into sub-objects, each owning one domain of per-conversation state.
|
|
31
31
|
|
|
32
|
-
**identity** (`SessionIdentity`):
|
|
32
|
+
**identity** (`SessionIdentity`): Stable session, owner, and resource identity for the conversation. See identity methods below.
|
|
33
33
|
|
|
34
34
|
**thread** (`SessionThread`): Active thread binding and thread/message reads. See thread methods below.
|
|
35
35
|
|
|
@@ -125,7 +125,23 @@ const usage = harness.session.getTokenUsage()
|
|
|
125
125
|
|
|
126
126
|
## Identity
|
|
127
127
|
|
|
128
|
-
`session.identity` owns the resource ID for the
|
|
128
|
+
`session.identity` owns the stable identifiers for the conversation: the resource ID, a session `id`, and an `ownerId`. The `id` and `ownerId` are stable for the life of the session and do not change when the resource ID is switched. They mirror the `id` and `ownerId` fields on `SessionRecord` in storage.
|
|
129
|
+
|
|
130
|
+
### `session.identity.getId()`
|
|
131
|
+
|
|
132
|
+
Return the stable session identifier.
|
|
133
|
+
|
|
134
|
+
```typescript
|
|
135
|
+
const sessionId = harness.session.identity.getId()
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
### `session.identity.getOwnerId()`
|
|
139
|
+
|
|
140
|
+
Return the stable owner identifier for the session.
|
|
141
|
+
|
|
142
|
+
```typescript
|
|
143
|
+
const ownerId = harness.session.identity.getOwnerId()
|
|
144
|
+
```
|
|
129
145
|
|
|
130
146
|
### `session.identity.getResourceId()`
|
|
131
147
|
|
|
@@ -143,7 +159,7 @@ Return the resource ID the session was created with.
|
|
|
143
159
|
const defaultResourceId = harness.session.identity.getDefaultResourceId()
|
|
144
160
|
```
|
|
145
161
|
|
|
146
|
-
To change the resource ID, use [`harness.setResourceId()`](https://mastra.ai/reference/harness/harness-class), which also tears down the active thread.
|
|
162
|
+
To change the resource ID, use [`harness.setResourceId()`](https://mastra.ai/reference/harness/harness-class), which also tears down the active thread. The session `id` and `ownerId` are not affected by resource switches.
|
|
147
163
|
|
|
148
164
|
## Thread
|
|
149
165
|
|
package/.docs/reference/index.md
CHANGED
|
@@ -118,6 +118,7 @@ The Reference section provides documentation of Mastra's API, including paramete
|
|
|
118
118
|
- [createScorer()](https://mastra.ai/reference/evals/create-scorer)
|
|
119
119
|
- [filterRun()](https://mastra.ai/reference/evals/filter-run)
|
|
120
120
|
- [MastraScorer](https://mastra.ai/reference/evals/mastra-scorer)
|
|
121
|
+
- [Quick Checks](https://mastra.ai/reference/evals/checks)
|
|
121
122
|
- [runEvals()](https://mastra.ai/reference/evals/run-evals)
|
|
122
123
|
- [Scorer Utils](https://mastra.ai/reference/evals/scorer-utils)
|
|
123
124
|
- [Answer Relevancy Scorer](https://mastra.ai/reference/evals/answer-relevancy)
|
|
@@ -44,7 +44,7 @@ OM performs thresholding with fast local token estimation. Text uses `tokenx`, a
|
|
|
44
44
|
|
|
45
45
|
**temporalMarkers** (`boolean`): Insert temporal-gap reminder markers before new user messages when the previous message in the thread is at least 10 minutes older. The marker is persisted in memory, emitted as an inline reminder event so clients can render it specially, and shown to the observer so it can anchor observations to when events occurred. (Default: `false`)
|
|
46
46
|
|
|
47
|
-
**retrieval** (`boolean | { vector?: boolean; scope?: 'thread' | 'resource' }`):
|
|
47
|
+
**retrieval** (`boolean | { vector?: boolean; scope?: 'thread' | 'resource' }`): Enable retrieval-mode observation groups as durable pointers to raw message history. \`true\` enables cross-thread browsing by default. \`{ vector: true }\` also enables semantic search using Memory's vector store and embedder. \`{ scope: 'thread' }\` restricts the recall tool to the current thread only. Default scope is \`'resource'\`. (Default: `false`)
|
|
48
48
|
|
|
49
49
|
**observation** (`ObservationalMemoryObservationConfig`): Configuration for the observation step. Controls when the Observer agent runs and how it behaves.
|
|
50
50
|
|
|
@@ -58,7 +58,7 @@ new MastraEditor({
|
|
|
58
58
|
|
|
59
59
|
**temporalMarkers** (`boolean`): Persist inline temporal gap markers for long pauses between messages.
|
|
60
60
|
|
|
61
|
-
**retrieval** (`boolean | { vector?: boolean; scope?: 'thread' | 'resource' }`):
|
|
61
|
+
**retrieval** (`boolean | { vector?: boolean; scope?: 'thread' | 'resource' }`): Enable retrieval-mode observation groups as durable pointers to raw message history.
|
|
62
62
|
|
|
63
63
|
**observation** (`SerializedObservationalMemoryObservationConfig`): Observer-step configuration (model, token thresholds, buffering).
|
|
64
64
|
|
|
@@ -78,7 +78,7 @@ const stream = await agent.stream('message for agent')
|
|
|
78
78
|
|
|
79
79
|
**options.onAbort** (`(event: any) => Promise<void> | void`): Callback function called when the stream is aborted.
|
|
80
80
|
|
|
81
|
-
**options.abortSignal** (`AbortSignal`): Signal object that allows you to abort the agent's execution. When the signal is aborted, all ongoing operations will be terminated.
|
|
81
|
+
**options.abortSignal** (`AbortSignal`): Signal object that allows you to abort the agent's execution. When the signal is aborted, all ongoing operations will be terminated, including any in-flight subagent runs the agent delegated to.
|
|
82
82
|
|
|
83
83
|
**options.activeTools** (`Array<keyof ToolSet> | undefined`): Array of active tool names that can be used during execution.
|
|
84
84
|
|
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,19 @@
|
|
|
1
1
|
# @mastra/mcp-docs-server
|
|
2
2
|
|
|
3
|
+
## 1.2.2-alpha.2
|
|
4
|
+
|
|
5
|
+
### Patch Changes
|
|
6
|
+
|
|
7
|
+
- Updated dependencies [[`7f9ae70`](https://github.com/mastra-ai/mastra/commit/7f9ae70826b047e5a66218f9e92f20e54a2d791f), [`1505c07`](https://github.com/mastra-ai/mastra/commit/1505c07603f6346bae12aa82f140e8b88ffea9ab), [`e940f09`](https://github.com/mastra-ai/mastra/commit/e940f099ef5d18b403e6f2b4937e086a4da857b1)]:
|
|
8
|
+
- @mastra/core@1.46.1-alpha.1
|
|
9
|
+
|
|
10
|
+
## 1.2.2-alpha.0
|
|
11
|
+
|
|
12
|
+
### Patch Changes
|
|
13
|
+
|
|
14
|
+
- Updated dependencies [[`fbeda0c`](https://github.com/mastra-ai/mastra/commit/fbeda0c0f35def07e6837936dd3a003b2b7c5172), [`307573b`](https://github.com/mastra-ai/mastra/commit/307573b9ff3149b70c79540dbc86f1319b180f29)]:
|
|
15
|
+
- @mastra/core@1.46.1-alpha.0
|
|
16
|
+
|
|
3
17
|
## 1.2.1
|
|
4
18
|
|
|
5
19
|
### Patch Changes
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@mastra/mcp-docs-server",
|
|
3
|
-
"version": "1.2.
|
|
3
|
+
"version": "1.2.2-alpha.2",
|
|
4
4
|
"description": "MCP server for accessing Mastra.ai documentation, changelogs, and news.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "dist/index.js",
|
|
@@ -28,8 +28,8 @@
|
|
|
28
28
|
"jsdom": "^26.1.0",
|
|
29
29
|
"local-pkg": "^1.1.2",
|
|
30
30
|
"zod": "^4.4.3",
|
|
31
|
-
"@mastra/
|
|
32
|
-
"@mastra/
|
|
31
|
+
"@mastra/mcp": "^1.12.0",
|
|
32
|
+
"@mastra/core": "1.46.1-alpha.1"
|
|
33
33
|
},
|
|
34
34
|
"devDependencies": {
|
|
35
35
|
"@hono/node-server": "^1.19.11",
|
|
@@ -47,7 +47,7 @@
|
|
|
47
47
|
"vitest": "4.1.8",
|
|
48
48
|
"@internal/lint": "0.0.108",
|
|
49
49
|
"@internal/types-builder": "0.0.83",
|
|
50
|
-
"@mastra/core": "1.46.
|
|
50
|
+
"@mastra/core": "1.46.1-alpha.1"
|
|
51
51
|
},
|
|
52
52
|
"homepage": "https://mastra.ai",
|
|
53
53
|
"repository": {
|