@mastra/mcp-docs-server 1.2.8-alpha.7 → 1.2.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.docs/docs/agent-builder/integrations.md +1 -3
- package/.docs/docs/agent-builder/overview.md +2 -0
- package/.docs/docs/agents/overview.md +17 -13
- package/.docs/docs/agents/skills.md +1 -3
- package/.docs/docs/agents/structured-output.md +2 -0
- package/.docs/docs/agents/using-tools.md +66 -49
- package/.docs/docs/browser/agent-browser.md +1 -0
- package/.docs/docs/browser/firecrawl.md +129 -0
- package/.docs/docs/browser/overview.md +16 -2
- package/.docs/docs/capabilities/channels/slack.md +111 -1
- package/.docs/docs/deployment/mastra-server.md +35 -0
- package/.docs/docs/deployment/monorepo.md +2 -0
- package/.docs/docs/deployment/overview.md +6 -0
- package/.docs/docs/deployment/sandbox.md +277 -0
- package/.docs/docs/editor/overview.md +2 -6
- package/.docs/docs/editor/tools.md +2 -6
- package/.docs/docs/evals/multi-turn.md +1 -1
- package/.docs/docs/evals/running-in-ci.md +3 -9
- package/.docs/docs/getting-started/build-with-ai.md +2 -6
- package/.docs/docs/getting-started/manual-install.md +1 -1
- package/.docs/docs/index.md +17 -13
- package/.docs/docs/long-running-agents/background-tasks.md +2 -2
- package/.docs/docs/long-running-agents/goals.md +3 -1
- package/.docs/docs/mastra-platform/database.md +12 -6
- package/.docs/docs/mastra-platform/deploy.md +14 -7
- package/.docs/docs/mastra-platform/environments.md +5 -4
- package/.docs/docs/mastra-platform/observability.md +4 -12
- package/.docs/docs/mastra-platform/overview.md +1 -1
- package/.docs/docs/mastra-platform/regions.md +73 -0
- package/.docs/docs/mastra-platform/workspace.md +111 -0
- package/.docs/docs/memory/observational-memory.md +77 -2
- package/.docs/docs/memory/overview.md +4 -0
- package/.docs/docs/observability/feedback.md +4 -0
- package/.docs/docs/observability/integrations/bridges/otel.md +1 -3
- package/.docs/docs/observability/integrations/exporters/otel.md +1 -3
- package/.docs/docs/server/auth/fga.md +40 -0
- package/.docs/docs/server/auth/simple-auth.md +1 -3
- package/.docs/docs/studio/overview.md +2 -0
- package/.docs/docs/workflows/control-flow.md +1 -3
- package/.docs/docs/workspace/filesystem.md +1 -0
- package/.docs/docs/workspace/sandbox.md +1 -0
- package/.docs/guides/getting-started/quickstart.md +40 -25
- package/.docs/guides/index.md +1 -1
- package/.docs/guides/migrations/vnext-to-standard-apis.md +1 -3
- package/.docs/models/environment-variables.md +3 -0
- package/.docs/models/gateways/netlify.md +3 -6
- package/.docs/models/gateways/openrouter.md +343 -344
- package/.docs/models/gateways/vercel.md +11 -5
- package/.docs/models/index.md +1 -1
- package/.docs/models/providers/aki-io.md +78 -0
- package/.docs/models/providers/alibaba-token-plan-cn.md +22 -21
- package/.docs/models/providers/alibaba-token-plan.md +22 -21
- package/.docs/models/providers/ambient.md +12 -10
- package/.docs/models/providers/baseten.md +3 -2
- package/.docs/models/providers/cline-pass.md +82 -0
- package/.docs/models/providers/cortecs.md +2 -1
- package/.docs/models/providers/crossmodel.md +2 -1
- package/.docs/models/providers/deepinfra.md +4 -7
- package/.docs/models/providers/deepseek.md +3 -1
- package/.docs/models/providers/empiriolabs.md +2 -1
- package/.docs/models/providers/evroc.md +3 -2
- package/.docs/models/providers/google.md +4 -2
- package/.docs/models/providers/huggingface.md +2 -1
- package/.docs/models/providers/inferx.md +3 -3
- package/.docs/models/providers/kenari.md +44 -29
- package/.docs/models/providers/kimi-for-coding.md +7 -9
- package/.docs/models/providers/llmgateway.md +4 -1
- package/.docs/models/providers/moonshotai-cn.md +3 -2
- package/.docs/models/providers/moonshotai.md +3 -2
- package/.docs/models/providers/nebius.md +3 -1
- package/.docs/models/providers/novita-ai.md +3 -1
- package/.docs/models/providers/opencode-go.md +6 -3
- package/.docs/models/providers/opencode.md +4 -2
- package/.docs/models/providers/poolside.md +2 -2
- package/.docs/models/providers/thinkingmachines.md +73 -0
- package/.docs/models/providers/togetherai.md +2 -6
- package/.docs/models/providers/wandb.md +2 -1
- package/.docs/models/providers/zenmux.md +3 -1
- package/.docs/models/providers.md +3 -0
- package/.docs/reference/agent-controller/session.md +13 -0
- package/.docs/reference/agents/agent.md +13 -1
- package/.docs/reference/agents/generate.md +18 -0
- package/.docs/reference/browser/firecrawl-browser.md +71 -0
- package/.docs/reference/cli/create-mastra.md +118 -43
- package/.docs/reference/cli/mastra.md +59 -5
- package/.docs/reference/code-sdk/mount-agent-controller.md +4 -4
- package/.docs/reference/core/getMCPServer.md +1 -3
- package/.docs/reference/core/getMCPServerById.md +1 -3
- package/.docs/reference/core/getWorkflow.md +1 -3
- package/.docs/reference/core/listMCPServers.md +2 -6
- package/.docs/reference/datasets/addItem.md +1 -3
- package/.docs/reference/datasets/addItems.md +1 -3
- package/.docs/reference/datasets/compareExperiments.md +1 -3
- package/.docs/reference/datasets/create.md +1 -3
- package/.docs/reference/datasets/dataset.md +2 -6
- package/.docs/reference/datasets/datasets-manager.md +5 -15
- package/.docs/reference/datasets/delete.md +1 -3
- package/.docs/reference/datasets/deleteExperiment.md +1 -3
- package/.docs/reference/datasets/deleteItem.md +1 -3
- package/.docs/reference/datasets/deleteItems.md +1 -3
- package/.docs/reference/datasets/get.md +1 -3
- package/.docs/reference/datasets/getDetails.md +1 -3
- package/.docs/reference/datasets/getExperiment.md +1 -3
- package/.docs/reference/datasets/getItem.md +1 -3
- package/.docs/reference/datasets/getItemHistory.md +1 -3
- package/.docs/reference/datasets/list.md +1 -3
- package/.docs/reference/datasets/listExperimentResults.md +1 -3
- package/.docs/reference/datasets/listExperiments.md +1 -3
- package/.docs/reference/datasets/listItems.md +1 -3
- package/.docs/reference/datasets/listVersions.md +1 -3
- package/.docs/reference/datasets/startExperiment.md +1 -3
- package/.docs/reference/datasets/startExperimentAsync.md +1 -3
- package/.docs/reference/datasets/update.md +1 -3
- package/.docs/reference/datasets/updateItem.md +1 -3
- package/.docs/reference/editor/mastra-editor.md +1 -3
- package/.docs/reference/evals/context-recall.md +205 -0
- package/.docs/reference/evals/create-scorer.md +3 -9
- package/.docs/reference/evals/mastra-scorer.md +1 -3
- package/.docs/reference/file-based-agents/tools.md +7 -6
- package/.docs/reference/index.md +5 -0
- package/.docs/reference/memory/observational-memory.md +21 -0
- package/.docs/reference/observability/tracing/exporters/posthog.md +39 -1
- package/.docs/reference/observability/tracing/processors/sensitive-data-filter.md +1 -3
- package/.docs/reference/processors/token-limiter-processor.md +1 -3
- package/.docs/reference/rag/database-config.md +12 -0
- package/.docs/reference/streaming/agents/stream.md +2 -0
- package/.docs/reference/templates/overview.md +6 -32
- package/.docs/reference/tools/create-tool.md +78 -52
- package/.docs/reference/tools/mcp-client.md +8 -24
- package/.docs/reference/tools/mcp-server.md +143 -44
- package/.docs/reference/tools/tavily.md +1 -1
- package/.docs/reference/tools/vector-query-tool.md +22 -0
- package/.docs/reference/vectors/mongodb.md +2 -0
- package/.docs/reference/vectors/turbopuffer.md +4 -0
- package/.docs/reference/voice/mistral.md +127 -0
- package/.docs/reference/workspace/daytona-sandbox.md +2 -6
- package/.docs/reference/workspace/platform-filesystem.md +182 -0
- package/.docs/reference/workspace/platform-sandbox.md +200 -0
- package/.docs/reference/workspace/railway-sandbox.md +37 -1
- package/CHANGELOG.md +78 -0
- package/dist/{chunk-GLPCVXXO.js → chunk-HGADBLKG.js} +2 -2
- package/dist/{chunk-GLPCVXXO.js.map → chunk-HGADBLKG.js.map} +1 -1
- package/dist/index.js +1 -1
- package/dist/stdio.js +1 -1
- package/dist/tools/docs.d.ts.map +1 -1
- package/package.json +8 -8
|
@@ -13,9 +13,7 @@ The `DatasetsManager` class provides the public API for managing datasets, inclu
|
|
|
13
13
|
```typescript
|
|
14
14
|
import { Mastra } from '@mastra/core'
|
|
15
15
|
|
|
16
|
-
const mastra = new Mastra({
|
|
17
|
-
/* storage config */
|
|
18
|
-
})
|
|
16
|
+
const mastra = new Mastra({/* storage config */})
|
|
19
17
|
|
|
20
18
|
// Create a dataset
|
|
21
19
|
const dataset = await mastra.datasets.create({
|
|
@@ -32,9 +30,7 @@ const { datasets } = await mastra.datasets.list()
|
|
|
32
30
|
```typescript
|
|
33
31
|
import { Mastra } from '@mastra/core'
|
|
34
32
|
|
|
35
|
-
const mastra = new Mastra({
|
|
36
|
-
/* storage config */
|
|
37
|
-
})
|
|
33
|
+
const mastra = new Mastra({/* storage config */})
|
|
38
34
|
|
|
39
35
|
// Get by ID
|
|
40
36
|
const dataset = await mastra.datasets.get({ id: 'dataset-id' })
|
|
@@ -50,9 +46,7 @@ Retrieves a specific experiment (run) by ID. Unlike `dataset.getExperiment()`, t
|
|
|
50
46
|
```typescript
|
|
51
47
|
import { Mastra } from '@mastra/core'
|
|
52
48
|
|
|
53
|
-
const mastra = new Mastra({
|
|
54
|
-
/* storage config */
|
|
55
|
-
})
|
|
49
|
+
const mastra = new Mastra({/* storage config */})
|
|
56
50
|
|
|
57
51
|
// Get experiment directly without knowing the dataset
|
|
58
52
|
const experiment = await mastra.datasets.getExperiment({
|
|
@@ -68,9 +62,7 @@ console.log(`Status: ${experiment.status}`)
|
|
|
68
62
|
```typescript
|
|
69
63
|
import { Mastra } from '@mastra/core'
|
|
70
64
|
|
|
71
|
-
const mastra = new Mastra({
|
|
72
|
-
/* storage config */
|
|
73
|
-
})
|
|
65
|
+
const mastra = new Mastra({/* storage config */})
|
|
74
66
|
|
|
75
67
|
const comparison = await mastra.datasets.compareExperiments({
|
|
76
68
|
experimentIds: ['exp-1', 'exp-2'],
|
|
@@ -83,9 +75,7 @@ const comparison = await mastra.datasets.compareExperiments({
|
|
|
83
75
|
`DatasetsManager` isn't instantiated directly. Access it from a `Mastra` instance:
|
|
84
76
|
|
|
85
77
|
```typescript
|
|
86
|
-
const mastra = new Mastra({
|
|
87
|
-
/* storage config */
|
|
88
|
-
})
|
|
78
|
+
const mastra = new Mastra({/* storage config */})
|
|
89
79
|
const datasetsManager = mastra.datasets
|
|
90
80
|
```
|
|
91
81
|
|
|
@@ -11,9 +11,7 @@ Deletes a dataset by ID, including all items, versions, and experiments.
|
|
|
11
11
|
```typescript
|
|
12
12
|
import { Mastra } from '@mastra/core'
|
|
13
13
|
|
|
14
|
-
const mastra = new Mastra({
|
|
15
|
-
/* storage config */
|
|
16
|
-
})
|
|
14
|
+
const mastra = new Mastra({/* storage config */})
|
|
17
15
|
|
|
18
16
|
await mastra.datasets.delete({ id: 'dataset-id' })
|
|
19
17
|
```
|
|
@@ -11,9 +11,7 @@ Deletes an experiment (run) by ID, including all associated results.
|
|
|
11
11
|
```typescript
|
|
12
12
|
import { Mastra } from '@mastra/core'
|
|
13
13
|
|
|
14
|
-
const mastra = new Mastra({
|
|
15
|
-
/* storage config */
|
|
16
|
-
})
|
|
14
|
+
const mastra = new Mastra({/* storage config */})
|
|
17
15
|
|
|
18
16
|
const dataset = await mastra.datasets.get({ id: 'dataset-id' })
|
|
19
17
|
|
|
@@ -11,9 +11,7 @@ Deletes a single item from the dataset by ID.
|
|
|
11
11
|
```typescript
|
|
12
12
|
import { Mastra } from '@mastra/core'
|
|
13
13
|
|
|
14
|
-
const mastra = new Mastra({
|
|
15
|
-
/* storage config */
|
|
16
|
-
})
|
|
14
|
+
const mastra = new Mastra({/* storage config */})
|
|
17
15
|
|
|
18
16
|
const dataset = await mastra.datasets.get({ id: 'dataset-id' })
|
|
19
17
|
|
|
@@ -11,9 +11,7 @@ Deletes multiple items from the dataset in a single bulk operation.
|
|
|
11
11
|
```typescript
|
|
12
12
|
import { Mastra } from '@mastra/core'
|
|
13
13
|
|
|
14
|
-
const mastra = new Mastra({
|
|
15
|
-
/* storage config */
|
|
16
|
-
})
|
|
14
|
+
const mastra = new Mastra({/* storage config */})
|
|
17
15
|
|
|
18
16
|
const dataset = await mastra.datasets.get({ id: 'dataset-id' })
|
|
19
17
|
|
|
@@ -11,9 +11,7 @@ Retrieves an existing dataset by ID. Throws a `MastraError` if the dataset doesn
|
|
|
11
11
|
```typescript
|
|
12
12
|
import { Mastra } from '@mastra/core'
|
|
13
13
|
|
|
14
|
-
const mastra = new Mastra({
|
|
15
|
-
/* storage config */
|
|
16
|
-
})
|
|
14
|
+
const mastra = new Mastra({/* storage config */})
|
|
17
15
|
|
|
18
16
|
const dataset = await mastra.datasets.get({ id: 'dataset-id' })
|
|
19
17
|
|
|
@@ -11,9 +11,7 @@ Retrieves the full dataset record from storage, including metadata, schemas, and
|
|
|
11
11
|
```typescript
|
|
12
12
|
import { Mastra } from '@mastra/core'
|
|
13
13
|
|
|
14
|
-
const mastra = new Mastra({
|
|
15
|
-
/* storage config */
|
|
16
|
-
})
|
|
14
|
+
const mastra = new Mastra({/* storage config */})
|
|
17
15
|
|
|
18
16
|
const dataset = await mastra.datasets.get({ id: 'dataset-id' })
|
|
19
17
|
|
|
@@ -11,9 +11,7 @@ Retrieves a specific experiment (run) by its ID.
|
|
|
11
11
|
```typescript
|
|
12
12
|
import { Mastra } from '@mastra/core'
|
|
13
13
|
|
|
14
|
-
const mastra = new Mastra({
|
|
15
|
-
/* storage config */
|
|
16
|
-
})
|
|
14
|
+
const mastra = new Mastra({/* storage config */})
|
|
17
15
|
|
|
18
16
|
const dataset = await mastra.datasets.get({ id: 'dataset-id' })
|
|
19
17
|
|
|
@@ -11,9 +11,7 @@ Retrieves a single dataset item by ID, optionally at a specific dataset version.
|
|
|
11
11
|
```typescript
|
|
12
12
|
import { Mastra } from '@mastra/core'
|
|
13
13
|
|
|
14
|
-
const mastra = new Mastra({
|
|
15
|
-
/* storage config */
|
|
16
|
-
})
|
|
14
|
+
const mastra = new Mastra({/* storage config */})
|
|
17
15
|
|
|
18
16
|
const dataset = await mastra.datasets.get({ id: 'dataset-id' })
|
|
19
17
|
|
|
@@ -11,9 +11,7 @@ Retrieves the full SCD-2 (Slowly Changing Dimension Type 2) history of a specifi
|
|
|
11
11
|
```typescript
|
|
12
12
|
import { Mastra } from '@mastra/core'
|
|
13
13
|
|
|
14
|
-
const mastra = new Mastra({
|
|
15
|
-
/* storage config */
|
|
16
|
-
})
|
|
14
|
+
const mastra = new Mastra({/* storage config */})
|
|
17
15
|
|
|
18
16
|
const dataset = await mastra.datasets.get({ id: 'dataset-id' })
|
|
19
17
|
|
|
@@ -11,9 +11,7 @@ Lists datasets with pagination and optional filters. Filters are applied at the
|
|
|
11
11
|
```typescript
|
|
12
12
|
import { Mastra } from '@mastra/core'
|
|
13
13
|
|
|
14
|
-
const mastra = new Mastra({
|
|
15
|
-
/* storage config */
|
|
16
|
-
})
|
|
14
|
+
const mastra = new Mastra({/* storage config */})
|
|
17
15
|
|
|
18
16
|
const { datasets, pagination } = await mastra.datasets.list({ page: 0, perPage: 10 })
|
|
19
17
|
|
|
@@ -11,9 +11,7 @@ Lists individual item results for a specific experiment with optional filters an
|
|
|
11
11
|
```typescript
|
|
12
12
|
import { Mastra } from '@mastra/core'
|
|
13
13
|
|
|
14
|
-
const mastra = new Mastra({
|
|
15
|
-
/* storage config */
|
|
16
|
-
})
|
|
14
|
+
const mastra = new Mastra({/* storage config */})
|
|
17
15
|
|
|
18
16
|
const dataset = await mastra.datasets.get({ id: 'dataset-id' })
|
|
19
17
|
|
|
@@ -11,9 +11,7 @@ Lists experiments (runs) for this dataset with optional filters and pagination.
|
|
|
11
11
|
```typescript
|
|
12
12
|
import { Mastra } from '@mastra/core'
|
|
13
13
|
|
|
14
|
-
const mastra = new Mastra({
|
|
15
|
-
/* storage config */
|
|
16
|
-
})
|
|
14
|
+
const mastra = new Mastra({/* storage config */})
|
|
17
15
|
|
|
18
16
|
const dataset = await mastra.datasets.get({ id: 'dataset-id' })
|
|
19
17
|
|
|
@@ -11,9 +11,7 @@ Lists items in the dataset. When only `version` is provided, returns a bare `Dat
|
|
|
11
11
|
```typescript
|
|
12
12
|
import { Mastra } from '@mastra/core'
|
|
13
13
|
|
|
14
|
-
const mastra = new Mastra({
|
|
15
|
-
/* storage config */
|
|
16
|
-
})
|
|
14
|
+
const mastra = new Mastra({/* storage config */})
|
|
17
15
|
|
|
18
16
|
const dataset = await mastra.datasets.get({ id: 'dataset-id' })
|
|
19
17
|
|
|
@@ -11,9 +11,7 @@ Lists all versions of the dataset with pagination.
|
|
|
11
11
|
```typescript
|
|
12
12
|
import { Mastra } from '@mastra/core'
|
|
13
13
|
|
|
14
|
-
const mastra = new Mastra({
|
|
15
|
-
/* storage config */
|
|
16
|
-
})
|
|
14
|
+
const mastra = new Mastra({/* storage config */})
|
|
17
15
|
|
|
18
16
|
const dataset = await mastra.datasets.get({ id: 'dataset-id' })
|
|
19
17
|
|
|
@@ -11,9 +11,7 @@ Runs an experiment on the dataset and waits for completion. Executes all items a
|
|
|
11
11
|
```typescript
|
|
12
12
|
import { Mastra } from '@mastra/core'
|
|
13
13
|
|
|
14
|
-
const mastra = new Mastra({
|
|
15
|
-
/* storage config */
|
|
16
|
-
})
|
|
14
|
+
const mastra = new Mastra({/* storage config */})
|
|
17
15
|
|
|
18
16
|
const dataset = await mastra.datasets.get({ id: 'dataset-id' })
|
|
19
17
|
|
|
@@ -11,9 +11,7 @@ Starts an experiment asynchronously (fire-and-forget). Returns immediately with
|
|
|
11
11
|
```typescript
|
|
12
12
|
import { Mastra } from '@mastra/core'
|
|
13
13
|
|
|
14
|
-
const mastra = new Mastra({
|
|
15
|
-
/* storage config */
|
|
16
|
-
})
|
|
14
|
+
const mastra = new Mastra({/* storage config */})
|
|
17
15
|
|
|
18
16
|
const dataset = await mastra.datasets.get({ id: 'dataset-id' })
|
|
19
17
|
|
|
@@ -12,9 +12,7 @@ Updates dataset metadata, name, description, and/or schemas. Use Standard JSON S
|
|
|
12
12
|
import { Mastra } from '@mastra/core'
|
|
13
13
|
import { z } from 'zod'
|
|
14
14
|
|
|
15
|
-
const mastra = new Mastra({
|
|
16
|
-
/* storage config */
|
|
17
|
-
})
|
|
15
|
+
const mastra = new Mastra({/* storage config */})
|
|
18
16
|
|
|
19
17
|
const dataset = await mastra.datasets.get({ id: 'dataset-id' })
|
|
20
18
|
|
|
@@ -11,9 +11,7 @@ Updates an existing item in the dataset. Only the provided fields are updated. U
|
|
|
11
11
|
```typescript
|
|
12
12
|
import { Mastra } from '@mastra/core'
|
|
13
13
|
|
|
14
|
-
const mastra = new Mastra({
|
|
15
|
-
/* storage config */
|
|
16
|
-
})
|
|
14
|
+
const mastra = new Mastra({/* storage config */})
|
|
17
15
|
|
|
18
16
|
const dataset = await mastra.datasets.get({ id: 'dataset-id' })
|
|
19
17
|
|
|
@@ -14,9 +14,7 @@ import { MastraEditor } from '@mastra/editor'
|
|
|
14
14
|
import { ComposioToolProvider } from '@mastra/editor/providers/composio'
|
|
15
15
|
|
|
16
16
|
export const mastra = new Mastra({
|
|
17
|
-
agents: {
|
|
18
|
-
/* your agents */
|
|
19
|
-
},
|
|
17
|
+
agents: {/* your agents */},
|
|
20
18
|
editor: new MastraEditor({
|
|
21
19
|
toolProviders: {
|
|
22
20
|
composio: new ComposioToolProvider({
|
|
@@ -0,0 +1,205 @@
|
|
|
1
|
+
> Discover all available pages from the documentation index: https://mastra.ai/llms.txt
|
|
2
|
+
|
|
3
|
+
# Context recall scorer
|
|
4
|
+
|
|
5
|
+
The `createContextRecallScorer()` function creates a scorer that evaluates how well retrieved context covers the claims in a ground-truth reference answer. It measures retrieval completeness by checking what fraction of the ground truth's claims are attributable to the retrieved context.
|
|
6
|
+
|
|
7
|
+
This scorer requires a ground-truth reference answer, making it suitable for labeled datasets in CI or test environments. When `groundTruth` is not provided in the run, the scorer returns a score of 0 rather than throwing an error.
|
|
8
|
+
|
|
9
|
+
## RAG retrieval evaluation
|
|
10
|
+
|
|
11
|
+
Ideal for evaluating retrieval completeness in RAG pipelines where:
|
|
12
|
+
|
|
13
|
+
- You need to verify the retriever fetches all necessary information
|
|
14
|
+
- You have labeled datasets with known-correct answers
|
|
15
|
+
- You want to catch regressions in what the retriever returns
|
|
16
|
+
|
|
17
|
+
## Dataset-driven testing
|
|
18
|
+
|
|
19
|
+
Use when running evaluations against curated test sets:
|
|
20
|
+
|
|
21
|
+
- CI pipelines with ground-truth labeled questions
|
|
22
|
+
- A/B testing retrieval strategies
|
|
23
|
+
- Benchmarking embedding models for coverage
|
|
24
|
+
|
|
25
|
+
## Parameters
|
|
26
|
+
|
|
27
|
+
**model** (`MastraModelConfig`): The language model to use for evaluating claim attribution
|
|
28
|
+
|
|
29
|
+
**options** (`ContextRecallMetricOptions`): Configuration options for the scorer
|
|
30
|
+
|
|
31
|
+
**Note**: Either `context` or `contextExtractor` must be provided. When both are provided, `contextExtractor` is used only if the run input and output are agent-shaped (`MastraDBMessage[]`); otherwise the scorer falls back to `context`.
|
|
32
|
+
|
|
33
|
+
## `.run()` returns
|
|
34
|
+
|
|
35
|
+
**score** (`number`): Recall score between 0 and scale (default 0-1), representing the fraction of ground-truth claims covered by the context
|
|
36
|
+
|
|
37
|
+
**reason** (`string`): Human-readable explanation of which ground-truth claims were and were not found in the context
|
|
38
|
+
|
|
39
|
+
## Scoring details
|
|
40
|
+
|
|
41
|
+
### Claim attribution
|
|
42
|
+
|
|
43
|
+
Context Recall uses a two-step LLM evaluation followed by a deterministic score calculation:
|
|
44
|
+
|
|
45
|
+
1. **Claim extraction**: The ground-truth answer is decomposed into atomic claims
|
|
46
|
+
2. **Attribution check**: Each claim is checked against the retrieval context for support
|
|
47
|
+
|
|
48
|
+
The score is then calculated as the ratio of attributed claims to total claims, multiplied by scale.
|
|
49
|
+
|
|
50
|
+
### Scoring formula
|
|
51
|
+
|
|
52
|
+
```text
|
|
53
|
+
Context Recall = attributed_claims / total_claims × scale
|
|
54
|
+
|
|
55
|
+
Where:
|
|
56
|
+
- attributed_claims = number of ground-truth claims supported by the context
|
|
57
|
+
- total_claims = total number of claims extracted from the ground truth
|
|
58
|
+
- Attribution is binary: a claim is either supported (yes) or not (no)
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
### Score interpretation
|
|
62
|
+
|
|
63
|
+
These ranges assume the default `scale` of 1. When using a custom scale, multiply accordingly.
|
|
64
|
+
|
|
65
|
+
- **0.9-1.0**: Excellent recall — context covers nearly all ground-truth claims
|
|
66
|
+
- **0.7-0.8**: Good recall — most claims are covered, minor gaps
|
|
67
|
+
- **0.4-0.6**: Moderate recall — significant information missing from context
|
|
68
|
+
- **0.1-0.3**: Poor recall — most ground-truth claims not found in context
|
|
69
|
+
- **0.0**: No recall — none of the ground-truth claims are in the context
|
|
70
|
+
|
|
71
|
+
### Reason analysis
|
|
72
|
+
|
|
73
|
+
The reason field explains:
|
|
74
|
+
|
|
75
|
+
- Which ground-truth claims were found in the context
|
|
76
|
+
- Which claims were missing and what information gaps exist
|
|
77
|
+
- Specific context pieces that supported attributed claims
|
|
78
|
+
|
|
79
|
+
### Optimization insights
|
|
80
|
+
|
|
81
|
+
Use results to:
|
|
82
|
+
|
|
83
|
+
- **Improve retrieval**: Identify what types of information the retriever misses
|
|
84
|
+
- **Tune chunk size**: Ensure chunks contain enough detail to cover ground-truth claims
|
|
85
|
+
- **Evaluate embeddings**: Test different embedding models for better information coverage
|
|
86
|
+
- **Expand knowledge base**: Add documents that cover frequently missed claims
|
|
87
|
+
|
|
88
|
+
### Example calculation
|
|
89
|
+
|
|
90
|
+
Given ground truth: "Einstein was born in 1879. He developed relativity. He won the Nobel Prize."
|
|
91
|
+
|
|
92
|
+
Claims extracted: 3
|
|
93
|
+
|
|
94
|
+
- "Einstein was born in 1879" → attributed (context mentions birthdate)
|
|
95
|
+
- "Einstein developed relativity" → attributed (context covers relativity)
|
|
96
|
+
- "Einstein won the Nobel Prize" → not attributed (context doesn't mention Nobel Prize)
|
|
97
|
+
|
|
98
|
+
Recall = 2/3 = 0.67
|
|
99
|
+
|
|
100
|
+
## Scorer configuration
|
|
101
|
+
|
|
102
|
+
### Dynamic context extraction
|
|
103
|
+
|
|
104
|
+
```typescript
|
|
105
|
+
const scorer = createContextRecallScorer({
|
|
106
|
+
model: 'openai/gpt-5.5',
|
|
107
|
+
options: {
|
|
108
|
+
contextExtractor: (input, output) => {
|
|
109
|
+
const query = input?.inputMessages?.[0]?.content || ''
|
|
110
|
+
const searchResults = vectorDB.search(query, { limit: 10 })
|
|
111
|
+
return searchResults.map(result => result.content)
|
|
112
|
+
},
|
|
113
|
+
scale: 1,
|
|
114
|
+
},
|
|
115
|
+
})
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
### Static context evaluation
|
|
119
|
+
|
|
120
|
+
```typescript
|
|
121
|
+
const scorer = createContextRecallScorer({
|
|
122
|
+
model: 'openai/gpt-5.5',
|
|
123
|
+
options: {
|
|
124
|
+
context: [
|
|
125
|
+
'Document 1: Einstein was born on 14 March 1879 in Ulm, Germany.',
|
|
126
|
+
'Document 2: Einstein published the theory of special relativity in 1905.',
|
|
127
|
+
'Document 3: Einstein moved to the United States in 1933.',
|
|
128
|
+
],
|
|
129
|
+
},
|
|
130
|
+
})
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
## Example
|
|
134
|
+
|
|
135
|
+
Evaluate RAG retrieval completeness against a labeled dataset:
|
|
136
|
+
|
|
137
|
+
```typescript
|
|
138
|
+
import { runEvals } from '@mastra/core/evals'
|
|
139
|
+
import { createContextRecallScorer } from '@mastra/evals/scorers/prebuilt'
|
|
140
|
+
import { myAgent } from './agent'
|
|
141
|
+
|
|
142
|
+
const scorer = createContextRecallScorer({
|
|
143
|
+
model: 'openai/gpt-5.5',
|
|
144
|
+
options: {
|
|
145
|
+
contextExtractor: (input, output) => {
|
|
146
|
+
// Extract context from tool invocation results in the agent output
|
|
147
|
+
return output
|
|
148
|
+
.filter(msg => msg?.role === 'assistant')
|
|
149
|
+
.flatMap(msg => msg?.content?.toolInvocations ?? [])
|
|
150
|
+
.filter((tool: any) => tool.state === 'result')
|
|
151
|
+
.map((tool: any) => JSON.stringify(tool.result))
|
|
152
|
+
},
|
|
153
|
+
},
|
|
154
|
+
})
|
|
155
|
+
|
|
156
|
+
const result = await runEvals({
|
|
157
|
+
data: [
|
|
158
|
+
{
|
|
159
|
+
input: 'What are the health benefits of green tea?',
|
|
160
|
+
groundTruth:
|
|
161
|
+
'Green tea contains antioxidants that reduce inflammation, L-theanine that improves focus, and catechins that boost metabolism.',
|
|
162
|
+
},
|
|
163
|
+
{
|
|
164
|
+
input: 'How does photosynthesis work?',
|
|
165
|
+
groundTruth:
|
|
166
|
+
'Photosynthesis converts sunlight into chemical energy using chlorophyll in chloroplasts, producing glucose and oxygen from carbon dioxide and water.',
|
|
167
|
+
},
|
|
168
|
+
],
|
|
169
|
+
scorers: [scorer],
|
|
170
|
+
target: myAgent,
|
|
171
|
+
onItemComplete: ({ scorerResults }) => {
|
|
172
|
+
console.log({
|
|
173
|
+
score: scorerResults[scorer.id].score,
|
|
174
|
+
reason: scorerResults[scorer.id].reason,
|
|
175
|
+
})
|
|
176
|
+
},
|
|
177
|
+
})
|
|
178
|
+
|
|
179
|
+
console.log(result.scores)
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
For more details on `runEvals`, see the [runEvals reference](https://mastra.ai/reference/evals/run-evals).
|
|
183
|
+
|
|
184
|
+
To add this scorer to an agent, see the [Scorers overview](https://mastra.ai/docs/evals/overview) guide.
|
|
185
|
+
|
|
186
|
+
## Comparison with context precision
|
|
187
|
+
|
|
188
|
+
Choose the right scorer for your needs:
|
|
189
|
+
|
|
190
|
+
| Use case | Context Recall | Context Precision |
|
|
191
|
+
| ------------------------- | ------------------------ | ----------------------------- |
|
|
192
|
+
| **What it measures** | Coverage of ground truth | Relevance of retrieved chunks |
|
|
193
|
+
| **Direction** | Ground truth → context | Context → ground truth |
|
|
194
|
+
| **Position sensitive** | No | Yes (rewards early placement) |
|
|
195
|
+
| **Requires ground truth** | Yes | Yes |
|
|
196
|
+
| **Failure mode caught** | Missing information | Irrelevant noise |
|
|
197
|
+
|
|
198
|
+
Use both together for a complete picture of retrieval quality: precision catches junk in the context, recall catches gaps.
|
|
199
|
+
|
|
200
|
+
## Related
|
|
201
|
+
|
|
202
|
+
- [Context Precision Scorer](https://mastra.ai/reference/evals/context-precision): Evaluates if retrieved context is relevant and well-ranked
|
|
203
|
+
- [Context Relevance Scorer](https://mastra.ai/reference/evals/context-relevance): Evaluates context usage and quality
|
|
204
|
+
- [Faithfulness Scorer](https://mastra.ai/reference/evals/faithfulness): Measures answer groundedness in context
|
|
205
|
+
- [Custom Scorers](https://mastra.ai/docs/evals/custom-scorers): Creating your own evaluation metrics
|
|
@@ -23,18 +23,12 @@ const scorer = createScorer({
|
|
|
23
23
|
instructions: 'You are an expert evaluator...',
|
|
24
24
|
},
|
|
25
25
|
})
|
|
26
|
-
.preprocess({
|
|
27
|
-
|
|
28
|
-
})
|
|
29
|
-
.analyze({
|
|
30
|
-
/* step config */
|
|
31
|
-
})
|
|
26
|
+
.preprocess({/* step config */})
|
|
27
|
+
.analyze({/* step config */})
|
|
32
28
|
.generateScore(({ run, results }) => {
|
|
33
29
|
// Return a number
|
|
34
30
|
})
|
|
35
|
-
.generateReason({
|
|
36
|
-
/* step config */
|
|
37
|
-
})
|
|
31
|
+
.generateReason({/* step config */})
|
|
38
32
|
```
|
|
39
33
|
|
|
40
34
|
## `createScorer` options
|
|
@@ -33,9 +33,7 @@ const result = await scorer.run({
|
|
|
33
33
|
input: 'What is machine learning?',
|
|
34
34
|
output: 'Machine learning is a subset of artificial intelligence...',
|
|
35
35
|
runId: 'optional-run-id',
|
|
36
|
-
requestContext: {
|
|
37
|
-
/* optional context */
|
|
38
|
-
},
|
|
36
|
+
requestContext: {/* optional context */},
|
|
39
37
|
})
|
|
40
38
|
```
|
|
41
39
|
|
|
@@ -16,16 +16,17 @@ import { z } from 'zod'
|
|
|
16
16
|
|
|
17
17
|
export default createTool({
|
|
18
18
|
id: 'get_weather',
|
|
19
|
-
description: 'Get the current weather for a
|
|
19
|
+
description: 'Get the current weather for a location.',
|
|
20
20
|
inputSchema: z.object({
|
|
21
|
-
|
|
21
|
+
location: z.string(),
|
|
22
22
|
}),
|
|
23
23
|
outputSchema: z.object({
|
|
24
|
-
|
|
25
|
-
|
|
24
|
+
location: z.string(),
|
|
25
|
+
temperatureCelsius: z.number(),
|
|
26
|
+
conditions: z.string(),
|
|
26
27
|
}),
|
|
27
|
-
execute: async ({
|
|
28
|
-
return {
|
|
28
|
+
execute: async ({ location }) => {
|
|
29
|
+
return { location, temperatureCelsius: 21, conditions: 'sunny' }
|
|
29
30
|
},
|
|
30
31
|
})
|
|
31
32
|
```
|
package/.docs/reference/index.md
CHANGED
|
@@ -57,6 +57,7 @@ The Reference section provides documentation of Mastra's API, including paramete
|
|
|
57
57
|
- [WorkOS](https://mastra.ai/reference/auth/workos)
|
|
58
58
|
- [AgentBrowser](https://mastra.ai/reference/browser/agent-browser)
|
|
59
59
|
- [BrowserViewer](https://mastra.ai/reference/browser/browser-viewer)
|
|
60
|
+
- [FirecrawlBrowser](https://mastra.ai/reference/browser/firecrawl-browser)
|
|
60
61
|
- [MastraBrowser Class](https://mastra.ai/reference/browser/mastra-browser)
|
|
61
62
|
- [StagehandBrowser](https://mastra.ai/reference/browser/stagehand-browser)
|
|
62
63
|
- [ChannelProvider](https://mastra.ai/reference/channels/channel-provider)
|
|
@@ -142,6 +143,7 @@ The Reference section provides documentation of Mastra's API, including paramete
|
|
|
142
143
|
- [Completeness](https://mastra.ai/reference/evals/completeness)
|
|
143
144
|
- [Content Similarity Scorer](https://mastra.ai/reference/evals/content-similarity)
|
|
144
145
|
- [Context Precision Scorer](https://mastra.ai/reference/evals/context-precision)
|
|
146
|
+
- [Context Recall Scorer](https://mastra.ai/reference/evals/context-recall)
|
|
145
147
|
- [Context Relevance Scorer](https://mastra.ai/reference/evals/context-relevance)
|
|
146
148
|
- [Faithfulness](https://mastra.ai/reference/evals/faithfulness)
|
|
147
149
|
- [Hallucination](https://mastra.ai/reference/evals/hallucination)
|
|
@@ -336,6 +338,7 @@ The Reference section provides documentation of Mastra's API, including paramete
|
|
|
336
338
|
- [Inworld Realtime](https://mastra.ai/reference/voice/inworld-realtime)
|
|
337
339
|
- [LiveKit](https://mastra.ai/reference/voice/livekit)
|
|
338
340
|
- [Mastra Voice](https://mastra.ai/reference/voice/mastra-voice)
|
|
341
|
+
- [Mistral](https://mastra.ai/reference/voice/mistral)
|
|
339
342
|
- [Murf](https://mastra.ai/reference/voice/murf)
|
|
340
343
|
- [OpenAI](https://mastra.ai/reference/voice/openai)
|
|
341
344
|
- [OpenAI Realtime](https://mastra.ai/reference/voice/openai-realtime)
|
|
@@ -392,6 +395,8 @@ The Reference section provides documentation of Mastra's API, including paramete
|
|
|
392
395
|
- [LocalSandbox](https://mastra.ai/reference/workspace/local-sandbox)
|
|
393
396
|
- [MesaFilesystem](https://mastra.ai/reference/workspace/mesa-filesystem)
|
|
394
397
|
- [ModalSandbox](https://mastra.ai/reference/workspace/modal-sandbox)
|
|
398
|
+
- [PlatformFilesystem](https://mastra.ai/reference/workspace/platform-filesystem)
|
|
399
|
+
- [PlatformSandbox](https://mastra.ai/reference/workspace/platform-sandbox)
|
|
395
400
|
- [RailwaySandbox](https://mastra.ai/reference/workspace/railway-sandbox)
|
|
396
401
|
- [S3Filesystem](https://mastra.ai/reference/workspace/s3-filesystem)
|
|
397
402
|
- [SandboxProcessManager](https://mastra.ai/reference/workspace/process-manager)
|
|
@@ -408,6 +408,27 @@ Setting `bufferTokens: false` disables both observation and reflection async buf
|
|
|
408
408
|
|
|
409
409
|
Observational Memory emits typed data parts during agent execution that clients can use for real-time UI feedback. These are streamed alongside the agent's response.
|
|
410
410
|
|
|
411
|
+
### Read extractor results
|
|
412
|
+
|
|
413
|
+
Both completion events carry extractor output in their `data` payload. The extractor fields are:
|
|
414
|
+
|
|
415
|
+
```typescript
|
|
416
|
+
interface DataOmObservationEndPart {
|
|
417
|
+
type: 'data-om-observation-end'
|
|
418
|
+
data: {
|
|
419
|
+
/** Whether the completed work was an observation or reflection */
|
|
420
|
+
operationType: 'observation' | 'reflection'
|
|
421
|
+
/** Values extracted during this OM operation, keyed by extractor slug */
|
|
422
|
+
extractedValues?: Record<string, unknown>
|
|
423
|
+
/** Extractor failures from this OM operation. Successful extractor values are still included */
|
|
424
|
+
extractionFailures?: Array<{ slug: string; error: string }>
|
|
425
|
+
// ...other fields documented in the tables below
|
|
426
|
+
}
|
|
427
|
+
}
|
|
428
|
+
```
|
|
429
|
+
|
|
430
|
+
Both extractor fields are optional. A completion can include values, failures, both, or neither. `data-om-observation-end` reports synchronous completion. `data-om-buffering-end` reports completed background work whose buffered content still awaits activation, although extractor metadata is already persisted. `DataOmBufferingEndPart` carries the same extractor fields, and both types are exported from `@mastra/memory/processors`. See [Read extracted values from a stream](https://mastra.ai/docs/memory/observational-memory) for a consumer example.
|
|
431
|
+
|
|
411
432
|
### `data-om-status`
|
|
412
433
|
|
|
413
434
|
Emitted once per agent loop step, before model generation. Provides a snapshot of the current memory state, including token usage for both context windows and the state of any async buffered content.
|
|
@@ -39,6 +39,14 @@ async exportTracingEvent(event: TracingEvent): Promise<void>
|
|
|
39
39
|
|
|
40
40
|
Exports a tracing event to PostHog.
|
|
41
41
|
|
|
42
|
+
### `onFeedbackEvent`
|
|
43
|
+
|
|
44
|
+
```typescript
|
|
45
|
+
async onFeedbackEvent(event: FeedbackEvent): Promise<void>
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
Exports a feedback event to PostHog. See [Feedback export](#feedback-export).
|
|
49
|
+
|
|
42
50
|
### flush
|
|
43
51
|
|
|
44
52
|
```typescript
|
|
@@ -92,4 +100,34 @@ const exporter = new PosthogExporter({
|
|
|
92
100
|
| `AGENT_RUN` | `$ai_span` |
|
|
93
101
|
| `WORKFLOW_RUN` | `$ai_span` |
|
|
94
102
|
| All other workflows | `$ai_span` |
|
|
95
|
-
| `GENERIC` | `$ai_span` |
|
|
103
|
+
| `GENERIC` | `$ai_span` |
|
|
104
|
+
|
|
105
|
+
## Feedback export
|
|
106
|
+
|
|
107
|
+
Feedback recorded with [`addFeedback()`](https://mastra.ai/docs/observability/feedback) is exported to PostHog as a native `$ai_feedback` event. PostHog shows it as **User feedback** on the linked trace. No extra configuration is needed.
|
|
108
|
+
|
|
109
|
+
```typescript
|
|
110
|
+
const trace = await mastra.observability.getRecordedTrace({ traceId })
|
|
111
|
+
await trace.addFeedback({
|
|
112
|
+
feedbackType: 'thumbs',
|
|
113
|
+
value: 'down',
|
|
114
|
+
comment: 'Wrong answer',
|
|
115
|
+
})
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
Feedback events map to these PostHog properties:
|
|
119
|
+
|
|
120
|
+
| Mastra feedback field | PostHog property |
|
|
121
|
+
| -------------------------------------- | ------------------- |
|
|
122
|
+
| `traceId` | `$ai_trace_id` |
|
|
123
|
+
| `comment` (or `value` when no comment) | `$ai_feedback_text` |
|
|
124
|
+
| `feedbackId` | `feedback_id` |
|
|
125
|
+
| `feedbackType` | `feedback_type` |
|
|
126
|
+
| `value` | `feedback_value` |
|
|
127
|
+
| `feedbackSource` | `feedback_source` |
|
|
128
|
+
| `spanId` | `span_id` |
|
|
129
|
+
| `sourceId` | `source_id` |
|
|
130
|
+
| `metadata.sessionId` | `$ai_session_id` |
|
|
131
|
+
| `metadata` (other keys) | custom properties |
|
|
132
|
+
|
|
133
|
+
Feedback without a `traceId` is dropped because PostHog anchors feedback to a trace through `$ai_trace_id`. The event's distinct ID resolves from `feedbackUserId`, then `metadata.userId`, then `defaultDistinctId`, then `anonymous`.
|