@mastra/mcp-docs-server 1.1.5 → 1.1.6

package/.docs/docs/observability/datasets/overview.md

@@ -0,0 +1,198 @@
+# Datasets Overview
+
+**Added in:** `@mastra/core@1.4.0`
+
+Datasets are collections of test cases that you run experiments against to measure how well your agents and workflows perform. Each mutation creates a new version, so you can reproduce past experiments exactly. Pair datasets with [scorers](https://mastra.ai/docs/evals/overview) to track quality across prompts, models, or code changes.
+
+## Usage
+
+### Configure storage
+
+Configure storage in your Mastra instance. Datasets require a storage adapter that provides the `datasets` domain:
+
+```typescript
+import { Mastra } from '@mastra/core'
+import { LibSQLStore } from '@mastra/libsql'
+
+export const mastra = new Mastra({
+  storage: new LibSQLStore({
+    id: 'my-store',
+    url: 'file:./mastra.db',
+  }),
+})
+```
+
+### Accessing the datasets API
+
+All dataset operations are available through `mastra.datasets`:
+
+```typescript
+const datasets = mastra.datasets
+
+// Create a dataset
+const dataset = await datasets.create({ name: 'my-dataset' })
+
+// Retrieve an existing dataset
+const existing = await datasets.get({ id: 'dataset-id' })
+
+// List all datasets
+const { datasets: all } = await datasets.list()
+```
+
+> **Info:** Visit the [`DatasetsManager` reference](https://mastra.ai/reference/datasets/datasets-manager) for the full list of methods.
+
+## Creating a dataset
+
+Call [`create()`](https://mastra.ai/reference/datasets/create) with a name and optional description:
+
+```typescript
+import { mastra } from '../index'
+
+const dataset = await mastra.datasets.create({
+  name: 'translation-pairs',
+  description: 'English to Spanish translation test cases',
+})
+
+console.log(dataset.id) // auto-generated UUID
+```
+
+### Defining schemas
+
+You can enforce the shape of `input` and `groundTruth` by passing Zod schemas. Mastra converts them to JSON Schema at creation time:
+
+```typescript
+import { z } from 'zod'
+import { mastra } from '../index'
+
+const dataset = await mastra.datasets.create({
+  name: 'translation-pairs',
+  inputSchema: z.object({
+    text: z.string(),
+    sourceLang: z.string(),
+    targetLang: z.string(),
+  }),
+  groundTruthSchema: z.object({
+    translation: z.string(),
+  }),
+})
+```
+
+Items that don't match the schema are rejected at insert time.
+
+## Adding items
+
+Use [`addItem()`](https://mastra.ai/reference/datasets/addItem) for a single item or [`addItems()`](https://mastra.ai/reference/datasets/addItems) to insert in bulk:
+
+```typescript
+// Single item
+await dataset.addItem({
+  input: { text: 'Hello', sourceLang: 'en', targetLang: 'es' },
+  groundTruth: { translation: 'Hola' },
+})
+
+// Bulk insert
+await dataset.addItems({
+  items: [
+    {
+      input: { text: 'Goodbye', sourceLang: 'en', targetLang: 'es' },
+      groundTruth: { translation: 'Adiós' },
+    },
+    {
+      input: { text: 'Thank you', sourceLang: 'en', targetLang: 'es' },
+      groundTruth: { translation: 'Gracias' },
+    },
+  ],
+})
+```
+
+## Updating and deleting items
+
+[`updateItem()`](https://mastra.ai/reference/datasets/updateItem), [`deleteItem()`](https://mastra.ai/reference/datasets/deleteItem), and [`deleteItems()`](https://mastra.ai/reference/datasets/deleteItems) let you modify or remove existing items by `itemId`:
+
+```typescript
+await dataset.updateItem({
+  itemId: 'item-abc-123',
+  groundTruth: { translation: '¡Hola!' },
+})
+
+await dataset.deleteItem({ itemId: 'item-abc-123' })
+
+await dataset.deleteItems({ itemIds: ['item-1', 'item-2'] })
+```
+
+## Listing and searching items
+
+[`listItems()`](https://mastra.ai/reference/datasets/listItems) supports pagination and full-text search:
+
+```typescript
+// Paginated list
+const { items, pagination } = await dataset.listItems({
+  page: 0,
+  perPage: 50,
+})
+
+// Full-text search
+const { items: matches } = await dataset.listItems({
+  search: 'Hello',
+})
+
+// List items at a specific version
+const v2Items = await dataset.listItems({ version: 2 })
+```
+
+## Versioning
+
+Every mutation to a dataset's items (add, update, or delete) bumps the dataset version. This lets you pin experiments to a specific snapshot of the data.
+
+### Listing versions
+
+Use [`listVersions()`](https://mastra.ai/reference/datasets/listVersions) to see the paginated history of versions:
+
+```typescript
+const { versions, pagination } = await dataset.listVersions()
+
+for (const v of versions) {
+  console.log(`Version ${v.version} — created ${v.createdAt}`)
+}
+```
+
+### Viewing item history
+
+See how a specific item changed across versions by calling [`getItemHistory()`](https://mastra.ai/reference/datasets/getItemHistory) with the `itemId`:
+
+```typescript
+const history = await dataset.getItemHistory({ itemId: 'item-abc-123' })
+
+for (const row of history) {
+  console.log(`Version ${row.datasetVersion}`, row.input, row.groundTruth)
+}
+```
+
+### Pinning to a version
+
+Fetch the exact items that existed at a past version:
+
+```typescript
+const items = await dataset.listItems({ version: 2 })
+```
+
+You can also pin experiments to a version, see [running experiments](https://mastra.ai/docs/observability/datasets/running-experiments).
+
+> **Info:** Visit the [`Dataset` reference](https://mastra.ai/reference/datasets/dataset) for the full list of methods and parameters.
+
+## Studio
+
+You can also manage datasets in [Mastra Studio](https://mastra.ai/docs/getting-started/studio). After opening Studio, select **Datasets** from the sidebar to see all your available datasets or create a new one.
+
+To get started, select **Create Dataset** and set a name, description, and optional schemas. After confirming, you'll see the dataset details page with two tabs: **Items** and [**Experiments**](https://mastra.ai/docs/observability/datasets/running-experiments).
+
+In the **Items** view you can add, update, and delete items, and view version history. Select **Add Item** to insert a new item with JSON editors for input and ground truth. From this view you can also import items in bulk from a CSV or JSON file. When importing, map each column to the corresponding dataset field.
+
+Select **Versions** to see the full history of changes to the dataset. After selecting **Compare Versions**, choose any two versions and select **Compare** to see a side-by-side diff of all items that were added, changed, or removed between those versions.
+
+## Related
+
+- [Running experiments](https://mastra.ai/docs/observability/datasets/running-experiments)
+- [Scorers overview](https://mastra.ai/docs/evals/overview)
+- [DatasetsManager reference](https://mastra.ai/reference/datasets/datasets-manager)
+- [Dataset reference](https://mastra.ai/reference/datasets/dataset)

package/.docs/docs/observability/datasets/running-experiments.md

@@ -0,0 +1,274 @@
+# Running Experiments
+
+**Added in:** `@mastra/core@1.4.0`
+
+An experiment runs every item in a dataset through a target (an agent, a workflow, or a scorer) and then optionally scores the outputs. Use a scorer as the target when you want to evaluate an LLM judge itself. Results are persisted to storage so you can compare runs across different prompts, models, or code changes.
+
+## Basic experiment
+
+Call [`startExperiment()`](https://mastra.ai/reference/datasets/startExperiment) with a target and scorers:
+
+```typescript
+import { mastra } from '../index'
+
+const dataset = await mastra.datasets.get({ id: 'translation-dataset-id' })
+
+const summary = await dataset.startExperiment({
+  name: 'gpt-5.1-baseline',
+  targetType: 'agent',
+  targetId: 'translation-agent',
+  scorers: ['accuracy', 'fluency'],
+})
+
+console.log(summary.status) // 'completed' | 'failed'
+console.log(summary.succeededCount) // number of items that ran successfully
+console.log(summary.failedCount) // number of items that failed
+```
+
+`startExperiment()` blocks until all items finish. For fire-and-forget execution, see [async experiments](#async-experiments).
+
+## Experiment targets
+
+You can point an experiment at a registered agent, workflow, or scorer.
+
+### Registered agent
+
+Point to an agent registered on your Mastra instance:
+
+```typescript
+const summary = await dataset.startExperiment({
+  name: 'agent-v2-eval',
+  targetType: 'agent',
+  targetId: 'translation-agent',
+  scorers: ['accuracy'],
+})
+```
+
+Each item's `input` is passed directly to `agent.generate()`, so it must be a `string`, `string[]`, or `CoreMessage[]`.
+
+### Registered workflow
+
+Point to a workflow registered on your Mastra instance:
+
+```typescript
+const summary = await dataset.startExperiment({
+  name: 'workflow-eval',
+  targetType: 'workflow',
+  targetId: 'translation-workflow',
+  scorers: ['accuracy'],
+})
+```
+
+The workflow receives each item's `input` as its trigger data.
+
+### Registered scorer
+
+Point to a scorer to evaluate an LLM judge against ground truth:
+
+```typescript
+const summary = await dataset.startExperiment({
+  name: 'judge-accuracy-eval',
+  targetType: 'scorer',
+  targetId: 'accuracy',
+})
+```
+
+The scorer receives each item's `input` and `groundTruth`. LLM-based judges can drift over time as underlying models change, so it's important to periodically realign them against known-good labels. A dataset gives you a stable benchmark to detect that drift.
+
+## Scoring results
+
+Scorers automatically run after each item's target execution. Pass scorer instances or registered scorer IDs:
+
+**Scorer IDs**:
+
+```typescript
+// Reference scorers registered on the Mastra instance
+const summary = await dataset.startExperiment({
+  name: 'with-registered-scorers',
+  targetType: 'agent',
+  targetId: 'translation-agent',
+  scorers: ['accuracy', 'fluency'],
+})
+```
+
+**Scorer instances**:
+
+```typescript
+import { createAnswerRelevancyScorer } from '@mastra/evals/scorers/prebuilt'
+
+const relevancy = createAnswerRelevancyScorer({ model: 'openai/gpt-4.1-nano' })
+
+const summary = await dataset.startExperiment({
+  name: 'with-scorer-instances',
+  targetType: 'agent',
+  targetId: 'translation-agent',
+  scorers: [relevancy],
+})
+```
+
+Each item's results include per-scorer scores:
+
+```typescript
+for (const item of summary.results) {
+  console.log(item.itemId, item.output)
+  for (const score of item.scores) {
+    console.log(`  ${score.scorerName}: ${score.score} — ${score.reason}`)
+  }
+}
+```
+
+> **Info:** Visit the [Scorers overview](https://mastra.ai/docs/evals/overview) for details on available and custom scorers.
+
+## Async experiments
+
+`startExperiment()` blocks until every item completes. For long-running datasets, use [`startExperimentAsync()`](https://mastra.ai/reference/datasets/startExperimentAsync) to start the experiment in the background:
+
+```typescript
+const { experimentId, status } = await dataset.startExperimentAsync({
+  name: 'large-dataset-run',
+  targetType: 'agent',
+  targetId: 'translation-agent',
+  scorers: ['accuracy'],
+})
+
+console.log(experimentId) // UUID
+console.log(status) // 'pending'
+```
+
+Poll for completion using [`getExperiment()`](https://mastra.ai/reference/datasets/getExperiment):
+
+```typescript
+let experiment = await dataset.getExperiment({ experimentId })
+
+while (experiment.status === 'pending' || experiment.status === 'running') {
+  await new Promise(resolve => setTimeout(resolve, 5000))
+  experiment = await dataset.getExperiment({ experimentId })
+}
+
+console.log(experiment.status) // 'completed' | 'failed'
+```
+
+## Configuration options
+
+### Concurrency
+
+Control how many items run in parallel (default: 5):
+
+```typescript
+const summary = await dataset.startExperiment({
+  targetType: 'agent',
+  targetId: 'translation-agent',
+  maxConcurrency: 10,
+})
+```
+
+### Timeouts and retries
+
+Set a per-item timeout (in milliseconds) and retry count:
+
+```typescript
+const summary = await dataset.startExperiment({
+  targetType: 'agent',
+  targetId: 'translation-agent',
+  itemTimeout: 30_000, // 30 seconds per item
+  maxRetries: 2, // retry failed items up to 2 times
+})
+```
+
+Retries use exponential backoff. Abort errors are never retried.
+
+### Aborting an experiment
+
+Pass an `AbortSignal` to cancel a running experiment:
+
+```typescript
+const controller = new AbortController()
+
+// Cancel after 60 seconds
+setTimeout(() => controller.abort(), 60_000)
+
+const summary = await dataset.startExperiment({
+  targetType: 'agent',
+  targetId: 'translation-agent',
+  signal: controller.signal,
+})
+```
+
+Remaining items are marked as skipped in the summary.
+
+### Pinning a dataset version
+
+Run against a specific snapshot of the dataset:
+
+```typescript
+const summary = await dataset.startExperiment({
+  targetType: 'agent',
+  targetId: 'translation-agent',
+  version: 3, // use items from dataset version 3
+})
+```
+
+## Viewing results
+
+### Listing experiments
+
+```typescript
+const { experiments, pagination } = await dataset.listExperiments({
+  page: 0,
+  perPage: 10,
+})
+
+for (const exp of experiments) {
+  console.log(`${exp.name} — ${exp.status} (${exp.succeededCount}/${exp.totalItems})`)
+}
+```
+
+### Experiment details
+
+```typescript
+const experiment = await dataset.getExperiment({
+  experimentId: 'exp-abc-123',
+})
+
+console.log(experiment.status)
+console.log(experiment.startedAt)
+console.log(experiment.completedAt)
+```
+
+### Item-level results
+
+```typescript
+const { results, pagination } = await dataset.listExperimentResults({
+  experimentId: 'exp-abc-123',
+  page: 0,
+  perPage: 50,
+})
+
+for (const result of results) {
+  console.log(result.itemId, result.output, result.error)
+}
+```
+
+## Understanding the summary
+
+`startExperiment()` returns an `ExperimentSummary` with counts and per-item results:
+
+- `completedWithErrors` is `true` when the experiment finished but some items failed.
+- Items cancelled via `signal` appear in `skippedCount`.
+
+> **Info:** Visit the [`startExperiment` reference](https://mastra.ai/reference/datasets/startExperiment) for the full parameter and return type documentation.
+
+## Studio
+
+You can also run experiments in [Mastra Studio](https://mastra.ai/docs/getting-started/studio). After you've added a dataset item, open it and select **Run Experiment** and configure the target, scorers, and options.
+
+After running an experiment, the **Experiments** tab shows all runs for that dataset (with status, counts, and timestamps). Select an experiment to see per-item results, scores, and execution traces.
+
+In the **Experiments** tab, select **Compare** and choose two or more experiments to compare their scores and results side by side.
+
+## Related
+
+- [Datasets overview](https://mastra.ai/docs/observability/datasets/overview)
+- [Scorers overview](https://mastra.ai/docs/evals/overview)
+- [`startExperiment` reference](https://mastra.ai/reference/datasets/startExperiment)
+- [`listExperimentResults` reference](https://mastra.ai/reference/datasets/listExperimentResults)

package/.docs/docs/observability/logging.md

@@ -0,0 +1,99 @@
+# Logging
+
+Mastra's logging system captures function execution, input data, and output responses in a structured format.
+
+When deploying to Mastra Cloud, logs are shown on the [Logs](https://mastra.ai/docs/mastra-cloud/observability) page. In self-hosted or custom environments, logs can be directed to files or external services depending on the configured transports.
+
+## Configuring logs with PinoLogger
+
+When [initializing a new Mastra project](https://mastra.ai/guides/getting-started/quickstart) using the CLI, `PinoLogger` is included by default.
+
+```typescript
+import { Mastra } from '@mastra/core/mastra'
+import { PinoLogger } from '@mastra/loggers'
+
+export const mastra = new Mastra({
+  logger: new PinoLogger({
+    name: 'Mastra',
+    level: 'info',
+  }),
+})
+```
+
+> **Info:** Visit [PinoLogger](https://mastra.ai/reference/logging/pino-logger) for all available configuration options.
+
+## Customizing logs
+
+Mastra provides access to a logger instance via the `mastra.getLogger()` method, available inside both workflow steps and tools. The logger supports standard severity levels: `debug`, `info`, `warn`, and `error`.
+
+### Logging from workflow steps
+
+Within a workflow step, access the logger via the `mastra` parameter inside the `execute` function. This allows you to log messages relevant to the step’s execution.
+
+```typescript
+import { createWorkflow, createStep } from "@mastra/core/workflows";
+import { z } from "zod";
+
+const step1 = createStep({
+  execute: async ({ mastra }) => {
+    const logger = mastra.getLogger();
+    logger.info("workflow info log");
+
+    return {
+      output: ""
+    };
+  }
+});
+
+export const testWorkflow = createWorkflow({...})
+  .then(step1)
+  .commit();
+```
+
+### Logging from tools
+
+Similarly, tools have access to the logger instance via the `mastra` parameter. Use this to log tool specific activity during execution.
+
+```typescript
+import { createTool } from '@mastra/core/tools'
+import { z } from 'zod'
+
+export const testTool = createTool({
+  execute: async (inputData, context) => {
+    const logger = context?.mastra.getLogger()
+    logger?.info('tool info log')
+
+    return {
+      output: '',
+    }
+  },
+})
+```
+
+### Logging with additional data
+
+Logger methods accept an optional second argument for additional data. This can be any value, such as an object, string, or number.
+
+In this example, the log message includes an object with a key of `agent` and a value of the `testAgent` instance.
+
+```typescript
+import { createWorkflow, createStep } from "@mastra/core/workflows";
+import { z } from "zod";
+
+const step1 = createStep({
+  execute: async ({ mastra }) => {
+    const testAgent = mastra.getAgent("testAgent");
+    const logger = mastra.getLogger();
+
+    logger.info("workflow info log", { agent: testAgent });
+
+    return {
+      output: ""
+    };
+  }
+});
+
+export const testWorkflow = createWorkflow({...})
+  .then(step1)
+  .commit();
+```

package/.docs/docs/observability/overview.md

@@ -0,0 +1,70 @@
+# Observability Overview
+
+Mastra provides observability features for AI applications. Monitor LLM operations, trace agent decisions, and debug complex workflows with tools that understand AI-specific patterns.
+
+## Key Features
+
+### Tracing
+
+Specialized tracing for AI operations that captures:
+
+- **Model interactions**: Token usage, latency, prompts, and completions
+- **Agent execution**: Decision paths, tool calls, and memory operations
+- **Workflow steps**: Branching logic, parallel execution, and step outputs
+- **Automatic instrumentation**: Tracing with decorators
+
+## Storage Requirements
+
+The `DefaultExporter` persists traces to your configured storage backend. Not all storage providers support observability—for the full list, see [Storage Provider Support](https://mastra.ai/docs/observability/tracing/exporters/default).
+
+For production environments with high traffic, we recommend using **ClickHouse** for the observability domain via [composite storage](https://mastra.ai/reference/storage/composite). See [Production Recommendations](https://mastra.ai/docs/observability/tracing/exporters/default) for details.
+
+## Quick Start
+
+Configure Observability in your Mastra instance:
+
+```typescript
+import { Mastra } from '@mastra/core'
+import { PinoLogger } from '@mastra/loggers'
+import { LibSQLStore } from '@mastra/libsql'
+import {
+  Observability,
+  DefaultExporter,
+  CloudExporter,
+  SensitiveDataFilter,
+} from '@mastra/observability'
+
+export const mastra = new Mastra({
+  logger: new PinoLogger(),
+  storage: new LibSQLStore({
+    id: 'mastra-storage',
+    url: 'file:./mastra.db', // Storage is required for tracing
+  }),
+  observability: new Observability({
+    configs: {
+      default: {
+        serviceName: 'mastra',
+        exporters: [
+          new DefaultExporter(), // Persists traces to storage for Mastra Studio
+          new CloudExporter(), // Sends traces to Mastra Cloud (if MASTRA_CLOUD_ACCESS_TOKEN is set)
+        ],
+        spanOutputProcessors: [
+          new SensitiveDataFilter(), // Redacts sensitive data like passwords, tokens, keys
+        ],
+      },
+    },
+  }),
+})
+```
+
+> **Serverless environments:** The `file:./mastra.db` storage URL uses the local filesystem, which doesn't work in serverless environments like Vercel, AWS Lambda, or Cloudflare Workers. For serverless deployments, use external storage. See the [Vercel deployment guide](https://mastra.ai/guides/deployment/vercel) for a complete example.
+
+With this basic setup, you will see Traces and Logs in both Studio and in Mastra Cloud.
+
+We also support various external tracing providers like MLflow, Langfuse, Braintrust, and any OpenTelemetry-compatible platform (Datadog, New Relic, SigNoz, etc.). See more about this in the [Tracing](https://mastra.ai/docs/observability/tracing/overview) documentation.
+
+## What's Next?
+
+- **[Set up Tracing](https://mastra.ai/docs/observability/tracing/overview)**: Configure tracing for your application
+- **[Configure Logging](https://mastra.ai/docs/observability/logging)**: Add structured logging
+- **[API Reference](https://mastra.ai/reference/observability/tracing/instances)**: Detailed configuration options