@mastra/mcp-docs-server 1.2.18-alpha.5 → 1.2.18-alpha.6

package/.docs/docs/datasets/running-experiments.md

@@ -254,7 +254,7 @@ The exported event types are `ExperimentEvent`, `ExperimentRunStartedEvent`, `Ex
 
 ## Lifecycle hooks
 
-Use lifecycle hooks to prepare state before a target runs and clean it up afterwards. This is useful when an item can't be evaluated against an empty environment. A run might need a fixture file copied into the agent's workspace, or a sandbox provisioned before the agent can touch it.
+Use lifecycle hooks to prepare state before a target runs and clean it up afterward. This is useful when an item can't be evaluated against an empty environment. A run might need a fixture file copied into the agent's workspace, or a sandbox provisioned before the agent can touch it.
 
 Hooks run at two levels. `beforeAll` and `afterAll` run once per experiment, and `beforeEach` and `afterEach` run once per item:
 
@@ -430,6 +430,91 @@ while (experiment.status === 'pending' || experiment.status === 'running') {
 console.log(experiment.status) // 'completed' | 'failed'
 ```
 
+## Caller-driven experiments
+
+`startExperiment()` puts Mastra in charge of the whole run. If a durable orchestrator (for example Temporal, Airflow, or a custom worker fleet) owns the loop instead, create the experiment first and drive it yourself in one of these shapes:
+
+- **Caller drives the loop, Mastra runs each item.** Create the experiment with a target, then call [`runExperimentItem()`](https://mastra.ai/reference/datasets/runExperimentItem) once per item. Mastra executes the target and runs the scorers, then persists the result.
+- **Caller runs everything.** Create the experiment without a target, execute and score items on your own infrastructure, and ingest each result with [`submitExperimentResult()`](https://mastra.ai/reference/datasets/submitExperimentResult).
+
+Both shapes finish with [`finalizeExperiment()`](https://mastra.ai/reference/datasets/finalizeExperiment), and both write to the same tables as native runs, so Studio views, comparisons, and review summaries work unchanged.
+
+### Run items server-side
+
+Create the experiment with [`createExperiment()`](https://mastra.ai/reference/datasets/createExperiment), passing the target and scorers. Pass your own `id` (for example a workflow run id) to make creation idempotent: retrying the call with the same id returns the existing experiment instead of failing.
+
+```typescript
+const dataset = await mastra.datasets.get({ id: 'translation-dataset-id' })
+
+const { experimentId, totalItems, datasetVersion } = await dataset.createExperiment({
+  id: 'temporal-wf-run-42', // optional: reuse your workflow run id for idempotent creates
+  targetType: 'agent',
+  targetId: 'translation-agent',
+  scorers: ['accuracy'],
+})
+```
+
+Then run each item from your orchestrator. Mastra resolves the item at the pinned dataset version and executes the target with the resolved scorers. The result upserts keyed by `(experimentId, itemId, attempt)`, so a retried activity converges on a single row:
+
+```typescript
+const { result, scores } = await dataset.runExperimentItem({
+  experimentId,
+  itemId: 'item-1',
+})
+```
+
+Retries and timeouts belong to your orchestrator: each `runExperimentItem` call executes the item exactly once. Scorers resolve with the same precedence as native runs: experiment `scorers` win over item `scorerIds`, which win over dataset `scorerIds`.
+
+### Ingest external results
+
+If your workers execute and score items themselves, create the experiment without a target:
+
+```typescript
+const { experimentId } = await dataset.createExperiment({
+  id: 'temporal-wf-run-42',
+  name: 'external-eval',
+})
+```
+
+Submit one result per item with [`submitExperimentResult()`](https://mastra.ai/reference/datasets/submitExperimentResult). Submissions are upserts keyed by `(experimentId, itemId, attempt)`, so a retried worker converges on a single row instead of duplicating results:
+
+```typescript
+await dataset.submitExperimentResult({
+  experimentId,
+  itemId: 'item-1',
+  output: { translation: 'Hola' },
+  scores: [{ scorerId: 'accuracy', score: 0.92, reason: 'Faithful translation' }],
+})
+```
+
+`input` and `groundTruth` default to the dataset item's values at the experiment's pinned dataset version. Inline `scores` are persisted to the scores store under the experiment, so they show up in comparisons alongside native scorer runs. Experiments created with a target reject `submitExperimentResult` so two writers can't race on the same rows.
+
+### Finalize
+
+When all items are done, call [`finalizeExperiment()`](https://mastra.ai/reference/datasets/finalizeExperiment). Mastra computes `succeededCount`, `failedCount`, and `skippedCount` from the persisted rows, so your workers never track completion bookkeeping. Finalization is idempotent. Calling it again returns the stored record:
+
+```typescript
+const experiment = await dataset.finalizeExperiment({ experimentId })
+
+console.log(experiment.status) // 'completed'
+console.log(experiment.succeededCount) // items with at least one attempt that didn't error
+console.log(experiment.failedCount) // items where every attempt errored
+console.log(experiment.skippedCount) // items never submitted
+```
+
+The same lifecycle is available over HTTP (`POST /api/datasets/:datasetId/experiments` with `start: false`, `POST .../experiments/:experimentId/items/:itemId/run`, `POST .../experiments/:experimentId/results`, `POST .../experiments/:experimentId/finalize`, see [Server routes](https://mastra.ai/reference/server/routes)) and through `@mastra/client-js` (see the [Datasets API](https://mastra.ai/reference/client-js/datasets)).
+
+### Retry and idempotency contract
+
+Caller-driven calls are designed to be retried aggressively (for example by a Temporal retry policy). The exact guarantees:
+
+- **Create is idempotent on `id`.** Calling `createExperiment` again with the same caller-supplied `id` returns the existing experiment. Reusing an `id` that belongs to another dataset or an experiment with a different target fails with `EXPERIMENT_ID_CONFLICT` (HTTP `409`).
+- **Results upsert on `(experimentId, itemId, attempt)`.** Re-submitting the same key updates the existing row, and the last write wins for all mutable fields. A retried worker never creates duplicates.
+- **`attempt` separates deliberate trials from retries.** Retries of the same run should reuse the same `attempt` (default `0`) so they converge. To record repeated trials of an item as separate rows, submit with `attempt: 0`, `attempt: 1`, and so on.
+- **The dataset version is pinned at creation.** Submissions are validated against the items visible at that version, so editing or deleting items afterward doesn't affect an in-flight experiment. Submitting an `itemId` that isn't visible at the pinned version fails with `404`.
+- **Finalize is idempotent and terminal.** Finalizing an already-completed experiment returns the stored record without recomputing. After finalization, further submissions are rejected with `EXPERIMENT_ALREADY_FINALIZED` (HTTP `409`).
+- **Counts are server-computed and per-item.** At finalize time Mastra rolls attempts up per item: `succeededCount` (at least one attempt without an error), `failedCount` (every attempt errored), `skippedCount` (never submitted). `succeededCount + failedCount + skippedCount === totalItems` always holds, and callers keep no completion bookkeeping. Attempt-level rows remain available via `listExperimentResults`.
+
 ## Configuration options
 
 ### Concurrency

package/.docs/docs/mastra-platform/deploy.md

@@ -28,7 +28,7 @@ A local `.env` file is optional. Environment variables stored on the platform ar
 
    On the first run the CLI prompts you to create the platform project (named after your `package.json`) and the `production` environment. Accept the prompts, or pass `--yes` to accept defaults without confirmation.
 
-2. The CLI runs a preflight check before anything ships. Storage that would fall back to a local file path (which doesn't survive on the platform's ephemeral filesystem) would normally block the deploy:
+2. The CLI runs a preflight check before every deploy. Storage that would fall back to a local file path (which doesn't survive on the platform's ephemeral filesystem) would normally block the deploy:
 
    ```text
    file:./mastra.db will be used at runtime because TURSO_DATABASE_URL is not set
@@ -48,6 +48,8 @@ A local `.env` file is optional. Environment variables stored on the platform ar
 
    The environment slug (`production` above) matches the environment the CLI would have deployed to — this is important because `mastra env db create` requires an environment argument in non-interactive shells when the project has more than one environment.
 
+   Preflight also catches database URLs that point at your local machine. A `.env` file with `REDIS_URL=redis://localhost:6379` works during development, but the deployed server can't reach your laptop — so the CLI warns and offers the same managed provisioning. If you decline, the deploy continues with your value as-is; if you accept, the managed database's connection variables take precedence at deploy time while your local `.env` keeps working for development.
+
    > **Note:** If preflight reports a hard-coded local path instead (`Build contains a host-local storage URL`), it can't offer the inline fix — guard the path with an environment variable first so the file is only used during local development:
    >
    > ```ts

package/.docs/integrations/observability/arize.md

@@ -2,7 +2,9 @@
 
 # Arize
 
-[Arize](https://arize.com/) provides observability platforms for AI applications through [Phoenix](https://phoenix.arize.com/) (open-source) and [Arize AX](https://arize.com/generative-ai/) (enterprise). The Arize exporter sends traces using OpenTelemetry and [OpenInference](https://github.com/Arize-ai/openinference/tree/main/spec) semantic conventions, compatible with any OpenTelemetry platform that supports OpenInference.
+[Arize AI](https://arize.com/?utm_source=mastra-docs\&utm_medium=partner\&utm_campaign=partner-docs\&utm_content=observability-arize) provides observability and evaluation for AI applications through [Arize Phoenix](https://arize.com/phoenix/) and [Arize AX](https://arize.com/products/ax/). Use Phoenix for a local or self-hosted open-source workflow, and use Arize AX for managed cloud or enterprise self-hosted production observability. The Arize exporter sends traces using OpenTelemetry and [OpenInference](https://github.com/Arize-ai/openinference/tree/main/spec) semantic conventions, compatible with any OpenTelemetry platform that supports OpenInference.
+
+For workflows that use traces to improve quality, see Arize's [agent evaluation guide](https://arize.com/guides/ai-agent-handbook/agent-evaluation/) and [LLM evaluation guide](https://arize.com/resources/llm-evaluation/).
 
 ## Installation
 
@@ -34,18 +36,18 @@ bun add @mastra/arize@latest
 
 ### Phoenix Setup
 
-Phoenix is an open-source observability platform that can be self-hosted or used via Phoenix Cloud.
+Phoenix is an open-source observability platform that can run locally or be self-hosted.
 
 #### Prerequisites
 
-1. **Phoenix Instance**: Deploy using Docker or sign up at [Phoenix Cloud](https://app.phoenix.arize.com/login)
+1. **Phoenix Instance**: Run Phoenix locally with Docker or connect to your self-hosted deployment
 2. **Endpoint**: Your Phoenix endpoint URL (ends in `/v1/traces`)
-3. **API Key**: Optional for unauthenticated instances, required for Phoenix Cloud
+3. **API Key**: Optional for unauthenticated instances, required for authenticated deployments
 4. **Environment Variables**: Set your configuration
 
 ```bash
 # Required
-PHOENIX_COLLECTOR_ENDPOINT=http://localhost:6006/v1/traces  # Or your Phoenix Cloud URL
+PHOENIX_COLLECTOR_ENDPOINT=http://localhost:6006/v1/traces  # Or your self-hosted Phoenix URL
 
 # Optional
 PHOENIX_API_KEY=your-api-key  # For authenticated Phoenix instances
@@ -112,7 +114,7 @@ export const mastra = new Mastra({
 
 ### Arize AX Setup
 
-Arize AX is an enterprise observability platform with advanced features for production AI systems.
+Arize AX is a managed cloud and enterprise self-hosted observability platform with advanced features for production AI systems.
 
 #### Prerequisites
 

package/.docs/reference/cli/mastra.md

@@ -541,11 +541,11 @@ mastra env db create --kind neon --name my-app-db --region aws-us-east-1 --share
 
 #### `--kind`
 
-Database provider (required). One of `turso` or `neon`.
+Database provider (required). One of `turso`, `neon`, or `redis`.
 
 #### `--name`
 
-Database name. Defaults to a name derived from the project slug (for example `my-app-db`).
+Database name. Defaults to a name derived from the project slug and provider (for example `my-app-turso` or `my-app-redis`).
 
 #### `--region`
 

package/.docs/reference/client-js/datasets.md

@@ -0,0 +1,146 @@
+> Discover all available pages from the documentation index: https://mastra.ai/llms.txt
+
+# Datasets API
+
+The Datasets API exposes Mastra's dataset and experiment routes from `MastraClient`. This page covers the caller-driven experiment methods, which let an orchestrator you own (for example a Temporal workflow) drive the experiment loop while Mastra acts as the system of record. Create the experiment, then either have Mastra execute each item server-side with `runExperimentItem` or ingest results you computed yourself with `submitExperimentResult`, and call finalize when the run is done.
+
+Item runs, result submission, and finalization are safe to retry. Creation is safe to retry only when the request includes a caller-supplied `id`; without one, each retry creates a new experiment.
+
+- `createDatasetExperiment` is idempotent on a caller-supplied `id`
+- `runExperimentItem` and `submitExperimentResult` upsert on `(experimentId, itemId, attempt)`
+- `finalizeExperiment` returns the stored record if the experiment is already completed
+
+## Usage example
+
+```typescript
+import { MastraClient } from '@mastra/client-js'
+
+const client = new MastraClient({
+  baseUrl: 'http://localhost:4111',
+})
+
+// 1. Create the experiment. Pass your orchestrator's run ID so a
+//    retried workflow converges on the same experiment.
+const { experimentId, totalItems, datasetVersion } = await client.createDatasetExperiment({
+  datasetId: 'clinical-triage-evals',
+  id: 'temporal-wf-run-42',
+  targetType: 'agent',
+  targetId: 'triage-agent',
+  scorerIds: ['clinical-judge'],
+})
+
+// 2. Run one item per activity. Mastra executes the agent, runs the
+//    scorers, and upserts the result. Retried activities converge on
+//    a single row per (experimentId, itemId, attempt).
+const { result, scores } = await client.runExperimentItem({
+  datasetId: 'clinical-triage-evals',
+  experimentId,
+  itemId: 'item-1',
+})
+
+// 3. Finalize. The server computes succeeded/failed/skipped counts
+//    from the persisted rows.
+const experiment = await client.finalizeExperiment({
+  datasetId: 'clinical-triage-evals',
+  experimentId,
+})
+```
+
+For pure ingestion, create the experiment without `targetType`/`targetId` and replace step 2 with `submitExperimentResult`.
+
+## createDatasetExperiment()
+
+Creates an experiment without starting a run. No runner is spawned, and the dataset version is pinned at creation time. Include a target when Mastra should execute items via `runExperimentItem`. Omit the target for pure ingestion via `submitExperimentResult`.
+
+**datasetId** (`string`): ID of the dataset the experiment runs against.
+
+**id** (`string`): Caller-supplied experiment ID (for example a workflow run ID). Makes creation idempotent on retry. Conflicting reuse of an ID returns a 409.
+
+**targetType** (`'agent' | 'workflow' | 'scorer'`): Type of target that runExperimentItem executes. Provide together with targetId, or omit both.
+
+**targetId** (`string`): ID of the registered target. Provide together with targetType.
+
+**scorerIds** (`string[]`): Run-level scorer IDs resolved server-side by runExperimentItem. Requires a target.
+
+**name** (`string`): Human-readable experiment name.
+
+**description** (`string`): Experiment description.
+
+**metadata** (`Record<string, unknown>`): Arbitrary metadata stored on the experiment.
+
+**version** (`number`): Dataset version to pin. Defaults to the current dataset version.
+
+**provenance** (`object`): Where the experiment came from (source, sourceId, sourceVersion, metadata).
+
+**grouping** (`object`): Grouping fields (experimentSetId, comparisonId, variantId, trialIndex) for organizing related runs.
+
+Returns `Promise<{ experimentId, status, totalItems, datasetVersion }>`.
+
+## runExperimentItem()
+
+Executes one experiment item server-side: Mastra runs the experiment's target against the item, runs the resolved scorers, and upserts the result keyed by `(experimentId, itemId, attempt)`. Requires an experiment created with a target. Scorers resolve with the same precedence as native runs: experiment `scorerIds` win over item `scorerIds`, which win over dataset `scorerIds`.
+
+**datasetId** (`string`): ID of the dataset the experiment runs against.
+
+**experimentId** (`string`): ID of an experiment created with a target.
+
+**itemId** (`string`): Dataset item to execute. Must exist at the pinned dataset version.
+
+**attempt** (`number`): Zero-based repetition index for repeated trials. Defaults to 0. Retries of the same attempt converge on one row.
+
+**requestContext** (`Record<string, unknown>`): Request context merged with the item's own request context (item values win).
+
+Returns `Promise<{ result, scores }>` with the persisted result row and the scores produced for the item.
+
+Calling it on a target-less experiment returns a `400`. Calling it after finalization returns a `409`.
+
+## submitExperimentResult()
+
+Submits (or re-submits) one externally computed item result for a target-less experiment. Submitting the same `(experimentId, itemId, attempt)` key again updates the existing row instead of creating a duplicate. Use a different `attempt` value to record deliberate repeated trials as separate rows.
+
+**datasetId** (`string`): ID of the dataset the experiment runs against.
+
+**experimentId** (`string`): ID of the target-less experiment.
+
+**itemId** (`string`): Dataset item this result belongs to. Must exist at the pinned dataset version.
+
+**attempt** (`number`): Zero-based repetition index for repeated trials. Defaults to 0. Retries of the same attempt converge on one row.
+
+**input** (`unknown`): Input replayed by the external runner. Defaults to the dataset item input.
+
+**output** (`unknown`): Output produced by the external runner.
+
+**groundTruth** (`unknown`): Ground truth. Defaults to the dataset item groundTruth.
+
+**error** (`{ message: string; stack?: string; code?: string } | null`): Failure info when the item run failed. Counted as failed at finalization.
+
+**startedAt** (`Date`): When the item run started.
+
+**completedAt** (`Date`): When the item run completed.
+
+**traceId** (`string`): Trace ID linking the result to observability data.
+
+**scores** (`Array<{ scorerId: string; scorerName?: string; score: number; reason?: string; metadata?: Record<string, unknown> }>`): Externally computed scores. Persisted keyed by runId = experimentId, so they appear in comparisons like scorer-produced scores.
+
+Returns `Promise<DatasetExperimentResult>`, the persisted result row.
+
+Submitting to an experiment that has a target returns a `400`. Submitting after finalization returns a `409`.
+
+## finalizeExperiment()
+
+Marks a caller-driven experiment completed. The server computes per-item counts from the persisted result rows, so the caller keeps no bookkeeping: `succeededCount` (at least one attempt without an error), `failedCount` (every attempt errored), and `skippedCount` (never submitted). Idempotent: finalizing an already-completed experiment returns the stored record.
+
+**datasetId** (`string`): ID of the dataset the experiment runs against.
+
+**experimentId** (`string`): ID of the experiment to finalize.
+
+Returns `Promise<DatasetExperiment>`, the updated experiment record.
+
+## Related
+
+- [Running experiments](https://mastra.ai/docs/datasets/running-experiments)
+- [dataset.createExperiment()](https://mastra.ai/reference/datasets/createExperiment)
+- [dataset.runExperimentItem()](https://mastra.ai/reference/datasets/runExperimentItem)
+- [dataset.submitExperimentResult()](https://mastra.ai/reference/datasets/submitExperimentResult)
+- [dataset.finalizeExperiment()](https://mastra.ai/reference/datasets/finalizeExperiment)
+- [Server routes](https://mastra.ai/reference/server/routes)

package/.docs/reference/configuration.md

@@ -731,6 +731,43 @@ export const mastra = new Mastra({
 })
 ```
 
+### server.handleShutdownSignals
+
+**Type:** `boolean`\
+**Default:** `true`
+
+Whether the server generated by `mastra dev` and `mastra build` installs its own `SIGINT`/`SIGTERM` handlers. By default, receiving a signal begins graceful shutdown. The generated server stops accepting new connections and drains in-flight requests before it runs `mastra.shutdown()` and exits the process (see [`server.drainTimeout`](#serverdraintimeout)).
+
+Set `handleShutdownSignals: false` to manage signals yourself, for example with a handler registered in your Mastra config module that calls `mastra.shutdown()`:
+
+```typescript
+import { Mastra } from '@mastra/core'
+
+export const mastra = new Mastra({
+  server: {
+    handleShutdownSignals: false,
+  },
+})
+
+let shuttingDown = false
+const shutdown = async () => {
+  if (shuttingDown) return
+  shuttingDown = true
+  // custom shutdown coordination
+  await mastra.shutdown()
+  process.exit(0)
+}
+
+process.once('SIGINT', shutdown)
+process.once('SIGTERM', shutdown)
+```
+
+Be aware of these caveats:
+
+- Without a handler of your own, Node.js default signal behavior applies: the process terminates immediately with no drain.
+- Your handler has no access to the HTTP server handle in the generated entry file, so it can't drain in-flight HTTP requests. If you only need a longer drain window, prefer [`server.drainTimeout`](#serverdraintimeout). For full control over the server lifecycle, use a [server adapter](https://mastra.ai/docs/server/server-adapters).
+- During `mastra dev`, the built-in handlers release storage resources between hot reloads (for example, DuckDB file locks). Disabling them without an equivalent handler can cause lock conflicts after a reload.
+
 ### server.host
 
 **Type:** `string`\
@@ -879,6 +916,27 @@ export const mastra = new Mastra({
 })
 ```
 
+### server.drainTimeout
+
+**Type:** `number`\
+**Default:** `5000` (5 seconds)
+
+Maximum time in milliseconds to drain in-flight requests after the server receives `SIGINT` or `SIGTERM`. The value must be finite and between `0` and `2147483647`. During this window the server stops accepting new connections and waits for in-flight requests (including active streams) to finish; if the deadline passes first, remaining HTTP connections are closed. Upgraded connections such as WebSockets end when the process exits. Either way, `mastra.shutdown()` then runs (bounded by its own 5 second limit) before the process exits. Setting `0` skips the drain entirely.
+
+Increase this when rolling deploys should let long-running agent turns finish. Configure it at least 5 seconds plus a safety margin below your platform's grace period so `mastra.shutdown()` can still complete. For example, account for `terminationGracePeriodSeconds` on Kubernetes:
+
+```typescript
+import { Mastra } from '@mastra/core'
+
+export const mastra = new Mastra({
+  server: {
+    drainTimeout: 600000, // 10 minutes
+  },
+})
+```
+
+Sending either shutdown signal again during the drain bypasses it and terminates the process immediately.
+
 ### server.studioBase
 
 **Type:** `string`\

package/.docs/reference/datasets/createExperiment.md

@@ -0,0 +1,76 @@
+> Discover all available pages from the documentation index: https://mastra.ai/llms.txt
+
+# dataset.createExperiment()
+
+**Added in:** `@mastra/core@1.61.0`
+
+Creates an experiment without starting a run. Your own orchestrator (for example a Temporal workflow) drives the loop in one of two shapes:
+
+- **With a target:** pass `targetType` and `targetId`, then call [`runExperimentItem()`](https://mastra.ai/reference/datasets/runExperimentItem) per item. Mastra executes the target and runs the scorers server-side.
+- **Without a target:** omit both and run everything on your own infrastructure, ingesting each result with [`submitExperimentResult()`](https://mastra.ai/reference/datasets/submitExperimentResult).
+
+Both shapes finish with [`finalizeExperiment()`](https://mastra.ai/reference/datasets/finalizeExperiment).
+
+The experiment pins the dataset version at creation time, so submissions are validated against a stable set of items even if the dataset changes afterwards.
+
+## Usage example
+
+```typescript
+import { Mastra } from '@mastra/core'
+
+const mastra = new Mastra({/* storage config */})
+
+const dataset = await mastra.datasets.get({ id: 'dataset-id' })
+
+const { experimentId, totalItems, datasetVersion } = await dataset.createExperiment({
+  id: 'temporal-wf-run-42', // optional: idempotent create on retry
+  targetType: 'agent',
+  targetId: 'translation-agent',
+  scorers: ['accuracy'],
+})
+```
+
+Passing your own `id` makes creation idempotent: calling it again with the same `id` returns the existing experiment instead of failing, so a retried workflow activity is safe. If the `id` belongs to an experiment on another dataset or an experiment with a different target, the call throws an `EXPERIMENT_ID_CONFLICT` error.
+
+`targetType` and `targetId` must be provided together, and the target must exist in the Mastra registry at create time. `scorers` requires a target because Mastra never scores target-less experiments; submit flat scores through `submitExperimentResult` instead.
+
+## Parameters
+
+**id** (`string`): Caller-supplied experiment ID (for example a workflow run ID). Makes creation idempotent on retry.
+
+**targetType** (`'agent' | 'workflow' | 'scorer'`): Type of target that runExperimentItem() executes. Provide together with targetId, or omit both for pure ingestion.
+
+**targetId** (`string`): ID of the registered target. Provide together with targetType.
+
+**scorers** (`string[]`): Run-level scorer IDs resolved server-side by runExperimentItem(). Requires a target.
+
+**name** (`string`): Human-readable experiment name.
+
+**description** (`string`): Experiment description.
+
+**metadata** (`Record<string, unknown>`): Arbitrary metadata stored on the experiment.
+
+**provenance** (`ExperimentProvenance`): Where the experiment came from (source system, ID, and version).
+
+**grouping** (`ExperimentGrouping`): Grouping fields (experimentSetId, comparisonId, variantId, trialIndex) for organizing related runs.
+
+**version** (`number`): Dataset version to pin. Defaults to the current dataset version.
+
+## Returns
+
+**result** (`Promise<object>`): Immediate response with experiment ID.
+
+**result.experimentId** (`string`): ID of the created (or existing) experiment.
+
+**result.status** (`ExperimentStatus`): 'running' for a new experiment; the stored status when an existing experiment is returned.
+
+**result.totalItems** (`number`): Number of dataset items visible at the pinned version.
+
+**result.datasetVersion** (`number`): The pinned dataset version.
+
+## Related
+
+- [dataset.runExperimentItem()](https://mastra.ai/reference/datasets/runExperimentItem)
+- [dataset.submitExperimentResult()](https://mastra.ai/reference/datasets/submitExperimentResult)
+- [dataset.finalizeExperiment()](https://mastra.ai/reference/datasets/finalizeExperiment)
+- [Running experiments](https://mastra.ai/docs/datasets/running-experiments)

package/.docs/reference/datasets/finalizeExperiment.md

@@ -0,0 +1,43 @@
+> Discover all available pages from the documentation index: https://mastra.ai/llms.txt
+
+# dataset.finalizeExperiment()
+
+**Added in:** `@mastra/core@1.61.0`
+
+Finalizes a caller-driven experiment created with [`createExperiment()`](https://mastra.ai/reference/datasets/createExperiment), with or without a target. Mastra computes the final counts server-side from the persisted result rows and marks the experiment `'completed'`, so the caller never tracks completion bookkeeping.
+
+Counts are per item, so `succeededCount + failedCount + skippedCount === totalItems` even when items have multiple attempts:
+
+- `succeededCount`: items where at least one attempt completed without an error
+- `failedCount`: items that were submitted but where every attempt errored
+- `skippedCount`: dataset items that were never submitted (any attempt)
+
+Attempt-level detail remains available through [`listExperimentResults()`](https://mastra.ai/reference/datasets/listExperimentResults).
+
+Finalization is idempotent: calling it on an already-finalized experiment returns the stored record without recomputing.
+
+## Usage example
+
+```typescript
+const experiment = await dataset.finalizeExperiment({ experimentId })
+
+console.log(experiment.status) // 'completed'
+console.log(experiment.succeededCount)
+console.log(experiment.failedCount)
+console.log(experiment.skippedCount)
+```
+
+## Parameters
+
+**experimentId** (`string`): ID of the experiment to finalize.
+
+## Returns
+
+Returns a `Promise<Experiment>`, the updated experiment record with final status, counts, and `completedAt` timestamp.
+
+## Related
+
+- [dataset.createExperiment()](https://mastra.ai/reference/datasets/createExperiment)
+- [dataset.runExperimentItem()](https://mastra.ai/reference/datasets/runExperimentItem)
+- [dataset.submitExperimentResult()](https://mastra.ai/reference/datasets/submitExperimentResult)
+- [Running experiments](https://mastra.ai/docs/datasets/running-experiments)

package/.docs/reference/datasets/runExperimentItem.md

@@ -0,0 +1,55 @@
+> Discover all available pages from the documentation index: https://mastra.ai/llms.txt
+
+# dataset.runExperimentItem()
+
+**Added in:** `@mastra/core@1.61.0`
+
+Executes one experiment item server-side: runs the experiment's target against the item, runs the resolved scorers, and upserts the result row keyed by `(experimentId, itemId, attempt)`.
+
+Built for caller-driven loops where a durable orchestrator (for example a Temporal workflow) fans out one call per item and owns retries and timeouts. A retried call converges on the same row. Requires an experiment created with a target via [`createExperiment()`](https://mastra.ai/reference/datasets/createExperiment).
+
+Scorers resolve with the same precedence as native runs: experiment `scorers` win over item `scorerIds`, which win over dataset `scorerIds`.
+
+## Usage example
+
+```typescript
+const { experimentId } = await dataset.createExperiment({
+  targetType: 'agent',
+  targetId: 'translation-agent',
+  scorers: ['accuracy'],
+})
+
+const { result, scores } = await dataset.runExperimentItem({
+  experimentId,
+  itemId: 'item-1',
+})
+
+console.log(result.output)
+console.log(scores)
+```
+
+Each call executes the item exactly once, with no internal retry loop. Calling it on a target-less experiment throws `EXPERIMENT_HAS_NO_TARGET`. Use [`submitExperimentResult()`](https://mastra.ai/reference/datasets/submitExperimentResult) for those. Calls after finalization throw `EXPERIMENT_ALREADY_FINALIZED`, and an `itemId` that isn't visible at the pinned dataset version throws `DATASET_ITEM_NOT_FOUND`.
+
+## Parameters
+
+**experimentId** (`string`): ID of an experiment created with a target.
+
+**itemId** (`string`): ID of the dataset item to execute. Must be visible at the pinned dataset version.
+
+**attempt** (`number`): Attempt number for repeated trials. Defaults to 0. Same (experimentId, itemId, attempt) upserts the existing row.
+
+**requestContext** (`Record<string, unknown>`): Request context merged with the item's own request context (item values win).
+
+## Returns
+
+**result** (`Promise<object>`): The persisted result and its scores.
+
+**result.result** (`ExperimentResult`): The persisted result row, including output, error, traceId, attempt, and timestamps.
+
+**result.scores** (`ScorerResult[]`): Scores produced by the resolved scorers for this item.
+
+## Related
+
+- [dataset.createExperiment()](https://mastra.ai/reference/datasets/createExperiment)
+- [dataset.finalizeExperiment()](https://mastra.ai/reference/datasets/finalizeExperiment)
+- [Running experiments](https://mastra.ai/docs/datasets/running-experiments)

package/.docs/reference/datasets/submitExperimentResult.md

@@ -0,0 +1,56 @@
+> Discover all available pages from the documentation index: https://mastra.ai/llms.txt
+
+# dataset.submitExperimentResult()
+
+**Added in:** `@mastra/core@1.61.0`
+
+Submits (or re-submits) one externally computed item result for a target-less experiment created with [`createExperiment()`](https://mastra.ai/reference/datasets/createExperiment).
+
+Submissions are upserts keyed by `(experimentId, itemId, attempt)`: a retried worker that submits the same key converges on a single row instead of duplicating results. Use a different `attempt` value to record repeated trials as separate rows.
+
+## Usage example
+
+```typescript
+const result = await dataset.submitExperimentResult({
+  experimentId,
+  itemId: 'item-1',
+  output: { translation: 'Hola' },
+  scores: [{ scorerId: 'accuracy', score: 0.92, reason: 'Faithful translation' }],
+})
+```
+
+The item must be visible at the experiment's pinned dataset version. Submissions to an experiment that has a target throw `EXPERIMENT_HAS_TARGET`, which prevents two writers from racing on the same rows. Submissions after finalization throw `EXPERIMENT_ALREADY_FINALIZED`.
+
+## Parameters
+
+**experimentId** (`string`): ID of the target-less experiment.
+
+**itemId** (`string`): ID of the dataset item this result belongs to.
+
+**attempt** (`number`): Attempt number for repeated trials. Defaults to 0. Same (experimentId, itemId, attempt) upserts the existing row.
+
+**input** (`unknown`): Input the worker ran. Defaults to the dataset item's input at the pinned version.
+
+**output** (`unknown`): Output produced by the external worker.
+
+**groundTruth** (`unknown`): Expected output. Defaults to the dataset item's ground truth at the pinned version.
+
+**error** (`object`): Error details (message, optional stack and code) when the item failed.
+
+**startedAt** (`Date`): When the worker started this item.
+
+**completedAt** (`Date`): When the worker finished this item.
+
+**traceId** (`string`): Trace ID linking the result to observability data.
+
+**scores** (`array`): Externally computed scores (scorerId, score, optional scorerName, reason, metadata). Persisted to the scores store under the experiment, so they appear in comparisons alongside native scorer runs. Score persistence is best-effort and never fails the submission.
+
+## Returns
+
+Returns a `Promise<ExperimentResult>`, the persisted result row, including its `id`, `attempt`, and timestamps.
+
+## Related
+
+- [dataset.createExperiment()](https://mastra.ai/reference/datasets/createExperiment)
+- [dataset.finalizeExperiment()](https://mastra.ai/reference/datasets/finalizeExperiment)
+- [Running experiments](https://mastra.ai/docs/datasets/running-experiments)

package/.docs/reference/index.md

@@ -68,6 +68,7 @@ The Reference section provides documentation of Mastra's API, including paramete
 - [mastra](https://mastra.ai/reference/cli/mastra)
 - [Agents API](https://mastra.ai/reference/client-js/agents)
 - [Conversations API](https://mastra.ai/reference/client-js/conversations)
+- [Datasets API](https://mastra.ai/reference/client-js/datasets)
 - [Error Handling](https://mastra.ai/reference/client-js/error-handling)
 - [Logs API](https://mastra.ai/reference/client-js/logs)
 - [Mastra Client SDK](https://mastra.ai/reference/client-js/mastra-client)
@@ -166,10 +167,12 @@ The Reference section provides documentation of Mastra's API, including paramete
 - [.addItems()](https://mastra.ai/reference/datasets/addItems)
 - [.compareExperiments()](https://mastra.ai/reference/datasets/compareExperiments)
 - [.create()](https://mastra.ai/reference/datasets/create)
+- [.createExperiment()](https://mastra.ai/reference/datasets/createExperiment)
 - [.delete()](https://mastra.ai/reference/datasets/delete)
 - [.deleteExperiment()](https://mastra.ai/reference/datasets/deleteExperiment)
 - [.deleteItem()](https://mastra.ai/reference/datasets/deleteItem)
 - [.deleteItems()](https://mastra.ai/reference/datasets/deleteItems)
+- [.finalizeExperiment()](https://mastra.ai/reference/datasets/finalizeExperiment)
 - [.get()](https://mastra.ai/reference/datasets/get)
 - [.getDetails()](https://mastra.ai/reference/datasets/getDetails)
 - [.getExperiment()](https://mastra.ai/reference/datasets/getExperiment)
@@ -180,8 +183,10 @@ The Reference section provides documentation of Mastra's API, including paramete
 - [.listExperiments()](https://mastra.ai/reference/datasets/listExperiments)
 - [.listItems()](https://mastra.ai/reference/datasets/listItems)
 - [.listVersions()](https://mastra.ai/reference/datasets/listVersions)
+- [.runExperimentItem()](https://mastra.ai/reference/datasets/runExperimentItem)
 - [.startExperiment()](https://mastra.ai/reference/datasets/startExperiment)
 - [.startExperimentAsync()](https://mastra.ai/reference/datasets/startExperimentAsync)
+- [.submitExperimentResult()](https://mastra.ai/reference/datasets/submitExperimentResult)
 - [.update()](https://mastra.ai/reference/datasets/update)
 - [.updateItem()](https://mastra.ai/reference/datasets/updateItem)
 - [config.ts](https://mastra.ai/reference/file-based-agents/config)

package/.docs/reference/pubsub/redis-streams.md

@@ -135,4 +135,14 @@ When a subscriber calls `nack`, the event is republished with an incremented `de
 
 Lease keys are namespaced under the same `keyPrefix` as topics, as `<keyPrefix>:lease:<key>`. All operations are atomic: `acquireLease` uses `SET NX PX` and refreshes its own TTL idempotently, while `releaseLease`, `renewLease`, and `transferLease` use Lua scripts that check ownership before mutating, so a concurrent renewal from another owner is never clobbered.
 
-You don't call these methods directly. Configuring `RedisStreamsPubSub` as the `pubsub` backend is enough for the runtime to detect and use the capability. See [`LeaseProvider`](https://mastra.ai/reference/pubsub/lease-provider) for the full method contract.
+You don't call these methods directly. Configuring `RedisStreamsPubSub` as the `pubsub` backend is enough for the runtime to detect and use the capability. See [`LeaseProvider`](https://mastra.ai/reference/pubsub/lease-provider) for the full method contract.
+
+## Managed provisioning
+
+When you deploy a project that reads `REDIS_URL` from the environment, the Mastra CLI's `deploy` preflight offers to attach a managed Redis on the Mastra platform in one step.
+
+If you accept, `mastra deploy` provisions the database and waits for it to reach the ready state before injecting `REDIS_URL` into the deploy so `new RedisStreamsPubSub({ url: process.env.REDIS_URL })` connects without extra setup.
+
+The prompt is shown only when three conditions hold: `REDIS_URL` is referenced by user code, the environment's value is missing or points to a localhost address, and the run is interactive (a terminal without the `--yes` flag). Non-interactive runs (CI, `--yes`) report the issue with a copy-pasteable `mastra env db create --kind redis` command instead of silently creating infrastructure.
+
+You can also provision a managed Redis up front with `mastra env db create --kind redis`. Once attached, the platform meters CPU seconds and memory GB-hours and shows them on the project's Database Usage page.

package/.docs/reference/server/routes.md

@@ -368,6 +368,117 @@ On authenticated servers, the read routes require the `stored-workflows:read` pe
 }
 ```
 
+## Datasets and experiments
+
+| Method   | Path                                                                   | Description                                                               |
+| -------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------- |
+| `GET`    | `/api/datasets`                                                        | List datasets                                                             |
+| `POST`   | `/api/datasets`                                                        | Create a dataset                                                          |
+| `GET`    | `/api/datasets/:datasetId`                                             | Get dataset by ID                                                         |
+| `PATCH`  | `/api/datasets/:datasetId`                                             | Update a dataset                                                          |
+| `DELETE` | `/api/datasets/:datasetId`                                             | Delete a dataset                                                          |
+| `GET`    | `/api/datasets/:datasetId/items`                                       | List dataset items                                                        |
+| `POST`   | `/api/datasets/:datasetId/items`                                       | Add a dataset item                                                        |
+| `GET`    | `/api/experiments`                                                     | List experiments across datasets                                          |
+| `GET`    | `/api/datasets/:datasetId/experiments`                                 | List experiments for a dataset                                            |
+| `POST`   | `/api/datasets/:datasetId/experiments`                                 | Trigger an experiment, or create one without starting it (`start: false`) |
+| `POST`   | `/api/datasets/:datasetId/experiments/:experimentId/items/:itemId/run` | Execute one experiment item server-side                                   |
+| `POST`   | `/api/datasets/:datasetId/experiments/:experimentId/results`           | Submit an externally computed item result                                 |
+| `POST`   | `/api/datasets/:datasetId/experiments/:experimentId/finalize`          | Finalize a caller-driven experiment                                       |
+| `GET`    | `/api/datasets/:datasetId/experiments/:experimentId`                   | Get experiment by ID                                                      |
+| `GET`    | `/api/datasets/:datasetId/experiments/:experimentId/results`           | List experiment results                                                   |
+| `POST`   | `/api/datasets/:datasetId/compare`                                     | Compare two experiments                                                   |
+
+### Caller-driven experiment routes
+
+Use these routes when your own orchestrator (for example a Temporal workflow) owns the experiment loop and Mastra acts as the system of record. Item runs, result submission, and finalization are safe to retry. Creation is safe to retry only when the request includes a caller-supplied `id`; without one, each retry creates a new experiment. Item runs and result submissions upsert on `(experimentId, itemId, attempt)`. Finalize returns the stored record if the experiment is already completed.
+
+#### Create experiment request body
+
+Post to `/api/datasets/:datasetId/experiments` with `start: false` to create the experiment without running it. Include `targetType` and `targetId` when Mastra should execute items via the run-item route. Omit both for pure ingestion via the results route.
+
+```typescript
+{
+  start?: boolean;              // false creates the experiment without running it. Defaults to true.
+  id?: string;                  // Caller-supplied experiment id (e.g. a workflow run id) for idempotent creates
+  targetType?: 'agent' | 'workflow' | 'scorer'; // Provide with targetId, or omit both
+  targetId?: string;
+  scorerIds?: string[];         // Run-level scorer ids, resolved server-side. Requires a target when start is false.
+  name?: string;
+  description?: string;
+  metadata?: Record<string, unknown>;
+  version?: number;             // Pin to a specific dataset version (defaults to latest)
+  provenance?: {
+    source?: string;
+    sourceId?: string;
+    sourceVersion?: string;
+    metadata?: Record<string, unknown>;
+  };
+  grouping?: {                  // Stable grouping dimensions for comparisons and repeated trials
+    experimentSetId?: string;
+    comparisonId?: string;
+    variantId?: string;
+    trialIndex?: number;
+  };
+}
+```
+
+Returns `{ experimentId, status, totalItems, datasetVersion }` when `start` is `false`.
+
+#### Run experiment item request body
+
+`POST /api/datasets/:datasetId/experiments/:experimentId/items/:itemId/run` executes the experiment's target against one item and runs the resolved scorers, then upserts the result. Requires an experiment created with a target.
+
+```typescript
+{
+  attempt?: number;             // Zero-based repetition index. Defaults to 0.
+  requestContext?: Record<string, unknown>; // Merged with the item's own request context (item wins)
+}
+```
+
+Returns `{ result, scores }` with the persisted result row and the scores produced for the item.
+
+#### Submit experiment result request body
+
+```typescript
+{
+  itemId: string;               // Dataset item this result belongs to
+  attempt?: number;             // Zero-based repetition index. Defaults to 0.
+  input?: unknown;              // Defaults to the dataset item input
+  output?: unknown;             // Output produced by the external runner
+  groundTruth?: unknown;        // Defaults to the dataset item groundTruth
+  error?: {                     // Failure info when the item run failed
+    message: string;
+    stack?: string;
+    code?: string;
+  } | null;
+  startedAt?: string;           // ISO date
+  completedAt?: string;         // ISO date
+  traceId?: string;
+  scores?: Array<{              // Externally computed scores, persisted keyed by runId = experimentId
+    scorerId: string;
+    scorerName?: string;
+    score: number;
+    reason?: string;
+    metadata?: Record<string, unknown>;
+  }>;
+}
+```
+
+Returns the persisted experiment result row.
+
+#### Finalize experiment
+
+`POST /api/datasets/:datasetId/experiments/:experimentId/finalize` takes no body. The server computes per-item counts from the persisted result rows and marks the experiment `completed`: `succeededCount` (at least one attempt without an error), `failedCount` (every attempt errored), and `skippedCount` (never submitted). Returns the updated experiment record.
+
+#### Caller-driven experiment error responses
+
+| Status | Condition                                                                                                                                                                                                                                                                                                                  |
+| ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `400`  | Result submitted to an experiment that has a target (`EXPERIMENT_HAS_TARGET`), item run requested on a target-less experiment (`EXPERIMENT_HAS_NO_TARGET`), `targetType` and `targetId` weren't provided together (`EXPERIMENT_INVALID_TARGET`), or the dataset has no items at the pinned version (`EXPERIMENT_NO_ITEMS`) |
+| `404`  | Dataset, experiment, dataset item, or target not found (`EXPERIMENT_TARGET_NOT_FOUND`)                                                                                                                                                                                                                                     |
+| `409`  | Caller-supplied experiment `id` already exists for a different dataset or target (`EXPERIMENT_ID_CONFLICT`), or a call arrived after finalization (`EXPERIMENT_ALREADY_FINALIZED`)                                                                                                                                         |
+
 ## MCP
 
 | Method | Path                               | Description        |

package/CHANGELOG.md

@@ -1,5 +1,12 @@
 # @mastra/mcp-docs-server
 
+## 1.2.18-alpha.6
+
+### Patch Changes
+
+- Updated dependencies [[`9267e9b`](https://github.com/mastra-ai/mastra/commit/9267e9b3d9c2fcf16936050495a787054c2431ab), [`acc3471`](https://github.com/mastra-ai/mastra/commit/acc3471de5f3fde8027ee4e355af292b2bc1bc30), [`b6a771e`](https://github.com/mastra-ai/mastra/commit/b6a771ef23d203ddb348efca8065eff65def8191), [`9267e9b`](https://github.com/mastra-ai/mastra/commit/9267e9b3d9c2fcf16936050495a787054c2431ab), [`26d4016`](https://github.com/mastra-ai/mastra/commit/26d40160ff7f7d8bf95fee2039a52cbc83863533), [`9267e9b`](https://github.com/mastra-ai/mastra/commit/9267e9b3d9c2fcf16936050495a787054c2431ab), [`9267e9b`](https://github.com/mastra-ai/mastra/commit/9267e9b3d9c2fcf16936050495a787054c2431ab), [`57c5103`](https://github.com/mastra-ai/mastra/commit/57c51035a2a36e3df3c4f32f46bb789a66ed5946)]:
+  - @mastra/core@1.61.0-alpha.3
+
 ## 1.2.18-alpha.4
 
 ### Patch Changes

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@mastra/mcp-docs-server",
-  "version": "1.2.18-alpha.5",
+  "version": "1.2.18-alpha.6",
   "description": "MCP server for accessing Mastra.ai documentation, changelogs, and news.",
   "type": "module",
   "main": "dist/index.js",
@@ -28,7 +28,7 @@
     "jsdom": "^26.1.0",
     "local-pkg": "^1.1.2",
     "zod": "^4.4.3",
-    "@mastra/core": "1.61.0-alpha.2",
+    "@mastra/core": "1.61.0-alpha.3",
     "@mastra/mcp": "^1.17.1-alpha.0"
   },
   "devDependencies": {
@@ -47,7 +47,7 @@
     "vitest": "4.1.10",
     "@internal/lint": "0.0.124",
     "@internal/types-builder": "0.0.99",
-    "@mastra/core": "1.61.0-alpha.2"
+    "@mastra/core": "1.61.0-alpha.3"
   },
   "homepage": "https://mastra.ai",
   "repository": {