@mastra/core 1.6.0 → 1.7.0-alpha.0

package/dist/docs/references/docs-observability-datasets-running-experiments.md

@@ -1,266 +0,0 @@
-# Running Experiments
-
-**Added in:** `@mastra/core@1.4.0`
-
-An experiment runs every item in a dataset through a target (an agent, a workflow, or a scorer) and then optionally scores the outputs. Use a scorer as the target when you want to evaluate an LLM judge itself. Results are persisted to storage so you can compare runs across different prompts, models, or code changes.
-
-## Basic experiment
-
-Call [`startExperiment()`](https://mastra.ai/reference/datasets/startExperiment) with a target and scorers:
-
-```typescript
-import { mastra } from "../index";
-
-const dataset = await mastra.datasets.get({ id: "translation-dataset-id" });
-
-const summary = await dataset.startExperiment({
-  name: "gpt-5.1-baseline",
-  targetType: "agent",
-  targetId: "translation-agent",
-  scorers: ["accuracy", "fluency"],
-});
-
-console.log(summary.status);          // 'completed' | 'failed'
-console.log(summary.succeededCount);  // number of items that ran successfully
-console.log(summary.failedCount);     // number of items that failed
-```
-
-`startExperiment()` blocks until all items finish. For fire-and-forget execution, see [async experiments](#async-experiments).
-
-## Experiment targets
-
-You can point an experiment at a registered agent, workflow, or scorer.
-
-### Registered agent
-
-Point to an agent registered on your Mastra instance:
-
-```typescript
-const summary = await dataset.startExperiment({
-  name: "agent-v2-eval",
-  targetType: "agent",
-  targetId: "translation-agent",
-  scorers: ["accuracy"],
-});
-```
-
-Each item's `input` is passed directly to `agent.generate()`, so it must be a `string`, `string[]`, or `CoreMessage[]`.
-
-### Registered workflow
-
-Point to a workflow registered on your Mastra instance:
-
-```typescript
-const summary = await dataset.startExperiment({
-  name: "workflow-eval",
-  targetType: "workflow",
-  targetId: "translation-workflow",
-  scorers: ["accuracy"],
-});
-```
-
-The workflow receives each item's `input` as its trigger data.
-
-### Registered scorer
-
-Point to a scorer to evaluate an LLM judge against ground truth:
-
-```typescript
-const summary = await dataset.startExperiment({
-  name: "judge-accuracy-eval",
-  targetType: "scorer",
-  targetId: "accuracy",
-});
-```
-
-The scorer receives each item's `input` and `groundTruth`. LLM-based judges can drift over time as underlying models change, so it's important to periodically realign them against known-good labels. A dataset gives you a stable benchmark to detect that drift.
-
-## Scoring results
-
-Scorers automatically run after each item's target execution. Pass scorer instances or registered scorer IDs:
-
-**Scorer IDs**:
-
-```typescript
-// Reference scorers registered on the Mastra instance
-const summary = await dataset.startExperiment({
-  name: "with-registered-scorers",
-  targetType: "agent",
-  targetId: "translation-agent",
-  scorers: ["accuracy", "fluency"],
-});
-```
-
-**Scorer instances**:
-
-```typescript
-import { createAnswerRelevancyScorer } from "@mastra/evals/scorers/prebuilt";
-
-const relevancy = createAnswerRelevancyScorer({ model: "openai/gpt-4.1-nano" });
-
-const summary = await dataset.startExperiment({
-  name: "with-scorer-instances",
-  targetType: "agent",
-  targetId: "translation-agent",
-  scorers: [relevancy],
-});
-```
-
-Each item's results include per-scorer scores:
-
-```typescript
-for (const item of summary.results) {
-  console.log(item.itemId, item.output);
-  for (const score of item.scores) {
-    console.log(`  ${score.scorerName}: ${score.score} — ${score.reason}`);
-  }
-}
-```
-
-> **Info:** Visit the [Scorers overview](https://mastra.ai/docs/evals/overview) for details on available and custom scorers.
-
-## Async experiments
-
-`startExperiment()` blocks until every item completes. For long-running datasets, use [`startExperimentAsync()`](https://mastra.ai/reference/datasets/startExperimentAsync) to start the experiment in the background:
-
-```typescript
-const { experimentId, status } = await dataset.startExperimentAsync({
-  name: "large-dataset-run",
-  targetType: "agent",
-  targetId: "translation-agent",
-  scorers: ["accuracy"],
-});
-
-console.log(experimentId); // UUID
-console.log(status);       // 'pending'
-```
-
-Poll for completion using [`getExperiment()`](https://mastra.ai/reference/datasets/getExperiment):
-
-```typescript
-let experiment = await dataset.getExperiment({ experimentId });
-
-while (experiment.status === "pending" || experiment.status === "running") {
-  await new Promise(resolve => setTimeout(resolve, 5000));
-  experiment = await dataset.getExperiment({ experimentId });
-}
-
-console.log(experiment.status); // 'completed' | 'failed'
-```
-
-## Configuration options
-
-### Concurrency
-
-Control how many items run in parallel (default: 5):
-
-```typescript
-const summary = await dataset.startExperiment({
-  targetType: "agent",
-  targetId: "translation-agent",
-  maxConcurrency: 10,
-});
-```
-
-### Timeouts and retries
-
-Set a per-item timeout (in milliseconds) and retry count:
-
-```typescript
-const summary = await dataset.startExperiment({
-  targetType: "agent",
-  targetId: "translation-agent",
-  itemTimeout: 30_000,  // 30 seconds per item
-  maxRetries: 2,        // retry failed items up to 2 times
-});
-```
-
-Retries use exponential backoff. Abort errors are never retried.
-
-### Aborting an experiment
-
-Pass an `AbortSignal` to cancel a running experiment:
-
-```typescript
-const controller = new AbortController();
-
-// Cancel after 60 seconds
-setTimeout(() => controller.abort(), 60_000);
-
-const summary = await dataset.startExperiment({
-  targetType: "agent",
-  targetId: "translation-agent",
-  signal: controller.signal,
-});
-```
-
-Remaining items are marked as skipped in the summary.
-
-### Pinning a dataset version
-
-Run against a specific snapshot of the dataset:
-
-```typescript
-const summary = await dataset.startExperiment({
-  targetType: "agent",
-  targetId: "translation-agent",
-  version: 3, // use items from dataset version 3
-});
-```
-
-## Viewing results
-
-### Listing experiments
-
-```typescript
-const { experiments, pagination } = await dataset.listExperiments({
-  page: 0,
-  perPage: 10,
-});
-
-for (const exp of experiments) {
-  console.log(`${exp.name} — ${exp.status} (${exp.succeededCount}/${exp.totalItems})`);
-}
-```
-
-### Experiment details
-
-```typescript
-const experiment = await dataset.getExperiment({
-  experimentId: "exp-abc-123",
-});
-
-console.log(experiment.status);
-console.log(experiment.startedAt);
-console.log(experiment.completedAt);
-```
-
-### Item-level results
-
-```typescript
-const { results, pagination } = await dataset.listExperimentResults({
-  experimentId: "exp-abc-123",
-  page: 0,
-  perPage: 50,
-});
-
-for (const result of results) {
-  console.log(result.itemId, result.output, result.error);
-}
-```
-
-## Understanding the summary
-
-`startExperiment()` returns an `ExperimentSummary` with counts and per-item results:
-
-- `completedWithErrors` is `true` when the experiment finished but some items failed.
-- Items cancelled via `signal` appear in `skippedCount`.
-
-> **Info:** Visit the [`startExperiment` reference](https://mastra.ai/reference/datasets/startExperiment) for the full parameter and return type documentation.
-
-## Related
-
-- [Datasets overview](https://mastra.ai/docs/observability/datasets/overview)
-- [Scorers overview](https://mastra.ai/docs/evals/overview)
-- [`startExperiment` reference](https://mastra.ai/reference/datasets/startExperiment)
-- [`listExperimentResults` reference](https://mastra.ai/reference/datasets/listExperimentResults)

package/dist/docs/references/docs-observability-logging.md

@@ -1,99 +0,0 @@
-# Logging
-
-Mastra's logging system captures function execution, input data, and output responses in a structured format.
-
-When deploying to Mastra Cloud, logs are shown on the [Logs](https://mastra.ai/docs/mastra-cloud/observability) page. In self-hosted or custom environments, logs can be directed to files or external services depending on the configured transports.
-
-## Configuring logs with PinoLogger
-
-When [initializing a new Mastra project](https://mastra.ai/guides/getting-started/quickstart) using the CLI, `PinoLogger` is included by default.
-
-```typescript
-import { Mastra } from "@mastra/core/mastra";
-import { PinoLogger } from "@mastra/loggers";
-
-export const mastra = new Mastra({
-  logger: new PinoLogger({
-    name: "Mastra",
-    level: "info",
-  }),
-});
-```
-
-> **Info:** Visit [PinoLogger](https://mastra.ai/reference/logging/pino-logger) for all available configuration options.
-
-## Customizing logs
-
-Mastra provides access to a logger instance via the `mastra.getLogger()` method, available inside both workflow steps and tools. The logger supports standard severity levels: `debug`, `info`, `warn`, and `error`.
-
-### Logging from workflow steps
-
-Within a workflow step, access the logger via the `mastra` parameter inside the `execute` function. This allows you to log messages relevant to the step’s execution.
-
-```typescript
-import { createWorkflow, createStep } from "@mastra/core/workflows";
-import { z } from "zod";
-
-const step1 = createStep({
-  execute: async ({ mastra }) => {
-    const logger = mastra.getLogger();
-    logger.info("workflow info log");
-
-    return {
-      output: ""
-    };
-  }
-});
-
-export const testWorkflow = createWorkflow({...})
-  .then(step1)
-  .commit();
-```
-
-### Logging from tools
-
-Similarly, tools have access to the logger instance via the `mastra` parameter. Use this to log tool specific activity during execution.
-
-```typescript
-import { createTool } from "@mastra/core/tools";
-import { z } from "zod";
-
-export const testTool = createTool({
-  execute: async (inputData, context) => {
-    const logger = context?.mastra.getLogger();
-    logger?.info("tool info log");
-
-    return {
-      output: "",
-    };
-  },
-});
-```
-
-### Logging with additional data
-
-Logger methods accept an optional second argument for additional data. This can be any value, such as an object, string, or number.
-
-In this example, the log message includes an object with a key of `agent` and a value of the `testAgent` instance.
-
-```typescript
-import { createWorkflow, createStep } from "@mastra/core/workflows";
-import { z } from "zod";
-
-const step1 = createStep({
-  execute: async ({ mastra }) => {
-    const testAgent = mastra.getAgent("testAgent");
-    const logger = mastra.getLogger();
-
-    logger.info("workflow info log", { agent: testAgent });
-
-    return {
-      output: ""
-    };
-  }
-});
-
-export const testWorkflow = createWorkflow({...})
-  .then(step1)
-  .commit();
-```

package/dist/docs/references/docs-observability-overview.md

@@ -1,70 +0,0 @@
-# Observability Overview
-
-Mastra provides observability features for AI applications. Monitor LLM operations, trace agent decisions, and debug complex workflows with tools that understand AI-specific patterns.
-
-## Key Features
-
-### Tracing
-
-Specialized tracing for AI operations that captures:
-
-- **Model interactions**: Token usage, latency, prompts, and completions
-- **Agent execution**: Decision paths, tool calls, and memory operations
-- **Workflow steps**: Branching logic, parallel execution, and step outputs
-- **Automatic instrumentation**: Tracing with decorators
-
-## Storage Requirements
-
-The `DefaultExporter` persists traces to your configured storage backend. Not all storage providers support observability—for the full list, see [Storage Provider Support](https://mastra.ai/docs/observability/tracing/exporters/default).
-
-For production environments with high traffic, we recommend using **ClickHouse** for the observability domain via [composite storage](https://mastra.ai/reference/storage/composite). See [Production Recommendations](https://mastra.ai/docs/observability/tracing/exporters/default) for details.
-
-## Quick Start
-
-Configure Observability in your Mastra instance:
-
-```typescript
-import { Mastra } from "@mastra/core";
-import { PinoLogger } from "@mastra/loggers";
-import { LibSQLStore } from "@mastra/libsql";
-import {
-  Observability,
-  DefaultExporter,
-  CloudExporter,
-  SensitiveDataFilter,
-} from "@mastra/observability";
-
-export const mastra = new Mastra({
-  logger: new PinoLogger(),
-  storage: new LibSQLStore({
-    id: 'mastra-storage',
-    url: "file:./mastra.db", // Storage is required for tracing
-  }),
-  observability: new Observability({
-    configs: {
-      default: {
-        serviceName: "mastra",
-        exporters: [
-          new DefaultExporter(), // Persists traces to storage for Mastra Studio
-          new CloudExporter(), // Sends traces to Mastra Cloud (if MASTRA_CLOUD_ACCESS_TOKEN is set)
-        ],
-        spanOutputProcessors: [
-          new SensitiveDataFilter(), // Redacts sensitive data like passwords, tokens, keys
-        ],
-      },
-    },
-  }),
-});
-```
-
-> **Serverless environments:** The `file:./mastra.db` storage URL uses the local filesystem, which doesn't work in serverless environments like Vercel, AWS Lambda, or Cloudflare Workers. For serverless deployments, use external storage. See the [Vercel deployment guide](https://mastra.ai/guides/deployment/vercel) for a complete example.
-
-With this basic setup, you will see Traces and Logs in both Studio and in Mastra Cloud.
-
-We also support various external tracing providers like MLflow, Langfuse, Braintrust, and any OpenTelemetry-compatible platform (Datadog, New Relic, SigNoz, etc.). See more about this in the [Tracing](https://mastra.ai/docs/observability/tracing/overview) documentation.
-
-## What's Next?
-
-- **[Set up Tracing](https://mastra.ai/docs/observability/tracing/overview)**: Configure tracing for your application
-- **[Configure Logging](https://mastra.ai/docs/observability/logging)**: Add structured logging
-- **[API Reference](https://mastra.ai/reference/observability/tracing/instances)**: Detailed configuration options

package/dist/docs/references/docs-observability-tracing-bridges-otel.md

@@ -1,209 +0,0 @@
-# OpenTelemetry Bridge
-
-> **Warning:** The OpenTelemetry Bridge is currently **experimental**. APIs and configuration options may change in future releases.
-
-The OpenTelemetry (OTEL) Bridge enables bidirectional integration between Mastra's tracing system and existing OpenTelemetry infrastructure. Unlike exporters that send trace data to external platforms, the bridge creates native OTEL spans that participate in your distributed tracing context.
-
-> **Looking to send traces without existing OTEL infrastructure?:** If you don't have existing OpenTelemetry instrumentation, the [OpenTelemetry Exporter](https://mastra.ai/docs/observability/tracing/exporters/otel) may be simpler — it sends traces directly without requiring an OTEL SDK setup.
-
-## When to Use the Bridge
-
-Use the OtelBridge when you:
-
-- Have existing OTEL instrumentation in your application (HTTP servers, database clients, etc.)
-- Want Mastra operations to appear as child spans of your existing OTEL traces
-- Need OTEL-instrumented code inside Mastra tools to maintain proper parent-child relationships
-- Are building a distributed system where trace context must propagate across services
-
-## How It Works
-
-The OtelBridge provides two-way integration:
-
-**From OTEL to Mastra:**
-
-- Reads from OTEL ambient context (AsyncLocalStorage) automatically
-- Inherits trace ID and parent span ID from active OTEL spans
-- Respects OTEL sampling decisions — if a trace is not sampled, Mastra won't create spans for it
-- No manual trace ID passing required when OTEL auto-instrumentation is active
-
-**From Mastra to OTEL:**
-
-- Creates native OTEL spans for Mastra operations (agents, LLM calls, tools, workflows)
-- Maintains proper parent-child relationships in distributed traces
-- Allows OTEL-instrumented code (HTTP clients, database calls) within Mastra operations to nest correctly
-
-## Installation
-
-**npm**:
-
-```bash
-npm install @mastra/otel-bridge
-```
-
-**pnpm**:
-
-```bash
-pnpm add @mastra/otel-bridge
-```
-
-**Yarn**:
-
-```bash
-yarn add @mastra/otel-bridge
-```
-
-**Bun**:
-
-```bash
-bun add @mastra/otel-bridge
-```
-
-The bridge works with your existing OpenTelemetry setup. Depending on your configuration, you may also need some of these packages:
-
-- `@opentelemetry/sdk-node` - Core Node.js SDK for OTEL
-- `@opentelemetry/auto-instrumentations-node` - Auto-instrumentation for common libraries
-- `@opentelemetry/exporter-trace-otlp-proto` - OTLP exporter (Protobuf over HTTP)
-- `@opentelemetry/exporter-trace-otlp-http` - OTLP exporter (JSON over HTTP)
-- `@opentelemetry/exporter-trace-otlp-grpc` - OTLP exporter (gRPC)
-- `@opentelemetry/sdk-trace-base` - Base tracing SDK (for BatchSpanProcessor, etc.)
-- `@opentelemetry/core` - Core utilities (for W3CTraceContextPropagator, etc.)
-
-## Configuration
-
-Using the OtelBridge requires two steps:
-
-1. Configure OpenTelemetry instrumentation in your application
-2. Add the OtelBridge to your Mastra observability config
-
-### Step 1: OpenTelemetry Instrumentation
-
-Create an instrumentation file that initializes OTEL. This must run before your application code:
-
-```typescript
-import { NodeSDK } from "@opentelemetry/sdk-node";
-import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
-import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
-import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
-import { W3CTraceContextPropagator } from "@opentelemetry/core";
-
-const sdk = new NodeSDK({
-  serviceName: "my-service",
-  spanProcessors: [
-    new BatchSpanProcessor(
-      new OTLPTraceExporter({
-        url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || "http://localhost:4318/v1/traces",
-      })
-    ),
-  ],
-  instrumentations: [getNodeAutoInstrumentations()],
-  textMapPropagator: new W3CTraceContextPropagator(),
-});
-
-sdk.start();
-
-export { sdk };
-```
-
-### Step 2: Mastra Configuration
-
-Add the OtelBridge to your Mastra observability config:
-
-```typescript
-import { Mastra } from "@mastra/core";
-import { Observability } from "@mastra/observability";
-import { OtelBridge } from "@mastra/otel-bridge";
-
-export const mastra = new Mastra({
-  observability: new Observability({
-    configs: {
-      default: {
-        serviceName: "my-service",
-        bridge: new OtelBridge(),
-      },
-    },
-  }),
-  agents: {
-    /* your agents */
-  },
-});
-```
-
-No Mastra exporters are required when using the bridge — traces are sent via your OTEL SDK configuration. You can optionally add Mastra exporters if you want to send traces to additional destinations.
-
-### Running Your Application
-
-Use the `--import` flag to ensure instrumentation loads before your application:
-
-```bash
-tsx --import ./instrumentation.ts ./src/index.ts
-```
-
-## Semantic Conventions
-
-The OtelBridge exports Mastra spans using [OpenTelemetry Semantic Conventions for GenAI v1.38.0](https://github.com/open-telemetry/semantic-conventions/tree/v1.38.0/docs/gen-ai). This includes standardized span names (`chat {model}`, `execute_tool {tool_name}`, etc.) and attributes (`gen_ai.usage.input_tokens`, `gen_ai.request.model`, etc.).
-
-For details on span naming and attributes, see the [OpenTelemetry Exporter semantic conventions](https://mastra.ai/docs/observability/tracing/exporters/otel).
-
-## Trace Hierarchy
-
-With the OtelBridge, your traces maintain proper hierarchy across OTEL and Mastra boundaries:
-
-```text
-HTTP POST /api/chat (from Hono middleware)
-└── agent.assistant (from Mastra via OtelBridge)
-    ├── chat gpt-4o (LLM call)
-    ├── tool.execute search (tool execution)
-    │   └── HTTP GET api.example.com (from OTEL auto-instrumentation)
-    └── chat gpt-4o (follow-up LLM call)
-```
-
-## Multi-Service Distributed Tracing
-
-The OtelBridge enables trace propagation across service boundaries. When Service A calls Service B via HTTP, trace context propagates automatically:
-
-```text
-Service A: HTTP POST /api/process
-└── HTTP POST service-b/api/analyze (outgoing call)
-
-Service B: HTTP POST /api/analyze (incoming call - same trace!)
-└── agent.analyzer (Mastra agent inherits trace context)
-    └── chat gpt-4o
-```
-
-Both services must have:
-
-1. OTEL instrumentation configured
-2. W3C Trace Context propagator enabled
-3. Mastra with OtelBridge configured
-
-## Using Tags
-
-Tags help you categorize and filter traces in your OTEL backend. Add tags when executing agents or workflows:
-
-```typescript
-const result = await agent.generate("Hello", {
-  tracingOptions: {
-    tags: ["production", "experiment-v2", "user-request"],
-  },
-});
-```
-
-Tags are exported as a JSON string in the `mastra.tags` span attribute for broad backend compatibility. Common use cases include:
-
-- Environment labels: `"production"`, `"staging"`
-- Experiment tracking: `"experiment-v1"`, `"control-group"`
-- Priority levels: `"priority-high"`, `"batch-job"`
-
-## Troubleshooting
-
-If traces aren't appearing or connecting as expected:
-
-- Verify OTEL SDK is initialized before Mastra (use `--import` flag or import at top of entry point)
-- Ensure the OtelBridge is added to your observability config
-- Check that your OTEL backend is running and accessible
-
-## Related
-
-- [Tracing Overview](https://mastra.ai/docs/observability/tracing/overview)
-- [OpenTelemetry Exporter](https://mastra.ai/docs/observability/tracing/exporters/otel) - For sending traces to OTEL backends
-- [OtelBridge Reference](https://mastra.ai/reference/observability/tracing/bridges/otel) - API documentation