@mastra/mcp-docs-server 1.1.4 → 1.1.5-alpha.0

package/.docs/reference/evals/run-evals.md

@@ -1,138 +0,0 @@
-# runEvals
-
-The `runEvals` function enables batch evaluation of agents and workflows by running multiple test cases against scorers concurrently. This is essential for systematic testing, performance analysis, and validation of AI systems.
-
-## Usage Example
-
-```typescript
-import { runEvals } from "@mastra/core/evals";
-import { myAgent } from "./agents/my-agent";
-import { myScorer1, myScorer2 } from "./scorers";
-
-const result = await runEvals({
-  target: myAgent,
-  data: [
-    { input: "What is machine learning?" },
-    { input: "Explain neural networks" },
-    { input: "How does AI work?" },
-  ],
-  scorers: [myScorer1, myScorer2],
-  concurrency: 2,
-  onItemComplete: ({ item, targetResult, scorerResults }) => {
-    console.log(`Completed: ${item.input}`);
-    console.log(`Scores:`, scorerResults);
-  },
-});
-
-console.log(`Average scores:`, result.scores);
-console.log(`Processed ${result.summary.totalItems} items`);
-```
-
-## Parameters
-
-**target:** (`Agent | Workflow`): The agent or workflow to evaluate.
-
-**data:** (`RunEvalsDataItem[]`): Array of test cases with input data and optional ground truth.
-
-**scorers:** (`MastraScorer[] | WorkflowScorerConfig`): Array of scorers for agents, or configuration object for workflows specifying scorers for the workflow and individual steps.
-
-**concurrency?:** (`number`): Number of test cases to run concurrently. (Default: `1`)
-
-**onItemComplete?:** (`function`): Callback function called after each test case completes. Receives item, target result, and scorer results.
-
-## Data Item Structure
-
-**input:** (`string | string[] | CoreMessage[] | any`): Input data for the target. For agents: messages or strings. For workflows: workflow input data.
-
-**groundTruth?:** (`any`): Expected or reference output for comparison during scoring.
-
-**requestContext?:** (`RequestContext`): Request Context to pass to the target during execution.
-
-**tracingContext?:** (`TracingContext`): Tracing context for observability and debugging.
-
-## Workflow Scorer Configuration
-
-For workflows, you can specify scorers at different levels using `WorkflowScorerConfig`:
-
-**workflow?:** (`MastraScorer[]`): Array of scorers to evaluate the entire workflow output.
-
-**steps?:** (`Record<string, MastraScorer[]>`): Object mapping step IDs to arrays of scorers for evaluating individual step outputs.
-
-## Returns
-
-**scores:** (`Record<string, any>`): Average scores across all test cases, organized by scorer name.
-
-**summary:** (`object`): Summary information about the experiment execution.
-
-**summary.totalItems:** (`number`): Total number of test cases processed.
-
-## Examples
-
-### Agent Evaluation
-
-```typescript
-import { createScorer, runEvals } from "@mastra/core/evals";
-
-const myScorer = createScorer({
-  id: "my-scorer",
-  description: "Check if Agent's response contains ground truth",
-  type: "agent",
-}).generateScore(({ run }) => {
-  const response = run.output[0]?.content || "";
-  const expectedResponse = run.groundTruth;
-  return response.includes(expectedResponse) ? 1 : 0;
-});
-
-const result = await runEvals({
-  target: chatAgent,
-  data: [
-    {
-      input: "What is AI?",
-      groundTruth:
-        "AI is a field of computer science that creates intelligent machines.",
-    },
-    {
-      input: "How does machine learning work?",
-      groundTruth:
-        "Machine learning uses algorithms to learn patterns from data.",
-    },
-  ],
-  scorers: [relevancyScorer],
-  concurrency: 3,
-});
-```
-
-### Workflow Evaluation
-
-```typescript
-const workflowResult = await runEvals({
-  target: myWorkflow,
-  data: [
-    { input: { query: "Process this data", priority: "high" } },
-    { input: { query: "Another task", priority: "low" } },
-  ],
-  scorers: {
-    workflow: [outputQualityScorer],
-    steps: {
-      "validation-step": [validationScorer],
-      "processing-step": [processingScorer],
-    },
-  },
-  onItemComplete: ({ item, targetResult, scorerResults }) => {
-    console.log(`Workflow completed for: ${item.inputData.query}`);
-    if (scorerResults.workflow) {
-      console.log("Workflow scores:", scorerResults.workflow);
-    }
-    if (scorerResults.steps) {
-      console.log("Step scores:", scorerResults.steps);
-    }
-  },
-});
-```
-
-## Related
-
-- [createScorer()](https://mastra.ai/reference/evals/create-scorer) - Create custom scorers for experiments
-- [MastraScorer](https://mastra.ai/reference/evals/mastra-scorer) - Learn about scorer structure and methods
-- [Custom Scorers](https://mastra.ai/docs/evals/custom-scorers) - Guide to building evaluation logic
-- [Scorers Overview](https://mastra.ai/docs/evals/overview) - Understanding scorer concepts

package/.docs/reference/evals/scorer-utils.md

@@ -1,330 +0,0 @@
-# Scorer Utils
-
-Mastra provides utility functions to help extract and process data from scorer run inputs and outputs. These utilities are particularly useful in the `preprocess` step of custom scorers.
-
-## Import
-
-```typescript
-import {
-  getAssistantMessageFromRunOutput,
-  getReasoningFromRunOutput,
-  getUserMessageFromRunInput,
-  getSystemMessagesFromRunInput,
-  getCombinedSystemPrompt,
-  extractToolCalls,
-  extractInputMessages,
-  extractAgentResponseMessages,
-} from "@mastra/evals/scorers/utils";
-```
-
-## Message Extraction
-
-### getAssistantMessageFromRunOutput
-
-Extracts the text content from the first assistant message in the run output.
-
-```typescript
-const scorer = createScorer({
-  id: "my-scorer",
-  description: "My scorer",
-  type: "agent",
-})
-  .preprocess(({ run }) => {
-    const response = getAssistantMessageFromRunOutput(run.output);
-    return { response };
-  })
-  .generateScore(({ results }) => {
-    return results.preprocessStepResult?.response ? 1 : 0;
-  });
-```
-
-**output?:** (`ScorerRunOutputForAgent`): The scorer run output (array of MastraDBMessage)
-
-**Returns:** `string | undefined` - The assistant message text, or undefined if no assistant message is found.
-
-### getUserMessageFromRunInput
-
-Extracts the text content from the first user message in the run input.
-
-```typescript
-.preprocess(({ run }) => {
-  const userMessage = getUserMessageFromRunInput(run.input);
-  return { userMessage };
-})
-```
-
-**input?:** (`ScorerRunInputForAgent`): The scorer run input containing input messages
-
-**Returns:** `string | undefined` - The user message text, or undefined if no user message is found.
-
-### extractInputMessages
-
-Extracts text content from all input messages as an array.
-
-```typescript
-.preprocess(({ run }) => {
-  const allUserMessages = extractInputMessages(run.input);
-  return { conversationHistory: allUserMessages.join("\n") };
-})
-```
-
-**Returns:** `string[]` - Array of text strings from each input message.
-
-### extractAgentResponseMessages
-
-Extracts text content from all assistant response messages as an array.
-
-```typescript
-.preprocess(({ run }) => {
-  const allResponses = extractAgentResponseMessages(run.output);
-  return { allResponses };
-})
-```
-
-**Returns:** `string[]` - Array of text strings from each assistant message.
-
-## Reasoning Extraction
-
-### getReasoningFromRunOutput
-
-Extracts reasoning text from the run output. This is particularly useful when evaluating responses from reasoning models like `deepseek-reasoner` that produce chain-of-thought reasoning.
-
-Reasoning can be stored in two places:
-
-1. `content.reasoning` - a string field on the message content
-2. `content.parts` - as parts with `type: 'reasoning'` containing `details`
-
-```typescript
-import {
-  getReasoningFromRunOutput,
-  getAssistantMessageFromRunOutput
-} from "@mastra/evals/scorers/utils";
-
-const reasoningQualityScorer = createScorer({
-  id: "reasoning-quality",
-  name: "Reasoning Quality",
-  description: "Evaluates the quality of model reasoning",
-  type: "agent",
-})
-  .preprocess(({ run }) => {
-    const reasoning = getReasoningFromRunOutput(run.output);
-    const response = getAssistantMessageFromRunOutput(run.output);
-    return { reasoning, response };
-  })
-  .analyze(({ results }) => {
-    const { reasoning } = results.preprocessStepResult || {};
-    return {
-      hasReasoning: !!reasoning,
-      reasoningLength: reasoning?.length || 0,
-      hasStepByStep: reasoning?.includes("step") || false,
-    };
-  })
-  .generateScore(({ results }) => {
-    const { hasReasoning, reasoningLength } = results.analyzeStepResult || {};
-    if (!hasReasoning) return 0;
-    // Score based on reasoning length (normalized to 0-1)
-    return Math.min(reasoningLength / 500, 1);
-  })
-  .generateReason(({ results, score }) => {
-    const { hasReasoning, reasoningLength } = results.analyzeStepResult || {};
-    if (!hasReasoning) {
-      return "No reasoning was provided by the model.";
-    }
-    return `Model provided ${reasoningLength} characters of reasoning. Score: ${score}`;
-  });
-```
-
-**output?:** (`ScorerRunOutputForAgent`): The scorer run output (array of MastraDBMessage)
-
-**Returns:** `string | undefined` - The reasoning text, or undefined if no reasoning is present.
-
-## System Message Extraction
-
-### getSystemMessagesFromRunInput
-
-Extracts all system messages from the run input, including both standard system messages and tagged system messages (specialized prompts like memory instructions).
-
-```typescript
-.preprocess(({ run }) => {
-  const systemMessages = getSystemMessagesFromRunInput(run.input);
-  return {
-    systemPromptCount: systemMessages.length,
-    systemPrompts: systemMessages
-  };
-})
-```
-
-**Returns:** `string[]` - Array of system message strings.
-
-### getCombinedSystemPrompt
-
-Combines all system messages into a single prompt string, joined with double newlines.
-
-```typescript
-.preprocess(({ run }) => {
-  const fullSystemPrompt = getCombinedSystemPrompt(run.input);
-  return { fullSystemPrompt };
-})
-```
-
-**Returns:** `string` - Combined system prompt string.
-
-## Tool Call Extraction
-
-### extractToolCalls
-
-Extracts information about all tool calls from the run output, including tool names, call IDs, and their positions in the message array.
-
-```typescript
-const toolUsageScorer = createScorer({
-  id: "tool-usage",
-  description: "Evaluates tool usage patterns",
-  type: "agent",
-})
-  .preprocess(({ run }) => {
-    const { tools, toolCallInfos } = extractToolCalls(run.output);
-    return {
-      toolsUsed: tools,
-      toolCount: tools.length,
-      toolDetails: toolCallInfos,
-    };
-  })
-  .generateScore(({ results }) => {
-    const { toolCount } = results.preprocessStepResult || {};
-    // Score based on appropriate tool usage
-    return toolCount > 0 ? 1 : 0;
-  });
-```
-
-**Returns:**
-
-```typescript
-{
-  tools: string[];           // Array of tool names
-  toolCallInfos: ToolCallInfo[];  // Detailed tool call information
-}
-```
-
-Where `ToolCallInfo` is:
-
-```typescript
-type ToolCallInfo = {
-  toolName: string;      // Name of the tool
-  toolCallId: string;    // Unique call identifier
-  messageIndex: number;  // Index in the output array
-  invocationIndex: number; // Index within message's tool invocations
-};
-```
-
-## Test Utilities
-
-These utilities help create test data for scorer development.
-
-### createTestMessage
-
-Creates a `MastraDBMessage` object for testing purposes.
-
-```typescript
-import { createTestMessage } from "@mastra/evals/scorers/utils";
-
-const userMessage = createTestMessage({
-  content: "What is the weather?",
-  role: "user",
-});
-
-const assistantMessage = createTestMessage({
-  content: "The weather is sunny.",
-  role: "assistant",
-  toolInvocations: [
-    {
-      toolCallId: "call-1",
-      toolName: "weatherTool",
-      args: { location: "London" },
-      result: { temp: 20 },
-      state: "result",
-    },
-  ],
-});
-```
-
-### createAgentTestRun
-
-Creates a complete test run object for testing scorers.
-
-```typescript
-import { createAgentTestRun, createTestMessage } from "@mastra/evals/scorers/utils";
-
-const testRun = createAgentTestRun({
-  inputMessages: [
-    createTestMessage({ content: "Hello", role: "user" }),
-  ],
-  output: [
-    createTestMessage({ content: "Hi there!", role: "assistant" }),
-  ],
-});
-
-// Run your scorer with the test data
-const result = await myScorer.run({
-  input: testRun.input,
-  output: testRun.output,
-});
-```
-
-## Complete Example
-
-Here's a complete example showing how to use multiple utilities together:
-
-```typescript
-import { createScorer } from "@mastra/core/evals";
-import {
-  getAssistantMessageFromRunOutput,
-  getReasoningFromRunOutput,
-  getUserMessageFromRunInput,
-  getCombinedSystemPrompt,
-  extractToolCalls,
-} from "@mastra/evals/scorers/utils";
-
-const comprehensiveScorer = createScorer({
-  id: "comprehensive-analysis",
-  name: "Comprehensive Analysis",
-  description: "Analyzes all aspects of an agent response",
-  type: "agent",
-})
-  .preprocess(({ run }) => {
-    // Extract all relevant data
-    const userMessage = getUserMessageFromRunInput(run.input);
-    const response = getAssistantMessageFromRunOutput(run.output);
-    const reasoning = getReasoningFromRunOutput(run.output);
-    const systemPrompt = getCombinedSystemPrompt(run.input);
-    const { tools, toolCallInfos } = extractToolCalls(run.output);
-
-    return {
-      userMessage,
-      response,
-      reasoning,
-      systemPrompt,
-      toolsUsed: tools,
-      toolCount: tools.length,
-    };
-  })
-  .generateScore(({ results }) => {
-    const { response, reasoning, toolCount } = results.preprocessStepResult || {};
-
-    let score = 0;
-    if (response && response.length > 0) score += 0.4;
-    if (reasoning) score += 0.3;
-    if (toolCount > 0) score += 0.3;
-
-    return score;
-  })
-  .generateReason(({ results, score }) => {
-    const { response, reasoning, toolCount } = results.preprocessStepResult || {};
-
-    const parts = [];
-    if (response) parts.push("provided a response");
-    if (reasoning) parts.push("included reasoning");
-    if (toolCount > 0) parts.push(`used ${toolCount} tool(s)`);
-
-    return `Score: ${score}. The agent ${parts.join(", ")}.`;
-  });
-```

package/.docs/reference/evals/textual-difference.md

@@ -1,113 +0,0 @@
-# Textual Difference Scorer
-
-The `createTextualDifferenceScorer()` function uses sequence matching to measure the textual differences between two strings. It provides detailed information about changes, including the number of operations needed to transform one text into another.
-
-## Parameters
-
-The `createTextualDifferenceScorer()` function does not take any options.
-
-This function returns an instance of the MastraScorer class. See the [MastraScorer reference](https://mastra.ai/reference/evals/mastra-scorer) for details on the `.run()` method and its input/output.
-
-## .run() Returns
-
-**runId:** (`string`): The id of the run (optional).
-
-**analyzeStepResult:** (`object`): Object with difference metrics: { confidence: number, changes: number, lengthDiff: number }
-
-**score:** (`number`): Similarity ratio (0-1) where 1 indicates identical texts.
-
-`.run()` returns a result in the following shape:
-
-```typescript
-{
-  runId: string,
-  analyzeStepResult: {
-    confidence: number,
-    ratio: number,
-    changes: number,
-    lengthDiff: number
-  },
-  score: number
-}
-```
-
-## Scoring Details
-
-The scorer calculates several measures:
-
-- **Similarity Ratio**: Based on sequence matching between texts (0-1)
-- **Changes**: Count of non-matching operations needed
-- **Length Difference**: Normalized difference in text lengths
-- **Confidence**: Inversely proportional to length difference
-
-### Scoring Process
-
-1. Analyzes textual differences:
-
-   - Performs sequence matching between input and output
-   - Counts the number of change operations required
-   - Measures length differences
-
-2. Calculates metrics:
-
-   - Computes similarity ratio
-   - Determines confidence score
-   - Combines into weighted score
-
-Final score: `(similarity_ratio * confidence) * scale`
-
-### Score interpretation
-
-A textual difference score between 0 and 1:
-
-- **1.0**: Identical texts – no differences detected.
-- **0.7–0.9**: Minor differences – few changes needed.
-- **0.4–0.6**: Moderate differences – noticeable changes required.
-- **0.1–0.3**: Major differences – extensive changes needed.
-- **0.0**: Completely different texts.
-
-## Example
-
-Measure textual differences between expected and actual agent outputs:
-
-```typescript
-import { runEvals } from "@mastra/core/evals";
-import { createTextualDifferenceScorer } from "@mastra/evals/scorers/prebuilt";
-import { myAgent } from "./agent";
-
-const scorer = createTextualDifferenceScorer();
-
-const result = await runEvals({
-  data: [
-    {
-      input: "Summarize the concept of recursion",
-      groundTruth:
-        "Recursion is when a function calls itself to solve a problem by breaking it into smaller subproblems.",
-    },
-    {
-      input: "What is the capital of France?",
-      groundTruth: "The capital of France is Paris.",
-    },
-  ],
-  scorers: [scorer],
-  target: myAgent,
-  onItemComplete: ({ scorerResults }) => {
-    console.log({
-      score: scorerResults[scorer.id].score,
-      groundTruth: scorerResults[scorer.id].groundTruth,
-    });
-  },
-});
-
-console.log(result.scores);
-```
-
-For more details on `runEvals`, see the [runEvals reference](https://mastra.ai/reference/evals/run-evals).
-
-To add this scorer to an agent, see the [Scorers overview](https://mastra.ai/docs/evals/overview) guide.
-
-## Related
-
-- [Content Similarity Scorer](https://mastra.ai/reference/evals/content-similarity)
-- [Completeness Scorer](https://mastra.ai/reference/evals/completeness)
-- [Keyword Coverage Scorer](https://mastra.ai/reference/evals/keyword-coverage)

package/.docs/reference/evals/tone-consistency.md

@@ -1,119 +0,0 @@
-# Tone Consistency Scorer
-
-The `createToneScorer()` function evaluates the text's emotional tone and sentiment consistency. It can operate in two modes: comparing tone between input/output pairs or analyzing tone stability within a single text.
-
-## Parameters
-
-The `createToneScorer()` function does not take any options.
-
-This function returns an instance of the MastraScorer class. See the [MastraScorer reference](https://mastra.ai/reference/evals/mastra-scorer) for details on the `.run()` method and its input/output.
-
-## .run() Returns
-
-**runId:** (`string`): The id of the run (optional).
-
-**analyzeStepResult:** (`object`): Object with tone metrics: { responseSentiment: number, referenceSentiment: number, difference: number } (for comparison mode) OR { avgSentiment: number, sentimentVariance: number } (for stability mode)
-
-**score:** (`number`): Tone consistency/stability score (0-1).
-
-`.run()` returns a result in the following shape:
-
-```typescript
-{
-  runId: string,
-  analyzeStepResult: {
-    responseSentiment?: number,
-    referenceSentiment?: number,
-    difference?: number,
-    avgSentiment?: number,
-    sentimentVariance?: number,
-  },
-  score: number
-}
-```
-
-## Scoring Details
-
-The scorer evaluates sentiment consistency through tone pattern analysis and mode-specific scoring.
-
-### Scoring Process
-
-1. Analyzes tone patterns:
-
-   - Extracts sentiment features
-   - Computes sentiment scores
-   - Measures tone variations
-
-2. Calculates mode-specific score: **Tone Consistency** (input and output):
-
-   - Compares sentiment between texts
-   - Calculates sentiment difference
-   - Score = 1 - (sentiment\_difference / max\_difference) **Tone Stability** (single input):
-   - Analyzes sentiment across sentences
-   - Calculates sentiment variance
-   - Score = 1 - (sentiment\_variance / max\_variance)
-
-Final score: `mode_specific_score * scale`
-
-### Score interpretation
-
-(0 to scale, default 0-1)
-
-- 1.0: Perfect tone consistency/stability
-- 0.7-0.9: Strong consistency with minor variations
-- 0.4-0.6: Moderate consistency with noticeable shifts
-- 0.1-0.3: Poor consistency with major tone changes
-- 0.0: No consistency - completely different tones
-
-### analyzeStepResult
-
-Object with tone metrics:
-
-- **responseSentiment**: Sentiment score for the response (comparison mode).
-- **referenceSentiment**: Sentiment score for the input/reference (comparison mode).
-- **difference**: Absolute difference between sentiment scores (comparison mode).
-- **avgSentiment**: Average sentiment across sentences (stability mode).
-- **sentimentVariance**: Variance of sentiment across sentences (stability mode).
-
-## Example
-
-Evaluate tone consistency between related agent responses:
-
-```typescript
-import { runEvals } from "@mastra/core/evals";
-import { createToneScorer } from "@mastra/evals/scorers/prebuilt";
-import { myAgent } from "./agent";
-
-const scorer = createToneScorer();
-
-const result = await runEvals({
-  data: [
-    {
-      input: "How was your experience with our service?",
-      groundTruth: "The service was excellent and exceeded expectations!",
-    },
-    {
-      input: "Tell me about the customer support",
-      groundTruth: "The support team was friendly and very helpful.",
-    },
-  ],
-  scorers: [scorer],
-  target: myAgent,
-  onItemComplete: ({ scorerResults }) => {
-    console.log({
-      score: scorerResults[scorer.id].score,
-    });
-  },
-});
-
-console.log(result.scores);
-```
-
-For more details on `runEvals`, see the [runEvals reference](https://mastra.ai/reference/evals/run-evals).
-
-To add this scorer to an agent, see the [Scorers overview](https://mastra.ai/docs/evals/overview) guide.
-
-## Related
-
-- [Content Similarity Scorer](https://mastra.ai/reference/evals/content-similarity)
-- [Toxicity Scorer](https://mastra.ai/reference/evals/toxicity)