@mastra/core 1.6.0 → 1.7.0

package/dist/docs/references/reference-evals-scorer-utils.md

@@ -1,330 +0,0 @@
-# Scorer Utils
-
-Mastra provides utility functions to help extract and process data from scorer run inputs and outputs. These utilities are particularly useful in the `preprocess` step of custom scorers.
-
-## Import
-
-```typescript
-import {
-  getAssistantMessageFromRunOutput,
-  getReasoningFromRunOutput,
-  getUserMessageFromRunInput,
-  getSystemMessagesFromRunInput,
-  getCombinedSystemPrompt,
-  extractToolCalls,
-  extractInputMessages,
-  extractAgentResponseMessages,
-} from "@mastra/evals/scorers/utils";
-```
-
-## Message Extraction
-
-### getAssistantMessageFromRunOutput
-
-Extracts the text content from the first assistant message in the run output.
-
-```typescript
-const scorer = createScorer({
-  id: "my-scorer",
-  description: "My scorer",
-  type: "agent",
-})
-  .preprocess(({ run }) => {
-    const response = getAssistantMessageFromRunOutput(run.output);
-    return { response };
-  })
-  .generateScore(({ results }) => {
-    return results.preprocessStepResult?.response ? 1 : 0;
-  });
-```
-
-**output?:** (`ScorerRunOutputForAgent`): The scorer run output (array of MastraDBMessage)
-
-**Returns:** `string | undefined` - The assistant message text, or undefined if no assistant message is found.
-
-### getUserMessageFromRunInput
-
-Extracts the text content from the first user message in the run input.
-
-```typescript
-.preprocess(({ run }) => {
-  const userMessage = getUserMessageFromRunInput(run.input);
-  return { userMessage };
-})
-```
-
-**input?:** (`ScorerRunInputForAgent`): The scorer run input containing input messages
-
-**Returns:** `string | undefined` - The user message text, or undefined if no user message is found.
-
-### extractInputMessages
-
-Extracts text content from all input messages as an array.
-
-```typescript
-.preprocess(({ run }) => {
-  const allUserMessages = extractInputMessages(run.input);
-  return { conversationHistory: allUserMessages.join("\n") };
-})
-```
-
-**Returns:** `string[]` - Array of text strings from each input message.
-
-### extractAgentResponseMessages
-
-Extracts text content from all assistant response messages as an array.
-
-```typescript
-.preprocess(({ run }) => {
-  const allResponses = extractAgentResponseMessages(run.output);
-  return { allResponses };
-})
-```
-
-**Returns:** `string[]` - Array of text strings from each assistant message.
-
-## Reasoning Extraction
-
-### getReasoningFromRunOutput
-
-Extracts reasoning text from the run output. This is particularly useful when evaluating responses from reasoning models like `deepseek-reasoner` that produce chain-of-thought reasoning.
-
-Reasoning can be stored in two places:
-
-1. `content.reasoning` - a string field on the message content
-2. `content.parts` - as parts with `type: 'reasoning'` containing `details`
-
-```typescript
-import {
-  getReasoningFromRunOutput,
-  getAssistantMessageFromRunOutput
-} from "@mastra/evals/scorers/utils";
-
-const reasoningQualityScorer = createScorer({
-  id: "reasoning-quality",
-  name: "Reasoning Quality",
-  description: "Evaluates the quality of model reasoning",
-  type: "agent",
-})
-  .preprocess(({ run }) => {
-    const reasoning = getReasoningFromRunOutput(run.output);
-    const response = getAssistantMessageFromRunOutput(run.output);
-    return { reasoning, response };
-  })
-  .analyze(({ results }) => {
-    const { reasoning } = results.preprocessStepResult || {};
-    return {
-      hasReasoning: !!reasoning,
-      reasoningLength: reasoning?.length || 0,
-      hasStepByStep: reasoning?.includes("step") || false,
-    };
-  })
-  .generateScore(({ results }) => {
-    const { hasReasoning, reasoningLength } = results.analyzeStepResult || {};
-    if (!hasReasoning) return 0;
-    // Score based on reasoning length (normalized to 0-1)
-    return Math.min(reasoningLength / 500, 1);
-  })
-  .generateReason(({ results, score }) => {
-    const { hasReasoning, reasoningLength } = results.analyzeStepResult || {};
-    if (!hasReasoning) {
-      return "No reasoning was provided by the model.";
-    }
-    return `Model provided ${reasoningLength} characters of reasoning. Score: ${score}`;
-  });
-```
-
-**output?:** (`ScorerRunOutputForAgent`): The scorer run output (array of MastraDBMessage)
-
-**Returns:** `string | undefined` - The reasoning text, or undefined if no reasoning is present.
-
-## System Message Extraction
-
-### getSystemMessagesFromRunInput
-
-Extracts all system messages from the run input, including both standard system messages and tagged system messages (specialized prompts like memory instructions).
-
-```typescript
-.preprocess(({ run }) => {
-  const systemMessages = getSystemMessagesFromRunInput(run.input);
-  return {
-    systemPromptCount: systemMessages.length,
-    systemPrompts: systemMessages
-  };
-})
-```
-
-**Returns:** `string[]` - Array of system message strings.
-
-### getCombinedSystemPrompt
-
-Combines all system messages into a single prompt string, joined with double newlines.
-
-```typescript
-.preprocess(({ run }) => {
-  const fullSystemPrompt = getCombinedSystemPrompt(run.input);
-  return { fullSystemPrompt };
-})
-```
-
-**Returns:** `string` - Combined system prompt string.
-
-## Tool Call Extraction
-
-### extractToolCalls
-
-Extracts information about all tool calls from the run output, including tool names, call IDs, and their positions in the message array.
-
-```typescript
-const toolUsageScorer = createScorer({
-  id: "tool-usage",
-  description: "Evaluates tool usage patterns",
-  type: "agent",
-})
-  .preprocess(({ run }) => {
-    const { tools, toolCallInfos } = extractToolCalls(run.output);
-    return {
-      toolsUsed: tools,
-      toolCount: tools.length,
-      toolDetails: toolCallInfos,
-    };
-  })
-  .generateScore(({ results }) => {
-    const { toolCount } = results.preprocessStepResult || {};
-    // Score based on appropriate tool usage
-    return toolCount > 0 ? 1 : 0;
-  });
-```
-
-**Returns:**
-
-```typescript
-{
-  tools: string[];           // Array of tool names
-  toolCallInfos: ToolCallInfo[];  // Detailed tool call information
-}
-```
-
-Where `ToolCallInfo` is:
-
-```typescript
-type ToolCallInfo = {
-  toolName: string;      // Name of the tool
-  toolCallId: string;    // Unique call identifier
-  messageIndex: number;  // Index in the output array
-  invocationIndex: number; // Index within message's tool invocations
-};
-```
-
-## Test Utilities
-
-These utilities help create test data for scorer development.
-
-### createTestMessage
-
-Creates a `MastraDBMessage` object for testing purposes.
-
-```typescript
-import { createTestMessage } from "@mastra/evals/scorers/utils";
-
-const userMessage = createTestMessage({
-  content: "What is the weather?",
-  role: "user",
-});
-
-const assistantMessage = createTestMessage({
-  content: "The weather is sunny.",
-  role: "assistant",
-  toolInvocations: [
-    {
-      toolCallId: "call-1",
-      toolName: "weatherTool",
-      args: { location: "London" },
-      result: { temp: 20 },
-      state: "result",
-    },
-  ],
-});
-```
-
-### createAgentTestRun
-
-Creates a complete test run object for testing scorers.
-
-```typescript
-import { createAgentTestRun, createTestMessage } from "@mastra/evals/scorers/utils";
-
-const testRun = createAgentTestRun({
-  inputMessages: [
-    createTestMessage({ content: "Hello", role: "user" }),
-  ],
-  output: [
-    createTestMessage({ content: "Hi there!", role: "assistant" }),
-  ],
-});
-
-// Run your scorer with the test data
-const result = await myScorer.run({
-  input: testRun.input,
-  output: testRun.output,
-});
-```
-
-## Complete Example
-
-Here's a complete example showing how to use multiple utilities together:
-
-```typescript
-import { createScorer } from "@mastra/core/evals";
-import {
-  getAssistantMessageFromRunOutput,
-  getReasoningFromRunOutput,
-  getUserMessageFromRunInput,
-  getCombinedSystemPrompt,
-  extractToolCalls,
-} from "@mastra/evals/scorers/utils";
-
-const comprehensiveScorer = createScorer({
-  id: "comprehensive-analysis",
-  name: "Comprehensive Analysis",
-  description: "Analyzes all aspects of an agent response",
-  type: "agent",
-})
-  .preprocess(({ run }) => {
-    // Extract all relevant data
-    const userMessage = getUserMessageFromRunInput(run.input);
-    const response = getAssistantMessageFromRunOutput(run.output);
-    const reasoning = getReasoningFromRunOutput(run.output);
-    const systemPrompt = getCombinedSystemPrompt(run.input);
-    const { tools, toolCallInfos } = extractToolCalls(run.output);
-
-    return {
-      userMessage,
-      response,
-      reasoning,
-      systemPrompt,
-      toolsUsed: tools,
-      toolCount: tools.length,
-    };
-  })
-  .generateScore(({ results }) => {
-    const { response, reasoning, toolCount } = results.preprocessStepResult || {};
-
-    let score = 0;
-    if (response && response.length > 0) score += 0.4;
-    if (reasoning) score += 0.3;
-    if (toolCount > 0) score += 0.3;
-
-    return score;
-  })
-  .generateReason(({ results, score }) => {
-    const { response, reasoning, toolCount } = results.preprocessStepResult || {};
-
-    const parts = [];
-    if (response) parts.push("provided a response");
-    if (reasoning) parts.push("included reasoning");
-    if (toolCount > 0) parts.push(`used ${toolCount} tool(s)`);
-
-    return `Score: ${score}. The agent ${parts.join(", ")}.`;
-  });
-```

package/dist/docs/references/reference-evals-textual-difference.md

@@ -1,113 +0,0 @@
-# Textual Difference Scorer
-
-The `createTextualDifferenceScorer()` function uses sequence matching to measure the textual differences between two strings. It provides detailed information about changes, including the number of operations needed to transform one text into another.
-
-## Parameters
-
-The `createTextualDifferenceScorer()` function does not take any options.
-
-This function returns an instance of the MastraScorer class. See the [MastraScorer reference](https://mastra.ai/reference/evals/mastra-scorer) for details on the `.run()` method and its input/output.
-
-## .run() Returns
-
-**runId:** (`string`): The id of the run (optional).
-
-**analyzeStepResult:** (`object`): Object with difference metrics: { confidence: number, changes: number, lengthDiff: number }
-
-**score:** (`number`): Similarity ratio (0-1) where 1 indicates identical texts.
-
-`.run()` returns a result in the following shape:
-
-```typescript
-{
-  runId: string,
-  analyzeStepResult: {
-    confidence: number,
-    ratio: number,
-    changes: number,
-    lengthDiff: number
-  },
-  score: number
-}
-```
-
-## Scoring Details
-
-The scorer calculates several measures:
-
-- **Similarity Ratio**: Based on sequence matching between texts (0-1)
-- **Changes**: Count of non-matching operations needed
-- **Length Difference**: Normalized difference in text lengths
-- **Confidence**: Inversely proportional to length difference
-
-### Scoring Process
-
-1. Analyzes textual differences:
-
-   - Performs sequence matching between input and output
-   - Counts the number of change operations required
-   - Measures length differences
-
-2. Calculates metrics:
-
-   - Computes similarity ratio
-   - Determines confidence score
-   - Combines into weighted score
-
-Final score: `(similarity_ratio * confidence) * scale`
-
-### Score interpretation
-
-A textual difference score between 0 and 1:
-
-- **1.0**: Identical texts – no differences detected.
-- **0.7–0.9**: Minor differences – few changes needed.
-- **0.4–0.6**: Moderate differences – noticeable changes required.
-- **0.1–0.3**: Major differences – extensive changes needed.
-- **0.0**: Completely different texts.
-
-## Example
-
-Measure textual differences between expected and actual agent outputs:
-
-```typescript
-import { runEvals } from "@mastra/core/evals";
-import { createTextualDifferenceScorer } from "@mastra/evals/scorers/prebuilt";
-import { myAgent } from "./agent";
-
-const scorer = createTextualDifferenceScorer();
-
-const result = await runEvals({
-  data: [
-    {
-      input: "Summarize the concept of recursion",
-      groundTruth:
-        "Recursion is when a function calls itself to solve a problem by breaking it into smaller subproblems.",
-    },
-    {
-      input: "What is the capital of France?",
-      groundTruth: "The capital of France is Paris.",
-    },
-  ],
-  scorers: [scorer],
-  target: myAgent,
-  onItemComplete: ({ scorerResults }) => {
-    console.log({
-      score: scorerResults[scorer.id].score,
-      groundTruth: scorerResults[scorer.id].groundTruth,
-    });
-  },
-});
-
-console.log(result.scores);
-```
-
-For more details on `runEvals`, see the [runEvals reference](https://mastra.ai/reference/evals/run-evals).
-
-To add this scorer to an agent, see the [Scorers overview](https://mastra.ai/docs/evals/overview) guide.
-
-## Related
-
-- [Content Similarity Scorer](https://mastra.ai/reference/evals/content-similarity)
-- [Completeness Scorer](https://mastra.ai/reference/evals/completeness)
-- [Keyword Coverage Scorer](https://mastra.ai/reference/evals/keyword-coverage)

package/dist/docs/references/reference-evals-tone-consistency.md

@@ -1,119 +0,0 @@
-# Tone Consistency Scorer
-
-The `createToneScorer()` function evaluates the text's emotional tone and sentiment consistency. It can operate in two modes: comparing tone between input/output pairs or analyzing tone stability within a single text.
-
-## Parameters
-
-The `createToneScorer()` function does not take any options.
-
-This function returns an instance of the MastraScorer class. See the [MastraScorer reference](https://mastra.ai/reference/evals/mastra-scorer) for details on the `.run()` method and its input/output.
-
-## .run() Returns
-
-**runId:** (`string`): The id of the run (optional).
-
-**analyzeStepResult:** (`object`): Object with tone metrics: { responseSentiment: number, referenceSentiment: number, difference: number } (for comparison mode) OR { avgSentiment: number, sentimentVariance: number } (for stability mode)
-
-**score:** (`number`): Tone consistency/stability score (0-1).
-
-`.run()` returns a result in the following shape:
-
-```typescript
-{
-  runId: string,
-  analyzeStepResult: {
-    responseSentiment?: number,
-    referenceSentiment?: number,
-    difference?: number,
-    avgSentiment?: number,
-    sentimentVariance?: number,
-  },
-  score: number
-}
-```
-
-## Scoring Details
-
-The scorer evaluates sentiment consistency through tone pattern analysis and mode-specific scoring.
-
-### Scoring Process
-
-1. Analyzes tone patterns:
-
-   - Extracts sentiment features
-   - Computes sentiment scores
-   - Measures tone variations
-
-2. Calculates mode-specific score: **Tone Consistency** (input and output):
-
-   - Compares sentiment between texts
-   - Calculates sentiment difference
-   - Score = 1 - (sentiment\_difference / max\_difference) **Tone Stability** (single input):
-   - Analyzes sentiment across sentences
-   - Calculates sentiment variance
-   - Score = 1 - (sentiment\_variance / max\_variance)
-
-Final score: `mode_specific_score * scale`
-
-### Score interpretation
-
-(0 to scale, default 0-1)
-
-- 1.0: Perfect tone consistency/stability
-- 0.7-0.9: Strong consistency with minor variations
-- 0.4-0.6: Moderate consistency with noticeable shifts
-- 0.1-0.3: Poor consistency with major tone changes
-- 0.0: No consistency - completely different tones
-
-### analyzeStepResult
-
-Object with tone metrics:
-
-- **responseSentiment**: Sentiment score for the response (comparison mode).
-- **referenceSentiment**: Sentiment score for the input/reference (comparison mode).
-- **difference**: Absolute difference between sentiment scores (comparison mode).
-- **avgSentiment**: Average sentiment across sentences (stability mode).
-- **sentimentVariance**: Variance of sentiment across sentences (stability mode).
-
-## Example
-
-Evaluate tone consistency between related agent responses:
-
-```typescript
-import { runEvals } from "@mastra/core/evals";
-import { createToneScorer } from "@mastra/evals/scorers/prebuilt";
-import { myAgent } from "./agent";
-
-const scorer = createToneScorer();
-
-const result = await runEvals({
-  data: [
-    {
-      input: "How was your experience with our service?",
-      groundTruth: "The service was excellent and exceeded expectations!",
-    },
-    {
-      input: "Tell me about the customer support",
-      groundTruth: "The support team was friendly and very helpful.",
-    },
-  ],
-  scorers: [scorer],
-  target: myAgent,
-  onItemComplete: ({ scorerResults }) => {
-    console.log({
-      score: scorerResults[scorer.id].score,
-    });
-  },
-});
-
-console.log(result.scores);
-```
-
-For more details on `runEvals`, see the [runEvals reference](https://mastra.ai/reference/evals/run-evals).
-
-To add this scorer to an agent, see the [Scorers overview](https://mastra.ai/docs/evals/overview) guide.
-
-## Related
-
-- [Content Similarity Scorer](https://mastra.ai/reference/evals/content-similarity)
-- [Toxicity Scorer](https://mastra.ai/reference/evals/toxicity)

package/dist/docs/references/reference-evals-toxicity.md

@@ -1,123 +0,0 @@
-# Toxicity Scorer
-
-The `createToxicityScorer()` function evaluates whether an LLM's output contains racist, biased, or toxic elements. It uses a judge-based system to analyze responses for various forms of toxicity including personal attacks, mockery, hate speech, dismissive statements, and threats.
-
-## Parameters
-
-The `createToxicityScorer()` function accepts a single options object with the following properties:
-
-**model:** (`LanguageModel`): Configuration for the model used to evaluate toxicity.
-
-**scale:** (`number`): Maximum score value (default is 1). (Default: `1`)
-
-This function returns an instance of the MastraScorer class. The `.run()` method accepts the same input as other scorers (see the [MastraScorer reference](https://mastra.ai/reference/evals/mastra-scorer)), but the return value includes LLM-specific fields as documented below.
-
-## .run() Returns
-
-**runId:** (`string`): The id of the run (optional).
-
-**analyzeStepResult:** (`object`): Object with verdicts: { verdicts: Array<{ verdict: 'yes' | 'no', reason: string }> }
-
-**analyzePrompt:** (`string`): The prompt sent to the LLM for the analyze step (optional).
-
-**score:** (`number`): Toxicity score (0 to scale, default 0-1).
-
-**reason:** (`string`): Detailed explanation of the toxicity assessment.
-
-**generateReasonPrompt:** (`string`): The prompt sent to the LLM for the generateReason step (optional).
-
-`.run()` returns a result in the following shape:
-
-```typescript
-{
-  runId: string,
-  analyzeStepResult: {
-    verdicts: Array<{ verdict: 'yes' | 'no', reason: string }>
-  },
-  analyzePrompt: string,
-  score: number,
-  reason: string,
-  reasonPrompt: string
-}
-```
-
-## Scoring Details
-
-The scorer evaluates toxicity through multiple aspects:
-
-- Personal attacks
-- Mockery or sarcasm
-- Hate speech
-- Dismissive statements
-- Threats or intimidation
-
-### Scoring Process
-
-1. Analyzes toxic elements:
-
-   - Identifies personal attacks and mockery
-   - Detects hate speech and threats
-   - Evaluates dismissive statements
-   - Assesses severity levels
-
-2. Calculates toxicity score:
-
-   - Weighs detected elements
-   - Combines severity ratings
-   - Normalizes to scale
-
-Final score: `(toxicity_weighted_sum / max_toxicity) * scale`
-
-### Score interpretation
-
-A toxicity score between 0 and 1:
-
-- **0.8–1.0**: Severe toxicity.
-- **0.4–0.7**: Moderate toxicity.
-- **0.1–0.3**: Mild toxicity.
-- **0.0**: No toxic elements detected.
-
-## Example
-
-Evaluate agent responses for toxic, biased, or harmful content:
-
-```typescript
-import { runEvals } from "@mastra/core/evals";
-import { createToxicityScorer } from "@mastra/evals/scorers/prebuilt";
-import { myAgent } from "./agent";
-
-const scorer = createToxicityScorer({ model: "openai/gpt-4o" });
-
-const result = await runEvals({
-  data: [
-    {
-      input: "What do you think about the new team member?",
-    },
-    {
-      input: "How was the meeting discussion?",
-    },
-    {
-      input: "Can you provide feedback on the project proposal?",
-    },
-  ],
-  scorers: [scorer],
-  target: myAgent,
-  onItemComplete: ({ scorerResults }) => {
-    console.log({
-      score: scorerResults[scorer.id].score,
-      reason: scorerResults[scorer.id].reason,
-    });
-  },
-});
-
-console.log(result.scores);
-```
-
-For more details on `runEvals`, see the [runEvals reference](https://mastra.ai/reference/evals/run-evals).
-
-To add this scorer to an agent, see the [Scorers overview](https://mastra.ai/docs/evals/overview) guide.
-
-## Related
-
-- [Tone Consistency Scorer](https://mastra.ai/reference/evals/tone-consistency)
-- [Bias Scorer](https://mastra.ai/reference/evals/bias)