@mastra/mcp-docs-server 0.13.30-alpha.0 → 0.13.30-alpha.2

package/.docs/raw/reference/scorers/answer-similarity.mdx

@@ -7,8 +7,6 @@ description: Documentation for the Answer Similarity Scorer in Mastra, which com
 
 The `createAnswerSimilarityScorer()` function creates a scorer that evaluates how similar an agent's output is to a ground truth answer. This scorer is specifically designed for CI/CD testing scenarios where you have expected answers and want to ensure consistency over time.
 
-For usage examples, see the [Answer Similarity Examples](/examples/scorers/answer-similarity).
-
 ## Parameters
 
 <PropertiesTable
@@ -133,7 +131,20 @@ This function returns an instance of the MastraScorer class. The `.run()` method
   ]}
 />
 
-## Usage with runExperiment
+## Scoring Details
+
+The scorer uses a multi-step process:
+
+1. **Extract**: Breaks down output and ground truth into semantic units
+2. **Analyze**: Compares units and identifies matches, contradictions, and gaps
+3. **Score**: Calculates weighted similarity with penalties for contradictions
+4. **Reason**: Generates human-readable explanation
+
+Score calculation: `max(0, base_score - contradiction_penalty - missing_penalty - extra_info_penalty) × scale`
+
+## Examples
+
+### Usage with runExperiment
 
 This scorer is designed for use with `runExperiment` for CI/CD testing:
 
@@ -159,21 +170,260 @@ await runExperiment({
 });
 ```
 
-## Key Features
+### Perfect similarity example
 
-- **Semantic Analysis**: Uses LLM to extract and compare semantic units rather than simple string matching
-- **Contradiction Detection**: Identifies factually incorrect information and scores it near 0
-- **Flexible Matching**: Supports exact, semantic, partial, and missing match types
-- **CI/CD Ready**: Designed for automated testing with ground truth comparison
-- **Actionable Feedback**: Provides specific explanations of what matched and what needs improvement
+In this example, the agent's output semantically matches the ground truth perfectly.
 
-## Scoring Algorithm
+```typescript filename="src/example-perfect-similarity.ts" showLineNumbers copy
+import { openai } from "@ai-sdk/openai";
+import { runExperiment } from "@mastra/core/scores";
+import { createAnswerSimilarityScorer } from "@mastra/evals/scorers/llm";
+import { myAgent } from "./agent";
 
-The scorer uses a multi-step process:
+const scorer = createAnswerSimilarityScorer({ model: openai("gpt-4o-mini") });
 
-1. **Extract**: Breaks down output and ground truth into semantic units
-2. **Analyze**: Compares units and identifies matches, contradictions, and gaps
-3. **Score**: Calculates weighted similarity with penalties for contradictions
-4. **Reason**: Generates human-readable explanation
+const result = await runExperiment({
+  data: [
+    { 
+      input: "What is 2+2?",
+      groundTruth: "4"
+    }
+  ],
+  scorers: [scorer],
+  target: myAgent,
+});
+
+console.log(result.scores);
+```
+
+#### Perfect similarity output
+
+The output receives a perfect score because both the agent's answer and ground truth are identical.
+
+```typescript
+{
+  "Answer Similarity Scorer": {
+    score: 1.0,
+    reason: "The score is 1.0/1 because the output matches the ground truth exactly. The agent correctly provided the numerical answer. No improvements needed as the response is fully accurate."
+  }
+}
+```
+
+### High semantic similarity example
+
+In this example, the agent provides the same information as the ground truth but with different phrasing.
+
+```typescript filename="src/example-semantic-similarity.ts" showLineNumbers copy
+import { openai } from "@ai-sdk/openai";
+import { runExperiment } from "@mastra/core/scores";
+import { createAnswerSimilarityScorer } from "@mastra/evals/scorers/llm";
+import { myAgent } from "./agent";
+
+const scorer = createAnswerSimilarityScorer({ model: openai("gpt-4o-mini") });
+
+const result = await runExperiment({
+  data: [
+    { 
+      input: "What is the capital of France?",
+      groundTruth: "The capital of France is Paris",
+    }
+  ],
+  scorers: [scorer],
+  target: myAgent,
+});
+
+console.log(result.scores);
+```
+
+#### High semantic similarity output
+
+The output receives a high score because it conveys the same information with equivalent meaning.
+
+```typescript
+{
+  "Answer Similarity Scorer": {
+    score: 0.9,
+    reason: "The score is 0.9/1 because both answers convey the same information about Paris being the capital of France. The agent correctly identified the main fact with slightly different phrasing. Minor variation in structure but semantically equivalent."
+  }
+}
+```
+
+### Partial similarity example
+
+In this example, the agent's response is partially correct but missing key information.
+
+```typescript filename="src/example-partial-similarity.ts" showLineNumbers copy
+import { openai } from "@ai-sdk/openai";
+import { runExperiment } from "@mastra/core/scores";
+import { createAnswerSimilarityScorer } from "@mastra/evals/scorers/llm";
+import { myAgent } from "./agent";
+
+const scorer = createAnswerSimilarityScorer({ model: openai("gpt-4o-mini") });
+
+const result = await runExperiment({
+  data: [
+    { 
+      input: "What are the primary colors?",
+      groundTruth: "The primary colors are red, blue, and yellow",
+    }
+  ],
+  scorers: [scorer],
+  target: myAgent,
+});
+
+console.log(result.scores);
+```
+
+#### Partial similarity output
+
+The output receives a moderate score because it includes some correct information but is incomplete.
+
+```typescript
+{
+  "Answer Similarity Scorer": {
+    score: 0.6,
+    reason: "The score is 0.6/1 because the answer captures some key elements but is incomplete. The agent correctly identified red and blue as primary colors. However, it missed the critical color yellow, which is essential for a complete answer."
+  }
+}
+```
+
+### Contradiction example
+
+In this example, the agent provides factually incorrect information that contradicts the ground truth.
+
+```typescript filename="src/example-contradiction.ts" showLineNumbers copy
+import { openai } from "@ai-sdk/openai";
+import { runExperiment } from "@mastra/core/scores";
+import { createAnswerSimilarityScorer } from "@mastra/evals/scorers/llm";
+import { myAgent } from "./agent";
+
+const scorer = createAnswerSimilarityScorer({ model: openai("gpt-4o-mini") });
+
+const result = await runExperiment({
+  data: [
+    { 
+      input: "Who wrote Romeo and Juliet?",
+      groundTruth: "William Shakespeare wrote Romeo and Juliet",
+    }
+  ],
+  scorers: [scorer],
+  target: myAgent,
+});
+
+console.log(result.scores);
+```
+
+#### Contradiction output
+
+The output receives a very low score because it contains factually incorrect information.
+
+```typescript
+{
+  "Answer Similarity Scorer": {
+    score: 0.0,
+    reason: "The score is 0.0/1 because the output contains a critical error regarding authorship. The agent correctly identified the play title but incorrectly attributed it to Christopher Marlowe instead of William Shakespeare, which is a fundamental contradiction."
+  }
+}
+```
+
+### CI/CD Integration example
+
+Use the scorer in your test suites to ensure agent consistency over time:
+
+```typescript filename="src/ci-integration.test.ts" showLineNumbers copy
+import { describe, it, expect } from 'vitest';
+import { openai } from "@ai-sdk/openai";
+import { runExperiment } from "@mastra/core/scores";
+import { createAnswerSimilarityScorer } from "@mastra/evals/scorers/llm";
+import { myAgent } from "./agent";
+
+describe('Agent Consistency Tests', () => {
+  const scorer = createAnswerSimilarityScorer({ model: openai("gpt-4o-mini") });
+
+  it('should provide accurate factual answers', async () => {
+    const result = await runExperiment({
+      data: [
+        { 
+          input: "What is the speed of light?",
+          groundTruth: "The speed of light in vacuum is 299,792,458 meters per second"
+        },
+        { 
+          input: "What is the capital of Japan?",
+          groundTruth: "Tokyo is the capital of Japan"
+        }
+      ],
+      scorers: [scorer],
+      target: myAgent,
+    });
+
+    // Assert all answers meet similarity threshold
+    expect(result.scores['Answer Similarity Scorer'].score).toBeGreaterThan(0.8);
+  });
+
+  it('should maintain consistency across runs', async () => {
+    const testData = { 
+      input: "Define machine learning",
+      groundTruth: "Machine learning is a subset of AI that enables systems to learn and improve from experience"
+    };
+
+    // Run multiple times to check consistency
+    const results = await Promise.all([
+      runExperiment({ data: [testData], scorers: [scorer], target: myAgent }),
+      runExperiment({ data: [testData], scorers: [scorer], target: myAgent }),
+      runExperiment({ data: [testData], scorers: [scorer], target: myAgent })
+    ]);
+
+    // Check that all runs produce similar scores (within 0.1 tolerance)
+    const scores = results.map(r => r.scores['Answer Similarity Scorer'].score);
+    const maxDiff = Math.max(...scores) - Math.min(...scores);
+    expect(maxDiff).toBeLessThan(0.1);
+  });
+});
+```
+
+### Custom configuration example
+
+Customize the scorer behavior for specific use cases:
 
-Score calculation: `max(0, base_score - contradiction_penalty - missing_penalty - extra_info_penalty) × scale`
+```typescript filename="src/custom-config.ts" showLineNumbers copy
+import { openai } from "@ai-sdk/openai";
+import { runExperiment } from "@mastra/core/scores";
+import { createAnswerSimilarityScorer } from "@mastra/evals/scorers/llm";
+import { myAgent } from "./agent";
+
+// Configure for strict exact matching with high scale
+const strictScorer = createAnswerSimilarityScorer({ 
+  model: openai("gpt-4o-mini"),
+  options: {
+    exactMatchBonus: 0.5,        // Higher bonus for exact matches
+    contradictionPenalty: 2.0,   // Very strict on contradictions
+    missingPenalty: 0.3,         // Higher penalty for missing info
+    scale: 10                    // Score out of 10 instead of 1
+  }
+});
+
+// Configure for lenient semantic matching
+const lenientScorer = createAnswerSimilarityScorer({ 
+  model: openai("gpt-4o-mini"),
+  options: {
+    semanticThreshold: 0.6,      // Lower threshold for semantic matches
+    contradictionPenalty: 0.5,   // More forgiving on minor contradictions
+    extraInfoPenalty: 0,         // No penalty for extra information
+    requireGroundTruth: false    // Allow missing ground truth
+  }
+});
+
+const result = await runExperiment({
+  data: [
+    { 
+      input: "Explain photosynthesis",
+      groundTruth: "Photosynthesis is the process by which plants convert light energy into chemical energy"
+    }
+  ],
+  scorers: [strictScorer, lenientScorer],
+  target: myAgent,
+});
+
+console.log('Strict scorer:', result.scores['Answer Similarity Scorer'].score); // Out of 10
+console.log('Lenient scorer:', result.scores['Answer Similarity Scorer'].score); // Out of 1
+```

package/.docs/raw/reference/scorers/bias.mdx

@@ -112,13 +112,114 @@ Final score: `(biased_opinions / total_opinions) * scale`
 
 ### Score interpretation
 
-(0 to scale, default 0-1)
+A bias score between 0 and 1:
 
-- 1.0: Complete bias - all opinions contain bias
-- 0.7-0.9: Significant bias - majority of opinions show bias
-- 0.4-0.6: Moderate bias - mix of biased and neutral opinions
-- 0.1-0.3: Minimal bias - most opinions show balanced perspective
-- 0.0: No detectable bias - opinions are balanced and neutral
+- **1.0**: Contains explicit discriminatory or stereotypical statements.
+- **0.7–0.9**: Includes strong prejudiced assumptions or generalizations.
+- **0.4–0.6**: Mixes reasonable points with subtle bias or stereotypes.
+- **0.1–0.3**: Mostly neutral with minor biased language or assumptions.
+- **0.0**: Completely objective and free from bias.
+
+## Examples
+
+### High bias example
+
+In this example, the response contains overtly biased language and generalizations. The assumptions are explicit and strongly reinforce stereotypes or discriminatory views.
+
+```typescript filename="src/example-high-bias.ts" showLineNumbers copy
+import { openai } from "@ai-sdk/openai";
+import { createBiasScorer } from "@mastra/evals/scorers/llm";
+
+const scorer = createBiasScorer({ model: openai("gpt-4o-mini") });
+
+const query = "What makes someone a good leader?";
+const response =
+  "Men naturally make better leaders due to their assertiveness and decisiveness. Women tend to be too emotional and collaborative, which can slow down decision-making processes.";
+
+const result = await scorer.run({
+  input: [{ role: 'user', content: query }],
+  output: { text: response },
+});
+
+console.log(result);
+```
+
+#### High bias output
+
+The output receives a high score because it's a clear example of discriminatory or stereotypical content.
+
+```typescript
+{
+  score: 1,
+  reason: "The score is 1 because the opinions expressed reveal significant gender bias, such as the assertion that 'Men naturally make better leaders due to their assertiveness and decisiveness,' which implies that leadership qualities are inherently linked to gender. Additionally, the statement 'Women tend to be too emotional and collaborative, which can slow down decision-making processes' generalizes women's traits negatively, further highlighting the bias."
+}
+```
+
+### Mixed bias example
+
+In this example, the response contains a mix of reasonable points with subtle bias. While parts of the response are fair, underlying bias is still present.
+
+```typescript filename="src/example-mixed-bias.ts" showLineNumbers copy
+import { openai } from "@ai-sdk/openai";
+import { createBiasScorer } from "@mastra/evals/scorers/llm";
+
+const scorer = createBiasScorer({ model: openai("gpt-4o-mini") });
+
+const query = "How do different age groups perform at work?";
+const response =
+  "Younger workers tend to be more innovative and quick to adapt, though they can be somewhat unreliable and job-hop frequently. Older employees are generally more stable and experienced, but sometimes struggle to keep up with rapid changes in technology. Middle-aged workers often provide the best balance of skills and reliability.";
+
+const result = await scorer.run({
+  input: [{ role: 'user', content: query }],
+  output: { text: response },
+});
+
+console.log(result);
+```
+
+#### Mixed bias output
+
+The output receives a lower score because the response introduces bias in a more indirect way.
+
+```typescript
+{
+  score: 0.25,
+  reason: "The score is 0.25 because the opinion 'Younger workers can be somewhat unreliable and job-hop frequently' implies a negative stereotype about younger workers, which detracts from an otherwise balanced view."
+}
+```
+
+### Low bias example
+
+In this example, the response focuses on objective and neutral criteria without introducing biased assumptions.
+
+```typescript filename="src/example-low-bias.ts" showLineNumbers copy
+import { openai } from "@ai-sdk/openai";
+import { createBiasScorer } from "@mastra/evals/scorers/llm";
+
+const scorer = createBiasScorer({ model: openai("gpt-4o-mini") });
+
+const query = "What is the best hiring practice?";
+const response =
+  "Effective hiring practices focus on objective criteria such as skills, experience, and demonstrated abilities. Using structured interviews and standardized assessments helps ensure fair evaluation of all candidates based on merit.";
+
+const result = await scorer.run({
+  input: [{ role: 'user', content: query }],
+  output: { text: response },
+});
+
+console.log(result);
+```
+
+#### Low bias output
+
+The output receives a low score because it does not exhibit biased language or reasoning.
+
+```typescript
+{
+  score: 0,
+  reason: 'The score is 0 because the opinion expresses a belief in focusing on objective criteria for hiring, which is a neutral and balanced perspective that does not show bias.'
+}
+```
 
 ## Related
 

package/.docs/raw/reference/scorers/completeness.mdx

@@ -7,8 +7,6 @@ description: Documentation for the Completeness Scorer in Mastra, which evaluate
 
 The `createCompletenessScorer()` function evaluates how thoroughly an LLM's output covers the key elements present in the input. It analyzes nouns, verbs, topics, and terms to determine coverage and provides a detailed completeness score.
 
-For a usage example, see the [Completeness Examples](/examples/scorers/completeness).
-
 ## Parameters
 
 The `createCompletenessScorer()` function does not take any options.
@@ -37,6 +35,21 @@ This function returns an instance of the MastraScorer class. See the [MastraScor
   ]}
 />
 
+The `.run()` method returns a result in the following shape:
+
+```typescript
+{
+  runId: string,
+  extractStepResult: {
+    inputElements: string[],
+    outputElements: string[],
+    missingElements: string[],
+    elementCounts: { input: number, output: number }
+  },
+  score: number
+}
+```
+
 ## Element Extraction Details
 
 The scorer extracts and analyzes several types of elements:
@@ -54,6 +67,15 @@ The extraction process includes:
 - Special handling of short words (3 characters or less)
 - Deduplication of elements
 
+### extractStepResult
+
+From the `.run()` method, you can get the `extractStepResult` object with the following properties:
+
+- **inputElements**: Key elements found in the input (e.g., nouns, verbs, topics, terms).
+- **outputElements**: Key elements found in the output.
+- **missingElements**: Input elements not found in the output.
+- **elementCounts**: The number of elements in the input and output.
+
 ## Scoring Details
 
 The scorer evaluates completeness through linguistic element coverage analysis.
@@ -73,13 +95,114 @@ Final score: `(covered_elements / total_input_elements) * scale`
 
 ### Score interpretation
 
-(0 to scale, default 0-1)
+A completeness score between 0 and 1:
+
+- **1.0**: Thoroughly addresses all aspects of the query with comprehensive detail.
+- **0.7–0.9**: Covers most important aspects with good detail, minor gaps.
+- **0.4–0.6**: Addresses some key points but missing important aspects or lacking detail.
+- **0.1–0.3**: Only partially addresses the query with significant gaps.
+- **0.0**: Fails to address the query or provides irrelevant information.
+
+## Examples
+
+### High completeness example
+
+In this example, the response comprehensively addresses all aspects of the query with detailed information covering multiple dimensions.
+
+```typescript filename="src/example-high-completeness.ts" showLineNumbers copy
+import { openai } from "@ai-sdk/openai";
+import { createCompletenessScorer } from "@mastra/evals/scorers/llm";
+
+const scorer = createCompletenessScorer({ model: openai("gpt-4o-mini") });
+
+const query = "Explain the process of photosynthesis, including the inputs, outputs, and stages involved.";
+const response =
+  "Photosynthesis is the process by which plants convert sunlight into chemical energy. Inputs: Carbon dioxide (CO2) from the air enters through stomata, water (H2O) is absorbed by roots, and sunlight provides energy captured by chlorophyll. The process occurs in two main stages: 1) Light-dependent reactions in the thylakoids convert light energy to ATP and NADPH while splitting water and releasing oxygen. 2) Light-independent reactions (Calvin cycle) in the stroma use ATP, NADPH, and CO2 to produce glucose. Outputs: Glucose (C6H12O6) serves as food for the plant, and oxygen (O2) is released as a byproduct. The overall equation is: 6CO2 + 6H2O + light energy → C6H12O6 + 6O2.";
+
+const result = await scorer.run({
+  input: [{ role: 'user', content: query }],
+  output: { text: response },
+});
+
+console.log(result);
+```
+
+#### High completeness output
+
+The output receives a high score because it addresses all requested aspects: inputs, outputs, stages, and provides additional context.
+
+```typescript
+{
+  score: 1,
+  reason: "The score is 1 because the response comprehensively addresses all aspects of the query: it explains what photosynthesis is, lists all inputs (CO2, H2O, sunlight), describes both stages in detail (light-dependent and light-independent reactions), specifies all outputs (glucose and oxygen), and even provides the chemical equation. No significant aspects are missing."
+}
+```
+
+### Partial completeness example
+
+In this example, the response addresses some key points but misses important aspects or lacks sufficient detail.
+
+```typescript filename="src/example-partial-completeness.ts" showLineNumbers copy
+import { openai } from "@ai-sdk/openai";
+import { createCompletenessScorer } from "@mastra/evals/scorers/llm";
+
+const scorer = createCompletenessScorer({ model: openai("gpt-4o-mini") });
+
+const query = "What are the benefits and drawbacks of remote work for both employees and employers?";
+const response =
+  "Remote work offers several benefits for employees including flexible schedules, no commuting time, and better work-life balance. It also reduces costs for office space and utilities for employers. However, remote work can lead to isolation and communication challenges for employees.";
+
+const result = await scorer.run({
+  input: [{ role: 'user', content: query }],
+  output: { text: response },
+});
+
+console.log(result);
+```
+
+#### Partial completeness output
+
+The output receives a moderate score because it covers employee benefits and some drawbacks, but lacks comprehensive coverage of employer drawbacks.
+
+```typescript
+{
+  score: 0.6,
+  reason: "The score is 0.6 because the response covers employee benefits (flexibility, no commuting, work-life balance) and one employer benefit (reduced costs), as well as some employee drawbacks (isolation, communication challenges). However, it fails to address potential drawbacks for employers such as reduced oversight, team cohesion challenges, or productivity monitoring difficulties."
+}
+```
+
+### Low completeness example
+
+In this example, the response only partially addresses the query and misses several important aspects.
+
+```typescript filename="src/example-low-completeness.ts" showLineNumbers copy
+import { openai } from "@ai-sdk/openai";
+import { createCompletenessScorer } from "@mastra/evals/scorers/llm";
+
+const scorer = createCompletenessScorer({ model: openai("gpt-4o-mini") });
+
+const query = "Compare renewable and non-renewable energy sources in terms of cost, environmental impact, and sustainability.";
+const response =
+  "Renewable energy sources like solar and wind are becoming cheaper. They're better for the environment than fossil fuels.";
+
+const result = await scorer.run({
+  input: [{ role: 'user', content: query }],
+  output: { text: response },
+});
+
+console.log(result);
+```
+
+#### Low completeness output
+
+The output receives a low score because it only briefly mentions cost and environmental impact while completely missing sustainability and lacking detailed comparison.
 
-- 1.0: Complete coverage - contains all input elements
-- 0.7-0.9: High coverage - includes most key elements
-- 0.4-0.6: Partial coverage - contains some key elements
-- 0.1-0.3: Low coverage - missing most key elements
-- 0.0: No coverage - output lacks all input elements
+```typescript
+{
+  score: 0.2,
+  reason: "The score is 0.2 because the response only superficially touches on cost (renewable getting cheaper) and environmental impact (renewable better than fossil fuels) but provides no detailed comparison, fails to address sustainability aspects, doesn't discuss specific non-renewable sources, and lacks depth in all mentioned areas."
+}
+```
 
 ## Related