@mastra/mcp-docs-server 0.13.39 → 1.0.0-beta.1

package/.docs/raw/reference/evals/bias.mdx

@@ -1,123 +1,89 @@
 ---
-title: "Reference: BiasMetric | Evals | Mastra Docs"
-description: Documentation for the Bias Metric in Mastra, which evaluates LLM outputs for various forms of bias, including gender, political, racial/ethnic, or geographical bias.
+title: "Reference: Bias Scorer | Evals | Mastra Docs"
+description: Documentation for the Bias Scorer in Mastra, which evaluates LLM outputs for various forms of bias, including gender, political, racial/ethnic, or geographical bias.
 ---
 
-# BiasMetric
+# Bias Scorer
 
-:::info Scorers
-This documentation refers to the legacy evals API. For the latest scorer features, see [Scorers](/docs/scorers/overview).
-:::
+The `createBiasScorer()` function accepts a single options object with the following properties:
 
-The `BiasMetric` class evaluates whether an LLM's output contains various forms of bias, including gender, political, racial/ethnic, or geographical bias. This metric is particularly useful for detecting unintended biases that may emerge after fine-tuning models or applying optimizations.
-
-## Basic Usage
-
-```typescript
-import { openai } from "@ai-sdk/openai";
-import { BiasMetric } from "@mastra/evals/llm";
-
-// Configure the model for evaluation
-const model = openai("gpt-4o-mini");
-
-const metric = new BiasMetric(model, {
-  scale: 1,
-});
-
-const result = await metric.measure(
-  "Describe leadership qualities in the workplace",
-  "Strong leaders are typically men who can make tough decisions...",
-);
-
-console.log(result.score); // Score from 0-1, higher means more biased
-```
-
-## Constructor Parameters
+## Parameters
 
 <PropertiesTable
   content={[
     {
       name: "model",
       type: "LanguageModel",
-      description: "Configuration for the model used to evaluate bias",
-      isOptional: false,
+      required: true,
+      description: "Configuration for the model used to evaluate bias.",
     },
-    {
-      name: "options",
-      type: "BiasMetricOptions",
-      description: "Configuration options for the metric",
-      isOptional: true,
-      defaultValue: "{ scale: 1 }",
-    },
-  ]}
-/>
-
-### BiasMetricOptions
-
-<PropertiesTable
-  content={[
     {
       name: "scale",
       type: "number",
-      description: "Maximum score value",
-      isOptional: true,
+      required: false,
       defaultValue: "1",
+      description: "Maximum score value.",
     },
   ]}
 />
 
-## measure() Parameters
+This function returns an instance of the MastraScorer class. The `.run()` method accepts the same input as other scorers (see the [MastraScorer reference](./mastra-scorer)), but the return value includes LLM-specific fields as documented below.
+
+## .run() Returns
 
 <PropertiesTable
   content={[
     {
-      name: "input",
+      name: "runId",
+      type: "string",
+      description: "The id of the run (optional).",
+    },
+    {
+      name: "preprocessStepResult",
+      type: "object",
+      description: "Object with extracted opinions: { opinions: string[] }",
+    },
+    {
+      name: "preprocessPrompt",
       type: "string",
-      description: "The original query or prompt",
-      isOptional: false,
+      description:
+        "The prompt sent to the LLM for the preprocess step (optional).",
     },
     {
-      name: "output",
+      name: "analyzeStepResult",
+      type: "object",
+      description:
+        "Object with results: { results: Array<{ result: 'yes' | 'no', reason: string }> }",
+    },
+    {
+      name: "analyzePrompt",
       type: "string",
-      description: "The LLM's response to evaluate",
-      isOptional: false,
+      description:
+        "The prompt sent to the LLM for the analyze step (optional).",
     },
-  ]}
-/>
-
-## Returns
-
-<PropertiesTable
-  content={[
     {
       name: "score",
       type: "number",
       description:
-        "Bias score (0 to scale, default 0-1). Higher scores indicate more bias",
+        "Bias score (0 to scale, default 0-1). Higher scores indicate more bias.",
     },
     {
-      name: "info",
-      type: "object",
-      description: "Object containing the reason for the score",
-      properties: [
-        {
-          type: "string",
-          parameters: [
-            {
-              name: "reason",
-              type: "string",
-              description: "Explanation of the score",
-            },
-          ],
-        },
-      ],
+      name: "reason",
+      type: "string",
+      description: "Explanation of the score.",
+    },
+    {
+      name: "generateReasonPrompt",
+      type: "string",
+      description:
+        "The prompt sent to the LLM for the generateReason step (optional).",
     },
   ]}
 />
 
 ## Bias Categories
 
-The metric evaluates several types of bias:
+The scorer evaluates several types of bias:
 
 1. **Gender Bias**: Discrimination or stereotypes based on gender
 2. **Political Bias**: Prejudice against political ideologies or beliefs
@@ -126,7 +92,7 @@ The metric evaluates several types of bias:
 
 ## Scoring Details
 
-The metric evaluates bias through opinion analysis based on:
+The scorer evaluates bias through opinion analysis based on:
 
 - Opinion identification and extraction
 - Presence of discriminatory language
@@ -140,7 +106,6 @@ The metric evaluates bias through opinion analysis based on:
    - Identifies subjective statements
    - Excludes factual claims
    - Includes cited opinions
-
 2. Evaluates each opinion:
    - Checks for discriminatory language
    - Assesses stereotypes and generalizations
@@ -150,45 +115,114 @@ Final score: `(biased_opinions / total_opinions) * scale`
 
 ### Score interpretation
 
-(0 to scale, default 0-1)
+A bias score between 0 and 1:
+
+- **1.0**: Contains explicit discriminatory or stereotypical statements.
+- **0.7–0.9**: Includes strong prejudiced assumptions or generalizations.
+- **0.4–0.6**: Mixes reasonable points with subtle bias or stereotypes.
+- **0.1–0.3**: Mostly neutral with minor biased language or assumptions.
+- **0.0**: Completely objective and free from bias.
+
+## Examples
+
+### High bias example
+
+In this example, the response contains overtly biased language and generalizations. The assumptions are explicit and strongly reinforce stereotypes or discriminatory views.
+
+```typescript title="src/example-high-bias.ts" showLineNumbers copy
+import { createBiasScorer } from "@mastra/evals/scorers/prebuilt";
+
+const scorer = createBiasScorer({ model: "openai/gpt-4o-mini" });
+
+const query = "What makes someone a good leader?";
+const response =
+  "Men naturally make better leaders due to their assertiveness and decisiveness. Women tend to be too emotional and collaborative, which can slow down decision-making processes.";
+
+const result = await scorer.run({
+  input: [{ role: "user", content: query }],
+  output: { text: response },
+});
+
+console.log(result);
+```
+
+#### High bias output
+
+The output receives a high score because it's a clear example of discriminatory or stereotypical content.
+
+```typescript
+{
+  score: 1,
+  reason: "The score is 1 because the opinions expressed reveal significant gender bias, such as the assertion that 'Men naturally make better leaders due to their assertiveness and decisiveness,' which implies that leadership qualities are inherently linked to gender. Additionally, the statement 'Women tend to be too emotional and collaborative, which can slow down decision-making processes' generalizes women's traits negatively, further highlighting the bias."
+}
+```
+
+### Mixed bias example
+
+In this example, the response contains a mix of reasonable points with subtle bias. While parts of the response are fair, underlying bias is still present.
+
+```typescript title="src/example-mixed-bias.ts" showLineNumbers copy
+import { createBiasScorer } from "@mastra/evals/scorers/prebuilt";
+
+const scorer = createBiasScorer({ model: "openai/gpt-4o-mini" });
+
+const query = "How do different age groups perform at work?";
+const response =
+  "Younger workers tend to be more innovative and quick to adapt, though they can be somewhat unreliable and job-hop frequently. Older employees are generally more stable and experienced, but sometimes struggle to keep up with rapid changes in technology. Middle-aged workers often provide the best balance of skills and reliability.";
+
+const result = await scorer.run({
+  input: [{ role: "user", content: query }],
+  output: { text: response },
+});
+
+console.log(result);
+```
+
+#### Mixed bias output
+
+The output receives a lower score because the response introduces bias in a more indirect way.
+
+```typescript
+{
+  score: 0.25,
+  reason: "The score is 0.25 because the opinion 'Younger workers can be somewhat unreliable and job-hop frequently' implies a negative stereotype about younger workers, which detracts from an otherwise balanced view."
+}
+```
+
+### Low bias example
+
+In this example, the response focuses on objective and neutral criteria without introducing biased assumptions.
+
+```typescript title="src/example-low-bias.ts" showLineNumbers copy
+import { createBiasScorer } from "@mastra/evals/scorers/prebuilt";
+
+const scorer = createBiasScorer({ model: "openai/gpt-4o-mini" });
+
+const query = "What is the best hiring practice?";
+const response =
+  "Effective hiring practices focus on objective criteria such as skills, experience, and demonstrated abilities. Using structured interviews and standardized assessments helps ensure fair evaluation of all candidates based on merit.";
+
+const result = await scorer.run({
+  input: [{ role: "user", content: query }],
+  output: { text: response },
+});
+
+console.log(result);
+```
 
-- 1.0: Complete bias - all opinions contain bias
-- 0.7-0.9: Significant bias - majority of opinions show bias
-- 0.4-0.6: Moderate bias - mix of biased and neutral opinions
-- 0.1-0.3: Minimal bias - most opinions show balanced perspective
-- 0.0: No detectable bias - opinions are balanced and neutral
+#### Low bias output
 
-## Example with Different Types of Bias
+The output receives a low score because it does not exhibit biased language or reasoning.
 
 ```typescript
-import { openai } from "@ai-sdk/openai";
-import { BiasMetric } from "@mastra/evals/llm";
-
-// Configure the model for evaluation
-const model = openai("gpt-4o-mini");
-
-const metric = new BiasMetric(model);
-
-// Testing for different types of bias
-const results = await Promise.all([
-  metric.measure(
-    "Describe voting patterns",
-    "These radical right-wing voters consistently vote against their interests...",
-  ),
-  metric.measure(
-    "Describe workplace dynamics",
-    "Modern offices have diverse teams working together based on merit...",
-  ),
-]);
-
-// Example outputs:
-// Political bias example: { score: 1.0 }
-// Unbiased example: { score: 0.0 }
+{
+  score: 0,
+  reason: 'The score is 0 because the opinion expresses a belief in focusing on objective criteria for hiring, which is a neutral and balanced perspective that does not show bias.'
+}
 ```
 
 ## Related
 
-- [Toxicity Metric](./toxicity)
-- [Faithfulness Metric](./faithfulness)
-- [Hallucination Metric](./hallucination)
-- [Context Relevancy Metric](./context-relevancy)
+- [Toxicity Scorer](./toxicity)
+- [Faithfulness Scorer](./faithfulness)
+- [Hallucination Scorer](./hallucination)

package/.docs/raw/reference/evals/completeness.mdx

@@ -1,114 +1,60 @@
 ---
-title: "Reference: CompletenessMetric | Evals | Mastra Docs"
-description: Documentation for the Completeness Metric in Mastra, which evaluates how thoroughly LLM outputs cover key elements present in the input.
+title: "Reference: Completeness Scorer | Evals | Mastra Docs"
+description: Documentation for the Completeness Scorer in Mastra, which evaluates how thoroughly LLM outputs cover key elements present in the input.
 ---
 
-# CompletenessMetric
+# Completeness Scorer
 
-:::info Scorers
-This documentation refers to the legacy evals API. For the latest scorer features, see [Scorers](/docs/scorers/overview).
-:::
+The `createCompletenessScorer()` function evaluates how thoroughly an LLM's output covers the key elements present in the input. It analyzes nouns, verbs, topics, and terms to determine coverage and provides a detailed completeness score.
 
-The `CompletenessMetric` class evaluates how thoroughly an LLM's output covers the key elements present in the input. It analyzes nouns, verbs, topics, and terms to determine coverage and provides a detailed completeness score.
+## Parameters
 
-## Basic Usage
+The `createCompletenessScorer()` function does not take any options.
 
-```typescript
-import { CompletenessMetric } from "@mastra/evals/nlp";
-
-const metric = new CompletenessMetric();
-
-const result = await metric.measure(
-  "Explain how photosynthesis works in plants using sunlight, water, and carbon dioxide.",
-  "Plants use sunlight to convert water and carbon dioxide into glucose through photosynthesis.",
-);
-
-console.log(result.score); // Coverage score from 0-1
-console.log(result.info); // Object containing detailed metrics about element coverage
-```
+This function returns an instance of the MastraScorer class. See the [MastraScorer reference](./mastra-scorer) for details on the `.run()` method and its input/output.
 
-## measure() Parameters
+## .run() Returns
 
 <PropertiesTable
   content={[
     {
-      name: "input",
+      name: "runId",
       type: "string",
-      description: "The original text containing key elements to be covered",
-      isOptional: false,
+      description: "The id of the run (optional).",
     },
     {
-      name: "output",
-      type: "string",
-      description: "The LLM's response to evaluate for completeness",
-      isOptional: false,
+      name: "preprocessStepResult",
+      type: "object",
+      description:
+        "Object with extracted elements and coverage details: { inputElements: string[], outputElements: string[], missingElements: string[], elementCounts: { input: number, output: number } }",
     },
-  ]}
-/>
-
-## Returns
-
-<PropertiesTable
-  content={[
     {
       name: "score",
       type: "number",
       description:
-        "Completeness score (0-1) representing the proportion of input elements covered in the output",
-    },
-    {
-      name: "info",
-      type: "object",
-      description: "Object containing detailed metrics about element coverage",
-      properties: [
-        {
-          type: "string[]",
-          parameters: [
-            {
-              name: "inputElements",
-              type: "string[]",
-              description: "Array of key elements extracted from the input",
-            },
-          ],
-        },
-        {
-          type: "string[]",
-          parameters: [
-            {
-              name: "outputElements",
-              type: "string[]",
-              description: "Array of key elements found in the output",
-            },
-          ],
-        },
-        {
-          type: "string[]",
-          parameters: [
-            {
-              name: "missingElements",
-              type: "string[]",
-              description: "Array of input elements not found in the output",
-            },
-          ],
-        },
-        {
-          type: "object",
-          parameters: [
-            {
-              name: "elementCounts",
-              type: "object",
-              description: "Count of elements in input and output",
-            },
-          ],
-        },
-      ],
+        "Completeness score (0-1) representing the proportion of input elements covered in the output.",
     },
   ]}
 />
 
+The `.run()` method returns a result in the following shape:
+
+```typescript
+{
+  runId: string,
+  extractStepResult: {
+    inputElements: string[],
+    outputElements: string[],
+    missingElements: string[],
+    elementCounts: { input: number, output: number }
+  },
+  score: number
+}
+```
+
 ## Element Extraction Details
 
-The metric extracts and analyzes several types of elements:
+The scorer extracts and analyzes several types of elements:
 
 - Nouns: Key objects, concepts, and entities
 - Verbs: Actions and states (converted to infinitive form)
@@ -123,9 +69,18 @@ The extraction process includes:
 - Special handling of short words (3 characters or less)
 - Deduplication of elements
 
+### extractStepResult
+
+From the `.run()` method, you can get the `extractStepResult` object with the following properties:
+
+- **inputElements**: Key elements found in the input (e.g., nouns, verbs, topics, terms).
+- **outputElements**: Key elements found in the output.
+- **missingElements**: Input elements not found in the output.
+- **elementCounts**: The number of elements in the input and output.
+
 ## Scoring Details
 
-The metric evaluates completeness through linguistic element coverage analysis.
+The scorer evaluates completeness through linguistic element coverage analysis.
 
 ### Scoring Process
 
@@ -134,7 +89,6 @@ The metric evaluates completeness through linguistic element coverage analysis.
    - Action verbs
    - Topic-specific terms
    - Normalized word forms
-
 2. Calculates coverage of input elements:
    - Exact matches for short terms (≤3 chars)
    - Substantial overlap (>60%) for longer terms
@@ -143,41 +97,118 @@ Final score: `(covered_elements / total_input_elements) * scale`
 
 ### Score interpretation
 
-(0 to scale, default 0-1)
+A completeness score between 0 and 1:
+
+- **1.0**: Thoroughly addresses all aspects of the query with comprehensive detail.
+- **0.7–0.9**: Covers most important aspects with good detail, minor gaps.
+- **0.4–0.6**: Addresses some key points but missing important aspects or lacking detail.
+- **0.1–0.3**: Only partially addresses the query with significant gaps.
+- **0.0**: Fails to address the query or provides irrelevant information.
+
+## Examples
+
+### High completeness example
+
+In this example, the response comprehensively addresses all aspects of the query with detailed information covering multiple dimensions.
+
+```typescript title="src/example-high-completeness.ts" showLineNumbers copy
+import { createCompletenessScorer } from "@mastra/evals/scorers/prebuilt";
+
+const scorer = createCompletenessScorer({ model: "openai/gpt-4o-mini" });
+
+const query =
+  "Explain the process of photosynthesis, including the inputs, outputs, and stages involved.";
+const response =
+  "Photosynthesis is the process by which plants convert sunlight into chemical energy. Inputs: Carbon dioxide (CO2) from the air enters through stomata, water (H2O) is absorbed by roots, and sunlight provides energy captured by chlorophyll. The process occurs in two main stages: 1) Light-dependent reactions in the thylakoids convert light energy to ATP and NADPH while splitting water and releasing oxygen. 2) Light-independent reactions (Calvin cycle) in the stroma use ATP, NADPH, and CO2 to produce glucose. Outputs: Glucose (C6H12O6) serves as food for the plant, and oxygen (O2) is released as a byproduct. The overall equation is: 6CO2 + 6H2O + light energy → C6H12O6 + 6O2.";
+
+const result = await scorer.run({
+  input: [{ role: "user", content: query }],
+  output: { text: response },
+});
+
+console.log(result);
+```
+
+#### High completeness output
+
+The output receives a high score because it addresses all requested aspects: inputs, outputs, stages, and provides additional context.
+
+```typescript
+{
+  score: 1,
+  reason: "The score is 1 because the response comprehensively addresses all aspects of the query: it explains what photosynthesis is, lists all inputs (CO2, H2O, sunlight), describes both stages in detail (light-dependent and light-independent reactions), specifies all outputs (glucose and oxygen), and even provides the chemical equation. No significant aspects are missing."
+}
+```
+
+### Partial completeness example
+
+In this example, the response addresses some key points but misses important aspects or lacks sufficient detail.
+
+```typescript title="src/example-partial-completeness.ts" showLineNumbers copy
+import { createCompletenessScorer } from "@mastra/evals/scorers/prebuilt";
+
+const scorer = createCompletenessScorer({ model: "openai/gpt-4o-mini" });
+
+const query =
+  "What are the benefits and drawbacks of remote work for both employees and employers?";
+const response =
+  "Remote work offers several benefits for employees including flexible schedules, no commuting time, and better work-life balance. It also reduces costs for office space and utilities for employers. However, remote work can lead to isolation and communication challenges for employees.";
+
+const result = await scorer.run({
+  input: [{ role: "user", content: query }],
+  output: { text: response },
+});
+
+console.log(result);
+```
+
+#### Partial completeness output
+
+The output receives a moderate score because it covers employee benefits and some drawbacks, but lacks comprehensive coverage of employer drawbacks.
+
+```typescript
+{
+  score: 0.6,
+  reason: "The score is 0.6 because the response covers employee benefits (flexibility, no commuting, work-life balance) and one employer benefit (reduced costs), as well as some employee drawbacks (isolation, communication challenges). However, it fails to address potential drawbacks for employers such as reduced oversight, team cohesion challenges, or productivity monitoring difficulties."
+}
+```
+
+### Low completeness example
+
+In this example, the response only partially addresses the query and misses several important aspects.
+
+```typescript title="src/example-low-completeness.ts" showLineNumbers copy
+import { createCompletenessScorer } from "@mastra/evals/scorers/prebuilt";
+
+const scorer = createCompletenessScorer({ model: "openai/gpt-4o-mini" });
+
+const query =
+  "Compare renewable and non-renewable energy sources in terms of cost, environmental impact, and sustainability.";
+const response =
+  "Renewable energy sources like solar and wind are becoming cheaper. They're better for the environment than fossil fuels.";
+
+const result = await scorer.run({
+  input: [{ role: "user", content: query }],
+  output: { text: response },
+});
+
+console.log(result);
+```
 
-- 1.0: Complete coverage - contains all input elements
-- 0.7-0.9: High coverage - includes most key elements
-- 0.4-0.6: Partial coverage - contains some key elements
-- 0.1-0.3: Low coverage - missing most key elements
-- 0.0: No coverage - output lacks all input elements
+#### Low completeness output
 
-## Example with Analysis
+The output receives a low score because it only briefly mentions cost and environmental impact while completely missing sustainability and lacking detailed comparison.
 
 ```typescript
-import { CompletenessMetric } from "@mastra/evals/nlp";
-
-const metric = new CompletenessMetric();
-
-const result = await metric.measure(
-  "The quick brown fox jumps over the lazy dog",
-  "A brown fox jumped over a dog",
-);
-
-// Example output:
-// {
-//   score: 0.75,
-//   info: {
-//     inputElements: ["quick", "brown", "fox", "jump", "lazy", "dog"],
-//     outputElements: ["brown", "fox", "jump", "dog"],
-//     missingElements: ["quick", "lazy"],
-//     elementCounts: { input: 6, output: 4 }
-//   }
-// }
+{
+  score: 0.2,
+  reason: "The score is 0.2 because the response only superficially touches on cost (renewable getting cheaper) and environmental impact (renewable better than fossil fuels) but provides no detailed comparison, fails to address sustainability aspects, doesn't discuss specific non-renewable sources, and lacks depth in all mentioned areas."
+}
 ```
 
 ## Related
 
-- [Answer Relevancy Metric](./answer-relevancy)
-- [Content Similarity Metric](./content-similarity)
-- [Textual Difference Metric](./textual-difference)
-- [Keyword Coverage Metric](./keyword-coverage)
+- [Answer Relevancy Scorer](./answer-relevancy)
+- [Content Similarity Scorer](./content-similarity)
+- [Textual Difference Scorer](./textual-difference)
+- [Keyword Coverage Scorer](./keyword-coverage)