@mastra/mcp-docs-server 0.0.0-error-handler-fix-20251020202607 → 0.0.0-execa-dynamic-import-20260304221256

package/.docs/raw/reference/evals/answer-relevancy.mdx

@@ -1,190 +0,0 @@
----
-title: "Reference: Answer Relevancy | Metrics | Evals | Mastra Docs"
-description: Documentation for the Answer Relevancy Metric in Mastra, which evaluates how well LLM outputs address the input query.
----
-
-import { ScorerCallout } from '@/components/scorer-callout'
-
-# AnswerRelevancyMetric
-
-<ScorerCallout />
-
-The `AnswerRelevancyMetric` class evaluates how well an LLM's output answers or addresses the input query. It uses a judge-based system to determine relevancy and provides detailed scoring and reasoning.
-
-## Basic Usage
-
-```typescript
-import { openai } from "@ai-sdk/openai";
-import { AnswerRelevancyMetric } from "@mastra/evals/llm";
-
-// Configure the model for evaluation
-const model = openai("gpt-4o-mini");
-
-const metric = new AnswerRelevancyMetric(model, {
-  uncertaintyWeight: 0.3,
-  scale: 1,
-});
-
-const result = await metric.measure(
-  "What is the capital of France?",
-  "Paris is the capital of France.",
-);
-
-console.log(result.score); // Score from 0-1
-console.log(result.info.reason); // Explanation of the score
-```
-
-## Constructor Parameters
-
-<PropertiesTable
-  content={[
-    {
-      name: "model",
-      type: "LanguageModel",
-      description: "Configuration for the model used to evaluate relevancy",
-      isOptional: false,
-    },
-    {
-      name: "options",
-      type: "AnswerRelevancyMetricOptions",
-      description: "Configuration options for the metric",
-      isOptional: true,
-      defaultValue: "{ uncertaintyWeight: 0.3, scale: 1 }",
-    },
-  ]}
-/>
-
-### AnswerRelevancyMetricOptions
-
-<PropertiesTable
-  content={[
-    {
-      name: "uncertaintyWeight",
-      type: "number",
-      description: "Weight given to 'unsure' verdicts in scoring (0-1)",
-      isOptional: true,
-      defaultValue: "0.3",
-    },
-    {
-      name: "scale",
-      type: "number",
-      description: "Maximum score value",
-      isOptional: true,
-      defaultValue: "1",
-    },
-  ]}
-/>
-
-## measure() Parameters
-
-<PropertiesTable
-  content={[
-    {
-      name: "input",
-      type: "string",
-      description: "The original query or prompt",
-      isOptional: false,
-    },
-    {
-      name: "output",
-      type: "string",
-      description: "The LLM's response to evaluate",
-      isOptional: false,
-    },
-  ]}
-/>
-
-## Returns
-
-<PropertiesTable
-  content={[
-    {
-      name: "score",
-      type: "number",
-      description: "Relevancy score (0 to scale, default 0-1)",
-    },
-    {
-      name: "info",
-      type: "object",
-      description: "Object containing the reason for the score",
-      properties: [
-        {
-          type: "string",
-          parameters: [
-            {
-              name: "reason",
-              type: "string",
-              description: "Explanation of the score",
-            },
-          ],
-        },
-      ],
-    },
-  ]}
-/>
-
-## Scoring Details
-
-The metric evaluates relevancy through query-answer alignment, considering completeness, accuracy, and detail level.
-
-### Scoring Process
-
-1. Statement Analysis:
-
-   - Breaks output into meaningful statements while preserving context
-   - Evaluates each statement against query requirements
-
-2. Evaluates relevance of each statement:
-   - "yes": Full weight for direct matches
-   - "unsure": Partial weight (default: 0.3) for approximate matches
-   - "no": Zero weight for irrelevant content
-
-Final score: `((direct + uncertainty * partial) / total_statements) * scale`
-
-### Score interpretation
-
-(0 to scale, default 0-1)
-
-- 1.0: Perfect relevance - complete and accurate
-- 0.7-0.9: High relevance - minor gaps or imprecisions
-- 0.4-0.6: Moderate relevance - significant gaps
-- 0.1-0.3: Low relevance - major issues
-- 0.0: No relevance - incorrect or off-topic
-
-## Example with Custom Configuration
-
-```typescript
-import { openai } from "@ai-sdk/openai";
-import { AnswerRelevancyMetric } from "@mastra/evals/llm";
-
-// Configure the model for evaluation
-const model = openai("gpt-4o-mini");
-
-const metric = new AnswerRelevancyMetric(model, {
-  uncertaintyWeight: 0.5, // Higher weight for uncertain verdicts
-  scale: 5, // Use 0-5 scale instead of 0-1
-});
-
-const result = await metric.measure(
-  "What are the benefits of exercise?",
-  "Regular exercise improves cardiovascular health, builds strength, and boosts mental wellbeing.",
-);
-
-// Example output:
-// {
-//   score: 4.5,
-//   info: {
-//     reason: "The score is 4.5 out of 5 because the response directly addresses the query
-//           with specific, accurate benefits of exercise. It covers multiple aspects
-//           (cardiovascular, muscular, and mental health) in a clear and concise manner.
-//           The answer is highly relevant and provides appropriate detail without
-//           including unnecessary information."
-//   }
-// }
-```
-
-## Related
-
-- [Prompt Alignment Metric](./prompt-alignment)
-- [Context Precision Metric](./context-precision)
-- [Faithfulness Metric](./faithfulness)

package/.docs/raw/reference/evals/bias.mdx

@@ -1,195 +0,0 @@
----
-title: "Reference: Bias | Output Metrics | Evals | Mastra Docs"
-description: Documentation for the Bias Metric in Mastra, which evaluates LLM outputs for various forms of bias, including gender, political, racial/ethnic, or geographical bias.
----
-
-import { ScorerCallout } from '@/components/scorer-callout'
-
-# BiasMetric
-
-<ScorerCallout />
-
-The `BiasMetric` class evaluates whether an LLM's output contains various forms of bias, including gender, political, racial/ethnic, or geographical bias. This metric is particularly useful for detecting unintended biases that may emerge after fine-tuning models or applying optimizations.
-
-## Basic Usage
-
-```typescript
-import { openai } from "@ai-sdk/openai";
-import { BiasMetric } from "@mastra/evals/llm";
-
-// Configure the model for evaluation
-const model = openai("gpt-4o-mini");
-
-const metric = new BiasMetric(model, {
-  scale: 1,
-});
-
-const result = await metric.measure(
-  "Describe leadership qualities in the workplace",
-  "Strong leaders are typically men who can make tough decisions...",
-);
-
-console.log(result.score); // Score from 0-1, higher means more biased
-```
-
-## Constructor Parameters
-
-<PropertiesTable
-  content={[
-    {
-      name: "model",
-      type: "LanguageModel",
-      description: "Configuration for the model used to evaluate bias",
-      isOptional: false,
-    },
-    {
-      name: "options",
-      type: "BiasMetricOptions",
-      description: "Configuration options for the metric",
-      isOptional: true,
-      defaultValue: "{ scale: 1 }",
-    },
-  ]}
-/>
-
-### BiasMetricOptions
-
-<PropertiesTable
-  content={[
-    {
-      name: "scale",
-      type: "number",
-      description: "Maximum score value",
-      isOptional: true,
-      defaultValue: "1",
-    },
-  ]}
-/>
-
-## measure() Parameters
-
-<PropertiesTable
-  content={[
-    {
-      name: "input",
-      type: "string",
-      description: "The original query or prompt",
-      isOptional: false,
-    },
-    {
-      name: "output",
-      type: "string",
-      description: "The LLM's response to evaluate",
-      isOptional: false,
-    },
-  ]}
-/>
-
-## Returns
-
-<PropertiesTable
-  content={[
-    {
-      name: "score",
-      type: "number",
-      description:
-        "Bias score (0 to scale, default 0-1). Higher scores indicate more bias",
-    },
-    {
-      name: "info",
-      type: "object",
-      description: "Object containing the reason for the score",
-      properties: [
-        {
-          type: "string",
-          parameters: [
-            {
-              name: "reason",
-              type: "string",
-              description: "Explanation of the score",
-            },
-          ],
-        },
-      ],
-    },
-  ]}
-/>
-
-## Bias Categories
-
-The metric evaluates several types of bias:
-
-1. **Gender Bias**: Discrimination or stereotypes based on gender
-2. **Political Bias**: Prejudice against political ideologies or beliefs
-3. **Racial/Ethnic Bias**: Discrimination based on race, ethnicity, or national origin
-4. **Geographical Bias**: Prejudice based on location or regional stereotypes
-
-## Scoring Details
-
-The metric evaluates bias through opinion analysis based on:
-
-- Opinion identification and extraction
-- Presence of discriminatory language
-- Use of stereotypes or generalizations
-- Balance in perspective presentation
-- Loaded or prejudicial terminology
-
-### Scoring Process
-
-1. Extracts opinions from text:
-
-   - Identifies subjective statements
-   - Excludes factual claims
-   - Includes cited opinions
-
-2. Evaluates each opinion:
-   - Checks for discriminatory language
-   - Assesses stereotypes and generalizations
-   - Analyzes perspective balance
-
-Final score: `(biased_opinions / total_opinions) * scale`
-
-### Score interpretation
-
-(0 to scale, default 0-1)
-
-- 1.0: Complete bias - all opinions contain bias
-- 0.7-0.9: Significant bias - majority of opinions show bias
-- 0.4-0.6: Moderate bias - mix of biased and neutral opinions
-- 0.1-0.3: Minimal bias - most opinions show balanced perspective
-- 0.0: No detectable bias - opinions are balanced and neutral
-
-## Example with Different Types of Bias
-
-```typescript
-import { openai } from "@ai-sdk/openai";
-import { BiasMetric } from "@mastra/evals/llm";
-
-// Configure the model for evaluation
-const model = openai("gpt-4o-mini");
-
-const metric = new BiasMetric(model);
-
-// Testing for different types of bias
-const results = await Promise.all([
-  metric.measure(
-    "Describe voting patterns",
-    "These radical right-wing voters consistently vote against their interests...",
-  ),
-  metric.measure(
-    "Describe workplace dynamics",
-    "Modern offices have diverse teams working together based on merit...",
-  ),
-]);
-
-// Example outputs:
-// Political bias example: { score: 1.0 }
-// Unbiased example: { score: 0.0 }
-```
-
-## Related
-
-- [Toxicity Metric](./toxicity)
-- [Faithfulness Metric](./faithfulness)
-- [Hallucination Metric](./hallucination)
-- [Context Relevancy Metric](./context-relevancy)

package/.docs/raw/reference/evals/completeness.mdx

@@ -1,184 +0,0 @@
----
-title: "Reference: Completeness | Metrics | Evals | Mastra Docs"
-description: Documentation for the Completeness Metric in Mastra, which evaluates how thoroughly LLM outputs cover key elements present in the input.
----
-
-import { ScorerCallout } from '@/components/scorer-callout'
-
-# CompletenessMetric
-
-<ScorerCallout />
-
-The `CompletenessMetric` class evaluates how thoroughly an LLM's output covers the key elements present in the input. It analyzes nouns, verbs, topics, and terms to determine coverage and provides a detailed completeness score.
-
-## Basic Usage
-
-```typescript
-import { CompletenessMetric } from "@mastra/evals/nlp";
-
-const metric = new CompletenessMetric();
-
-const result = await metric.measure(
-  "Explain how photosynthesis works in plants using sunlight, water, and carbon dioxide.",
-  "Plants use sunlight to convert water and carbon dioxide into glucose through photosynthesis.",
-);
-
-console.log(result.score); // Coverage score from 0-1
-console.log(result.info); // Object containing detailed metrics about element coverage
-```
-
-## measure() Parameters
-
-<PropertiesTable
-  content={[
-    {
-      name: "input",
-      type: "string",
-      description: "The original text containing key elements to be covered",
-      isOptional: false,
-    },
-    {
-      name: "output",
-      type: "string",
-      description: "The LLM's response to evaluate for completeness",
-      isOptional: false,
-    },
-  ]}
-/>
-
-## Returns
-
-<PropertiesTable
-  content={[
-    {
-      name: "score",
-      type: "number",
-      description:
-        "Completeness score (0-1) representing the proportion of input elements covered in the output",
-    },
-    {
-      name: "info",
-      type: "object",
-      description: "Object containing detailed metrics about element coverage",
-      properties: [
-        {
-          type: "string[]",
-          parameters: [
-            {
-              name: "inputElements",
-              type: "string[]",
-              description: "Array of key elements extracted from the input",
-            },
-          ],
-        },
-        {
-          type: "string[]",
-          parameters: [
-            {
-              name: "outputElements",
-              type: "string[]",
-              description: "Array of key elements found in the output",
-            },
-          ],
-        },
-        {
-          type: "string[]",
-          parameters: [
-            {
-              name: "missingElements",
-              type: "string[]",
-              description: "Array of input elements not found in the output",
-            },
-          ],
-        },
-        {
-          type: "object",
-          parameters: [
-            {
-              name: "elementCounts",
-              type: "object",
-              description: "Count of elements in input and output",
-            },
-          ],
-        },
-      ],
-    },
-  ]}
-/>
-
-## Element Extraction Details
-
-The metric extracts and analyzes several types of elements:
-
-- Nouns: Key objects, concepts, and entities
-- Verbs: Actions and states (converted to infinitive form)
-- Topics: Main subjects and themes
-- Terms: Individual significant words
-
-The extraction process includes:
-
-- Normalization of text (removing diacritics, converting to lowercase)
-- Splitting camelCase words
-- Handling of word boundaries
-- Special handling of short words (3 characters or less)
-- Deduplication of elements
-
-## Scoring Details
-
-The metric evaluates completeness through linguistic element coverage analysis.
-
-### Scoring Process
-
-1. Extracts key elements:
-
-   - Nouns and named entities
-   - Action verbs
-   - Topic-specific terms
-   - Normalized word forms
-
-2. Calculates coverage of input elements:
-   - Exact matches for short terms (≤3 chars)
-   - Substantial overlap (>60%) for longer terms
-
-Final score: `(covered_elements / total_input_elements) * scale`
-
-### Score interpretation
-
-(0 to scale, default 0-1)
-
-- 1.0: Complete coverage - contains all input elements
-- 0.7-0.9: High coverage - includes most key elements
-- 0.4-0.6: Partial coverage - contains some key elements
-- 0.1-0.3: Low coverage - missing most key elements
-- 0.0: No coverage - output lacks all input elements
-
-## Example with Analysis
-
-```typescript
-import { CompletenessMetric } from "@mastra/evals/nlp";
-
-const metric = new CompletenessMetric();
-
-const result = await metric.measure(
-  "The quick brown fox jumps over the lazy dog",
-  "A brown fox jumped over a dog",
-);
-
-// Example output:
-// {
-//   score: 0.75,
-//   info: {
-//     inputElements: ["quick", "brown", "fox", "jump", "lazy", "dog"],
-//     outputElements: ["brown", "fox", "jump", "dog"],
-//     missingElements: ["quick", "lazy"],
-//     elementCounts: { input: 6, output: 4 }
-//   }
-// }
-```
-
-## Related
-
-- [Answer Relevancy Metric](./answer-relevancy)
-- [Content Similarity Metric](./content-similarity)
-- [Textual Difference Metric](./textual-difference)
-- [Keyword Coverage Metric](./keyword-coverage)