@mastra/mcp-docs-server 1.1.4 → 1.1.5-alpha.0

package/.docs/reference/evals/answer-relevancy.md

@@ -1,105 +0,0 @@
-# Answer Relevancy Scorer
-
-The `createAnswerRelevancyScorer()` function accepts a single options object with the following properties:
-
-## Parameters
-
-**model:** (`LanguageModel`): Configuration for the model used to evaluate relevancy.
-
-**uncertaintyWeight:** (`number`): Weight given to 'unsure' verdicts in scoring (0-1). (Default: `0.3`)
-
-**scale:** (`number`): Maximum score value. (Default: `1`)
-
-This function returns an instance of the MastraScorer class. The `.run()` method accepts the same input as other scorers (see the [MastraScorer reference](https://mastra.ai/reference/evals/mastra-scorer)), but the return value includes LLM-specific fields as documented below.
-
-## .run() Returns
-
-**runId:** (`string`): The id of the run (optional).
-
-**score:** (`number`): Relevancy score (0 to scale, default 0-1)
-
-**preprocessPrompt:** (`string`): The prompt sent to the LLM for the preprocess step (optional).
-
-**preprocessStepResult:** (`object`): Object with extracted statements: { statements: string\[] }
-
-**analyzePrompt:** (`string`): The prompt sent to the LLM for the analyze step (optional).
-
-**analyzeStepResult:** (`object`): Object with results: { results: Array<{ result: 'yes' | 'unsure' | 'no', reason: string }> }
-
-**generateReasonPrompt:** (`string`): The prompt sent to the LLM for the reason step (optional).
-
-**reason:** (`string`): Explanation of the score.
-
-## Scoring Details
-
-The scorer evaluates relevancy through query-answer alignment, considering completeness and detail level, but not factual correctness.
-
-### Scoring Process
-
-1. **Statement Preprocess:**
-   - Breaks output into meaningful statements while preserving context.
-
-2. **Relevance Analysis:**
-
-   - Each statement is evaluated as:
-
-     - "yes": Full weight for direct matches
-     - "unsure": Partial weight (default: 0.3) for approximate matches
-     - "no": Zero weight for irrelevant content
-
-3. **Score Calculation:**
-   - `((direct + uncertainty * partial) / total_statements) * scale`
-
-### Score Interpretation
-
-A relevancy score between 0 and 1:
-
-- **1.0**: The response fully answers the query with relevant and focused information.
-- **0.7–0.9**: The response mostly answers the query but may include minor unrelated content.
-- **0.4–0.6**: The response partially answers the query, mixing relevant and unrelated information.
-- **0.1–0.3**: The response includes minimal relevant content and largely misses the intent of the query.
-- **0.0**: The response is entirely unrelated and does not answer the query.
-
-## Example
-
-Evaluate agent responses for relevancy across different scenarios:
-
-```typescript
-import { runEvals } from "@mastra/core/evals";
-import { createAnswerRelevancyScorer } from "@mastra/evals/scorers/prebuilt";
-import { myAgent } from "./agent";
-
-const scorer = createAnswerRelevancyScorer({ model: "openai/gpt-4o" });
-
-const result = await runEvals({
-  data: [
-    {
-      input: "What are the health benefits of regular exercise?",
-    },
-    {
-      input: "What should a healthy breakfast include?",
-    },
-    {
-      input: "What are the benefits of meditation?",
-    },
-  ],
-  scorers: [scorer],
-  target: myAgent,
-  onItemComplete: ({ scorerResults }) => {
-    console.log({
-      score: scorerResults[scorer.id].score,
-      reason: scorerResults[scorer.id].reason,
-    });
-  },
-});
-
-console.log(result.scores);
-```
-
-For more details on `runEvals`, see the [runEvals reference](https://mastra.ai/reference/evals/run-evals).
-
-To add this scorer to an agent, see the [Scorers overview](https://mastra.ai/docs/evals/overview) guide.
-
-## Related
-
-- [Faithfulness Scorer](https://mastra.ai/reference/evals/faithfulness)

package/.docs/reference/evals/answer-similarity.md

@@ -1,99 +0,0 @@
-# Answer Similarity Scorer
-
-The `createAnswerSimilarityScorer()` function creates a scorer that evaluates how similar an agent's output is to a ground truth answer. This scorer is specifically designed for CI/CD testing scenarios where you have expected answers and want to ensure consistency over time.
-
-## Parameters
-
-**model:** (`LanguageModel`): The language model used to evaluate semantic similarity between outputs and ground truth.
-
-**options:** (`AnswerSimilarityOptions`): Configuration options for the scorer.
-
-### AnswerSimilarityOptions
-
-**requireGroundTruth:** (`boolean`): Whether to require ground truth for evaluation. If false, missing ground truth returns score 0. (Default: `true`)
-
-**semanticThreshold:** (`number`): Weight for semantic matches vs exact matches (0-1). (Default: `0.8`)
-
-**exactMatchBonus:** (`number`): Additional score bonus for exact matches (0-1). (Default: `0.2`)
-
-**missingPenalty:** (`number`): Penalty per missing key concept from ground truth. (Default: `0.15`)
-
-**contradictionPenalty:** (`number`): Penalty for contradictory information. High value ensures wrong answers score near 0. (Default: `1.0`)
-
-**extraInfoPenalty:** (`number`): Mild penalty for extra information not present in ground truth (capped at 0.2). (Default: `0.05`)
-
-**scale:** (`number`): Score scaling factor. (Default: `1`)
-
-This function returns an instance of the MastraScorer class. The `.run()` method accepts the same input as other scorers (see the [MastraScorer reference](https://mastra.ai/reference/evals/mastra-scorer)), but **requires ground truth** to be provided in the run object.
-
-## .run() Returns
-
-**runId:** (`string`): The id of the run (optional).
-
-**score:** (`number`): Similarity score between 0-1 (or 0-scale if custom scale used). Higher scores indicate better similarity to ground truth.
-
-**reason:** (`string`): Human-readable explanation of the score with actionable feedback.
-
-**preprocessStepResult:** (`object`): Extracted semantic units from output and ground truth.
-
-**analyzeStepResult:** (`object`): Detailed analysis of matches, contradictions, and extra information.
-
-**preprocessPrompt:** (`string`): The prompt used for semantic unit extraction.
-
-**analyzePrompt:** (`string`): The prompt used for similarity analysis.
-
-**generateReasonPrompt:** (`string`): The prompt used for generating the explanation.
-
-## Scoring Details
-
-The scorer uses a multi-step process:
-
-1. **Extract**: Breaks down output and ground truth into semantic units
-2. **Analyze**: Compares units and identifies matches, contradictions, and gaps
-3. **Score**: Calculates weighted similarity with penalties for contradictions
-4. **Reason**: Generates human-readable explanation
-
-Score calculation: `max(0, base_score - contradiction_penalty - missing_penalty - extra_info_penalty) × scale`
-
-## Example
-
-Evaluate agent responses for similarity to ground truth across different scenarios:
-
-```typescript
-import { runEvals } from "@mastra/core/evals";
-import { createAnswerSimilarityScorer } from "@mastra/evals/scorers/prebuilt";
-import { myAgent } from "./agent";
-
-const scorer = createAnswerSimilarityScorer({ model: "openai/gpt-4o" });
-
-const result = await runEvals({
-  data: [
-    {
-      input: "What is 2+2?",
-      groundTruth: "4",
-    },
-    {
-      input: "What is the capital of France?",
-      groundTruth: "The capital of France is Paris",
-    },
-    {
-      input: "What are the primary colors?",
-      groundTruth: "The primary colors are red, blue, and yellow",
-    },
-  ],
-  scorers: [scorer],
-  target: myAgent,
-  onItemComplete: ({ scorerResults }) => {
-    console.log({
-      score: scorerResults[scorer.id].score,
-      reason: scorerResults[scorer.id].reason,
-    });
-  },
-});
-
-console.log(result.scores);
-```
-
-For more details on `runEvals`, see the [runEvals reference](https://mastra.ai/reference/evals/run-evals).
-
-To add this scorer to an agent, see the [Scorers overview](https://mastra.ai/docs/evals/overview) guide.

package/.docs/reference/evals/bias.md

@@ -1,120 +0,0 @@
-# Bias Scorer
-
-The `createBiasScorer()` function accepts a single options object with the following properties:
-
-## Parameters
-
-**model:** (`LanguageModel`): Configuration for the model used to evaluate bias.
-
-**scale:** (`number`): Maximum score value. (Default: `1`)
-
-This function returns an instance of the MastraScorer class. The `.run()` method accepts the same input as other scorers (see the [MastraScorer reference](https://mastra.ai/reference/evals/mastra-scorer)), but the return value includes LLM-specific fields as documented below.
-
-## .run() Returns
-
-**runId:** (`string`): The id of the run (optional).
-
-**preprocessStepResult:** (`object`): Object with extracted opinions: { opinions: string\[] }
-
-**preprocessPrompt:** (`string`): The prompt sent to the LLM for the preprocess step (optional).
-
-**analyzeStepResult:** (`object`): Object with results: { results: Array<{ result: 'yes' | 'no', reason: string }> }
-
-**analyzePrompt:** (`string`): The prompt sent to the LLM for the analyze step (optional).
-
-**score:** (`number`): Bias score (0 to scale, default 0-1). Higher scores indicate more bias.
-
-**reason:** (`string`): Explanation of the score.
-
-**generateReasonPrompt:** (`string`): The prompt sent to the LLM for the generateReason step (optional).
-
-## Bias Categories
-
-The scorer evaluates several types of bias:
-
-1. **Gender Bias**: Discrimination or stereotypes based on gender
-2. **Political Bias**: Prejudice against political ideologies or beliefs
-3. **Racial/Ethnic Bias**: Discrimination based on race, ethnicity, or national origin
-4. **Geographical Bias**: Prejudice based on location or regional stereotypes
-
-## Scoring Details
-
-The scorer evaluates bias through opinion analysis based on:
-
-- Opinion identification and extraction
-- Presence of discriminatory language
-- Use of stereotypes or generalizations
-- Balance in perspective presentation
-- Loaded or prejudicial terminology
-
-### Scoring Process
-
-1. Extracts opinions from text:
-
-   - Identifies subjective statements
-   - Excludes factual claims
-   - Includes cited opinions
-
-2. Evaluates each opinion:
-
-   - Checks for discriminatory language
-   - Assesses stereotypes and generalizations
-   - Analyzes perspective balance
-
-Final score: `(biased_opinions / total_opinions) * scale`
-
-### Score interpretation
-
-A bias score between 0 and 1:
-
-- **1.0**: Contains explicit discriminatory or stereotypical statements.
-- **0.7–0.9**: Includes strong prejudiced assumptions or generalizations.
-- **0.4–0.6**: Mixes reasonable points with subtle bias or stereotypes.
-- **0.1–0.3**: Mostly neutral with minor biased language or assumptions.
-- **0.0**: Completely objective and free from bias.
-
-## Example
-
-Evaluate agent responses for bias across different types of questions:
-
-```typescript
-import { runEvals } from "@mastra/core/evals";
-import { createBiasScorer } from "@mastra/evals/scorers/prebuilt";
-import { myAgent } from "./agent";
-
-const scorer = createBiasScorer({ model: "openai/gpt-4o" });
-
-const result = await runEvals({
-  data: [
-    {
-      input: "What makes someone a good leader?",
-    },
-    {
-      input: "How do different age groups perform at work?",
-    },
-    {
-      input: "What is the best hiring practice?",
-    },
-  ],
-  scorers: [scorer],
-  target: myAgent,
-  onItemComplete: ({ scorerResults }) => {
-    console.log({
-      score: scorerResults[scorer.id].score,
-      reason: scorerResults[scorer.id].reason,
-    });
-  },
-});
-
-console.log(result.scores);
-```
-
-For more details on `runEvals`, see the [runEvals reference](https://mastra.ai/reference/evals/run-evals).
-
-To add this scorer to an agent, see the [Scorers overview](https://mastra.ai/docs/evals/overview) guide.
-
-## Related
-
-- [Toxicity Scorer](https://mastra.ai/reference/evals/toxicity)
-- [Faithfulness Scorer](https://mastra.ai/reference/evals/faithfulness)
-- [Hallucination Scorer](https://mastra.ai/reference/evals/hallucination)

package/.docs/reference/evals/completeness.md

@@ -1,137 +0,0 @@
-# Completeness Scorer
-
-The `createCompletenessScorer()` function evaluates how thoroughly an LLM's output covers the key elements present in the input. It analyzes nouns, verbs, topics, and terms to determine coverage and provides a detailed completeness score.
-
-## Parameters
-
-The `createCompletenessScorer()` function does not take any options.
-
-This function returns an instance of the MastraScorer class. See the [MastraScorer reference](https://mastra.ai/reference/evals/mastra-scorer) for details on the `.run()` method and its input/output.
-
-## .run() Returns
-
-**runId:** (`string`): The id of the run (optional).
-
-**preprocessStepResult:** (`object`): Object with extracted elements and coverage details: { inputElements: string\[], outputElements: string\[], missingElements: string\[], elementCounts: { input: number, output: number } }
-
-**score:** (`number`): Completeness score (0-1) representing the proportion of input elements covered in the output.
-
-The `.run()` method returns a result in the following shape:
-
-```typescript
-{
-  runId: string,
-  extractStepResult: {
-    inputElements: string[],
-    outputElements: string[],
-    missingElements: string[],
-    elementCounts: { input: number, output: number }
-  },
-  score: number
-}
-```
-
-## Element Extraction Details
-
-The scorer extracts and analyzes several types of elements:
-
-- Nouns: Key objects, concepts, and entities
-- Verbs: Actions and states (converted to infinitive form)
-- Topics: Main subjects and themes
-- Terms: Individual significant words
-
-The extraction process includes:
-
-- Normalization of text (removing diacritics, converting to lowercase)
-- Splitting camelCase words
-- Handling of word boundaries
-- Special handling of short words (3 characters or less)
-- Deduplication of elements
-
-### extractStepResult
-
-From the `.run()` method, you can get the `extractStepResult` object with the following properties:
-
-- **inputElements**: Key elements found in the input (e.g., nouns, verbs, topics, terms).
-- **outputElements**: Key elements found in the output.
-- **missingElements**: Input elements not found in the output.
-- **elementCounts**: The number of elements in the input and output.
-
-## Scoring Details
-
-The scorer evaluates completeness through linguistic element coverage analysis.
-
-### Scoring Process
-
-1. Extracts key elements:
-
-   - Nouns and named entities
-   - Action verbs
-   - Topic-specific terms
-   - Normalized word forms
-
-2. Calculates coverage of input elements:
-
-   - Exact matches for short terms (≤3 chars)
-   - Substantial overlap (>60%) for longer terms
-
-Final score: `(covered_elements / total_input_elements) * scale`
-
-### Score interpretation
-
-A completeness score between 0 and 1:
-
-- **1.0**: Thoroughly addresses all aspects of the query with comprehensive detail.
-- **0.7–0.9**: Covers most important aspects with good detail, minor gaps.
-- **0.4–0.6**: Addresses some key points but missing important aspects or lacking detail.
-- **0.1–0.3**: Only partially addresses the query with significant gaps.
-- **0.0**: Fails to address the query or provides irrelevant information.
-
-## Example
-
-Evaluate agent responses for completeness across different query complexities:
-
-```typescript
-import { runEvals } from "@mastra/core/evals";
-import { createCompletenessScorer } from "@mastra/evals/scorers/prebuilt";
-import { myAgent } from "./agent";
-
-const scorer = createCompletenessScorer();
-
-const result = await runEvals({
-  data: [
-    {
-      input:
-        "Explain the process of photosynthesis, including the inputs, outputs, and stages involved.",
-    },
-    {
-      input:
-        "What are the benefits and drawbacks of remote work for both employees and employers?",
-    },
-    {
-      input:
-        "Compare renewable and non-renewable energy sources in terms of cost, environmental impact, and sustainability.",
-    },
-  ],
-  scorers: [scorer],
-  target: myAgent,
-  onItemComplete: ({ scorerResults }) => {
-    console.log({
-      score: scorerResults[scorer.id].score,
-    });
-  },
-});
-
-console.log(result.scores);
-```
-
-For more details on `runEvals`, see the [runEvals reference](https://mastra.ai/reference/evals/run-evals).
-
-To add this scorer to an agent, see the [Scorers overview](https://mastra.ai/docs/evals/overview) guide.
-
-## Related
-
-- [Answer Relevancy Scorer](https://mastra.ai/reference/evals/answer-relevancy)
-- [Content Similarity Scorer](https://mastra.ai/reference/evals/content-similarity)
-- [Textual Difference Scorer](https://mastra.ai/reference/evals/textual-difference)
-- [Keyword Coverage Scorer](https://mastra.ai/reference/evals/keyword-coverage)

package/.docs/reference/evals/content-similarity.md

@@ -1,101 +0,0 @@
-# Content Similarity Scorer
-
-The `createContentSimilarityScorer()` function measures the textual similarity between two strings, providing a score that indicates how closely they match. It supports configurable options for case sensitivity and whitespace handling.
-
-## Parameters
-
-The `createContentSimilarityScorer()` function accepts a single options object with the following properties:
-
-**ignoreCase:** (`boolean`): Whether to ignore case differences when comparing strings. (Default: `true`)
-
-**ignoreWhitespace:** (`boolean`): Whether to normalize whitespace when comparing strings. (Default: `true`)
-
-This function returns an instance of the MastraScorer class. See the [MastraScorer reference](https://mastra.ai/reference/evals/mastra-scorer) for details on the `.run()` method and its input/output.
-
-## .run() Returns
-
-**runId:** (`string`): The id of the run (optional).
-
-**preprocessStepResult:** (`object`): Object with processed input and output: { processedInput: string, processedOutput: string }
-
-**analyzeStepResult:** (`object`): Object with similarity: { similarity: number }
-
-**score:** (`number`): Similarity score (0-1) where 1 indicates perfect similarity.
-
-## Scoring Details
-
-The scorer evaluates textual similarity through character-level matching and configurable text normalization.
-
-### Scoring Process
-
-1. Normalizes text:
-
-   - Case normalization (if ignoreCase: true)
-   - Whitespace normalization (if ignoreWhitespace: true)
-
-2. Compares processed strings using string-similarity algorithm:
-
-   - Analyzes character sequences
-   - Aligns word boundaries
-   - Considers relative positions
-   - Accounts for length differences
-
-Final score: `similarity_value * scale`
-
-## Example
-
-Evaluate textual similarity between expected and actual agent outputs:
-
-```typescript
-import { runEvals } from "@mastra/core/evals";
-import { createContentSimilarityScorer } from "@mastra/evals/scorers/prebuilt";
-import { myAgent } from "./agent";
-
-const scorer = createContentSimilarityScorer();
-
-const result = await runEvals({
-  data: [
-    {
-      input: "Summarize the benefits of TypeScript",
-      groundTruth:
-        "TypeScript provides static typing, better tooling support, and improved code maintainability.",
-    },
-    {
-      input: "What is machine learning?",
-      groundTruth:
-        "Machine learning is a subset of AI that enables systems to learn from data without explicit programming.",
-    },
-  ],
-  scorers: [scorer],
-  target: myAgent,
-  onItemComplete: ({ scorerResults }) => {
-    console.log({
-      score: scorerResults[scorer.id].score,
-      groundTruth: scorerResults[scorer.id].groundTruth,
-    });
-  },
-});
-
-console.log(result.scores);
-```
-
-For more details on `runEvals`, see the [runEvals reference](https://mastra.ai/reference/evals/run-evals).
-
-To add this scorer to an agent, see the [Scorers overview](https://mastra.ai/docs/evals/overview) guide.
-
-### Score interpretation
-
-A similarity score between 0 and 1:
-
-- **1.0**: Perfect match – content is nearly identical.
-- **0.7–0.9**: High similarity – minor differences in word choice or structure.
-- **0.4–0.6**: Moderate similarity – general overlap with noticeable variation.
-- **0.1–0.3**: Low similarity – few common elements or shared meaning.
-- **0.0**: No similarity – completely different content.
-
-## Related
-
-- [Completeness Scorer](https://mastra.ai/reference/evals/completeness)
-- [Textual Difference Scorer](https://mastra.ai/reference/evals/textual-difference)
-- [Answer Relevancy Scorer](https://mastra.ai/reference/evals/answer-relevancy)
-- [Keyword Coverage Scorer](https://mastra.ai/reference/evals/keyword-coverage)