@mastra/mcp-docs-server 0.13.7-alpha.0 → 0.13.7-alpha.2

package/.docs/raw/reference/scorers/bias.mdx

@@ -0,0 +1,127 @@
+---
+title: "Reference: Bias | Scorers | Mastra Docs"
+description: Documentation for the Bias Scorer in Mastra, which evaluates LLM outputs for various forms of bias, including gender, political, racial/ethnic, or geographical bias.
+---
+
+# Bias Scorer
+The `createBiasScorer()` function accepts a single options object with the following properties:
+
+For a usage example, see the [Bias Examples](/examples/scorers/bias).
+
+## Parameters
+
+
+<PropertiesTable
+  content={[
+    {
+      name: "model",
+      type: "LanguageModel",
+      required: true,
+      description: "Configuration for the model used to evaluate bias.",
+    },
+    {
+      name: "scale",
+      type: "number",
+      required: false,
+      defaultValue: "1",
+      description: "Maximum score value.",
+    },
+  ]}
+/>
+
+This function returns an instance of the MastraScorer class. The `.run()` method accepts the same input as other scorers (see the [MastraScorer reference](./mastra-scorer)), but the return value includes LLM-specific fields as documented below.
+
+## .run() Returns
+
+<PropertiesTable
+  content={[
+    {
+      name: "runId",
+      type: "string",
+      description: "The id of the run (optional).",
+    },
+    {
+      name: "extractStepResult",
+      type: "object",
+      description: "Object with extracted opinions: { opinions: string[] }",
+    },
+    {
+      name: "extractPrompt",
+      type: "string",
+      description: "The prompt sent to the LLM for the extract step (optional).",
+    },
+    {
+      name: "analyzeStepResult",
+      type: "object",
+      description: "Object with results: { results: Array<{ result: 'yes' | 'no', reason: string }> }",
+    },
+    {
+      name: "analyzePrompt",
+      type: "string",
+      description: "The prompt sent to the LLM for the analyze step (optional).",
+    },
+    {
+      name: "score",
+      type: "number",
+      description: "Bias score (0 to scale, default 0-1). Higher scores indicate more bias.",
+    },
+    {
+      name: "reason",
+      type: "string",
+      description: "Explanation of the score.",
+    },
+    {
+      name: "reasonPrompt",
+      type: "string",
+      description: "The prompt sent to the LLM for the reason step (optional).",
+    },
+  ]}
+/>
+
+## Bias Categories
+
+The scorer evaluates several types of bias:
+
+1. **Gender Bias**: Discrimination or stereotypes based on gender
+2. **Political Bias**: Prejudice against political ideologies or beliefs
+3. **Racial/Ethnic Bias**: Discrimination based on race, ethnicity, or national origin
+4. **Geographical Bias**: Prejudice based on location or regional stereotypes
+
+## Scoring Details
+
+The scorer evaluates bias through opinion analysis based on:
+
+- Opinion identification and extraction
+- Presence of discriminatory language
+- Use of stereotypes or generalizations
+- Balance in perspective presentation
+- Loaded or prejudicial terminology
+
+### Scoring Process
+
+1. Extracts opinions from text:
+   - Identifies subjective statements
+   - Excludes factual claims
+   - Includes cited opinions
+2. Evaluates each opinion:
+   - Checks for discriminatory language
+   - Assesses stereotypes and generalizations
+   - Analyzes perspective balance
+
+Final score: `(biased_opinions / total_opinions) * scale`
+
+### Score interpretation
+
+(0 to scale, default 0-1)
+
+- 1.0: Complete bias - all opinions contain bias
+- 0.7-0.9: Significant bias - majority of opinions show bias
+- 0.4-0.6: Moderate bias - mix of biased and neutral opinions
+- 0.1-0.3: Minimal bias - most opinions show balanced perspective
+- 0.0: No detectable bias - opinions are balanced and neutral
+
+## Related
+
+- [Toxicity Scorer](./toxicity)
+- [Faithfulness Scorer](./faithfulness)
+- [Hallucination Scorer](./hallucination)

package/.docs/raw/reference/scorers/completeness.mdx

@@ -0,0 +1,89 @@
+---
+title: "Reference: Completeness | Scorers | Mastra Docs"
+description: Documentation for the Completeness Scorer in Mastra, which evaluates how thoroughly LLM outputs cover key elements present in the input.
+---
+
+# Completeness Scorer
+
+The `createCompletenessScorer()` function evaluates how thoroughly an LLM's output covers the key elements present in the input. It analyzes nouns, verbs, topics, and terms to determine coverage and provides a detailed completeness score.
+
+For a usage example, see the [Completeness Examples](/examples/scorers/completeness).
+
+## Parameters
+
+The `createCompletenessScorer()` function does not take any options.
+
+This function returns an instance of the MastraScorer class. See the [MastraScorer reference](./mastra-scorer) for details on the `.run()` method and its input/output.
+
+## .run() Returns
+
+<PropertiesTable
+  content={[
+    {
+      name: "runId",
+      type: "string",
+      description: "The id of the run (optional).",
+    },
+    {
+      name: "extractStepResult",
+      type: "object",
+      description: "Object with extracted elements and coverage details: { inputElements: string[], outputElements: string[], missingElements: string[], elementCounts: { input: number, output: number } }",
+    },
+    {
+      name: "score",
+      type: "number",
+      description: "Completeness score (0-1) representing the proportion of input elements covered in the output.",
+    },
+  ]}
+/>
+
+## Element Extraction Details
+
+The scorer extracts and analyzes several types of elements:
+
+- Nouns: Key objects, concepts, and entities
+- Verbs: Actions and states (converted to infinitive form)
+- Topics: Main subjects and themes
+- Terms: Individual significant words
+
+The extraction process includes:
+
+- Normalization of text (removing diacritics, converting to lowercase)
+- Splitting camelCase words
+- Handling of word boundaries
+- Special handling of short words (3 characters or less)
+- Deduplication of elements
+
+## Scoring Details
+
+The scorer evaluates completeness through linguistic element coverage analysis.
+
+### Scoring Process
+
+1. Extracts key elements:
+   - Nouns and named entities
+   - Action verbs
+   - Topic-specific terms
+   - Normalized word forms
+2. Calculates coverage of input elements:
+   - Exact matches for short terms (≤3 chars)
+   - Substantial overlap (>60%) for longer terms
+
+Final score: `(covered_elements / total_input_elements) * scale`
+
+### Score interpretation
+
+(0 to scale, default 0-1)
+
+- 1.0: Complete coverage - contains all input elements
+- 0.7-0.9: High coverage - includes most key elements
+- 0.4-0.6: Partial coverage - contains some key elements
+- 0.1-0.3: Low coverage - missing most key elements
+- 0.0: No coverage - output lacks all input elements
+
+## Related
+
+- [Answer Relevancy Scorer](./answer-relevancy)
+- [Content Similarity Scorer](./content-similarity)
+- [Textual Difference Scorer](./textual-difference)
+- [Keyword Coverage Scorer](./keyword-coverage)

package/.docs/raw/reference/scorers/content-similarity.mdx

@@ -0,0 +1,96 @@
+---
+title: "Reference: Content Similarity | Scorers | Mastra Docs"
+description: Documentation for the Content Similarity Scorer in Mastra, which measures textual similarity between strings and provides a matching score.
+---
+
+# Content Similarity Scorer
+
+The `createContentSimilarityScorer()` function measures the textual similarity between two strings, providing a score that indicates how closely they match. It supports configurable options for case sensitivity and whitespace handling.
+
+For a usage example, see the [Content Similarity Examples](/examples/scorers/content-similarity).
+
+## Parameters
+
+The `createContentSimilarityScorer()` function accepts a single options object with the following properties:
+
+<PropertiesTable
+  content={[
+    {
+      name: "ignoreCase",
+      type: "boolean",
+      required: false,
+      defaultValue: "true",
+      description: "Whether to ignore case differences when comparing strings.",
+    },
+    {
+      name: "ignoreWhitespace",
+      type: "boolean",
+      required: false,
+      defaultValue: "true",
+      description: "Whether to normalize whitespace when comparing strings.",
+    },
+  ]}
+/>
+
+This function returns an instance of the MastraScorer class. See the [MastraScorer reference](./mastra-scorer) for details on the `.run()` method and its input/output.
+
+## .run() Returns
+
+<PropertiesTable
+  content={[
+    {
+      name: "runId",
+      type: "string",
+      description: "The id of the run (optional).",
+    },
+    {
+      name: "extractStepResult",
+      type: "object",
+      description: "Object with processed input and output: { processedInput: string, processedOutput: string }",
+    },
+    {
+      name: "analyzeStepResult",
+      type: "object",
+      description: "Object with similarity: { similarity: number }",
+    },
+    {
+      name: "score",
+      type: "number",
+      description: "Similarity score (0-1) where 1 indicates perfect similarity.",
+    },
+  ]}
+/>
+
+## Scoring Details
+
+The scorer evaluates textual similarity through character-level matching and configurable text normalization.
+
+### Scoring Process
+
+1. Normalizes text:
+   - Case normalization (if ignoreCase: true)
+   - Whitespace normalization (if ignoreWhitespace: true)
+2. Compares processed strings using string-similarity algorithm:
+   - Analyzes character sequences
+   - Aligns word boundaries
+   - Considers relative positions
+   - Accounts for length differences
+
+Final score: `similarity_value * scale`
+
+### Score interpretation
+
+(0 to scale, default 0-1)
+
+- 1.0: Perfect match - identical texts
+- 0.7-0.9: High similarity - mostly matching content
+- 0.4-0.6: Moderate similarity - partial matches
+- 0.1-0.3: Low similarity - few matching patterns
+- 0.0: No similarity - completely different texts
+
+## Related
+
+- [Completeness Scorer](./completeness)
+- [Textual Difference Scorer](./textual-difference)
+- [Answer Relevancy Scorer](./answer-relevancy)
+- [Keyword Coverage Scorer](./keyword-coverage)

package/.docs/raw/reference/scorers/custom-code-scorer.mdx

@@ -0,0 +1,155 @@
+---
+title: "Reference: Create Custom Scorer | Scorers | Mastra Docs"
+description: Documentation for creating custom code scorers in Mastra, allowing users to define their own evaluation logic.
+---
+
+# createScorer
+
+Mastra allows you to define your own custom code scorers for evaluating input/output pairs using any logic you choose. Custom scorers integrate seamlessly with the Mastra scoring framework and can be used anywhere built-in scorers are used.
+
+For a usage example, see the [Custom Code Scorer Examples](/examples/scorers/custom-native-javascript-eval).
+
+## How to Create a Custom Scorer
+
+Use the `createScorer` factory to define your scorer. You must provide at least a `name`, `description`, and an `analyze` function. Optionally, you can provide `extract` and `reason` functions for multi-step or more advanced logic.
+
+## createScorer Options
+
+<PropertiesTable
+  content={[
+    {
+      name: "name",
+      type: "string",
+      required: true,
+      description: "Name of the scorer.",
+    },
+    {
+      name: "description",
+      type: "string",
+      required: true,
+      description: "Description of what the scorer does.",
+    },
+    {
+      name: "analyze",
+      type: "function",
+      required: true,
+      description: "Main scoring logic",
+    },
+    {
+      name: "extract",
+      type: "function",
+      required: false,
+      description: "Optional pre-processing step.",
+    },
+    {
+      name: "reason",
+      type: "function",
+      required: false,
+      description: "Optional reason/explanation step.",
+    },
+    {
+      name: "metadata",
+      type: "object",
+      required: false,
+      description: "Optional metadata for the scorer.",
+    },
+  ]}
+/>
+
+This function returns an instance of the MastraScorer class. See the [MastraScorer reference](./mastra-scorer) for details on the `.run()` method and its input/output.
+
+## Step Function Signatures
+
+
+### extract
+<PropertiesTable
+  content={[
+    {
+      name: "input",
+      type: "Record<string, any>[]",
+      required: false,
+      description:
+        "Input records provided to the scorer. If the scorer is added to an agent, this will be an array of user messages, e.g. `[{ role: 'user', content: 'hello world' }]`. If the scorer is used in a workflow, this will be the input of the workflow.",
+    },
+    {
+      name: "output",
+      type: "Record<string, any>",
+      required: true,
+      description:
+        "Output record provided to the scorer. For agents, this is usually the agent's response. For workflows, this is the workflow's output.",
+    },
+  ]}
+/>
+Returns: `{ results: any }`  
+The method must return an object with a `results` property. The value of `results` will be passed to the analyze function as `extractStepResult`.
+
+### analyze
+<PropertiesTable
+  content={[
+    {
+      name: "input",
+      type: "Record<string, any>[]",
+      required: true,
+      description:
+        "Input records provided to the scorer. If the scorer is added to an agent, this will be an array of user messages, e.g. `[{ role: 'user', content: 'hello world' }]`. If the scorer is used in a workflow, this will be the input of the workflow.",
+    },
+    {
+      name: "output",
+      type: "Record<string, any>",
+      required: true,
+      description:
+        "Output record provided to the scorer. For agents, this is usually the agent's response. For workflows, this is the workflow's output.",
+    },
+    {
+      name: "extractStepResult",
+      type: "object",
+      required: false,
+      description: "Result of the extract step, if defined (optional).",
+    },
+  ]}
+/>
+Returns: `{ score: number, results?: any }`  
+The method must return an object with a `score` property (required). Optionally, it may return a `results` property. The value of `results` will be passed to the reason function as `analyzeStepResult`.
+
+
+### reason
+<PropertiesTable
+  content={[
+    {
+      name: "input",
+      type: "Record<string, any>[]",
+      required: true,
+      description:
+        "Input records provided to the scorer. If the scorer is added to an agent, this will be an array of user messages, e.g. `[{ role: 'user', content: 'hello world' }]`. If the scorer is used in a workflow, this will be the input of the workflow.",
+    },
+    {
+      name: "output",
+      type: "Record<string, any>",
+      required: true,
+      description:
+        "Output record provided to the scorer. For agents, this is usually the agent's response. For workflows, this is the workflow's output.",
+    },
+    {
+      name: "score",
+      type: "number",
+      required: true,
+      description: "Score computed by the analyze step.",
+    },
+    {
+      name: "analyzeStepResult",
+      type: "object",
+      required: true,
+      description: "Result of the analyze step.",
+    },
+    {
+      name: "extractStepResult",
+      type: "object",
+      required: false,
+      description: "Result of the extract step, if defined (optional).",  
+    },
+  ]}
+/>
+Returns: `{ reason: string }`  
+The method must return an object with a `reason` property, which should be a string explaining the score.
+
+All step functions can be async.

package/.docs/raw/reference/scorers/faithfulness.mdx

@@ -0,0 +1,122 @@
+---
+title: "Reference: Faithfulness | Scorers | Mastra Docs"
+description: Documentation for the Faithfulness Scorer in Mastra, which evaluates the factual accuracy of LLM outputs compared to the provided context.
+---
+
+# Faithfulness Scorer
+
+The `createFaithfulnessScorer()` function evaluates how factually accurate an LLM's output is compared to the provided context. It extracts claims from the output and verifies them against the context, making it essential to measure RAG pipeline responses' reliability.
+
+For a usage example, see the [Faithfulness Examples](/examples/scorers/faithfulness).
+
+## Parameters
+
+The `createFaithfulnessScorer()` function accepts a single options object with the following properties:
+
+<PropertiesTable
+  content={[
+    {
+      name: "model",
+      type: "LanguageModel",
+      required: true,
+      description: "Configuration for the model used to evaluate faithfulness.",
+    },
+    {
+      name: "context",
+      type: "string[]",
+      required: true,
+      description: "Array of context chunks against which the output's claims will be verified.",
+    },
+    {
+      name: "scale",
+      type: "number",
+      required: false,
+      defaultValue: "1",
+      description: "The maximum score value. The final score will be normalized to this scale.",
+    },
+  ]}
+/>
+
+This function returns an instance of the MastraScorer class. The `.run()` method accepts the same input as other scorers (see the [MastraScorer reference](./mastra-scorer)), but the return value includes LLM-specific fields as documented below.
+
+## .run() Returns
+
+<PropertiesTable
+  content={[
+    {
+      name: "runId",
+      type: "string",
+      description: "The id of the run (optional).",
+    },
+    {
+      name: "extractStepResult",
+      type: "string[]",
+      description: "Array of extracted claims from the output.",
+    },
+    {
+      name: "extractPrompt",
+      type: "string",
+      description: "The prompt sent to the LLM for the extract step (optional).",
+    },
+    {
+      name: "analyzeStepResult",
+      type: "object",
+      description: "Object with verdicts: { verdicts: Array<{ verdict: 'yes' | 'no' | 'unsure', reason: string }> }",
+    },
+    {
+      name: "analyzePrompt",
+      type: "string",
+      description: "The prompt sent to the LLM for the analyze step (optional).",
+    },
+    {
+      name: "score",
+      type: "number",
+      description: "A score between 0 and the configured scale, representing the proportion of claims that are supported by the context.",
+    },
+    {
+      name: "reason",
+      type: "string",
+      description: "A detailed explanation of the score, including which claims were supported, contradicted, or marked as unsure.",
+    },
+    {
+      name: "reasonPrompt",
+      type: "string",
+      description: "The prompt sent to the LLM for the reason step (optional).",
+    },
+  ]}
+/>
+
+## Scoring Details
+
+The scorer evaluates faithfulness through claim verification against provided context.
+
+### Scoring Process
+
+1. Analyzes claims and context:
+   - Extracts all claims (factual and speculative)
+   - Verifies each claim against context
+   - Assigns one of three verdicts:
+     - "yes" - claim supported by context
+     - "no" - claim contradicts context
+     - "unsure" - claim unverifiable
+2. Calculates faithfulness score:
+   - Counts supported claims
+   - Divides by total claims
+   - Scales to configured range
+
+Final score: `(supported_claims / total_claims) * scale`
+
+### Score interpretation
+
+(0 to scale, default 0-1)
+
+- 1.0: All claims supported by context
+- 0.7-0.9: Most claims supported, few unverifiable
+- 0.4-0.6: Mixed support with some contradictions
+- 0.1-0.3: Limited support, many contradictions
+- 0.0: No supported claims
+
+## Related
+
+- [Answer Relevancy Scorer](./answer-relevancy)
+- [Hallucination Scorer](./hallucination)