@mastra/mcp-docs-server 0.13.7-alpha.0 → 0.13.7-alpha.2

package/.docs/raw/reference/scorers/hallucination.mdx

@@ -0,0 +1,133 @@
+---
+title: "Reference: Hallucination | Scorers | Mastra Docs"
+description: Documentation for the Hallucination Scorer in Mastra, which evaluates the factual correctness of LLM outputs by identifying contradictions with provided context.
+---
+
+# Hallucination Scorer
+
+The `createHallucinationScorer()` function evaluates whether an LLM generates factually correct information by comparing its output against the provided context. This scorer measures hallucination by identifying direct contradictions between the context and the output.
+
+For a usage example, see the [Hallucination Examples](/examples/scorers/hallucination).
+
+## Parameters
+
+The `createHallucinationScorer()` function accepts a single options object with the following properties:
+
+<PropertiesTable
+  content={[
+    {
+      name: "model",
+      type: "LanguageModel",
+      required: true,
+      description: "Configuration for the model used to evaluate hallucination.",
+    },
+    {
+      name: "scale",
+      type: "number",
+      required: false,
+      defaultValue: "1",
+      description: "Maximum score value.",
+    },
+  ]}
+/>
+
+This function returns an instance of the MastraScorer class. The `.run()` method accepts the same input as other scorers (see the [MastraScorer reference](./mastra-scorer)), but the return value includes LLM-specific fields as documented below.
+
+## .run() Returns
+
+<PropertiesTable
+  content={[
+    {
+      name: "runId",
+      type: "string",
+      description: "The id of the run (optional).",
+    },
+    {
+      name: "extractStepResult",
+      type: "object",
+      description: "Object with extracted claims: { claims: string[] }",
+    },
+    {
+      name: "extractPrompt",
+      type: "string",
+      description: "The prompt sent to the LLM for the extract step (optional).",
+    },
+    {
+      name: "analyzeStepResult",
+      type: "object",
+      description: "Object with verdicts: { verdicts: Array<{ statement: string, verdict: 'yes' | 'no', reason: string }> }",
+    },
+    {
+      name: "analyzePrompt",
+      type: "string",
+      description: "The prompt sent to the LLM for the analyze step (optional).",
+    },
+    {
+      name: "score",
+      type: "number",
+      description: "Hallucination score (0 to scale, default 0-1).",
+    },
+    {
+      name: "reason",
+      type: "string",
+      description: "Detailed explanation of the score and identified contradictions.",
+    },
+    {
+      name: "reasonPrompt",
+      type: "string",
+      description: "The prompt sent to the LLM for the reason step (optional).",
+    },
+  ]}
+/>
+
+## Scoring Details
+
+The scorer evaluates hallucination through contradiction detection and unsupported claim analysis.
+
+### Scoring Process
+
+1. Analyzes factual content:
+   - Extracts statements from context
+   - Identifies numerical values and dates
+   - Maps statement relationships
+2. Analyzes output for hallucinations:
+   - Compares against context statements
+   - Marks direct conflicts as hallucinations
+   - Identifies unsupported claims as hallucinations
+   - Evaluates numerical accuracy
+   - Considers approximation context
+3. Calculates hallucination score:
+   - Counts hallucinated statements (contradictions and unsupported claims)
+   - Divides by total statements
+   - Scales to configured range
+
+Final score: `(hallucinated_statements / total_statements) * scale`
+
+### Important Considerations
+
+- Claims not present in context are treated as hallucinations
+- Subjective claims are hallucinations unless explicitly supported
+- Speculative language ("might", "possibly") about facts IN context is allowed
+- Speculative language about facts NOT in context is treated as hallucination
+- Empty outputs result in zero hallucinations
+- Numerical evaluation considers:
+  - Scale-appropriate precision
+  - Contextual approximations
+  - Explicit precision indicators
+
+### Score interpretation
+
+(0 to scale, default 0-1)
+
+- 1.0: Complete hallucination - contradicts all context statements
+- 0.75: High hallucination - contradicts 75% of context statements
+- 0.5: Moderate hallucination - contradicts half of context statements
+- 0.25: Low hallucination - contradicts 25% of context statements
+- 0.0: No hallucination - output aligns with all context statements
+
+**Note:** The score represents the degree of hallucination - lower scores indicate better factual alignment with the provided context
+
+## Related
+
+- [Faithfulness Scorer](./faithfulness)
+- [Answer Relevancy Scorer](./answer-relevancy)

package/.docs/raw/reference/scorers/keyword-coverage.mdx

@@ -0,0 +1,92 @@
+---
+title: "Reference: Keyword Coverage | Scorers | Mastra Docs"
+description: Documentation for the Keyword Coverage Scorer in Mastra, which evaluates how well LLM outputs cover important keywords from the input.
+---
+
+# Keyword Coverage Scorer
+
+The `createKeywordCoverageScorer()` function evaluates how well an LLM's output covers the important keywords from the input. It analyzes keyword presence and matches while ignoring common words and stop words.
+
+For a usage example, see the [Keyword Coverage Examples](/examples/scorers/keyword-coverage).
+
+## Parameters
+
+The `createKeywordCoverageScorer()` function does not take any options.
+
+This function returns an instance of the MastraScorer class. See the [MastraScorer reference](./mastra-scorer) for details on the `.run()` method and its input/output.
+
+## .run() Returns
+
+<PropertiesTable
+  content={[
+    {
+      name: "runId",
+      type: "string",
+      description: "The id of the run (optional).",
+    },
+    {
+      name: "extractStepResult",
+      type: "object",
+      description: "Object with extracted keywords: { referenceKeywords: Set<string>, responseKeywords: Set<string> }",
+    },
+    {
+      name: "analyzeStepResult",
+      type: "object",
+      description: "Object with keyword coverage: { totalKeywords: number, matchedKeywords: number }",
+    },
+    {
+      name: "score",
+      type: "number",
+      description: "Coverage score (0-1) representing the proportion of matched keywords.",
+    },
+  ]}
+/>
+
+## Scoring Details
+
+The scorer evaluates keyword coverage by matching keywords with the following features:
+
+- Common word and stop word filtering (e.g., "the", "a", "and")
+- Case-insensitive matching
+- Word form variation handling
+- Special handling of technical terms and compound words
+
+### Scoring Process
+
+1. Processes keywords from input and output:
+   - Filters out common words and stop words
+   - Normalizes case and word forms
+   - Handles special terms and compounds
+2. Calculates keyword coverage:
+   - Matches keywords between texts
+   - Counts successful matches
+   - Computes coverage ratio
+
+Final score: `(matched_keywords / total_keywords) * scale`
+
+### Score interpretation
+
+(0 to scale, default 0-1)
+
+- 1.0: Perfect keyword coverage
+- 0.7-0.9: Good coverage with most keywords present
+- 0.4-0.6: Moderate coverage with some keywords missing
+- 0.1-0.3: Poor coverage with many keywords missing
+- 0.0: No keyword matches
+
+## Special Cases
+
+The scorer handles several special cases:
+
+- Empty input/output: Returns score of 1.0 if both empty, 0.0 if only one is empty
+- Single word: Treated as a single keyword
+- Technical terms: Preserves compound technical terms (e.g., "React.js", "machine learning")
+- Case differences: "JavaScript" matches "javascript"
+- Common words: Ignored in scoring to focus on meaningful keywords
+
+## Related
+
+- [Completeness Scorer](./completeness)
+- [Content Similarity Scorer](./content-similarity)
+- [Answer Relevancy Scorer](./answer-relevancy)
+- [Textual Difference Scorer](./textual-difference)

package/.docs/raw/reference/scorers/llm-scorer.mdx

@@ -0,0 +1,210 @@
+---
+title: "Reference: createLLMScorer | Scorers | Mastra Docs"
+description: Documentation for creating LLM-based scorers in Mastra, allowing users to define evaluation logic using language models.
+---
+
+# createLLMScorer
+
+The `createLLMScorer()` function lets you define custom scorers that use a language model (LLM) as a judge for evaluation. LLM scorers are ideal for tasks where you want to use prompt-based evaluation, such as answer relevancy, faithfulness, or custom prompt-based metrics. LLM scorers integrate seamlessly with the Mastra scoring framework and can be used anywhere built-in scorers are used.
+
+For a usage example, see the [Custom LLM Judge Examples](/examples/scorers/custom-llm-judge-eval).
+
+## createLLMScorer Options
+
+<PropertiesTable
+  content={[
+    {
+      name: "name",
+      type: "string",
+      required: true,
+      description: "Name of the scorer.",
+    },
+    {
+      name: "description",
+      type: "string",
+      required: true,
+      description: "Description of what the scorer does.",
+    },
+    {
+      name: "judge",
+      type: "object",
+      required: true,
+      description: "Judge configuration object. Must include a model and instructions (system prompt). See Judge Object section below.",
+    },
+    {
+      name: "extract",
+      type: "object",
+      required: false,
+      description: "(Optional) Extraction step configuration object. See Extract Object section below.",
+    },
+    {
+      name: "analyze",
+      type: "object",
+      required: true,
+      description: "Analysis step configuration object. See Analyze Object section below.",
+    },
+    {
+      name: "reason",
+      type: "object",
+      required: false,
+      description: "(Optional) Reason step configuration object. See Reason Object section below.",
+    },
+    {
+      name: "calculateScore",
+      type: "function",
+      required: true,
+      description: "Function: ({ run }) => number. Computes the final score from the analyze step result.",
+    },
+  ]}
+/>
+
+This function returns an instance of the MastraScorer class. See the [MastraScorer reference](./mastra-scorer) for details on the `.run()` method and its input/output.
+
+## Judge Object
+<PropertiesTable
+  content={[
+    {
+      name: "model",
+      type: "LanguageModel",
+      required: true,
+      description: "The LLM model instance to use for evaluation.",
+    },
+    {
+      name: "instructions",
+      type: "string",
+      required: true,
+      description: "System prompt/instructions for the LLM.",
+    },
+  ]}
+/>
+
+## Extract Object
+<PropertiesTable
+  content={[
+    {
+      name: "description",
+      type: "string",
+      required: true,
+      description: "Description of the extract step.",
+    },
+    {
+      name: "judge",
+      type: "object",
+      required: false,
+      description: "(Optional) LLM judge for this step (can override main judge/model). See Judge Object section.",
+    },
+    {
+      name: "outputSchema",
+      type: "ZodSchema",
+      required: true,
+      description: "Zod schema for the expected output of the extract step.",
+    },
+    {
+      name: "createPrompt",
+      type: "function",
+      required: true,
+      description: "Function: ({ run: ScoringInput }) => string. Returns the prompt for the LLM.",
+    },
+  ]}
+/>
+
+## Analyze Object
+<PropertiesTable
+  content={[
+    {
+      name: "description",
+      type: "string",
+      required: true,
+      description: "Description of the analyze step.",
+    },
+    {
+      name: "judge",
+      type: "object",
+      required: false,
+      description: "(Optional) LLM judge for this step (can override main judge/model). See Judge Object section.",
+    },
+    {
+      name: "outputSchema",
+      type: "ZodSchema",
+      required: true,
+      description: "Zod schema for the expected output of the analyze step.",
+    },
+    {
+      name: "createPrompt",
+      type: "function",
+      required: true,
+      description: "Function: ({ run: ScoringInput & { extractStepResult } }) => string. Returns the LLM prompt.",
+    },
+  ]}
+/>
+
+## Calculate Score Function
+
+The `calculateScore` function converts the LLM's structured analysis into a numerical score. This function receives the results from previous steps but not the score itself (since that's what it calculates).
+
+<PropertiesTable
+  content={[
+    {
+      name: "input",
+      type: "Record<string, any>[]",
+      required: true,
+      description:
+        "Input records provided to the scorer. If the scorer is added to an agent, this will be an array of user messages, e.g. `[{ role: 'user', content: 'hello world' }]`. If the scorer is used in a workflow, this will be the input of the workflow.",
+    },
+    {
+      name: "output",
+      type: "Record<string, any>",
+      required: true,
+      description:
+        "Output record provided to the scorer. For agents, this is usually the agent's response. For workflows, this is the workflow's output.",
+    },
+    {
+      name: "runtimeContext",
+      type: "object",
+      required: false,
+      description: "Runtime context from the agent or workflow step being evaluated (optional).",
+    },
+    {
+      name: "extractStepResult",
+      type: "object",
+      required: false,
+      description: "Result of the extract step, if defined (optional).",
+    },
+    {
+      name: "analyzeStepResult",
+      type: "object",
+      required: true,
+      description: "Structured result from the analyze step, conforming to the outputSchema defined in the analyze step.",
+    },
+  ]}
+/>
+
+Returns: `number`  
+The function must return a numerical score, typically in the 0-1 range where 1 represents the best possible score.
+
+## Reason Object
+<PropertiesTable
+  content={[
+    {
+      name: "description",
+      type: "string",
+      required: true,
+      description: "Description of the reason step.",
+    },
+    {
+      name: "judge",
+      type: "object",
+      required: false,
+      description: "(Optional) LLM judge for this step (can override main judge/model). See Judge Object section.",
+    },
+    {
+      name: "createPrompt",
+      type: "function",
+      required: true,
+      description: "Function: ({ run }) => string. `run` includes input, output, extractStepResult, analyzeStepResult, and score. Returns the prompt for the LLM.",
+    },
+  ]}
+/>
+
+LLM scorers may also include step-specific prompt fields in the return value, such as `extractPrompt`, `analyzePrompt`, and `reasonPrompt`.
+

package/.docs/raw/reference/scorers/mastra-scorer.mdx

@@ -0,0 +1,218 @@
+---
+title: "Reference: MastraScorer | Scorers | Mastra Docs"
+description: Documentation for the MastraScorer base class in Mastra, which provides the foundation for all custom and built-in scorers.
+---
+
+# MastraScorer
+
+The `MastraScorer` class is the base class for all scorers in Mastra. It provides a standard `.run()` method for evaluating input/output pairs and supports multi-step scoring workflows. Most users will use `createScorer` or `createLLMScorer`, but advanced users can subclass or instantiate `MastraScorer` directly for full control.
+
+## Constructor Options
+
+<PropertiesTable
+  content={[
+    {
+      name: "name",
+      type: "string",
+      required: true,
+      description: "Name of the scorer.",
+    },
+    {
+      name: "description",
+      type: "string",
+      required: true,
+      description: "Description of what the scorer does.",
+    },
+    {
+      name: "extract",
+      type: "function",
+      required: false,
+      description: "Optional extraction step. See extract step signature below.",
+    },
+    {
+      name: "analyze",
+      type: "function",
+      required: true,
+      description: "Main scoring logic. See analyze step signature below.",
+    },
+    {
+      name: "reason",
+      type: "function",
+      required: false,
+      description: "Optional reason/explanation step. See reason step signature below.",
+    },
+    {
+      name: "metadata",
+      type: "object",
+      required: false,
+      description: "Optional metadata for the scorer.",
+    },
+    {
+      name: "isLLMScorer",
+      type: "boolean",
+      required: false,
+      description: "(Internal) Used to distinguish LLM scorers.",
+    },
+  ]}
+/>
+
+## Step Function Signatures
+
+### extract
+<PropertiesTable
+  content={[
+    {
+      name: "input",
+      type: "Record<string, any>[]",
+      required: false,
+      description: "Input records provided to the scorer. If the scorer is added to an agent, this will be an array of user messages, e.g. `[{ role: 'user', content: 'hello world' }]`. If the scorer is used in a workflow, this will be the input of the workflow.",
+    },
+    {
+      name: "output",
+      type: "Record<string, any>",
+      required: true,
+      description: "Output record provided to the scorer. For agents, this is usually the agent's response. For workflows, this is the workflow's output.",
+    },
+  ]}
+/>
+Returns: `{ results: any }`  
+The method must return an object with a `results` property. The value of `results` will be passed to the analyze function as `extractStepResult`.
+
+### analyze
+<PropertiesTable
+  content={[
+    {
+      name: "input",
+      type: "Record<string, any>[]",
+      required: true,
+      description: "Input records provided to the scorer. If the scorer is added to an agent, this will be an array of user messages, e.g. `[{ role: 'user', content: 'hello world' }]`. If the scorer is used in a workflow, this will be the input of the workflow.",
+    },
+    {
+      name: "output",
+      type: "Record<string, any>",
+      required: true,
+      description: "Output record provided to the scorer. For agents, this is usually the agent's response. For workflows, this is the workflow's output.",
+    },
+    {
+      name: "extractStepResult",
+      type: "object",
+      required: false,
+      description: "Result of the extract step, if defined (optional).",
+    },
+  ]}
+/>
+Returns: `{ score: number, results?: any }`  
+The method must return an object with a `score` property (required). Optionally, it may return a `results` property. The value of `results` will be passed to the reason function as `analyzeStepResult`.
+
+### reason
+<PropertiesTable
+  content={[
+    {
+      name: "input",
+      type: "Record<string, any>[]",
+      required: true,
+      description: "Input records provided to the scorer. If the scorer is added to an agent, this will be an array of user messages, e.g. `[{ role: 'user', content: 'hello world' }]`. If the scorer is used in a workflow, this will be the input of the workflow.",
+    },
+    {
+      name: "output",
+      type: "Record<string, any>",
+      required: true,
+      description: "Output record provided to the scorer. For agents, this is usually the agent's response. For workflows, this is the workflow's output.",
+    },
+    {
+      name: "score",
+      type: "number",
+      required: true,
+      description: "Score computed by the analyze step.",
+    },
+    {
+      name: "analyzeStepResult",
+      type: "object",
+      required: true,
+      description: "Result of the analyze step.",
+    },
+    {
+      name: "extractStepResult",
+      type: "object",
+      required: false,
+      description: "Result of the extract step, if defined (optional).",  
+    },
+  ]}
+/>
+Returns: `{ reason: string }`  
+The method must return an object with a `reason` property, which should be a string explaining the score.
+
+All step functions can be async.
+
+## .run() Input
+
+<PropertiesTable
+  content={[
+    {
+      name: "runId",
+      type: "string",
+      required: false,
+      description: "The id of the run (optional).",
+    },
+    {
+      name: "input",
+      type: "Record<string, any>[]",
+      required: true,
+      description: "An array of records. This should contain user messages or the data to be evaluated.",
+    },
+    {
+      name: "output",
+      type: "Record<string, any>",
+      required: true,
+      description: "A record. This should contain the output to be evaluated.",
+    },
+    {
+      name: "additionalContext",
+      type: "Record<string, any>",
+      required: false,
+      description: "Additional context for the run (optional).",
+    },
+    {
+      name: "runtimeContext",
+      type: "Record<string, any>",
+      required: false,
+      description: "Runtime context for the run (optional).",
+    },
+  ]}
+/>
+
+## .run() Returns
+
+<PropertiesTable
+  content={[
+    {
+      name: "runId",
+      type: "string",
+      description: "The id of the run (optional).",
+    },
+    {
+      name: "extractStepResult",
+      type: "object",
+      description: "Result of the extract step, if defined (optional).",
+    },
+    {
+      name: "analyzeStepResult",
+      type: "object",
+      description: "Result of the analyze step (custom structure defined by your scorer).",
+    },
+    {
+      name: "score",
+      type: "number",
+      description: "Score computed by your analyze function.",
+    },
+    {
+      name: "reason",
+      type: "string",
+      description: "Reason/explanation for the score, if defined (optional).",
+    },
+  ]}
+/>
+
+## Integration
+
+MastraScorer instances can be used for agents and workflow steps