@mastra/mcp-docs-server 0.13.6 → 0.13.7-alpha.1

package/.docs/raw/reference/scorers/faithfulness.mdx

@@ -0,0 +1,123 @@
+---
+title: "Reference: Faithfulness | Scorers | Mastra Docs"
+description: Documentation for the Faithfulness Scorer in Mastra, which evaluates the factual accuracy of LLM outputs compared to the provided context.
+---
+
+# Faithfulness Scorer
+
+The `createFaithfulnessScorer()` function evaluates how factually accurate an LLM's output is compared to the provided context. It extracts claims from the output and verifies them against the context, making it essential to measure RAG pipeline responses' reliability.
+
+For a usage example, see the [Faithfulness Examples](/examples/scorers/faithfulness).
+
+## Parameters
+
+The `createFaithfulnessScorer()` function accepts a single options object with the following properties:
+
+<PropertiesTable
+  content={[
+    {
+      name: "model",
+      type: "LanguageModel",
+      required: true,
+      description: "Configuration for the model used to evaluate faithfulness.",
+    },
+    {
+      name: "context",
+      type: "string[]",
+      required: true,
+      description: "Array of context chunks against which the output's claims will be verified.",
+    },
+    {
+      name: "scale",
+      type: "number",
+      required: false,
+      defaultValue: "1",
+      description: "The maximum score value. The final score will be normalized to this scale.",
+    },
+  ]}
+/>
+
+This function returns an instance of the MastraScorer class. The `.run()` method accepts the same input as other scorers (see the [MastraScorer reference](./mastra-scorer)), but the return value includes LLM-specific fields as documented below.
+
+## .run() Returns
+
+<PropertiesTable
+  content={[
+    {
+      name: "runId",
+      type: "string",
+      description: "The id of the run (optional).",
+    },
+    {
+      name: "extractStepResult",
+      type: "string[]",
+      description: "Array of extracted claims from the output.",
+    },
+    {
+      name: "extractPrompt",
+      type: "string",
+      description: "The prompt sent to the LLM for the extract step (optional).",
+    },
+    {
+      name: "analyzeStepResult",
+      type: "object",
+      description: "Object with verdicts: { verdicts: Array<{ verdict: 'yes' | 'no' | 'unsure', reason: string }> }",
+    },
+    {
+      name: "analyzePrompt",
+      type: "string",
+      description: "The prompt sent to the LLM for the analyze step (optional).",
+    },
+    {
+      name: "score",
+      type: "number",
+      description: "A score between 0 and the configured scale, representing the proportion of claims that are supported by the context.",
+    },
+    {
+      name: "reason",
+      type: "string",
+      description: "A detailed explanation of the score, including which claims were supported, contradicted, or marked as unsure.",
+    },
+    {
+      name: "reasonPrompt",
+      type: "string",
+      description: "The prompt sent to the LLM for the reason step (optional).",
+    },
+  ]}
+/>
+
+## Scoring Details
+
+The scorer evaluates faithfulness through claim verification against provided context.
+
+### Scoring Process
+
+1. Analyzes claims and context:
+   - Extracts all claims (factual and speculative)
+   - Verifies each claim against context
+   - Assigns one of three verdicts:
+     - "yes" - claim supported by context
+     - "no" - claim contradicts context
+     - "unsure" - claim unverifiable
+2. Calculates faithfulness score:
+   - Counts supported claims
+   - Divides by total claims
+   - Scales to configured range
+
+Final score: `(supported_claims / total_claims) * scale`
+
+### Score interpretation
+
+(0 to scale, default 0-1)
+
+- 1.0: All claims supported by context
+- 0.7-0.9: Most claims supported, few unverifiable
+- 0.4-0.6: Mixed support with some contradictions
+- 0.1-0.3: Limited support, many contradictions
+- 0.0: No supported claims
+
+## Related
+
+- [Answer Relevancy Scorer](./answer-relevancy)
+- [Hallucination Scorer](./hallucination)
+- [Context Relevancy Scorer](./context-relevancy)

package/.docs/raw/reference/scorers/hallucination.mdx

@@ -0,0 +1,135 @@
+---
+title: "Reference: Hallucination | Scorers | Mastra Docs"
+description: Documentation for the Hallucination Scorer in Mastra, which evaluates the factual correctness of LLM outputs by identifying contradictions with provided context.
+---
+
+# Hallucination Scorer
+
+The `createHallucinationScorer()` function evaluates whether an LLM generates factually correct information by comparing its output against the provided context. This scorer measures hallucination by identifying direct contradictions between the context and the output.
+
+For a usage example, see the [Hallucination Examples](/examples/scorers/hallucination).
+
+## Parameters
+
+The `createHallucinationScorer()` function accepts a single options object with the following properties:
+
+<PropertiesTable
+  content={[
+    {
+      name: "model",
+      type: "LanguageModel",
+      required: true,
+      description: "Configuration for the model used to evaluate hallucination.",
+    },
+    {
+      name: "scale",
+      type: "number",
+      required: false,
+      defaultValue: "1",
+      description: "Maximum score value.",
+    },
+  ]}
+/>
+
+This function returns an instance of the MastraScorer class. The `.run()` method accepts the same input as other scorers (see the [MastraScorer reference](./mastra-scorer)), but the return value includes LLM-specific fields as documented below.
+
+## .run() Returns
+
+<PropertiesTable
+  content={[
+    {
+      name: "runId",
+      type: "string",
+      description: "The id of the run (optional).",
+    },
+    {
+      name: "extractStepResult",
+      type: "object",
+      description: "Object with extracted claims: { claims: string[] }",
+    },
+    {
+      name: "extractPrompt",
+      type: "string",
+      description: "The prompt sent to the LLM for the extract step (optional).",
+    },
+    {
+      name: "analyzeStepResult",
+      type: "object",
+      description: "Object with verdicts: { verdicts: Array<{ statement: string, verdict: 'yes' | 'no', reason: string }> }",
+    },
+    {
+      name: "analyzePrompt",
+      type: "string",
+      description: "The prompt sent to the LLM for the analyze step (optional).",
+    },
+    {
+      name: "score",
+      type: "number",
+      description: "Hallucination score (0 to scale, default 0-1).",
+    },
+    {
+      name: "reason",
+      type: "string",
+      description: "Detailed explanation of the score and identified contradictions.",
+    },
+    {
+      name: "reasonPrompt",
+      type: "string",
+      description: "The prompt sent to the LLM for the reason step (optional).",
+    },
+  ]}
+/>
+
+## Scoring Details
+
+The scorer evaluates hallucination through contradiction detection and unsupported claim analysis.
+
+### Scoring Process
+
+1. Analyzes factual content:
+   - Extracts statements from context
+   - Identifies numerical values and dates
+   - Maps statement relationships
+2. Analyzes output for hallucinations:
+   - Compares against context statements
+   - Marks direct conflicts as hallucinations
+   - Identifies unsupported claims as hallucinations
+   - Evaluates numerical accuracy
+   - Considers approximation context
+3. Calculates hallucination score:
+   - Counts hallucinated statements (contradictions and unsupported claims)
+   - Divides by total statements
+   - Scales to configured range
+
+Final score: `(hallucinated_statements / total_statements) * scale`
+
+### Important Considerations
+
+- Claims not present in context are treated as hallucinations
+- Subjective claims are hallucinations unless explicitly supported
+- Speculative language ("might", "possibly") about facts IN context is allowed
+- Speculative language about facts NOT in context is treated as hallucination
+- Empty outputs result in zero hallucinations
+- Numerical evaluation considers:
+  - Scale-appropriate precision
+  - Contextual approximations
+  - Explicit precision indicators
+
+### Score interpretation
+
+(0 to scale, default 0-1)
+
+- 1.0: Complete hallucination - contradicts all context statements
+- 0.75: High hallucination - contradicts 75% of context statements
+- 0.5: Moderate hallucination - contradicts half of context statements
+- 0.25: Low hallucination - contradicts 25% of context statements
+- 0.0: No hallucination - output aligns with all context statements
+
+**Note:** The score represents the degree of hallucination - lower scores indicate better factual alignment with the provided context
+
+## Related
+
+- [Faithfulness Scorer](./faithfulness)
+- [Answer Relevancy Scorer](./answer-relevancy)
+- [Context Precision Scorer](./context-precision)
+- [Context Relevancy Scorer](./context-relevancy)

package/.docs/raw/reference/scorers/keyword-coverage.mdx

@@ -0,0 +1,92 @@
+---
+title: "Reference: Keyword Coverage | Scorers | Mastra Docs"
+description: Documentation for the Keyword Coverage Scorer in Mastra, which evaluates how well LLM outputs cover important keywords from the input.
+---
+
+# Keyword Coverage Scorer
+
+The `createKeywordCoverageScorer()` function evaluates how well an LLM's output covers the important keywords from the input. It analyzes keyword presence and matches while ignoring common words and stop words.
+
+For a usage example, see the [Keyword Coverage Examples](/examples/scorers/keyword-coverage).
+
+## Parameters
+
+The `createKeywordCoverageScorer()` function does not take any options.
+
+This function returns an instance of the MastraScorer class. See the [MastraScorer reference](./mastra-scorer) for details on the `.run()` method and its input/output.
+
+## .run() Returns
+
+<PropertiesTable
+  content={[
+    {
+      name: "runId",
+      type: "string",
+      description: "The id of the run (optional).",
+    },
+    {
+      name: "extractStepResult",
+      type: "object",
+      description: "Object with extracted keywords: { referenceKeywords: Set<string>, responseKeywords: Set<string> }",
+    },
+    {
+      name: "analyzeStepResult",
+      type: "object",
+      description: "Object with keyword coverage: { totalKeywords: number, matchedKeywords: number }",
+    },
+    {
+      name: "score",
+      type: "number",
+      description: "Coverage score (0-1) representing the proportion of matched keywords.",
+    },
+  ]}
+/>
+
+## Scoring Details
+
+The scorer evaluates keyword coverage by matching keywords with the following features:
+
+- Common word and stop word filtering (e.g., "the", "a", "and")
+- Case-insensitive matching
+- Word form variation handling
+- Special handling of technical terms and compound words
+
+### Scoring Process
+
+1. Processes keywords from input and output:
+   - Filters out common words and stop words
+   - Normalizes case and word forms
+   - Handles special terms and compounds
+2. Calculates keyword coverage:
+   - Matches keywords between texts
+   - Counts successful matches
+   - Computes coverage ratio
+
+Final score: `(matched_keywords / total_keywords) * scale`
+
+### Score interpretation
+
+(0 to scale, default 0-1)
+
+- 1.0: Perfect keyword coverage
+- 0.7-0.9: Good coverage with most keywords present
+- 0.4-0.6: Moderate coverage with some keywords missing
+- 0.1-0.3: Poor coverage with many keywords missing
+- 0.0: No keyword matches
+
+## Special Cases
+
+The scorer handles several special cases:
+
+- Empty input/output: Returns score of 1.0 if both empty, 0.0 if only one is empty
+- Single word: Treated as a single keyword
+- Technical terms: Preserves compound technical terms (e.g., "React.js", "machine learning")
+- Case differences: "JavaScript" matches "javascript"
+- Common words: Ignored in scoring to focus on meaningful keywords
+
+## Related
+
+- [Completeness Scorer](./completeness)
+- [Content Similarity Scorer](./content-similarity)
+- [Answer Relevancy Scorer](./answer-relevancy)
+- [Textual Difference Scorer](./textual-difference)

package/.docs/raw/reference/scorers/llm-scorer.mdx

@@ -0,0 +1,166 @@
+---
+title: "Reference: createLLMScorer | Scorers | Mastra Docs"
+description: Documentation for creating LLM-based scorers in Mastra, allowing users to define evaluation logic using language models.
+---
+
+# LLM Scorer
+
+The `createLLMScorer()` function lets you define custom scorers that use a language model (LLM) as a judge for evaluation. LLM scorers are ideal for tasks where you want to use prompt-based evaluation, such as answer relevancy, faithfulness, or custom prompt-based metrics. LLM scorers integrate seamlessly with the Mastra scoring framework and can be used anywhere built-in scorers are used.
+
+For a usage example, see the [Custom LLM Judge Examples](/examples/scorers/custom-llm-judge-eval).
+
+## createLLMScorer Options
+
+<PropertiesTable
+  content={[
+    {
+      name: "name",
+      type: "string",
+      required: true,
+      description: "Name of the scorer.",
+    },
+    {
+      name: "description",
+      type: "string",
+      required: true,
+      description: "Description of what the scorer does.",
+    },
+    {
+      name: "judge",
+      type: "object",
+      required: true,
+      description: "Judge configuration object. Must include a model and instructions (system prompt). See Judge Object section below.",
+    },
+    {
+      name: "extract",
+      type: "object",
+      required: false,
+      description: "(Optional) Extraction step configuration object. See Extract Object section below.",
+    },
+    {
+      name: "analyze",
+      type: "object",
+      required: true,
+      description: "Analysis step configuration object. See Analyze Object section below.",
+    },
+    {
+      name: "reason",
+      type: "object",
+      required: false,
+      description: "(Optional) Reason step configuration object. See Reason Object section below.",
+    },
+    {
+      name: "calculateScore",
+      type: "function",
+      required: true,
+      description: "Function: ({ run }) => number. Computes the final score from the analyze step result.",
+    },
+  ]}
+/>
+
+This function returns an instance of the MastraScorer class. See the [MastraScorer reference](./mastra-scorer) for details on the `.run()` method and its input/output.
+
+## Judge Object
+<PropertiesTable
+  content={[
+    {
+      name: "model",
+      type: "LanguageModel",
+      required: true,
+      description: "The LLM model instance to use for evaluation.",
+    },
+    {
+      name: "instructions",
+      type: "string",
+      required: true,
+      description: "System prompt/instructions for the LLM.",
+    },
+  ]}
+/>
+
+## Extract Object
+<PropertiesTable
+  content={[
+    {
+      name: "description",
+      type: "string",
+      required: true,
+      description: "Description of the extract step.",
+    },
+    {
+      name: "judge",
+      type: "object",
+      required: false,
+      description: "(Optional) LLM judge for this step (can override main judge/model). See Judge Object section.",
+    },
+    {
+      name: "outputSchema",
+      type: "ZodSchema",
+      required: true,
+      description: "Zod schema for the expected output of the extract step.",
+    },
+    {
+      name: "createPrompt",
+      type: "function",
+      required: true,
+      description: "Function: ({ run: ScoringInput }) => string. Returns the prompt for the LLM.",
+    },
+  ]}
+/>
+
+## Analyze Object
+<PropertiesTable
+  content={[
+    {
+      name: "description",
+      type: "string",
+      required: true,
+      description: "Description of the analyze step.",
+    },
+    {
+      name: "judge",
+      type: "object",
+      required: false,
+      description: "(Optional) LLM judge for this step (can override main judge/model). See Judge Object section.",
+    },
+    {
+      name: "outputSchema",
+      type: "ZodSchema",
+      required: true,
+      description: "Zod schema for the expected output of the analyze step.",
+    },
+    {
+      name: "createPrompt",
+      type: "function",
+      required: true,
+      description: "Function: ({ run: ScoringInput & { extractStepResult } }) => string. Returns the LLM prompt.",
+    },
+  ]}
+/>
+
+## Reason Object
+<PropertiesTable
+  content={[
+    {
+      name: "description",
+      type: "string",
+      required: true,
+      description: "Description of the reason step.",
+    },
+    {
+      name: "judge",
+      type: "object",
+      required: false,
+      description: "(Optional) LLM judge for this step (can override main judge/model). See Judge Object section.",
+    },
+    {
+      name: "createPrompt",
+      type: "function",
+      required: true,
+      description: "Function: ({ run }) => string. `run` includes input, output, extractStepResult, analyzeStepResult, and score. Returns the prompt for the LLM.",
+    },
+  ]}
+/>
+
+LLM scorers may also include step-specific prompt fields in the return value, such as `extractPrompt`, `analyzePrompt`, and `reasonPrompt`.
+