@mastra/mcp-docs-server 0.0.0-error-handler-fix-20251020202607 → 0.0.0-execa-dynamic-import-20260304221256

package/.docs/raw/reference/scorers/faithfulness.mdx

@@ -1,233 +0,0 @@
----
-title: "Reference: Faithfulness | Scorers | Mastra Docs"
-description: Documentation for the Faithfulness Scorer in Mastra, which evaluates the factual accuracy of LLM outputs compared to the provided context.
----
-
-# Faithfulness Scorer
-
-The `createFaithfulnessScorer()` function evaluates how factually accurate an LLM's output is compared to the provided context. It extracts claims from the output and verifies them against the context, making it essential to measure RAG pipeline responses' reliability.
-
-## Parameters
-
-The `createFaithfulnessScorer()` function accepts a single options object with the following properties:
-
-<PropertiesTable
-  content={[
-    {
-      name: "model",
-      type: "LanguageModel",
-      required: true,
-      description: "Configuration for the model used to evaluate faithfulness.",
-    },
-    {
-      name: "context",
-      type: "string[]",
-      required: true,
-      description: "Array of context chunks against which the output's claims will be verified.",
-    },
-    {
-      name: "scale",
-      type: "number",
-      required: false,
-      defaultValue: "1",
-      description: "The maximum score value. The final score will be normalized to this scale.",
-    },
-  ]}
-/>
-
-This function returns an instance of the MastraScorer class. The `.run()` method accepts the same input as other scorers (see the [MastraScorer reference](./mastra-scorer)), but the return value includes LLM-specific fields as documented below.
-
-## .run() Returns
-
-<PropertiesTable
-  content={[
-    {
-      name: "runId",
-      type: "string",
-      description: "The id of the run (optional).",
-    },
-    {
-      name: "preprocessStepResult",
-      type: "string[]",
-      description: "Array of extracted claims from the output.",
-    },
-    {
-      name: "preprocessPrompt",
-      type: "string",
-      description: "The prompt sent to the LLM for the preprocess step (optional).",
-    },
-    {
-      name: "analyzeStepResult",
-      type: "object",
-      description: "Object with verdicts: { verdicts: Array<{ verdict: 'yes' | 'no' | 'unsure', reason: string }> }",
-    },
-    {
-      name: "analyzePrompt",
-      type: "string",
-      description: "The prompt sent to the LLM for the analyze step (optional).",
-    },
-    {
-      name: "score",
-      type: "number",
-      description: "A score between 0 and the configured scale, representing the proportion of claims that are supported by the context.",
-    },
-    {
-      name: "reason",
-      type: "string",
-      description: "A detailed explanation of the score, including which claims were supported, contradicted, or marked as unsure.",
-    },
-    {
-      name: "generateReasonPrompt",
-      type: "string",
-      description: "The prompt sent to the LLM for the generateReason step (optional).",
-    },
-  ]}
-/>
-
-## Scoring Details
-
-The scorer evaluates faithfulness through claim verification against provided context.
-
-### Scoring Process
-
-1. Analyzes claims and context:
-   - Extracts all claims (factual and speculative)
-   - Verifies each claim against context
-   - Assigns one of three verdicts:
-     - "yes" - claim supported by context
-     - "no" - claim contradicts context
-     - "unsure" - claim unverifiable
-2. Calculates faithfulness score:
-   - Counts supported claims
-   - Divides by total claims
-   - Scales to configured range
-
-Final score: `(supported_claims / total_claims) * scale`
-
-### Score interpretation
-
-A faithfulness score between 0 and 1:
-
-- **1.0**: All claims are accurate and directly supported by the context.
-- **0.7–0.9**: Most claims are correct, with minor additions or omissions.
-- **0.4–0.6**: Some claims are supported, but others are unverifiable.
-- **0.1–0.3**: Most of the content is inaccurate or unsupported.
-- **0.0**: All claims are false or contradict the context.
-
-## Examples
-
-### High faithfulness example
-
-In this example, the response closely aligns with the context. Each statement in the output is verifiable and supported by the provided context entries, resulting in a high score.
-
-```typescript filename="src/example-high-faithfulness.ts" showLineNumbers copy
-import { createFaithfulnessScorer } from "@mastra/evals/scorers/llm";
-
-const scorer = createFaithfulnessScorer({ model: 'openai/gpt-4o-mini', options: {
-  context: [
-    "The Tesla Model 3 was launched in 2017.",
-    "It has a range of up to 358 miles.",
-    "The base model accelerates 0-60 mph in 5.8 seconds."
-  ]
-});
-
-const query = "Tell me about the Tesla Model 3.";
-const response = "The Tesla Model 3 was introduced in 2017. It can travel up to 358 miles on a single charge and the base version goes from 0 to 60 mph in 5.8 seconds.";
-
-const result = await scorer.run({
-  input: [{ role: 'user', content: query }],
-  output: { text: response },
-});
-
-console.log(result);
-```
-
-#### High faithfulness output
-
-The output receives a score of 1 because all the information it provides can be directly traced to the context. There are no missing or contradictory facts.
-
-```typescript
-{
-  score: 1,
-  reason: 'The score is 1 because all claims made in the output are supported by the provided context.'
-}
-```
-
-### Mixed faithfulness example
-
-In this example, there are a mix of supported and unsupported claims. Some parts of the response are backed by the context, while others introduce new information not found in the source material.
-
-```typescript filename="src/example-mixed-faithfulness.ts" showLineNumbers copy
-import { createFaithfulnessScorer } from "@mastra/evals/scorers/llm";
-
-const scorer = createFaithfulnessScorer({ model: 'openai/gpt-4o-mini', options: {
-  context: [
-    "Python was created by Guido van Rossum.",
-    "The first version was released in 1991.",
-    "Python emphasizes code readability."
-  ]
-});
-
-const query = "What can you tell me about Python?";
-const response = "Python was created by Guido van Rossum and released in 1991. It is the most popular programming language today and is used by millions of developers worldwide.";
-
-const result = await scorer.run({
-  input: [{ role: 'user', content: query }],
-  output: { text: response },
-});
-
-console.log(result);
-```
-
-#### Mixed faithfulness output
-
-The score is lower because only a portion of the response is verifiable. While some claims match the context, others are unconfirmed or out of scope, reducing the overall faithfulness.
-
-```typescript
-{
-  score: 0.5,
-  reason: "The score is 0.5 because while two claims are supported by the context (Python was created by Guido van Rossum and Python was released in 1991), the other two claims regarding Python's popularity and usage cannot be verified as they are not mentioned in the context."
-}
-```
-
-### Low faithfulness example
-
-In this example, the response directly contradicts the context. None of the claims are supported, and several conflict with the facts provided.
-
-```typescript filename="src/example-low-faithfulness.ts" showLineNumbers copy
-import { createFaithfulnessScorer } from "@mastra/evals/scorers/llm";
-
-const scorer = createFaithfulnessScorer({ model: 'openai/gpt-4o-mini', options: {
-  context: [
-    "Mars is the fourth planet from the Sun.",
-    "It has a thin atmosphere of mostly carbon dioxide.",
-    "Two small moons orbit Mars: Phobos and Deimos."
-  ]
-});
-
-const query = "What do we know about Mars?";
-const response = "Mars is the third planet from the Sun. It has a thick atmosphere rich in oxygen and nitrogen, and is orbited by three large moons.";
-
-const result = await scorer.run({
-  input: [{ role: 'user', content: query }],
-  output: { text: response },
-});
-
-console.log(result);
-```
-
-#### Low faithfulness output
-
-Each claim is inaccurate or conflicts with the context, resulting in a score of 0.
-
-```typescript
-{
-  score: 0,
-  reason: "The score is 0 because all claims made in the output contradict the provided context. The output states that Mars is the third planet from the Sun, while the context clearly states it is the fourth. Additionally, it claims that Mars has a thick atmosphere rich in oxygen and nitrogen, contradicting the context's description of a thin atmosphere mostly composed of carbon dioxide. Finally, the output mentions that Mars is orbited by three large moons, while the context specifies that it has only two small moons, Phobos and Deimos. Therefore, there are no supported claims, leading to a score of 0."
-}
-```
-
-## Related
-
-- [Answer Relevancy Scorer](./answer-relevancy)
-- [Hallucination Scorer](./hallucination)

package/.docs/raw/reference/scorers/hallucination.mdx

@@ -1,247 +0,0 @@
----
-title: "Reference: Hallucination | Scorers | Mastra Docs"
-description: Documentation for the Hallucination Scorer in Mastra, which evaluates the factual correctness of LLM outputs by identifying contradictions with provided context.
----
-
-# Hallucination Scorer
-
-The `createHallucinationScorer()` function evaluates whether an LLM generates factually correct information by comparing its output against the provided context. This scorer measures hallucination by identifying direct contradictions between the context and the output.
-
-## Parameters
-
-The `createHallucinationScorer()` function accepts a single options object with the following properties:
-
-<PropertiesTable
-  content={[
-    {
-      name: "model",
-      type: "LanguageModel",
-      required: true,
-      description: "Configuration for the model used to evaluate hallucination.",
-    },
-    {
-      name: "scale",
-      type: "number",
-      required: false,
-      defaultValue: "1",
-      description: "Maximum score value.",
-    },
-  ]}
-/>
-
-This function returns an instance of the MastraScorer class. The `.run()` method accepts the same input as other scorers (see the [MastraScorer reference](./mastra-scorer)), but the return value includes LLM-specific fields as documented below.
-
-## .run() Returns
-
-<PropertiesTable
-  content={[
-    {
-      name: "runId",
-      type: "string",
-      description: "The id of the run (optional).",
-    },
-    {
-      name: "preprocessStepResult",
-      type: "object",
-      description: "Object with extracted claims: { claims: string[] }",
-    },
-    {
-      name: "preprocessPrompt",
-      type: "string",
-      description: "The prompt sent to the LLM for the preprocess step (optional).",
-    },
-    {
-      name: "analyzeStepResult",
-      type: "object",
-      description: "Object with verdicts: { verdicts: Array<{ statement: string, verdict: 'yes' | 'no', reason: string }> }",
-    },
-    {
-      name: "analyzePrompt",
-      type: "string",
-      description: "The prompt sent to the LLM for the analyze step (optional).",
-    },
-    {
-      name: "score",
-      type: "number",
-      description: "Hallucination score (0 to scale, default 0-1).",
-    },
-    {
-      name: "reason",
-      type: "string",
-      description: "Detailed explanation of the score and identified contradictions.",
-    },
-    {
-      name: "generateReasonPrompt",
-      type: "string",
-      description: "The prompt sent to the LLM for the generateReason step (optional).",
-    },
-  ]}
-/>
-
-## Scoring Details
-
-The scorer evaluates hallucination through contradiction detection and unsupported claim analysis.
-
-### Scoring Process
-
-1. Analyzes factual content:
-   - Extracts statements from context
-   - Identifies numerical values and dates
-   - Maps statement relationships
-2. Analyzes output for hallucinations:
-   - Compares against context statements
-   - Marks direct conflicts as hallucinations
-   - Identifies unsupported claims as hallucinations
-   - Evaluates numerical accuracy
-   - Considers approximation context
-3. Calculates hallucination score:
-   - Counts hallucinated statements (contradictions and unsupported claims)
-   - Divides by total statements
-   - Scales to configured range
-
-Final score: `(hallucinated_statements / total_statements) * scale`
-
-### Important Considerations
-
-- Claims not present in context are treated as hallucinations
-- Subjective claims are hallucinations unless explicitly supported
-- Speculative language ("might", "possibly") about facts IN context is allowed
-- Speculative language about facts NOT in context is treated as hallucination
-- Empty outputs result in zero hallucinations
-- Numerical evaluation considers:
-  - Scale-appropriate precision
-  - Contextual approximations
-  - Explicit precision indicators
-
-### Score interpretation
-
-A hallucination score between 0 and 1:
-
-- **0.0**: No hallucination — all claims match the context.
-- **0.3–0.4**: Low hallucination — a few contradictions.
-- **0.5–0.6**: Mixed hallucination — several contradictions.
-- **0.7–0.8**: High hallucination — many contradictions.
-- **0.9–1.0**: Complete hallucination — most or all claims contradict the context.
-
-**Note:** The score represents the degree of hallucination - lower scores indicate better factual alignment with the provided context
-
-## Examples
-
-### No hallucination example
-
-In this example, the response is fully aligned with the provided context. All claims are factually correct and directly supported by the source material, resulting in a low hallucination score.
-
-```typescript filename="src/example-no-hallucination.ts" showLineNumbers copy
-import { createHallucinationScorer } from "@mastra/evals/scorers/llm";
-
-const scorer = createHallucinationScorer({ model: 'openai/gpt-4o-mini', options: {
-  context: [
-    "The iPhone was first released in 2007.",
-    "Steve Jobs unveiled it at Macworld.",
-    "The original model had a 3.5-inch screen."
-  ]
-});
-
-const query = "When was the first iPhone released?";
-const response = "The iPhone was first released in 2007, when Steve Jobs unveiled it at Macworld. The original iPhone featured a 3.5-inch screen.";
-
-const result = await scorer.run({
-  input: [{ role: 'user', content: query }],
-  output: { text: response },
-});
-
-console.log(result);
-```
-
-#### No hallucination output
-
-The response receives a score of 0 because there are no contradictions. Every statement is consistent with the context, and no new or fabricated information has been introduced.
-
-```typescript
-{
-  score: 0,
-  reason: 'The score is 0 because none of the statements from the context were contradicted by the output.'
-}
-```
-
-### Mixed hallucination example
-
-In this example, the response includes both accurate and inaccurate claims. Some details align with the context, while others directly contradict it—such as inflated numbers or incorrect locations. These contradictions increase the hallucination score.
-
-```typescript filename="src/example-mixed-hallucination.ts" showLineNumbers copy
-import { createHallucinationScorer } from "@mastra/evals/scorers/llm";
-
-const scorer = createHallucinationScorer({ model: 'openai/gpt-4o-mini', options: {
-  context: [
-    "The first Star Wars movie was released in 1977.",
-    "It was directed by George Lucas.",
-    "The film earned $775 million worldwide.",
-    "The movie was filmed in Tunisia and England."
-  ]
-});
-
-const query = "Tell me about the first Star Wars movie.";
-const response = "The first Star Wars movie came out in 1977 and was directed by George Lucas. It made over $1 billion at the box office and was filmed entirely in California.";
-
-const result = await scorer.run({
-  input: [{ role: 'user', content: query }],
-  output: { text: response },
-});
-
-console.log(result);
-```
-
-#### Mixed hallucination output
-
-The Scorer assigns a mid-range score because parts of the response conflict with the context. While some facts are correct, others are inaccurate or fabricated, reducing overall reliability.
-
-```typescript
-{
-  score: 0.5,
-  reason: 'The score is 0.5 because two out of four statements from the output were contradicted by claims in the context, indicating a balance of accurate and inaccurate information.'
-}
-```
-
-### Complete hallucination example
-
-In this example, the response contradicts every key fact in the context. None of the claims can be verified, and all presented details are factually incorrect.
-
-```typescript filename="src/example-complete-hallucination.ts" showLineNumbers copy
-import { createHallucinationScorer } from "@mastra/evals/scorers/llm";
-
-const scorer = createHallucinationScorer({ model: 'openai/gpt-4o-mini', options: {
-  context: [
-    "The Wright brothers made their first flight in 1903.",
-    "The flight lasted 12 seconds.",
-    "It covered a distance of 120 feet."
-  ]
-});
-
-const query = "When did the Wright brothers first fly?";
-const response = "The Wright brothers achieved their historic first flight in 1908. The flight lasted about 2 minutes and covered nearly a mile.";
-
-const result = await scorer.run({
-  input: [{ role: 'user', content: query }],
-  output: { text: response },
-});
-
-console.log(result);
-
-```
-
-#### Complete hallucination output
-
-The Scorer assigns a score of 1 because every statement in the response conflicts with the context. The details are fabricated or inaccurate across the board.
-
-```typescript
-{
-  score: 1,
-  reason: 'The score is 1.0 because all three statements from the output directly contradict the context: the first flight was in 1903, not 1908; it lasted 12 seconds, not about 2 minutes; and it covered 120 feet, not nearly a mile.'
-}
-```
-
-
-## Related
-
-- [Faithfulness Scorer](./faithfulness)
-- [Answer Relevancy Scorer](./answer-relevancy)

package/.docs/raw/reference/scorers/keyword-coverage.mdx

@@ -1,224 +0,0 @@
----
-title: "Reference: Keyword Coverage | Scorers | Mastra Docs"
-description: Documentation for the Keyword Coverage Scorer in Mastra, which evaluates how well LLM outputs cover important keywords from the input.
----
-
-# Keyword Coverage Scorer
-
-The `createKeywordCoverageScorer()` function evaluates how well an LLM's output covers the important keywords from the input. It analyzes keyword presence and matches while ignoring common words and stop words.
-
-## Parameters
-
-The `createKeywordCoverageScorer()` function does not take any options.
-
-This function returns an instance of the MastraScorer class. See the [MastraScorer reference](./mastra-scorer) for details on the `.run()` method and its input/output.
-
-## .run() Returns
-
-<PropertiesTable
-  content={[
-    {
-      name: "runId",
-      type: "string",
-      description: "The id of the run (optional).",
-    },
-    {
-      name: "preprocessStepResult",
-      type: "object",
-      description: "Object with extracted keywords: { referenceKeywords: Set<string>, responseKeywords: Set<string> }",
-    },
-    {
-      name: "analyzeStepResult",
-      type: "object",
-      description: "Object with keyword coverage: { totalKeywords: number, matchedKeywords: number }",
-    },
-    {
-      name: "score",
-      type: "number",
-      description: "Coverage score (0-1) representing the proportion of matched keywords.",
-    },
-  ]}
-/>
-
-`.run()` returns a result in the following shape:
-
-```typescript
-{
-  runId: string,
-  extractStepResult: {
-    referenceKeywords: Set<string>,
-    responseKeywords: Set<string>
-  },
-  analyzeStepResult: {
-    totalKeywords: number,
-    matchedKeywords: number
-  },
-  score: number
-}
-```
-
-## Scoring Details
-
-The scorer evaluates keyword coverage by matching keywords with the following features:
-
-- Common word and stop word filtering (e.g., "the", "a", "and")
-- Case-insensitive matching
-- Word form variation handling
-- Special handling of technical terms and compound words
-
-### Scoring Process
-
-1. Processes keywords from input and output:
-   - Filters out common words and stop words
-   - Normalizes case and word forms
-   - Handles special terms and compounds
-2. Calculates keyword coverage:
-   - Matches keywords between texts
-   - Counts successful matches
-   - Computes coverage ratio
-
-Final score: `(matched_keywords / total_keywords) * scale`
-
-### Score interpretation
-
-A coverage score between 0 and 1:
-
-- **1.0**: Complete coverage – all keywords present.
-- **0.7–0.9**: High coverage – most keywords included.
-- **0.4–0.6**: Partial coverage – some keywords present.
-- **0.1–0.3**: Low coverage – few keywords matched.
-- **0.0**: No coverage – no keywords found.
-
-### Special Cases
-
-The scorer handles several special cases:
-
-- Empty input/output: Returns score of 1.0 if both empty, 0.0 if only one is empty
-- Single word: Treated as a single keyword
-- Technical terms: Preserves compound technical terms (e.g., "React.js", "machine learning")
-- Case differences: "JavaScript" matches "javascript"
-- Common words: Ignored in scoring to focus on meaningful keywords
-
-## Examples
-
-### Full coverage example
-
-In this example, the response fully reflects the key terms from the input. All required keywords are present, resulting in complete coverage with no omissions.
-
-```typescript filename="src/example-full-keyword-coverage.ts" showLineNumbers copy
-import { createKeywordCoverageScorer } from "@mastra/evals/scorers/code";
-
-const scorer = createKeywordCoverageScorer();
-
-const input = 'JavaScript frameworks like React and Vue';
-const output = 'Popular JavaScript frameworks include React and Vue for web development';
-
-const result = await scorer.run({
-  input: [{ role: 'user', content: input }],
-  output: { role: 'assistant', text: output },
-});
-
-console.log('Score:', result.score);
-console.log('AnalyzeStepResult:', result.analyzeStepResult);
-```
-
-#### Full coverage output
-
-A score of 1 indicates that all expected keywords were found in the response. The `analyzeStepResult` field confirms that the number of matched keywords equals the total number extracted from the input.
-
-```typescript
-{
-  score: 1,
-  analyzeStepResult: {
-    totalKeywords: 4,
-    matchedKeywords: 4
-  }
-}
-```
-
-### Partial coverage example
-
-In this example, the response includes some, but not all, of the important keywords from the input. The score reflects partial coverage, with key terms either missing or only partially matched.
-
-```typescript filename="src/example-partial-keyword-coverage.ts" showLineNumbers copy
-import { createKeywordCoverageScorer } from "@mastra/evals/scorers/code";
-
-const scorer = createKeywordCoverageScorer();
-
-const input = 'TypeScript offers interfaces, generics, and type inference';
-const output = 'TypeScript provides type inference and some advanced features';
-
-const result = await scorer.run({
-  input: [{ role: 'user', content: input }],
-  output: { role: 'assistant', text: output },
-});
-
-console.log('Score:', result.score);
-console.log('AnalyzeStepResult:', result.analyzeStepResult);
-```
-
-#### Partial coverage output
-
-A score of 0.5 indicates that only half of the expected keywords were found in the response. The `analyzeStepResult` field shows how many terms were matched compared to the total identified in the input.
-
-```typescript
-{
-  score: 0.5,
-  analyzeStepResult: {
-    totalKeywords: 6,
-    matchedKeywords: 3
-  }
-}
-```
-
-### Minimal coverage example
-
-In this example, the response includes very few of the important keywords from the input. The score reflects minimal coverage, with most key terms missing or unaccounted for.
-
-```typescript filename="src/example-minimal-keyword-coverage.ts" showLineNumbers copy
-import { createKeywordCoverageScorer } from "@mastra/evals/scorers/code";
-
-const scorer = createKeywordCoverageScorer();
-
-const input = 'Machine learning models require data preprocessing, feature engineering, and hyperparameter tuning';
-const output = 'Data preparation is important for models';
-
-const result = await scorer.run({
-  input: [{ role: 'user', content: input }],
-  output: { role: 'assistant', text: output },
-});
-
-console.log('Score:', result.score);
-console.log('AnalyzeStepResult:', result.analyzeStepResult);
-```
-
-#### Minimal coverage output
-
-A low score indicates that only a small number of the expected keywords were present in the response. The `analyzeStepResult` field highlights the gap between total and matched keywords, signaling insufficient coverage.
-
-```typescript
-{
-  score: 0.2,
-  analyzeStepResult: {
-    totalKeywords: 10,
-    matchedKeywords: 2
-  }
-}
-```
-
-### Metric configuration
-
-You can create a `KeywordCoverageMetric` instance with default settings. No additional configuration is required.
-
-```typescript
-const metric = new KeywordCoverageMetric();
-```
-
-> See [KeywordCoverageScorer](/reference/scorers/keyword-coverage.mdx) for a full list of configuration options.
-
-## Related
-
-- [Completeness Scorer](./completeness)
-- [Content Similarity Scorer](./content-similarity)
-- [Answer Relevancy Scorer](./answer-relevancy)
-- [Textual Difference Scorer](./textual-difference)