@mastra/mcp-docs-server 0.13.30-alpha.0 → 0.13.30-alpha.1

package/.docs/raw/reference/scorers/context-relevance.mdx

@@ -9,6 +9,22 @@ import { PropertiesTable } from "@/components/properties-table";
 
 The `createContextRelevanceScorerLLM()` function creates a scorer that evaluates how relevant and useful provided context was for generating agent responses. It uses weighted relevance levels and applies penalties for unused high-relevance context and missing information.
 
+It is especially useful for these use cases:
+
+**Content Generation Evaluation**
+
+Best for evaluating context quality in:
+- Chat systems where context usage matters
+- RAG pipelines needing nuanced relevance assessment
+- Systems where missing context affects quality
+
+**Context Selection Optimization**
+
+Use when optimizing for:
+- Comprehensive context coverage
+- Effective context utilization
+- Identifying context gaps
+
 ## Parameters
 
 <PropertiesTable
@@ -74,9 +90,7 @@ The `createContextRelevanceScorerLLM()` function creates a scorer that evaluates
   ]}
 />
 
-:::note
-Either `context` or `contextExtractor` must be provided. If both are provided, `contextExtractor` takes precedence.
-:::
+Note: Either `context` or `contextExtractor` must be provided. If both are provided, `contextExtractor` takes precedence.
 
 ## .run() Returns
 
@@ -129,12 +143,30 @@ Final Score = max(0, Base Score - Usage Penalty - Missing Penalty) × scale
 - `maxMissingContextPenalty` = 0.5 (maximum 50% penalty for missing context)
 - `scale` = 1
 
-### Score Interpretation
+### Score interpretation
+
+- **0.9-1.0**: Excellent - all context highly relevant and used
+- **0.7-0.8**: Good - mostly relevant with minor gaps
+- **0.4-0.6**: Mixed - significant irrelevant or unused context
+- **0.2-0.3**: Poor - mostly irrelevant context
+- **0.0-0.1**: Very poor - no relevant context found
+
+### Reason analysis
+
+The reason field provides insights on:
+- Relevance level of each context piece (high/medium/low/none)
+- Which context was actually used in the response
+- Penalties applied for unused high-relevance context (configurable via `unusedHighRelevanceContext`)
+- Missing context that would have improved the response (penalized via `missingContextPerItem` up to `maxMissingContextPenalty`)
 
-- **0.9-1.0** = Excellent relevance with minimal gaps
-- **0.7-0.8** = Good relevance with some unused or missing context
-- **0.4-0.6** = Mixed relevance with significant gaps
-- **0.0-0.3** = Poor relevance or mostly irrelevant context
+### Optimization strategies
+
+Use results to improve your system:
+- **Filter irrelevant context**: Remove low/none relevance pieces before processing
+- **Ensure context usage**: Make sure high-relevance context is incorporated
+- **Fill context gaps**: Add missing information identified by the scorer
+- **Balance context size**: Find optimal amount of context for best relevance
+- **Tune penalty sensitivity**: Adjust `unusedHighRelevanceContext`, `missingContextPerItem`, and `maxMissingContextPenalty` based on your application's tolerance for unused or missing context
 
 ### Difference from Context Precision
 
@@ -146,35 +178,76 @@ Final Score = max(0, Base Score - Usage Penalty - Missing Penalty) × scale
 | **Usage** | Tracks and penalizes unused context | Not considered |
 | **Missing** | Identifies and penalizes gaps | Not evaluated |
 
-## Usage Examples
+## Scorer configuration
+
+### Custom penalty configuration
 
-### Basic Configuration
+Control how penalties are applied for unused and missing context:
 
 ```typescript
-const scorer = createContextRelevanceScorerLLM({
-  model: openai('gpt-4o'),
+import { openai } from '@ai-sdk/openai';
+import { createContextRelevanceScorerLLM } from '@mastra/evals';
+
+// Stricter penalty configuration
+const strictScorer = createContextRelevanceScorerLLM({
+  model: openai('gpt-4o-mini'),
   options: {
-    context: ['Einstein won the Nobel Prize for his work on the photoelectric effect'],
+    context: [
+      'Einstein won the Nobel Prize for photoelectric effect',
+      'He developed the theory of relativity',
+      'Einstein was born in Germany',
+    ],
+    penalties: {
+      unusedHighRelevanceContext: 0.2, // 20% penalty per unused high-relevance context
+      missingContextPerItem: 0.25, // 25% penalty per missing context item
+      maxMissingContextPenalty: 0.6, // Maximum 60% penalty for missing context
+    },
     scale: 1,
   },
 });
-```
 
-### Custom Penalty Configuration
-
-```typescript
-const scorer = createContextRelevanceScorerLLM({
-  model: openai('gpt-4o'),
+// Lenient penalty configuration
+const lenientScorer = createContextRelevanceScorerLLM({
+  model: openai('gpt-4o-mini'),
   options: {
-    context: ['Context information...'],
+    context: [
+      'Einstein won the Nobel Prize for photoelectric effect',
+      'He developed the theory of relativity',
+      'Einstein was born in Germany',
+    ],
     penalties: {
-      unusedHighRelevanceContext: 0.05, // Lower penalty for unused context
-      missingContextPerItem: 0.2, // Higher penalty per missing item
-      maxMissingContextPenalty: 0.4, // Lower maximum penalty cap
+      unusedHighRelevanceContext: 0.05, // 5% penalty per unused high-relevance context
+      missingContextPerItem: 0.1, // 10% penalty per missing context item
+      maxMissingContextPenalty: 0.3, // Maximum 30% penalty for missing context
     },
-    scale: 2, // Double the final score
+    scale: 1,
   },
 });
+
+const testRun = {
+  input: {
+    inputMessages: [
+      {
+        id: '1',
+        role: 'user',
+        content: 'What did Einstein achieve in physics?',
+      },
+    ],
+  },
+  output: [
+    {
+      id: '2',
+      role: 'assistant',
+      content: 'Einstein won the Nobel Prize for his work on the photoelectric effect.',
+    },
+  ],
+};
+
+const strictResult = await strictScorer.run(testRun);
+const lenientResult = await lenientScorer.run(testRun);
+
+console.log('Strict penalties:', strictResult.score); // Lower score due to unused context
+console.log('Lenient penalties:', lenientResult.score); // Higher score, less penalty
 ```
 
 ### Dynamic Context Extraction
@@ -201,19 +274,329 @@ const scorer = createContextRelevanceScorerLLM({
 });
 ```
 
-## Usage Patterns
+### Custom scale factor
 
-### Content Generation Evaluation
-Best for evaluating context quality in:
-- Chat systems where context usage matters
-- RAG pipelines needing nuanced relevance assessment
-- Systems where missing context affects quality
+```typescript
+const scorer = createContextRelevanceScorerLLM({
+  model: openai('gpt-4o-mini'),
+  options: {
+    context: [
+      'Relevant information...',
+      'Supporting details...',
+    ],
+    scale: 100, // Scale scores from 0-100 instead of 0-1
+  },
+});
 
-### Context Selection Optimization
-Use when optimizing for:
-- Comprehensive context coverage
-- Effective context utilization
-- Identifying context gaps
+// Result will be scaled: score: 85 instead of 0.85
+```
+
+### Combining multiple context sources
+
+```typescript
+const scorer = createContextRelevanceScorerLLM({
+  model: openai('gpt-4o-mini'),
+  options: {
+    contextExtractor: (input, output) => {
+      const query = input?.inputMessages?.[0]?.content || '';
+      
+      // Combine from multiple sources
+      const kbContext = knowledgeBase.search(query);
+      const docContext = documentStore.retrieve(query);
+      const cacheContext = contextCache.get(query);
+      
+      return [
+        ...kbContext,
+        ...docContext,
+        ...cacheContext,
+      ];
+    },
+    scale: 1,
+  },
+});
+```
+
+## Examples
+
+### High relevance example
+
+This example shows excellent context relevance where all context directly supports the response:
+
+```typescript
+import { openai } from '@ai-sdk/openai';
+import { createContextRelevanceScorerLLM } from '@mastra/evals';
+
+const scorer = createContextRelevanceScorerLLM({
+  model: openai('gpt-4o-mini'),
+  options: {
+    context: [
+      'Einstein won the Nobel Prize for his discovery of the photoelectric effect in 1921.',
+      'He published his theory of special relativity in 1905.',
+      'His general relativity theory, published in 1915, revolutionized our understanding of gravity.',
+    ],
+    scale: 1,
+  },
+});
+
+const result = await scorer.run({
+  input: {
+    inputMessages: [
+      {
+        id: '1',
+        role: 'user',
+        content: 'What were Einstein\'s major scientific achievements?',
+      },
+    ],
+  },
+  output: [
+    {
+      id: '2',
+      role: 'assistant',
+      content: 'Einstein\'s major achievements include the Nobel Prize for the photoelectric effect, special relativity in 1905, and general relativity in 1915.',
+    },
+  ],
+});
+
+console.log(result);
+// Output:
+// {
+//   score: 1.0,
+//   reason: "The score is 1.0 because all context pieces are highly relevant to Einstein's achievements and were effectively used in generating the comprehensive response."
+// }
+```
+
+### Mixed relevance example
+
+This example shows moderate relevance with some context being irrelevant or unused:
+
+```typescript
+import { openai } from '@ai-sdk/openai';
+import { createContextRelevanceScorerLLM } from '@mastra/evals';
+
+const scorer = createContextRelevanceScorerLLM({
+  model: openai('gpt-4o-mini'),
+  options: {
+    context: [
+      'Solar eclipses occur when the Moon blocks the Sun.',
+      'The Moon moves between the Earth and Sun during eclipses.',
+      'The Moon is visible at night.',
+      'Stars twinkle due to atmospheric interference.',
+      'Total eclipses can last up to 7.5 minutes.',
+    ],
+    scale: 1,
+  },
+});
+
+const result = await scorer.run({
+  input: {
+    inputMessages: [
+      {
+        id: '1',
+        role: 'user',
+        content: 'What causes solar eclipses?',
+      },
+    ],
+  },
+  output: [
+    {
+      id: '2',
+      role: 'assistant',
+      content: 'Solar eclipses happen when the Moon moves between Earth and the Sun, blocking sunlight.',
+    },
+  ],
+});
+
+console.log(result);
+// Output with default penalties:
+// {
+//   score: 0.64,
+//   reason: "The score is 0.64 because contexts 1 and 2 are highly relevant and used, context 5 is relevant but unused (10% penalty), while contexts 3 and 4 are irrelevant."
+// }
+
+// With custom penalty configuration
+const customScorer = createContextRelevanceScorerLLM({
+  model: openai('gpt-4o-mini'),
+  options: {
+    context: [
+      'Solar eclipses occur when the Moon blocks the Sun.',
+      'The Moon moves between the Earth and Sun during eclipses.',
+      'The Moon is visible at night.',
+      'Stars twinkle due to atmospheric interference.',
+      'Total eclipses can last up to 7.5 minutes.',
+    ],
+    penalties: {
+      unusedHighRelevanceContext: 0.05, // Lower penalty for unused context
+      missingContextPerItem: 0.1,
+      maxMissingContextPenalty: 0.3,
+    },
+  },
+});
+
+const customResult = await customScorer.run({
+  input: { inputMessages: [{ id: '1', role: 'user', content: 'What causes solar eclipses?' }] },
+  output: [{ id: '2', role: 'assistant', content: 'Solar eclipses happen when the Moon moves between Earth and the Sun, blocking sunlight.' }],
+});
+
+console.log(customResult);
+// Output with lenient penalties:
+// {
+//   score: 0.69, // Higher score due to reduced penalty for unused context
+//   reason: "The score is 0.69 because contexts 1 and 2 are highly relevant and used, context 5 is relevant but unused (5% penalty), while contexts 3 and 4 are irrelevant."
+// }
+```
+
+### Low relevance example
+
+This example shows poor context relevance with mostly irrelevant information:
+
+```typescript
+import { openai } from '@ai-sdk/openai';
+import { createContextRelevanceScorerLLM } from '@mastra/evals';
+
+const scorer = createContextRelevanceScorerLLM({
+  model: openai('gpt-4o-mini'),
+  options: {
+    context: [
+      'The Great Barrier Reef is located in Australia.',
+      'Coral reefs need warm water to survive.',
+      'Many fish species live in coral reefs.',
+      'Australia has six states and two territories.',
+      'The capital of Australia is Canberra.',
+    ],
+    scale: 1,
+  },
+});
+
+const result = await scorer.run({
+  input: {
+    inputMessages: [
+      {
+        id: '1',
+        role: 'user',
+        content: 'What is the capital of Australia?',
+      },
+    ],
+  },
+  output: [
+    {
+      id: '2',
+      role: 'assistant',
+      content: 'The capital of Australia is Canberra.',
+    },
+  ],
+});
+
+console.log(result);
+// Output:
+// {
+//   score: 0.26,
+//   reason: "The score is 0.26 because only context 5 is relevant to the query about Australia's capital, while the other contexts about reefs are completely irrelevant."
+// }
+```
+
+### Dynamic context extraction
+
+Extract context dynamically based on the run input:
+
+```typescript
+import { openai } from '@ai-sdk/openai';
+import { createContextRelevanceScorerLLM } from '@mastra/evals';
+
+const scorer = createContextRelevanceScorerLLM({
+  model: openai('gpt-4o-mini'),
+  options: {
+    contextExtractor: (input, output) => {
+      // Extract query from input
+      const query = input?.inputMessages?.[0]?.content || '';
+      
+      // Dynamically retrieve context based on query
+      if (query.toLowerCase().includes('einstein')) {
+        return [
+          'Einstein developed E=mc²',
+          'He won the Nobel Prize in 1921',
+          'His theories revolutionized physics',
+        ];
+      }
+      
+      if (query.toLowerCase().includes('climate')) {
+        return [
+          'Global temperatures are rising',
+          'CO2 levels affect climate',
+          'Renewable energy reduces emissions',
+        ];
+      }
+      
+      return ['General knowledge base entry'];
+    },
+    penalties: {
+      unusedHighRelevanceContext: 0.15, // 15% penalty for unused relevant context
+      missingContextPerItem: 0.2, // 20% penalty per missing context item
+      maxMissingContextPenalty: 0.4, // Cap at 40% total missing context penalty
+    },
+    scale: 1,
+  },
+});
+```
+
+### RAG system integration
+
+Integrate with RAG pipelines to evaluate retrieved context:
+
+```typescript
+import { openai } from '@ai-sdk/openai';
+import { createContextRelevanceScorerLLM } from '@mastra/evals';
+
+const scorer = createContextRelevanceScorerLLM({
+  model: openai('gpt-4o-mini'),
+  options: {
+    contextExtractor: (input, output) => {
+      // Extract from RAG retrieval results
+      const ragResults = input.metadata?.ragResults || [];
+      
+      // Return the text content of retrieved documents
+      return ragResults
+        .filter(doc => doc.relevanceScore > 0.5)
+        .map(doc => doc.content);
+    },
+    penalties: {
+      unusedHighRelevanceContext: 0.12, // Moderate penalty for unused RAG context
+      missingContextPerItem: 0.18, // Higher penalty for missing information in RAG
+      maxMissingContextPenalty: 0.45, // Slightly higher cap for RAG systems
+    },
+    scale: 1,
+  },
+});
+
+// Evaluate RAG system performance
+const evaluateRAG = async (testCases) => {
+  const results = [];
+  
+  for (const testCase of testCases) {
+    const score = await scorer.run(testCase);
+    results.push({
+      query: testCase.input.inputMessages[0].content,
+      relevanceScore: score.score,
+      feedback: score.reason,
+      unusedContext: score.reason.includes('unused'),
+      missingContext: score.reason.includes('missing'),
+    });
+  }
+  
+  return results;
+};
+```
+
+## Comparison with Context Precision
+
+Choose the right scorer for your needs:
+
+| Use Case | Context Relevance | Context Precision |
+|----------|-------------------|-------------------|
+| **RAG evaluation** | When usage matters | When ranking matters |
+| **Context quality** | Nuanced levels | Binary relevance |
+| **Missing detection** | ✓ Identifies gaps | ✗ Not evaluated |
+| **Usage tracking** | ✓ Tracks utilization | ✗ Not considered |
+| **Position sensitivity** | ✗ Position agnostic | ✓ Rewards early placement |
 
 ## Related
 

package/.docs/raw/reference/scorers/faithfulness.mdx

@@ -7,8 +7,6 @@ description: Documentation for the Faithfulness Scorer in Mastra, which evaluate
 
 The `createFaithfulnessScorer()` function evaluates how factually accurate an LLM's output is compared to the provided context. It extracts claims from the output and verifies them against the context, making it essential to measure RAG pipeline responses' reliability.
 
-For a usage example, see the [Faithfulness Examples](/examples/scorers/faithfulness).
-
 ## Parameters
 
 The `createFaithfulnessScorer()` function accepts a single options object with the following properties:
@@ -108,13 +106,129 @@ Final score: `(supported_claims / total_claims) * scale`
 
 ### Score interpretation
 
-(0 to scale, default 0-1)
+A faithfulness score between 0 and 1:
+
+- **1.0**: All claims are accurate and directly supported by the context.
+- **0.7–0.9**: Most claims are correct, with minor additions or omissions.
+- **0.4–0.6**: Some claims are supported, but others are unverifiable.
+- **0.1–0.3**: Most of the content is inaccurate or unsupported.
+- **0.0**: All claims are false or contradict the context.
+
+## Examples
+
+### High faithfulness example
+
+In this example, the response closely aligns with the context. Each statement in the output is verifiable and supported by the provided context entries, resulting in a high score.
+
+```typescript filename="src/example-high-faithfulness.ts" showLineNumbers copy
+import { openai } from "@ai-sdk/openai";
+import { createFaithfulnessScorer } from "@mastra/evals/scorers/llm";
+
+const scorer = createFaithfulnessScorer({ model: openai("gpt-4o-mini"), options: {
+  context: [
+    "The Tesla Model 3 was launched in 2017.",
+    "It has a range of up to 358 miles.",
+    "The base model accelerates 0-60 mph in 5.8 seconds."
+  ]
+});
+
+const query = "Tell me about the Tesla Model 3.";
+const response = "The Tesla Model 3 was introduced in 2017. It can travel up to 358 miles on a single charge and the base version goes from 0 to 60 mph in 5.8 seconds.";
+
+const result = await scorer.run({
+  input: [{ role: 'user', content: query }],
+  output: { text: response },
+});
+
+console.log(result);
+```
+
+#### High faithfulness output
+
+The output receives a score of 1 because all the information it provides can be directly traced to the context. There are no missing or contradictory facts.
+
+```typescript
+{
+  score: 1,
+  reason: 'The score is 1 because all claims made in the output are supported by the provided context.'
+}
+```
+
+### Mixed faithfulness example
+
+In this example, there are a mix of supported and unsupported claims. Some parts of the response are backed by the context, while others introduce new information not found in the source material.
+
+```typescript filename="src/example-mixed-faithfulness.ts" showLineNumbers copy
+import { openai } from "@ai-sdk/openai";
+import { createFaithfulnessScorer } from "@mastra/evals/scorers/llm";
+
+const scorer = createFaithfulnessScorer({ model: openai("gpt-4o-mini"), options: {
+  context: [
+    "Python was created by Guido van Rossum.",
+    "The first version was released in 1991.",
+    "Python emphasizes code readability."
+  ]
+});
+
+const query = "What can you tell me about Python?";
+const response = "Python was created by Guido van Rossum and released in 1991. It is the most popular programming language today and is used by millions of developers worldwide.";
+
+const result = await scorer.run({
+  input: [{ role: 'user', content: query }],
+  output: { text: response },
+});
+
+console.log(result);
+```
+
+#### Mixed faithfulness output
+
+The score is lower because only a portion of the response is verifiable. While some claims match the context, others are unconfirmed or out of scope, reducing the overall faithfulness.
+
+```typescript
+{
+  score: 0.5,
+  reason: "The score is 0.5 because while two claims are supported by the context (Python was created by Guido van Rossum and Python was released in 1991), the other two claims regarding Python's popularity and usage cannot be verified as they are not mentioned in the context."
+}
+```
+
+### Low faithfulness example
+
+In this example, the response directly contradicts the context. None of the claims are supported, and several conflict with the facts provided.
+
+```typescript filename="src/example-low-faithfulness.ts" showLineNumbers copy
+import { openai } from "@ai-sdk/openai";
+import { createFaithfulnessScorer } from "@mastra/evals/scorers/llm";
+
+const scorer = createFaithfulnessScorer({ model: openai("gpt-4o-mini"), options: {
+  context: [
+    "Mars is the fourth planet from the Sun.",
+    "It has a thin atmosphere of mostly carbon dioxide.",
+    "Two small moons orbit Mars: Phobos and Deimos."
+  ]
+});
+
+const query = "What do we know about Mars?";
+const response = "Mars is the third planet from the Sun. It has a thick atmosphere rich in oxygen and nitrogen, and is orbited by three large moons.";
+
+const result = await scorer.run({
+  input: [{ role: 'user', content: query }],
+  output: { text: response },
+});
+
+console.log(result);
+```
+
+#### Low faithfulness output
+
+Each claim is inaccurate or conflicts with the context, resulting in a score of 0.
 
-- 1.0: All claims supported by context
-- 0.7-0.9: Most claims supported, few unverifiable
-- 0.4-0.6: Mixed support with some contradictions
-- 0.1-0.3: Limited support, many contradictions
-- 0.0: No supported claims
+```typescript
+{
+  score: 0,
+  reason: "The score is 0 because all claims made in the output contradict the provided context. The output states that Mars is the third planet from the Sun, while the context clearly states it is the fourth. Additionally, it claims that Mars has a thick atmosphere rich in oxygen and nitrogen, contradicting the context's description of a thin atmosphere mostly composed of carbon dioxide. Finally, the output mentions that Mars is orbited by three large moons, while the context specifies that it has only two small moons, Phobos and Deimos. Therefore, there are no supported claims, leading to a score of 0."
+}
+```
 
 ## Related