@mastra/mcp-docs-server 0.13.29 → 0.13.30-alpha.1

package/.docs/raw/reference/scorers/content-similarity.mdx

@@ -7,8 +7,6 @@ description: Documentation for the Content Similarity Scorer in Mastra, which me
 
 The `createContentSimilarityScorer()` function measures the textual similarity between two strings, providing a score that indicates how closely they match. It supports configurable options for case sensitivity and whitespace handling.
 
-For a usage example, see the [Content Similarity Examples](/examples/scorers/content-similarity).
-
 ## Parameters
 
 The `createContentSimilarityScorer()` function accepts a single options object with the following properties:
@@ -78,15 +76,116 @@ The scorer evaluates textual similarity through character-level matching and con
 
 Final score: `similarity_value * scale`
 
+## Examples
+
+### High similarity example
+
+In this example, the response closely resembles the query in both structure and meaning. Minor differences in tense and phrasing do not significantly affect the overall similarity.
+
+```typescript filename="src/example-high-similarity.ts" showLineNumbers copy
+import { createContentSimilarityScorer } from "@mastra/evals/scorers/llm";
+
+const scorer = createContentSimilarityScorer();
+
+const query = "The quick brown fox jumps over the lazy dog.";
+const response = "A quick brown fox jumped over a lazy dog.";
+
+const result = await scorer.run({
+  input: [{ role: 'user', content: query }],
+  output: { text: response },
+});
+
+console.log(result);
+```
+
+#### High similarity output
+
+The output receives a high score because the response preserves the intent and content of the query with only subtle wording changes.
+
+```typescript
+{
+  score: 0.7761194029850746,
+  analyzeStepResult: {
+    similarity: 0.7761194029850746
+  },
+}
+```
+
+### Moderate similarity example
+
+In this example, the response shares some conceptual overlap with the query but diverges in structure and wording. Key elements remain present, but the phrasing introduces moderate variation.
+
+```typescript filename="src/example-moderate-similarity.ts" showLineNumbers copy
+import { createContentSimilarityScorer } from "@mastra/evals/scorers/llm";
+
+const scorer = createContentSimilarityScorer();
+
+const query = "A brown fox quickly leaps across a sleeping dog.";
+const response = "The quick brown fox jumps over the lazy dog.";
+
+const result = await scorer.run({
+  input: [{ role: 'user', content: query }],
+  output: { text: response },
+});
+
+console.log(result);
+```
+
+#### Moderate similarity output
+
+The output receives a mid-range score because the response captures the general idea of the query, though it differs enough in wording to reduce overall similarity.
+
+```typescript
+{
+  score: 0.40540540540540543,
+  analyzeStepResult: {
+    similarity: 0.40540540540540543
+  }
+}
+```
+
+### Low similarity example
+
+In this example, the response and query are unrelated in meaning, despite having a similar grammatical structure. There is little to no shared content overlap.
+
+```typescript filename="src/example-low-similarity.ts" showLineNumbers copy
+import { createContentSimilarityScorer } from "@mastra/evals/scorers/llm";
+
+const scorer = createContentSimilarityScorer();
+
+const query = "The cat sleeps on the windowsill.";
+const response = "The quick brown fox jumps over the lazy dog.";
+
+const result = await scorer.run({
+  input: [{ role: 'user', content: query }],
+  output: { text: response },
+});
+
+console.log(result);
+```
+
+#### Low similarity output
+
+The output receives a low score because the response does not align with the content or intent of the query.
+
+```typescript
+{
+  score: 0.25806451612903225,
+  analyzeStepResult: {
+    similarity: 0.25806451612903225
+  },
+}
+```
+
 ### Score interpretation
 
-(0 to scale, default 0-1)
+A similarity score between 0 and 1:
 
-- 1.0: Perfect match - identical texts
-- 0.7-0.9: High similarity - mostly matching content
-- 0.4-0.6: Moderate similarity - partial matches
-- 0.1-0.3: Low similarity - few matching patterns
-- 0.0: No similarity - completely different texts
+- **1.0**: Perfect match – content is nearly identical.
+- **0.7–0.9**: High similarity – minor differences in word choice or structure.
+- **0.4–0.6**: Moderate similarity – general overlap with noticeable variation.
+- **0.1–0.3**: Low similarity – few common elements or shared meaning.
+- **0.0**: No similarity – completely different content.
 
 ## Related
 

package/.docs/raw/reference/scorers/context-precision.mdx

@@ -9,6 +9,22 @@ import { PropertiesTable } from "@/components/properties-table";
 
 The `createContextPrecisionScorer()` function creates a scorer that evaluates how relevant and well-positioned retrieved context pieces are for generating expected outputs. It uses **Mean Average Precision (MAP)** to reward systems that place relevant context earlier in the sequence.
 
+It is especially useful for these use cases:
+
+**RAG System Evaluation**
+
+Ideal for evaluating retrieved context in RAG pipelines where:
+- Context ordering matters for model performance
+- You need to measure retrieval quality beyond simple relevance
+- Early relevant context is more valuable than later relevant context
+
+**Context Window Optimization**
+
+Use when optimizing context selection for:
+- Limited context windows
+- Token budget constraints  
+- Multi-step reasoning tasks
+
 ## Parameters
 
 <PropertiesTable
@@ -48,9 +64,8 @@ The `createContextPrecisionScorer()` function creates a scorer that evaluates ho
   ]}
 />
 
-:::note
-Either `context` or `contextExtractor` must be provided. If both are provided, `contextExtractor` takes precedence.
-:::
+
+**Note**: Either `context` or `contextExtractor` must be provided. If both are provided, `contextExtractor` takes precedence.
 
 ## .run() Returns
 
@@ -93,10 +108,26 @@ Where:
 
 ### Score Interpretation
 
-- **1.0** = Perfect precision (all relevant context appears first)
-- **0.5-0.9** = Good precision with some relevant context well-positioned
-- **0.1-0.4** = Poor precision with relevant context buried or scattered
-- **0.0** = No relevant context found
+- **0.9-1.0**: Excellent precision - all relevant context early in sequence
+- **0.7-0.8**: Good precision - most relevant context well-positioned  
+- **0.4-0.6**: Moderate precision - relevant context mixed with irrelevant
+- **0.1-0.3**: Poor precision - little relevant context or poorly positioned
+- **0.0**: No relevant context found
+
+### Reason analysis
+
+The reason field explains:
+- Which context pieces were deemed relevant/irrelevant
+- How positioning affected the MAP calculation
+- Specific relevance criteria used in evaluation
+
+### Optimization insights
+
+Use results to:
+- **Improve retrieval**: Filter out irrelevant context before ranking
+- **Optimize ranking**: Ensure relevant context appears early
+- **Tune chunk size**: Balance context detail vs. relevance precision
+- **Evaluate embeddings**: Test different embedding models for better retrieval
 
 ### Example Calculation
 
@@ -109,19 +140,204 @@ Given context: `[relevant, irrelevant, relevant, irrelevant]`
 
 MAP = (1.0 + 0.67) / 2 = 0.835 ≈ **0.83**
 
-## Usage Patterns
+## Scorer configuration
 
-### RAG System Evaluation
-Ideal for evaluating retrieved context in RAG pipelines where:
-- Context ordering matters for model performance
-- You need to measure retrieval quality beyond simple relevance
-- Early relevant context is more valuable than later relevant context
+### Dynamic context extraction
 
-### Context Window Optimization
-Use when optimizing context selection for:
-- Limited context windows
-- Token budget constraints  
-- Multi-step reasoning tasks
+```typescript
+const scorer = createContextPrecisionScorer({
+  model: openai('gpt-4o-mini'),
+  options: {
+    contextExtractor: (input, output) => {
+      // Extract context dynamically based on the query
+      const query = input?.inputMessages?.[0]?.content || '';
+      
+      // Example: Retrieve from a vector database
+      const searchResults = vectorDB.search(query, { limit: 10 });
+      return searchResults.map(result => result.content);
+    },
+    scale: 1,
+  },
+});
+```
+
+### Large context evaluation
+
+```typescript
+const scorer = createContextPrecisionScorer({
+  model: openai('gpt-4o-mini'), 
+  options: {
+    context: [
+      // Simulate retrieved documents from vector database
+      'Document 1: Highly relevant content...',
+      'Document 2: Somewhat related content...',
+      'Document 3: Tangentially related...',
+      'Document 4: Not relevant...',
+      'Document 5: Highly relevant content...',
+      // ... up to dozens of context pieces
+    ],
+  },
+});
+```
+
+## Examples
+
+### High precision example
+
+This example shows perfect context precision where all relevant context appears early:
+
+```typescript
+import { openai } from '@ai-sdk/openai';
+import { createContextPrecisionScorer } from '@mastra/evals';
+
+const scorer = createContextPrecisionScorer({
+  model: openai('gpt-4o-mini'),
+  options: {
+    context: [
+      'Photosynthesis is the process by which plants convert sunlight, carbon dioxide, and water into glucose and oxygen.',
+      'The process occurs in the chloroplasts of plant cells, specifically in the thylakoids.',
+      'Light-dependent reactions happen in the thylakoid membranes, while the Calvin cycle occurs in the stroma.',
+    ],
+    scale: 1,
+  },
+});
+
+const result = await scorer.run({
+  input: {
+    inputMessages: [
+      {
+        id: '1',
+        role: 'user',
+        content: 'How does photosynthesis work in plants?',
+      },
+    ],
+  },
+  output: [
+    {
+      id: '2', 
+      role: 'assistant',
+      content: 'Photosynthesis is the process where plants convert sunlight, CO2, and water into glucose and oxygen using chloroplasts.',
+    },
+  ],
+});
+
+console.log(result);
+// Output:
+// {
+//   score: 1.0,
+//   reason: "The score is 1.0 because all context pieces are highly relevant to explaining photosynthesis and are optimally ordered to support the expected output."
+// }
+```
+
+### Mixed precision example
+
+This example shows moderate precision with both relevant and irrelevant context:
+
+```typescript
+import { openai } from '@ai-sdk/openai';
+import { createContextPrecisionScorer } from '@mastra/evals';
+
+const scorer = createContextPrecisionScorer({
+  model: openai('gpt-4o-mini'),
+  options: {
+    context: [
+      'Regular exercise improves cardiovascular health by strengthening the heart muscle.',
+      'A balanced diet should include fruits, vegetables, and whole grains.',
+      'Physical activity releases endorphins which improve mood and reduce stress.',
+      'The average person should drink 8 glasses of water per day.',
+      'Exercise also helps maintain healthy body weight and muscle mass.',
+    ],
+    scale: 1,
+  },
+});
+
+const result = await scorer.run({
+  input: {
+    inputMessages: [
+      {
+        id: '1',
+        role: 'user', 
+        content: 'What are the mental and physical benefits of exercise?',
+      },
+    ],
+  },
+  output: [
+    {
+      id: '2',
+      role: 'assistant', 
+      content: 'Exercise provides cardiovascular benefits, improves mood through endorphin release, and helps maintain healthy body composition.',
+    },
+  ],
+});
+
+console.log(result);
+// Output:
+// {
+//   score: 0.72,
+//   reason: "The score is 0.72 because contexts 1, 3, and 5 are relevant to exercise benefits, but irrelevant contexts about diet and hydration reduce the precision score."
+// }
+```
+
+### Low precision example  
+
+This example shows poor context precision with mostly irrelevant context:
+
+```typescript
+import { openai } from '@ai-sdk/openai';
+import { createContextPrecisionScorer } from '@mastra/evals';
+
+const scorer = createContextPrecisionScorer({
+  model: openai('gpt-4o-mini'),
+  options: {
+    context: [
+      'The weather forecast shows sunny skies this weekend.',
+      'Coffee is one of the world\'s most popular beverages.',
+      'Machine learning requires large amounts of training data.',
+      'Cats typically sleep 12-16 hours per day.',
+      'The capital of France is Paris.',
+    ],
+    scale: 1,
+  },
+});
+
+const result = await scorer.run({
+  input: {
+    inputMessages: [
+      {
+        id: '1',
+        role: 'user',
+        content: 'How does photosynthesis work?',
+      },
+    ],
+  },
+  output: [
+    {
+      id: '2',
+      role: 'assistant',
+      content: 'Photosynthesis is the process by which plants convert sunlight into energy using chlorophyll.',
+    },
+  ],
+});
+
+console.log(result);
+// Output:
+// {
+//   score: 0.0,
+//   reason: "The score is 0.0 because none of the retrieved context pieces are relevant to explaining photosynthesis."
+// }
+```
+
+## Comparison with Context Relevance
+
+Choose the right scorer for your needs:
+
+| Use Case | Context Relevance | Context Precision |
+|----------|-------------------|-------------------|
+| **RAG evaluation** | When usage matters | When ranking matters |
+| **Context quality** | Nuanced levels | Binary relevance |
+| **Missing detection** | ✓ Identifies gaps | ✗ Not evaluated |
+| **Usage tracking** | ✓ Tracks utilization | ✗ Not considered |
+| **Position sensitivity** | ✗ Position agnostic | ✓ Rewards early placement |
 
 ## Related