@mastra/mcp-docs-server 1.1.5 → 1.1.6

package/.docs/reference/evals/context-precision.md

@@ -0,0 +1,196 @@
+# Context Precision Scorer
+
+The `createContextPrecisionScorer()` function creates a scorer that evaluates how relevant and well-positioned retrieved context pieces are for generating expected outputs. It uses **Mean Average Precision (MAP)** to reward systems that place relevant context earlier in the sequence.
+
+It is especially useful for these use cases:
+
+## RAG System Evaluation
+
+Ideal for evaluating retrieved context in RAG pipelines where:
+
+- Context ordering matters for model performance
+- You need to measure retrieval quality beyond simple relevance
+- Early relevant context is more valuable than later relevant context
+
+## Context Window Optimization
+
+Use when optimizing context selection for:
+
+- Limited context windows
+- Token budget constraints
+- Multi-step reasoning tasks
+
+## Parameters
+
+**model:** (`MastraModelConfig`): The language model to use for evaluating context relevance
+
+**options:** (`ContextPrecisionMetricOptions`): Configuration options for the scorer
+
+**Note**: Either `context` or `contextExtractor` must be provided. If both are provided, `contextExtractor` takes precedence.
+
+## .run() Returns
+
+**score:** (`number`): Mean Average Precision score between 0 and scale (default 0-1)
+
+**reason:** (`string`): Human-readable explanation of the context precision evaluation
+
+## Scoring Details
+
+### Mean Average Precision (MAP)
+
+Context Precision uses **Mean Average Precision** to evaluate both relevance and positioning:
+
+1. **Context Evaluation**: Each context piece is classified as relevant or irrelevant for generating the expected output
+2. **Precision Calculation**: For each relevant context at position `i`, precision = `relevant_items_so_far / (i + 1)`
+3. **Average Precision**: Sum all precision values and divide by total relevant items
+4. **Final Score**: Multiply by scale factor and round to 2 decimals
+
+### Scoring Formula
+
+```text
+MAP = (Σ Precision@k) / R
+
+Where:
+- Precision@k = (relevant items in positions 1...k) / k
+- R = total number of relevant items
+- Only calculated at positions where relevant items appear
+```
+
+### Score Interpretation
+
+- **0.9-1.0**: Excellent precision - all relevant context early in sequence
+- **0.7-0.8**: Good precision - most relevant context well-positioned
+- **0.4-0.6**: Moderate precision - relevant context mixed with irrelevant
+- **0.1-0.3**: Poor precision - little relevant context or poorly positioned
+- **0.0**: No relevant context found
+
+### Reason analysis
+
+The reason field explains:
+
+- Which context pieces were deemed relevant/irrelevant
+- How positioning affected the MAP calculation
+- Specific relevance criteria used in evaluation
+
+### Optimization insights
+
+Use results to:
+
+- **Improve retrieval**: Filter out irrelevant context before ranking
+- **Optimize ranking**: Ensure relevant context appears early
+- **Tune chunk size**: Balance context detail vs. relevance precision
+- **Evaluate embeddings**: Test different embedding models for better retrieval
+
+### Example Calculation
+
+Given context: `[relevant, irrelevant, relevant, irrelevant]`
+
+- Position 0: Relevant → Precision = 1/1 = 1.0
+- Position 1: Skip (irrelevant)
+- Position 2: Relevant → Precision = 2/3 = 0.67
+- Position 3: Skip (irrelevant)
+
+MAP = (1.0 + 0.67) / 2 = 0.835 ≈ **0.83**
+
+## Scorer configuration
+
+### Dynamic context extraction
+
+```typescript
+const scorer = createContextPrecisionScorer({
+  model: 'openai/gpt-5.1',
+  options: {
+    contextExtractor: (input, output) => {
+      // Extract context dynamically based on the query
+      const query = input?.inputMessages?.[0]?.content || ''
+
+      // Example: Retrieve from a vector database
+      const searchResults = vectorDB.search(query, { limit: 10 })
+      return searchResults.map(result => result.content)
+    },
+    scale: 1,
+  },
+})
+```
+
+### Large context evaluation
+
+```typescript
+const scorer = createContextPrecisionScorer({
+  model: 'openai/gpt-5.1',
+  options: {
+    context: [
+      // Simulate retrieved documents from vector database
+      'Document 1: Highly relevant content...',
+      'Document 2: Somewhat related content...',
+      'Document 3: Tangentially related...',
+      'Document 4: Not relevant...',
+      'Document 5: Highly relevant content...',
+      // ... up to dozens of context pieces
+    ],
+  },
+})
+```
+
+## Example
+
+Evaluate RAG system context retrieval precision for different queries:
+
+```typescript
+import { runEvals } from '@mastra/core/evals'
+import { createContextPrecisionScorer } from '@mastra/evals/scorers/prebuilt'
+import { myAgent } from './agent'
+
+const scorer = createContextPrecisionScorer({
+  model: 'openai/gpt-4o',
+  options: {
+    contextExtractor: (input, output) => {
+      // Extract context from agent's retrieved documents
+      return output.metadata?.retrievedContext || []
+    },
+  },
+})
+
+const result = await runEvals({
+  data: [
+    {
+      input: 'How does photosynthesis work in plants?',
+    },
+    {
+      input: 'What are the mental and physical benefits of exercise?',
+    },
+  ],
+  scorers: [scorer],
+  target: myAgent,
+  onItemComplete: ({ scorerResults }) => {
+    console.log({
+      score: scorerResults[scorer.id].score,
+      reason: scorerResults[scorer.id].reason,
+    })
+  },
+})
+
+console.log(result.scores)
+```
+
+For more details on `runEvals`, see the [runEvals reference](https://mastra.ai/reference/evals/run-evals).
+
+To add this scorer to an agent, see the [Scorers overview](https://mastra.ai/docs/evals/overview) guide.
+
+## Comparison with Context Relevance
+
+Choose the right scorer for your needs:
+
+| Use Case                 | Context Relevance    | Context Precision         |
+| ------------------------ | -------------------- | ------------------------- |
+| **RAG evaluation**       | When usage matters   | When ranking matters      |
+| **Context quality**      | Nuanced levels       | Binary relevance          |
+| **Missing detection**    | ✓ Identifies gaps    | ✗ Not evaluated           |
+| **Usage tracking**       | ✓ Tracks utilization | ✗ Not considered          |
+| **Position sensitivity** | ✗ Position agnostic  | ✓ Rewards early placement |
+
+## Related
+
+- [Answer Relevancy Scorer](https://mastra.ai/reference/evals/answer-relevancy) - Evaluates if answers address the question
+- [Faithfulness Scorer](https://mastra.ai/reference/evals/faithfulness) - Measures answer groundedness in context
+- [Custom Scorers](https://mastra.ai/docs/evals/custom-scorers) - Creating your own evaluation metrics