@mastra/mcp-docs-server 0.13.39 → 1.0.0-beta.1

package/.docs/raw/reference/evals/context-precision.mdx

@@ -1,195 +1,352 @@
 ---
-title: "Reference: ContextPrecisionMetric | Evals | Mastra Docs"
-description: Documentation for the Context Precision Metric in Mastra, which evaluates the relevance and precision of retrieved context nodes for generating expected outputs.
+title: "Reference: Context Precision Scorer | Evals | Mastra Docs"
+description: Documentation for the Context Precision Scorer in Mastra. Evaluates the relevance and precision of retrieved context for generating expected outputs using Mean Average Precision.
 ---
 
-# ContextPrecisionMetric
+import PropertiesTable from "@site/src/components/PropertiesTable";
 
-:::info Scorers
-This documentation refers to the legacy evals API. For the latest scorer features, see [Scorers](/docs/scorers/overview).
-:::
+# Context Precision Scorer
 
-The `ContextPrecisionMetric` class evaluates how relevant and precise the retrieved context nodes are for generating the expected output. It uses a judge-based system to analyze each context piece's contribution and provides weighted scoring based on position.
+The `createContextPrecisionScorer()` function creates a scorer that evaluates how relevant and well-positioned retrieved context pieces are for generating expected outputs. It uses **Mean Average Precision (MAP)** to reward systems that place relevant context earlier in the sequence.
 
-## Basic Usage
+It is especially useful for these use cases:
 
-```typescript
-import { openai } from "@ai-sdk/openai";
-import { ContextPrecisionMetric } from "@mastra/evals/llm";
+**RAG System Evaluation**
 
-// Configure the model for evaluation
-const model = openai("gpt-4o-mini");
+Ideal for evaluating retrieved context in RAG pipelines where:
 
-const metric = new ContextPrecisionMetric(model, {
-  context: [
-    "Photosynthesis is a biological process used by plants to create energy from sunlight.",
-    "Plants need water and nutrients from the soil to grow.",
-    "The process of photosynthesis produces oxygen as a byproduct.",
-  ],
-});
+- Context ordering matters for model performance
+- You need to measure retrieval quality beyond simple relevance
+- Early relevant context is more valuable than later relevant context
 
-const result = await metric.measure(
-  "What is photosynthesis?",
-  "Photosynthesis is the process by which plants convert sunlight into energy.",
-);
+**Context Window Optimization**
 
-console.log(result.score); // Precision score from 0-1
-console.log(result.info.reason); // Explanation of the score
-```
+Use when optimizing context selection for:
 
-## Constructor Parameters
+- Limited context windows
+- Token budget constraints
+- Multi-step reasoning tasks
+
+## Parameters
 
 <PropertiesTable
   content={[
     {
       name: "model",
-      type: "LanguageModel",
-      description:
-        "Configuration for the model used to evaluate context relevance",
-      isOptional: false,
+      type: "MastraModelConfig",
+      description: "The language model to use for evaluating context relevance",
+      required: true,
     },
     {
       name: "options",
       type: "ContextPrecisionMetricOptions",
-      description: "Configuration options for the metric",
-      isOptional: false,
+      description: "Configuration options for the scorer",
+      required: true,
+      children: [
+        {
+          name: "context",
+          type: "string[]",
+          description: "Array of context pieces to evaluate for relevance",
+          required: false,
+        },
+        {
+          name: "contextExtractor",
+          type: "(input, output) => string[]",
+          description:
+            "Function to dynamically extract context from the run input and output",
+          required: false,
+        },
+        {
+          name: "scale",
+          type: "number",
+          description: "Scale factor to multiply the final score (default: 1)",
+          required: false,
+        },
+      ],
     },
   ]}
 />
 
-### ContextPrecisionMetricOptions
+**Note**: Either `context` or `contextExtractor` must be provided. If both are provided, `contextExtractor` takes precedence.
+
+## .run() Returns
 
 <PropertiesTable
   content={[
     {
-      name: "scale",
+      name: "score",
       type: "number",
-      description: "Maximum score value",
-      isOptional: true,
-      defaultValue: "1",
+      description:
+        "Mean Average Precision score between 0 and scale (default 0-1)",
     },
     {
-      name: "context",
-      type: "string[]",
-      description: "Array of context pieces in their retrieval order",
-      isOptional: false,
+      name: "reason",
+      type: "string",
+      description:
+        "Human-readable explanation of the context precision evaluation",
     },
   ]}
 />
 
-## measure() Parameters
+## Scoring Details
 
-<PropertiesTable
-  content={[
-    {
-      name: "input",
-      type: "string",
-      description: "The original query or prompt",
-      isOptional: false,
-    },
-    {
-      name: "output",
-      type: "string",
-      description: "The generated response to evaluate",
-      isOptional: false,
-    },
-  ]}
-/>
+### Mean Average Precision (MAP)
 
-## Returns
+Context Precision uses **Mean Average Precision** to evaluate both relevance and positioning:
 
-<PropertiesTable
-  content={[
-    {
-      name: "score",
-      type: "number",
-      description: "Precision score (0 to scale, default 0-1)",
-    },
-    {
-      name: "info",
-      type: "object",
-      description: "Object containing the reason for the score",
-      properties: [
-        {
-          type: "string",
-          parameters: [
-            {
-              name: "reason",
-              type: "string",
-              description: "Detailed explanation of the score",
-            },
-          ],
-        },
-      ],
+1. **Context Evaluation**: Each context piece is classified as relevant or irrelevant for generating the expected output
+2. **Precision Calculation**: For each relevant context at position `i`, precision = `relevant_items_so_far / (i + 1)`
+3. **Average Precision**: Sum all precision values and divide by total relevant items
+4. **Final Score**: Multiply by scale factor and round to 2 decimals
+
+### Scoring Formula
+
+```
+MAP = (Σ Precision@k) / R
+
+Where:
+- Precision@k = (relevant items in positions 1...k) / k
+- R = total number of relevant items
+- Only calculated at positions where relevant items appear
+```
+
+### Score Interpretation
+
+- **0.9-1.0**: Excellent precision - all relevant context early in sequence
+- **0.7-0.8**: Good precision - most relevant context well-positioned
+- **0.4-0.6**: Moderate precision - relevant context mixed with irrelevant
+- **0.1-0.3**: Poor precision - little relevant context or poorly positioned
+- **0.0**: No relevant context found
+
+### Reason analysis
+
+The reason field explains:
+
+- Which context pieces were deemed relevant/irrelevant
+- How positioning affected the MAP calculation
+- Specific relevance criteria used in evaluation
+
+### Optimization insights
+
+Use results to:
+
+- **Improve retrieval**: Filter out irrelevant context before ranking
+- **Optimize ranking**: Ensure relevant context appears early
+- **Tune chunk size**: Balance context detail vs. relevance precision
+- **Evaluate embeddings**: Test different embedding models for better retrieval
+
+### Example Calculation
+
+Given context: `[relevant, irrelevant, relevant, irrelevant]`
+
+- Position 0: Relevant → Precision = 1/1 = 1.0
+- Position 1: Skip (irrelevant)
+- Position 2: Relevant → Precision = 2/3 = 0.67
+- Position 3: Skip (irrelevant)
+
+MAP = (1.0 + 0.67) / 2 = 0.835 ≈ **0.83**
+
+## Scorer configuration
+
+### Dynamic context extraction
+
+```typescript
+const scorer = createContextPrecisionScorer({
+  model: "openai/gpt-4o-mini",
+  options: {
+    contextExtractor: (input, output) => {
+      // Extract context dynamically based on the query
+      const query = input?.inputMessages?.[0]?.content || "";
+
+      // Example: Retrieve from a vector database
+      const searchResults = vectorDB.search(query, { limit: 10 });
+      return searchResults.map((result) => result.content);
     },
-  ]}
-/>
+    scale: 1,
+  },
+});
+```
 
-## Scoring Details
+### Large context evaluation
 
-The metric evaluates context precision through binary relevance assessment and Mean Average Precision (MAP) scoring.
+```typescript
+const scorer = createContextPrecisionScorer({
+  model: "openai/gpt-4o-mini",
+  options: {
+    context: [
+      // Simulate retrieved documents from vector database
+      "Document 1: Highly relevant content...",
+      "Document 2: Somewhat related content...",
+      "Document 3: Tangentially related...",
+      "Document 4: Not relevant...",
+      "Document 5: Highly relevant content...",
+      // ... up to dozens of context pieces
+    ],
+  },
+});
+```
 
-### Scoring Process
+## Examples
 
-1. Assigns binary relevance scores:
-   - Relevant context: 1
-   - Irrelevant context: 0
+### High precision example
 
-2. Calculates Mean Average Precision:
-   - Computes precision at each position
-   - Weights earlier positions more heavily
-   - Normalizes to configured scale
+This example shows perfect context precision where all relevant context appears early:
 
-Final score: `Mean Average Precision * scale`
+```typescript
+import { createContextPrecisionScorer } from "@mastra/evals";
+
+const scorer = createContextPrecisionScorer({
+  model: "openai/gpt-4o-mini",
+  options: {
+    context: [
+      "Photosynthesis is the process by which plants convert sunlight, carbon dioxide, and water into glucose and oxygen.",
+      "The process occurs in the chloroplasts of plant cells, specifically in the thylakoids.",
+      "Light-dependent reactions happen in the thylakoid membranes, while the Calvin cycle occurs in the stroma.",
+    ],
+    scale: 1,
+  },
+});
 
-### Score interpretation
+const result = await scorer.run({
+  input: {
+    inputMessages: [
+      {
+        id: "1",
+        role: "user",
+        content: "How does photosynthesis work in plants?",
+      },
+    ],
+  },
+  output: [
+    {
+      id: "2",
+      role: "assistant",
+      content:
+        "Photosynthesis is the process where plants convert sunlight, CO2, and water into glucose and oxygen using chloroplasts.",
+    },
+  ],
+});
 
-(0 to scale, default 0-1)
+console.log(result);
+// Output:
+// {
+//   score: 1.0,
+//   reason: "The score is 1.0 because all context pieces are highly relevant to explaining photosynthesis and are optimally ordered to support the expected output."
+// }
+```
 
-- 1.0: All relevant context in optimal order
-- 0.7-0.9: Mostly relevant context with good ordering
-- 0.4-0.6: Mixed relevance or suboptimal ordering
-- 0.1-0.3: Limited relevance or poor ordering
-- 0.0: No relevant context
+### Mixed precision example
 
-## Example with Analysis
+This example shows moderate precision with both relevant and irrelevant context:
 
 ```typescript
-import { openai } from "@ai-sdk/openai";
-import { ContextPrecisionMetric } from "@mastra/evals/llm";
-
-// Configure the model for evaluation
-const model = openai("gpt-4o-mini");
-
-const metric = new ContextPrecisionMetric(model, {
-  context: [
-    "Exercise strengthens the heart and improves blood circulation.",
-    "A balanced diet is important for health.",
-    "Regular physical activity reduces stress and anxiety.",
-    "Exercise equipment can be expensive.",
+import { createContextPrecisionScorer } from "@mastra/evals";
+
+const scorer = createContextPrecisionScorer({
+  model: "openai/gpt-4o-mini",
+  options: {
+    context: [
+      "Regular exercise improves cardiovascular health by strengthening the heart muscle.",
+      "A balanced diet should include fruits, vegetables, and whole grains.",
+      "Physical activity releases endorphins which improve mood and reduce stress.",
+      "The average person should drink 8 glasses of water per day.",
+      "Exercise also helps maintain healthy body weight and muscle mass.",
+    ],
+    scale: 1,
+  },
+});
+
+const result = await scorer.run({
+  input: {
+    inputMessages: [
+      {
+        id: "1",
+        role: "user",
+        content: "What are the mental and physical benefits of exercise?",
+      },
+    ],
+  },
+  output: [
+    {
+      id: "2",
+      role: "assistant",
+      content:
+        "Exercise provides cardiovascular benefits, improves mood through endorphin release, and helps maintain healthy body composition.",
+    },
   ],
 });
 
-const result = await metric.measure(
-  "What are the benefits of exercise?",
-  "Regular exercise improves cardiovascular health and mental wellbeing.",
-);
+console.log(result);
+// Output:
+// {
+//   score: 0.72,
+//   reason: "The score is 0.72 because contexts 1, 3, and 5 are relevant to exercise benefits, but irrelevant contexts about diet and hydration reduce the precision score."
+// }
+```
+
+### Low precision example
+
+This example shows poor context precision with mostly irrelevant context:
+
+```typescript
+import { createContextPrecisionScorer } from "@mastra/evals";
+
+const scorer = createContextPrecisionScorer({
+  model: "openai/gpt-4o-mini",
+  options: {
+    context: [
+      "The weather forecast shows sunny skies this weekend.",
+      "Coffee is one of the world's most popular beverages.",
+      "Machine learning requires large amounts of training data.",
+      "Cats typically sleep 12-16 hours per day.",
+      "The capital of France is Paris.",
+    ],
+    scale: 1,
+  },
+});
+
+const result = await scorer.run({
+  input: {
+    inputMessages: [
+      {
+        id: "1",
+        role: "user",
+        content: "How does photosynthesis work?",
+      },
+    ],
+  },
+  output: [
+    {
+      id: "2",
+      role: "assistant",
+      content:
+        "Photosynthesis is the process by which plants convert sunlight into energy using chlorophyll.",
+    },
+  ],
+});
 
-// Example output:
+console.log(result);
+// Output:
 // {
-//   score: 0.75,
-//   info: {
-//     reason: "The score is 0.75 because the first and third contexts are highly relevant
-//           to the benefits mentioned in the output, while the second and fourth contexts
-//           are not directly related to exercise benefits. The relevant contexts are well-positioned
-//           at the beginning and middle of the sequence."
-//   }
+//   score: 0.0,
+//   reason: "The score is 0.0 because none of the retrieved context pieces are relevant to explaining photosynthesis."
 // }
 ```
 
+## Comparison with Context Relevance
+
+Choose the right scorer for your needs:
+
+| Use Case                 | Context Relevance    | Context Precision         |
+| ------------------------ | -------------------- | ------------------------- |
+| **RAG evaluation**       | When usage matters   | When ranking matters      |
+| **Context quality**      | Nuanced levels       | Binary relevance          |
+| **Missing detection**    | ✓ Identifies gaps    | ✗ Not evaluated           |
+| **Usage tracking**       | ✓ Tracks utilization | ✗ Not considered          |
+| **Position sensitivity** | ✗ Position agnostic  | ✓ Rewards early placement |
+
 ## Related
 
-- [Answer Relevancy Metric](./answer-relevancy)
-- [Context Position Metric](./context-position)
-- [Completeness Metric](./completeness)
-- [Context Relevancy Metric](./context-relevancy)
+- [Answer Relevancy Scorer](/reference/v1/evals/answer-relevancy) - Evaluates if answers address the question
+- [Faithfulness Scorer](/reference/v1/evals/faithfulness) - Measures answer groundedness in context
+- [Custom Scorers](/docs/v1/evals/custom-scorers) - Creating your own evaluation metrics

package/.docs/raw/reference/{scorers → evals}/context-relevance.mdx

@@ -1,5 +1,5 @@
 ---
-title: "Reference: Context Relevance Scorer | Scorers | Mastra Docs"
+title: "Reference: Context Relevance Scorer | Evals | Mastra Docs"
 description: Documentation for the Context Relevance Scorer in Mastra. Evaluates the relevance and utility of provided context for generating agent responses using weighted relevance scoring.
 ---
 
@@ -561,7 +561,7 @@ const scorer = createContextRelevanceScorerLLM({
   options: {
     contextExtractor: (input, output) => {
       // Extract from RAG retrieval results
-      const ragResults = input.metadata?.ragResults || [];
+      const ragResults = inputData.metadata?.ragResults || [];
 
       // Return the text content of retrieved documents
       return ragResults
@@ -584,7 +584,7 @@ const evaluateRAG = async (testCases) => {
   for (const testCase of testCases) {
     const score = await scorer.run(testCase);
     results.push({
-      query: testCase.input.inputMessages[0].content,
+      query: testCase.inputData.inputMessages[0].content,
       relevanceScore: score.score,
       feedback: score.reason,
       unusedContext: score.reason.includes("unused"),
@@ -610,6 +610,6 @@ Choose the right scorer for your needs:
 
 ## Related
 
-- [Context Precision Scorer](/reference/scorers/context-precision) - Evaluates context ranking using MAP
-- [Faithfulness Scorer](/reference/scorers/faithfulness) - Measures answer groundedness in context
-- [Custom Scorers](/docs/scorers/custom-scorers) - Creating your own evaluation metrics
+- [Context Precision Scorer](/reference/v1/evals/context-precision) - Evaluates context ranking using MAP
+- [Faithfulness Scorer](/reference/v1/evals/faithfulness) - Measures answer groundedness in context
+- [Custom Scorers](/docs/v1/evals/custom-scorers) - Creating your own evaluation metrics