@mastra/mcp-docs-server 0.13.39 → 1.0.0-beta.0

package/.docs/raw/reference/scorers/context-precision.mdx

@@ -1,352 +0,0 @@
----
-title: "Reference: Context Precision Scorer | Scorers | Mastra Docs"
-description: Documentation for the Context Precision Scorer in Mastra. Evaluates the relevance and precision of retrieved context for generating expected outputs using Mean Average Precision.
----
-
-import PropertiesTable from "@site/src/components/PropertiesTable";
-
-# Context Precision Scorer
-
-The `createContextPrecisionScorer()` function creates a scorer that evaluates how relevant and well-positioned retrieved context pieces are for generating expected outputs. It uses **Mean Average Precision (MAP)** to reward systems that place relevant context earlier in the sequence.
-
-It is especially useful for these use cases:
-
-**RAG System Evaluation**
-
-Ideal for evaluating retrieved context in RAG pipelines where:
-
-- Context ordering matters for model performance
-- You need to measure retrieval quality beyond simple relevance
-- Early relevant context is more valuable than later relevant context
-
-**Context Window Optimization**
-
-Use when optimizing context selection for:
-
-- Limited context windows
-- Token budget constraints
-- Multi-step reasoning tasks
-
-## Parameters
-
-<PropertiesTable
-  content={[
-    {
-      name: "model",
-      type: "MastraModelConfig",
-      description: "The language model to use for evaluating context relevance",
-      required: true,
-    },
-    {
-      name: "options",
-      type: "ContextPrecisionMetricOptions",
-      description: "Configuration options for the scorer",
-      required: true,
-      children: [
-        {
-          name: "context",
-          type: "string[]",
-          description: "Array of context pieces to evaluate for relevance",
-          required: false,
-        },
-        {
-          name: "contextExtractor",
-          type: "(input, output) => string[]",
-          description:
-            "Function to dynamically extract context from the run input and output",
-          required: false,
-        },
-        {
-          name: "scale",
-          type: "number",
-          description: "Scale factor to multiply the final score (default: 1)",
-          required: false,
-        },
-      ],
-    },
-  ]}
-/>
-
-**Note**: Either `context` or `contextExtractor` must be provided. If both are provided, `contextExtractor` takes precedence.
-
-## .run() Returns
-
-<PropertiesTable
-  content={[
-    {
-      name: "score",
-      type: "number",
-      description:
-        "Mean Average Precision score between 0 and scale (default 0-1)",
-    },
-    {
-      name: "reason",
-      type: "string",
-      description:
-        "Human-readable explanation of the context precision evaluation",
-    },
-  ]}
-/>
-
-## Scoring Details
-
-### Mean Average Precision (MAP)
-
-Context Precision uses **Mean Average Precision** to evaluate both relevance and positioning:
-
-1. **Context Evaluation**: Each context piece is classified as relevant or irrelevant for generating the expected output
-2. **Precision Calculation**: For each relevant context at position `i`, precision = `relevant_items_so_far / (i + 1)`
-3. **Average Precision**: Sum all precision values and divide by total relevant items
-4. **Final Score**: Multiply by scale factor and round to 2 decimals
-
-### Scoring Formula
-
-```
-MAP = (Σ Precision@k) / R
-
-Where:
-- Precision@k = (relevant items in positions 1...k) / k
-- R = total number of relevant items
-- Only calculated at positions where relevant items appear
-```
-
-### Score Interpretation
-
-- **0.9-1.0**: Excellent precision - all relevant context early in sequence
-- **0.7-0.8**: Good precision - most relevant context well-positioned
-- **0.4-0.6**: Moderate precision - relevant context mixed with irrelevant
-- **0.1-0.3**: Poor precision - little relevant context or poorly positioned
-- **0.0**: No relevant context found
-
-### Reason analysis
-
-The reason field explains:
-
-- Which context pieces were deemed relevant/irrelevant
-- How positioning affected the MAP calculation
-- Specific relevance criteria used in evaluation
-
-### Optimization insights
-
-Use results to:
-
-- **Improve retrieval**: Filter out irrelevant context before ranking
-- **Optimize ranking**: Ensure relevant context appears early
-- **Tune chunk size**: Balance context detail vs. relevance precision
-- **Evaluate embeddings**: Test different embedding models for better retrieval
-
-### Example Calculation
-
-Given context: `[relevant, irrelevant, relevant, irrelevant]`
-
-- Position 0: Relevant → Precision = 1/1 = 1.0
-- Position 1: Skip (irrelevant)
-- Position 2: Relevant → Precision = 2/3 = 0.67
-- Position 3: Skip (irrelevant)
-
-MAP = (1.0 + 0.67) / 2 = 0.835 ≈ **0.83**
-
-## Scorer configuration
-
-### Dynamic context extraction
-
-```typescript
-const scorer = createContextPrecisionScorer({
-  model: "openai/gpt-4o-mini",
-  options: {
-    contextExtractor: (input, output) => {
-      // Extract context dynamically based on the query
-      const query = input?.inputMessages?.[0]?.content || "";
-
-      // Example: Retrieve from a vector database
-      const searchResults = vectorDB.search(query, { limit: 10 });
-      return searchResults.map((result) => result.content);
-    },
-    scale: 1,
-  },
-});
-```
-
-### Large context evaluation
-
-```typescript
-const scorer = createContextPrecisionScorer({
-  model: "openai/gpt-4o-mini",
-  options: {
-    context: [
-      // Simulate retrieved documents from vector database
-      "Document 1: Highly relevant content...",
-      "Document 2: Somewhat related content...",
-      "Document 3: Tangentially related...",
-      "Document 4: Not relevant...",
-      "Document 5: Highly relevant content...",
-      // ... up to dozens of context pieces
-    ],
-  },
-});
-```
-
-## Examples
-
-### High precision example
-
-This example shows perfect context precision where all relevant context appears early:
-
-```typescript
-import { createContextPrecisionScorer } from "@mastra/evals";
-
-const scorer = createContextPrecisionScorer({
-  model: "openai/gpt-4o-mini",
-  options: {
-    context: [
-      "Photosynthesis is the process by which plants convert sunlight, carbon dioxide, and water into glucose and oxygen.",
-      "The process occurs in the chloroplasts of plant cells, specifically in the thylakoids.",
-      "Light-dependent reactions happen in the thylakoid membranes, while the Calvin cycle occurs in the stroma.",
-    ],
-    scale: 1,
-  },
-});
-
-const result = await scorer.run({
-  input: {
-    inputMessages: [
-      {
-        id: "1",
-        role: "user",
-        content: "How does photosynthesis work in plants?",
-      },
-    ],
-  },
-  output: [
-    {
-      id: "2",
-      role: "assistant",
-      content:
-        "Photosynthesis is the process where plants convert sunlight, CO2, and water into glucose and oxygen using chloroplasts.",
-    },
-  ],
-});
-
-console.log(result);
-// Output:
-// {
-//   score: 1.0,
-//   reason: "The score is 1.0 because all context pieces are highly relevant to explaining photosynthesis and are optimally ordered to support the expected output."
-// }
-```
-
-### Mixed precision example
-
-This example shows moderate precision with both relevant and irrelevant context:
-
-```typescript
-import { createContextPrecisionScorer } from "@mastra/evals";
-
-const scorer = createContextPrecisionScorer({
-  model: "openai/gpt-4o-mini",
-  options: {
-    context: [
-      "Regular exercise improves cardiovascular health by strengthening the heart muscle.",
-      "A balanced diet should include fruits, vegetables, and whole grains.",
-      "Physical activity releases endorphins which improve mood and reduce stress.",
-      "The average person should drink 8 glasses of water per day.",
-      "Exercise also helps maintain healthy body weight and muscle mass.",
-    ],
-    scale: 1,
-  },
-});
-
-const result = await scorer.run({
-  input: {
-    inputMessages: [
-      {
-        id: "1",
-        role: "user",
-        content: "What are the mental and physical benefits of exercise?",
-      },
-    ],
-  },
-  output: [
-    {
-      id: "2",
-      role: "assistant",
-      content:
-        "Exercise provides cardiovascular benefits, improves mood through endorphin release, and helps maintain healthy body composition.",
-    },
-  ],
-});
-
-console.log(result);
-// Output:
-// {
-//   score: 0.72,
-//   reason: "The score is 0.72 because contexts 1, 3, and 5 are relevant to exercise benefits, but irrelevant contexts about diet and hydration reduce the precision score."
-// }
-```
-
-### Low precision example
-
-This example shows poor context precision with mostly irrelevant context:
-
-```typescript
-import { createContextPrecisionScorer } from "@mastra/evals";
-
-const scorer = createContextPrecisionScorer({
-  model: "openai/gpt-4o-mini",
-  options: {
-    context: [
-      "The weather forecast shows sunny skies this weekend.",
-      "Coffee is one of the world's most popular beverages.",
-      "Machine learning requires large amounts of training data.",
-      "Cats typically sleep 12-16 hours per day.",
-      "The capital of France is Paris.",
-    ],
-    scale: 1,
-  },
-});
-
-const result = await scorer.run({
-  input: {
-    inputMessages: [
-      {
-        id: "1",
-        role: "user",
-        content: "How does photosynthesis work?",
-      },
-    ],
-  },
-  output: [
-    {
-      id: "2",
-      role: "assistant",
-      content:
-        "Photosynthesis is the process by which plants convert sunlight into energy using chlorophyll.",
-    },
-  ],
-});
-
-console.log(result);
-// Output:
-// {
-//   score: 0.0,
-//   reason: "The score is 0.0 because none of the retrieved context pieces are relevant to explaining photosynthesis."
-// }
-```
-
-## Comparison with Context Relevance
-
-Choose the right scorer for your needs:
-
-| Use Case                 | Context Relevance    | Context Precision         |
-| ------------------------ | -------------------- | ------------------------- |
-| **RAG evaluation**       | When usage matters   | When ranking matters      |
-| **Context quality**      | Nuanced levels       | Binary relevance          |
-| **Missing detection**    | ✓ Identifies gaps    | ✗ Not evaluated           |
-| **Usage tracking**       | ✓ Tracks utilization | ✗ Not considered          |
-| **Position sensitivity** | ✗ Position agnostic  | ✓ Rewards early placement |
-
-## Related
-
-- [Answer Relevancy Scorer](/reference/scorers/answer-relevancy) - Evaluates if answers address the question
-- [Faithfulness Scorer](/reference/scorers/faithfulness) - Measures answer groundedness in context
-- [Custom Scorers](/docs/scorers/custom-scorers) - Creating your own evaluation metrics

package/.docs/raw/reference/scorers/faithfulness.mdx

@@ -1,241 +0,0 @@
----
-title: "Reference: Faithfulness Scorer | Scorers | Mastra Docs"
-description: Documentation for the Faithfulness Scorer in Mastra, which evaluates the factual accuracy of LLM outputs compared to the provided context.
----
-
-# Faithfulness Scorer
-
-The `createFaithfulnessScorer()` function evaluates how factually accurate an LLM's output is compared to the provided context. It extracts claims from the output and verifies them against the context, making it essential to measure RAG pipeline responses' reliability.
-
-## Parameters
-
-The `createFaithfulnessScorer()` function accepts a single options object with the following properties:
-
-<PropertiesTable
-  content={[
-    {
-      name: "model",
-      type: "LanguageModel",
-      required: true,
-      description: "Configuration for the model used to evaluate faithfulness.",
-    },
-    {
-      name: "context",
-      type: "string[]",
-      required: true,
-      description:
-        "Array of context chunks against which the output's claims will be verified.",
-    },
-    {
-      name: "scale",
-      type: "number",
-      required: false,
-      defaultValue: "1",
-      description:
-        "The maximum score value. The final score will be normalized to this scale.",
-    },
-  ]}
-/>
-
-This function returns an instance of the MastraScorer class. The `.run()` method accepts the same input as other scorers (see the [MastraScorer reference](./mastra-scorer)), but the return value includes LLM-specific fields as documented below.
-
-## .run() Returns
-
-<PropertiesTable
-  content={[
-    {
-      name: "runId",
-      type: "string",
-      description: "The id of the run (optional).",
-    },
-    {
-      name: "preprocessStepResult",
-      type: "string[]",
-      description: "Array of extracted claims from the output.",
-    },
-    {
-      name: "preprocessPrompt",
-      type: "string",
-      description:
-        "The prompt sent to the LLM for the preprocess step (optional).",
-    },
-    {
-      name: "analyzeStepResult",
-      type: "object",
-      description:
-        "Object with verdicts: { verdicts: Array<{ verdict: 'yes' | 'no' | 'unsure', reason: string }> }",
-    },
-    {
-      name: "analyzePrompt",
-      type: "string",
-      description:
-        "The prompt sent to the LLM for the analyze step (optional).",
-    },
-    {
-      name: "score",
-      type: "number",
-      description:
-        "A score between 0 and the configured scale, representing the proportion of claims that are supported by the context.",
-    },
-    {
-      name: "reason",
-      type: "string",
-      description:
-        "A detailed explanation of the score, including which claims were supported, contradicted, or marked as unsure.",
-    },
-    {
-      name: "generateReasonPrompt",
-      type: "string",
-      description:
-        "The prompt sent to the LLM for the generateReason step (optional).",
-    },
-  ]}
-/>
-
-## Scoring Details
-
-The scorer evaluates faithfulness through claim verification against provided context.
-
-### Scoring Process
-
-1. Analyzes claims and context:
-   - Extracts all claims (factual and speculative)
-   - Verifies each claim against context
-   - Assigns one of three verdicts:
-     - "yes" - claim supported by context
-     - "no" - claim contradicts context
-     - "unsure" - claim unverifiable
-2. Calculates faithfulness score:
-   - Counts supported claims
-   - Divides by total claims
-   - Scales to configured range
-
-Final score: `(supported_claims / total_claims) * scale`
-
-### Score interpretation
-
-A faithfulness score between 0 and 1:
-
-- **1.0**: All claims are accurate and directly supported by the context.
-- **0.7–0.9**: Most claims are correct, with minor additions or omissions.
-- **0.4–0.6**: Some claims are supported, but others are unverifiable.
-- **0.1–0.3**: Most of the content is inaccurate or unsupported.
-- **0.0**: All claims are false or contradict the context.
-
-## Examples
-
-### High faithfulness example
-
-In this example, the response closely aligns with the context. Each statement in the output is verifiable and supported by the provided context entries, resulting in a high score.
-
-```typescript title="src/example-high-faithfulness.ts" showLineNumbers copy
-import { createFaithfulnessScorer } from "@mastra/evals/scorers/llm";
-
-const scorer = createFaithfulnessScorer({ model: 'openai/gpt-4o-mini', options: {
-  context: [
-    "The Tesla Model 3 was launched in 2017.",
-    "It has a range of up to 358 miles.",
-    "The base model accelerates 0-60 mph in 5.8 seconds."
-  ]
-});
-
-const query = "Tell me about the Tesla Model 3.";
-const response = "The Tesla Model 3 was introduced in 2017. It can travel up to 358 miles on a single charge and the base version goes from 0 to 60 mph in 5.8 seconds.";
-
-const result = await scorer.run({
-  input: [{ role: 'user', content: query }],
-  output: { text: response },
-});
-
-console.log(result);
-```
-
-#### High faithfulness output
-
-The output receives a score of 1 because all the information it provides can be directly traced to the context. There are no missing or contradictory facts.
-
-```typescript
-{
-  score: 1,
-  reason: 'The score is 1 because all claims made in the output are supported by the provided context.'
-}
-```
-
-### Mixed faithfulness example
-
-In this example, there are a mix of supported and unsupported claims. Some parts of the response are backed by the context, while others introduce new information not found in the source material.
-
-```typescript title="src/example-mixed-faithfulness.ts" showLineNumbers copy
-import { createFaithfulnessScorer } from "@mastra/evals/scorers/llm";
-
-const scorer = createFaithfulnessScorer({ model: 'openai/gpt-4o-mini', options: {
-  context: [
-    "Python was created by Guido van Rossum.",
-    "The first version was released in 1991.",
-    "Python emphasizes code readability."
-  ]
-});
-
-const query = "What can you tell me about Python?";
-const response = "Python was created by Guido van Rossum and released in 1991. It is the most popular programming language today and is used by millions of developers worldwide.";
-
-const result = await scorer.run({
-  input: [{ role: 'user', content: query }],
-  output: { text: response },
-});
-
-console.log(result);
-```
-
-#### Mixed faithfulness output
-
-The score is lower because only a portion of the response is verifiable. While some claims match the context, others are unconfirmed or out of scope, reducing the overall faithfulness.
-
-```typescript
-{
-  score: 0.5,
-  reason: "The score is 0.5 because while two claims are supported by the context (Python was created by Guido van Rossum and Python was released in 1991), the other two claims regarding Python's popularity and usage cannot be verified as they are not mentioned in the context."
-}
-```
-
-### Low faithfulness example
-
-In this example, the response directly contradicts the context. None of the claims are supported, and several conflict with the facts provided.
-
-```typescript title="src/example-low-faithfulness.ts" showLineNumbers copy
-import { createFaithfulnessScorer } from "@mastra/evals/scorers/llm";
-
-const scorer = createFaithfulnessScorer({ model: 'openai/gpt-4o-mini', options: {
-  context: [
-    "Mars is the fourth planet from the Sun.",
-    "It has a thin atmosphere of mostly carbon dioxide.",
-    "Two small moons orbit Mars: Phobos and Deimos."
-  ]
-});
-
-const query = "What do we know about Mars?";
-const response = "Mars is the third planet from the Sun. It has a thick atmosphere rich in oxygen and nitrogen, and is orbited by three large moons.";
-
-const result = await scorer.run({
-  input: [{ role: 'user', content: query }],
-  output: { text: response },
-});
-
-console.log(result);
-```
-
-#### Low faithfulness output
-
-Each claim is inaccurate or conflicts with the context, resulting in a score of 0.
-
-```typescript
-{
-  score: 0,
-  reason: "The score is 0 because all claims made in the output contradict the provided context. The output states that Mars is the third planet from the Sun, while the context clearly states it is the fourth. Additionally, it claims that Mars has a thick atmosphere rich in oxygen and nitrogen, contradicting the context's description of a thin atmosphere mostly composed of carbon dioxide. Finally, the output mentions that Mars is orbited by three large moons, while the context specifies that it has only two small moons, Phobos and Deimos. Therefore, there are no supported claims, leading to a score of 0."
-}
-```
-
-## Related
-
-- [Answer Relevancy Scorer](./answer-relevancy)
-- [Hallucination Scorer](./hallucination)