@mastra/mcp-docs-server 0.0.0-commonjs-20250414101718

package/.docs/raw/reference/evals/context-relevancy.mdx

@@ -0,0 +1,188 @@
+---
+title: "Reference: Context Relevancy | Evals | Mastra Docs"
+description: Documentation for the Context Relevancy Metric, which evaluates the relevance of retrieved context in RAG pipelines.
+---
+
+# ContextRelevancyMetric
+
+The `ContextRelevancyMetric` class evaluates the quality of your RAG (Retrieval-Augmented Generation) pipeline's retriever by measuring how relevant the retrieved context is to the input query. It uses an LLM-based evaluation system that first extracts statements from the context and then assesses their relevance to the input.
+
+## Basic Usage
+
+```typescript
+import { openai } from "@ai-sdk/openai";
+import { ContextRelevancyMetric } from "@mastra/evals/llm";
+
+// Configure the model for evaluation
+const model = openai("gpt-4o-mini");
+
+const metric = new ContextRelevancyMetric(model, {
+  context: [
+    "All data is encrypted at rest and in transit",
+    "Two-factor authentication is mandatory",
+    "The platform supports multiple languages",
+    "Our offices are located in San Francisco"
+  ]
+});
+
+const result = await metric.measure(
+  "What are our product's security features?",
+  "Our product uses encryption and requires 2FA.",
+  );
+
+console.log(result.score); // Score from 0-1
+console.log(result.info.reason); // Explanation of the relevancy assessment
+```
+
+## Constructor Parameters
+
+<PropertiesTable
+  content={[
+    {
+      name: "model",
+      type: "LanguageModel",
+      description: "Configuration for the model used to evaluate context relevancy",
+      isOptional: false,
+    },
+    {
+      name: "options",
+      type: "ContextRelevancyMetricOptions",
+      description: "Configuration options for the metric",
+      isOptional: false,
+    }
+  ]}
+/>
+
+### ContextRelevancyMetricOptions
+
+<PropertiesTable
+  content={[
+    {
+      name: "scale",
+      type: "number",
+      description: "Maximum score value",
+      isOptional: true,
+      defaultValue: "1",
+    },
+    {
+      name: "context",
+      type: "string[]",
+      description: "Array of retrieved context documents used to generate the response",
+      isOptional: false,
+    }
+  ]}
+/>
+
+## measure() Parameters
+
+<PropertiesTable
+  content={[
+    {
+      name: "input",
+      type: "string",
+      description: "The original query or prompt",
+      isOptional: false,
+    },
+    {
+      name: "output",
+      type: "string",
+      description: "The LLM's response to evaluate",
+      isOptional: false,
+    }
+  ]}
+/>
+
+## Returns
+
+<PropertiesTable
+  content={[
+    {
+      name: "score",
+      type: "number",
+      description: "Context relevancy score (0 to scale, default 0-1)",
+    },
+    {
+      name: "info",
+      type: "object",
+      description: "Object containing the reason for the score",
+      properties: [
+        {
+          type: "string",
+          parameters: [
+            {
+              name: "reason",
+              type: "string",
+              description: "Detailed explanation of the relevancy assessment",
+            }
+          ]
+        }
+      ]
+    }
+  ]}
+/>
+
+## Scoring Details
+
+The metric evaluates how well retrieved context matches the query through binary relevance classification.
+
+### Scoring Process
+
+1. Extracts statements from context:
+   - Breaks down context into meaningful units
+   - Preserves semantic relationships
+
+2. Evaluates statement relevance:
+   - Assesses each statement against query
+   - Counts relevant statements
+   - Calculates relevance ratio
+
+Final score: `(relevant_statements / total_statements) * scale`
+
+### Score interpretation
+(0 to scale, default 0-1)
+- 1.0: Perfect relevancy - all retrieved context is relevant
+- 0.7-0.9: High relevancy - most context is relevant with few irrelevant pieces
+- 0.4-0.6: Moderate relevancy - a mix of relevant and irrelevant context
+- 0.1-0.3: Low relevancy - mostly irrelevant context
+- 0.0: No relevancy - completely irrelevant context
+
+## Example with Custom Configuration
+
+```typescript
+import { openai } from "@ai-sdk/openai";
+import { ContextRelevancyMetric } from "@mastra/evals/llm";
+
+// Configure the model for evaluation
+const model = openai("gpt-4o-mini");
+
+const metric = new ContextRelevancyMetric(model, {
+  scale: 100, // Use 0-100 scale instead of 0-1
+  context: [
+    "Basic plan costs $10/month",
+    "Pro plan includes advanced features at $30/month",
+    "Enterprise plan has custom pricing",
+    "Our company was founded in 2020",
+    "We have offices worldwide"
+  ]
+});
+
+const result = await metric.measure(
+  "What are our pricing plans?",
+  "We offer Basic, Pro, and Enterprise plans.",
+);
+
+// Example output:
+// {
+//   score: 60,
+//   info: {
+//     reason: "3 out of 5 statements are relevant to pricing plans. The statements about 
+//           company founding and office locations are not relevant to the pricing query."
+//   }
+// }
+```
+
+## Related
+
+- [Contextual Recall Metric](./contextual-recall)
+- [Context Precision Metric](./context-precision)
+- [Context Position Metric](./context-position) 

package/.docs/raw/reference/evals/contextual-recall.mdx

@@ -0,0 +1,191 @@
+---
+title: "Reference: Contextual Recall | Metrics | Evals | Mastra Docs"
+description: Documentation for the Contextual Recall Metric, which evaluates the completeness of LLM responses in incorporating relevant context.
+---
+
+# ContextualRecallMetric
+
+The `ContextualRecallMetric` class evaluates how effectively an LLM's response incorporates all relevant information from the provided context. It measures whether important information from the reference documents was successfully included in the response, focusing on completeness rather than precision.
+
+## Basic Usage
+
+```typescript
+import { openai } from "@ai-sdk/openai";
+import { ContextualRecallMetric } from "@mastra/evals/llm";
+
+// Configure the model for evaluation
+const model = openai("gpt-4o-mini");
+
+const metric = new ContextualRecallMetric(model, {
+  context: [
+    "Product features: cloud synchronization capability",
+    "Offline mode available for all users",
+    "Supports multiple devices simultaneously",
+    "End-to-end encryption for all data"
+  ]
+});
+
+const result = await metric.measure(
+  "What are the key features of the product?",
+  "The product includes cloud sync, offline mode, and multi-device support.",
+);
+
+console.log(result.score); // Score from 0-1
+```
+
+## Constructor Parameters
+
+<PropertiesTable
+  content={[
+    {
+      name: "model",
+      type: "LanguageModel",
+      description: "Configuration for the model used to evaluate contextual recall",
+      isOptional: false,
+    },
+    {
+      name: "options",
+      type: "ContextualRecallMetricOptions",
+      description: "Configuration options for the metric",
+      isOptional: false,
+    }
+  ]}
+/>
+
+### ContextualRecallMetricOptions
+
+<PropertiesTable
+  content={[
+    {
+      name: "scale",
+      type: "number",
+      description: "Maximum score value",
+      isOptional: true,
+      defaultValue: "1",
+    },
+    {
+      name: "context",
+      type: "string[]",
+      description: "Array of reference documents or pieces of information to check against",
+      isOptional: false,
+    }
+  ]}
+/>
+
+## measure() Parameters
+
+<PropertiesTable
+  content={[
+    {
+      name: "input",
+      type: "string",
+      description: "The original query or prompt",
+      isOptional: false,
+    },
+    {
+      name: "output",
+      type: "string",
+      description: "The LLM's response to evaluate",
+      isOptional: false,
+    }
+  ]}
+/>
+
+## Returns
+
+<PropertiesTable
+  content={[
+    {
+      name: "score",
+      type: "number",
+      description: "Recall score (0 to scale, default 0-1)",
+    },
+    {
+      name: "info",
+      type: "object",
+      description: "Object containing the reason for the score",
+      properties: [
+        {
+          type: "string",
+          parameters: [
+            {
+              name: "reason",
+              type: "string",
+              description: "Detailed explanation of the score",
+            }
+          ]
+        }
+      ]
+    }
+  ]}
+/>
+
+## Scoring Details
+
+The metric evaluates recall through comparison of response content against relevant context items.
+
+### Scoring Process
+
+1. Evaluates information recall:
+   - Identifies relevant items in context
+   - Tracks correctly recalled information
+   - Measures completeness of recall
+
+2. Calculates recall score:
+   - Counts correctly recalled items
+   - Compares against total relevant items
+   - Computes coverage ratio
+
+Final score: `(correctly_recalled_items / total_relevant_items) * scale`
+
+### Score interpretation
+(0 to scale, default 0-1)
+- 1.0: Perfect recall - all relevant information included
+- 0.7-0.9: High recall - most relevant information included
+- 0.4-0.6: Moderate recall - some relevant information missed
+- 0.1-0.3: Low recall - significant information missed
+- 0.0: No recall - no relevant information included
+
+## Example with Custom Configuration
+
+```typescript
+import { openai } from "@ai-sdk/openai";
+import { ContextualRecallMetric } from "@mastra/evals/llm";
+
+// Configure the model for evaluation
+const model = openai("gpt-4o-mini");
+
+const metric = new ContextualRecallMetric(
+  model,
+  {
+    scale: 100, // Use 0-100 scale instead of 0-1
+    context: [
+      "All data is encrypted at rest and in transit",
+      "Two-factor authentication (2FA) is mandatory",
+      "Regular security audits are performed",
+      "Incident response team available 24/7"
+    ]
+  }
+);
+
+const result = await metric.measure(
+  "Summarize the company's security measures",
+  "The company implements encryption for data protection and requires 2FA for all users.",
+);
+
+// Example output:
+// {
+//   score: 50, // Only half of the security measures were mentioned
+//   info: {
+//     reason: "The score is 50 because only half of the security measures were mentioned 
+//           in the response. The response missed the regular security audits and incident 
+//           response team information."
+//   }
+// }
+```
+
+## Related
+
++ [Context Relevancy Metric](./context-relevancy) 
++ [Completeness Metric](./completeness)
++ [Summarization Metric](./summarization)

package/.docs/raw/reference/evals/faithfulness.mdx

@@ -0,0 +1,193 @@
+---
+title: "Reference: Faithfulness | Metrics | Evals | Mastra Docs"
+description: Documentation for the Faithfulness Metric in Mastra, which evaluates the factual accuracy of LLM outputs compared to the provided context.
+---
+
+# FaithfulnessMetric Reference
+
+The `FaithfulnessMetric` in Mastra evaluates how factually accurate an LLM's output is compared to the provided context. It extracts claims from the output and verifies them against the context, making it essential to measure RAG pipeline responses' reliability.
+
+## Basic Usage
+
+```typescript
+import { openai } from "@ai-sdk/openai";
+import { FaithfulnessMetric } from "@mastra/evals/llm";
+
+// Configure the model for evaluation
+const model = openai("gpt-4o-mini");
+
+const metric = new FaithfulnessMetric(model, {
+  context: [
+    "The company was established in 1995.",
+    "Currently employs around 450-550 people.",
+  ],
+});
+
+const result = await metric.measure(
+  "Tell me about the company.",
+  "The company was founded in 1995 and has 500 employees.",
+);
+
+console.log(result.score); // 1.0
+console.log(result.info.reason); // "All claims are supported by the context."
+```
+
+## Constructor Parameters
+
+<PropertiesTable
+  content={[
+    {
+      name: "model",
+      type: "LanguageModel",
+      description: "Configuration for the model used to evaluate faithfulness.",
+      isOptional: false,
+    },
+    {
+      name: "options",
+      type: "FaithfulnessMetricOptions",
+      description: "Additional options for configuring the metric.",
+      isOptional: false,
+    },
+  ]}
+/>
+
+### FaithfulnessMetricOptions
+
+<PropertiesTable
+  content={[
+    {
+      name: "scale",
+      type: "number",
+      description:
+        "The maximum score value. The final score will be normalized to this scale.",
+      isOptional: false,
+      defaultValue: "1",
+    },
+    {
+      name: "context",
+      type: "string[]",
+      description:
+        "Array of context chunks against which the output's claims will be verified.",
+      isOptional: false,
+    },
+  ]}
+/>
+
+## measure() Parameters
+
+<PropertiesTable
+  content={[
+    {
+      name: "input",
+      type: "string",
+      description: "The original query or prompt given to the LLM.",
+      isOptional: false,
+    },
+    {
+      name: "output",
+      type: "string",
+      description: "The LLM's response to be evaluated for faithfulness.",
+      isOptional: false,
+    },
+  ]}
+/>
+
+## Returns
+
+<PropertiesTable
+  content={[
+    {
+      name: "score",
+      type: "number",
+      description:
+        "A score between 0 and the configured scale, representing the proportion of claims that are supported by the context.",
+    },
+    {
+      name: "info",
+      type: "object",
+      description: "Object containing the reason for the score",
+      properties: [
+        {
+          type: "string",
+          parameters: [
+            {
+              name: "reason",
+              type: "string",
+              description:
+                "A detailed explanation of the score, including which claims were supported, contradicted, or marked as unsure.",
+            },
+          ],
+        },
+      ],
+    },
+  ]}
+/>
+
+## Scoring Details
+
+The metric evaluates faithfulness through claim verification against provided context.
+
+### Scoring Process
+
+1. Analyzes claims and context:
+   - Extracts all claims (factual and speculative)
+   - Verifies each claim against context
+   - Assigns one of three verdicts:
+     - "yes" - claim supported by context
+     - "no" - claim contradicts context
+     - "unsure" - claim unverifiable
+
+2. Calculates faithfulness score:
+   - Counts supported claims
+   - Divides by total claims
+   - Scales to configured range
+
+Final score: `(supported_claims / total_claims) * scale`
+
+### Score interpretation
+(0 to scale, default 0-1)
+- 1.0: All claims supported by context
+- 0.7-0.9: Most claims supported, few unverifiable
+- 0.4-0.6: Mixed support with some contradictions
+- 0.1-0.3: Limited support, many contradictions
+- 0.0: No supported claims
+
+## Advanced Example
+
+```typescript
+import { openai } from "@ai-sdk/openai";
+import { FaithfulnessMetric } from "@mastra/evals/llm";
+
+// Configure the model for evaluation
+const model = openai("gpt-4o-mini");
+
+const metric = new FaithfulnessMetric(model, {
+  context: [
+    "The company had 100 employees in 2020.",
+    "Current employee count is approximately 500.",
+  ],
+});
+
+// Example with mixed claim types
+const result = await metric.measure(
+  "What's the company's growth like?",
+  "The company has grown from 100 employees in 2020 to 500 now, and might expand to 1000 by next year.",
+);
+
+// Example output:
+// {
+//   score: 0.67,
+//   info: {
+//     reason: "The score is 0.67 because two claims are supported by the context
+//           (initial employee count of 100 in 2020 and current count of 500),
+//           while the future expansion claim is marked as unsure as it cannot
+//           be verified against the context."
+//   }
+// }
+```
+
+### Related
+
+- [Answer Relevancy Metric](./answer-relevancy)
+- [Hallucination Metric](./hallucination)
+- [Context Relevancy Metric](./context-relevancy)