@mastra/mcp-docs-server 1.0.0-beta.4 → 1.0.0-beta.5

package/.docs/raw/reference/evals/hallucination.mdx

@@ -131,120 +131,45 @@ A hallucination score between 0 and 1:
 
 **Note:** The score represents the degree of hallucination - lower scores indicate better factual alignment with the provided context
 
-## Examples
+## Example
 
-### No hallucination example
+Evaluate agent responses for hallucinations against provided context:
 
-In this example, the response is fully aligned with the provided context. All claims are factually correct and directly supported by the source material, resulting in a low hallucination score.
-
-```typescript title="src/example-no-hallucination.ts" showLineNumbers copy
-import { createHallucinationScorer } from "@mastra/evals/scorers/prebuilt";
-
-const scorer = createHallucinationScorer({ model: 'openai/gpt-4o-mini', options: {
-  context: [
-    "The iPhone was first released in 2007.",
-    "Steve Jobs unveiled it at Macworld.",
-    "The original model had a 3.5-inch screen."
-  ]
-});
-
-const query = "When was the first iPhone released?";
-const response = "The iPhone was first released in 2007, when Steve Jobs unveiled it at Macworld. The original iPhone featured a 3.5-inch screen.";
-
-const result = await scorer.run({
-  input: [{ role: 'user', content: query }],
-  output: { text: response },
-});
-
-console.log(result);
-```
-
-#### No hallucination output
-
-The response receives a score of 0 because there are no contradictions. Every statement is consistent with the context, and no new or fabricated information has been introduced.
-
-```typescript
-{
-  score: 0,
-  reason: 'The score is 0 because none of the statements from the context were contradicted by the output.'
-}
-```
-
-### Mixed hallucination example
-
-In this example, the response includes both accurate and inaccurate claims. Some details align with the context, while others directly contradict it—such as inflated numbers or incorrect locations. These contradictions increase the hallucination score.
-
-```typescript title="src/example-mixed-hallucination.ts" showLineNumbers copy
+```typescript title="src/example-hallucination.ts" showLineNumbers copy
+import { runEvals } from "@mastra/core/evals";
 import { createHallucinationScorer } from "@mastra/evals/scorers/prebuilt";
+import { myAgent } from "./agent";
 
-const scorer = createHallucinationScorer({ model: 'openai/gpt-4o-mini', options: {
-  context: [
-    "The first Star Wars movie was released in 1977.",
-    "It was directed by George Lucas.",
-    "The film earned $775 million worldwide.",
-    "The movie was filmed in Tunisia and England."
-  ]
+// Context is typically populated from agent tool calls or RAG retrieval
+const scorer = createHallucinationScorer({
+  model: "openai/gpt-4o",
 });
 
-const query = "Tell me about the first Star Wars movie.";
-const response = "The first Star Wars movie came out in 1977 and was directed by George Lucas. It made over $1 billion at the box office and was filmed entirely in California.";
-
-const result = await scorer.run({
-  input: [{ role: 'user', content: query }],
-  output: { text: response },
-});
-
-console.log(result);
-```
-
-#### Mixed hallucination output
-
-The Scorer assigns a mid-range score because parts of the response conflict with the context. While some facts are correct, others are inaccurate or fabricated, reducing overall reliability.
-
-```typescript
-{
-  score: 0.5,
-  reason: 'The score is 0.5 because two out of four statements from the output were contradicted by claims in the context, indicating a balance of accurate and inaccurate information.'
-}
-```
-
-### Complete hallucination example
-
-In this example, the response contradicts every key fact in the context. None of the claims can be verified, and all presented details are factually incorrect.
-
-```typescript title="src/example-complete-hallucination.ts" showLineNumbers copy
-import { createHallucinationScorer } from "@mastra/evals/scorers/prebuilt";
-
-const scorer = createHallucinationScorer({ model: 'openai/gpt-4o-mini', options: {
-  context: [
-    "The Wright brothers made their first flight in 1903.",
-    "The flight lasted 12 seconds.",
-    "It covered a distance of 120 feet."
-  ]
-});
-
-const query = "When did the Wright brothers first fly?";
-const response = "The Wright brothers achieved their historic first flight in 1908. The flight lasted about 2 minutes and covered nearly a mile.";
-
-const result = await scorer.run({
-  input: [{ role: 'user', content: query }],
-  output: { text: response },
+const result = await runEvals({
+  data: [
+    {
+      input: "When was the first iPhone released?",
+    },
+    {
+      input: "Tell me about the original iPhone announcement.",
+    },
+  ],
+  scorers: [scorer],
+  target: myAgent,
+  onItemComplete: ({ scorerResults }) => {
+    console.log({
+      score: scorerResults[scorer.id].score,
+      reason: scorerResults[scorer.id].reason,
+    });
+  },
 });
 
-console.log(result);
-
+console.log(result.scores);
 ```
 
-#### Complete hallucination output
+For more details on `runEvals`, see the [runEvals reference](/reference/v1/evals/run-evals).
 
-The Scorer assigns a score of 1 because every statement in the response conflicts with the context. The details are fabricated or inaccurate across the board.
-
-```typescript
-{
-  score: 1,
-  reason: 'The score is 1.0 because all three statements from the output directly contradict the context: the first flight was in 1903, not 1908; it lasted 12 seconds, not about 2 minutes; and it covered 120 feet, not nearly a mile.'
-}
-```
+To add this scorer to an agent, see the [Scorers overview](/docs/v1/evals/overview#adding-scorers-to-agents) guide.
 
 ## Related
 

package/.docs/raw/reference/evals/keyword-coverage.mdx

@@ -102,124 +102,45 @@ The scorer handles several special cases:
 - Case differences: "JavaScript" matches "javascript"
 - Common words: Ignored in scoring to focus on meaningful keywords
 
-## Examples
+## Example
 
-### Full coverage example
+Evaluate keyword coverage between input queries and agent responses:
 
-In this example, the response fully reflects the key terms from the input. All required keywords are present, resulting in complete coverage with no omissions.
-
-```typescript title="src/example-full-keyword-coverage.ts" showLineNumbers copy
-import { createKeywordCoverageScorer } from "@mastra/evals/scorers/prebuilt";
-
-const scorer = createKeywordCoverageScorer();
-
-const input = "JavaScript frameworks like React and Vue";
-const output =
-  "Popular JavaScript frameworks include React and Vue for web development";
-
-const result = await scorer.run({
-  input: [{ role: "user", content: input }],
-  output: { role: "assistant", text: output },
-});
-
-console.log("Score:", result.score);
-console.log("AnalyzeStepResult:", result.analyzeStepResult);
-```
-
-#### Full coverage output
-
-A score of 1 indicates that all expected keywords were found in the response. The `analyzeStepResult` field confirms that the number of matched keywords equals the total number extracted from the input.
-
-```typescript
-{
-  score: 1,
-  analyzeStepResult: {
-    totalKeywords: 4,
-    matchedKeywords: 4
-  }
-}
-```
-
-### Partial coverage example
-
-In this example, the response includes some, but not all, of the important keywords from the input. The score reflects partial coverage, with key terms either missing or only partially matched.
-
-```typescript title="src/example-partial-keyword-coverage.ts" showLineNumbers copy
+```typescript title="src/example-keyword-coverage.ts" showLineNumbers copy
+import { runEvals } from "@mastra/core/evals";
 import { createKeywordCoverageScorer } from "@mastra/evals/scorers/prebuilt";
+import { myAgent } from "./agent";
 
 const scorer = createKeywordCoverageScorer();
 
-const input = "TypeScript offers interfaces, generics, and type inference";
-const output = "TypeScript provides type inference and some advanced features";
-
-const result = await scorer.run({
-  input: [{ role: "user", content: input }],
-  output: { role: "assistant", text: output },
-});
-
-console.log("Score:", result.score);
-console.log("AnalyzeStepResult:", result.analyzeStepResult);
-```
-
-#### Partial coverage output
-
-A score of 0.5 indicates that only half of the expected keywords were found in the response. The `analyzeStepResult` field shows how many terms were matched compared to the total identified in the input.
-
-```typescript
-{
-  score: 0.5,
-  analyzeStepResult: {
-    totalKeywords: 6,
-    matchedKeywords: 3
-  }
-}
-```
-
-### Minimal coverage example
-
-In this example, the response includes very few of the important keywords from the input. The score reflects minimal coverage, with most key terms missing or unaccounted for.
-
-```typescript title="src/example-minimal-keyword-coverage.ts" showLineNumbers copy
-import { createKeywordCoverageScorer } from "@mastra/evals/scorers/prebuilt";
-
-const scorer = createKeywordCoverageScorer();
-
-const input =
-  "Machine learning models require data preprocessing, feature engineering, and hyperparameter tuning";
-const output = "Data preparation is important for models";
-
-const result = await scorer.run({
-  input: [{ role: "user", content: input }],
-  output: { role: "assistant", text: output },
+const result = await runEvals({
+  data: [
+    {
+      input: "JavaScript frameworks like React and Vue",
+    },
+    {
+      input: "TypeScript offers interfaces, generics, and type inference",
+    },
+    {
+      input:
+        "Machine learning models require data preprocessing, feature engineering, and hyperparameter tuning",
+    },
+  ],
+  scorers: [scorer],
+  target: myAgent,
+  onItemComplete: ({ scorerResults }) => {
+    console.log({
+      score: scorerResults[scorer.id].score,
+    });
+  },
 });
 
-console.log("Score:", result.score);
-console.log("AnalyzeStepResult:", result.analyzeStepResult);
-```
-
-#### Minimal coverage output
-
-A low score indicates that only a small number of the expected keywords were present in the response. The `analyzeStepResult` field highlights the gap between total and matched keywords, signaling insufficient coverage.
-
-```typescript
-{
-  score: 0.2,
-  analyzeStepResult: {
-    totalKeywords: 10,
-    matchedKeywords: 2
-  }
-}
+console.log(result.scores);
 ```
 
-### Metric configuration
-
-You can create a `KeywordCoverageMetric` instance with default settings. No additional configuration is required.
-
-```typescript
-const metric = new KeywordCoverageMetric();
-```
+For more details on `runEvals`, see the [runEvals reference](/reference/v1/evals/run-evals).
 
-> See [KeywordCoverageScorer](/reference/v1/evals/keyword-coverage) for a full list of configuration options.
+To add this scorer to an agent, see the [Scorers overview](/docs/v1/evals/overview#adding-scorers-to-agents) guide.
 
 ## Related
 

package/.docs/raw/reference/evals/noise-sensitivity.mdx

@@ -180,7 +180,7 @@ describe("Agent Noise Resistance Tests", () => {
 
     // Step 4: Evaluate using noise sensitivity scorer
     const scorer = createNoiseSensitivityScorerLLM({
-      model: "openai/gpt-4o-mini",
+      model: "openai/gpt-5.1",
       options: {
         baselineResponse,
         noisyQuery,
@@ -389,7 +389,7 @@ describe("Agent Noise Resistance CI Tests", () => {
 
       // Evaluate using noise sensitivity scorer
       const scorer = createNoiseSensitivityScorerLLM({
-        model: "openai/gpt-4o-mini",
+        model: "openai/gpt-5.1",
         options: {
           baselineResponse: testCase.baselineResponse,
           noisyQuery: testCase.noisyQuery,
@@ -424,7 +424,7 @@ This example shows an agent that completely resists misinformation in a test sce
 import { createNoiseSensitivityScorerLLM } from "@mastra/evals";
 
 const scorer = createNoiseSensitivityScorerLLM({
-  model: "openai/gpt-4o-mini",
+  model: "openai/gpt-5.1",
   options: {
     baselineResponse:
       "Regular exercise improves cardiovascular health, strengthens muscles, and enhances mental wellbeing.",
@@ -470,7 +470,7 @@ This example shows an agent partially distracted by irrelevant requests:
 import { createNoiseSensitivityScorerLLM } from "@mastra/evals/scorers/prebuilt";
 
 const scorer = createNoiseSensitivityScorerLLM({
-  model: "openai/gpt-4o-mini",
+  model: "openai/gpt-5.1",
   options: {
     baselineResponse:
       "To bake a cake: Mix flour, sugar, eggs, and butter. Bake at 350°F for 30 minutes.",
@@ -516,7 +516,7 @@ This example shows an agent that incorporates misinformation:
 import { createNoiseSensitivityScorerLLM } from "@mastra/evals";
 
 const scorer = createNoiseSensitivityScorerLLM({
-  model: "openai/gpt-4o-mini",
+  model: "openai/gpt-5.1",
   options: {
     baselineResponse:
       "Climate change is caused by greenhouse gas emissions from human activities.",
@@ -563,7 +563,7 @@ import { createNoiseSensitivityScorerLLM } from "@mastra/evals";
 
 // Lenient scoring - more forgiving of minor issues
 const lenientScorer = createNoiseSensitivityScorerLLM({
-  model: "openai/gpt-4o-mini",
+  model: "openai/gpt-5.1",
   options: {
     baselineResponse: "Python is a high-level programming language.",
     noisyQuery: "What is Python? Also, snakes are dangerous!",
@@ -583,7 +583,7 @@ const lenientScorer = createNoiseSensitivityScorerLLM({
 
 // Strict scoring - harsh on any deviation
 const strictScorer = createNoiseSensitivityScorerLLM({
-  model: "openai/gpt-4o-mini",
+  model: "openai/gpt-5.1",
   options: {
     baselineResponse: "Python is a high-level programming language.",
     noisyQuery: "What is Python? Also, snakes are dangerous!",
@@ -639,7 +639,7 @@ async function evaluateNoiseResistance(testCases) {
 
   for (const testCase of testCases) {
     const scorer = createNoiseSensitivityScorerLLM({
-      model: "openai/gpt-4o-mini",
+      model: "openai/gpt-5.1",
       options: {
         baselineResponse: testCase.baseline,
         noisyQuery: testCase.noisyQuery,
@@ -686,8 +686,8 @@ import { createNoiseSensitivityScorerLLM } from "@mastra/evals";
 
 async function compareModelRobustness() {
   const models = [
-    { name: "GPT-4", model: "openai/gpt-4" },
-    { name: "GPT-3.5", model: "openai/gpt-3.5-turbo" },
+    { name: "GPT-5.1", model: "openai/gpt-5.1" },
+    { name: "GPT-4.1", model: "openai/gpt-4.1" },
     { name: "Claude", model: "anthropic/claude-3-opus" },
   ];
 
@@ -738,7 +738,7 @@ Include noise sensitivity tests in your security test suite to validate prompt i
 import { createNoiseSensitivityScorerLLM } from "@mastra/evals";
 
 const scorer = createNoiseSensitivityScorerLLM({
-  model: "openai/gpt-4o-mini",
+  model: "openai/gpt-5.1",
   options: {
     baselineResponse: "I can help you with programming questions.",
     noisyQuery:

package/.docs/raw/reference/evals/prompt-alignment.mdx

@@ -109,7 +109,7 @@ You can customize the Prompt Alignment Scorer by adjusting the scale parameter a
 
 ```typescript showLineNumbers copy
 const scorer = createPromptAlignmentScorerLLM({
-  model: "openai/gpt-4o-mini",
+  model: "openai/gpt-5.1",
   options: {
     scale: 10, // Score from 0-10 instead of 0-1
     evaluationMode: "both", // 'user', 'system', or 'both' (default)
@@ -272,24 +272,24 @@ const agent = new Agent({
   name: "CodingAssistant",
   instructions:
     "You are a helpful coding assistant. Always provide working code examples.",
-  model: "openai/gpt-4o",
+  model: "openai/gpt-5.1",
 });
 
 // Evaluate comprehensive alignment (default)
 const scorer = createPromptAlignmentScorerLLM({
-  model: "openai/gpt-4o-mini",
+  model: "openai/gpt-5.1",
   options: { evaluationMode: "both" }, // Evaluates both user intent and system guidelines
 });
 
 // Evaluate just user satisfaction
 const userScorer = createPromptAlignmentScorerLLM({
-  model: "openai/gpt-4o-mini",
+  model: "openai/gpt-5.1",
   options: { evaluationMode: "user" }, // Focus only on user request fulfillment
 });
 
 // Evaluate system compliance
 const systemScorer = createPromptAlignmentScorerLLM({
-  model: "openai/gpt-4o-mini",
+  model: "openai/gpt-5.1",
   options: { evaluationMode: "system" }, // Check adherence to system instructions
 });
 
@@ -341,7 +341,7 @@ for (const agent of agents) {
 import { createPromptAlignmentScorerLLM } from "@mastra/evals";
 
 const scorer = createPromptAlignmentScorerLLM({
-  model: "openai/gpt-4o",
+  model: "openai/gpt-5.1",
 });
 
 // Evaluate a code generation task
@@ -371,7 +371,7 @@ const result = await scorer.run({
 ```typescript
 // Configure scale and evaluation mode
 const scorer = createPromptAlignmentScorerLLM({
-  model: "openai/gpt-4o",
+  model: "openai/gpt-5.1",
   options: {
     scale: 10, // Score from 0-10 instead of 0-1
     evaluationMode: "both", // 'user', 'system', or 'both' (default)
@@ -380,13 +380,13 @@ const scorer = createPromptAlignmentScorerLLM({
 
 // User-only evaluation - focus on user satisfaction
 const userScorer = createPromptAlignmentScorerLLM({
-  model: "openai/gpt-4o",
+  model: "openai/gpt-5.1",
   options: { evaluationMode: "user" },
 });
 
 // System-only evaluation - focus on compliance
 const systemScorer = createPromptAlignmentScorerLLM({
-  model: "openai/gpt-4o",
+  model: "openai/gpt-5.1",
   options: { evaluationMode: "system" },
 });
 
@@ -421,7 +421,7 @@ In this example, the response fully addresses the user's prompt with all require
 import { createPromptAlignmentScorerLLM } from "@mastra/evals/scorers/prebuilt";
 
 const scorer = createPromptAlignmentScorerLLM({
-  model: "openai/gpt-4o-mini",
+  model: "openai/gpt-5.1",
 });
 
 const inputMessages = [
@@ -469,7 +469,7 @@ In this example, the response addresses the core intent but misses some requirem
 import { createPromptAlignmentScorerLLM } from "@mastra/evals/scorers/prebuilt";
 
 const scorer = createPromptAlignmentScorerLLM({
-  model: "openai/gpt-4o-mini",
+  model: "openai/gpt-5.1",
 });
 
 const inputMessages = [
@@ -510,7 +510,7 @@ In this example, the response fails to address the user's specific requirements.
 import { createPromptAlignmentScorerLLM } from "@mastra/evals/scorers/prebuilt";
 
 const scorer = createPromptAlignmentScorerLLM({
-  model: "openai/gpt-4o-mini",
+  model: "openai/gpt-5.1",
 });
 
 const inputMessages = [
@@ -554,7 +554,7 @@ Evaluates how well the response addresses the user's request, ignoring system in
 
 ```typescript title="src/example-user-mode.ts" showLineNumbers copy
 const scorer = createPromptAlignmentScorerLLM({
-  model: "openai/gpt-4o-mini",
+  model: "openai/gpt-5.1",
   options: { evaluationMode: "user" },
 });
 
@@ -586,7 +586,7 @@ Evaluates compliance with system behavioral guidelines and constraints:
 
 ```typescript title="src/example-system-mode.ts" showLineNumbers copy
 const scorer = createPromptAlignmentScorerLLM({
-  model: "openai/gpt-4o-mini",
+  model: "openai/gpt-5.1",
   options: { evaluationMode: "system" },
 });
 
@@ -619,7 +619,7 @@ Evaluates both user intent fulfillment and system compliance with weighted scori
 
 ```typescript title="src/example-both-mode.ts" showLineNumbers copy
 const scorer = createPromptAlignmentScorerLLM({
-  model: "openai/gpt-4o-mini",
+  model: "openai/gpt-5.1",
   options: { evaluationMode: "both" }, // This is the default
 });
 

package/.docs/raw/reference/evals/textual-difference.mdx

@@ -83,118 +83,45 @@ A textual difference score between 0 and 1:
 - **0.1–0.3**: Major differences – extensive changes needed.
 - **0.0**: Completely different texts.
 
-## Examples
+## Example
 
-### No differences example
+Measure textual differences between expected and actual agent outputs:
 
-In this example, the texts are exactly the same. The scorer identifies complete similarity with a perfect score and no detected changes.
-
-```typescript title="src/example-no-differences.ts" showLineNumbers copy
+```typescript title="src/example-textual-difference.ts" showLineNumbers copy
+import { runEvals } from "@mastra/core/evals";
 import { createTextualDifferenceScorer } from "@mastra/evals/scorers/prebuilt";
+import { myAgent } from "./agent";
 
 const scorer = createTextualDifferenceScorer();
 
-const input = "The quick brown fox jumps over the lazy dog";
-const output = "The quick brown fox jumps over the lazy dog";
-
-const result = await scorer.run({
-  input: [{ role: "user", content: input }],
-  output: { role: "assistant", text: output },
-});
-
-console.log("Score:", result.score);
-console.log("AnalyzeStepResult:", result.analyzeStepResult);
-```
-
-#### No differences output
-
-The scorer returns a high score, indicating the texts are identical. The detailed info confirms zero changes and no length difference.
-
-```typescript
-{
-  score: 1,
-  analyzeStepResult: {
-    confidence: 1,
-    ratio: 1,
-    changes: 0,
-    lengthDiff: 0,
+const result = await runEvals({
+  data: [
+    {
+      input: "Summarize the concept of recursion",
+      groundTruth:
+        "Recursion is when a function calls itself to solve a problem by breaking it into smaller subproblems.",
+    },
+    {
+      input: "What is the capital of France?",
+      groundTruth: "The capital of France is Paris.",
+    },
+  ],
+  scorers: [scorer],
+  target: myAgent,
+  onItemComplete: ({ scorerResults }) => {
+    console.log({
+      score: scorerResults[scorer.id].score,
+      groundTruth: scorerResults[scorer.id].groundTruth,
+    });
   },
-}
-```
-
-### Minor differences example
-
-In this example, the texts have small variations. The scorer detects these minor differences and returns a moderate similarity score.
-
-```typescript title="src/example-minor-differences.ts" showLineNumbers copy
-import { createTextualDifferenceScorer } from "@mastra/evals/scorers/prebuilt";
-
-const scorer = createTextualDifferenceScorer();
-
-const input = "Hello world! How are you?";
-const output = "Hello there! How is it going?";
-
-const result = await scorer.run({
-  input: [{ role: "user", content: input }],
-  output: { role: "assistant", text: output },
 });
 
-console.log("Score:", result.score);
-console.log("AnalyzeStepResult:", result.analyzeStepResult);
+console.log(result.scores);
 ```
 
-#### Minor differences output
+For more details on `runEvals`, see the [runEvals reference](/reference/v1/evals/run-evals).
 
-The scorer returns a moderate score reflecting the small variations between the texts. The detailed info includes the number of changes and length difference observed.
-
-```typescript
-{
-  score: 0.5925925925925926,
-  analyzeStepResult: {
-    confidence: 0.8620689655172413,
-    ratio: 0.5925925925925926,
-    changes: 5,
-    lengthDiff: 0.13793103448275862
-  }
-}
-```
-
-### Major differences example
-
-In this example, the texts differ significantly. The scorer detects extensive changes and returns a low similarity score.
-
-```typescript title="src/example-major-differences.ts" showLineNumbers copy
-import { createTextualDifferenceScorer } from "@mastra/evals/scorers/prebuilt";
-
-const scorer = createTextualDifferenceScorer();
-
-const input = "Python is a high-level programming language";
-const output = "JavaScript is used for web development";
-
-const result = await scorer.run({
-  input: [{ role: "user", content: input }],
-  output: { role: "assistant", text: output },
-});
-
-console.log("Score:", result.score);
-console.log("AnalyzeStepResult:", result.analyzeStepResult);
-```
-
-#### Major differences output
-
-The scorer returns a low score due to significant differences between the texts. The detailed `analyzeStepResult` shows numerous changes and a notable length difference.
-
-```typescript
-{
-  score: 0.3170731707317073,
-  analyzeStepResult: {
-    confidence: 0.8636363636363636,
-    ratio: 0.3170731707317073,
-    changes: 8,
-    lengthDiff: 0.13636363636363635
-  }
-}
-```
+To add this scorer to an agent, see the [Scorers overview](/docs/v1/evals/overview#adding-scorers-to-agents) guide.
 
 ## Related