@mastra/mcp-docs-server 0.13.39 → 1.0.0-beta.1

package/.docs/raw/reference/evals/faithfulness.mdx

@@ -1,105 +1,76 @@
 ---
-title: "Reference: FaithfulnessMetric Reference | Evals | Mastra Docs"
-description: Documentation for the Faithfulness Metric in Mastra, which evaluates the factual accuracy of LLM outputs compared to the provided context.
+title: "Reference: Faithfulness Scorer | Evals | Mastra Docs"
+description: Documentation for the Faithfulness Scorer in Mastra, which evaluates the factual accuracy of LLM outputs compared to the provided context.
 ---
 
-# FaithfulnessMetric Reference
+# Faithfulness Scorer
 
-:::info Scorers
-This documentation refers to the legacy evals API. For the latest scorer features, see [Scorers](/docs/scorers/overview).
-:::
+The `createFaithfulnessScorer()` function evaluates how factually accurate an LLM's output is compared to the provided context. It extracts claims from the output and verifies them against the context, making it essential to measure RAG pipeline responses' reliability.
 
-The `FaithfulnessMetric` in Mastra evaluates how factually accurate an LLM's output is compared to the provided context. It extracts claims from the output and verifies them against the context, making it essential to measure RAG pipeline responses' reliability.
+## Parameters
 
-## Basic Usage
-
-```typescript
-import { openai } from "@ai-sdk/openai";
-import { FaithfulnessMetric } from "@mastra/evals/llm";
-
-// Configure the model for evaluation
-const model = openai("gpt-4o-mini");
-
-const metric = new FaithfulnessMetric(model, {
-  context: [
-    "The company was established in 1995.",
-    "Currently employs around 450-550 people.",
-  ],
-});
-
-const result = await metric.measure(
-  "Tell me about the company.",
-  "The company was founded in 1995 and has 500 employees.",
-);
-
-console.log(result.score); // 1.0
-console.log(result.info.reason); // "All claims are supported by the context."
-```
-
-## Constructor Parameters
+The `createFaithfulnessScorer()` function accepts a single options object with the following properties:
 
 <PropertiesTable
   content={[
     {
       name: "model",
       type: "LanguageModel",
+      required: true,
       description: "Configuration for the model used to evaluate faithfulness.",
-      isOptional: false,
     },
     {
-      name: "options",
-      type: "FaithfulnessMetricOptions",
-      description: "Additional options for configuring the metric.",
-      isOptional: false,
+      name: "context",
+      type: "string[]",
+      required: true,
+      description:
+        "Array of context chunks against which the output's claims will be verified.",
     },
-  ]}
-/>
-
-### FaithfulnessMetricOptions
-
-<PropertiesTable
-  content={[
     {
       name: "scale",
       type: "number",
-      description:
-        "The maximum score value. The final score will be normalized to this scale.",
-      isOptional: false,
+      required: false,
       defaultValue: "1",
-    },
-    {
-      name: "context",
-      type: "string[]",
       description:
-        "Array of context chunks against which the output's claims will be verified.",
-      isOptional: false,
+        "The maximum score value. The final score will be normalized to this scale.",
     },
   ]}
 />
 
-## measure() Parameters
+This function returns an instance of the MastraScorer class. The `.run()` method accepts the same input as other scorers (see the [MastraScorer reference](./mastra-scorer)), but the return value includes LLM-specific fields as documented below.
+
+## .run() Returns
 
 <PropertiesTable
   content={[
     {
-      name: "input",
+      name: "runId",
       type: "string",
-      description: "The original query or prompt given to the LLM.",
-      isOptional: false,
+      description: "The id of the run (optional).",
+    },
+    {
+      name: "preprocessStepResult",
+      type: "string[]",
+      description: "Array of extracted claims from the output.",
     },
     {
-      name: "output",
+      name: "preprocessPrompt",
       type: "string",
-      description: "The LLM's response to be evaluated for faithfulness.",
-      isOptional: false,
+      description:
+        "The prompt sent to the LLM for the preprocess step (optional).",
+    },
+    {
+      name: "analyzeStepResult",
+      type: "object",
+      description:
+        "Object with verdicts: { verdicts: Array<{ verdict: 'yes' | 'no' | 'unsure', reason: string }> }",
+    },
+    {
+      name: "analyzePrompt",
+      type: "string",
+      description:
+        "The prompt sent to the LLM for the analyze step (optional).",
     },
-  ]}
-/>
-
-## Returns
-
-<PropertiesTable
-  content={[
     {
       name: "score",
       type: "number",
@@ -107,29 +78,23 @@ console.log(result.info.reason); // "All claims are supported by the context."
         "A score between 0 and the configured scale, representing the proportion of claims that are supported by the context.",
     },
     {
-      name: "info",
-      type: "object",
-      description: "Object containing the reason for the score",
-      properties: [
-        {
-          type: "string",
-          parameters: [
-            {
-              name: "reason",
-              type: "string",
-              description:
-                "A detailed explanation of the score, including which claims were supported, contradicted, or marked as unsure.",
-            },
-          ],
-        },
-      ],
+      name: "reason",
+      type: "string",
+      description:
+        "A detailed explanation of the score, including which claims were supported, contradicted, or marked as unsure.",
+    },
+    {
+      name: "generateReasonPrompt",
+      type: "string",
+      description:
+        "The prompt sent to the LLM for the generateReason step (optional).",
     },
   ]}
 />
 
 ## Scoring Details
 
-The metric evaluates faithfulness through claim verification against provided context.
+The scorer evaluates faithfulness through claim verification against provided context.
 
 ### Scoring Process
 
@@ -140,7 +105,6 @@ The metric evaluates faithfulness through claim verification against provided co
      - "yes" - claim supported by context
      - "no" - claim contradicts context
      - "unsure" - claim unverifiable
-
 2. Calculates faithfulness score:
    - Counts supported claims
    - Divides by total claims
@@ -150,50 +114,128 @@ Final score: `(supported_claims / total_claims) * scale`
 
 ### Score interpretation
 
-(0 to scale, default 0-1)
+A faithfulness score between 0 and 1:
+
+- **1.0**: All claims are accurate and directly supported by the context.
+- **0.7–0.9**: Most claims are correct, with minor additions or omissions.
+- **0.4–0.6**: Some claims are supported, but others are unverifiable.
+- **0.1–0.3**: Most of the content is inaccurate or unsupported.
+- **0.0**: All claims are false or contradict the context.
+
+## Examples
+
+### High faithfulness example
+
+In this example, the response closely aligns with the context. Each statement in the output is verifiable and supported by the provided context entries, resulting in a high score.
+
+```typescript title="src/example-high-faithfulness.ts" showLineNumbers copy
+import { createFaithfulnessScorer } from "@mastra/evals/scorers/prebuilt";
 
-- 1.0: All claims supported by context
-- 0.7-0.9: Most claims supported, few unverifiable
-- 0.4-0.6: Mixed support with some contradictions
-- 0.1-0.3: Limited support, many contradictions
-- 0.0: No supported claims
+const scorer = createFaithfulnessScorer({ model: 'openai/gpt-4o-mini', options: {
+  context: [
+    "The Tesla Model 3 was launched in 2017.",
+    "It has a range of up to 358 miles.",
+    "The base model accelerates 0-60 mph in 5.8 seconds."
+  ]
+});
+
+const query = "Tell me about the Tesla Model 3.";
+const response = "The Tesla Model 3 was introduced in 2017. It can travel up to 358 miles on a single charge and the base version goes from 0 to 60 mph in 5.8 seconds.";
+
+const result = await scorer.run({
+  input: [{ role: 'user', content: query }],
+  output: { text: response },
+});
+
+console.log(result);
+```
+
+#### High faithfulness output
 
-## Advanced Example
+The output receives a score of 1 because all the information it provides can be directly traced to the context. There are no missing or contradictory facts.
 
 ```typescript
-import { openai } from "@ai-sdk/openai";
-import { FaithfulnessMetric } from "@mastra/evals/llm";
+{
+  score: 1,
+  reason: 'The score is 1 because all claims made in the output are supported by the provided context.'
+}
+```
 
-// Configure the model for evaluation
-const model = openai("gpt-4o-mini");
+### Mixed faithfulness example
 
-const metric = new FaithfulnessMetric(model, {
+In this example, there are a mix of supported and unsupported claims. Some parts of the response are backed by the context, while others introduce new information not found in the source material.
+
+```typescript title="src/example-mixed-faithfulness.ts" showLineNumbers copy
+import { createFaithfulnessScorer } from "@mastra/evals/scorers/prebuilt";
+
+const scorer = createFaithfulnessScorer({ model: 'openai/gpt-4o-mini', options: {
   context: [
-    "The company had 100 employees in 2020.",
-    "Current employee count is approximately 500.",
-  ],
+    "Python was created by Guido van Rossum.",
+    "The first version was released in 1991.",
+    "Python emphasizes code readability."
+  ]
 });
 
-// Example with mixed claim types
-const result = await metric.measure(
-  "What's the company's growth like?",
-  "The company has grown from 100 employees in 2020 to 500 now, and might expand to 1000 by next year.",
-);
-
-// Example output:
-// {
-//   score: 0.67,
-//   info: {
-//     reason: "The score is 0.67 because two claims are supported by the context
-//           (initial employee count of 100 in 2020 and current count of 500),
-//           while the future expansion claim is marked as unsure as it cannot
-//           be verified against the context."
-//   }
-// }
+const query = "What can you tell me about Python?";
+const response = "Python was created by Guido van Rossum and released in 1991. It is the most popular programming language today and is used by millions of developers worldwide.";
+
+const result = await scorer.run({
+  input: [{ role: 'user', content: query }],
+  output: { text: response },
+});
+
+console.log(result);
+```
+
+#### Mixed faithfulness output
+
+The score is lower because only a portion of the response is verifiable. While some claims match the context, others are unconfirmed or out of scope, reducing the overall faithfulness.
+
+```typescript
+{
+  score: 0.5,
+  reason: "The score is 0.5 because while two claims are supported by the context (Python was created by Guido van Rossum and Python was released in 1991), the other two claims regarding Python's popularity and usage cannot be verified as they are not mentioned in the context."
+}
+```
+
+### Low faithfulness example
+
+In this example, the response directly contradicts the context. None of the claims are supported, and several conflict with the facts provided.
+
+```typescript title="src/example-low-faithfulness.ts" showLineNumbers copy
+import { createFaithfulnessScorer } from "@mastra/evals/scorers/prebuilt";
+
+const scorer = createFaithfulnessScorer({ model: 'openai/gpt-4o-mini', options: {
+  context: [
+    "Mars is the fourth planet from the Sun.",
+    "It has a thin atmosphere of mostly carbon dioxide.",
+    "Two small moons orbit Mars: Phobos and Deimos."
+  ]
+});
+
+const query = "What do we know about Mars?";
+const response = "Mars is the third planet from the Sun. It has a thick atmosphere rich in oxygen and nitrogen, and is orbited by three large moons.";
+
+const result = await scorer.run({
+  input: [{ role: 'user', content: query }],
+  output: { text: response },
+});
+
+console.log(result);
+```
+
+#### Low faithfulness output
+
+Each claim is inaccurate or conflicts with the context, resulting in a score of 0.
+
+```typescript
+{
+  score: 0,
+  reason: "The score is 0 because all claims made in the output contradict the provided context. The output states that Mars is the third planet from the Sun, while the context clearly states it is the fourth. Additionally, it claims that Mars has a thick atmosphere rich in oxygen and nitrogen, contradicting the context's description of a thin atmosphere mostly composed of carbon dioxide. Finally, the output mentions that Mars is orbited by three large moons, while the context specifies that it has only two small moons, Phobos and Deimos. Therefore, there are no supported claims, leading to a score of 0."
+}
 ```
 
-### Related
+## Related
 
-- [Answer Relevancy Metric](./answer-relevancy)
-- [Hallucination Metric](./hallucination)
-- [Context Relevancy Metric](./context-relevancy)
+- [Answer Relevancy Scorer](./answer-relevancy)
+- [Hallucination Scorer](./hallucination)