@mastra/core 1.7.0 → 1.8.0

package/dist/docs/references/reference-evals-create-scorer.md

@@ -0,0 +1,270 @@
+# createScorer
+
+Mastra provides a unified `createScorer` factory that allows you to define custom scorers for evaluating input/output pairs. You can use either native JavaScript functions or LLM-based prompt objects for each evaluation step. Custom scorers can be added to Agents and Workflow steps.
+
+## How to Create a Custom Scorer
+
+Use the `createScorer` factory to define your scorer with a name, description, and optional judge configuration. Then chain step methods to build your evaluation pipeline. You must provide at least a `generateScore` step.
+
+**Prompt object steps** are step configurations expressed as objects with `description` + `createPrompt` (and `outputSchema` for `preprocess`/`analyze`). These steps invoke the judge LLM. **Function steps** are plain functions and never call the judge.
+
+```typescript
+import { createScorer } from '@mastra/core/evals'
+
+const scorer = createScorer({
+  id: 'my-custom-scorer',
+  name: 'My Custom Scorer', // Optional, defaults to id
+  description: 'Evaluates responses based on custom criteria',
+  type: 'agent', // Optional: for agent evaluation with automatic typing
+  judge: {
+    model: myModel,
+    instructions: 'You are an expert evaluator...',
+  },
+})
+  .preprocess({
+    /* step config */
+  })
+  .analyze({
+    /* step config */
+  })
+  .generateScore(({ run, results }) => {
+    // Return a number
+  })
+  .generateReason({
+    /* step config */
+  })
+```
+
+## createScorer Options
+
+**id:** (`string`): Unique identifier for the scorer. Used as the name if \`name\` is not provided.
+
+**name?:** (`string`): Name of the scorer. Defaults to \`id\` if not provided.
+
+**description:** (`string`): Description of what the scorer does.
+
+**judge?:** (`object`): Optional judge configuration for LLM-based steps. See Judge Object section below.
+
+**type?:** (`string`): Type specification for input/output. Use 'agent' for automatic agent types. For custom types, use the generic approach instead.
+
+This function returns a scorer builder that you can chain step methods onto. See the [MastraScorer reference](https://mastra.ai/reference/evals/mastra-scorer) for details on the `.run()` method and its input/output.
+
+## Judge Object
+
+**model:** (`LanguageModel`): The LLM model instance to use for evaluation.
+
+**instructions:** (`string`): System prompt/instructions for the LLM.
+
+The judge only runs for steps defined as **prompt objects** (`preprocess`, `analyze`, `generateScore`, `generateReason` in prompt mode). If you use function steps only, the judge is never called and there is no LLM output to inspect. In that case, any score/reason must be produced by your functions.
+
+When a prompt-object step runs, its structured LLM output is stored in the corresponding result field (`preprocessStepResult`, `analyzeStepResult`, or the value consumed by `calculateScore` in `generateScore`).
+
+## Type Safety
+
+You can specify input/output types when creating scorers for better type inference and IntelliSense support:
+
+### Agent Type Shortcut
+
+For evaluating agents, use `type: 'agent'` to automatically get the correct types for agent input/output:
+
+```typescript
+import { createScorer } from '@mastra/core/evals'
+
+// Agent scorer with automatic typing
+const agentScorer = createScorer({
+  id: 'agent-response-quality',
+  description: 'Evaluates agent responses',
+  type: 'agent', // Automatically provides ScorerRunInputForAgent/ScorerRunOutputForAgent
+})
+  .preprocess(({ run }) => {
+    // run.input is automatically typed as ScorerRunInputForAgent
+    const userMessage = run.inputData.inputMessages[0]?.content
+    return { userMessage }
+  })
+  .generateScore(({ run, results }) => {
+    // run.output is automatically typed as ScorerRunOutputForAgent
+    const response = run.output[0]?.content
+    return response.length > 10 ? 1.0 : 0.5
+  })
+```
+
+### Custom Types with Generics
+
+For custom input/output types, use the generic approach:
+
+```typescript
+import { createScorer } from '@mastra/core/evals'
+
+type CustomInput = { query: string; context: string[] }
+type CustomOutput = { answer: string; confidence: number }
+
+const customScorer = createScorer<CustomInput, CustomOutput>({
+  id: 'custom-scorer',
+  description: 'Evaluates custom data',
+}).generateScore(({ run }) => {
+  // run.input is typed as CustomInput
+  // run.output is typed as CustomOutput
+  return run.output.confidence
+})
+```
+
+### Built-in Agent Types
+
+- **`ScorerRunInputForAgent`** - Contains `inputMessages`, `rememberedMessages`, `systemMessages`, and `taggedSystemMessages` for agent evaluation
+- **`ScorerRunOutputForAgent`** - Array of agent response messages
+
+Using these types provides autocomplete, compile-time validation, and better documentation for your scoring logic.
+
+## Trace Scoring with Agent Types
+
+When you use `type: 'agent'`, your scorer is compatible for both adding directly to agents and scoring traces from agent interactions. The scorer automatically transforms trace data into the proper agent input/output format:
+
+```typescript
+const agentTraceScorer = createScorer({
+  id: 'agent-trace-length',
+  description: 'Evaluates agent response length',
+  type: 'agent',
+}).generateScore(({ run }) => {
+  // Trace data is automatically transformed to agent format
+  const userMessages = run.inputData.inputMessages
+  const agentResponse = run.output[0]?.content
+
+  // Score based on response length
+  return agentResponse?.length > 50 ? 0 : 1
+})
+
+// Register with Mastra for trace scoring
+const mastra = new Mastra({
+  scorers: {
+    agentTraceScorer,
+  },
+})
+```
+
+## Step Method Signatures
+
+### preprocess
+
+Optional preprocessing step that can extract or transform data before analysis.
+
+**Function Mode:** Function: `({ run, results }) => any`
+
+**run.input:** (`any`): Input records provided to the scorer. If the scorer is added to an agent, this will be an array of user messages, e.g. \`\[{ role: 'user', content: 'hello world' }]\`. If the scorer is used in a workflow, this will be the input of the workflow.
+
+**run.output:** (`any`): Output record provided to the scorer. For agents, this is usually the agent's response. For workflows, this is the workflow's output.
+
+**run.runId:** (`string`): Unique identifier for this scoring run.
+
+**run.requestContext?:** (`object`): Request Context from the agent or workflow step being evaluated (optional).
+
+**results:** (`object`): Empty object (no previous steps).
+
+Returns: `any`\
+The method can return any value. The returned value will be available to subsequent steps as `preprocessStepResult`.
+
+**Prompt Object Mode:**
+
+**description:** (`string`): Description of what this preprocessing step does.
+
+**outputSchema:** (`ZodSchema`): Zod schema for the expected output of the preprocess step.
+
+**createPrompt:** (`function`): Function: ({ run, results }) => string. Returns the prompt for the LLM.
+
+**judge?:** (`object`): (Optional) LLM judge for this step (can override main judge). See Judge Object section.
+
+### analyze
+
+Optional analysis step that processes the input/output and any preprocessed data.
+
+**Function Mode:** Function: `({ run, results }) => any`
+
+**run.input:** (`any`): Input records provided to the scorer. If the scorer is added to an agent, this will be an array of user messages, e.g. \`\[{ role: 'user', content: 'hello world' }]\`. If the scorer is used in a workflow, this will be the input of the workflow.
+
+**run.output:** (`any`): Output record provided to the scorer. For agents, this is usually the agent's response. For workflows, this is the workflow's output.
+
+**run.runId:** (`string`): Unique identifier for this scoring run.
+
+**run.requestContext?:** (`object`): Request Context from the agent or workflow step being evaluated (optional).
+
+**results.preprocessStepResult?:** (`any`): Result from preprocess step, if defined (optional).
+
+Returns: `any`\
+The method can return any value. The returned value will be available to subsequent steps as `analyzeStepResult`.
+
+**Prompt Object Mode:**
+
+**description:** (`string`): Description of what this analysis step does.
+
+**outputSchema:** (`ZodSchema`): Zod schema for the expected output of the analyze step.
+
+**createPrompt:** (`function`): Function: ({ run, results }) => string. Returns the prompt for the LLM.
+
+**judge?:** (`object`): (Optional) LLM judge for this step (can override main judge). See Judge Object section.
+
+### generateScore
+
+**Required** step that computes the final numerical score.
+
+**Function Mode:** Function: `({ run, results }) => number`
+
+**run.input:** (`any`): Input records provided to the scorer. If the scorer is added to an agent, this will be an array of user messages, e.g. \`\[{ role: 'user', content: 'hello world' }]\`. If the scorer is used in a workflow, this will be the input of the workflow.
+
+**run.output:** (`any`): Output record provided to the scorer. For agents, this is usually the agent's response. For workflows, this is the workflow's output.
+
+**run.runId:** (`string`): Unique identifier for this scoring run.
+
+**run.requestContext?:** (`object`): Request Context from the agent or workflow step being evaluated (optional).
+
+**results.preprocessStepResult?:** (`any`): Result from preprocess step, if defined (optional).
+
+**results.analyzeStepResult?:** (`any`): Result from analyze step, if defined (optional).
+
+Returns: `number`\
+The method must return a numerical score.
+
+**Prompt Object Mode:**
+
+**description:** (`string`): Description of what this scoring step does.
+
+**outputSchema:** (`ZodSchema`): Zod schema for the expected output of the generateScore step.
+
+**createPrompt:** (`function`): Function: ({ run, results }) => string. Returns the prompt for the LLM.
+
+**judge?:** (`object`): (Optional) LLM judge for this step (can override main judge). See Judge Object section.
+
+When using prompt object mode, you must also provide a `calculateScore` function to convert the LLM output to a numerical score:
+
+**calculateScore:** (`function`): Function: ({ run, results, analyzeStepResult }) => number. Converts the LLM's structured output into a numerical score.
+
+### generateReason
+
+Optional step that provides an explanation for the score.
+
+**Function Mode:** Function: `({ run, results, score }) => string`
+
+**run.input:** (`any`): Input records provided to the scorer. If the scorer is added to an agent, this will be an array of user messages, e.g. \`\[{ role: 'user', content: 'hello world' }]\`. If the scorer is used in a workflow, this will be the input of the workflow.
+
+**run.output:** (`any`): Output record provided to the scorer. For agents, this is usually the agent's response. For workflows, this is the workflow's output.
+
+**run.runId:** (`string`): Unique identifier for this scoring run.
+
+**run.requestContext?:** (`object`): Request Context from the agent or workflow step being evaluated (optional).
+
+**results.preprocessStepResult?:** (`any`): Result from preprocess step, if defined (optional).
+
+**results.analyzeStepResult?:** (`any`): Result from analyze step, if defined (optional).
+
+**score:** (`number`): Score computed by the generateScore step.
+
+Returns: `string`\
+The method must return a string explaining the score.
+
+**Prompt Object Mode:**
+
+**description:** (`string`): Description of what this reasoning step does.
+
+**createPrompt:** (`function`): Function: ({ run, results, score }) => string. Returns the prompt for the LLM.
+
+**judge?:** (`object`): (Optional) LLM judge for this step (can override main judge). See Judge Object section.
+
+All step functions can be async.

package/dist/docs/references/reference-evals-faithfulness.md

@@ -0,0 +1,114 @@
+# Faithfulness Scorer
+
+The `createFaithfulnessScorer()` function evaluates how factually accurate an LLM's output is compared to the provided context. It extracts claims from the output and verifies them against the context, making it essential to measure RAG pipeline responses' reliability.
+
+## Parameters
+
+The `createFaithfulnessScorer()` function accepts a single options object with the following properties:
+
+**model:** (`LanguageModel`): Configuration for the model used to evaluate faithfulness.
+
+**context:** (`string[]`): Array of context chunks against which the output's claims will be verified.
+
+**scale:** (`number`): The maximum score value. The final score will be normalized to this scale. (Default: `1`)
+
+This function returns an instance of the MastraScorer class. The `.run()` method accepts the same input as other scorers (see the [MastraScorer reference](https://mastra.ai/reference/evals/mastra-scorer)), but the return value includes LLM-specific fields as documented below.
+
+## .run() Returns
+
+**runId:** (`string`): The id of the run (optional).
+
+**preprocessStepResult:** (`string[]`): Array of extracted claims from the output.
+
+**preprocessPrompt:** (`string`): The prompt sent to the LLM for the preprocess step (optional).
+
+**analyzeStepResult:** (`object`): Object with verdicts: { verdicts: Array<{ verdict: 'yes' | 'no' | 'unsure', reason: string }> }
+
+**analyzePrompt:** (`string`): The prompt sent to the LLM for the analyze step (optional).
+
+**score:** (`number`): A score between 0 and the configured scale, representing the proportion of claims that are supported by the context.
+
+**reason:** (`string`): A detailed explanation of the score, including which claims were supported, contradicted, or marked as unsure.
+
+**generateReasonPrompt:** (`string`): The prompt sent to the LLM for the generateReason step (optional).
+
+## Scoring Details
+
+The scorer evaluates faithfulness through claim verification against provided context.
+
+### Scoring Process
+
+1. Analyzes claims and context:
+
+   - Extracts all claims (factual and speculative)
+
+   - Verifies each claim against context
+
+   - Assigns one of three verdicts:
+
+     - "yes" - claim supported by context
+     - "no" - claim contradicts context
+     - "unsure" - claim unverifiable
+
+2. Calculates faithfulness score:
+
+   - Counts supported claims
+   - Divides by total claims
+   - Scales to configured range
+
+Final score: `(supported_claims / total_claims) * scale`
+
+### Score interpretation
+
+A faithfulness score between 0 and 1:
+
+- **1.0**: All claims are accurate and directly supported by the context.
+- **0.7–0.9**: Most claims are correct, with minor additions or omissions.
+- **0.4–0.6**: Some claims are supported, but others are unverifiable.
+- **0.1–0.3**: Most of the content is inaccurate or unsupported.
+- **0.0**: All claims are false or contradict the context.
+
+## Example
+
+Evaluate agent responses for faithfulness to provided context:
+
+```typescript
+import { runEvals } from '@mastra/core/evals'
+import { createFaithfulnessScorer } from '@mastra/evals/scorers/prebuilt'
+import { myAgent } from './agent'
+
+// Context is typically populated from agent tool calls or RAG retrieval
+const scorer = createFaithfulnessScorer({
+  model: 'openai/gpt-4o',
+})
+
+const result = await runEvals({
+  data: [
+    {
+      input: 'Tell me about the Tesla Model 3.',
+    },
+    {
+      input: 'What are the key features of this electric vehicle?',
+    },
+  ],
+  scorers: [scorer],
+  target: myAgent,
+  onItemComplete: ({ scorerResults }) => {
+    console.log({
+      score: scorerResults[scorer.id].score,
+      reason: scorerResults[scorer.id].reason,
+    })
+  },
+})
+
+console.log(result.scores)
+```
+
+For more details on `runEvals`, see the [runEvals reference](https://mastra.ai/reference/evals/run-evals).
+
+To add this scorer to an agent, see the [Scorers overview](https://mastra.ai/docs/evals/overview) guide.
+
+## Related
+
+- [Answer Relevancy Scorer](https://mastra.ai/reference/evals/answer-relevancy)
+- [Hallucination Scorer](https://mastra.ai/reference/evals/hallucination)

package/dist/docs/references/reference-evals-hallucination.md

@@ -0,0 +1,213 @@
+# Hallucination Scorer
+
+The `createHallucinationScorer()` function evaluates whether an LLM generates factually correct information by comparing its output against the provided context. This scorer measures hallucination by identifying direct contradictions between the context and the output.
+
+## Parameters
+
+The `createHallucinationScorer()` function accepts a single options object with the following properties:
+
+**model:** (`LanguageModel`): Configuration for the model used to evaluate hallucination.
+
+**options.scale:** (`number`): Maximum score value. (Default: `1`)
+
+**options.context:** (`string[]`): Static context strings to use as ground truth for hallucination detection.
+
+**options.getContext:** (`(params: GetContextParams) => string[] | Promise<string[]>`): A hook to dynamically resolve context at runtime. Takes priority over static context. Useful for live scoring where context (like tool results) is only available when the scorer runs.
+
+This function returns an instance of the MastraScorer class. The `.run()` method accepts the same input as other scorers (see the [MastraScorer reference](https://mastra.ai/reference/evals/mastra-scorer)), but the return value includes LLM-specific fields as documented below.
+
+### GetContextParams
+
+The `getContext` hook receives the following parameters:
+
+**run:** (`GetContextRun`): The scorer run containing input, output, runId, requestContext, and tracingContext.
+
+**results:** (`Record<string, any>`): Accumulated results from previous steps (e.g., preprocessStepResult with extracted claims).
+
+**score:** (`number`): The computed score. Only present when called from the generateReason step.
+
+**step:** (`'analyze' | 'generateReason'`): Which step is calling the hook. Useful for caching context between calls.
+
+## .run() Returns
+
+**runId:** (`string`): The id of the run (optional).
+
+**preprocessStepResult:** (`object`): Object with extracted claims: { claims: string\[] }
+
+**preprocessPrompt:** (`string`): The prompt sent to the LLM for the preprocess step (optional).
+
+**analyzeStepResult:** (`object`): Object with verdicts: { verdicts: Array<{ statement: string, verdict: 'yes' | 'no', reason: string }> }
+
+**analyzePrompt:** (`string`): The prompt sent to the LLM for the analyze step (optional).
+
+**score:** (`number`): Hallucination score (0 to scale, default 0-1).
+
+**reason:** (`string`): Detailed explanation of the score and identified contradictions.
+
+**generateReasonPrompt:** (`string`): The prompt sent to the LLM for the generateReason step (optional).
+
+## Scoring Details
+
+The scorer evaluates hallucination through contradiction detection and unsupported claim analysis.
+
+### Scoring Process
+
+1. Analyzes factual content:
+
+   - Extracts statements from context
+   - Identifies numerical values and dates
+   - Maps statement relationships
+
+2. Analyzes output for hallucinations:
+
+   - Compares against context statements
+   - Marks direct conflicts as hallucinations
+   - Identifies unsupported claims as hallucinations
+   - Evaluates numerical accuracy
+   - Considers approximation context
+
+3. Calculates hallucination score:
+
+   - Counts hallucinated statements (contradictions and unsupported claims)
+   - Divides by total statements
+   - Scales to configured range
+
+Final score: `(hallucinated_statements / total_statements) * scale`
+
+### Important Considerations
+
+- Claims not present in context are treated as hallucinations
+
+- Subjective claims are hallucinations unless explicitly supported
+
+- Speculative language ("might", "possibly") about facts IN context is allowed
+
+- Speculative language about facts NOT in context is treated as hallucination
+
+- Empty outputs result in zero hallucinations
+
+- Numerical evaluation considers:
+
+  - Scale-appropriate precision
+  - Contextual approximations
+  - Explicit precision indicators
+
+### Score interpretation
+
+A hallucination score between 0 and 1:
+
+- **0.0**: No hallucination — all claims match the context.
+- **0.3–0.4**: Low hallucination — a few contradictions.
+- **0.5–0.6**: Mixed hallucination — several contradictions.
+- **0.7–0.8**: High hallucination — many contradictions.
+- **0.9–1.0**: Complete hallucination — most or all claims contradict the context.
+
+**Note:** The score represents the degree of hallucination - lower scores indicate better factual alignment with the provided context
+
+## Examples
+
+### Static Context
+
+Use static context when you have known ground truth to compare against:
+
+```typescript
+import { createHallucinationScorer } from '@mastra/evals/scorers/prebuilt'
+
+const scorer = createHallucinationScorer({
+  model: 'openai/gpt-4o',
+  options: {
+    context: [
+      'The first iPhone was announced on January 9, 2007.',
+      'It was released on June 29, 2007.',
+      'Steve Jobs introduced it at Macworld.',
+    ],
+  },
+})
+```
+
+### Dynamic Context with getContext
+
+Use `getContext` for live scoring scenarios where context comes from tool results:
+
+```typescript
+import { createHallucinationScorer } from '@mastra/evals/scorers/prebuilt'
+import { extractToolResults } from '@mastra/evals/scorers'
+
+const scorer = createHallucinationScorer({
+  model: 'openai/gpt-4o',
+  options: {
+    getContext: ({ run, step }) => {
+      // Extract tool results as context
+      const toolResults = extractToolResults(run.output)
+      return toolResults.map(t => JSON.stringify({ tool: t.toolName, result: t.result }))
+    },
+  },
+})
+```
+
+### Live Scoring with Agent
+
+Attach the scorer to an agent for live evaluation:
+
+```typescript
+import { Agent } from '@mastra/core/agent'
+import { createHallucinationScorer } from '@mastra/evals/scorers/prebuilt'
+import { extractToolResults } from '@mastra/evals/scorers'
+
+const hallucinationScorer = createHallucinationScorer({
+  model: 'openai/gpt-4o',
+  options: {
+    getContext: ({ run }) => {
+      const toolResults = extractToolResults(run.output)
+      return toolResults.map(t => JSON.stringify({ tool: t.toolName, result: t.result }))
+    },
+  },
+})
+
+const agent = new Agent({
+  name: 'my-agent',
+  model: 'openai/gpt-4o',
+  instructions: 'You are a helpful assistant.',
+  evals: {
+    scorers: [hallucinationScorer],
+  },
+})
+```
+
+### Batch Evaluation with runEvals
+
+```typescript
+import { runEvals } from '@mastra/core/evals'
+import { createHallucinationScorer } from '@mastra/evals/scorers/prebuilt'
+import { myAgent } from './agent'
+
+const scorer = createHallucinationScorer({
+  model: 'openai/gpt-4o',
+  options: {
+    context: ['Known fact 1', 'Known fact 2'],
+  },
+})
+
+const result = await runEvals({
+  data: [{ input: 'Tell me about topic A' }, { input: 'Tell me about topic B' }],
+  scorers: [scorer],
+  target: myAgent,
+  onItemComplete: ({ scorerResults }) => {
+    console.log({
+      score: scorerResults[scorer.id].score,
+      reason: scorerResults[scorer.id].reason,
+    })
+  },
+})
+
+console.log(result.scores)
+```
+
+For more details on `runEvals`, see the [runEvals reference](https://mastra.ai/reference/evals/run-evals).
+
+To add this scorer to an agent, see the [Scorers overview](https://mastra.ai/docs/evals/overview) guide.
+
+## Related
+
+- [Faithfulness Scorer](https://mastra.ai/reference/evals/faithfulness)
+- [Answer Relevancy Scorer](https://mastra.ai/reference/evals/answer-relevancy)