@mastra/core 1.7.0 → 1.8.0

package/dist/docs/references/reference-evals-keyword-coverage.md

@@ -0,0 +1,128 @@
+# Keyword Coverage Scorer
+
+The `createKeywordCoverageScorer()` function evaluates how well an LLM's output covers the important keywords from the input. It analyzes keyword presence and matches while ignoring common words and stop words.
+
+## Parameters
+
+The `createKeywordCoverageScorer()` function does not take any options.
+
+This function returns an instance of the MastraScorer class. See the [MastraScorer reference](https://mastra.ai/reference/evals/mastra-scorer) for details on the `.run()` method and its input/output.
+
+## .run() Returns
+
+**runId:** (`string`): The id of the run (optional).
+
+**preprocessStepResult:** (`object`): Object with extracted keywords: { referenceKeywords: Set\<string>, responseKeywords: Set\<string> }
+
+**analyzeStepResult:** (`object`): Object with keyword coverage: { totalKeywords: number, matchedKeywords: number }
+
+**score:** (`number`): Coverage score (0-1) representing the proportion of matched keywords.
+
+`.run()` returns a result in the following shape:
+
+```typescript
+{
+  runId: string,
+  extractStepResult: {
+    referenceKeywords: Set<string>,
+    responseKeywords: Set<string>
+  },
+  analyzeStepResult: {
+    totalKeywords: number,
+    matchedKeywords: number
+  },
+  score: number
+}
+```
+
+## Scoring Details
+
+The scorer evaluates keyword coverage by matching keywords with the following features:
+
+- Common word and stop word filtering (e.g., "the", "a", "and")
+- Case-insensitive matching
+- Word form variation handling
+- Special handling of technical terms and compound words
+
+### Scoring Process
+
+1. Processes keywords from input and output:
+
+   - Filters out common words and stop words
+   - Normalizes case and word forms
+   - Handles special terms and compounds
+
+2. Calculates keyword coverage:
+
+   - Matches keywords between texts
+   - Counts successful matches
+   - Computes coverage ratio
+
+Final score: `(matched_keywords / total_keywords) * scale`
+
+### Score interpretation
+
+A coverage score between 0 and 1:
+
+- **1.0**: Complete coverage – all keywords present.
+- **0.7–0.9**: High coverage – most keywords included.
+- **0.4–0.6**: Partial coverage – some keywords present.
+- **0.1–0.3**: Low coverage – few keywords matched.
+- **0.0**: No coverage – no keywords found.
+
+### Special Cases
+
+The scorer handles several special cases:
+
+- Empty input/output: Returns score of 1.0 if both empty, 0.0 if only one is empty
+- Single word: Treated as a single keyword
+- Technical terms: Preserves compound technical terms (e.g., "React.js", "machine learning")
+- Case differences: "JavaScript" matches "javascript"
+- Common words: Ignored in scoring to focus on meaningful keywords
+
+## Example
+
+Evaluate keyword coverage between input queries and agent responses:
+
+```typescript
+import { runEvals } from '@mastra/core/evals'
+import { createKeywordCoverageScorer } from '@mastra/evals/scorers/prebuilt'
+import { myAgent } from './agent'
+
+const scorer = createKeywordCoverageScorer()
+
+const result = await runEvals({
+  data: [
+    {
+      input: 'JavaScript frameworks like React and Vue',
+    },
+    {
+      input: 'TypeScript offers interfaces, generics, and type inference',
+    },
+    {
+      input:
+        'Machine learning models require data preprocessing, feature engineering, and hyperparameter tuning',
+    },
+  ],
+  scorers: [scorer],
+  target: myAgent,
+  onItemComplete: ({ scorerResults }) => {
+    console.log({
+      score: scorerResults[scorer.id].score,
+    })
+  },
+})
+
+console.log(result.scores)
+```
+
+For more details on `runEvals`, see the [runEvals reference](https://mastra.ai/reference/evals/run-evals).
+
+To add this scorer to an agent, see the [Scorers overview](https://mastra.ai/docs/evals/overview) guide.
+
+## Related
+
+- [Completeness Scorer](https://mastra.ai/reference/evals/completeness)
+- [Content Similarity Scorer](https://mastra.ai/reference/evals/content-similarity)
+- [Answer Relevancy Scorer](https://mastra.ai/reference/evals/answer-relevancy)
+- [Textual Difference Scorer](https://mastra.ai/reference/evals/textual-difference)

package/dist/docs/references/reference-evals-mastra-scorer.md

@@ -0,0 +1,123 @@
+# MastraScorer
+
+The `MastraScorer` class is the base class for all scorers in Mastra. It provides a standard `.run()` method for evaluating input/output pairs and supports multi-step scoring workflows with preprocess → analyze → generateScore → generateReason execution flow.
+
+**Note:** Most users should use [`createScorer`](https://mastra.ai/reference/evals/create-scorer) to create scorer instances. Direct instantiation of `MastraScorer` is not recommended.
+
+## How to Get a MastraScorer Instance
+
+Use the `createScorer` factory function, which returns a `MastraScorer` instance:
+
+```typescript
+import { createScorer } from '@mastra/core/evals'
+
+const scorer = createScorer({
+  name: 'My Custom Scorer',
+  description: 'Evaluates responses based on custom criteria',
+}).generateScore(({ run, results }) => {
+  // scoring logic
+  return 0.85
+})
+
+// scorer is now a MastraScorer instance
+```
+
+## .run() Method
+
+The `.run()` method is the primary way to execute your scorer and evaluate input/output pairs. It processes the data through your defined steps (preprocess → analyze → generateScore → generateReason) and returns a comprehensive result object with the score, reasoning, and intermediate results.
+
+```typescript
+const result = await scorer.run({
+  input: 'What is machine learning?',
+  output: 'Machine learning is a subset of artificial intelligence...',
+  runId: 'optional-run-id',
+  requestContext: {
+    /* optional context */
+  },
+})
+```
+
+## .run() Input
+
+**input:** (`any`): Input data to be evaluated. Can be any type depending on your scorer's requirements.
+
+**output:** (`any`): Output data to be evaluated. Can be any type depending on your scorer's requirements.
+
+**runId:** (`string`): Optional unique identifier for this scoring run.
+
+**requestContext:** (`any`): Optional request context from the agent or workflow step being evaluated.
+
+**groundTruth:** (`any`): Optional expected or reference output for comparison during scoring. Automatically passed when using runEvals.
+
+## .run() Returns
+
+**runId:** (`string`): The unique identifier for this scoring run.
+
+**score:** (`number`): Numerical score computed by the generateScore step.
+
+**reason:** (`string`): Explanation for the score, if generateReason step was defined (optional).
+
+**preprocessStepResult:** (`any`): Result of the preprocess step, if defined (optional).
+
+**analyzeStepResult:** (`any`): Result of the analyze step, if defined (optional).
+
+**preprocessPrompt:** (`string`): Preprocess prompt, if defined (optional).
+
+**analyzePrompt:** (`string`): Analyze prompt, if defined (optional).
+
+**generateScorePrompt:** (`string`): Generate score prompt, if defined (optional).
+
+**generateReasonPrompt:** (`string`): Generate reason prompt, if defined (optional).
+
+## Step Execution Flow
+
+When you call `.run()`, the MastraScorer executes the defined steps in this order:
+
+1. **preprocess** (optional) - Extracts or transforms data
+2. **analyze** (optional) - Processes the input/output and preprocessed data
+3. **generateScore** (required) - Computes the numerical score
+4. **generateReason** (optional) - Provides explanation for the score
+
+Each step receives the results from previous steps, allowing you to build complex evaluation pipelines.
+
+## Usage Example
+
+```typescript
+const scorer = createScorer({
+  name: 'Quality Scorer',
+  description: 'Evaluates response quality',
+})
+  .preprocess(({ run }) => {
+    // Extract key information
+    return { wordCount: run.output.split(' ').length }
+  })
+  .analyze(({ run, results }) => {
+    // Analyze the response
+    const hasSubstance = results.preprocessStepResult.wordCount > 10
+    return { hasSubstance }
+  })
+  .generateScore(({ results }) => {
+    // Calculate score
+    return results.analyzeStepResult.hasSubstance ? 1.0 : 0.0
+  })
+  .generateReason(({ score, results }) => {
+    // Explain the score
+    const wordCount = results.preprocessStepResult.wordCount
+    return `Score: ${score}. Response has ${wordCount} words.`
+  })
+
+// Use the scorer
+const result = await scorer.run({
+  input: 'What is machine learning?',
+  output: 'Machine learning is a subset of artificial intelligence...',
+})
+
+console.log(result.score) // 1.0
+console.log(result.reason) // "Score: 1.0. Response has 12 words."
+```
+
+## Integration
+
+MastraScorer instances can be used for agents and workflow steps
+
+See the [createScorer reference](https://mastra.ai/reference/evals/create-scorer) for detailed information on defining custom scoring logic.

package/dist/docs/references/reference-evals-run-evals.md

@@ -0,0 +1,179 @@
+# runEvals
+
+The `runEvals` function enables batch evaluation of agents and workflows by running multiple test cases against scorers concurrently. This is essential for systematic testing, performance analysis, and validation of AI systems.
+
+## Usage Example
+
+```typescript
+import { runEvals } from '@mastra/core/evals'
+import { myAgent } from './agents/my-agent'
+import { myScorer1, myScorer2 } from './scorers'
+
+const result = await runEvals({
+  target: myAgent,
+  data: [
+    { input: 'What is machine learning?' },
+    { input: 'Explain neural networks' },
+    { input: 'How does AI work?' },
+  ],
+  scorers: [myScorer1, myScorer2],
+  targetOptions: { maxSteps: 5 },
+  concurrency: 2,
+  onItemComplete: ({ item, targetResult, scorerResults }) => {
+    console.log(`Completed: ${item.input}`)
+    console.log(`Scores:`, scorerResults)
+  },
+})
+
+console.log(`Average scores:`, result.scores)
+console.log(`Processed ${result.summary.totalItems} items`)
+```
+
+## Parameters
+
+**target:** (`Agent | Workflow`): The agent or workflow to evaluate.
+
+**data:** (`RunEvalsDataItem[]`): Array of test cases with input data and optional ground truth.
+
+**scorers:** (`MastraScorer[] | WorkflowScorerConfig`): Array of scorers for agents, or configuration object for workflows specifying scorers for the workflow and individual steps.
+
+**targetOptions?:** (`AgentExecutionOptions | WorkflowRunOptions`): Options forwarded to the target during execution. For agents: options passed to agent.generate() (e.g. maxSteps, modelSettings, instructions). For workflows: options passed to run.start() (e.g. perStep, outputOptions, initialState).
+
+**concurrency?:** (`number`): Number of test cases to run concurrently. (Default: `1`)
+
+**onItemComplete?:** (`function`): Callback function called after each test case completes. Receives item, target result, and scorer results.
+
+## Data Item Structure
+
+**input:** (`string | string[] | CoreMessage[] | any`): Input data for the target. For agents: messages or strings. For workflows: workflow input data.
+
+**groundTruth?:** (`any`): Expected or reference output for comparison during scoring.
+
+**requestContext?:** (`RequestContext`): Request Context to pass to the target during execution.
+
+**tracingContext?:** (`TracingContext`): Tracing context for observability and debugging.
+
+**startOptions?:** (`WorkflowRunOptions`): Per-item workflow run options (e.g. initialState, perStep, outputOptions). Merged on top of targetOptions, so per-item values take precedence. Only applicable when the target is a workflow.
+
+## Workflow Scorer Configuration
+
+For workflows, you can specify scorers at different levels using `WorkflowScorerConfig`:
+
+**workflow?:** (`MastraScorer[]`): Array of scorers to evaluate the entire workflow output.
+
+**steps?:** (`Record<string, MastraScorer[]>`): Object mapping step IDs to arrays of scorers for evaluating individual step outputs.
+
+## Returns
+
+**scores:** (`Record<string, any>`): Average scores across all test cases, organized by scorer name.
+
+**summary:** (`object`): Summary information about the experiment execution.
+
+**summary.totalItems:** (`number`): Total number of test cases processed.
+
+## Examples
+
+### Agent Evaluation
+
+```typescript
+import { createScorer, runEvals } from '@mastra/core/evals'
+
+const myScorer = createScorer({
+  id: 'my-scorer',
+  description: "Check if Agent's response contains ground truth",
+  type: 'agent',
+}).generateScore(({ run }) => {
+  const response = run.output[0]?.content || ''
+  const expectedResponse = run.groundTruth
+  return response.includes(expectedResponse) ? 1 : 0
+})
+
+const result = await runEvals({
+  target: chatAgent,
+  data: [
+    {
+      input: 'What is AI?',
+      groundTruth: 'AI is a field of computer science that creates intelligent machines.',
+    },
+    {
+      input: 'How does machine learning work?',
+      groundTruth: 'Machine learning uses algorithms to learn patterns from data.',
+    },
+  ],
+  scorers: [relevancyScorer],
+  concurrency: 3,
+})
+```
+
+### Agent with targetOptions
+
+Pass execution options like `maxSteps` or `modelSettings` to customize agent behavior during evaluation:
+
+```typescript
+const result = await runEvals({
+  target: chatAgent,
+  data: [{ input: 'Summarize this article' }, { input: 'Translate to French' }],
+  scorers: [relevancyScorer],
+  targetOptions: {
+    maxSteps: 5,
+    modelSettings: { temperature: 0 },
+  },
+})
+```
+
+### Workflow Evaluation
+
+```typescript
+const workflowResult = await runEvals({
+  target: myWorkflow,
+  data: [
+    { input: { query: 'Process this data', priority: 'high' } },
+    { input: { query: 'Another task', priority: 'low' } },
+  ],
+  scorers: {
+    workflow: [outputQualityScorer],
+    steps: {
+      'validation-step': [validationScorer],
+      'processing-step': [processingScorer],
+    },
+  },
+  onItemComplete: ({ item, targetResult, scorerResults }) => {
+    console.log(`Workflow completed for: ${item.inputData.query}`)
+    if (scorerResults.workflow) {
+      console.log('Workflow scores:', scorerResults.workflow)
+    }
+    if (scorerResults.steps) {
+      console.log('Step scores:', scorerResults.steps)
+    }
+  },
+})
+```
+
+### Workflow with per-item startOptions
+
+Use `startOptions` on individual data items to customize each workflow run. Per-item values take precedence over `targetOptions`:
+
+```typescript
+const result = await runEvals({
+  target: myWorkflow,
+  data: [
+    {
+      input: { query: 'hello' },
+      startOptions: { initialState: { counter: 1 } },
+    },
+    {
+      input: { query: 'world' },
+      startOptions: { initialState: { counter: 2 } },
+    },
+  ],
+  scorers: [outputQualityScorer],
+  targetOptions: { perStep: true },
+})
+```
+
+## Related
+
+- [createScorer()](https://mastra.ai/reference/evals/create-scorer) - Create custom scorers for experiments
+- [MastraScorer](https://mastra.ai/reference/evals/mastra-scorer) - Learn about scorer structure and methods
+- [Custom Scorers](https://mastra.ai/docs/evals/custom-scorers) - Guide to building evaluation logic
+- [Scorers Overview](https://mastra.ai/docs/evals/overview) - Understanding scorer concepts