@mastra/core 1.6.0 → 1.7.0-alpha.0

package/dist/docs/references/reference-evals-keyword-coverage.md

@@ -1,128 +0,0 @@
-# Keyword Coverage Scorer
-
-The `createKeywordCoverageScorer()` function evaluates how well an LLM's output covers the important keywords from the input. It analyzes keyword presence and matches while ignoring common words and stop words.
-
-## Parameters
-
-The `createKeywordCoverageScorer()` function does not take any options.
-
-This function returns an instance of the MastraScorer class. See the [MastraScorer reference](https://mastra.ai/reference/evals/mastra-scorer) for details on the `.run()` method and its input/output.
-
-## .run() Returns
-
-**runId:** (`string`): The id of the run (optional).
-
-**preprocessStepResult:** (`object`): Object with extracted keywords: { referenceKeywords: Set\<string>, responseKeywords: Set\<string> }
-
-**analyzeStepResult:** (`object`): Object with keyword coverage: { totalKeywords: number, matchedKeywords: number }
-
-**score:** (`number`): Coverage score (0-1) representing the proportion of matched keywords.
-
-`.run()` returns a result in the following shape:
-
-```typescript
-{
-  runId: string,
-  extractStepResult: {
-    referenceKeywords: Set<string>,
-    responseKeywords: Set<string>
-  },
-  analyzeStepResult: {
-    totalKeywords: number,
-    matchedKeywords: number
-  },
-  score: number
-}
-```
-
-## Scoring Details
-
-The scorer evaluates keyword coverage by matching keywords with the following features:
-
-- Common word and stop word filtering (e.g., "the", "a", "and")
-- Case-insensitive matching
-- Word form variation handling
-- Special handling of technical terms and compound words
-
-### Scoring Process
-
-1. Processes keywords from input and output:
-
-   - Filters out common words and stop words
-   - Normalizes case and word forms
-   - Handles special terms and compounds
-
-2. Calculates keyword coverage:
-
-   - Matches keywords between texts
-   - Counts successful matches
-   - Computes coverage ratio
-
-Final score: `(matched_keywords / total_keywords) * scale`
-
-### Score interpretation
-
-A coverage score between 0 and 1:
-
-- **1.0**: Complete coverage – all keywords present.
-- **0.7–0.9**: High coverage – most keywords included.
-- **0.4–0.6**: Partial coverage – some keywords present.
-- **0.1–0.3**: Low coverage – few keywords matched.
-- **0.0**: No coverage – no keywords found.
-
-### Special Cases
-
-The scorer handles several special cases:
-
-- Empty input/output: Returns score of 1.0 if both empty, 0.0 if only one is empty
-- Single word: Treated as a single keyword
-- Technical terms: Preserves compound technical terms (e.g., "React.js", "machine learning")
-- Case differences: "JavaScript" matches "javascript"
-- Common words: Ignored in scoring to focus on meaningful keywords
-
-## Example
-
-Evaluate keyword coverage between input queries and agent responses:
-
-```typescript
-import { runEvals } from "@mastra/core/evals";
-import { createKeywordCoverageScorer } from "@mastra/evals/scorers/prebuilt";
-import { myAgent } from "./agent";
-
-const scorer = createKeywordCoverageScorer();
-
-const result = await runEvals({
-  data: [
-    {
-      input: "JavaScript frameworks like React and Vue",
-    },
-    {
-      input: "TypeScript offers interfaces, generics, and type inference",
-    },
-    {
-      input:
-        "Machine learning models require data preprocessing, feature engineering, and hyperparameter tuning",
-    },
-  ],
-  scorers: [scorer],
-  target: myAgent,
-  onItemComplete: ({ scorerResults }) => {
-    console.log({
-      score: scorerResults[scorer.id].score,
-    });
-  },
-});
-
-console.log(result.scores);
-```
-
-For more details on `runEvals`, see the [runEvals reference](https://mastra.ai/reference/evals/run-evals).
-
-To add this scorer to an agent, see the [Scorers overview](https://mastra.ai/docs/evals/overview) guide.
-
-## Related
-
-- [Completeness Scorer](https://mastra.ai/reference/evals/completeness)
-- [Content Similarity Scorer](https://mastra.ai/reference/evals/content-similarity)
-- [Answer Relevancy Scorer](https://mastra.ai/reference/evals/answer-relevancy)
-- [Textual Difference Scorer](https://mastra.ai/reference/evals/textual-difference)

package/dist/docs/references/reference-evals-mastra-scorer.md

@@ -1,123 +0,0 @@
-# MastraScorer
-
-The `MastraScorer` class is the base class for all scorers in Mastra. It provides a standard `.run()` method for evaluating input/output pairs and supports multi-step scoring workflows with preprocess → analyze → generateScore → generateReason execution flow.
-
-**Note:** Most users should use [`createScorer`](https://mastra.ai/reference/evals/create-scorer) to create scorer instances. Direct instantiation of `MastraScorer` is not recommended.
-
-## How to Get a MastraScorer Instance
-
-Use the `createScorer` factory function, which returns a `MastraScorer` instance:
-
-```typescript
-import { createScorer } from "@mastra/core/evals";
-
-const scorer = createScorer({
-  name: "My Custom Scorer",
-  description: "Evaluates responses based on custom criteria",
-}).generateScore(({ run, results }) => {
-  // scoring logic
-  return 0.85;
-});
-
-// scorer is now a MastraScorer instance
-```
-
-## .run() Method
-
-The `.run()` method is the primary way to execute your scorer and evaluate input/output pairs. It processes the data through your defined steps (preprocess → analyze → generateScore → generateReason) and returns a comprehensive result object with the score, reasoning, and intermediate results.
-
-```typescript
-const result = await scorer.run({
-  input: "What is machine learning?",
-  output: "Machine learning is a subset of artificial intelligence...",
-  runId: "optional-run-id",
-  requestContext: {
-    /* optional context */
-  },
-});
-```
-
-## .run() Input
-
-**input:** (`any`): Input data to be evaluated. Can be any type depending on your scorer's requirements.
-
-**output:** (`any`): Output data to be evaluated. Can be any type depending on your scorer's requirements.
-
-**runId:** (`string`): Optional unique identifier for this scoring run.
-
-**requestContext:** (`any`): Optional request context from the agent or workflow step being evaluated.
-
-**groundTruth:** (`any`): Optional expected or reference output for comparison during scoring. Automatically passed when using runEvals.
-
-## .run() Returns
-
-**runId:** (`string`): The unique identifier for this scoring run.
-
-**score:** (`number`): Numerical score computed by the generateScore step.
-
-**reason:** (`string`): Explanation for the score, if generateReason step was defined (optional).
-
-**preprocessStepResult:** (`any`): Result of the preprocess step, if defined (optional).
-
-**analyzeStepResult:** (`any`): Result of the analyze step, if defined (optional).
-
-**preprocessPrompt:** (`string`): Preprocess prompt, if defined (optional).
-
-**analyzePrompt:** (`string`): Analyze prompt, if defined (optional).
-
-**generateScorePrompt:** (`string`): Generate score prompt, if defined (optional).
-
-**generateReasonPrompt:** (`string`): Generate reason prompt, if defined (optional).
-
-## Step Execution Flow
-
-When you call `.run()`, the MastraScorer executes the defined steps in this order:
-
-1. **preprocess** (optional) - Extracts or transforms data
-2. **analyze** (optional) - Processes the input/output and preprocessed data
-3. **generateScore** (required) - Computes the numerical score
-4. **generateReason** (optional) - Provides explanation for the score
-
-Each step receives the results from previous steps, allowing you to build complex evaluation pipelines.
-
-## Usage Example
-
-```typescript
-const scorer = createScorer({
-  name: "Quality Scorer",
-  description: "Evaluates response quality",
-})
-  .preprocess(({ run }) => {
-    // Extract key information
-    return { wordCount: run.output.split(" ").length };
-  })
-  .analyze(({ run, results }) => {
-    // Analyze the response
-    const hasSubstance = results.preprocessStepResult.wordCount > 10;
-    return { hasSubstance };
-  })
-  .generateScore(({ results }) => {
-    // Calculate score
-    return results.analyzeStepResult.hasSubstance ? 1.0 : 0.0;
-  })
-  .generateReason(({ score, results }) => {
-    // Explain the score
-    const wordCount = results.preprocessStepResult.wordCount;
-    return `Score: ${score}. Response has ${wordCount} words.`;
-  });
-
-// Use the scorer
-const result = await scorer.run({
-  input: "What is machine learning?",
-  output: "Machine learning is a subset of artificial intelligence...",
-});
-
-console.log(result.score); // 1.0
-console.log(result.reason); // "Score: 1.0. Response has 12 words."
-```
-
-## Integration
-
-MastraScorer instances can be used for agents and workflow steps
-
-See the [createScorer reference](https://mastra.ai/reference/evals/create-scorer) for detailed information on defining custom scoring logic.

package/dist/docs/references/reference-evals-run-evals.md

@@ -1,138 +0,0 @@
-# runEvals
-
-The `runEvals` function enables batch evaluation of agents and workflows by running multiple test cases against scorers concurrently. This is essential for systematic testing, performance analysis, and validation of AI systems.
-
-## Usage Example
-
-```typescript
-import { runEvals } from "@mastra/core/evals";
-import { myAgent } from "./agents/my-agent";
-import { myScorer1, myScorer2 } from "./scorers";
-
-const result = await runEvals({
-  target: myAgent,
-  data: [
-    { input: "What is machine learning?" },
-    { input: "Explain neural networks" },
-    { input: "How does AI work?" },
-  ],
-  scorers: [myScorer1, myScorer2],
-  concurrency: 2,
-  onItemComplete: ({ item, targetResult, scorerResults }) => {
-    console.log(`Completed: ${item.input}`);
-    console.log(`Scores:`, scorerResults);
-  },
-});
-
-console.log(`Average scores:`, result.scores);
-console.log(`Processed ${result.summary.totalItems} items`);
-```
-
-## Parameters
-
-**target:** (`Agent | Workflow`): The agent or workflow to evaluate.
-
-**data:** (`RunEvalsDataItem[]`): Array of test cases with input data and optional ground truth.
-
-**scorers:** (`MastraScorer[] | WorkflowScorerConfig`): Array of scorers for agents, or configuration object for workflows specifying scorers for the workflow and individual steps.
-
-**concurrency?:** (`number`): Number of test cases to run concurrently. (Default: `1`)
-
-**onItemComplete?:** (`function`): Callback function called after each test case completes. Receives item, target result, and scorer results.
-
-## Data Item Structure
-
-**input:** (`string | string[] | CoreMessage[] | any`): Input data for the target. For agents: messages or strings. For workflows: workflow input data.
-
-**groundTruth?:** (`any`): Expected or reference output for comparison during scoring.
-
-**requestContext?:** (`RequestContext`): Request Context to pass to the target during execution.
-
-**tracingContext?:** (`TracingContext`): Tracing context for observability and debugging.
-
-## Workflow Scorer Configuration
-
-For workflows, you can specify scorers at different levels using `WorkflowScorerConfig`:
-
-**workflow?:** (`MastraScorer[]`): Array of scorers to evaluate the entire workflow output.
-
-**steps?:** (`Record<string, MastraScorer[]>`): Object mapping step IDs to arrays of scorers for evaluating individual step outputs.
-
-## Returns
-
-**scores:** (`Record<string, any>`): Average scores across all test cases, organized by scorer name.
-
-**summary:** (`object`): Summary information about the experiment execution.
-
-**summary.totalItems:** (`number`): Total number of test cases processed.
-
-## Examples
-
-### Agent Evaluation
-
-```typescript
-import { createScorer, runEvals } from "@mastra/core/evals";
-
-const myScorer = createScorer({
-  id: "my-scorer",
-  description: "Check if Agent's response contains ground truth",
-  type: "agent",
-}).generateScore(({ run }) => {
-  const response = run.output[0]?.content || "";
-  const expectedResponse = run.groundTruth;
-  return response.includes(expectedResponse) ? 1 : 0;
-});
-
-const result = await runEvals({
-  target: chatAgent,
-  data: [
-    {
-      input: "What is AI?",
-      groundTruth:
-        "AI is a field of computer science that creates intelligent machines.",
-    },
-    {
-      input: "How does machine learning work?",
-      groundTruth:
-        "Machine learning uses algorithms to learn patterns from data.",
-    },
-  ],
-  scorers: [relevancyScorer],
-  concurrency: 3,
-});
-```
-
-### Workflow Evaluation
-
-```typescript
-const workflowResult = await runEvals({
-  target: myWorkflow,
-  data: [
-    { input: { query: "Process this data", priority: "high" } },
-    { input: { query: "Another task", priority: "low" } },
-  ],
-  scorers: {
-    workflow: [outputQualityScorer],
-    steps: {
-      "validation-step": [validationScorer],
-      "processing-step": [processingScorer],
-    },
-  },
-  onItemComplete: ({ item, targetResult, scorerResults }) => {
-    console.log(`Workflow completed for: ${item.inputData.query}`);
-    if (scorerResults.workflow) {
-      console.log("Workflow scores:", scorerResults.workflow);
-    }
-    if (scorerResults.steps) {
-      console.log("Step scores:", scorerResults.steps);
-    }
-  },
-});
-```
-
-## Related
-
-- [createScorer()](https://mastra.ai/reference/evals/create-scorer) - Create custom scorers for experiments
-- [MastraScorer](https://mastra.ai/reference/evals/mastra-scorer) - Learn about scorer structure and methods
-- [Custom Scorers](https://mastra.ai/docs/evals/custom-scorers) - Guide to building evaluation logic
-- [Scorers Overview](https://mastra.ai/docs/evals/overview) - Understanding scorer concepts