@mastra/mcp-docs-server 1.1.17-alpha.7 → 1.1.17

package/.docs/reference/evals/run-evals.md

@@ -35,7 +35,7 @@ console.log(`Processed ${result.summary.totalItems} items`)
 
 **data** (`RunEvalsDataItem[]`): Array of test cases with input data and optional ground truth.
 
-**scorers** (`MastraScorer[] | WorkflowScorerConfig`): Array of scorers for agents, or configuration object for workflows specifying scorers for the workflow and individual steps.
+**scorers** (`MastraScorer[] | AgentScorerConfig | WorkflowScorerConfig`): Scorers to use. A flat array applies all scorers to the raw output. For agents, an \`AgentScorerConfig\` object separates agent-level and trajectory scorers. For workflows, a \`WorkflowScorerConfig\` object specifies scorers for the workflow, individual steps, and trajectory.
 
 **targetOptions** (`AgentExecutionOptions | WorkflowRunOptions`): Options forwarded to the target during execution. For agents: options passed to agent.generate() (e.g. maxSteps, modelSettings, instructions). For workflows: options passed to run.start() (e.g. perStep, outputOptions, initialState).
 
@@ -49,20 +49,32 @@ console.log(`Processed ${result.summary.totalItems} items`)
 
 **groundTruth** (`any`): Expected or reference output for comparison during scoring.
 
+**expectedTrajectory** (`TrajectoryExpectation`): Expected trajectory configuration for trajectory scoring. Includes expected steps, ordering, efficiency budgets, blacklists, and tool failure tolerance. Passed to trajectory scorers as \`run.expectedTrajectory\`. Overrides the static defaults in scorer constructors.
+
 **requestContext** (`RequestContext`): Request Context to pass to the target during execution.
 
 **tracingContext** (`TracingContext`): Tracing context for observability and debugging.
 
 **startOptions** (`WorkflowRunOptions`): Per-item workflow run options (e.g. initialState, perStep, outputOptions). Merged on top of targetOptions, so per-item values take precedence. Only applicable when the target is a workflow.
 
+## Agent scorer configuration
+
+For agents, use `AgentScorerConfig` to separate agent-level scorers from trajectory scorers:
+
+**agent** (`MastraScorer[]`): Scorers that receive the raw agent output (MastraDBMessage\[]). Use for evaluating response quality, content, etc.
+
+**trajectory** (`MastraScorer[]`): Scorers that receive a pre-extracted Trajectory object. When storage is configured, the pipeline extracts a hierarchical trajectory from observability traces (including nested tool calls and model generations). Otherwise, it falls back to extracting tool calls from agent messages.
+
 ## Workflow scorer configuration
 
-For workflows, you can specify scorers at different levels using `WorkflowScorerConfig`:
+For workflows, use `WorkflowScorerConfig` to specify scorers at different levels:
 
-**workflow** (`MastraScorer[]`): Array of scorers to evaluate the entire workflow output.
+**workflow** (`MastraScorer[]`): Scorers to evaluate the entire workflow output.
 
 **steps** (`Record<string, MastraScorer[]>`): Object mapping step IDs to arrays of scorers for evaluating individual step outputs.
 
+**trajectory** (`MastraScorer[]`): Scorers that receive a pre-extracted Trajectory from the workflow execution. When storage is configured, the pipeline extracts a hierarchical trajectory from observability traces (including nested agent runs and tool calls within workflow steps). Otherwise, it falls back to extracting step results from the workflow output.
+
 ## Returns
 
 **scores** (`Record<string, any>`): Average scores across all test cases, organized by scorer name.
@@ -105,6 +117,36 @@ const result = await runEvals({
 })
 ```
 
+### Agent trajectory evaluation
+
+Use `AgentScorerConfig` to evaluate both the agent response and its tool-calling trajectory:
+
+```typescript
+import { runEvals } from '@mastra/core/evals'
+import { createTrajectoryAccuracyScorerCode } from '@mastra/evals/scorers/code/trajectory'
+
+const trajectoryScorer = createTrajectoryAccuracyScorerCode()
+
+const result = await runEvals({
+  target: chatAgent,
+  data: [
+    {
+      input: 'What is the weather in London?',
+      expectedTrajectory: {
+        steps: [{ stepType: 'tool_call', name: 'weatherTool' }],
+      },
+    },
+  ],
+  scorers: {
+    // agent: [responseQualityScorer], // Optional: add agent-level scorers
+    trajectory: [trajectoryScorer],
+  },
+})
+
+// result.scores.agent — average agent-level scores
+// result.scores.trajectory — average trajectory scores
+```
+
 ### Agent with `targetOptions`
 
 Pass execution options like `maxSteps` or `modelSettings` to customize agent behavior during evaluation:
@@ -149,6 +191,37 @@ const workflowResult = await runEvals({
 })
 ```
 
+### Workflow trajectory evaluation
+
+Add trajectory scoring to workflow evaluations to validate step execution order:
+
+```typescript
+const workflowResult = await runEvals({
+  target: myWorkflow,
+  data: [
+    {
+      input: { query: 'Process this data' },
+      expectedTrajectory: {
+        steps: [
+          { stepType: 'workflow_step', name: 'validate' },
+          { stepType: 'workflow_step', name: 'process' },
+          { stepType: 'workflow_step', name: 'output' },
+        ],
+      },
+    },
+  ],
+  scorers: {
+    workflow: [outputQualityScorer],
+    steps: {
+      validate: [validationScorer],
+    },
+    trajectory: [trajectoryScorer],
+  },
+})
+
+// result.scores.trajectory — workflow trajectory scores
+```
+
 ### Workflow with per-item `startOptions`
 
 Use `startOptions` on individual data items to customize each workflow run. Per-item values take precedence over `targetOptions`:
@@ -175,5 +248,7 @@ const result = await runEvals({
 
 - [createScorer()](https://mastra.ai/reference/evals/create-scorer) - Create custom scorers for experiments
 - [MastraScorer](https://mastra.ai/reference/evals/mastra-scorer) - Learn about scorer structure and methods
+- [Trajectory Accuracy](https://mastra.ai/reference/evals/trajectory-accuracy) - Built-in trajectory evaluation scorers
+- [Scorer Utilities](https://mastra.ai/reference/evals/scorer-utils) - Helper functions for extracting trajectory data
 - [Custom Scorers](https://mastra.ai/docs/evals/custom-scorers) - Guide to building evaluation logic
 - [Scorers Overview](https://mastra.ai/docs/evals/overview) - Understanding scorer concepts

package/.docs/reference/evals/scorer-utils.md

@@ -14,9 +14,21 @@ import {
   extractToolCalls,
   extractInputMessages,
   extractAgentResponseMessages,
+  compareTrajectories,
+  createTrajectoryTestRun,
 } from '@mastra/evals/scorers/utils'
 ```
 
+Trajectory extraction functions are available from `@mastra/core/evals`:
+
+```typescript
+import {
+  extractTrajectory,
+  extractWorkflowTrajectory,
+  extractTrajectoryFromTrace,
+} from '@mastra/core/evals'
+```
+
 ## Message extraction
 
 ### `getAssistantMessageFromRunOutput`
@@ -266,6 +278,182 @@ const result = await myScorer.run({
 })
 ```
 
+## Trajectory utilities
+
+### `extractTrajectory`
+
+Extracts a `Trajectory` from agent output messages (`MastraDBMessage[]`). Converts tool invocations into `ToolCallStep` objects. The `runEvals` pipeline calls this automatically for trajectory scorers — you only need it for direct testing.
+
+Available from `@mastra/core/evals`.
+
+```typescript
+import { extractTrajectory } from '@mastra/core/evals'
+
+const trajectory = extractTrajectory(agentOutputMessages)
+// trajectory.steps — ToolCallStep[] extracted from toolInvocations
+// trajectory.rawOutput — the original MastraDBMessage[] array
+```
+
+**Returns:** `Trajectory` — Contains `steps: TrajectoryStep[]`, `totalDurationMs`, and `rawOutput`.
+
+### `extractWorkflowTrajectory`
+
+Extracts a `Trajectory` from workflow step results. Converts `StepResult` records into `WorkflowStepStep` objects, respecting the execution path ordering.
+
+Available from `@mastra/core/evals`.
+
+```typescript
+import { extractWorkflowTrajectory } from '@mastra/core/evals'
+
+const trajectory = extractWorkflowTrajectory(
+  workflowResult.steps, // Record<string, StepResult>
+  workflowResult.stepExecutionPath, // string[] (optional)
+)
+// trajectory.steps — WorkflowStepStep[] in execution order
+```
+
+**Returns:** `Trajectory` — Contains `steps: TrajectoryStep[]`, `totalDurationMs`, and `rawWorkflowResult`.
+
+### `extractTrajectoryFromTrace`
+
+Builds a hierarchical `Trajectory` from observability trace spans (`SpanRecord[]`). Reconstructs the parent-child span tree and maps each span to the appropriate `TrajectoryStep` discriminated union type with nested `children`.
+
+This is the preferred extraction method when storage is available. The `runEvals` pipeline calls this automatically when the target's `Mastra` instance has a configured storage backend. It produces richer trajectories than `extractTrajectory` or `extractWorkflowTrajectory` because it captures the full execution tree, including nested agent runs, tool calls, and model generations.
+
+Available from `@mastra/core/evals`.
+
+```typescript
+import { extractTrajectoryFromTrace } from '@mastra/core/evals'
+
+// After fetching a trace from the observability store
+const traceData = await observabilityStore.getTrace({ traceId })
+const trajectory = extractTrajectoryFromTrace(traceData.spans, rootSpanId)
+// trajectory.steps — hierarchical TrajectoryStep[] with children
+```
+
+**Parameters:**
+
+- `spans` (`SpanRecord[]`) — Array of span records from a trace query.
+- `rootSpanId` (`string`, optional) — Span ID to use as the starting point. When omitted, uses spans with no parent.
+
+**Returns:** `Trajectory` — Contains `steps: TrajectoryStep[]` with recursive `children` and `totalDurationMs`.
+
+#### Span type mapping
+
+| Span type              | Trajectory step type   | Key fields extracted                                          |
+| ---------------------- | ---------------------- | ------------------------------------------------------------- |
+| `TOOL_CALL`            | `tool_call`            | `toolArgs`, `toolResult`, `success`                           |
+| `MCP_TOOL_CALL`        | `mcp_tool_call`        | `toolArgs`, `toolResult`, `mcpServer`, `success`              |
+| `MODEL_GENERATION`     | `model_generation`     | `modelId`, `promptTokens`, `completionTokens`, `finishReason` |
+| `AGENT_RUN`            | `agent_run`            | `agentId` (from entity ID)                                    |
+| `WORKFLOW_RUN`         | `workflow_run`         | `workflowId` (from entity ID)                                 |
+| `WORKFLOW_STEP`        | `workflow_step`        | `output`                                                      |
+| `WORKFLOW_CONDITIONAL` | `workflow_conditional` | `conditionCount`, `selectedSteps`                             |
+| `WORKFLOW_PARALLEL`    | `workflow_parallel`    | `branchCount`, `parallelSteps`                                |
+| `WORKFLOW_LOOP`        | `workflow_loop`        | `loopType`, `totalIterations`                                 |
+| `WORKFLOW_SLEEP`       | `workflow_sleep`       | `sleepDurationMs`, `sleepType`                                |
+| `WORKFLOW_WAIT_EVENT`  | `workflow_wait_event`  | `eventName`, `eventReceived`                                  |
+| `PROCESSOR_RUN`        | `processor_run`        | `processorId`                                                 |
+
+Spans with types `GENERIC`, `MODEL_STEP`, `MODEL_CHUNK`, and `WORKFLOW_CONDITIONAL_EVAL` are skipped as noise.
+
+### `compareTrajectories`
+
+Compares an actual trajectory against an expected trajectory and returns a detailed comparison result. Used internally by `createTrajectoryAccuracyScorerCode`.
+
+The `expected` parameter accepts either a `Trajectory` (actual trajectory) or `{ steps: ExpectedStep[] }`. When using `ExpectedStep[]`, you can match by name only, name + stepType, or include data for comparison. See [Expected steps](https://mastra.ai/reference/evals/trajectory-accuracy) for details.
+
+```typescript
+import { compareTrajectories } from '@mastra/evals/scorers/utils'
+
+// Using ExpectedStep[] (recommended for expectations)
+// Data fields (e.g. toolArgs) are auto-compared when present on expected steps
+const result = compareTrajectories(
+  actualTrajectory,
+  { steps: [{ name: 'search' }, { name: 'summarize', stepType: 'tool_call' }] },
+  { allowRepeatedSteps: true },
+)
+// result.score — 0.0 to 1.0
+// result.missingSteps — step names not found
+// result.extraSteps — unexpected step names
+// result.outOfOrderSteps — steps found but in wrong order
+```
+
+**Returns:** `TrajectoryComparisonResult`
+
+### `createTrajectoryTestRun`
+
+Creates a test run object for trajectory scorers. Wraps a `Trajectory` into the expected `ScorerRun` format.
+
+```typescript
+import { createTrajectoryTestRun } from '@mastra/evals/scorers/utils'
+
+const run = createTrajectoryTestRun({
+  steps: [
+    { stepType: 'tool_call', name: 'search', toolArgs: { q: 'test' } },
+    { stepType: 'tool_call', name: 'summarize' },
+  ],
+})
+
+const result = await trajectoryScorer.run(run)
+```
+
+### `checkTrajectoryEfficiency`
+
+Evaluates trajectory efficiency against step, token, and duration budgets. Also detects redundant calls (same tool with same arguments).
+
+```typescript
+import { checkTrajectoryEfficiency } from '@mastra/evals/scorers/utils'
+
+const result = checkTrajectoryEfficiency(trajectory, {
+  maxSteps: 5,
+  maxTotalTokens: 2000,
+  maxTotalDurationMs: 5000,
+  noRedundantCalls: true,
+})
+// result.score — 1.0 if within all budgets, lower with penalties
+// result.redundantCalls — duplicate tool+args combos
+// result.overStepBudget — true if maxSteps exceeded
+// result.overTokenBudget — true if maxTotalTokens exceeded
+// result.overDurationBudget — true if maxTotalDurationMs exceeded
+```
+
+**Returns:** `TrajectoryEfficiencyResult`
+
+### `checkTrajectoryBlacklist`
+
+Checks whether a trajectory contains forbidden tools or tool call sequences.
+
+```typescript
+import { checkTrajectoryBlacklist } from '@mastra/evals/scorers/utils'
+
+const result = checkTrajectoryBlacklist(trajectory, {
+  blacklistedTools: ['deleteAll', 'admin-override'],
+  blacklistedSequences: [['escalate', 'admin-override']],
+})
+// result.score — 1.0 if no violations, 0.0 if any found
+// result.violatedTools — blacklisted tools that were called
+// result.violatedSequences — blacklisted sequences that were detected
+```
+
+**Returns:** `TrajectoryBlacklistResult`
+
+### `analyzeToolFailures`
+
+Detects tool failure patterns including retries, fallbacks, and argument corrections.
+
+```typescript
+import { analyzeToolFailures } from '@mastra/evals/scorers/utils'
+
+const result = analyzeToolFailures(trajectory, {
+  maxRetriesPerTool: 2,
+})
+// result.score — 1.0 if no failure patterns, lower if patterns detected
+// result.patterns — detected patterns (retry, fallback, arg_correction)
+```
+
+**Returns:** `ToolFailureAnalysisResult`
+
 ## Complete example
 
 Here's a complete example showing how to use multiple utilities together: