@mastra/evals 1.1.2 → 1.2.0-alpha.1

package/CHANGELOG.md

@@ -1,5 +1,81 @@
 # @mastra/evals
 
+## 1.2.0-alpha.1
+
+### Patch Changes
+
+- **Configurable weights**: Add `weights` option to `createTrajectoryScorerCode` for controlling how dimension scores are combined. Defaults to `{ accuracy: 0.4, efficiency: 0.3, toolFailures: 0.2, blacklist: 0.1 }`. ([#14740](https://github.com/mastra-ai/mastra/pull/14740))
+
+  ```ts
+  const scorer = createTrajectoryScorerCode({
+    defaults: { steps: [{ name: 'search' }], maxSteps: 5 },
+    weights: { accuracy: 0.6, efficiency: 0.2, toolFailures: 0.1, blacklist: 0.1 },
+  });
+  ```
+
+  **ExpectedStep redesign**: `ExpectedStep` is now a discriminated union mirroring `TrajectoryStep`. When you specify a `stepType`, you get autocomplete for that variant's fields (e.g., `toolArgs` for `tool_call`, `modelId` for `model_generation`). The old `data: Record<string, unknown>` field is replaced by direct variant fields.
+
+  ```ts
+  // Before: { name: 'search', stepType: 'tool_call', data: { input: { query: 'weather' } } }
+  // After:
+  { name: 'search', stepType: 'tool_call', toolArgs: { query: 'weather' } }
+  ```
+
+  **Remove `compareStepData`**: The `compareStepData` option is removed from `compareTrajectories`, `TrajectoryExpectation`, and all scorers. Data fields are now auto-compared when present on expected steps — if you specify `toolArgs` on an `ExpectedStep`, it will be compared against the actual step. If you omit it, only name and stepType are matched.
+
+  Also fixes documentation inaccuracies in `trajectory-accuracy.mdx` and `scorer-utils.mdx`.
+
+- Updated dependencies [[`e333b77`](https://github.com/mastra-ai/mastra/commit/e333b77e2d76ba57ccec1818e08cebc1993469ff), [`60a224d`](https://github.com/mastra-ai/mastra/commit/60a224dd497240e83698cfa5bfd02e3d1d854844), [`949b7bf`](https://github.com/mastra-ai/mastra/commit/949b7bfd4e40f2b2cba7fef5eb3f108a02cfe938), [`d084b66`](https://github.com/mastra-ai/mastra/commit/d084b6692396057e83c086b954c1857d20b58a14), [`79c699a`](https://github.com/mastra-ai/mastra/commit/79c699acf3cd8a77e11c55530431f48eb48456e9), [`62757b6`](https://github.com/mastra-ai/mastra/commit/62757b6db6e8bb86569d23ad0b514178f57053f8), [`3d70b0b`](https://github.com/mastra-ai/mastra/commit/3d70b0b3524d817173ad870768f259c06d61bd23), [`3b45a13`](https://github.com/mastra-ai/mastra/commit/3b45a138d09d040779c0aba1edbbfc1b57442d23), [`8127d96`](https://github.com/mastra-ai/mastra/commit/8127d96280492e335d49b244501088dfdd59a8f1)]:
+  - @mastra/core@1.18.0-alpha.3
+
+## 1.2.0-alpha.0
+
+### Minor Changes
+
+- **Trajectory scorers**: Added scorers for evaluating agent and workflow execution paths. ([#14697](https://github.com/mastra-ai/mastra/pull/14697))
+  - `createTrajectoryScorerCode` — unified scorer that evaluates accuracy, efficiency, blacklist violations, and tool failure patterns in a single pass. Supports per-item expectations from datasets with static defaults. Nested `ExpectedStep.children` configs allow recursive evaluation with different rules per hierarchy level.
+  - `createTrajectoryAccuracyScorerCode` — deterministic accuracy scorer with strict, relaxed, and unordered ordering modes.
+  - `createTrajectoryAccuracyScorerLLM` — LLM-based scorer for semantic trajectory evaluation.
+
+  **Utility functions:**
+  - `extractTrajectory` / `extractWorkflowTrajectory` — Convert agent runs and workflow executions into structured trajectories
+  - `extractTrajectoryFromTrace` — Build hierarchical trajectories from observability trace spans, including nested agent/tool calls
+  - `compareTrajectories` — Compare actual vs. expected trajectories with configurable ordering and data matching. Accepts `ExpectedStep[]` for simpler expected step definitions
+  - `checkTrajectoryEfficiency` — Evaluate step counts, token usage, and duration against budgets
+  - `checkTrajectoryBlacklist` — Detect forbidden tools or tool sequences
+  - `analyzeToolFailures` — Detect retry patterns, fallbacks, and argument corrections
+
+  **Example — unified scorer with defaults:**
+
+  ```ts
+  import { createTrajectoryScorerCode } from '@mastra/evals/scorers';
+
+  const scorer = createTrajectoryScorerCode({
+    defaults: {
+      ordering: 'strict',
+      steps: [
+        { name: 'validate-input' },
+        {
+          name: 'research-agent',
+          stepType: 'agent_run',
+          children: {
+            ordering: 'unordered',
+            steps: [{ name: 'search' }, { name: 'summarize' }],
+          },
+        },
+        { name: 'save-result' },
+      ],
+      maxSteps: 10,
+      blacklistedTools: ['deleteAll'],
+    },
+  });
+  ```
+
+### Patch Changes
+
+- Updated dependencies [[`dc9fc19`](https://github.com/mastra-ai/mastra/commit/dc9fc19da4437f6b508cc355f346a8856746a76b), [`260fe12`](https://github.com/mastra-ai/mastra/commit/260fe1295fe7354e39d6def2775e0797a7a277f0)]:
+  - @mastra/core@1.18.0-alpha.1
+
 ## 1.1.2
 
 ### Patch Changes
@@ -2369,7 +2445,7 @@
 - 876b8a2: Rename difference metric to textual difference metric
 - 1bbec77: Reorganized evals into nlp and llm
 - 35764f4: Added workflow for eval tests
-- 8769a62: Split core into seperate entry fils
+- 8769a62: Split core into separate entry files
 - aea3c13: Fix evals export for llm and nlp
 - 4f1d1a1: Enforce types ann cleanup package.json
 - 202d404: Added instructions when generating evals
@@ -2783,7 +2859,7 @@
 ### Patch Changes
 
 - 9625602: Use mastra core splitted bundles in other packages
-- 8769a62: Split core into seperate entry fils
+- 8769a62: Split core into separate entry files
 - Updated dependencies [30322ce]
 - Updated dependencies [78eec7c]
 - Updated dependencies [9625602]

package/LICENSE.md

@@ -1,3 +1,18 @@
+Portions of this software are licensed as follows:
+
+- All content that resides under any directory named "ee/" within this
+  repository, including but not limited to:
+  - `packages/core/src/auth/ee/`
+  - `packages/server/src/server/auth/ee/`
+    is licensed under the license defined in `ee/LICENSE`.
+
+- All third-party components incorporated into the Mastra Software are
+  licensed under the original license provided by the owner of the
+  applicable component.
+
+- Content outside of the above-mentioned directories or restrictions is
+  available under the "Apache License 2.0" as defined below.
+
 # Apache License 2.0
 
 Copyright (c) 2025 Kepler Software, Inc.

package/dist/chunk-AY4K3J4R.cjs

@@ -0,0 +1,581 @@
+'use strict';
+
+var requestContext = require('@mastra/core/request-context');
+var evals = require('@mastra/core/evals');
+
+// src/scorers/utils.ts
+function getTextContentFromMastraDBMessage(message) {
+  if (typeof message.content.content === "string" && message.content.content !== "") {
+    return message.content.content;
+  }
+  if (message.content.parts && Array.isArray(message.content.parts)) {
+    const textParts = message.content.parts.filter((p) => p.type === "text");
+    return textParts.length > 0 ? textParts[textParts.length - 1]?.text || "" : "";
+  }
+  return "";
+}
+var roundToTwoDecimals = (num) => {
+  return Math.round((num + Number.EPSILON) * 100) / 100;
+};
+function isCloserTo(value, target1, target2) {
+  return Math.abs(value - target1) < Math.abs(value - target2);
+}
+var createTestRun = (input, output, additionalContext, requestContext) => {
+  return {
+    input: [{ role: "user", content: input }],
+    output: { role: "assistant", text: output },
+    additionalContext: additionalContext ?? {},
+    requestContext: requestContext ?? {}
+  };
+};
+var getUserMessageFromRunInput = (input) => {
+  const message = input?.inputMessages.find(({ role }) => role === "user");
+  return message ? getTextContentFromMastraDBMessage(message) : void 0;
+};
+var getSystemMessagesFromRunInput = (input) => {
+  const systemMessages = [];
+  if (input?.systemMessages) {
+    systemMessages.push(
+      ...input.systemMessages.map((msg) => {
+        if (typeof msg.content === "string") {
+          return msg.content;
+        } else if (Array.isArray(msg.content)) {
+          return msg.content.filter((part) => part.type === "text").map((part) => part.text || "").join(" ");
+        }
+        return "";
+      }).filter((content) => content)
+    );
+  }
+  if (input?.taggedSystemMessages) {
+    Object.values(input.taggedSystemMessages).forEach((messages) => {
+      messages.forEach((msg) => {
+        if (typeof msg.content === "string") {
+          systemMessages.push(msg.content);
+        }
+      });
+    });
+  }
+  return systemMessages;
+};
+var getCombinedSystemPrompt = (input) => {
+  const systemMessages = getSystemMessagesFromRunInput(input);
+  return systemMessages.join("\n\n");
+};
+var getAssistantMessageFromRunOutput = (output) => {
+  const message = output?.find(({ role }) => role === "assistant");
+  return message ? getTextContentFromMastraDBMessage(message) : void 0;
+};
+var getReasoningFromRunOutput = (output) => {
+  if (!output) return void 0;
+  const message = output.find(({ role }) => role === "assistant");
+  if (!message) return void 0;
+  if (message.content.reasoning) {
+    return message.content.reasoning;
+  }
+  const reasoningParts = message.content.parts?.filter((p) => p.type === "reasoning");
+  if (reasoningParts && reasoningParts.length > 0) {
+    const reasoningTexts = reasoningParts.map((p) => {
+      if (p.details && Array.isArray(p.details)) {
+        return p.details.filter((d) => d.type === "text").map((d) => d.text).join("");
+      }
+      return p.reasoning || "";
+    }).filter(Boolean);
+    return reasoningTexts.length > 0 ? reasoningTexts.join("\n") : void 0;
+  }
+  return void 0;
+};
+var createToolInvocation = ({
+  toolCallId,
+  toolName,
+  args,
+  result,
+  state = "result"
+}) => {
+  return {
+    toolCallId,
+    toolName,
+    args,
+    result,
+    state
+  };
+};
+function createTestMessage({
+  content,
+  role,
+  id = "test-message",
+  toolInvocations = []
+}) {
+  return {
+    id,
+    role,
+    content: {
+      format: 2,
+      parts: [{ type: "text", text: content }],
+      content,
+      ...toolInvocations.length > 0 && {
+        toolInvocations: toolInvocations.map((ti) => ({
+          toolCallId: ti.toolCallId,
+          toolName: ti.toolName,
+          args: ti.args,
+          result: ti.result,
+          state: ti.state
+        }))
+      }
+    },
+    createdAt: /* @__PURE__ */ new Date()
+  };
+}
+var createAgentTestRun = ({
+  inputMessages = [],
+  output,
+  rememberedMessages = [],
+  systemMessages = [],
+  taggedSystemMessages = {},
+  requestContext: requestContext$1 = new requestContext.RequestContext(),
+  runId = crypto.randomUUID()
+}) => {
+  return {
+    input: {
+      inputMessages,
+      rememberedMessages,
+      systemMessages,
+      taggedSystemMessages
+    },
+    output,
+    requestContext: requestContext$1,
+    runId
+  };
+};
+var createTrajectoryTestRun = ({
+  inputMessages = [],
+  trajectory,
+  rememberedMessages = [],
+  systemMessages = [],
+  taggedSystemMessages = {},
+  requestContext: requestContext$1 = new requestContext.RequestContext(),
+  runId = crypto.randomUUID(),
+  expectedTrajectory
+}) => {
+  return {
+    input: {
+      inputMessages,
+      rememberedMessages,
+      systemMessages,
+      taggedSystemMessages
+    },
+    output: trajectory,
+    expectedTrajectory,
+    requestContext: requestContext$1,
+    runId
+  };
+};
+function extractToolCalls(output) {
+  const toolCalls = [];
+  const toolCallInfos = [];
+  for (let messageIndex = 0; messageIndex < output.length; messageIndex++) {
+    const message = output[messageIndex];
+    if (message?.content?.toolInvocations) {
+      for (let invocationIndex = 0; invocationIndex < message.content.toolInvocations.length; invocationIndex++) {
+        const invocation = message.content.toolInvocations[invocationIndex];
+        if (invocation && invocation.toolName && (invocation.state === "result" || invocation.state === "call")) {
+          toolCalls.push(invocation.toolName);
+          toolCallInfos.push({
+            toolName: invocation.toolName,
+            toolCallId: invocation.toolCallId || `${messageIndex}-${invocationIndex}`,
+            messageIndex,
+            invocationIndex
+          });
+        }
+      }
+    }
+  }
+  return { tools: toolCalls, toolCallInfos };
+}
+var extractInputMessages = (runInput) => {
+  return runInput?.inputMessages?.map((msg) => getTextContentFromMastraDBMessage(msg)) || [];
+};
+var extractAgentResponseMessages = (runOutput) => {
+  return runOutput.filter((msg) => msg.role === "assistant").map((msg) => getTextContentFromMastraDBMessage(msg));
+};
+function extractToolResults(output) {
+  const results = [];
+  for (const message of output) {
+    const toolInvocations = message?.content?.toolInvocations;
+    if (!toolInvocations) continue;
+    for (const invocation of toolInvocations) {
+      if (invocation.state === "result" && invocation.result !== void 0) {
+        results.push({
+          toolName: invocation.toolName,
+          toolCallId: invocation.toolCallId || "",
+          args: invocation.args || {},
+          result: invocation.result
+        });
+      }
+    }
+  }
+  return results;
+}
+function compareTrajectories(actual, expected, options = {}) {
+  const { allowRepeatedSteps = true, ordering = "relaxed" } = options;
+  const normalizedExpected = {
+    steps: expected.steps
+  };
+  if (normalizedExpected.steps.length === 0) {
+    return {
+      score: actual.steps.length === 0 ? 1 : 0,
+      matchedSteps: 0,
+      totalExpectedSteps: 0,
+      totalActualSteps: actual.steps.length,
+      missingSteps: [],
+      extraSteps: actual.steps.map((s) => s.name),
+      outOfOrderSteps: [],
+      repeatedSteps: []
+    };
+  }
+  const actualNames = actual.steps.map((s) => s.name);
+  const nameCounts = /* @__PURE__ */ new Map();
+  for (const name of actualNames) {
+    nameCounts.set(name, (nameCounts.get(name) || 0) + 1);
+  }
+  const repeatedSteps = [...nameCounts.entries()].filter(([_, count]) => count > 1).map(([name]) => name);
+  if (ordering === "strict") {
+    return compareStrictOrder(actual, normalizedExpected, { allowRepeatedSteps, repeatedSteps });
+  }
+  if (ordering === "unordered") {
+    return compareUnorderedPresence(actual, normalizedExpected, { allowRepeatedSteps, repeatedSteps });
+  }
+  return compareRelaxedOrder(actual, normalizedExpected, { allowRepeatedSteps, repeatedSteps });
+}
+function compareStrictOrder(actual, expected, opts) {
+  const actualNames = actual.steps.map((s) => s.name);
+  const expectedNames = expected.steps.map((s) => s.name);
+  let matchedSteps = 0;
+  const outOfOrderSteps = [];
+  const matchedExpectedIndices = /* @__PURE__ */ new Set();
+  const maxLen = Math.max(actualNames.length, expectedNames.length);
+  for (let i = 0; i < maxLen; i++) {
+    const actualName = actualNames[i];
+    const expectedName = expectedNames[i];
+    if (actualName === expectedName) {
+      if (actual.steps[i] && expected.steps[i]) {
+        if (expectedStepMatches(actual.steps[i], expected.steps[i])) {
+          matchedSteps++;
+          matchedExpectedIndices.add(i);
+        }
+      } else {
+        matchedSteps++;
+        matchedExpectedIndices.add(i);
+      }
+    } else if (actualName && expectedNames.includes(actualName)) {
+      outOfOrderSteps.push(actualName);
+    }
+  }
+  const missingSteps = expectedNames.filter((_, i) => !matchedExpectedIndices.has(i));
+  const extraSteps = actualNames.filter((name) => !expectedNames.includes(name));
+  let score = matchedSteps / expected.steps.length;
+  if (actualNames.length > expectedNames.length) {
+    const extraPenalty = (actualNames.length - expectedNames.length) / expectedNames.length;
+    score = Math.max(0, score - extraPenalty * 0.5);
+  }
+  if (!opts.allowRepeatedSteps && opts.repeatedSteps.length > 0) {
+    score = Math.max(0, score - opts.repeatedSteps.length * 0.1);
+  }
+  return {
+    score: roundToTwoDecimals(Math.max(0, Math.min(1, score))),
+    matchedSteps,
+    totalExpectedSteps: expected.steps.length,
+    totalActualSteps: actual.steps.length,
+    missingSteps,
+    extraSteps,
+    outOfOrderSteps,
+    repeatedSteps: opts.repeatedSteps
+  };
+}
+function compareRelaxedOrder(actual, expected, opts) {
+  const actualNames = actual.steps.map((s) => s.name);
+  const expectedNames = expected.steps.map((s) => s.name);
+  let matchedSteps = 0;
+  let lastMatchedIndex = -1;
+  const outOfOrderSteps = [];
+  const matchedExpectedIndices = /* @__PURE__ */ new Set();
+  for (let i = 0; i < expectedNames.length; i++) {
+    const expectedName = expectedNames[i];
+    let found = false;
+    for (let j = lastMatchedIndex + 1; j < actualNames.length; j++) {
+      if (actualNames[j] === expectedName) {
+        if (actual.steps[j] && expected.steps[i]) {
+          if (expectedStepMatches(actual.steps[j], expected.steps[i])) {
+            matchedSteps++;
+            lastMatchedIndex = j;
+            matchedExpectedIndices.add(i);
+            found = true;
+            break;
+          }
+        } else {
+          matchedSteps++;
+          lastMatchedIndex = j;
+          matchedExpectedIndices.add(i);
+          found = true;
+          break;
+        }
+      }
+    }
+    if (!found) {
+      if (actualNames.includes(expectedName)) {
+        outOfOrderSteps.push(expectedName);
+      }
+    }
+  }
+  const missingSteps = expectedNames.filter((_, i) => !matchedExpectedIndices.has(i));
+  const expectedSet = new Set(expectedNames);
+  const extraSteps = actualNames.filter((name) => !expectedSet.has(name));
+  let score = matchedSteps / expected.steps.length;
+  if (!opts.allowRepeatedSteps && opts.repeatedSteps.length > 0) {
+    score = Math.max(0, score - opts.repeatedSteps.length * 0.1);
+  }
+  return {
+    score: roundToTwoDecimals(Math.max(0, Math.min(1, score))),
+    matchedSteps,
+    totalExpectedSteps: expected.steps.length,
+    totalActualSteps: actual.steps.length,
+    missingSteps,
+    extraSteps,
+    outOfOrderSteps,
+    repeatedSteps: opts.repeatedSteps
+  };
+}
+var COMPARABLE_FIELDS_BY_TYPE = {
+  tool_call: ["toolArgs", "toolResult", "success"],
+  mcp_tool_call: ["toolArgs", "toolResult", "mcpServer", "success"],
+  model_generation: ["modelId", "promptTokens", "completionTokens", "finishReason"],
+  agent_run: ["agentId"],
+  workflow_step: ["stepId", "status", "output"],
+  workflow_run: ["workflowId", "status"],
+  workflow_conditional: ["conditionCount", "selectedSteps"],
+  workflow_parallel: ["branchCount", "parallelSteps"],
+  workflow_loop: ["loopType", "totalIterations"],
+  workflow_sleep: ["sleepDurationMs", "sleepType"],
+  workflow_wait_event: ["eventName", "eventReceived"],
+  processor_run: ["processorId"]
+};
+function expectedStepMatches(actual, expected) {
+  if (actual.name !== expected.name) return false;
+  if (expected.stepType && actual.stepType !== expected.stepType) return false;
+  if (expected.stepType) {
+    const fields = COMPARABLE_FIELDS_BY_TYPE[expected.stepType] ?? [];
+    for (const field of fields) {
+      const expectedVal = expected[field];
+      if (expectedVal === void 0) continue;
+      const actualVal = actual[field];
+      if (actualVal === void 0) return false;
+      try {
+        if (JSON.stringify(actualVal) !== JSON.stringify(expectedVal)) return false;
+      } catch {
+        return false;
+      }
+    }
+  }
+  return true;
+}
+function compareUnorderedPresence(actual, expected, opts) {
+  const actualNames = actual.steps.map((s) => s.name);
+  const expectedNames = expected.steps.map((s) => s.name);
+  let matchedSteps = 0;
+  const matchedExpectedIndices = /* @__PURE__ */ new Set();
+  const usedIndices = /* @__PURE__ */ new Set();
+  for (let i = 0; i < expected.steps.length; i++) {
+    const expectedStep = expected.steps[i];
+    for (let j = 0; j < actual.steps.length; j++) {
+      if (!usedIndices.has(j) && expectedStepMatches(actual.steps[j], expectedStep)) {
+        matchedSteps++;
+        matchedExpectedIndices.add(i);
+        usedIndices.add(j);
+        break;
+      }
+    }
+  }
+  const missingSteps = expectedNames.filter((_, i) => !matchedExpectedIndices.has(i));
+  const expectedSet = new Set(expectedNames);
+  const extraSteps = actualNames.filter((name) => !expectedSet.has(name));
+  let score = matchedSteps / expected.steps.length;
+  if (!opts.allowRepeatedSteps && opts.repeatedSteps.length > 0) {
+    score = Math.max(0, score - opts.repeatedSteps.length * 0.1);
+  }
+  return {
+    score: roundToTwoDecimals(Math.max(0, Math.min(1, score))),
+    matchedSteps,
+    totalExpectedSteps: expected.steps.length,
+    totalActualSteps: actual.steps.length,
+    missingSteps,
+    extraSteps,
+    outOfOrderSteps: [],
+    // ordering not checked in unordered mode
+    repeatedSteps: opts.repeatedSteps
+  };
+}
+function checkTrajectoryEfficiency(trajectory, options = {}) {
+  const { maxSteps, maxTotalTokens, maxTotalDurationMs, noRedundantCalls = true } = options;
+  const totalSteps = trajectory.steps.length;
+  let totalTokens = 0;
+  for (const step of trajectory.steps) {
+    if (step.stepType === "model_generation") {
+      totalTokens += (step.promptTokens ?? 0) + (step.completionTokens ?? 0);
+    }
+  }
+  const totalDurationMs = trajectory.totalDurationMs ?? trajectory.steps.reduce((sum, s) => sum + (s.durationMs ?? 0), 0);
+  const redundantCalls = [];
+  if (noRedundantCalls) {
+    for (let i = 1; i < trajectory.steps.length; i++) {
+      const prev = trajectory.steps[i - 1];
+      const curr = trajectory.steps[i];
+      if (prev.name === curr.name && prev.stepType === curr.stepType && (prev.stepType === "tool_call" || prev.stepType === "mcp_tool_call")) {
+        const prevArgs = prev.toolArgs;
+        const currArgs = curr.toolArgs;
+        try {
+          if (JSON.stringify(prevArgs) === JSON.stringify(currArgs)) {
+            redundantCalls.push({ name: curr.name, index: i });
+          }
+        } catch {
+        }
+      }
+    }
+  }
+  const overStepBudget = maxSteps !== void 0 && totalSteps > maxSteps;
+  const overTokenBudget = maxTotalTokens !== void 0 && totalTokens > maxTotalTokens;
+  const overDurationBudget = maxTotalDurationMs !== void 0 && totalDurationMs > maxTotalDurationMs;
+  const dimensions = [];
+  if (maxSteps !== void 0) {
+    dimensions.push(overStepBudget ? Math.max(0, 1 - (totalSteps - maxSteps) / maxSteps) : 1);
+  }
+  if (maxTotalTokens !== void 0) {
+    dimensions.push(overTokenBudget ? Math.max(0, 1 - (totalTokens - maxTotalTokens) / maxTotalTokens) : 1);
+  }
+  if (maxTotalDurationMs !== void 0) {
+    dimensions.push(
+      overDurationBudget ? Math.max(0, 1 - (totalDurationMs - maxTotalDurationMs) / maxTotalDurationMs) : 1
+    );
+  }
+  if (noRedundantCalls) {
+    dimensions.push(redundantCalls.length === 0 ? 1 : Math.max(0, 1 - redundantCalls.length * 0.2));
+  }
+  const score = dimensions.length > 0 ? dimensions.reduce((a, b) => a + b, 0) / dimensions.length : 1;
+  return {
+    score: roundToTwoDecimals(Math.max(0, Math.min(1, score))),
+    totalSteps,
+    overStepBudget,
+    totalTokens,
+    overTokenBudget,
+    totalDurationMs,
+    overDurationBudget,
+    redundantCalls
+  };
+}
+function checkTrajectoryBlacklist(trajectory, options = {}) {
+  const { blacklistedTools = [], blacklistedSequences = [] } = options;
+  const violatedTools = [];
+  const violatedSequences = [];
+  const stepNames = trajectory.steps.map((s) => s.name);
+  for (const forbidden of blacklistedTools) {
+    if (stepNames.includes(forbidden)) {
+      violatedTools.push(forbidden);
+    }
+  }
+  for (const sequence of blacklistedSequences) {
+    if (sequence.length === 0) continue;
+    for (let i = 0; i <= stepNames.length - sequence.length; i++) {
+      let match = true;
+      for (let j = 0; j < sequence.length; j++) {
+        if (stepNames[i + j] !== sequence[j]) {
+          match = false;
+          break;
+        }
+      }
+      if (match) {
+        violatedSequences.push(sequence);
+        break;
+      }
+    }
+  }
+  const hasViolations = violatedTools.length > 0 || violatedSequences.length > 0;
+  return {
+    score: hasViolations ? 0 : 1,
+    violatedTools,
+    violatedSequences
+  };
+}
+function analyzeToolFailures(trajectory, options = {}) {
+  const { maxRetriesPerTool = 2 } = options;
+  const patterns = [];
+  let totalRetries = 0;
+  const toolCallSteps = trajectory.steps.filter((s) => s.stepType === "tool_call" || s.stepType === "mcp_tool_call");
+  if (toolCallSteps.length === 0) {
+    return { score: 1, patterns: [], totalRetries: 0, excessiveRetryTools: [] };
+  }
+  let i = 0;
+  while (i < toolCallSteps.length) {
+    const currentTool = toolCallSteps[i];
+    let retryCount = 0;
+    let j = i + 1;
+    while (j < toolCallSteps.length && toolCallSteps[j].name === currentTool.name) {
+      const prevStep = toolCallSteps[j - 1];
+      if (prevStep.success === false) {
+        retryCount++;
+      }
+      j++;
+    }
+    if (retryCount > 0) {
+      const nextDifferentTool = j < toolCallSteps.length ? toolCallSteps[j] : void 0;
+      const lastRetry = toolCallSteps[j - 1];
+      const lastSuccess = lastRetry.success !== false;
+      patterns.push({
+        toolName: currentTool.name,
+        retryCount,
+        fellBackToAlternative: nextDifferentTool !== void 0 && !lastSuccess,
+        alternativeTool: nextDifferentTool !== void 0 && !lastSuccess ? nextDifferentTool.name : void 0,
+        eventuallySucceeded: lastSuccess
+      });
+      totalRetries += retryCount;
+    }
+    i = j;
+  }
+  const excessiveRetryTools = patterns.filter((p) => p.retryCount > maxRetriesPerTool).map((p) => p.toolName);
+  let score = 1;
+  if (toolCallSteps.length > 0) {
+    const excessRetries = patterns.reduce((sum, p) => sum + Math.max(0, p.retryCount - maxRetriesPerTool), 0);
+    score = Math.max(0, 1 - excessRetries * 0.2);
+  }
+  return {
+    score: roundToTwoDecimals(Math.max(0, Math.min(1, score))),
+    patterns,
+    totalRetries,
+    excessiveRetryTools
+  };
+}
+
+Object.defineProperty(exports, "extractTrajectory", {
+  enumerable: true,
+  get: function () { return evals.extractTrajectory; }
+});
+exports.analyzeToolFailures = analyzeToolFailures;
+exports.checkTrajectoryBlacklist = checkTrajectoryBlacklist;
+exports.checkTrajectoryEfficiency = checkTrajectoryEfficiency;
+exports.compareTrajectories = compareTrajectories;
+exports.createAgentTestRun = createAgentTestRun;
+exports.createTestMessage = createTestMessage;
+exports.createTestRun = createTestRun;
+exports.createToolInvocation = createToolInvocation;
+exports.createTrajectoryTestRun = createTrajectoryTestRun;
+exports.extractAgentResponseMessages = extractAgentResponseMessages;
+exports.extractInputMessages = extractInputMessages;
+exports.extractToolCalls = extractToolCalls;
+exports.extractToolResults = extractToolResults;
+exports.getAssistantMessageFromRunOutput = getAssistantMessageFromRunOutput;
+exports.getCombinedSystemPrompt = getCombinedSystemPrompt;
+exports.getReasoningFromRunOutput = getReasoningFromRunOutput;
+exports.getSystemMessagesFromRunInput = getSystemMessagesFromRunInput;
+exports.getTextContentFromMastraDBMessage = getTextContentFromMastraDBMessage;
+exports.getUserMessageFromRunInput = getUserMessageFromRunInput;
+exports.isCloserTo = isCloserTo;
+exports.roundToTwoDecimals = roundToTwoDecimals;
+//# sourceMappingURL=chunk-AY4K3J4R.cjs.map
+//# sourceMappingURL=chunk-AY4K3J4R.cjs.map