@mastra/evals 0.0.0-error-handler-fix-20251020202607 → 0.0.0-execa-dynamic-import-20260304221256

package/README.md

@@ -1,6 +1,11 @@
 # @mastra/evals
 
-A comprehensive evaluation framework for assessing AI model outputs across multiple dimensions.
+`@mastra/evals` ships a collection of scoring utilities you can run locally or inside your own evaluation pipelines. These scorers come in two flavors:
+
+- **LLM scorers** – leverage a judge model (e.g. OpenAI, Anthropic) to rate responses for qualities such as faithfulness or toxicity.
+- **Code/NLP scorers** – deterministic heuristics (keyword coverage, similarity, etc.) that do not require an external model.
+
+The scorers do not persist results or integrate with Mastra Storage; you decide where and how to record outcomes.
 
 ## Installation
 
@@ -8,171 +13,26 @@ A comprehensive evaluation framework for assessing AI model outputs across multi
 npm install @mastra/evals
 ```
 
-## Overview
-
-`@mastra/evals` provides a suite of evaluation metrics for assessing AI model outputs. The package includes both LLM-based and NLP-based metrics, enabling both automated and model-assisted evaluation of AI responses.
-
-## Features
-
-### LLM-Based Metrics
-
-1. **Answer Relevancy**
-   - Evaluates how well an answer addresses the input question
-   - Considers uncertainty weighting for more nuanced scoring
-   - Returns detailed reasoning for scores
-
-2. **Bias Detection**
-   - Identifies potential biases in model outputs
-   - Analyzes opinions and statements for bias indicators
-   - Provides explanations for detected biases
-   - Configurable scoring scale
-
-3. **Context Precision & Relevancy**
-   - Assesses how well responses use provided context
-   - Evaluates accuracy of context usage
-   - Measures relevance of context to the response
-   - Analyzes context positioning in responses
-
-4. **Faithfulness**
-   - Verifies that responses are faithful to provided context
-   - Detects hallucinations or fabricated information
-   - Evaluates claims against provided context
-   - Provides detailed analysis of faithfulness breaches
-
-5. **Prompt Alignment**
-   - Measures how well responses follow given instructions
-   - Evaluates adherence to multiple instruction criteria
-   - Provides per-instruction scoring
-   - Supports custom instruction sets
-
-6. **Toxicity**
-   - Detects toxic or harmful content in responses
-   - Provides detailed reasoning for toxicity verdicts
-   - Configurable scoring thresholds
-   - Considers both input and output context
+## Quick Start
 
-### NLP-Based Metrics
+```ts
+import { createFaithfulnessScorer, createContentSimilarityScorer } from '@mastra/evals/scorers/prebuilt';
 
-1. **Completeness**
-   - Analyzes structural completeness of responses
-   - Identifies missing elements from input requirements
-   - Provides detailed element coverage analysis
-   - Tracks input-output element ratios
-
-2. **Content Similarity**
-   - Measures text similarity between inputs and outputs
-   - Configurable for case and whitespace sensitivity
-   - Returns normalized similarity scores
-   - Uses string comparison algorithms for accuracy
-
-3. **Keyword Coverage**
-   - Tracks presence of key terms from input in output
-   - Provides detailed keyword matching statistics
-   - Calculates coverage ratios
-   - Useful for ensuring comprehensive responses
-
-## Usage
-
-### Basic Example
-
-```typescript
-import { ContentSimilarityMetric, ToxicityMetric } from '@mastra/evals';
-
-// Initialize metrics
-const similarityMetric = new ContentSimilarityMetric({
-  ignoreCase: true,
-  ignoreWhitespace: true,
+const faithfulness = createFaithfulnessScorer({
+   model: 'openai/gpt-4o-mini')
 });
 
-const toxicityMetric = new ToxicityMetric({
-  model: openai('gpt-4'),
-  scale: 1, // Optional: adjust scoring scale
-});
-
-// Evaluate outputs
-const input = 'What is the capital of France?';
-const output = 'Paris is the capital of France.';
-
-const similarityResult = await similarityMetric.measure(input, output);
-const toxicityResult = await toxicityMetric.measure(input, output);
-
-console.log('Similarity Score:', similarityResult.score);
-console.log('Toxicity Score:', toxicityResult.score);
-```
+const similarity = createContentSimilarityScorer({ ignoreCase: true });
 
-### Context-Aware Evaluation
+const answer = 'Paris is the capital of France.';
+const context = ['Paris is the capital of France', 'France is in Europe'];
 
-```typescript
-import { FaithfulnessMetric } from '@mastra/evals';
+const faithfulnessScore = await faithfulness.score({ answer, context });
 
-// Initialize with context
-const faithfulnessMetric = new FaithfulnessMetric({
-  model: openai('gpt-4'),
-  context: ['Paris is the capital of France', 'Paris has a population of 2.2 million'],
-  scale: 1,
+const similarityScore = similarity.score({
+   input: context[0],
+   output: answer
 });
 
-// Evaluate response against context
-const result = await faithfulnessMetric.measure(
-  'Tell me about Paris',
-  'Paris is the capital of France with 2.2 million residents',
-);
-
-console.log('Faithfulness Score:', result.score);
-console.log('Reasoning:', result.reason);
+console.log({ faithfulnessScore, similarityScore });
 ```
-
-## Metric Results
-
-Each metric returns a standardized result object containing:
-
-- `score`: Normalized score (typically 0-1)
-- `info`: Detailed information about the evaluation
-- Additional metric-specific data (e.g., matched keywords, missing elements)
-
-Some metrics also provide:
-
-- `reason`: Detailed explanation of the score
-- `verdicts`: Individual judgments that contributed to the final score
-
-## Telemetry and Logging
-
-The package includes built-in telemetry and logging capabilities:
-
-- Automatic evaluation tracking through Mastra Storage
-- Integration with OpenTelemetry for performance monitoring
-- Detailed evaluation traces for debugging
-
-```typescript
-import { attachListeners } from '@mastra/evals';
-
-// Enable basic evaluation tracking
-await attachListeners();
-
-// Store evals in Mastra Storage (if storage is enabled)
-await attachListeners(mastra);
-// Note: When using in-memory storage, evaluations are isolated to the test process.
-// When using file storage, evaluations are persisted and can be queried later.
-```
-
-## Environment Variables
-
-Required for LLM-based metrics:
-
-- `OPENAI_API_KEY`: For OpenAI model access
-- Additional provider keys as needed (Cohere, Anthropic, etc.)
-
-## Package Exports
-
-```typescript
-// Main package exports
-import { evaluate } from '@mastra/evals';
-// NLP-specific metrics
-import { ContentSimilarityMetric } from '@mastra/evals/nlp';
-```
-
-## Related Packages
-
-- `@mastra/core`: Core framework functionality
-- `@mastra/engine`: LLM execution engine
-- `@mastra/mcp`: Model Context Protocol integration

package/dist/chunk-OEOE7ZHN.js

@@ -0,0 +1,195 @@
+import { RequestContext } from '@mastra/core/request-context';
+
+// src/scorers/utils.ts
+function getTextContentFromMastraDBMessage(message) {
+  if (typeof message.content.content === "string" && message.content.content !== "") {
+    return message.content.content;
+  }
+  if (message.content.parts && Array.isArray(message.content.parts)) {
+    const textParts = message.content.parts.filter((p) => p.type === "text");
+    return textParts.length > 0 ? textParts[textParts.length - 1]?.text || "" : "";
+  }
+  return "";
+}
+var roundToTwoDecimals = (num) => {
+  return Math.round((num + Number.EPSILON) * 100) / 100;
+};
+function isCloserTo(value, target1, target2) {
+  return Math.abs(value - target1) < Math.abs(value - target2);
+}
+var createTestRun = (input, output, additionalContext, requestContext) => {
+  return {
+    input: [{ role: "user", content: input }],
+    output: { role: "assistant", text: output },
+    additionalContext: additionalContext ?? {},
+    requestContext: requestContext ?? {}
+  };
+};
+var getUserMessageFromRunInput = (input) => {
+  const message = input?.inputMessages.find(({ role }) => role === "user");
+  return message ? getTextContentFromMastraDBMessage(message) : void 0;
+};
+var getSystemMessagesFromRunInput = (input) => {
+  const systemMessages = [];
+  if (input?.systemMessages) {
+    systemMessages.push(
+      ...input.systemMessages.map((msg) => {
+        if (typeof msg.content === "string") {
+          return msg.content;
+        } else if (Array.isArray(msg.content)) {
+          return msg.content.filter((part) => part.type === "text").map((part) => part.text || "").join(" ");
+        }
+        return "";
+      }).filter((content) => content)
+    );
+  }
+  if (input?.taggedSystemMessages) {
+    Object.values(input.taggedSystemMessages).forEach((messages) => {
+      messages.forEach((msg) => {
+        if (typeof msg.content === "string") {
+          systemMessages.push(msg.content);
+        }
+      });
+    });
+  }
+  return systemMessages;
+};
+var getCombinedSystemPrompt = (input) => {
+  const systemMessages = getSystemMessagesFromRunInput(input);
+  return systemMessages.join("\n\n");
+};
+var getAssistantMessageFromRunOutput = (output) => {
+  const message = output?.find(({ role }) => role === "assistant");
+  return message ? getTextContentFromMastraDBMessage(message) : void 0;
+};
+var getReasoningFromRunOutput = (output) => {
+  if (!output) return void 0;
+  const message = output.find(({ role }) => role === "assistant");
+  if (!message) return void 0;
+  if (message.content.reasoning) {
+    return message.content.reasoning;
+  }
+  const reasoningParts = message.content.parts?.filter((p) => p.type === "reasoning");
+  if (reasoningParts && reasoningParts.length > 0) {
+    const reasoningTexts = reasoningParts.map((p) => {
+      if (p.details && Array.isArray(p.details)) {
+        return p.details.filter((d) => d.type === "text").map((d) => d.text).join("");
+      }
+      return p.reasoning || "";
+    }).filter(Boolean);
+    return reasoningTexts.length > 0 ? reasoningTexts.join("\n") : void 0;
+  }
+  return void 0;
+};
+var createToolInvocation = ({
+  toolCallId,
+  toolName,
+  args,
+  result,
+  state = "result"
+}) => {
+  return {
+    toolCallId,
+    toolName,
+    args,
+    result,
+    state
+  };
+};
+function createTestMessage({
+  content,
+  role,
+  id = "test-message",
+  toolInvocations = []
+}) {
+  return {
+    id,
+    role,
+    content: {
+      format: 2,
+      parts: [{ type: "text", text: content }],
+      content,
+      ...toolInvocations.length > 0 && {
+        toolInvocations: toolInvocations.map((ti) => ({
+          toolCallId: ti.toolCallId,
+          toolName: ti.toolName,
+          args: ti.args,
+          result: ti.result,
+          state: ti.state
+        }))
+      }
+    },
+    createdAt: /* @__PURE__ */ new Date()
+  };
+}
+var createAgentTestRun = ({
+  inputMessages = [],
+  output,
+  rememberedMessages = [],
+  systemMessages = [],
+  taggedSystemMessages = {},
+  requestContext = new RequestContext(),
+  runId = crypto.randomUUID()
+}) => {
+  return {
+    input: {
+      inputMessages,
+      rememberedMessages,
+      systemMessages,
+      taggedSystemMessages
+    },
+    output,
+    requestContext,
+    runId
+  };
+};
+function extractToolCalls(output) {
+  const toolCalls = [];
+  const toolCallInfos = [];
+  for (let messageIndex = 0; messageIndex < output.length; messageIndex++) {
+    const message = output[messageIndex];
+    if (message?.content?.toolInvocations) {
+      for (let invocationIndex = 0; invocationIndex < message.content.toolInvocations.length; invocationIndex++) {
+        const invocation = message.content.toolInvocations[invocationIndex];
+        if (invocation && invocation.toolName && (invocation.state === "result" || invocation.state === "call")) {
+          toolCalls.push(invocation.toolName);
+          toolCallInfos.push({
+            toolName: invocation.toolName,
+            toolCallId: invocation.toolCallId || `${messageIndex}-${invocationIndex}`,
+            messageIndex,
+            invocationIndex
+          });
+        }
+      }
+    }
+  }
+  return { tools: toolCalls, toolCallInfos };
+}
+var extractInputMessages = (runInput) => {
+  return runInput?.inputMessages?.map((msg) => getTextContentFromMastraDBMessage(msg)) || [];
+};
+var extractAgentResponseMessages = (runOutput) => {
+  return runOutput.filter((msg) => msg.role === "assistant").map((msg) => getTextContentFromMastraDBMessage(msg));
+};
+function extractToolResults(output) {
+  const results = [];
+  for (const message of output) {
+    const toolInvocations = message?.content?.toolInvocations;
+    if (!toolInvocations) continue;
+    for (const invocation of toolInvocations) {
+      if (invocation.state === "result" && invocation.result !== void 0) {
+        results.push({
+          toolName: invocation.toolName,
+          toolCallId: invocation.toolCallId || "",
+          args: invocation.args || {},
+          result: invocation.result
+        });
+      }
+    }
+  }
+  return results;
+}
+
+export { createAgentTestRun, createTestMessage, createTestRun, createToolInvocation, extractAgentResponseMessages, extractInputMessages, extractToolCalls, extractToolResults, getAssistantMessageFromRunOutput, getCombinedSystemPrompt, getReasoningFromRunOutput, getSystemMessagesFromRunInput, getTextContentFromMastraDBMessage, getUserMessageFromRunInput, isCloserTo, roundToTwoDecimals };
+//# sourceMappingURL=chunk-OEOE7ZHN.js.map
+//# sourceMappingURL=chunk-OEOE7ZHN.js.map

package/dist/chunk-OEOE7ZHN.js.map

@@ -0,0 +1 @@
+{"version":3,"sources":["../src/scorers/utils.ts"],"names":[],"mappings":";;;AAyBO,SAAS,kCAAkC,OAAA,EAAkC;AAClF,EAAA,IAAI,OAAO,QAAQ,OAAA,CAAQ,OAAA,KAAY,YAAY,OAAA,CAAQ,OAAA,CAAQ,YAAY,EAAA,EAAI;AACjF,IAAA,OAAO,QAAQ,OAAA,CAAQ,OAAA;AAAA,EACzB;AACA,EAAA,IAAI,OAAA,CAAQ,QAAQ,KAAA,IAAS,KAAA,CAAM,QAAQ,OAAA,CAAQ,OAAA,CAAQ,KAAK,CAAA,EAAG;AAEjE,IAAA,MAAM,SAAA,GAAY,QAAQ,OAAA,CAAQ,KAAA,CAAM,OAAO,CAAA,CAAA,KAAK,CAAA,CAAE,SAAS,MAAM,CAAA;AACrE,IAAA,OAAO,SAAA,CAAU,SAAS,CAAA,GAAI,SAAA,CAAU,UAAU,MAAA,GAAS,CAAC,CAAA,EAAG,IAAA,IAAQ,EAAA,GAAK,EAAA;AAAA,EAC9E;AACA,EAAA,OAAO,EAAA;AACT;AAgBO,IAAM,kBAAA,GAAqB,CAAC,GAAA,KAAgB;AACjD,EAAA,OAAO,KAAK,KAAA,CAAA,CAAO,GAAA,GAAM,MAAA,CAAO,OAAA,IAAW,GAAG,CAAA,GAAI,GAAA;AACpD;AAgBO,SAAS,UAAA,CAAW,KAAA,EAAe,OAAA,EAAiB,OAAA,EAA0B;AACnF,EAAA,OAAO,IAAA,CAAK,IAAI,KAAA,GAAQ,OAAO,IAAI,IAAA,CAAK,GAAA,CAAI,QAAQ,OAAO,CAAA;AAC7D;AA6CO,IAAM,aAAA,GAAgB,CAC3B,KAAA,EACA,MAAA,EACA,mBACA,cAAA,KACiB;AACjB,EAAA,OAAO;AAAA,IACL,OAAO,CAAC,EAAE,MAAM,MAAA,EAAQ,OAAA,EAAS,OAAO,CAAA;AAAA,IACxC,MAAA,EAAQ,EAAE,IAAA,EAAM,WAAA,EAAa,MAAM,MAAA,EAAO;AAAA,IAC1C,iBAAA,EAAmB,qBAAqB,EAAC;AAAA,IACzC,cAAA,EAAgB,kBAAkB;AAAC,GACrC;AACF;AAmBO,IAAM,0BAAA,GAA6B,CAAC,KAAA,KAAuD;AAChG,EAAA,MAAM,OAAA,GAAU,OAAO,aAAA,CAAc,IAAA,CAAK,CAAC,EAAE,IAAA,EAAK,KAAM,IAAA,KAAS,MAAM,CAAA;AACvE,EAAA,OAAO,OAAA,GAAU,iCAAA,CAAkC,OAAO,CAAA,GAAI,MAAA;AAChE;AAoBO,IAAM,6BAAA,GAAgC,CAAC,KAAA,KAA6C;AACzF,EAAA,MAAM,iBAA2B,EAAC;AAGlC,EAAA,IAAI,OAAO,cAAA,EAAgB;AACzB,IAAA,cAAA,CAAe,IAAA;AAAA,MACb,GAAG,KAAA,CAAM,cAAA,CACN,GAAA,CAAI,CAAA,GAAA,KAAO;AAEV,QAAA,IAAI,OAAO,GAAA,CAAI,OAAA,KAAY,QAAA,EAAU;AACnC,UAAA,OAAO,GAAA,CAAI,OAAA;AAAA,QACb,CAAA,MAAA,IAAW,KAAA,CAAM,OAAA,CAAQ,GAAA,CAAI,OAAO,CAAA,EAAG;AAErC,UAAA,OAAO,IAAI,OAAA,CACR,MAAA,CAAO,CAAC,IAAA,KAAc,KAAK,IAAA,KAAS,MAAM,CAAA,CAC1C,GAAA,CAAI,CAAC,IAAA,KAAc,IAAA,CAAK,QAAQ,EAAE,CAAA,CAClC,KAAK,GAAG,CAAA;AAAA,QACb;AACA,QAAA,OAAO,EAAA;AAAA,MACT,CAAC,CAAA,CACA,MAAA,CAAO,CAAA,OAAA,KAAW,OAAO;AAAA,KAC9B;AAAA,EACF;AAGA,EAAA,IAAI,OAAO,oBAAA,EAAsB;AAC/B,IAAA,MAAA,CAAO,MAAA,CAAO,KAAA,CAAM,oBAAoB,CAAA,CAAE,QAAQ,CAAA,QAAA,KAAY;AAC5D,MAAA,QAAA,CAAS,QAAQ,CAAA,GAAA,KAAO;AACtB,QAAA,IAAI,OAAO,GAAA,CAAI,OAAA,KAAY,QAAA,EAAU;AACnC,UAAA,cAAA,CAAe,IAAA,CAAK,IAAI,OAAO,CAAA;AAAA,QACjC;AAAA,MACF,CAAC,CAAA;AAAA,IACH,CAAC,CAAA;AAAA,EACH;AAEA,EAAA,OAAO,cAAA;AACT;AAmBO,IAAM,uBAAA,GAA0B,CAAC,KAAA,KAA2C;AACjF,EAAA,MAAM,cAAA,GAAiB,8BAA8B,KAAK,CAAA;AAC1D,EAAA,OAAO,cAAA,CAAe,KAAK,MAAM,CAAA;AACnC;AAmBO,IAAM,gCAAA,GAAmC,CAAC,MAAA,KAAqC;AACpF,EAAA,MAAM,OAAA,GAAU,QAAQ,IAAA,CAAK,CAAC,EAAE,IAAA,EAAK,KAAM,SAAS,WAAW,CAAA;AAC/D,EAAA,OAAO,OAAA,GAAU,iCAAA,CAAkC,OAAO,CAAA,GAAI,MAAA;AAChE;AAiCO,IAAM,yBAAA,GAA4B,CAAC,MAAA,KAAyD;AACjG,EAAA,IAAI,CAAC,QAAQ,OAAO,MAAA;AAEpB,EAAA,MAAM,OAAA,GAAU,OAAO,IAAA,CAAK,CAAC,EAAE,IAAA,EAAK,KAAM,SAAS,WAAW,CAAA;AAC9D,EAAA,IAAI,CAAC,SAAS,OAAO,MAAA;AAGrB,EAAA,IAAI,OAAA,CAAQ,QAAQ,SAAA,EAAW;AAC7B,IAAA,OAAO,QAAQ,OAAA,CAAQ,SAAA;AAAA,EACzB;AAIA,EAAA,MAAM,cAAA,GAAiB,QAAQ,OAAA,CAAQ,KAAA,EAAO,OAAO,CAAC,CAAA,KAAW,CAAA,CAAE,IAAA,KAAS,WAAW,CAAA;AACvF,EAAA,IAAI,cAAA,IAAkB,cAAA,CAAe,MAAA,GAAS,CAAA,EAAG;AAC/C,IAAA,MAAM,cAAA,GAAiB,cAAA,CACpB,GAAA,CAAI,CAAC,CAAA,KAAW;AAEf,MAAA,IAAI,EAAE,OAAA,IAAW,KAAA,CAAM,OAAA,CAAQ,CAAA,CAAE,OAAO,CAAA,EAAG;AACzC,QAAA,OAAO,EAAE,OAAA,CACN,MAAA,CAAO,CAAC,CAAA,KAAW,EAAE,IAAA,KAAS,MAAM,CAAA,CACpC,GAAA,CAAI,CAAC,CAAA,KAAW,CAAA,CAAE,IAAI,CAAA,CACtB,KAAK,EAAE,CAAA;AAAA,MACZ;AACA,MAAA,OAAO,EAAE,SAAA,IAAa,EAAA;AAAA,IACxB,CAAC,CAAA,CACA,MAAA,CAAO,OAAO,CAAA;AAEjB,IAAA,OAAO,eAAe,MAAA,GAAS,CAAA,GAAI,cAAA,CAAe,IAAA,CAAK,IAAI,CAAA,GAAI,MAAA;AAAA,EACjE;AAEA,EAAA,OAAO,MAAA;AACT;AAuBO,IAAM,uBAAuB,CAAC;AAAA,EACnC,UAAA;AAAA,EACA,QAAA;AAAA,EACA,IAAA;AAAA,EACA,MAAA;AAAA,EACA,KAAA,GAAQ;AACV,CAAA,KAMuH;AACrH,EAAA,OAAO;AAAA,IACL,UAAA;AAAA,IACA,QAAA;AAAA,IACA,IAAA;AAAA,IACA,MAAA;AAAA,IACA;AAAA,GACF;AACF;AAmCO,SAAS,iBAAA,CAAkB;AAAA,EAChC,OAAA;AAAA,EACA,IAAA;AAAA,EACA,EAAA,GAAK,cAAA;AAAA,EACL,kBAAkB;AACpB,CAAA,EAWoB;AAClB,EAAA,OAAO;AAAA,IACL,EAAA;AAAA,IACA,IAAA;AAAA,IACA,OAAA,EAAS;AAAA,MACP,MAAA,EAAQ,CAAA;AAAA,MACR,OAAO,CAAC,EAAE,MAAM,MAAA,EAAQ,IAAA,EAAM,SAAS,CAAA;AAAA,MACvC,OAAA;AAAA,MACA,GAAI,eAAA,CAAgB,MAAA,GAAS,CAAA,IAAK;AAAA,QAChC,eAAA,EAAiB,eAAA,CAAgB,GAAA,CAAI,CAAA,EAAA,MAAO;AAAA,UAC1C,YAAY,EAAA,CAAG,UAAA;AAAA,UACf,UAAU,EAAA,CAAG,QAAA;AAAA,UACb,MAAM,EAAA,CAAG,IAAA;AAAA,UACT,QAAQ,EAAA,CAAG,MAAA;AAAA,UACX,OAAO,EAAA,CAAG;AAAA,SACZ,CAAE;AAAA;AACJ,KACF;AAAA,IACA,SAAA,sBAAe,IAAA;AAAK,GACtB;AACF;AA+BO,IAAM,qBAAqB,CAAC;AAAA,EACjC,gBAAgB,EAAC;AAAA,EACjB,MAAA;AAAA,EACA,qBAAqB,EAAC;AAAA,EACtB,iBAAiB,EAAC;AAAA,EAClB,uBAAuB,EAAC;AAAA,EACxB,cAAA,GAAiB,IAAI,cAAA,EAAe;AAAA,EACpC,KAAA,GAAQ,OAAO,UAAA;AACjB,CAAA,KAaK;AACH,EAAA,OAAO;AAAA,IACL,KAAA,EAAO;AAAA,MACL,aAAA;AAAA,MACA,kBAAA;AAAA,MACA,cAAA;AAAA,MACA;AAAA,KACF;AAAA,IACA,MAAA;AAAA,IACA,cAAA;AAAA,IACA;AAAA,GACF;AACF;AAqCO,SAAS,iBAAiB,MAAA,EAAqF;AACpH,EAAA,MAAM,YAAsB,EAAC;AAC7B,EAAA,MAAM,gBAAgC,EAAC;AAEvC,EAAA,KAAA,IAAS,YAAA,GAAe,CAAA,EAAG,YAAA,GAAe,MAAA,CAAO,QAAQ,YAAA,EAAA,EAAgB;AACvE,IAAA,MAAM,OAAA,GAAU,OAAO,YAAY,CAAA;AAEnC,IAAA,IAAI,OAAA,EAAS,SAAS,eAAA,EAAiB;AACrC,MAAA,KAAA,IAAS,kBAAkB,CAAA,EAAG,eAAA,GAAkB,QAAQ,OAAA,CAAQ,eAAA,CAAgB,QAAQ,eAAA,EAAA,EAAmB;AACzG,QAAA,MAAM,UAAA,GAAa,OAAA,CAAQ,OAAA,CAAQ,eAAA,CAAgB,eAAe,CAAA;AAClE,QAAA,IAAI,UAAA,IAAc,WAAW,QAAA,KAAa,UAAA,CAAW,UAAU,QAAA,IAAY,UAAA,CAAW,UAAU,MAAA,CAAA,EAAS;AACvG,UAAA,SAAA,CAAU,IAAA,CAAK,WAAW,QAAQ,CAAA;AAClC,UAAA,aAAA,CAAc,IAAA,CAAK;AAAA,YACjB,UAAU,UAAA,CAAW,QAAA;AAAA,YACrB,YAAY,UAAA,CAAW,UAAA,IAAc,CAAA,EAAG,YAAY,IAAI,eAAe,CAAA,CAAA;AAAA,YACvE,YAAA;AAAA,YACA;AAAA,WACD,CAAA;AAAA,QACH;AAAA,MACF;AAAA,IACF;AAAA,EACF;AAEA,EAAA,OAAO,EAAE,KAAA,EAAO,SAAA,EAAW,aAAA,EAAc;AAC3C;AAiBO,IAAM,oBAAA,GAAuB,CAAC,QAAA,KAA2D;AAC9F,EAAA,OAAO,QAAA,EAAU,eAAe,GAAA,CAAI,CAAA,GAAA,KAAO,kCAAkC,GAAG,CAAC,KAAK,EAAC;AACzF;AAmBO,IAAM,4BAAA,GAA+B,CAAC,SAAA,KAAiD;AAC5F,EAAA,OAAO,SAAA,CAAU,MAAA,CAAO,CAAA,GAAA,KAAO,GAAA,CAAI,IAAA,KAAS,WAAW,CAAA,CAAE,GAAA,CAAI,CAAA,GAAA,KAAO,iCAAA,CAAkC,GAAG,CAAC,CAAA;AAC5G;AAyCO,SAAS,mBAAmB,MAAA,EAAmD;AACpF,EAAA,MAAM,UAA4B,EAAC;AAEnC,EAAA,KAAA,MAAW,WAAW,MAAA,EAAQ;AAC5B,IAAA,MAAM,eAAA,GAAkB,SAAS,OAAA,EAAS,eAAA;AAC1C,IAAA,IAAI,CAAC,eAAA,EAAiB;AAEtB,IAAA,KAAA,MAAW,cAAc,eAAA,EAAiB;AACxC,MAAA,IAAI,UAAA,CAAW,KAAA,KAAU,QAAA,IAAY,UAAA,CAAW,WAAW,MAAA,EAAW;AACpE,QAAA,OAAA,CAAQ,IAAA,CAAK;AAAA,UACX,UAAU,UAAA,CAAW,QAAA;AAAA,UACrB,UAAA,EAAY,WAAW,UAAA,IAAc,EAAA;AAAA,UACrC,IAAA,EAAM,UAAA,CAAW,IAAA,IAAQ,EAAC;AAAA,UAC1B,QAAQ,UAAA,CAAW;AAAA,SACpB,CAAA;AAAA,MACH;AAAA,IACF;AAAA,EACF;AAEA,EAAA,OAAO,OAAA;AACT","file":"chunk-OEOE7ZHN.js","sourcesContent":["import type { MastraDBMessage } from '@mastra/core/agent';\nimport type { ScorerRunInputForAgent, ScorerRunOutputForAgent, ScoringInput } from '@mastra/core/evals';\nimport { RequestContext } from '@mastra/core/request-context';\n\n/**\n * Extracts text content from a MastraDBMessage.\n *\n * This function matches the logic used in `MessageList.mastraDBMessageToAIV4UIMessage`.\n * It first checks for a string `content.content` field, then falls back to extracting\n * text from the `parts` array (returning only the last text part, like AI SDK does).\n *\n * @param message - The MastraDBMessage to extract text from\n * @returns The extracted text content, or an empty string if no text is found\n *\n * @example\n * ```ts\n * const message: MastraDBMessage = {\n *   id: 'msg-1',\n *   role: 'assistant',\n *   content: { format: 2, parts: [{ type: 'text', text: 'Hello!' }] },\n *   createdAt: new Date(),\n * };\n * const text = getTextContentFromMastraDBMessage(message); // 'Hello!'\n * ```\n */\nexport function getTextContentFromMastraDBMessage(message: MastraDBMessage): string {\n  if (typeof message.content.content === 'string' && message.content.content !== '') {\n    return message.content.content;\n  }\n  if (message.content.parts && Array.isArray(message.content.parts)) {\n    // Return only the last text part like AI SDK does\n    const textParts = message.content.parts.filter(p => p.type === 'text');\n    return textParts.length > 0 ? textParts[textParts.length - 1]?.text || '' : '';\n  }\n  return '';\n}\n\n/**\n * Rounds a number to two decimal places.\n *\n * Uses `Number.EPSILON` to handle floating-point precision issues.\n *\n * @param num - The number to round\n * @returns The number rounded to two decimal places\n *\n * @example\n * ```ts\n * roundToTwoDecimals(0.1 + 0.2); // 0.3\n * roundToTwoDecimals(1.005); // 1.01\n * ```\n */\nexport const roundToTwoDecimals = (num: number) => {\n  return Math.round((num + Number.EPSILON) * 100) / 100;\n};\n\n/**\n * Determines if a value is closer to the first target than the second.\n *\n * @param value - The value to compare\n * @param target1 - The first target value\n * @param target2 - The second target value\n * @returns `true` if `value` is closer to `target1` than `target2`\n *\n * @example\n * ```ts\n * isCloserTo(0.6, 1, 0); // true (0.6 is closer to 1)\n * isCloserTo(0.3, 1, 0); // false (0.3 is closer to 0)\n * ```\n */\nexport function isCloserTo(value: number, target1: number, target2: number): boolean {\n  return Math.abs(value - target1) < Math.abs(value - target2);\n}\n\n/**\n * Represents a test case for scorer evaluation.\n */\nexport type TestCase = {\n  /** The input text to evaluate */\n  input: string;\n  /** The output text to evaluate */\n  output: string;\n  /** The expected result of the evaluation */\n  expectedResult: {\n    /** The expected score */\n    score: number;\n    /** The optional expected reason */\n    reason?: string;\n  };\n};\n\n/**\n * Represents a test case with additional context for scorer evaluation.\n */\nexport type TestCaseWithContext = TestCase & {\n  /** Additional context strings for the evaluation */\n  context: string[];\n};\n\n/**\n * Creates a scoring input object for testing purposes.\n *\n * @param input - The user input text\n * @param output - The assistant output text\n * @param additionalContext - Optional additional context data\n * @param requestContext - Optional request context data\n * @returns A ScoringInput object ready for use in scorer tests\n *\n * @example\n * ```ts\n * const run = createTestRun(\n *   'What is 2+2?',\n *   'The answer is 4.',\n *   { topic: 'math' }\n * );\n * ```\n */\nexport const createTestRun = (\n  input: string,\n  output: string,\n  additionalContext?: Record<string, any>,\n  requestContext?: Record<string, any>,\n): ScoringInput => {\n  return {\n    input: [{ role: 'user', content: input }],\n    output: { role: 'assistant', text: output },\n    additionalContext: additionalContext ?? {},\n    requestContext: requestContext ?? {},\n  };\n};\n\n/**\n * Extracts the user message text from a scorer run input.\n *\n * Finds the first message with role 'user' and extracts its text content.\n *\n * @param input - The scorer run input containing input messages\n * @returns The user message text, or `undefined` if no user message is found\n *\n * @example\n * ```ts\n * const scorer = createScorer({ ... })\n *   .preprocess(({ run }) => {\n *     const userText = getUserMessageFromRunInput(run.input);\n *     return { userText };\n *   });\n * ```\n */\nexport const getUserMessageFromRunInput = (input?: ScorerRunInputForAgent): string | undefined => {\n  const message = input?.inputMessages.find(({ role }) => role === 'user');\n  return message ? getTextContentFromMastraDBMessage(message) : undefined;\n};\n\n/**\n * Extracts all system messages from a scorer run input.\n *\n * Collects text from both standard system messages and tagged system messages\n * (specialized system prompts like memory instructions).\n *\n * @param input - The scorer run input containing system messages\n * @returns An array of system message strings\n *\n * @example\n * ```ts\n * const scorer = createScorer({ ... })\n *   .preprocess(({ run }) => {\n *     const systemMessages = getSystemMessagesFromRunInput(run.input);\n *     return { systemPrompt: systemMessages.join('\\n') };\n *   });\n * ```\n */\nexport const getSystemMessagesFromRunInput = (input?: ScorerRunInputForAgent): string[] => {\n  const systemMessages: string[] = [];\n\n  // Add standard system messages\n  if (input?.systemMessages) {\n    systemMessages.push(\n      ...input.systemMessages\n        .map(msg => {\n          // Handle different content types - extract text if it's an array of parts\n          if (typeof msg.content === 'string') {\n            return msg.content;\n          } else if (Array.isArray(msg.content)) {\n            // Extract text from parts array\n            return msg.content\n              .filter((part: any) => part.type === 'text')\n              .map((part: any) => part.text || '')\n              .join(' ');\n          }\n          return '';\n        })\n        .filter(content => content),\n    );\n  }\n\n  // Add tagged system messages (these are specialized system prompts)\n  if (input?.taggedSystemMessages) {\n    Object.values(input.taggedSystemMessages).forEach(messages => {\n      messages.forEach(msg => {\n        if (typeof msg.content === 'string') {\n          systemMessages.push(msg.content);\n        }\n      });\n    });\n  }\n\n  return systemMessages;\n};\n\n/**\n * Combines all system messages into a single prompt string.\n *\n * Joins all system messages (standard and tagged) with double newlines.\n *\n * @param input - The scorer run input containing system messages\n * @returns A combined system prompt string\n *\n * @example\n * ```ts\n * const scorer = createScorer({ ... })\n *   .preprocess(({ run }) => {\n *     const systemPrompt = getCombinedSystemPrompt(run.input);\n *     return { systemPrompt };\n *   });\n * ```\n */\nexport const getCombinedSystemPrompt = (input?: ScorerRunInputForAgent): string => {\n  const systemMessages = getSystemMessagesFromRunInput(input);\n  return systemMessages.join('\\n\\n');\n};\n\n/**\n * Extracts the assistant message text from a scorer run output.\n *\n * Finds the first message with role 'assistant' and extracts its text content.\n *\n * @param output - The scorer run output (array of MastraDBMessage)\n * @returns The assistant message text, or `undefined` if no assistant message is found\n *\n * @example\n * ```ts\n * const scorer = createScorer({ ... })\n *   .preprocess(({ run }) => {\n *     const response = getAssistantMessageFromRunOutput(run.output);\n *     return { response };\n *   });\n * ```\n */\nexport const getAssistantMessageFromRunOutput = (output?: ScorerRunOutputForAgent) => {\n  const message = output?.find(({ role }) => role === 'assistant');\n  return message ? getTextContentFromMastraDBMessage(message) : undefined;\n};\n\n/**\n * Extracts reasoning text from a scorer run output.\n *\n * This function extracts reasoning content from assistant messages, which is\n * produced by reasoning models like `deepseek-reasoner`. The reasoning can be\n * stored in two places:\n * 1. `content.reasoning` - a string field on the message content\n * 2. `content.parts` - as parts with `type: 'reasoning'` containing `details`\n *\n * @param output - The scorer run output (array of MastraDBMessage)\n * @returns The reasoning text, or `undefined` if no reasoning is present\n *\n * @example\n * ```ts\n * const reasoningScorer = createScorer({\n *   id: 'reasoning-scorer',\n *   name: 'Reasoning Quality',\n *   description: 'Evaluates the quality of model reasoning',\n *   type: 'agent',\n * })\n *   .preprocess(({ run }) => {\n *     const reasoning = getReasoningFromRunOutput(run.output);\n *     const response = getAssistantMessageFromRunOutput(run.output);\n *     return { reasoning, response };\n *   })\n *   .generateScore(({ results }) => {\n *     // Score based on reasoning quality\n *     return results.preprocessStepResult?.reasoning ? 1 : 0;\n *   });\n * ```\n */\nexport const getReasoningFromRunOutput = (output?: ScorerRunOutputForAgent): string | undefined => {\n  if (!output) return undefined;\n\n  const message = output.find(({ role }) => role === 'assistant');\n  if (!message) return undefined;\n\n  // Check for reasoning in content.reasoning (string format)\n  if (message.content.reasoning) {\n    return message.content.reasoning;\n  }\n\n  // Check for reasoning in parts with type 'reasoning'\n  // Reasoning models store reasoning in parts as { type: 'reasoning', details: [{ type: 'text', text: '...' }] }\n  const reasoningParts = message.content.parts?.filter((p: any) => p.type === 'reasoning');\n  if (reasoningParts && reasoningParts.length > 0) {\n    const reasoningTexts = reasoningParts\n      .map((p: any) => {\n        // The reasoning text can be in p.reasoning or in p.details[].text\n        if (p.details && Array.isArray(p.details)) {\n          return p.details\n            .filter((d: any) => d.type === 'text')\n            .map((d: any) => d.text)\n            .join('');\n        }\n        return p.reasoning || '';\n      })\n      .filter(Boolean);\n\n    return reasoningTexts.length > 0 ? reasoningTexts.join('\\n') : undefined;\n  }\n\n  return undefined;\n};\n\n/**\n * Creates a tool invocation object for testing purposes.\n *\n * @param options - The tool invocation configuration\n * @param options.toolCallId - Unique identifier for the tool call\n * @param options.toolName - Name of the tool being called\n * @param options.args - Arguments passed to the tool\n * @param options.result - Result returned by the tool\n * @param options.state - State of the invocation (default: 'result')\n * @returns A tool invocation object\n *\n * @example\n * ```ts\n * const invocation = createToolInvocation({\n *   toolCallId: 'call-123',\n *   toolName: 'weatherTool',\n *   args: { location: 'London' },\n *   result: { temperature: 20, condition: 'sunny' },\n * });\n * ```\n */\nexport const createToolInvocation = ({\n  toolCallId,\n  toolName,\n  args,\n  result,\n  state = 'result',\n}: {\n  toolCallId: string;\n  toolName: string;\n  args: Record<string, any>;\n  result: Record<string, any>;\n  state?: 'call' | 'partial-call' | 'result';\n}): { toolCallId: string; toolName: string; args: Record<string, any>; result: Record<string, any>; state: string } => {\n  return {\n    toolCallId,\n    toolName,\n    args,\n    result,\n    state,\n  };\n};\n\n/**\n * Creates a MastraDBMessage object for testing purposes.\n *\n * Supports optional tool invocations for testing tool call scenarios.\n *\n * @param options - The message configuration\n * @param options.content - The text content of the message\n * @param options.role - The role of the message sender ('user', 'assistant', or 'system')\n * @param options.id - Optional message ID (default: 'test-message')\n * @param options.toolInvocations - Optional array of tool invocations\n * @returns A MastraDBMessage object\n *\n * @example\n * ```ts\n * const message = createTestMessage({\n *   content: 'Hello, how can I help?',\n *   role: 'assistant',\n * });\n *\n * // With tool invocations\n * const messageWithTools = createTestMessage({\n *   content: 'Let me check the weather.',\n *   role: 'assistant',\n *   toolInvocations: [{\n *     toolCallId: 'call-1',\n *     toolName: 'weatherTool',\n *     args: { location: 'Paris' },\n *     result: { temp: 22 },\n *     state: 'result',\n *   }],\n * });\n * ```\n */\nexport function createTestMessage({\n  content,\n  role,\n  id = 'test-message',\n  toolInvocations = [],\n}: {\n  content: string;\n  role: 'user' | 'assistant' | 'system';\n  id?: string;\n  toolInvocations?: Array<{\n    toolCallId: string;\n    toolName: string;\n    args: Record<string, any>;\n    result: Record<string, any>;\n    state: any;\n  }>;\n}): MastraDBMessage {\n  return {\n    id,\n    role,\n    content: {\n      format: 2,\n      parts: [{ type: 'text', text: content }],\n      content,\n      ...(toolInvocations.length > 0 && {\n        toolInvocations: toolInvocations.map(ti => ({\n          toolCallId: ti.toolCallId,\n          toolName: ti.toolName,\n          args: ti.args,\n          result: ti.result,\n          state: ti.state,\n        })),\n      }),\n    },\n    createdAt: new Date(),\n  };\n}\n\n/**\n * Creates a complete agent test run object for testing scorers.\n *\n * Provides a convenient way to construct the full run object that scorers receive,\n * including input messages, output, system messages, and request context.\n *\n * @param options - The test run configuration\n * @param options.inputMessages - Array of input messages (default: [])\n * @param options.output - The output messages (required)\n * @param options.rememberedMessages - Array of remembered messages from memory (default: [])\n * @param options.systemMessages - Array of system messages (default: [])\n * @param options.taggedSystemMessages - Tagged system messages map (default: {})\n * @param options.requestContext - Request context (default: new RequestContext())\n * @param options.runId - Unique run ID (default: random UUID)\n * @returns A complete test run object\n *\n * @example\n * ```ts\n * const testRun = createAgentTestRun({\n *   inputMessages: [createTestMessage({ content: 'Hello', role: 'user' })],\n *   output: [createTestMessage({ content: 'Hi there!', role: 'assistant' })],\n * });\n *\n * const result = await scorer.run({\n *   input: testRun.input,\n *   output: testRun.output,\n * });\n * ```\n */\nexport const createAgentTestRun = ({\n  inputMessages = [],\n  output,\n  rememberedMessages = [],\n  systemMessages = [],\n  taggedSystemMessages = {},\n  requestContext = new RequestContext(),\n  runId = crypto.randomUUID(),\n}: {\n  inputMessages?: ScorerRunInputForAgent['inputMessages'];\n  output: ScorerRunOutputForAgent;\n  rememberedMessages?: ScorerRunInputForAgent['rememberedMessages'];\n  systemMessages?: ScorerRunInputForAgent['systemMessages'];\n  taggedSystemMessages?: ScorerRunInputForAgent['taggedSystemMessages'];\n  requestContext?: RequestContext;\n  runId?: string;\n}): {\n  input: ScorerRunInputForAgent;\n  output: ScorerRunOutputForAgent;\n  requestContext: RequestContext;\n  runId: string;\n} => {\n  return {\n    input: {\n      inputMessages,\n      rememberedMessages,\n      systemMessages,\n      taggedSystemMessages,\n    },\n    output,\n    requestContext,\n    runId,\n  };\n};\n\n/**\n * Information about a tool call extracted from scorer output.\n */\nexport type ToolCallInfo = {\n  /** Name of the tool that was called */\n  toolName: string;\n  /** Unique identifier for the tool call */\n  toolCallId: string;\n  /** Index of the message containing this tool call */\n  messageIndex: number;\n  /** Index of the invocation within the message's tool invocations */\n  invocationIndex: number;\n};\n\n/**\n * Extracts all tool calls from a scorer run output.\n *\n * Iterates through all messages and their tool invocations to collect\n * information about tools that were called (with state 'result' or 'call').\n *\n * @param output - The scorer run output (array of MastraDBMessage)\n * @returns An object containing tool names and detailed tool call info\n *\n * @example\n * ```ts\n * const scorer = createScorer({ ... })\n *   .preprocess(({ run }) => {\n *     const { tools, toolCallInfos } = extractToolCalls(run.output);\n *     return {\n *       toolsUsed: tools,\n *       toolCount: tools.length,\n *     };\n *   });\n * ```\n */\nexport function extractToolCalls(output: ScorerRunOutputForAgent): { tools: string[]; toolCallInfos: ToolCallInfo[] } {\n  const toolCalls: string[] = [];\n  const toolCallInfos: ToolCallInfo[] = [];\n\n  for (let messageIndex = 0; messageIndex < output.length; messageIndex++) {\n    const message = output[messageIndex];\n    // Tool invocations are now nested under content\n    if (message?.content?.toolInvocations) {\n      for (let invocationIndex = 0; invocationIndex < message.content.toolInvocations.length; invocationIndex++) {\n        const invocation = message.content.toolInvocations[invocationIndex];\n        if (invocation && invocation.toolName && (invocation.state === 'result' || invocation.state === 'call')) {\n          toolCalls.push(invocation.toolName);\n          toolCallInfos.push({\n            toolName: invocation.toolName,\n            toolCallId: invocation.toolCallId || `${messageIndex}-${invocationIndex}`,\n            messageIndex,\n            invocationIndex,\n          });\n        }\n      }\n    }\n  }\n\n  return { tools: toolCalls, toolCallInfos };\n}\n\n/**\n * Extracts text content from all input messages.\n *\n * @param runInput - The scorer run input\n * @returns An array of text strings from each input message\n *\n * @example\n * ```ts\n * const scorer = createScorer({ ... })\n *   .preprocess(({ run }) => {\n *     const messages = extractInputMessages(run.input);\n *     return { allUserMessages: messages.join('\\n') };\n *   });\n * ```\n */\nexport const extractInputMessages = (runInput: ScorerRunInputForAgent | undefined): string[] => {\n  return runInput?.inputMessages?.map(msg => getTextContentFromMastraDBMessage(msg)) || [];\n};\n\n/**\n * Extracts text content from all assistant response messages.\n *\n * Filters for messages with role 'assistant' and extracts their text content.\n *\n * @param runOutput - The scorer run output (array of MastraDBMessage)\n * @returns An array of text strings from each assistant message\n *\n * @example\n * ```ts\n * const scorer = createScorer({ ... })\n *   .preprocess(({ run }) => {\n *     const responses = extractAgentResponseMessages(run.output);\n *     return { allResponses: responses.join('\\n') };\n *   });\n * ```\n */\nexport const extractAgentResponseMessages = (runOutput: ScorerRunOutputForAgent): string[] => {\n  return runOutput.filter(msg => msg.role === 'assistant').map(msg => getTextContentFromMastraDBMessage(msg));\n};\n\n/**\n * Information about a tool result extracted from scorer output.\n */\nexport type ToolResultInfo = {\n  /** Name of the tool that was called */\n  toolName: string;\n  /** Unique identifier for the tool call */\n  toolCallId: string;\n  /** Arguments passed to the tool */\n  args: Record<string, any>;\n  /** Result returned by the tool */\n  result: any;\n};\n\n/**\n * Extracts tool results from a scorer run output.\n *\n * Returns structured objects that can be used with the hallucination scorer's\n * `getContext` hook or for other scorer logic.\n *\n * @param output - The scorer run output (array of MastraDBMessage)\n * @returns An array of ToolResultInfo objects\n *\n * @example\n * ```ts\n * import { extractToolResults } from '@mastra/evals/scorers';\n * import { createHallucinationScorer } from '@mastra/evals/scorers/prebuilt';\n *\n * const scorer = createHallucinationScorer({\n *   model: openai('gpt-4o'),\n *   options: {\n *     getContext: (run) => {\n *       const toolResults = extractToolResults(run.output);\n *       return toolResults.map(t => JSON.stringify({ tool: t.toolName, result: t.result }));\n *     },\n *   },\n * });\n * ```\n */\nexport function extractToolResults(output: ScorerRunOutputForAgent): ToolResultInfo[] {\n  const results: ToolResultInfo[] = [];\n\n  for (const message of output) {\n    const toolInvocations = message?.content?.toolInvocations;\n    if (!toolInvocations) continue;\n\n    for (const invocation of toolInvocations) {\n      if (invocation.state === 'result' && invocation.result !== undefined) {\n        results.push({\n          toolName: invocation.toolName,\n          toolCallId: invocation.toolCallId || '',\n          args: invocation.args || {},\n          result: invocation.result,\n        });\n      }\n    }\n  }\n\n  return results;\n}\n"]}

package/dist/{chunk-QKR2PMLZ.cjs → chunk-W3U7MMDX.cjs}

@@ -1,24 +1,35 @@
 'use strict';
 
-var runtimeContext = require('@mastra/core/runtime-context');
+var requestContext = require('@mastra/core/request-context');
 
 // src/scorers/utils.ts
+function getTextContentFromMastraDBMessage(message) {
+  if (typeof message.content.content === "string" && message.content.content !== "") {
+    return message.content.content;
+  }
+  if (message.content.parts && Array.isArray(message.content.parts)) {
+    const textParts = message.content.parts.filter((p) => p.type === "text");
+    return textParts.length > 0 ? textParts[textParts.length - 1]?.text || "" : "";
+  }
+  return "";
+}
 var roundToTwoDecimals = (num) => {
   return Math.round((num + Number.EPSILON) * 100) / 100;
 };
 function isCloserTo(value, target1, target2) {
   return Math.abs(value - target1) < Math.abs(value - target2);
 }
-var createTestRun = (input, output, additionalContext, runtimeContext) => {
+var createTestRun = (input, output, additionalContext, requestContext) => {
   return {
     input: [{ role: "user", content: input }],
     output: { role: "assistant", text: output },
     additionalContext: additionalContext ?? {},
-    runtimeContext: runtimeContext ?? {}
+    requestContext: requestContext ?? {}
   };
 };
 var getUserMessageFromRunInput = (input) => {
-  return input?.inputMessages.find(({ role }) => role === "user")?.content;
+  const message = input?.inputMessages.find(({ role }) => role === "user");
+  return message ? getTextContentFromMastraDBMessage(message) : void 0;
 };
 var getSystemMessagesFromRunInput = (input) => {
   const systemMessages = [];
@@ -50,7 +61,27 @@ var getCombinedSystemPrompt = (input) => {
   return systemMessages.join("\n\n");
 };
 var getAssistantMessageFromRunOutput = (output) => {
-  return output?.find(({ role }) => role === "assistant")?.content;
+  const message = output?.find(({ role }) => role === "assistant");
+  return message ? getTextContentFromMastraDBMessage(message) : void 0;
+};
+var getReasoningFromRunOutput = (output) => {
+  if (!output) return void 0;
+  const message = output.find(({ role }) => role === "assistant");
+  if (!message) return void 0;
+  if (message.content.reasoning) {
+    return message.content.reasoning;
+  }
+  const reasoningParts = message.content.parts?.filter((p) => p.type === "reasoning");
+  if (reasoningParts && reasoningParts.length > 0) {
+    const reasoningTexts = reasoningParts.map((p) => {
+      if (p.details && Array.isArray(p.details)) {
+        return p.details.filter((d) => d.type === "text").map((d) => d.text).join("");
+      }
+      return p.reasoning || "";
+    }).filter(Boolean);
+    return reasoningTexts.length > 0 ? reasoningTexts.join("\n") : void 0;
+  }
+  return void 0;
 };
 var createToolInvocation = ({
   toolCallId,
@@ -67,27 +98,39 @@ var createToolInvocation = ({
     state
   };
 };
-var createUIMessage = ({
+function createTestMessage({
   content,
   role,
   id = "test-message",
   toolInvocations = []
-}) => {
+}) {
   return {
     id,
     role,
-    content,
-    parts: [{ type: "text", text: content }],
-    toolInvocations
+    content: {
+      format: 2,
+      parts: [{ type: "text", text: content }],
+      content,
+      ...toolInvocations.length > 0 && {
+        toolInvocations: toolInvocations.map((ti) => ({
+          toolCallId: ti.toolCallId,
+          toolName: ti.toolName,
+          args: ti.args,
+          result: ti.result,
+          state: ti.state
+        }))
+      }
+    },
+    createdAt: /* @__PURE__ */ new Date()
   };
-};
+}
 var createAgentTestRun = ({
   inputMessages = [],
   output,
   rememberedMessages = [],
   systemMessages = [],
   taggedSystemMessages = {},
-  runtimeContext: runtimeContext$1 = new runtimeContext.RuntimeContext(),
+  requestContext: requestContext$1 = new requestContext.RequestContext(),
   runId = crypto.randomUUID()
 }) => {
   return {
@@ -98,7 +141,7 @@ var createAgentTestRun = ({
       taggedSystemMessages
     },
     output,
-    runtimeContext: runtimeContext$1,
+    requestContext: requestContext$1,
     runId
   };
 };
@@ -107,9 +150,9 @@ function extractToolCalls(output) {
   const toolCallInfos = [];
   for (let messageIndex = 0; messageIndex < output.length; messageIndex++) {
     const message = output[messageIndex];
-    if (message?.toolInvocations) {
-      for (let invocationIndex = 0; invocationIndex < message.toolInvocations.length; invocationIndex++) {
-        const invocation = message.toolInvocations[invocationIndex];
+    if (message?.content?.toolInvocations) {
+      for (let invocationIndex = 0; invocationIndex < message.content.toolInvocations.length; invocationIndex++) {
+        const invocation = message.content.toolInvocations[invocationIndex];
         if (invocation && invocation.toolName && (invocation.state === "result" || invocation.state === "call")) {
           toolCalls.push(invocation.toolName);
           toolCallInfos.push({
@@ -125,24 +168,45 @@ function extractToolCalls(output) {
   return { tools: toolCalls, toolCallInfos };
 }
 var extractInputMessages = (runInput) => {
-  return runInput?.inputMessages?.map((msg) => msg.content) || [];
+  return runInput?.inputMessages?.map((msg) => getTextContentFromMastraDBMessage(msg)) || [];
 };
 var extractAgentResponseMessages = (runOutput) => {
-  return runOutput.filter((msg) => msg.role === "assistant").map((msg) => msg.content);
+  return runOutput.filter((msg) => msg.role === "assistant").map((msg) => getTextContentFromMastraDBMessage(msg));
 };
+function extractToolResults(output) {
+  const results = [];
+  for (const message of output) {
+    const toolInvocations = message?.content?.toolInvocations;
+    if (!toolInvocations) continue;
+    for (const invocation of toolInvocations) {
+      if (invocation.state === "result" && invocation.result !== void 0) {
+        results.push({
+          toolName: invocation.toolName,
+          toolCallId: invocation.toolCallId || "",
+          args: invocation.args || {},
+          result: invocation.result
+        });
+      }
+    }
+  }
+  return results;
+}
 
 exports.createAgentTestRun = createAgentTestRun;
+exports.createTestMessage = createTestMessage;
 exports.createTestRun = createTestRun;
 exports.createToolInvocation = createToolInvocation;
-exports.createUIMessage = createUIMessage;
 exports.extractAgentResponseMessages = extractAgentResponseMessages;
 exports.extractInputMessages = extractInputMessages;
 exports.extractToolCalls = extractToolCalls;
+exports.extractToolResults = extractToolResults;
 exports.getAssistantMessageFromRunOutput = getAssistantMessageFromRunOutput;
 exports.getCombinedSystemPrompt = getCombinedSystemPrompt;
+exports.getReasoningFromRunOutput = getReasoningFromRunOutput;
 exports.getSystemMessagesFromRunInput = getSystemMessagesFromRunInput;
+exports.getTextContentFromMastraDBMessage = getTextContentFromMastraDBMessage;
 exports.getUserMessageFromRunInput = getUserMessageFromRunInput;
 exports.isCloserTo = isCloserTo;
 exports.roundToTwoDecimals = roundToTwoDecimals;
-//# sourceMappingURL=chunk-QKR2PMLZ.cjs.map
-//# sourceMappingURL=chunk-QKR2PMLZ.cjs.map
+//# sourceMappingURL=chunk-W3U7MMDX.cjs.map
+//# sourceMappingURL=chunk-W3U7MMDX.cjs.map