@mastra/deepeval 0.0.0

package/CHANGELOG.md

@@ -0,0 +1,33 @@
+# @mastra/deepeval
+
+## 0.1.0-alpha.0
+
+### Minor Changes
+
+- Added the `@mastra/deepeval` observability exporter to send Mastra traces to Confident AI for evaluation and monitoring. ([#20599](https://github.com/mastra-ai/mastra/pull/20599))
+
+  Register it in your Mastra observability config:
+
+  ```typescript
+  import { Mastra } from '@mastra/core';
+  import { Observability } from '@mastra/observability';
+  import { DeepEvalExporter } from '@mastra/deepeval';
+
+  export const mastra = new Mastra({
+    observability: new Observability({
+      configs: {
+        deepeval: {
+          serviceName: 'my-service',
+          exporters: [new DeepEvalExporter()],
+        },
+      },
+    }),
+  });
+  ```
+
+  Set `CONFIDENT_API_KEY` (and optionally `CONFIDENT_TRACE_ENVIRONMENT`) to send traces. Mastra spans map to Confident AI's `AGENT`, `LLM`, `TOOL`, `RETRIEVER`, and `CUSTOM` span types, with model, token counts, tool calls, and metric collections carried through.
+
+### Patch Changes
+
+- Updated dependencies [[`89200ba`](https://github.com/mastra-ai/mastra/commit/89200bafa05444bb7949b363ce7b743e29867561), [`c950138`](https://github.com/mastra-ai/mastra/commit/c950138e72e4f317a40187e3800588731ab790ce), [`063c8b2`](https://github.com/mastra-ai/mastra/commit/063c8b2eb14e4e5ca021779bc33e8c3c031c8604), [`f4e964c`](https://github.com/mastra-ai/mastra/commit/f4e964cad57057301d6bed5c55bcdd730175b941), [`1f7bbd7`](https://github.com/mastra-ai/mastra/commit/1f7bbd7785a8d230aad02454ecabeb4a0b2cc96f), [`e47ff36`](https://github.com/mastra-ai/mastra/commit/e47ff36945720f4ee4caa09f6e83514d7d188608), [`fb9a6ac`](https://github.com/mastra-ai/mastra/commit/fb9a6ac11c9560518742ece60b49d6b062845fd3), [`aa2cec8`](https://github.com/mastra-ai/mastra/commit/aa2cec8501f634d51c2f3ebfb3dd3aa7af8d2ca2), [`2adf8eb`](https://github.com/mastra-ai/mastra/commit/2adf8eb4a70ed2b6cff2dd39281496ea0e025fac), [`8264611`](https://github.com/mastra-ai/mastra/commit/8264611510e421b818bc7395dc2ae4d9c2d518b2), [`44fc98b`](https://github.com/mastra-ai/mastra/commit/44fc98b9d1242aa87a3ab44bdce9e9f12c44d8c9), [`0f2ef41`](https://github.com/mastra-ai/mastra/commit/0f2ef4118da022e4f30dac4e9856cc3a8c97671c)]:
+  - @mastra/core@1.57.0-alpha.1

package/README.md

@@ -0,0 +1,91 @@
+# @mastra/deepeval
+
+[Confident AI](https://www.confident-ai.com/) observability exporter for Mastra applications. Sends your Mastra traces to Confident AI for evaluation and monitoring, built on the DeepEval SDK.
+
+## Installation
+
+```bash
+npm install @mastra/deepeval
+```
+
+## Usage
+
+### Zero-Config Setup
+
+The exporter automatically reads credentials from environment variables:
+
+```bash
+# Required
+CONFIDENT_API_KEY=confident_...
+
+# Optional
+CONFIDENT_TRACE_ENVIRONMENT=production  # Defaults to "development"
+```
+
+```typescript
+import { Observability } from '@mastra/observability';
+import { DeepEvalExporter } from '@mastra/deepeval';
+
+const mastra = new Mastra({
+  ...,
+  observability: new Observability({
+    configs: {
+      deepeval: {
+        serviceName: 'my-service',
+        exporters: [new DeepEvalExporter()],
+      },
+    },
+  }),
+});
+```
+
+### Explicit Configuration
+
+You can also pass credentials directly:
+
+```typescript
+import { Observability } from '@mastra/observability';
+import { DeepEvalExporter } from '@mastra/deepeval';
+
+const mastra = new Mastra({
+  ...,
+  observability: new Observability({
+    configs: {
+      deepeval: {
+        serviceName: 'my-service',
+        exporters: [
+          new DeepEvalExporter({
+            apiKey: 'confident_...',
+            environment: 'production',
+          }),
+        ],
+      },
+    },
+  }),
+});
+```
+
+### Configuration Options
+
+| Option                    | Type                     | Description                                                                   |
+| ------------------------- | ------------------------ | ----------------------------------------------------------------------------- |
+| `apiKey`                  | `string`                 | Confident AI API key. Defaults to the `CONFIDENT_API_KEY` env var             |
+| `environment`             | `string`                 | Trace environment. Defaults to `CONFIDENT_TRACE_ENVIRONMENT` or `development` |
+| `name`                    | `string`                 | Trace name. Defaults to the Mastra `serviceName`                              |
+| `tags`                    | `string[]`               | Tags applied to every trace                                                   |
+| `metadata`                | `Record<string, any>`    | Metadata applied to every trace                                               |
+| `metricCollection`        | `string`                 | Trace-level metric collection                                                 |
+| `llmMetricCollection`     | `string`                 | Metric collection applied to LLM spans                                        |
+| `agentMetricCollection`   | `string`                 | Metric collection applied to agent spans                                      |
+| `toolMetricCollectionMap` | `Record<string, string>` | Metric collection applied per tool name                                       |
+| `traceCaptureSink`        | `(trace) => void`        | Receives completed traces locally instead of posting (for offline evaluation) |
+
+## Features
+
+### Tracing
+
+- **Native Confident AI spans**: Mastra spans become `LLM`, `TOOL`, `RETRIEVER`, `AGENT`, and `CUSTOM` spans
+- **Type-specific fields**: Extracts model, token counts, tool calls, and per-type metadata
+- **Error tracking**: Automatic error status and message tracking
+- **Hierarchical traces**: Maintains parent-child relationships and posts the trace once the root span ends
+- **Metric collections**: Attach Confident AI metric collections at the trace and per-span-type level

package/package.json

@@ -0,0 +1,64 @@
+{
+  "name": "@mastra/deepeval",
+  "version": "0.0.0",
+  "description": "DeepEval (Confident AI) observability provider for Mastra - includes tracing and future observability features",
+  "type": "module",
+  "main": "dist/index.js",
+  "types": "dist/index.d.ts",
+  "files": [
+    "dist",
+    "CHANGELOG.md"
+  ],
+  "exports": {
+    ".": {
+      "import": {
+        "types": "./dist/index.d.ts",
+        "default": "./dist/index.js"
+      },
+      "require": {
+        "types": "./dist/index.d.ts",
+        "default": "./dist/index.cjs"
+      }
+    },
+    "./package.json": "./package.json"
+  },
+  "scripts": {
+    "build": "tsup --silent --config tsup.config.ts",
+    "build:watch": "pnpm build --watch",
+    "test": "vitest run",
+    "test:watch": "vitest watch",
+    "lint": "eslint ."
+  },
+  "license": "Apache-2.0",
+  "dependencies": {
+    "@mastra/observability": "workspace:*",
+    "deepeval": "^0.1.32"
+  },
+  "devDependencies": {
+    "@internal/lint": "workspace:*",
+    "@internal/types-builder": "workspace:*",
+    "@mastra/core": "workspace:*",
+    "@types/node": "22.20.1",
+    "@vitest/coverage-v8": "catalog:",
+    "@vitest/ui": "catalog:",
+    "eslint": "^10.4.1",
+    "tsup": "^8.5.1",
+    "typescript": "catalog:",
+    "vitest": "catalog:"
+  },
+  "peerDependencies": {
+    "@mastra/core": ">=1.16.0-0 <2.0.0-0"
+  },
+  "homepage": "https://mastra.ai",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/mastra-ai/mastra.git",
+    "directory": "observability/deepeval"
+  },
+  "bugs": {
+    "url": "https://github.com/mastra-ai/mastra/issues"
+  },
+  "engines": {
+    "node": ">=22.13.0"
+  }
+}