@mastra/evals 0.1.0-alpha.5 → 0.1.0-alpha.50

package/CHANGELOG.md

@@ -1,5 +1,377 @@
 # @mastra/evals
 
+## 0.1.0-alpha.50
+
+### Patch Changes
+
+- Updated dependencies [5ee67d3]
+- Updated dependencies [95a4697]
+  - @mastra/core@0.2.0-alpha.108
+
+## 0.1.0-alpha.49
+
+### Patch Changes
+
+- Updated dependencies [66a5392]
+  - @mastra/core@0.2.0-alpha.107
+
+## 0.1.0-alpha.48
+
+### Patch Changes
+
+- a8a459a: Updated Evals table UI
+- Updated dependencies [6f2c0f5]
+- Updated dependencies [a8a459a]
+  - @mastra/core@0.2.0-alpha.106
+
+## 0.1.0-alpha.47
+
+### Patch Changes
+
+- Updated dependencies [1420ae2]
+- Updated dependencies [99f1847]
+  - @mastra/core@0.2.0-alpha.105
+
+## 0.1.0-alpha.46
+
+### Patch Changes
+
+- 5fdc87c: Update evals storage in attachListeners
+- b97ca96: Tracing into default storage
+- 72d1990: Updated evals table schema
+- Updated dependencies [5fdc87c]
+- Updated dependencies [b97ca96]
+- Updated dependencies [72d1990]
+- Updated dependencies [cf6d825]
+- Updated dependencies [10870bc]
+  - @mastra/core@0.2.0-alpha.104
+
+## 0.1.0-alpha.45
+
+### Patch Changes
+
+- Updated dependencies [4534e77]
+  - @mastra/core@0.2.0-alpha.103
+
+## 0.1.0-alpha.44
+
+### Patch Changes
+
+- Updated dependencies [a9345f9]
+  - @mastra/core@0.2.0-alpha.102
+
+## 0.1.0-alpha.43
+
+### Patch Changes
+
+- 4f1d1a1: Enforce types ann cleanup package.json
+- Updated dependencies [66a03ec]
+- Updated dependencies [4f1d1a1]
+  - @mastra/core@0.2.0-alpha.101
+
+## 0.1.0-alpha.42
+
+### Patch Changes
+
+- Updated dependencies [9d1796d]
+  - @mastra/core@0.2.0-alpha.100
+
+## 0.1.0-alpha.41
+
+### Patch Changes
+
+- Updated dependencies [7d83b92]
+  - @mastra/core@0.2.0-alpha.99
+
+## 0.1.0-alpha.40
+
+### Patch Changes
+
+- 70dabd9: Fix broken publish
+- 202d404: Added instructions when generating evals
+- Updated dependencies [70dabd9]
+- Updated dependencies [202d404]
+  - @mastra/core@0.2.0-alpha.98
+
+## 0.1.0-alpha.39
+
+### Patch Changes
+
+- 7892533: Updated test evals to use Mastra Storage
+- d641d91: Fix exports for @mastra/evals
+- Updated dependencies [07c069d]
+- Updated dependencies [7892533]
+- Updated dependencies [e6d8055]
+- Updated dependencies [5950de5]
+- Updated dependencies [df843d3]
+- Updated dependencies [a870123]
+  - @mastra/core@0.2.0-alpha.97
+
+## 0.1.0-alpha.38
+
+### Patch Changes
+
+- Updated dependencies [74b3078]
+  - @mastra/core@0.2.0-alpha.96
+
+## 0.1.0-alpha.37
+
+### Patch Changes
+
+- Updated dependencies [9fb59d6]
+  - @mastra/core@0.2.0-alpha.95
+
+## 0.1.0-alpha.36
+
+### Minor Changes
+
+- 8b416d9: Breaking changes
+
+### Patch Changes
+
+- 9c10484: update all packages
+- Updated dependencies [9c10484]
+- Updated dependencies [8b416d9]
+  - @mastra/core@0.2.0-alpha.94
+
+## 0.1.0-alpha.35
+
+### Patch Changes
+
+- Updated dependencies [5285356]
+  - @mastra/core@0.2.0-alpha.93
+
+## 0.1.0-alpha.34
+
+### Patch Changes
+
+- Updated dependencies [4d4f6b6]
+  - @mastra/core@0.2.0-alpha.92
+
+## 0.1.0-alpha.33
+
+### Patch Changes
+
+- Updated dependencies [d7d465a]
+- Updated dependencies [d7d465a]
+- Updated dependencies [2017553]
+- Updated dependencies [a10b7a3]
+- Updated dependencies [16e5b04]
+  - @mastra/core@0.2.0-alpha.91
+
+## 0.1.0-alpha.32
+
+### Patch Changes
+
+- Updated dependencies [8151f44]
+- Updated dependencies [e897f1c]
+- Updated dependencies [3700be1]
+  - @mastra/core@0.2.0-alpha.90
+
+## 0.1.0-alpha.31
+
+### Patch Changes
+
+- Updated dependencies [27275c9]
+  - @mastra/core@0.2.0-alpha.89
+
+## 0.1.0-alpha.30
+
+### Patch Changes
+
+- Updated dependencies [ccbc581]
+  - @mastra/core@0.2.0-alpha.88
+
+## 0.1.0-alpha.29
+
+### Patch Changes
+
+- Updated dependencies [7365b6c]
+  - @mastra/core@0.2.0-alpha.87
+
+## 0.1.0-alpha.28
+
+### Minor Changes
+
+- 5916f9d: Update deps from fixed to ^
+
+### Patch Changes
+
+- Updated dependencies [6fa4bd2]
+- Updated dependencies [e2e76de]
+- Updated dependencies [7f24c29]
+- Updated dependencies [67637ba]
+- Updated dependencies [04f3171]
+  - @mastra/core@0.2.0-alpha.86
+
+## 0.1.0-alpha.27
+
+### Patch Changes
+
+- Updated dependencies [e9d1b47]
+  - @mastra/core@0.2.0-alpha.85
+
+## 0.1.0-alpha.26
+
+### Patch Changes
+
+- Updated dependencies [2f17a5f]
+- Updated dependencies [cb290ee]
+- Updated dependencies [b4d7416]
+- Updated dependencies [38b7f66]
+  - @mastra/core@0.2.0-alpha.84
+
+## 0.1.0-alpha.25
+
+### Patch Changes
+
+- 9625602: Use mastra core splitted bundles in other packages
+- 8769a62: Split core into seperate entry fils
+- Updated dependencies [30322ce]
+- Updated dependencies [78eec7c]
+- Updated dependencies [9625602]
+- Updated dependencies [8769a62]
+  - @mastra/core@0.2.0-alpha.83
+
+## 0.1.0-alpha.24
+
+### Patch Changes
+
+- Updated dependencies [73d112c]
+  - @mastra/core@0.1.27-alpha.82
+
+## 0.1.0-alpha.23
+
+### Patch Changes
+
+- Updated dependencies [9fb3039]
+  - @mastra/core@0.1.27-alpha.81
+
+## 0.1.0-alpha.22
+
+### Patch Changes
+
+- cb2e997: Bundle evals package with tsup
+
+## 0.1.0-alpha.21
+
+### Patch Changes
+
+- Updated dependencies [327ece7]
+  - @mastra/core@0.1.27-alpha.80
+
+## 0.1.0-alpha.20
+
+### Patch Changes
+
+- Updated dependencies [21fe536]
+  - @mastra/core@0.1.27-alpha.79
+
+## 0.1.0-alpha.19
+
+### Patch Changes
+
+- Updated dependencies [685108a]
+- Updated dependencies [685108a]
+  - @mastra/core@0.1.27-alpha.78
+
+## 0.1.0-alpha.18
+
+### Patch Changes
+
+- Updated dependencies [8105fae]
+  - @mastra/core@0.1.27-alpha.77
+
+## 0.1.0-alpha.17
+
+### Patch Changes
+
+- Updated dependencies [ae7bf94]
+- Updated dependencies [ae7bf94]
+  - @mastra/core@0.1.27-alpha.76
+
+## 0.1.0-alpha.16
+
+### Patch Changes
+
+- Updated dependencies [23dcb23]
+  - @mastra/core@0.1.27-alpha.75
+
+## 0.1.0-alpha.15
+
+### Patch Changes
+
+- Updated dependencies [7b87567]
+  - @mastra/core@0.1.27-alpha.74
+
+## 0.1.0-alpha.14
+
+### Patch Changes
+
+- Updated dependencies [3427b95]
+  - @mastra/core@0.1.27-alpha.73
+
+## 0.1.0-alpha.13
+
+### Patch Changes
+
+- 06b2c0a: Update summarization prompt and fix eval input
+- Updated dependencies [e4d4ede]
+- Updated dependencies [06b2c0a]
+  - @mastra/core@0.1.27-alpha.72
+
+## 0.1.0-alpha.12
+
+### Patch Changes
+
+- Updated dependencies [d9c8dd0]
+  - @mastra/core@0.1.27-alpha.71
+
+## 0.1.0-alpha.11
+
+### Patch Changes
+
+- bdaf834: publish packages
+
+## 0.1.0-alpha.10
+
+### Patch Changes
+
+- Updated dependencies [dd6d87f]
+- Updated dependencies [04434b6]
+  - @mastra/core@0.1.27-alpha.70
+
+## 0.1.0-alpha.9
+
+### Patch Changes
+
+- 1944807: Unified logger and major step in better logs
+- 9ade36e: Changed measure for evals, added endpoints, attached metrics to agent, added ui for evals in playground, and updated docs
+- Updated dependencies [1944807]
+- Updated dependencies [9ade36e]
+  - @mastra/core@0.1.27-alpha.69
+
+## 0.1.0-alpha.8
+
+### Patch Changes
+
+- Updated dependencies [0be7181]
+- Updated dependencies [0be7181]
+  - @mastra/core@0.1.27-alpha.68
+
+## 0.1.0-alpha.7
+
+### Patch Changes
+
+- Updated dependencies [c8ff2f5]
+  - @mastra/core@0.1.27-alpha.67
+
+## 0.1.0-alpha.6
+
+### Patch Changes
+
+- aea3c13: Fix evals export for llm and nlp
+
 ## 0.1.0-alpha.5
 
 ### Minor Changes

package/README.md

@@ -0,0 +1,185 @@
+# @mastra/evals
+
+A comprehensive evaluation framework for assessing AI model outputs across multiple dimensions.
+
+## Installation
+
+```bash
+npm install @mastra/evals
+```
+
+## Overview
+
+`@mastra/evals` provides a suite of evaluation metrics for assessing AI model outputs. The package includes both LLM-based and NLP-based metrics, enabling both automated and model-assisted evaluation of AI responses.
+
+## Features
+
+### LLM-Based Metrics
+
+1. **Answer Relevancy**
+
+   - Evaluates how well an answer addresses the input question
+   - Considers uncertainty weighting for more nuanced scoring
+   - Returns detailed reasoning for scores
+
+2. **Bias Detection**
+
+   - Identifies potential biases in model outputs
+   - Analyzes opinions and statements for bias indicators
+   - Provides explanations for detected biases
+   - Configurable scoring scale
+
+3. **Context Precision & Relevancy**
+
+   - Assesses how well responses use provided context
+   - Evaluates accuracy of context usage
+   - Measures relevance of context to the response
+   - Analyzes context positioning in responses
+
+4. **Faithfulness**
+
+   - Verifies that responses are faithful to provided context
+   - Detects hallucinations or fabricated information
+   - Evaluates claims against provided context
+   - Provides detailed analysis of faithfulness breaches
+
+5. **Prompt Alignment**
+
+   - Measures how well responses follow given instructions
+   - Evaluates adherence to multiple instruction criteria
+   - Provides per-instruction scoring
+   - Supports custom instruction sets
+
+6. **Toxicity**
+   - Detects toxic or harmful content in responses
+   - Provides detailed reasoning for toxicity verdicts
+   - Configurable scoring thresholds
+   - Considers both input and output context
+
+### NLP-Based Metrics
+
+1. **Completeness**
+
+   - Analyzes structural completeness of responses
+   - Identifies missing elements from input requirements
+   - Provides detailed element coverage analysis
+   - Tracks input-output element ratios
+
+2. **Content Similarity**
+
+   - Measures text similarity between inputs and outputs
+   - Configurable for case and whitespace sensitivity
+   - Returns normalized similarity scores
+   - Uses string comparison algorithms for accuracy
+
+3. **Keyword Coverage**
+   - Tracks presence of key terms from input in output
+   - Provides detailed keyword matching statistics
+   - Calculates coverage ratios
+   - Useful for ensuring comprehensive responses
+
+## Usage
+
+### Basic Example
+
+```typescript
+import { ContentSimilarityMetric, ToxicityMetric } from '@mastra/evals';
+
+// Initialize metrics
+const similarityMetric = new ContentSimilarityMetric({
+  ignoreCase: true,
+  ignoreWhitespace: true,
+});
+
+const toxicityMetric = new ToxicityMetric({
+  model: openai('gpt-4'),
+  scale: 1, // Optional: adjust scoring scale
+});
+
+// Evaluate outputs
+const input = 'What is the capital of France?';
+const output = 'Paris is the capital of France.';
+
+const similarityResult = await similarityMetric.measure(input, output);
+const toxicityResult = await toxicityMetric.measure(input, output);
+
+console.log('Similarity Score:', similarityResult.score);
+console.log('Toxicity Score:', toxicityResult.score);
+```
+
+### Context-Aware Evaluation
+
+```typescript
+import { FaithfulnessMetric } from '@mastra/evals';
+
+// Initialize with context
+const faithfulnessMetric = new FaithfulnessMetric({
+  model: openai('gpt-4'),
+  context: ['Paris is the capital of France', 'Paris has a population of 2.2 million'],
+  scale: 1,
+});
+
+// Evaluate response against context
+const result = await faithfulnessMetric.measure(
+  'Tell me about Paris',
+  'Paris is the capital of France with 2.2 million residents',
+);
+
+console.log('Faithfulness Score:', result.score);
+console.log('Reasoning:', result.reason);
+```
+
+## Metric Results
+
+Each metric returns a standardized result object containing:
+
+- `score`: Normalized score (typically 0-1)
+- `info`: Detailed information about the evaluation
+- Additional metric-specific data (e.g., matched keywords, missing elements)
+
+Some metrics also provide:
+
+- `reason`: Detailed explanation of the score
+- `verdicts`: Individual judgments that contributed to the final score
+
+## Telemetry and Logging
+
+The package includes built-in telemetry and logging capabilities:
+
+- Automatic evaluation tracking through Mastra Storage
+- Integration with OpenTelemetry for performance monitoring
+- Detailed evaluation traces for debugging
+
+```typescript
+import { attachListeners } from '@mastra/evals';
+
+// Enable basic evaluation tracking
+await attachListeners();
+
+// Store evals in Mastra Storage (if storage is enabled)
+await attachListeners(mastra);
+// Note: When using in-memory storage, evaluations are isolated to the test process.
+// When using file storage, evaluations are persisted and can be queried later.
+```
+
+## Environment Variables
+
+Required for LLM-based metrics:
+
+- `OPENAI_API_KEY`: For OpenAI model access
+- Additional provider keys as needed (Cohere, Anthropic, etc.)
+
+## Package Exports
+
+```typescript
+// Main package exports
+import { evaluate } from '@mastra/evals';
+// NLP-specific metrics
+import { ContentSimilarityMetric } from '@mastra/evals/nlp';
+```
+
+## Related Packages
+
+- `@mastra/core`: Core framework functionality
+- `@mastra/engine`: LLM execution engine
+- `@mastra/mcp`: Model Context Protocol integration