@mastra/evals 1.5.0-alpha.1 → 1.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (4) hide show
  1. package/CHANGELOG.md +50 -0
  2. package/dist/docs/SKILL.md +1 -1
  3. package/dist/docs/assets/SOURCE_MAP.json +1 -1
  4. package/package.json +7 -7
package/CHANGELOG.md CHANGED
@@ -1,5 +1,55 @@
1
1
  # @mastra/evals
2
2
 
3
+ ## 1.5.0
4
+
5
+ ### Minor Changes
6
+
7
+ - Added `checks`, a new namespace of micro-scorers for common eval assertions. ([#18392](https://github.com/mastra-ai/mastra/pull/18392))
8
+
9
+ **What changed**
10
+ - Added text checks: `includes`, `excludes`, `equals`, `matches`, and `similarity`.
11
+ - Added tool checks: `calledTool`, `didNotCall`, `toolOrder`, `maxToolCalls`, `usedNoTools`, and `noToolErrors`.
12
+ - You can now import checks from `@mastra/evals/checks`.
13
+
14
+ **Example**
15
+
16
+ ```ts
17
+ import { checks } from '@mastra/evals/checks';
18
+
19
+ const scorers = [
20
+ checks.includes('sunny'),
21
+ checks.calledTool('get_weather'),
22
+ checks.toolOrder(['search', 'summarize']),
23
+ checks.noToolErrors(),
24
+ ];
25
+ ```
26
+
27
+ ### Patch Changes
28
+
29
+ - **Gates and verdict for `runEvals`** ([#18394](https://github.com/mastra-ai/mastra/pull/18394))
30
+
31
+ New optional `gates` field accepts scorers that must score 1.0 for the run to pass. Scorers can now use a `{ scorer, threshold }` form to set pass/fail thresholds. `threshold` accepts a number (minimum) or `{ min, max }` for range-based checks (e.g. hallucination where high = bad). The result includes `verdict`, `gateResults`, and `thresholdResults`. Fully backward compatible.
32
+
33
+ ```ts
34
+ import { runEvals } from '@mastra/core/evals';
35
+ import { checks } from '@mastra/evals/checks';
36
+
37
+ const result = await runEvals({
38
+ data: [{ input: 'What is the weather?' }],
39
+ target: weatherAgent,
40
+ gates: [checks.calledTool('get_weather')],
41
+ scorers: [
42
+ { scorer: faithfulnessScorer, threshold: 0.7 },
43
+ { scorer: hallucinationScorer, threshold: { max: 0.3 } },
44
+ ],
45
+ });
46
+
47
+ result.verdict; // 'passed' | 'scored' | 'failed'
48
+ ```
49
+
50
+ - Updated dependencies [[`86623c1`](https://github.com/mastra-ai/mastra/commit/86623c1adf7d22de32cc916dda17f4155184db36), [`023766f`](https://github.com/mastra-ai/mastra/commit/023766f44d59b30a50f3a381e33eddde8ab56c00), [`0200e75`](https://github.com/mastra-ai/mastra/commit/0200e7552d02d4221cd6040bf4eddf189a97a156), [`7c9dd77`](https://github.com/mastra-ai/mastra/commit/7c9dd77bd18cb8dc72797e25f1a0fbdc71a11347), [`7f9ae70`](https://github.com/mastra-ai/mastra/commit/7f9ae70826b047e5a66218f9e92f20e54a2d791f), [`a0509c7`](https://github.com/mastra-ai/mastra/commit/a0509c731a08aa3ed626557c5338126362856f57), [`06e0d63`](https://github.com/mastra-ai/mastra/commit/06e0d63d42bc2a202e18bc091f3781f409f5e6fb), [`bf3fe49`](https://github.com/mastra-ai/mastra/commit/bf3fe49f9467dbbdb8f9eaf74e0f7971ffb19559), [`01caf93`](https://github.com/mastra-ai/mastra/commit/01caf93d71ae2c1e65f49735cafb531975187426), [`438a971`](https://github.com/mastra-ai/mastra/commit/438a9715c8b4398e5eaf8914a1f19dc8a85dc1de), [`9990965`](https://github.com/mastra-ai/mastra/commit/999096571635a83b42ef40841fd7028cfa630779), [`77518cc`](https://github.com/mastra-ai/mastra/commit/77518ccb5bb8cc684875081e64213dc85cffdbee), [`fbeda0c`](https://github.com/mastra-ai/mastra/commit/fbeda0c0f35def07e6837936dd3a003b2b7c5172), [`8a68844`](https://github.com/mastra-ai/mastra/commit/8a688443013816105a09f89c6afa34b5ff13e26d), [`bb2a13b`](https://github.com/mastra-ai/mastra/commit/bb2a13bb4b32e6bb807200fe7b18ae8fa4322118), [`24ceaea`](https://github.com/mastra-ai/mastra/commit/24ceaea0bdd8609cabbab764380608ca6621a194), [`a73cd1a`](https://github.com/mastra-ai/mastra/commit/a73cd1a62a5e4ca023dcc39ba150029f4f1f74c1), [`c0ffa3c`](https://github.com/mastra-ai/mastra/commit/c0ffa3c897ccd326de880df734740a7f0681a18f), [`462a769`](https://github.com/mastra-ai/mastra/commit/462a769da61850862ca1be3d74134d33078ee6a7), [`0504bf5`](https://github.com/mastra-ai/mastra/commit/0504bf5e8cffc571a4b343326178de371e6f859b), [`0b5cc47`](https://github.com/mastra-ai/mastra/commit/0b5cc4726dc18d9a685a27520db39ff1b36bb89a), [`87f38a3`](https://github.com/mastra-ai/mastra/commit/87f38a3de03e24731f2dd6f8ed6a60b6722b85a1), [`d5fa3cd`](https://github.com/mastra-ai/mastra/commit/d5fa3cda1788c3cb93a361a3c6ec47de6ba21e98), [`fe98ef2`](https://github.com/mastra-ai/mastra/commit/fe98ef2e66dbfcbd7d645c88c9ee1e67b458a136), [`6ccf67b`](https://github.com/mastra-ai/mastra/commit/6ccf67bf075753754927a57bc2e1734ba2c820c5), [`793ea0f`](https://github.com/mastra-ai/mastra/commit/793ea0f52f831178837f21c83af6af93bf4ce638), [`825d8de`](https://github.com/mastra-ai/mastra/commit/825d8def9fa64c2bcc3d8dd6b49e09342c3ac5c7), [`507a5c4`](https://github.com/mastra-ai/mastra/commit/507a5c461bdc3136ad80744c0efbb55ce1f18f97), [`5afe423`](https://github.com/mastra-ai/mastra/commit/5afe423e4badf040f1b0d4525183a856fcb8146e), [`307573b`](https://github.com/mastra-ai/mastra/commit/307573b9ff3149b70c79540dbc86f1319b180f29), [`79b3626`](https://github.com/mastra-ai/mastra/commit/79b3626f8d647307eb07c8da14c9073c2699719d), [`c2c1d7b`](https://github.com/mastra-ai/mastra/commit/c2c1d7bb61d2602955f14ed3952f807c2d6eb576), [`86623c1`](https://github.com/mastra-ai/mastra/commit/86623c1adf7d22de32cc916dda17f4155184db36), [`1505c07`](https://github.com/mastra-ai/mastra/commit/1505c07603f6346bae12aa82f140e8b88ffea9ab), [`f328049`](https://github.com/mastra-ai/mastra/commit/f3280498c324afd2a8d36cd828f5b9f94a2dddc1), [`e545228`](https://github.com/mastra-ai/mastra/commit/e54522856934a5dc030b7b6385771e3548020d59), [`3eb852e`](https://github.com/mastra-ai/mastra/commit/3eb852e5435bc908b800193498103dc724f455b0), [`ffa09e7`](https://github.com/mastra-ai/mastra/commit/ffa09e772a5c92270eabe2090fc42d45bd8ec4b7), [`8c9f1c0`](https://github.com/mastra-ai/mastra/commit/8c9f1c0361d89066f9bcd14a2f69e761b01766c8), [`461a7c5`](https://github.com/mastra-ai/mastra/commit/461a7c501449295287f4f0ee4b0b42344f39fcf8), [`4211472`](https://github.com/mastra-ai/mastra/commit/4211472a5a2bd319c60cd2e42d9109c3eef7ac1c), [`9e45902`](https://github.com/mastra-ai/mastra/commit/9e4590208e745055cecca202e2db0e5c65e17d3c), [`5c0df77`](https://github.com/mastra-ai/mastra/commit/5c0df776c40efa420f8c07a2f3ee66010296618e), [`e940f09`](https://github.com/mastra-ai/mastra/commit/e940f099ef5d18b403e6f2b4937e086a4da857b1)]:
51
+ - @mastra/core@1.47.0
52
+
3
53
  ## 1.5.0-alpha.1
4
54
 
5
55
  ### Patch Changes
package/dist/docs/SKILL.md CHANGED
@@ -3,7 +3,7 @@ name: mastra-evals
3
3
  description: Documentation for @mastra/evals. Use when working with @mastra/evals APIs, configuration, or implementation.
4
4
  metadata:
5
5
  package: "@mastra/evals"
6
- version: "1.5.0-alpha.1"
6
+ version: "1.5.0"
7
7
  ---
8
8
 
9
9
  ## When to use
package/dist/docs/assets/SOURCE_MAP.json CHANGED
@@ -1,5 +1,5 @@
1
1
  {
2
- "version": "1.5.0-alpha.1",
2
+ "version": "1.5.0",
3
3
  "package": "@mastra/evals",
4
4
  "exports": {},
5
5
  "modules": {}
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@mastra/evals",
3
- "version": "1.5.0-alpha.1",
3
+ "version": "1.5.0",
4
4
  "description": "",
5
5
  "type": "module",
6
6
  "files": [
@@ -88,12 +88,12 @@
88
88
  "typescript": "^6.0.3",
89
89
  "vitest": "4.1.8",
90
90
  "zod": "^4.4.3",
91
- "@internal/ai-sdk-v5": "0.0.55",
92
- "@internal/lint": "0.0.108",
93
- "@internal/llm-recorder": "0.0.44",
94
- "@internal/types-builder": "0.0.83",
95
- "@internal/test-utils": "0.0.44",
96
- "@mastra/core": "1.47.0-alpha.2"
91
+ "@internal/lint": "0.0.109",
92
+ "@internal/ai-sdk-v5": "0.0.56",
93
+ "@internal/llm-recorder": "0.0.45",
94
+ "@internal/test-utils": "0.0.45",
95
+ "@internal/types-builder": "0.0.84",
96
+ "@mastra/core": "1.47.0"
97
97
  },
98
98
  "engines": {
99
99
  "node": ">=22.13.0"