@mastra/evals 1.7.0-alpha.0 → 1.7.0

package/CHANGELOG.md

@@ -1,5 +1,34 @@
 # @mastra/evals
 
+## 1.7.0
+
+### Minor Changes
+
+- Added a `summarization` scorer to `@mastra/evals`. It grades a summary on two axes and returns the lower score, so a summary cannot pass by being faithful but empty, or thorough but wrong. ([#20293](https://github.com/mastra-ai/mastra/pull/20293))
+
+  **Alignment** checks that every claim in the summary is supported by the source text. **Coverage** draws closed-ended questions from the source and answers them using the summary alone, in a separate call that never receives the source, so a missing fact cannot be answered from the source instead. The final score is `min(alignment, coverage) × scale`, and the reason names the axis that produced it.
+
+  The source text defaults to the user message of the run input. Pass `source` or `sourceExtractor` when the text being summarized comes from somewhere else, such as a tool result. `maxQuestions` bounds the coverage questions so cost does not grow with document length.
+
+  ```ts
+  import { createSummarizationScorer } from '@mastra/evals/scorers/prebuilt';
+
+  const scorer = createSummarizationScorer({
+    model: 'openai/gpt-5.5',
+    options: { maxQuestions: 10 },
+  });
+
+  const result = await scorer.run(run);
+  result.score;
+  ```
+
+  This restores the summarization metric that was removed with the legacy evals system, rebuilt on the scorers pipeline.
+
+### Patch Changes
+
+- Updated dependencies [[`4844167`](https://github.com/mastra-ai/mastra/commit/4844167cff2d5ec5004e94edd34970833040fa3f), [`c5e56ff`](https://github.com/mastra-ai/mastra/commit/c5e56ff3bcabdf062708f2d48744fec304df6792), [`594f7b2`](https://github.com/mastra-ai/mastra/commit/594f7b28f5263fb9982fd50d95c471fb971ea984), [`7f4e26d`](https://github.com/mastra-ai/mastra/commit/7f4e26dd57bd9b23c278ea21235ab823a3810a6c), [`311f943`](https://github.com/mastra-ai/mastra/commit/311f943bee60e8fdf5c84499ea50e884276c936c), [`322daa6`](https://github.com/mastra-ai/mastra/commit/322daa6d90552909204044790d850958f6745fed), [`db4e6ff`](https://github.com/mastra-ai/mastra/commit/db4e6ff744503112eb64deeaf6c2b54bf26a54c7), [`5faf93f`](https://github.com/mastra-ai/mastra/commit/5faf93f03e19daea394b9e2a923f2e4f833407f2), [`82201f7`](https://github.com/mastra-ai/mastra/commit/82201f75fae8e050a8de2df08b74875ee74c6b83), [`cadaa13`](https://github.com/mastra-ai/mastra/commit/cadaa1372e1077c8e85eb64c5499ba8803caa323), [`0c89896`](https://github.com/mastra-ai/mastra/commit/0c8989673fb7d106837098398131e570c6023b68), [`6d19a65`](https://github.com/mastra-ai/mastra/commit/6d19a6517f5da3911023d446b7e2d5dad8adb1cb), [`23b4238`](https://github.com/mastra-ai/mastra/commit/23b423844ad0bcf2a502a68dd62866d6160f9f6d), [`80ad891`](https://github.com/mastra-ai/mastra/commit/80ad891f8cd10379aa5b5af7510c763783b2ab56), [`fb18da5`](https://github.com/mastra-ai/mastra/commit/fb18da56fc35689ae370621a8f10b5b0d8606e20), [`fb18da5`](https://github.com/mastra-ai/mastra/commit/fb18da56fc35689ae370621a8f10b5b0d8606e20), [`e320a76`](https://github.com/mastra-ai/mastra/commit/e320a763feaf65c6be3cebecf746defcbde161b3), [`03b4918`](https://github.com/mastra-ai/mastra/commit/03b4918c80d188ce375334c393e131c6e94bd7eb), [`14ef73a`](https://github.com/mastra-ai/mastra/commit/14ef73a4bbd73e7808414816eb0628ce1d80b5d7), [`b582f7f`](https://github.com/mastra-ai/mastra/commit/b582f7fa2f9c1f87d19efc63d344fbe5dda2608c), [`0a6598b`](https://github.com/mastra-ai/mastra/commit/0a6598bde80bde008986ad6616bed9632b9294cb), [`06000d7`](https://github.com/mastra-ai/mastra/commit/06000d73712911572e913b8a83339270296d0a22), [`1d677d5`](https://github.com/mastra-ai/mastra/commit/1d677d5f99d7db403f7828585e8c25f299f72628), [`9e1dad8`](https://github.com/mastra-ai/mastra/commit/9e1dad8f7b1cab2bb7ade90e5b7561f24577b88a), [`2f43145`](https://github.com/mastra-ai/mastra/commit/2f4314504c03cbba280414ac81ba3197448ee6b0), [`4e35a56`](https://github.com/mastra-ai/mastra/commit/4e35a56cdf8d74a5ff6d5eda01f2c1deaf6cc7be), [`d94b8e1`](https://github.com/mastra-ai/mastra/commit/d94b8e1cee67416d518a8c30099040061bef6a1c), [`93e28ec`](https://github.com/mastra-ai/mastra/commit/93e28ecce9031c02397e0ae8406593e5c7a95883), [`729dab4`](https://github.com/mastra-ai/mastra/commit/729dab408faccfaef0cbb048e5a4338f9172847e), [`484003d`](https://github.com/mastra-ai/mastra/commit/484003d33ff59330c86b19863e4a38732d7e4155), [`3de0188`](https://github.com/mastra-ai/mastra/commit/3de0188bfaf9a9c09c95fe322b53838cf52c70b6), [`34d34d8`](https://github.com/mastra-ai/mastra/commit/34d34d8c811df512fef4dd5459f79b7821be1866), [`b582f7f`](https://github.com/mastra-ai/mastra/commit/b582f7fa2f9c1f87d19efc63d344fbe5dda2608c), [`933d291`](https://github.com/mastra-ai/mastra/commit/933d291146b789c19442ad206f94da3e4be90c64), [`a1cb98d`](https://github.com/mastra-ai/mastra/commit/a1cb98d11990b560b98482292a1f34aa1a2d9092), [`598ad82`](https://github.com/mastra-ai/mastra/commit/598ad82d41c41389a686338a1d0e50b7400e1938), [`1fd6aad`](https://github.com/mastra-ai/mastra/commit/1fd6aad1ea4a9d32f65efa832307c35e981a4c0a)]:
+  - @mastra/core@1.56.0
+
 ## 1.7.0-alpha.0
 
 ### Minor Changes

package/dist/docs/SKILL.md

@@ -3,7 +3,7 @@ name: mastra-evals
 description: Documentation for @mastra/evals. Use when working with @mastra/evals APIs, configuration, or implementation.
 metadata:
   package: "@mastra/evals"
-  version: "1.7.0-alpha.0"
+  version: "1.7.0"
 ---
 
 ## When to use

package/dist/docs/assets/SOURCE_MAP.json

@@ -1,5 +1,5 @@
 {
-  "version": "1.7.0-alpha.0",
+  "version": "1.7.0",
   "package": "@mastra/evals",
   "exports": {},
   "modules": {}

package/dist/docs/references/reference-evals-summarization.md

@@ -2,7 +2,7 @@
 
 # Summarization scorer
 
-The `createSummarizationScorer()` function creates a scorer that evaluates a summary on two axes: whether every claim it makes is supported by the source text, and whether it preserves the information the source states. The final score is the lower of the two, so a summary cannot pass by being faithful but empty, or thorough but wrong.
+The `createSummarizationScorer()` function creates a scorer that evaluates a summary on two axes: whether every claim it makes is supported by the source text, and whether it preserves the information the source states. The final score is the lower of the two, so a summary can't pass by being faithful but empty, or thorough but wrong.
 
 The summary is the agent's last message that carries text, and the source text defaults to the first user message of the run input. Pass `source` or `sourceExtractor` when the text being summarized lives somewhere else, such as a tool result.
 
@@ -96,13 +96,13 @@ These ranges assume the default `scale` of 1. When using a custom scale, multipl
 
 - **0.9-1.0**: Excellent summary, faithful to the source and covering its main points
 - **0.7-0.8**: Good summary with a small omission or an unsupported detail
-- **0.4-0.6**: Moderate summary, either missing significant information or drifting from the source
+- **0.4-0.6**: Moderate summary, either missing important information or drifting from the source
 - **0.1-0.3**: Poor summary, most of the source is lost or contradicted
-- **0.0**: The summary supports no claims, answers no questions, or produced nothing to judge
+- **0.0**: The summary produced nothing to judge, or it failed to support any claims. A summary that answers no questions also receives this score
 
 ### Reading the two axes
 
-Both axes leave their verdicts on the run result: the alignment verdicts on the preprocess step, and the coverage verdicts on the analyze step. Each verdict carries the claim or question it belongs to and the reason behind it. The two failure modes look different:
+Both axes leave their verdicts on the run result: the alignment verdicts on the preprocess step, and the coverage verdicts on the analyze step. Each verdict carries the claim or question it belongs to and the reason behind it. A low alignment score has a different meaning from a low coverage score:
 
 - A low alignment score with high coverage means the summary invents or distorts detail
 - A low coverage score with high alignment means the summary is accurate but leaves too much out
@@ -115,7 +115,7 @@ Length plays no part in the score. A summary that repeats the source word for wo
 
 ### Cost
 
-Each evaluation makes three model calls. `maxQuestions` bounds the coverage half of the work, which otherwise grows with source length. Raise it for long documents where ten questions cannot represent the content.
+Each evaluation makes three model calls. `maxQuestions` bounds the coverage half of the work, which otherwise grows with source length. Raise it for long documents where ten questions can't represent the content.
 
 ## Scorer configuration
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@mastra/evals",
-  "version": "1.7.0-alpha.0",
+  "version": "1.7.0",
   "description": "",
   "type": "module",
   "files": [
@@ -88,12 +88,12 @@
     "typescript": "^6.0.3",
     "vitest": "4.1.10",
     "zod": "^4.4.3",
-    "@internal/ai-sdk-v5": "0.0.66",
-    "@internal/lint": "0.0.119",
-    "@internal/llm-recorder": "0.0.55",
-    "@internal/types-builder": "0.0.94",
-    "@internal/test-utils": "0.0.55",
-    "@mastra/core": "1.56.0-alpha.3"
+    "@internal/ai-sdk-v5": "0.0.67",
+    "@internal/lint": "0.0.120",
+    "@internal/llm-recorder": "0.0.56",
+    "@internal/test-utils": "0.0.56",
+    "@mastra/core": "1.56.0",
+    "@internal/types-builder": "0.0.95"
   },
   "engines": {
     "node": ">=22.13.0"