vitest-evals 0.1.1 → 0.1.3

package/README.md

@@ -1,8 +1,8 @@
 # vitest-evals
 
-This project is a WIP prototype of extending vitest to support basic _Evals_ functionality. Evals are a type of testing that is most commonly deployed to _evaluate_ the results of calls to language models. This allows you to utilize them with a pattern of testing you're familiar with, working well with your existing continuous integration toolchain.
+This project is a prototype of extending vitest to support basic _Evals_ functionality. Evals are a type of testing that is most commonly deployed to _evaluate_ the results of calls to language models. This allows you to utilize them with a pattern of testing you're familiar with, working well with your existing continuous integration toolchain.
 
-This is heavily inspired by [Evalite](https://www.evalite.dev/), but opts for a vitest-native approach to maximize the compatibility of the existing ecosystem.
+This is heavily inspired by [Evalite](https://www.evalite.dev/), but opts for a vitest-native approach to maximize the compatibility of the existing ecosystem. This means you can use it with your existing toolchain, including reporting such as code coverage and xunit.
 
 ## Use
 
@@ -10,104 +10,173 @@ This is heavily inspired by [Evalite](https://www.evalite.dev/), but opts for a
 npm install -D vitest-evals
 ```
 
-### Dedicated Test Suites
+You've likely already got a mechanism for passing the user input into your model, for example:
 
 ```javascript
-// Inspired by the `evalite` API
+async function answerQuestion(prompt: string) {
+  const model = openai("gpt-4o");
+  const { text } = await generateText({
+    model,
+    prompt,
+  })
+  return text;
+}
+```
+
+You'll use this as the `task` within your evals, and then you simply need to define a set of scenarios
+and a way to validate if the LLM is responding as you desire:
+
+```javascript
+import { describeEval } from "vitest-evals";
+import { Factuality } from "autoevals";
+
 describeEval("my evals", {
   data: async () => {
+    // The scenarios you wish to evaluate
     return [
       input: "What is the capital of France?",
       expected: "Paris",
     ];
   },
-  task: async (input) => {
-    // TODO: query an LLM using a factuality checker
-    const output = 'Paris';
-    return output;
-  },
-  scorers: [checkFactuality],
-  threshold: 0.8,
-  // timeout: 10000, // default timeout increased from 5s to 10s
+
+  task: answerQuestion,
+
+  // Scorers determine if the response was acceptable - in this case we're using
+  // a secondary LLM prompt to judge the response of the first.
+  scorers: [Factuality],
+
+  // The threshold required for the average score for this eval to pass. This will be
+  // based on the scorers you've provided, and in the case of Factuality, we might be
+  // ok with a 60% score (see the implementation for why).
+  threshold: 0.6,
+
+  // The timeout for each test. Defaults to 10s. You may need to increase this if your model
+  // provider has high latency or you're using a large number of scorers.
+  // timeout: 10000,
+
+  // A check to determine if these tests should run. This is helpful to control tests so they only
+  // in certain situations, for example if a model providers API key is defined.
+  // skipIf: () => !process.env.OPENAI_API_KEY
 })
 ```
 
 ### Existing Test Suites
 
 ```javascript
-// import vitest-evals to expose expect().toEval();
+// import `vitest-evals` to expose `expect().toEval()`
+// This can also be done via `setupFiles` pattern in `vitest`.
 import "vitest-evals";
-import { generateObject } from "ai";
-
-function askTheLLM(input: string) {
-  // TODO: query an LLM using a factuality checker
-  const output = 'Paris';
-  return output;
-}
+import { Factuality } from "autoevals";
 
 describe("my test suite", () => {
   it("kind of works", () => {
-    expect("What is the capital of France?").toEval("Paris", askTheLLM, checkFactuality, 0.8)
+    expect("What is the capital of France?").toEval("Paris", answerQuestion, Factuality, 0.8)
   });
 });
 ```
 
-### Factuality Scoring
+### Scoring
+
+Scorers are compatible with the `autoevals` interface, but are also simple to implement on your own:
 
 ```javascript
-export const checkFactuality = async (opts: {
+export const Contains = async (opts: {
   input: string;
   expected: string;
   output: string;
 }) => {
-  const { object } = await generateObject({
-    model,
-    /**
-     * Prompt taken from autoevals:
-     *
-     * {@link https://github.com/braintrustdata/autoevals/blob/5aa20a0a9eb8fc9e07e9e5722ebf71c68d082f32/templates/factuality.yaml}
-     */
-    prompt: `
-      You are comparing a submitted answer to an expert answer on a given question. Here is the data:
-      [BEGIN DATA]
-      ************
-      [Question]: ${opts.input}
-      ************
-      [Expert]: ${opts.expected}
-      ************
-      [Submission]: ${opts.output}
-      ************
-      [END DATA]
-
-      Compare the factual content of the submitted answer with the expert answer. Ignore any differences in style, grammar, or punctuation.
-      The submitted answer may either be a subset or superset of the expert answer, or it may conflict with it. Determine which case applies. Answer the question by selecting one of the following options:
-      (A) The submitted answer is a subset of the expert answer and is fully consistent with it.
-      (B) The submitted answer is a superset of the expert answer and is fully consistent with it.
-      (C) The submitted answer contains all the same details as the expert answer.
-      (D) There is a disagreement between the submitted answer and the expert answer.
-      (E) The answers differ, but these differences don't matter from the perspective of factuality.
-    `,
-    schema: z.object({
-      answer: z.enum(["A", "B", "C", "D", "E"]).describe("Your selection."),
-      rationale: z
-        .string()
-        .describe("Why you chose this answer. Be very detailed."),
-    }),
-  });
-
-  const scores = {
-    A: 0.4,
-    B: 0.6,
-    C: 1,
-    D: 0,
-    E: 1,
+  return {
+    score: output.indexOf(expected) !== -1 ? 1.0 : 0.0,
   };
+}
+```
 
-  return {
-    score: scores[object.answer],
-    metadata: {
-      rationale: object.rationale,
-    },
+For something more realistic, here's a reimplementation of the Factuality scorer from `autoevals`, with some flexibility
+on the model, enabling you to evaluate against multiple models:
+
+```javascript
+import { generateObject, type LanguageModel } from "ai";
+import { z } from "zod";
+
+/**
+ * A Factuality checker utilizing the `ai` SDK based on the implementation in `autoevals`.
+ * 
+ * @param model - The language model to utilize (via `ai`).
+ * 
+ * @example
+ * ```javascript
+ * import { openai } from "@ai-sdk/openai";
+ *
+ * scorers: [Factuality(openai("gpt-4o"))]
+ * ```
+ */
+export function Factuality(model: LanguageModel) {
+  return async (opts: {
+    input: string;
+    expected: string;
+    output: string;
+  }) => {
+    const { object } = await generateObject({
+      model,
+      /**
+       * Prompt implementation from `autoevals`:
+       *
+       * {@link https://github.com/braintrustdata/autoevals/blob/5aa20a0a9eb8fc9e07e9e5722ebf71c68d082f32/templates/factuality.yaml}
+       */
+      prompt: `
+        You are comparing a submitted answer to an expert answer on a given question. Here is the data:
+        [BEGIN DATA]
+        ************
+        [Question]: ${opts.input}
+        ************
+        [Expert]: ${opts.expected}
+        ************
+        [Submission]: ${opts.output}
+        ************
+        [END DATA]
+
+        Compare the factual content of the submitted answer with the expert answer. Ignore any differences in style, grammar, or punctuation.
+        The submitted answer may either be a subset or superset of the expert answer, or it may conflict with it. Determine which case applies. Answer the question by selecting one of the following options:
+        (A) The submitted answer is a subset of the expert answer and is fully consistent with it.
+        (B) The submitted answer is a superset of the expert answer and is fully consistent with it.
+        (C) The submitted answer contains all the same details as the expert answer.
+        (D) There is a disagreement between the submitted answer and the expert answer.
+        (E) The answers differ, but these differences don't matter from the perspective of factuality.
+      `,
+      schema: z.object({
+        answer: z.enum(["A", "B", "C", "D", "E"]).describe("Your selection."),
+        rationale: z
+          .string()
+          .describe("Why you chose this answer. Be very detailed."),
+      }),
+    });
+
+    const scores = {
+      A: 0.4,
+      B: 0.6,
+      C: 1,
+      D: 0,
+      E: 1,
+    };
+
+    return {
+      score: scores[object.answer],
+      metadata: {
+        rationale: object.rationale,
+      },
+    };
   };
-};
+}
+```
+
+## Development
+
+Nothing fancy here.
+
+```javascript
+pnpm install
+```
+
+```javascript
+pnpm test
 ```

package/dist/index.d.mts

@@ -4,14 +4,15 @@ type TaskFn = (input: string) => Promise<string>;
 type Score = {
     score: number | null;
     metadata?: {
-        rationale: string;
+        rationale?: string;
+        output?: string;
     };
 };
 type ScoreFn = (opts: {
     input: string;
-    expected: string;
     output: string;
-}) => Promise<Score>;
+    expected?: string;
+}) => Promise<Score> | Score;
 type ToEval<R = unknown> = (expected: string, taskFn: TaskFn, scoreFn: ScoreFn, threshold?: number) => Promise<R>;
 interface EvalMatchers<R = unknown> {
     toEval: ToEval<R>;
@@ -22,15 +23,47 @@ declare module "vitest" {
     interface AsymmetricMatchersContaining extends EvalMatchers {
     }
 }
-declare function describeEval(name: string, { data, task, scorers, threshold, timeout, }: {
+/**
+ * Creates a test suite for evaluating language model outputs.
+ *
+ * @param name - The name of the test suite
+ * @param options - Configuration options
+ * @param options.data - Async function that returns an array of test cases with input and expected values
+ * @param options.task - Function that processes the input and returns the model output
+ * @param options.skipIf - Optional function that determines if tests should be skipped
+ * @param options.scorers - Array of scoring functions that evaluate model outputs
+ * @param options.threshold - Minimum acceptable average score (0-1), defaults to 1.0
+ * @param options.timeout - Test timeout in milliseconds, defaults to 10000
+ *
+ * @example
+ * ```javascript
+ * describeEval("capital cities test", {
+ *   data: async () => [{
+ *     input: "What is the capital of France?",
+ *     expected: "Paris"
+ *   }],
+ *   task: async (input) => {
+ *     // Query LLM here
+ *     return "Paris";
+ *   },
+ *   scorers: [checkFactuality],
+ *   threshold: 0.8
+ * });
+ * ```
+ */
+declare function describeEval(name: string, { data, task, skipIf, scorers, threshold, timeout, }: {
     data: () => Promise<{
         input: string;
         expected: string;
     }[]>;
     task: TaskFn;
+    skipIf?: () => boolean;
     scorers: ScoreFn[];
     threshold?: number | null;
     timeout?: number;
 }): vitest.SuiteCollector<object>;
+declare function formatScores(scores: (Score & {
+    name: string;
+})[]): string;
 
-export { type EvalMatchers, type Score, type ScoreFn, type TaskFn, type ToEval, describeEval };
+export { type EvalMatchers, type Score, type ScoreFn, type TaskFn, type ToEval, describeEval, formatScores };

package/dist/index.d.ts

@@ -4,14 +4,15 @@ type TaskFn = (input: string) => Promise<string>;
 type Score = {
     score: number | null;
     metadata?: {
-        rationale: string;
+        rationale?: string;
+        output?: string;
     };
 };
 type ScoreFn = (opts: {
     input: string;
-    expected: string;
     output: string;
-}) => Promise<Score>;
+    expected?: string;
+}) => Promise<Score> | Score;
 type ToEval<R = unknown> = (expected: string, taskFn: TaskFn, scoreFn: ScoreFn, threshold?: number) => Promise<R>;
 interface EvalMatchers<R = unknown> {
     toEval: ToEval<R>;
@@ -22,15 +23,47 @@ declare module "vitest" {
     interface AsymmetricMatchersContaining extends EvalMatchers {
     }
 }
-declare function describeEval(name: string, { data, task, scorers, threshold, timeout, }: {
+/**
+ * Creates a test suite for evaluating language model outputs.
+ *
+ * @param name - The name of the test suite
+ * @param options - Configuration options
+ * @param options.data - Async function that returns an array of test cases with input and expected values
+ * @param options.task - Function that processes the input and returns the model output
+ * @param options.skipIf - Optional function that determines if tests should be skipped
+ * @param options.scorers - Array of scoring functions that evaluate model outputs
+ * @param options.threshold - Minimum acceptable average score (0-1), defaults to 1.0
+ * @param options.timeout - Test timeout in milliseconds, defaults to 10000
+ *
+ * @example
+ * ```javascript
+ * describeEval("capital cities test", {
+ *   data: async () => [{
+ *     input: "What is the capital of France?",
+ *     expected: "Paris"
+ *   }],
+ *   task: async (input) => {
+ *     // Query LLM here
+ *     return "Paris";
+ *   },
+ *   scorers: [checkFactuality],
+ *   threshold: 0.8
+ * });
+ * ```
+ */
+declare function describeEval(name: string, { data, task, skipIf, scorers, threshold, timeout, }: {
     data: () => Promise<{
         input: string;
         expected: string;
     }[]>;
     task: TaskFn;
+    skipIf?: () => boolean;
     scorers: ScoreFn[];
     threshold?: number | null;
     timeout?: number;
 }): vitest.SuiteCollector<object>;
+declare function formatScores(scores: (Score & {
+    name: string;
+})[]): string;
 
-export { type EvalMatchers, type Score, type ScoreFn, type TaskFn, type ToEval, describeEval };
+export { type EvalMatchers, type Score, type ScoreFn, type TaskFn, type ToEval, describeEval, formatScores };

package/dist/index.js

@@ -1,8 +1,25 @@
 "use strict";
 var __defProp = Object.defineProperty;
+var __defProps = Object.defineProperties;
 var __getOwnPropDesc = Object.getOwnPropertyDescriptor;
+var __getOwnPropDescs = Object.getOwnPropertyDescriptors;
 var __getOwnPropNames = Object.getOwnPropertyNames;
+var __getOwnPropSymbols = Object.getOwnPropertySymbols;
 var __hasOwnProp = Object.prototype.hasOwnProperty;
+var __propIsEnum = Object.prototype.propertyIsEnumerable;
+var __defNormalProp = (obj, key, value) => key in obj ? __defProp(obj, key, { enumerable: true, configurable: true, writable: true, value }) : obj[key] = value;
+var __spreadValues = (a, b) => {
+  for (var prop in b || (b = {}))
+    if (__hasOwnProp.call(b, prop))
+      __defNormalProp(a, prop, b[prop]);
+  if (__getOwnPropSymbols)
+    for (var prop of __getOwnPropSymbols(b)) {
+      if (__propIsEnum.call(b, prop))
+        __defNormalProp(a, prop, b[prop]);
+    }
+  return a;
+};
+var __spreadProps = (a, b) => __defProps(a, __getOwnPropDescs(b));
 var __export = (target, all) => {
   for (var name in all)
     __defProp(target, name, { get: all[name], enumerable: true });
@@ -40,18 +57,45 @@ var __async = (__this, __arguments, generator) => {
 // src/index.ts
 var index_exports = {};
 __export(index_exports, {
-  describeEval: () => describeEval
+  describeEval: () => describeEval,
+  formatScores: () => formatScores
 });
 module.exports = __toCommonJS(index_exports);
 var import_vitest = require("vitest");
 var import_vitest2 = require("vitest");
 import_vitest.expect.extend({
+  /**
+   * Evaluates a language model output against an expected answer using a scoring function.
+   *
+   * @param expected - The expected (ground truth) answer
+   * @param taskFn - Async function that processes the input and returns the model output
+   * @param scoreFn - Function that evaluates the model output against the expected answer
+   * @param threshold - Minimum acceptable score (0-1), defaults to 1.0
+   *
+   * @example
+   * ```javascript
+   * test("checks capital of France", async () => {
+   *   expect("What is the capital of France?").toEval(
+   *     "Paris",
+   *     async (input) => {
+   *       // Query LLM here
+   *       return "Paris";
+   *     },
+   *     checkFactuality,
+   *     0.8
+   *   );
+   * });
+   * ```
+   */
   toEval: function toEval(input, expected, taskFn, scoreFn, threshold = 1) {
     return __async(this, null, function* () {
       var _a;
       const { isNot } = this;
       const output = yield taskFn(input);
-      const result = yield scoreFn({ input, expected, output });
+      let result = scoreFn({ input, expected, output });
+      if (result instanceof Promise) {
+        result = yield result;
+      }
       return {
         pass: ((_a = result.score) != null ? _a : 0) >= threshold,
         message: () => `Score: ${result.score} ${isNot ? "<" : ">"} ${threshold}
@@ -63,6 +107,7 @@ ${result.metadata ? `Rationale: ${result.metadata.rationale}` : ""}`
 function describeEval(name, {
   data,
   task,
+  skipIf,
   scorers,
   threshold = 1,
   // increase default test timeout as 5s is usually not enough for
@@ -70,14 +115,27 @@ function describeEval(name, {
   timeout = 1e4
 }) {
   return (0, import_vitest.describe)(name, () => __async(this, null, function* () {
+    const testFn = skipIf ? import_vitest.test.skipIf(skipIf()) : import_vitest.test;
     for (const { input, expected } of yield data()) {
-      (0, import_vitest.it)(
+      testFn(
         input,
+        {
+          timeout
+        },
         () => __async(this, null, function* () {
           const output = yield task(input);
           const scores = yield Promise.all(
-            scorers.map((scorer) => scorer({ input, expected, output }))
+            scorers.map((scorer) => {
+              const result = scorer({ input, expected, output });
+              if (result instanceof Promise) {
+                return result;
+              }
+              return new Promise((resolve) => resolve(result));
+            })
           );
+          const scoresWithName = scores.map((s, i) => __spreadProps(__spreadValues({}, s), {
+            name: scorers[i].name
+          }));
           const avgScore = scores.reduce((acc, s) => {
             var _a;
             return acc + ((_a = s.score) != null ? _a : 0);
@@ -86,19 +144,35 @@ function describeEval(name, {
             (0, import_vitest.assert)(
               avgScore >= threshold,
               `Score: ${avgScore} below threshold: ${threshold}
-Output: ${output}`
+Output: ${output}
+${formatScores(
+                scoresWithName
+              )}`
             );
           }
-        }),
-        {
-          timeout
-        }
+        })
       );
     }
   }));
 }
+function formatScores(scores) {
+  return scores.sort((a, b) => {
+    var _a, _b;
+    return ((_a = a.score) != null ? _a : 0) - ((_b = b.score) != null ? _b : 0);
+  }).map((s) => {
+    var _a, _b, _c, _d;
+    const scoreLine = `${s.name || "Unknown"} [${((_a = s.score) != null ? _a : 0).toFixed(1)}]`;
+    if (((_b = s.score) != null ? _b : 0) < 1 && ((_c = s.metadata) == null ? void 0 : _c.rationale) || ((_d = s.metadata) == null ? void 0 : _d.output)) {
+      return `${scoreLine}${s.metadata.rationale ? `
+Rationale: ${s.metadata.rationale}` : ""}${s.metadata.output ? `
+Output: ${s.metadata.output}` : ""}`;
+    }
+    return scoreLine;
+  }).join("\n\n");
+}
 // Annotate the CommonJS export names for ESM import in node:
 0 && (module.exports = {
-  describeEval
+  describeEval,
+  formatScores
 });
 //# sourceMappingURL=index.js.map

package/dist/index.js.map

@@ -1 +1 @@
-{"version":3,"sources":["../src/index.ts"],"sourcesContent":["import { assert, describe, expect, it } from \"vitest\";\nimport \"vitest\";\n\nexport type TaskFn = (input: string) => Promise<string>;\n\nexport type Score = {\n  score: number | null;\n  metadata?: {\n    rationale: string;\n  };\n};\n\nexport type ScoreFn = (opts: {\n  input: string;\n  expected: string;\n  output: string;\n}) => Promise<Score>;\n\nexport type ToEval<R = unknown> = (\n  expected: string,\n  taskFn: TaskFn,\n  scoreFn: ScoreFn,\n  threshold?: number,\n) => Promise<R>;\n\nexport interface EvalMatchers<R = unknown> {\n  toEval: ToEval<R>;\n}\n\ndeclare module \"vitest\" {\n  interface Assertion<T = any> extends EvalMatchers<T> {}\n  interface AsymmetricMatchersContaining extends EvalMatchers {}\n}\n\nexpect.extend({\n  toEval: async function toEval(\n    input: string,\n    expected: string,\n    taskFn: TaskFn,\n    scoreFn: ScoreFn,\n    threshold = 1.0,\n  ) {\n    const { isNot } = this;\n\n    const output = await taskFn(input);\n\n    const result = await scoreFn({ input, expected, output });\n\n    return {\n      pass: (result.score ?? 0) >= threshold,\n      message: () =>\n        `Score: ${result.score} ${isNot ? \"<\" : \">\"} ${threshold}\\n${\n          result.metadata ? `Rationale: ${result.metadata.rationale}` : \"\"\n        }`,\n    };\n  },\n});\n\nexport function describeEval(\n  name: string,\n  {\n    data,\n    task,\n    scorers,\n    threshold = 1.0,\n    // increase default test timeout as 5s is usually not enough for\n    // a single factuality check\n    timeout = 10000,\n  }: {\n    data: () => Promise<{ input: string; expected: string }[]>;\n    task: TaskFn;\n    scorers: ScoreFn[];\n    threshold?: number | null;\n    timeout?: number;\n  },\n) {\n  return describe(name, async () => {\n    // TODO: should data just be a generator?\n    for (const { input, expected } of await data()) {\n      it(\n        input,\n        async () => {\n          const output = await task(input);\n\n          const scores = await Promise.all(\n            scorers.map((scorer) => scorer({ input, expected, output })),\n          );\n\n          const avgScore =\n            scores.reduce((acc, s) => acc + (s.score ?? 0), 0) / scores.length;\n          if (threshold) {\n            assert(\n              avgScore >= threshold,\n              `Score: ${avgScore} below threshold: ${threshold}\\nOutput: ${output}`,\n            );\n          }\n        },\n        {\n          timeout,\n        },\n      );\n    }\n  });\n}\n"],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,oBAA6C;AAC7C,IAAAA,iBAAO;AAiCP,qBAAO,OAAO;AAAA,EACZ,QAAQ,SAAe,OACrB,OACA,UACA,QACA,SACA,YAAY,GACZ;AAAA;AAzCJ;AA0CI,YAAM,EAAE,MAAM,IAAI;AAElB,YAAM,SAAS,MAAM,OAAO,KAAK;AAEjC,YAAM,SAAS,MAAM,QAAQ,EAAE,OAAO,UAAU,OAAO,CAAC;AAExD,aAAO;AAAA,QACL,QAAO,YAAO,UAAP,YAAgB,MAAM;AAAA,QAC7B,SAAS,MACP,UAAU,OAAO,KAAK,IAAI,QAAQ,MAAM,GAAG,IAAI,SAAS;AAAA,EACtD,OAAO,WAAW,cAAc,OAAO,SAAS,SAAS,KAAK,EAChE;AAAA,MACJ;AAAA,IACF;AAAA;AACF,CAAC;AAEM,SAAS,aACd,MACA;AAAA,EACE;AAAA,EACA;AAAA,EACA;AAAA,EACA,YAAY;AAAA;AAAA;AAAA,EAGZ,UAAU;AACZ,GAOA;AACA,aAAO,wBAAS,MAAM,MAAY;AAEhC,eAAW,EAAE,OAAO,SAAS,KAAK,MAAM,KAAK,GAAG;AAC9C;AAAA,QACE;AAAA,QACA,MAAY;AACV,gBAAM,SAAS,MAAM,KAAK,KAAK;AAE/B,gBAAM,SAAS,MAAM,QAAQ;AAAA,YAC3B,QAAQ,IAAI,CAAC,WAAW,OAAO,EAAE,OAAO,UAAU,OAAO,CAAC,CAAC;AAAA,UAC7D;AAEA,gBAAM,WACJ,OAAO,OAAO,CAAC,KAAK,MAAG;AAzFnC;AAyFsC,2BAAO,OAAE,UAAF,YAAW;AAAA,aAAI,CAAC,IAAI,OAAO;AAC9D,cAAI,WAAW;AACb;AAAA,cACE,YAAY;AAAA,cACZ,UAAU,QAAQ,qBAAqB,SAAS;AAAA,UAAa,MAAM;AAAA,YACrE;AAAA,UACF;AAAA,QACF;AAAA,QACA;AAAA,UACE;AAAA,QACF;AAAA,MACF;AAAA,IACF;AAAA,EACF,EAAC;AACH;","names":["import_vitest"]}
+{"version":3,"sources":["../src/index.ts"],"sourcesContent":["import { assert, describe, expect, test } from \"vitest\";\nimport \"vitest\";\n\nexport type TaskFn = (input: string) => Promise<string>;\n\nexport type Score = {\n  score: number | null;\n  metadata?: {\n    rationale?: string;\n    output?: string;\n  };\n};\n\nexport type ScoreFn = (opts: {\n  input: string;\n  output: string;\n  expected?: string;\n}) => Promise<Score> | Score;\n\nexport type ToEval<R = unknown> = (\n  expected: string,\n  taskFn: TaskFn,\n  scoreFn: ScoreFn,\n  threshold?: number,\n) => Promise<R>;\n\nexport interface EvalMatchers<R = unknown> {\n  toEval: ToEval<R>;\n}\n\ndeclare module \"vitest\" {\n  interface Assertion<T = any> extends EvalMatchers<T> {}\n  interface AsymmetricMatchersContaining extends EvalMatchers {}\n}\n\nexpect.extend({\n  /**\n   * Evaluates a language model output against an expected answer using a scoring function.\n   *\n   * @param expected - The expected (ground truth) answer\n   * @param taskFn - Async function that processes the input and returns the model output\n   * @param scoreFn - Function that evaluates the model output against the expected answer\n   * @param threshold - Minimum acceptable score (0-1), defaults to 1.0\n   *\n   * @example\n   * ```javascript\n   * test(\"checks capital of France\", async () => {\n   *   expect(\"What is the capital of France?\").toEval(\n   *     \"Paris\",\n   *     async (input) => {\n   *       // Query LLM here\n   *       return \"Paris\";\n   *     },\n   *     checkFactuality,\n   *     0.8\n   *   );\n   * });\n   * ```\n   */\n  toEval: async function toEval(\n    input: string,\n    expected: string,\n    taskFn: TaskFn,\n    scoreFn: ScoreFn,\n    threshold = 1.0,\n  ) {\n    const { isNot } = this;\n\n    const output = await taskFn(input);\n\n    let result = scoreFn({ input, expected, output });\n    if (result instanceof Promise) {\n      result = await result;\n    }\n\n    return {\n      pass: (result.score ?? 0) >= threshold,\n      message: () =>\n        `Score: ${result.score} ${isNot ? \"<\" : \">\"} ${threshold}\\n${\n          result.metadata ? `Rationale: ${result.metadata.rationale}` : \"\"\n        }`,\n    };\n  },\n});\n\n/**\n * Creates a test suite for evaluating language model outputs.\n *\n * @param name - The name of the test suite\n * @param options - Configuration options\n * @param options.data - Async function that returns an array of test cases with input and expected values\n * @param options.task - Function that processes the input and returns the model output\n * @param options.skipIf - Optional function that determines if tests should be skipped\n * @param options.scorers - Array of scoring functions that evaluate model outputs\n * @param options.threshold - Minimum acceptable average score (0-1), defaults to 1.0\n * @param options.timeout - Test timeout in milliseconds, defaults to 10000\n *\n * @example\n * ```javascript\n * describeEval(\"capital cities test\", {\n *   data: async () => [{\n *     input: \"What is the capital of France?\",\n *     expected: \"Paris\"\n *   }],\n *   task: async (input) => {\n *     // Query LLM here\n *     return \"Paris\";\n *   },\n *   scorers: [checkFactuality],\n *   threshold: 0.8\n * });\n * ```\n */\nexport function describeEval(\n  name: string,\n  {\n    data,\n    task,\n    skipIf,\n    scorers,\n    threshold = 1.0,\n    // increase default test timeout as 5s is usually not enough for\n    // a single factuality check\n    timeout = 10000,\n  }: {\n    data: () => Promise<{ input: string; expected: string }[]>;\n    task: TaskFn;\n    skipIf?: () => boolean;\n    scorers: ScoreFn[];\n    threshold?: number | null;\n    timeout?: number;\n  },\n) {\n  return describe(name, async () => {\n    const testFn = skipIf ? test.skipIf(skipIf()) : test;\n    // TODO: should data just be a generator?\n    for (const { input, expected } of await data()) {\n      testFn(\n        input,\n        {\n          timeout,\n        },\n        async () => {\n          const output = await task(input);\n\n          const scores = await Promise.all(\n            scorers.map((scorer) => {\n              const result = scorer({ input, expected, output });\n              if (result instanceof Promise) {\n                return result;\n              }\n              return new Promise<Score>((resolve) => resolve(result));\n            }),\n          );\n          const scoresWithName = scores.map((s, i) => ({\n            ...s,\n            name: scorers[i].name,\n          }));\n\n          const avgScore =\n            scores.reduce((acc, s) => acc + (s.score ?? 0), 0) / scores.length;\n          if (threshold) {\n            assert(\n              avgScore >= threshold,\n              `Score: ${avgScore} below threshold: ${threshold}\\nOutput: ${output}\\n${formatScores(\n                scoresWithName,\n              )}`,\n            );\n          }\n        },\n      );\n    }\n  });\n}\n\nexport function formatScores(scores: (Score & { name: string })[]) {\n  return scores\n    .sort((a, b) => (a.score ?? 0) - (b.score ?? 0))\n    .map((s) => {\n      const scoreLine = `${s.name || \"Unknown\"} [${(s.score ?? 0).toFixed(1)}]`;\n      if (\n        ((s.score ?? 0) < 1.0 && s.metadata?.rationale) ||\n        s.metadata?.output\n      ) {\n        return `${scoreLine}${\n          s.metadata.rationale ? `\\nRationale: ${s.metadata.rationale}` : \"\"\n        }${s.metadata.output ? `\\nOutput: ${s.metadata.output}` : \"\"}`;\n      }\n      return scoreLine;\n    })\n    .join(\"\\n\\n\");\n}\n"],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,oBAA+C;AAC/C,IAAAA,iBAAO;AAkCP,qBAAO,OAAO;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,EAwBZ,QAAQ,SAAe,OACrB,OACA,UACA,QACA,SACA,YAAY,GACZ;AAAA;AAjEJ;AAkEI,YAAM,EAAE,MAAM,IAAI;AAElB,YAAM,SAAS,MAAM,OAAO,KAAK;AAEjC,UAAI,SAAS,QAAQ,EAAE,OAAO,UAAU,OAAO,CAAC;AAChD,UAAI,kBAAkB,SAAS;AAC7B,iBAAS,MAAM;AAAA,MACjB;AAEA,aAAO;AAAA,QACL,QAAO,YAAO,UAAP,YAAgB,MAAM;AAAA,QAC7B,SAAS,MACP,UAAU,OAAO,KAAK,IAAI,QAAQ,MAAM,GAAG,IAAI,SAAS;AAAA,EACtD,OAAO,WAAW,cAAc,OAAO,SAAS,SAAS,KAAK,EAChE;AAAA,MACJ;AAAA,IACF;AAAA;AACF,CAAC;AA8BM,SAAS,aACd,MACA;AAAA,EACE;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA,YAAY;AAAA;AAAA;AAAA,EAGZ,UAAU;AACZ,GAQA;AACA,aAAO,wBAAS,MAAM,MAAY;AAChC,UAAM,SAAS,SAAS,mBAAK,OAAO,OAAO,CAAC,IAAI;AAEhD,eAAW,EAAE,OAAO,SAAS,KAAK,MAAM,KAAK,GAAG;AAC9C;AAAA,QACE;AAAA,QACA;AAAA,UACE;AAAA,QACF;AAAA,QACA,MAAY;AACV,gBAAM,SAAS,MAAM,KAAK,KAAK;AAE/B,gBAAM,SAAS,MAAM,QAAQ;AAAA,YAC3B,QAAQ,IAAI,CAAC,WAAW;AACtB,oBAAM,SAAS,OAAO,EAAE,OAAO,UAAU,OAAO,CAAC;AACjD,kBAAI,kBAAkB,SAAS;AAC7B,uBAAO;AAAA,cACT;AACA,qBAAO,IAAI,QAAe,CAAC,YAAY,QAAQ,MAAM,CAAC;AAAA,YACxD,CAAC;AAAA,UACH;AACA,gBAAM,iBAAiB,OAAO,IAAI,CAAC,GAAG,MAAO,iCACxC,IADwC;AAAA,YAE3C,MAAM,QAAQ,CAAC,EAAE;AAAA,UACnB,EAAE;AAEF,gBAAM,WACJ,OAAO,OAAO,CAAC,KAAK,MAAG;AAhKnC;AAgKsC,2BAAO,OAAE,UAAF,YAAW;AAAA,aAAI,CAAC,IAAI,OAAO;AAC9D,cAAI,WAAW;AACb;AAAA,cACE,YAAY;AAAA,cACZ,UAAU,QAAQ,qBAAqB,SAAS;AAAA,UAAa,MAAM;AAAA,EAAK;AAAA,gBACtE;AAAA,cACF,CAAC;AAAA,YACH;AAAA,UACF;AAAA,QACF;AAAA,MACF;AAAA,IACF;AAAA,EACF,EAAC;AACH;AAEO,SAAS,aAAa,QAAsC;AACjE,SAAO,OACJ,KAAK,CAAC,GAAG,MAAG;AAjLjB;AAiLqB,oBAAE,UAAF,YAAW,OAAM,OAAE,UAAF,YAAW;AAAA,GAAE,EAC9C,IAAI,CAAC,MAAM;AAlLhB;AAmLM,UAAM,YAAY,GAAG,EAAE,QAAQ,SAAS,OAAM,OAAE,UAAF,YAAW,GAAG,QAAQ,CAAC,CAAC;AACtE,UACI,OAAE,UAAF,YAAW,KAAK,OAAO,OAAE,aAAF,mBAAY,gBACrC,OAAE,aAAF,mBAAY,SACZ;AACA,aAAO,GAAG,SAAS,GACjB,EAAE,SAAS,YAAY;AAAA,aAAgB,EAAE,SAAS,SAAS,KAAK,EAClE,GAAG,EAAE,SAAS,SAAS;AAAA,UAAa,EAAE,SAAS,MAAM,KAAK,EAAE;AAAA,IAC9D;AACA,WAAO;AAAA,EACT,CAAC,EACA,KAAK,MAAM;AAChB;","names":["import_vitest"]}

package/dist/index.mjs

@@ -1,3 +1,22 @@
+var __defProp = Object.defineProperty;
+var __defProps = Object.defineProperties;
+var __getOwnPropDescs = Object.getOwnPropertyDescriptors;
+var __getOwnPropSymbols = Object.getOwnPropertySymbols;
+var __hasOwnProp = Object.prototype.hasOwnProperty;
+var __propIsEnum = Object.prototype.propertyIsEnumerable;
+var __defNormalProp = (obj, key, value) => key in obj ? __defProp(obj, key, { enumerable: true, configurable: true, writable: true, value }) : obj[key] = value;
+var __spreadValues = (a, b) => {
+  for (var prop in b || (b = {}))
+    if (__hasOwnProp.call(b, prop))
+      __defNormalProp(a, prop, b[prop]);
+  if (__getOwnPropSymbols)
+    for (var prop of __getOwnPropSymbols(b)) {
+      if (__propIsEnum.call(b, prop))
+        __defNormalProp(a, prop, b[prop]);
+    }
+  return a;
+};
+var __spreadProps = (a, b) => __defProps(a, __getOwnPropDescs(b));
 var __async = (__this, __arguments, generator) => {
   return new Promise((resolve, reject) => {
     var fulfilled = (value) => {
@@ -20,15 +39,41 @@ var __async = (__this, __arguments, generator) => {
 };
 
 // src/index.ts
-import { assert, describe, expect, it } from "vitest";
+import { assert, describe, expect, test } from "vitest";
 import "vitest";
 expect.extend({
+  /**
+   * Evaluates a language model output against an expected answer using a scoring function.
+   *
+   * @param expected - The expected (ground truth) answer
+   * @param taskFn - Async function that processes the input and returns the model output
+   * @param scoreFn - Function that evaluates the model output against the expected answer
+   * @param threshold - Minimum acceptable score (0-1), defaults to 1.0
+   *
+   * @example
+   * ```javascript
+   * test("checks capital of France", async () => {
+   *   expect("What is the capital of France?").toEval(
+   *     "Paris",
+   *     async (input) => {
+   *       // Query LLM here
+   *       return "Paris";
+   *     },
+   *     checkFactuality,
+   *     0.8
+   *   );
+   * });
+   * ```
+   */
   toEval: function toEval(input, expected, taskFn, scoreFn, threshold = 1) {
     return __async(this, null, function* () {
       var _a;
       const { isNot } = this;
       const output = yield taskFn(input);
-      const result = yield scoreFn({ input, expected, output });
+      let result = scoreFn({ input, expected, output });
+      if (result instanceof Promise) {
+        result = yield result;
+      }
       return {
         pass: ((_a = result.score) != null ? _a : 0) >= threshold,
         message: () => `Score: ${result.score} ${isNot ? "<" : ">"} ${threshold}
@@ -40,6 +85,7 @@ ${result.metadata ? `Rationale: ${result.metadata.rationale}` : ""}`
 function describeEval(name, {
   data,
   task,
+  skipIf,
   scorers,
   threshold = 1,
   // increase default test timeout as 5s is usually not enough for
@@ -47,14 +93,27 @@ function describeEval(name, {
   timeout = 1e4
 }) {
   return describe(name, () => __async(this, null, function* () {
+    const testFn = skipIf ? test.skipIf(skipIf()) : test;
     for (const { input, expected } of yield data()) {
-      it(
+      testFn(
         input,
+        {
+          timeout
+        },
         () => __async(this, null, function* () {
           const output = yield task(input);
           const scores = yield Promise.all(
-            scorers.map((scorer) => scorer({ input, expected, output }))
+            scorers.map((scorer) => {
+              const result = scorer({ input, expected, output });
+              if (result instanceof Promise) {
+                return result;
+              }
+              return new Promise((resolve) => resolve(result));
+            })
           );
+          const scoresWithName = scores.map((s, i) => __spreadProps(__spreadValues({}, s), {
+            name: scorers[i].name
+          }));
           const avgScore = scores.reduce((acc, s) => {
             var _a;
             return acc + ((_a = s.score) != null ? _a : 0);
@@ -63,18 +122,34 @@ function describeEval(name, {
             assert(
               avgScore >= threshold,
               `Score: ${avgScore} below threshold: ${threshold}
-Output: ${output}`
+Output: ${output}
+${formatScores(
+                scoresWithName
+              )}`
             );
           }
-        }),
-        {
-          timeout
-        }
+        })
       );
     }
   }));
 }
+function formatScores(scores) {
+  return scores.sort((a, b) => {
+    var _a, _b;
+    return ((_a = a.score) != null ? _a : 0) - ((_b = b.score) != null ? _b : 0);
+  }).map((s) => {
+    var _a, _b, _c, _d;
+    const scoreLine = `${s.name || "Unknown"} [${((_a = s.score) != null ? _a : 0).toFixed(1)}]`;
+    if (((_b = s.score) != null ? _b : 0) < 1 && ((_c = s.metadata) == null ? void 0 : _c.rationale) || ((_d = s.metadata) == null ? void 0 : _d.output)) {
+      return `${scoreLine}${s.metadata.rationale ? `
+Rationale: ${s.metadata.rationale}` : ""}${s.metadata.output ? `
+Output: ${s.metadata.output}` : ""}`;
+    }
+    return scoreLine;
+  }).join("\n\n");
+}
 export {
-  describeEval
+  describeEval,
+  formatScores
 };
 //# sourceMappingURL=index.mjs.map

package/dist/index.mjs.map

@@ -1 +1 @@
-{"version":3,"sources":["../src/index.ts"],"sourcesContent":["import { assert, describe, expect, it } from \"vitest\";\nimport \"vitest\";\n\nexport type TaskFn = (input: string) => Promise<string>;\n\nexport type Score = {\n  score: number | null;\n  metadata?: {\n    rationale: string;\n  };\n};\n\nexport type ScoreFn = (opts: {\n  input: string;\n  expected: string;\n  output: string;\n}) => Promise<Score>;\n\nexport type ToEval<R = unknown> = (\n  expected: string,\n  taskFn: TaskFn,\n  scoreFn: ScoreFn,\n  threshold?: number,\n) => Promise<R>;\n\nexport interface EvalMatchers<R = unknown> {\n  toEval: ToEval<R>;\n}\n\ndeclare module \"vitest\" {\n  interface Assertion<T = any> extends EvalMatchers<T> {}\n  interface AsymmetricMatchersContaining extends EvalMatchers {}\n}\n\nexpect.extend({\n  toEval: async function toEval(\n    input: string,\n    expected: string,\n    taskFn: TaskFn,\n    scoreFn: ScoreFn,\n    threshold = 1.0,\n  ) {\n    const { isNot } = this;\n\n    const output = await taskFn(input);\n\n    const result = await scoreFn({ input, expected, output });\n\n    return {\n      pass: (result.score ?? 0) >= threshold,\n      message: () =>\n        `Score: ${result.score} ${isNot ? \"<\" : \">\"} ${threshold}\\n${\n          result.metadata ? `Rationale: ${result.metadata.rationale}` : \"\"\n        }`,\n    };\n  },\n});\n\nexport function describeEval(\n  name: string,\n  {\n    data,\n    task,\n    scorers,\n    threshold = 1.0,\n    // increase default test timeout as 5s is usually not enough for\n    // a single factuality check\n    timeout = 10000,\n  }: {\n    data: () => Promise<{ input: string; expected: string }[]>;\n    task: TaskFn;\n    scorers: ScoreFn[];\n    threshold?: number | null;\n    timeout?: number;\n  },\n) {\n  return describe(name, async () => {\n    // TODO: should data just be a generator?\n    for (const { input, expected } of await data()) {\n      it(\n        input,\n        async () => {\n          const output = await task(input);\n\n          const scores = await Promise.all(\n            scorers.map((scorer) => scorer({ input, expected, output })),\n          );\n\n          const avgScore =\n            scores.reduce((acc, s) => acc + (s.score ?? 0), 0) / scores.length;\n          if (threshold) {\n            assert(\n              avgScore >= threshold,\n              `Score: ${avgScore} below threshold: ${threshold}\\nOutput: ${output}`,\n            );\n          }\n        },\n        {\n          timeout,\n        },\n      );\n    }\n  });\n}\n"],"mappings":";;;;;;;;;;;;;;;;;;;;;;AAAA,SAAS,QAAQ,UAAU,QAAQ,UAAU;AAC7C,OAAO;AAiCP,OAAO,OAAO;AAAA,EACZ,QAAQ,SAAe,OACrB,OACA,UACA,QACA,SACA,YAAY,GACZ;AAAA;AAzCJ;AA0CI,YAAM,EAAE,MAAM,IAAI;AAElB,YAAM,SAAS,MAAM,OAAO,KAAK;AAEjC,YAAM,SAAS,MAAM,QAAQ,EAAE,OAAO,UAAU,OAAO,CAAC;AAExD,aAAO;AAAA,QACL,QAAO,YAAO,UAAP,YAAgB,MAAM;AAAA,QAC7B,SAAS,MACP,UAAU,OAAO,KAAK,IAAI,QAAQ,MAAM,GAAG,IAAI,SAAS;AAAA,EACtD,OAAO,WAAW,cAAc,OAAO,SAAS,SAAS,KAAK,EAChE;AAAA,MACJ;AAAA,IACF;AAAA;AACF,CAAC;AAEM,SAAS,aACd,MACA;AAAA,EACE;AAAA,EACA;AAAA,EACA;AAAA,EACA,YAAY;AAAA;AAAA;AAAA,EAGZ,UAAU;AACZ,GAOA;AACA,SAAO,SAAS,MAAM,MAAY;AAEhC,eAAW,EAAE,OAAO,SAAS,KAAK,MAAM,KAAK,GAAG;AAC9C;AAAA,QACE;AAAA,QACA,MAAY;AACV,gBAAM,SAAS,MAAM,KAAK,KAAK;AAE/B,gBAAM,SAAS,MAAM,QAAQ;AAAA,YAC3B,QAAQ,IAAI,CAAC,WAAW,OAAO,EAAE,OAAO,UAAU,OAAO,CAAC,CAAC;AAAA,UAC7D;AAEA,gBAAM,WACJ,OAAO,OAAO,CAAC,KAAK,MAAG;AAzFnC;AAyFsC,2BAAO,OAAE,UAAF,YAAW;AAAA,aAAI,CAAC,IAAI,OAAO;AAC9D,cAAI,WAAW;AACb;AAAA,cACE,YAAY;AAAA,cACZ,UAAU,QAAQ,qBAAqB,SAAS;AAAA,UAAa,MAAM;AAAA,YACrE;AAAA,UACF;AAAA,QACF;AAAA,QACA;AAAA,UACE;AAAA,QACF;AAAA,MACF;AAAA,IACF;AAAA,EACF,EAAC;AACH;","names":[]}
+{"version":3,"sources":["../src/index.ts"],"sourcesContent":["import { assert, describe, expect, test } from \"vitest\";\nimport \"vitest\";\n\nexport type TaskFn = (input: string) => Promise<string>;\n\nexport type Score = {\n  score: number | null;\n  metadata?: {\n    rationale?: string;\n    output?: string;\n  };\n};\n\nexport type ScoreFn = (opts: {\n  input: string;\n  output: string;\n  expected?: string;\n}) => Promise<Score> | Score;\n\nexport type ToEval<R = unknown> = (\n  expected: string,\n  taskFn: TaskFn,\n  scoreFn: ScoreFn,\n  threshold?: number,\n) => Promise<R>;\n\nexport interface EvalMatchers<R = unknown> {\n  toEval: ToEval<R>;\n}\n\ndeclare module \"vitest\" {\n  interface Assertion<T = any> extends EvalMatchers<T> {}\n  interface AsymmetricMatchersContaining extends EvalMatchers {}\n}\n\nexpect.extend({\n  /**\n   * Evaluates a language model output against an expected answer using a scoring function.\n   *\n   * @param expected - The expected (ground truth) answer\n   * @param taskFn - Async function that processes the input and returns the model output\n   * @param scoreFn - Function that evaluates the model output against the expected answer\n   * @param threshold - Minimum acceptable score (0-1), defaults to 1.0\n   *\n   * @example\n   * ```javascript\n   * test(\"checks capital of France\", async () => {\n   *   expect(\"What is the capital of France?\").toEval(\n   *     \"Paris\",\n   *     async (input) => {\n   *       // Query LLM here\n   *       return \"Paris\";\n   *     },\n   *     checkFactuality,\n   *     0.8\n   *   );\n   * });\n   * ```\n   */\n  toEval: async function toEval(\n    input: string,\n    expected: string,\n    taskFn: TaskFn,\n    scoreFn: ScoreFn,\n    threshold = 1.0,\n  ) {\n    const { isNot } = this;\n\n    const output = await taskFn(input);\n\n    let result = scoreFn({ input, expected, output });\n    if (result instanceof Promise) {\n      result = await result;\n    }\n\n    return {\n      pass: (result.score ?? 0) >= threshold,\n      message: () =>\n        `Score: ${result.score} ${isNot ? \"<\" : \">\"} ${threshold}\\n${\n          result.metadata ? `Rationale: ${result.metadata.rationale}` : \"\"\n        }`,\n    };\n  },\n});\n\n/**\n * Creates a test suite for evaluating language model outputs.\n *\n * @param name - The name of the test suite\n * @param options - Configuration options\n * @param options.data - Async function that returns an array of test cases with input and expected values\n * @param options.task - Function that processes the input and returns the model output\n * @param options.skipIf - Optional function that determines if tests should be skipped\n * @param options.scorers - Array of scoring functions that evaluate model outputs\n * @param options.threshold - Minimum acceptable average score (0-1), defaults to 1.0\n * @param options.timeout - Test timeout in milliseconds, defaults to 10000\n *\n * @example\n * ```javascript\n * describeEval(\"capital cities test\", {\n *   data: async () => [{\n *     input: \"What is the capital of France?\",\n *     expected: \"Paris\"\n *   }],\n *   task: async (input) => {\n *     // Query LLM here\n *     return \"Paris\";\n *   },\n *   scorers: [checkFactuality],\n *   threshold: 0.8\n * });\n * ```\n */\nexport function describeEval(\n  name: string,\n  {\n    data,\n    task,\n    skipIf,\n    scorers,\n    threshold = 1.0,\n    // increase default test timeout as 5s is usually not enough for\n    // a single factuality check\n    timeout = 10000,\n  }: {\n    data: () => Promise<{ input: string; expected: string }[]>;\n    task: TaskFn;\n    skipIf?: () => boolean;\n    scorers: ScoreFn[];\n    threshold?: number | null;\n    timeout?: number;\n  },\n) {\n  return describe(name, async () => {\n    const testFn = skipIf ? test.skipIf(skipIf()) : test;\n    // TODO: should data just be a generator?\n    for (const { input, expected } of await data()) {\n      testFn(\n        input,\n        {\n          timeout,\n        },\n        async () => {\n          const output = await task(input);\n\n          const scores = await Promise.all(\n            scorers.map((scorer) => {\n              const result = scorer({ input, expected, output });\n              if (result instanceof Promise) {\n                return result;\n              }\n              return new Promise<Score>((resolve) => resolve(result));\n            }),\n          );\n          const scoresWithName = scores.map((s, i) => ({\n            ...s,\n            name: scorers[i].name,\n          }));\n\n          const avgScore =\n            scores.reduce((acc, s) => acc + (s.score ?? 0), 0) / scores.length;\n          if (threshold) {\n            assert(\n              avgScore >= threshold,\n              `Score: ${avgScore} below threshold: ${threshold}\\nOutput: ${output}\\n${formatScores(\n                scoresWithName,\n              )}`,\n            );\n          }\n        },\n      );\n    }\n  });\n}\n\nexport function formatScores(scores: (Score & { name: string })[]) {\n  return scores\n    .sort((a, b) => (a.score ?? 0) - (b.score ?? 0))\n    .map((s) => {\n      const scoreLine = `${s.name || \"Unknown\"} [${(s.score ?? 0).toFixed(1)}]`;\n      if (\n        ((s.score ?? 0) < 1.0 && s.metadata?.rationale) ||\n        s.metadata?.output\n      ) {\n        return `${scoreLine}${\n          s.metadata.rationale ? `\\nRationale: ${s.metadata.rationale}` : \"\"\n        }${s.metadata.output ? `\\nOutput: ${s.metadata.output}` : \"\"}`;\n      }\n      return scoreLine;\n    })\n    .join(\"\\n\\n\");\n}\n"],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAAA,SAAS,QAAQ,UAAU,QAAQ,YAAY;AAC/C,OAAO;AAkCP,OAAO,OAAO;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA;AAAA,EAwBZ,QAAQ,SAAe,OACrB,OACA,UACA,QACA,SACA,YAAY,GACZ;AAAA;AAjEJ;AAkEI,YAAM,EAAE,MAAM,IAAI;AAElB,YAAM,SAAS,MAAM,OAAO,KAAK;AAEjC,UAAI,SAAS,QAAQ,EAAE,OAAO,UAAU,OAAO,CAAC;AAChD,UAAI,kBAAkB,SAAS;AAC7B,iBAAS,MAAM;AAAA,MACjB;AAEA,aAAO;AAAA,QACL,QAAO,YAAO,UAAP,YAAgB,MAAM;AAAA,QAC7B,SAAS,MACP,UAAU,OAAO,KAAK,IAAI,QAAQ,MAAM,GAAG,IAAI,SAAS;AAAA,EACtD,OAAO,WAAW,cAAc,OAAO,SAAS,SAAS,KAAK,EAChE;AAAA,MACJ;AAAA,IACF;AAAA;AACF,CAAC;AA8BM,SAAS,aACd,MACA;AAAA,EACE;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA,YAAY;AAAA;AAAA;AAAA,EAGZ,UAAU;AACZ,GAQA;AACA,SAAO,SAAS,MAAM,MAAY;AAChC,UAAM,SAAS,SAAS,KAAK,OAAO,OAAO,CAAC,IAAI;AAEhD,eAAW,EAAE,OAAO,SAAS,KAAK,MAAM,KAAK,GAAG;AAC9C;AAAA,QACE;AAAA,QACA;AAAA,UACE;AAAA,QACF;AAAA,QACA,MAAY;AACV,gBAAM,SAAS,MAAM,KAAK,KAAK;AAE/B,gBAAM,SAAS,MAAM,QAAQ;AAAA,YAC3B,QAAQ,IAAI,CAAC,WAAW;AACtB,oBAAM,SAAS,OAAO,EAAE,OAAO,UAAU,OAAO,CAAC;AACjD,kBAAI,kBAAkB,SAAS;AAC7B,uBAAO;AAAA,cACT;AACA,qBAAO,IAAI,QAAe,CAAC,YAAY,QAAQ,MAAM,CAAC;AAAA,YACxD,CAAC;AAAA,UACH;AACA,gBAAM,iBAAiB,OAAO,IAAI,CAAC,GAAG,MAAO,iCACxC,IADwC;AAAA,YAE3C,MAAM,QAAQ,CAAC,EAAE;AAAA,UACnB,EAAE;AAEF,gBAAM,WACJ,OAAO,OAAO,CAAC,KAAK,MAAG;AAhKnC;AAgKsC,2BAAO,OAAE,UAAF,YAAW;AAAA,aAAI,CAAC,IAAI,OAAO;AAC9D,cAAI,WAAW;AACb;AAAA,cACE,YAAY;AAAA,cACZ,UAAU,QAAQ,qBAAqB,SAAS;AAAA,UAAa,MAAM;AAAA,EAAK;AAAA,gBACtE;AAAA,cACF,CAAC;AAAA,YACH;AAAA,UACF;AAAA,QACF;AAAA,MACF;AAAA,IACF;AAAA,EACF,EAAC;AACH;AAEO,SAAS,aAAa,QAAsC;AACjE,SAAO,OACJ,KAAK,CAAC,GAAG,MAAG;AAjLjB;AAiLqB,oBAAE,UAAF,YAAW,OAAM,OAAE,UAAF,YAAW;AAAA,GAAE,EAC9C,IAAI,CAAC,MAAM;AAlLhB;AAmLM,UAAM,YAAY,GAAG,EAAE,QAAQ,SAAS,OAAM,OAAE,UAAF,YAAW,GAAG,QAAQ,CAAC,CAAC;AACtE,UACI,OAAE,UAAF,YAAW,KAAK,OAAO,OAAE,aAAF,mBAAY,gBACrC,OAAE,aAAF,mBAAY,SACZ;AACA,aAAO,GAAG,SAAS,GACjB,EAAE,SAAS,YAAY;AAAA,aAAgB,EAAE,SAAS,SAAS,KAAK,EAClE,GAAG,EAAE,SAAS,SAAS;AAAA,UAAa,EAAE,SAAS,MAAM,KAAK,EAAE;AAAA,IAC9D;AACA,WAAO;AAAA,EACT,CAAC,EACA,KAAK,MAAM;AAChB;","names":[]}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "vitest-evals",
-  "version": "0.1.1",
+  "version": "0.1.3",
   "sideEffects": false,
   "types": "./dist/index.d.ts",
   "main": "./dist/index.js",
@@ -37,6 +37,7 @@
   },
   "devDependencies": {
     "@biomejs/biome": "^1.9.4",
+    "autoevals": "^0.0.127",
     "lint-staged": "^15.5.0",
     "simple-git-hooks": "^2.12.1",
     "tsup": "^8.4.0",