promptfoo 0.1.0

package/LICENSE

@@ -0,0 +1,19 @@
+Copyright (c) Ian Webster 2023
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,353 @@
+# promptfoo
+
+`promptfoo` is a library and command-line tool that helps you evaluate LLM prompt & model quality with a systematic approach to comparing model outputs.
+
+With promptfoo, you can:
+
+- **Test multiple prompts** against predefined test cases
+- **Evaluate quality and catch regressions** by comparing LLM outputs side-by-side
+- **Speed up evaluations** by running tests concurrently
+- Use as a command line tool, or integrate into your workflow as a library
+- Use OpenAI API models (built-in support), or integrate custom API providers for any LLM API
+
+## Usage (command line)
+
+To evaluate prompts using `promptfoo`, use the following command:
+
+```bash
+npx promptfoo eval -p <prompt_paths...> -o <output_path> -r <provider> [-v <vars_path>] [-j <max_concurrency] [-c <config_path>]
+```
+
+- `<prompt_paths...>`: Paths to prompt file(s)
+- `<output_path>`: Path to output CSV, JSON, YAML, or HTML file. Defaults to terminal output
+- `<provider>`: One or more of: `openai:<model_name>`, or filesystem path to custom API caller module
+- `<vars_path>` (optional): Path to CSV, JSON, or YAML file with prompt variables
+- `<max_concurrency>` (optional): Number of simultaneous API requests. Defaults to 3
+- `<config_path>` (optional): Path to configuration file
+
+### Examples
+
+#### Prompt quality
+
+In this example, we evaluate whether adding adjectives to the personality of an assistant bot affects the responses:
+
+```bash
+npx promptfoo eval -p prompts.txt -v vars.csv -r openai:gpt-3.5-turbo
+```
+
+![Peek 2023-05-01 13-53](https://user-images.githubusercontent.com/310310/235529431-f4d5c395-d569-448e-9697-cd637e0372a5.gif)
+
+<!--
+<img width="1362" alt="Side-by-side evaluation of LLM prompt quality, terminal output" src="https://user-images.githubusercontent.com/310310/235329207-e8c22459-5f51-4fee-9714-1b602ac3d7ca.png">
+
+![Side-by-side evaluation of LLM prompt quality, html output](https://user-images.githubusercontent.com/310310/235483444-4ddb832d-e103-4b9c-a862-b0d6cc11cdc0.png)
+-->
+
+This command will evaluate the prompts in `prompts.txt`, substituing the variable values from `vars.csv`, and output results in your terminal.
+
+Have a look at the setup and full output [here](https://github.com/typpo/promptfoo/tree/main/examples/assistant-cli).
+
+You can run the command without an `-o` option to output in your terminal ([example](https://user-images.githubusercontent.com/310310/235329207-e8c22459-5f51-4fee-9714-1b602ac3d7ca.png)), or use `-o` to specify an HTML ([example](https://user-images.githubusercontent.com/310310/235483444-4ddb832d-e103-4b9c-a862-b0d6cc11cdc0.png)), CSV ([example](https://docs.google.com/spreadsheets/d/1nanoj3_TniWrDl1Sj-qYqIMD6jwm5FBy15xPFdUTsmI/edit?usp=sharing)), JSON ([example](https://github.com/typpo/promptfoo/blob/main/examples/simple-cli/output.json)), or YAML output.
+
+#### Model quality
+
+In this example, we evaluate the difference between GPT 3 and GPT 4 outputs for a given prompt:
+
+```bash
+npx promptfoo eval -p prompts.txt -r openai:gpt-3.5-turbo openai:gpt-4 -o output.html
+```
+
+Produces this HTML table:
+
+![Side-by-side evaluation of LLM model quality, gpt3 vs gpt4, html output](https://user-images.githubusercontent.com/310310/235490527-e0c31f40-00a0-493a-8afc-8ed6322bb5ca.png)
+
+Full setup and output [here](https://github.com/typpo/promptfoo/tree/main/examples/gpt-3.5-vs-4).
+
+## Usage (as a library)
+
+You can also use `promptfoo` as a library in your project by importing the `evaluate` function. The function takes the following parameters:
+
+- `providers`: a list of provider strings or `ApiProvider` objects, or just a single string or `ApiProvider`.
+- `options`: the prompts and variables you want to test:
+
+  ```typescript
+  {
+    prompts: string[];
+    vars?: Record<string, string>;
+  }
+  ```
+
+### Example
+
+`promptfoo` exports an `evaluate` function that you can use to run prompt evaluations.
+
+```javascript
+import promptfoo from 'promptfoo';
+
+const options = {
+  prompts: ['Rephrase this in French: {{body}}', 'Rephrase this like a pirate: {{body}}'],
+  vars: [{ body: 'Hello world' }, { body: "I'm hungry" }],
+};
+
+(async () => {
+  const summary = await promptfoo.evaluate('openai:gpt-3.5-turbo', options);
+  console.log(summary);
+})();
+```
+
+This code imports the `promptfoo` library, defines the evaluation options, and then calls the `evaluate` function with these options. The results are logged to the console:
+
+```js
+{
+  "results": [
+    {
+      "prompt": {
+        "raw": "Rephrase this in French: Hello world",
+        "display": "Rephrase this in French: {{body}}"
+      },
+      "vars": {
+        "body": "Hello world"
+      },
+      "response": {
+        "output": "Bonjour le monde",
+        "tokenUsage": {
+          "total": 19,
+          "prompt": 16,
+          "completion": 3
+        }
+      }
+    },
+    // ...
+  ],
+  "stats": {
+    "successes": 4,
+    "failures": 0,
+    "tokenUsage": {
+      "total": 120,
+      "prompt": 72,
+      "completion": 48
+    }
+  },
+  "table": [
+    // ...
+  ]
+}
+```
+
+[See full example here](https://github.com/typpo/promptfoo/tree/main/examples/simple-import)
+
+## Configuration
+
+### Prompt Files
+
+Prompt files are plain text files that contain the prompts you want to test. If you have only one file, you can include multiple prompts in the file, separated by the delimiter `---`. If you have multiple files, each prompt should be in a separate file.
+
+You can use [Nunjucks](https://mozilla.github.io/nunjucks/) templating syntax to include variables in your prompts, which will be replaced with actual values from the `vars` CSV file during evaluation.
+
+Example of a single prompt file with multiple prompts (`prompts.txt`):
+
+```
+Translate the following text to French: "{{text}}"
+---
+Translate the following text to German: "{{text}}"
+```
+
+Example of multiple prompt files:
+
+- `prompt1.txt`:
+
+  ```
+  Translate the following text to French: "{{text}}"
+  ```
+
+- `prompt2.txt`:
+
+  ```
+  Translate the following text to German: "{{text}}"
+  ```
+
+### Vars File
+
+The Vars file is a CSV, JSON, or YAML file that contains the values for the variables used in the prompts. The first row of the CSV file should contain the variable names, and each subsequent row should contain the corresponding values for each test case.
+
+Vars are substituted by [Nunjucks](https://mozilla.github.io/nunjucks/) templating syntax into prompts.
+
+Example of a vars file (`vars.csv`):
+
+```
+text
+"Hello, world!"
+"Goodbye, everyone!"
+```
+
+Example of a vars file (`vars.json`):
+
+```json
+[{ "text": "Hello, world!" }, { "text": "Goodbye, everyone!" }]
+```
+
+### Expected Value
+
+You can specify an expected value for each test case to evaluate the success or failure of the model's output. To do this, add a special field called `__expected` in the `vars` file. The `__expected` field supports three types of value comparisons:
+
+1. If the expected value starts with `eval:`, it will evaluate the contents as the body of a JavaScript function defined like: `function(output) { <eval> }`. The function should return a boolean value, where `true` indicates success and `false` indicates failure.
+
+2. If the expected value starts with `grade:`, it will call the `gradeOutput(prompt, output)` function. You should assume this function exists and returns a boolean value, where `true` indicates success and `false` indicates failure.
+
+3. Otherwise, it attempts an exact string match comparison between the expected value and the model's output.
+
+Example of a vars file with the `__expected` field (`vars.csv`):
+
+```
+text,__expected
+"Hello, world!","Bonjour le monde"
+"Goodbye, everyone!","eval:return output.includes('Au revoir');"
+```
+
+Example of a vars file with the `__expected` field (`vars.json`):
+
+```json
+[
+  { "text": "Hello, world!", "__expected": "Bonjour le monde" },
+  { "text": "Goodbye, everyone!", "__expected": "eval:output.includes('Au revoir');" }
+]
+```
+
+When the `__expected` field is provided, the success and failure statistics in the evaluation summary will be based on whether the expected criteria are met.
+
+For more advanced test cases, we recommend using a testing framework like [Jest](https://jestjs.io/) or [Mocha](https://mochajs.org/) and using promptfoo as a library.
+
+### Output File
+
+The results of the evaluation are written to this file. Each record in the output file corresponds to a test case and includes the original prompt, the output generated by the LLM, and the values of the variables used in the test case.
+
+For example outputs, see the [examples/](https://github.com/typpo/promptfoo/tree/main/examples) directory.
+
+### Configuration File
+
+You can specify any option in a configuration file (e.g., `.promptfoorc`, `promptfoo.config.json`). This can help you avoid repetitive command-line options and simplify the CLI invocation.
+
+Example of a configuration file (`promptfoo.config.json`):
+
+```json
+{
+  "provider": "openai:chat",
+  "vars": "/path/to/vars.csv"
+}
+```
+
+## Installation
+
+1. Clone the repository:
+
+```bash
+git clone https://github.com/typpo/promptfoo.git
+```
+
+2. Install the dependencies:
+
+```bash
+npm install
+```
+
+3. Link the CLI tool:
+
+```bash
+npm link
+```
+
+### Example
+
+```bash
+promptfoo eval -p prompt1.txt prompt2.txt -o results.csv -r openai:chat -v vars.csv
+```
+
+## API Providers
+
+`promptfoo` supports OpenAI API models out of the box. To use a custom API provider, create a custom module that implements the `ApiProvider` interface and pass the path to the module as the `provider` option.
+
+### OpenAI API
+
+To use the OpenAI API, set the `OPENAI_API_KEY` environment variable or pass the API key as an argument to the constructor.
+
+Example:
+
+```bash
+export OPENAI_API_KEY=your_api_key_here
+```
+
+Other OpenAI-related environment variables are supported:
+
+- `OPENAI_TEMPERATURE` - temperature model parameter, defaults to 0
+- `OPENAI_MAX_TOKENS` - max_tokens model parameter, defaults to 1024
+
+The OpenAI provider supports the following model formats:
+
+- `openai:chat` - defaults to gpt-3.5-turbo
+- `openai:completion` - defaults to `text-davinci-003`
+- `openai:<model name>` - uses a specific model name (mapped automatically to chat or completion endpoint)
+- `openai:chat:<model name>` - uses any model name against the chat endpoint
+- `openai:completion:<model name>` - uses any model name against the completion endpoint
+
+The `openai:<endpoint>:<model>` construction is useful if OpenAI releases a new model, or if you have a custom model. For example, if OpenAI releases gpt-5 chat completion, you could begin using it immediately with `openai:chat:gpt-5`.
+
+### Custom API Provider
+
+To create a custom API provider, implement the `ApiProvider` interface in a separate module. Here is the interface:
+
+```javascript
+export interface ApiProvider {
+  id: () => string;
+  callApi: (prompt: string) => Promise<ProviderResult>;
+}
+```
+
+Below is an example of a custom API provider that returns a predefined output and token usage:
+
+```javascript
+// customApiProvider.js
+class CustomApiProvider {
+  id() {
+    return 'my-custom-api';
+  }
+
+  async callApi(prompt) {
+    // Add your custom API logic here
+
+    return {
+      // Required
+      output: 'Model output',
+
+      // Optional
+      tokenUsage: {
+        total: 10,
+        prompt: 5,
+        completion: 5,
+      },
+    };
+  }
+}
+
+module.exports.default = CustomApiProvider;
+```
+
+To use the custom API provider with `promptfoo`, pass the path to the module as the `provider` option in the CLI invocation:
+
+```bash
+promptfoo eval -p prompt1.txt prompt2.txt -o results.csv  -v vars.csv -r ./customApiProvider.js
+```
+
+This command will evaluate the prompts using the custom API provider and save the results to the specified CSV file.
+
+## Development
+
+Contributions are welcome! Please feel free to submit a pull request or open an issue.
+
+`promptfoo` includes several npm scripts to make development easier and more efficient. To use these scripts, run `npm run <script_name>` in the project directory.
+
+Here are some of the available scripts:
+
+- `build`: Transpile TypeScript files to JavaScript
+- `watch`: Continuously watch and transpile TypeScript files on changes
+- `test`: Run test suite
+- `test:watch`: Continuously run test suite on changes

package/dist/__mocks__/esm.d.ts

@@ -0,0 +1,2 @@
+export declare function getDirectory(): string;
+//# sourceMappingURL=esm.d.ts.map

package/dist/__mocks__/esm.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"esm.d.ts","sourceRoot":"","sources":["../../src/__mocks__/esm.ts"],"names":[],"mappings":"AAAA,wBAAgB,YAAY,WAE3B"}

package/dist/__mocks__/esm.js

@@ -0,0 +1,4 @@
+export function getDirectory() {
+    return '/test/dir';
+}
+//# sourceMappingURL=esm.js.map

package/dist/__mocks__/esm.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"esm.js","sourceRoot":"","sources":["../../src/__mocks__/esm.ts"],"names":[],"mappings":"AAAA,MAAM,UAAU,YAAY;IAC1B,OAAO,WAAW,CAAC;AACrB,CAAC"}

package/dist/esm.d.ts

@@ -0,0 +1,2 @@
+export declare function getDirectory(): string;
+//# sourceMappingURL=esm.d.ts.map

package/dist/esm.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"esm.d.ts","sourceRoot":"","sources":["../src/esm.ts"],"names":[],"mappings":"AAKA,wBAAgB,YAAY,IAAI,MAAM,CAIrC"}

package/dist/esm.js

@@ -0,0 +1,9 @@
+// esm-specific crap that needs to get mocked out in tests
+import path from 'path';
+import { fileURLToPath } from 'url';
+export function getDirectory() {
+    // @ts-ignore: Jest chokes on this
+    const __filename = fileURLToPath(import.meta.url);
+    return path.dirname(__filename);
+}
+//# sourceMappingURL=esm.js.map

package/dist/esm.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"esm.js","sourceRoot":"","sources":["../src/esm.ts"],"names":[],"mappings":"AAAA,0DAA0D;AAE1D,OAAO,IAAI,MAAM,MAAM,CAAC;AACxB,OAAO,EAAE,aAAa,EAAE,MAAM,KAAK,CAAC;AAEpC,MAAM,UAAU,YAAY;IAC1B,kCAAkC;IAClC,MAAM,UAAU,GAAG,aAAa,CAAC,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;IAClD,OAAO,IAAI,CAAC,OAAO,CAAC,UAAU,CAAC,CAAC;AAClC,CAAC"}

package/dist/evaluator.d.ts

@@ -0,0 +1,3 @@
+import { EvaluateOptions, EvaluateSummary } from './types.js';
+export declare function evaluate(options: EvaluateOptions): Promise<EvaluateSummary>;
+//# sourceMappingURL=evaluator.d.ts.map

package/dist/evaluator.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"evaluator.d.ts","sourceRoot":"","sources":["../src/evaluator.ts"],"names":[],"mappings":"AAKA,OAAO,EAAE,eAAe,EAAE,eAAe,EAAuC,MAAM,YAAY,CAAC;AAiEnG,wBAAsB,QAAQ,CAAC,OAAO,EAAE,eAAe,GAAG,OAAO,CAAC,eAAe,CAAC,CAoIjF"}

package/dist/evaluator.js

@@ -0,0 +1,162 @@
+import async from 'async';
+import nunjucks from 'nunjucks';
+const DEFAULT_MAX_CONCURRENCY = 3;
+function checkExpectedValue(expected, output) {
+    if (expected.startsWith('eval:')) {
+        const evalBody = expected.slice(5);
+        const evalFunction = new Function('output', `return ${evalBody}`);
+        return evalFunction(output);
+    }
+    else if (expected.startsWith('grade:')) {
+        // NYI
+        return false;
+    }
+    else {
+        return expected === output;
+    }
+}
+async function runEval({ provider, prompt, vars, includeProviderId, }) {
+    vars = vars || {};
+    const renderedPrompt = nunjucks.renderString(prompt, vars);
+    // Note that we're using original prompt, not renderedPrompt
+    const promptDisplay = includeProviderId ? `[${provider.id()}] ${prompt}` : prompt;
+    const setup = {
+        prompt: {
+            raw: renderedPrompt,
+            display: promptDisplay,
+        },
+        vars,
+    };
+    try {
+        const response = await provider.callApi(renderedPrompt);
+        const success = vars.__expected ? checkExpectedValue(vars.__expected, response.output) : true;
+        const ret = {
+            ...setup,
+            response,
+            success,
+        };
+        if (!success) {
+            ret.error = `Expected ${vars.__expected}, got "${response.output}"`;
+        }
+        return ret;
+    }
+    catch (err) {
+        return {
+            ...setup,
+            error: String(err),
+            success: false,
+        };
+    }
+}
+export async function evaluate(options) {
+    const prompts = [];
+    const results = [];
+    for (const promptContent of options.prompts) {
+        for (const provider of options.providers) {
+            prompts.push({
+                raw: promptContent,
+                display: options.providers.length > 1 ? `[${provider.id()}] ${promptContent}` : promptContent,
+            });
+        }
+    }
+    const vars = options.vars && options.vars.length > 0 ? options.vars : [{}];
+    const varsWithExpectedKeyRemoved = vars.map((v) => {
+        const ret = { ...v };
+        delete ret.__expected;
+        return ret;
+    });
+    const isTest = vars[0].__expected;
+    const table = [
+        isTest
+            ? [
+                'RESULT',
+                [...prompts.map((p) => p.display), ...Object.keys(varsWithExpectedKeyRemoved[0])],
+            ].flat()
+            : [...prompts.map((p) => p.display), ...Object.keys(varsWithExpectedKeyRemoved[0])],
+    ];
+    const stats = {
+        successes: 0,
+        failures: 0,
+        tokenUsage: {
+            total: 0,
+            prompt: 0,
+            completion: 0,
+        },
+    };
+    let progressbar;
+    if (options.showProgressBar) {
+        const totalNumRuns = options.prompts.length * options.providers.length * (options.vars?.length || 1);
+        const cliProgress = await import('cli-progress');
+        progressbar = new cliProgress.SingleBar({
+            format: 'Eval: [{bar}] {percentage}% | ETA: {eta}s | {value}/{total} | {provider} "{prompt}" {vars}',
+        }, cliProgress.Presets.shades_classic);
+        progressbar.start(totalNumRuns, 0, {
+            provider: '',
+            prompt: '',
+            vars: '',
+        });
+    }
+    const runEvalOptions = [];
+    for (const row of vars) {
+        for (const promptContent of options.prompts) {
+            for (const provider of options.providers) {
+                runEvalOptions.push({
+                    provider,
+                    prompt: promptContent,
+                    vars: row,
+                    includeProviderId: options.providers.length > 1,
+                });
+            }
+        }
+    }
+    const combinedOutputs = new Array(vars.length).fill(null).map(() => []);
+    await async.forEachOfLimit(runEvalOptions, options.maxConcurrency || DEFAULT_MAX_CONCURRENCY, async (options, index) => {
+        const row = await runEval(options);
+        results.push(row);
+        if (row.error) {
+            stats.failures++;
+        }
+        else {
+            if (row.success) {
+                stats.successes++;
+            }
+            else {
+                stats.failures++;
+            }
+            stats.tokenUsage.total += row.response?.tokenUsage?.total || 0;
+            stats.tokenUsage.prompt += row.response?.tokenUsage?.prompt || 0;
+            stats.tokenUsage.completion += row.response?.tokenUsage?.completion || 0;
+        }
+        if (progressbar) {
+            progressbar.increment({
+                provider: options.provider.id(),
+                prompt: options.prompt.slice(0, 10),
+                vars: Object.entries(options.vars || {})
+                    .map(([k, v]) => `${k}=${v}`)
+                    .join(' ')
+                    .slice(0, 10),
+            });
+        }
+        // Bookkeeping for table
+        if (typeof index !== 'number') {
+            throw new Error('Expected index to be a number');
+        }
+        const combinedOutputIndex = Math.floor(index / prompts.length);
+        combinedOutputs[combinedOutputIndex].push(row.response?.output || '');
+    });
+    if (progressbar) {
+        progressbar.stop();
+    }
+    if (isTest) {
+        table.push(...combinedOutputs.map((output, index) => [
+            results[index].success ? 'PASS' : `FAIL: ${results[index].error}`,
+            ...output,
+            ...Object.values(varsWithExpectedKeyRemoved[index]),
+        ]));
+    }
+    else {
+        table.push(...combinedOutputs.map((output, index) => [...output, ...Object.values(vars[index])]));
+    }
+    return { results, stats, table };
+}
+//# sourceMappingURL=evaluator.js.map

package/dist/evaluator.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"evaluator.js","sourceRoot":"","sources":["../src/evaluator.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,MAAM,OAAO,CAAC;AAC1B,OAAO,QAAQ,MAAM,UAAU,CAAC;AAahC,MAAM,uBAAuB,GAAG,CAAC,CAAC;AAElC,SAAS,kBAAkB,CAAC,QAAgB,EAAE,MAAc;IAC1D,IAAI,QAAQ,CAAC,UAAU,CAAC,OAAO,CAAC,EAAE;QAChC,MAAM,QAAQ,GAAG,QAAQ,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;QACnC,MAAM,YAAY,GAAG,IAAI,QAAQ,CAAC,QAAQ,EAAE,UAAU,QAAQ,EAAE,CAAC,CAAC;QAClE,OAAO,YAAY,CAAC,MAAM,CAAC,CAAC;KAC7B;SAAM,IAAI,QAAQ,CAAC,UAAU,CAAC,QAAQ,CAAC,EAAE;QACxC,MAAM;QACN,OAAO,KAAK,CAAC;KACd;SAAM;QACL,OAAO,QAAQ,KAAK,MAAM,CAAC;KAC5B;AACH,CAAC;AAED,KAAK,UAAU,OAAO,CAAC,EACrB,QAAQ,EACR,MAAM,EACN,IAAI,EACJ,iBAAiB,GACF;IACf,IAAI,GAAG,IAAI,IAAI,EAAE,CAAC;IAClB,MAAM,cAAc,GAAG,QAAQ,CAAC,YAAY,CAAC,MAAM,EAAE,IAAI,CAAC,CAAC;IAE3D,4DAA4D;IAC5D,MAAM,aAAa,GAAG,iBAAiB,CAAC,CAAC,CAAC,IAAI,QAAQ,CAAC,EAAE,EAAE,KAAK,MAAM,EAAE,CAAC,CAAC,CAAC,MAAM,CAAC;IAElF,MAAM,KAAK,GAAG;QACZ,MAAM,EAAE;YACN,GAAG,EAAE,cAAc;YACnB,OAAO,EAAE,aAAa;SACvB;QACD,IAAI;KACL,CAAC;IAEF,IAAI;QACF,MAAM,QAAQ,GAAG,MAAM,QAAQ,CAAC,OAAO,CAAC,cAAc,CAAC,CAAC;QACxD,MAAM,OAAO,GAAG,IAAI,CAAC,UAAU,CAAC,CAAC,CAAC,kBAAkB,CAAC,IAAI,CAAC,UAAU,EAAE,QAAQ,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;QAC9F,MAAM,GAAG,GAAmB;YAC1B,GAAG,KAAK;YACR,QAAQ;YACR,OAAO;SACR,CAAC;QACF,IAAI,CAAC,OAAO,EAAE;YACZ,GAAG,CAAC,KAAK,GAAG,YAAY,IAAI,CAAC,UAAU,UAAU,QAAQ,CAAC,MAAM,GAAG,CAAC;SACrE;QACD,OAAO,GAAG,CAAC;KACZ;IAAC,OAAO,GAAG,EAAE;QACZ,OAAO;YACL,GAAG,KAAK;YACR,KAAK,EAAE,MAAM,CAAC,GAAG,CAAC;YAClB,OAAO,EAAE,KAAK;SACf,CAAC;KACH;AACH,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,QAAQ,CAAC,OAAwB;IACrD,MAAM,OAAO,GAAa,EAAE,CAAC;IAC7B,MAAM,OAAO,GAAqB,EAAE,CAAC;IAErC,KAAK,MAAM,aAAa,IAAI,OAAO,CAAC,OAAO,EAAE;QAC3C,KAAK,MAAM,QAAQ,IAAI,OAAO,CAAC,SAAS,EAAE;YACxC,OAAO,CAAC,IAAI,CAAC;gBACX,GAAG,EAAE,aAAa;gBAClB,OAAO,EACL,OAAO,CAAC,SAAS,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,QAAQ,CAAC,EAAE,EAAE,KAAK,aAAa,EAAE,CAAC,CAAC,CAAC,aAAa;aACvF,CAAC,CAAC;SACJ;KACF;IAED,MAAM,IAAI,GAAG,OAAO,CAAC,IAAI,IAAI,OAAO,CAAC,IAAI,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC;IAC3E,MAAM,0BAA0B,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE;QAChD,MAAM,GAAG,GAAG,EAAE,GAAG,CAAC,EAAE,CAAC;QACrB,OAAO,GAAG,CAAC,UAAU,CAAC;QACtB,OAAO,GAAG,CAAC;IACb,CAAC,CAAC,CAAC;IACH,MAAM,MAAM,GAAG,IAAI,CAAC,CAAC,CAAC,CAAC,UAAU,CAAC;IAClC,MAAM,KAAK,GAAe;QACxB,MAAM;YACJ,CAAC,CAAC;gBACE,QAAQ;gBACR,CAAC,GAAG,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,OAAO,CAAC,EAAE,GAAG,MAAM,CAAC,IAAI,CAAC,0BAA0B,CAAC,CAAC,CAAC,CAAC,CAAC;aAClF,CAAC,IAAI,EAAE;YACV,CAAC,CAAC,CAAC,GAAG,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,OAAO,CAAC,EAAE,GAAG,MAAM,CAAC,IAAI,CAAC,0BAA0B,CAAC,CAAC,CAAC,CAAC,CAAC;KACtF,CAAC;IAEF,MAAM,KAAK,GAAG;QACZ,SAAS,EAAE,CAAC;QACZ,QAAQ,EAAE,CAAC;QACX,UAAU,EAAE;YACV,KAAK,EAAE,CAAC;YACR,MAAM,EAAE,CAAC;YACT,UAAU,EAAE,CAAC;SACd;KACF,CAAC;IAEF,IAAI,WAAkC,CAAC;IACvC,IAAI,OAAO,CAAC,eAAe,EAAE;QAC3B,MAAM,YAAY,GAChB,OAAO,CAAC,OAAO,CAAC,MAAM,GAAG,OAAO,CAAC,SAAS,CAAC,MAAM,GAAG,CAAC,OAAO,CAAC,IAAI,EAAE,MAAM,IAAI,CAAC,CAAC,CAAC;QAClF,MAAM,WAAW,GAAG,MAAM,MAAM,CAAC,cAAc,CAAC,CAAC;QACjD,WAAW,GAAG,IAAI,WAAW,CAAC,SAAS,CACrC;YACE,MAAM,EACJ,4FAA4F;SAC/F,EACD,WAAW,CAAC,OAAO,CAAC,cAAc,CACnC,CAAC;QACF,WAAW,CAAC,KAAK,CAAC,YAAY,EAAE,CAAC,EAAE;YACjC,QAAQ,EAAE,EAAE;YACZ,MAAM,EAAE,EAAE;YACV,IAAI,EAAE,EAAE;SACT,CAAC,CAAC;KACJ;IAED,MAAM,cAAc,GAAqB,EAAE,CAAC;IAC5C,KAAK,MAAM,GAAG,IAAI,IAAI,EAAE;QACtB,KAAK,MAAM,aAAa,IAAI,OAAO,CAAC,OAAO,EAAE;YAC3C,KAAK,MAAM,QAAQ,IAAI,OAAO,CAAC,SAAS,EAAE;gBACxC,cAAc,CAAC,IAAI,CAAC;oBAClB,QAAQ;oBACR,MAAM,EAAE,aAAa;oBACrB,IAAI,EAAE,GAAG;oBACT,iBAAiB,EAAE,OAAO,CAAC,SAAS,CAAC,MAAM,GAAG,CAAC;iBAChD,CAAC,CAAC;aACJ;SACF;KACF;IAED,MAAM,eAAe,GAAe,IAAI,KAAK,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,GAAG,EAAE,CAAC,EAAE,CAAC,CAAC;IACpF,MAAM,KAAK,CAAC,cAAc,CACxB,cAAc,EACd,OAAO,CAAC,cAAc,IAAI,uBAAuB,EACjD,KAAK,EAAE,OAAuB,EAAE,KAAsB,EAAE,EAAE;QACxD,MAAM,GAAG,GAAG,MAAM,OAAO,CAAC,OAAO,CAAC,CAAC;QACnC,OAAO,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;QAClB,IAAI,GAAG,CAAC,KAAK,EAAE;YACb,KAAK,CAAC,QAAQ,EAAE,CAAC;SAClB;aAAM;YACL,IAAI,GAAG,CAAC,OAAO,EAAE;gBACf,KAAK,CAAC,SAAS,EAAE,CAAC;aACnB;iBAAM;gBACL,KAAK,CAAC,QAAQ,EAAE,CAAC;aAClB;YACD,KAAK,CAAC,UAAU,CAAC,KAAK,IAAI,GAAG,CAAC,QAAQ,EAAE,UAAU,EAAE,KAAK,IAAI,CAAC,CAAC;YAC/D,KAAK,CAAC,UAAU,CAAC,MAAM,IAAI,GAAG,CAAC,QAAQ,EAAE,UAAU,EAAE,MAAM,IAAI,CAAC,CAAC;YACjE,KAAK,CAAC,UAAU,CAAC,UAAU,IAAI,GAAG,CAAC,QAAQ,EAAE,UAAU,EAAE,UAAU,IAAI,CAAC,CAAC;SAC1E;QAED,IAAI,WAAW,EAAE;YACf,WAAW,CAAC,SAAS,CAAC;gBACpB,QAAQ,EAAE,OAAO,CAAC,QAAQ,CAAC,EAAE,EAAE;gBAC/B,MAAM,EAAE,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC;gBACnC,IAAI,EAAE,MAAM,CAAC,OAAO,CAAC,OAAO,CAAC,IAAI,IAAI,EAAE,CAAC;qBACrC,GAAG,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,EAAE,EAAE,CAAC,GAAG,CAAC,IAAI,CAAC,EAAE,CAAC;qBAC5B,IAAI,CAAC,GAAG,CAAC;qBACT,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC;aAChB,CAAC,CAAC;SACJ;QAED,wBAAwB;QACxB,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE;YAC7B,MAAM,IAAI,KAAK,CAAC,+BAA+B,CAAC,CAAC;SAClD;QACD,MAAM,mBAAmB,GAAG,IAAI,CAAC,KAAK,CAAC,KAAK,GAAG,OAAO,CAAC,MAAM,CAAC,CAAC;QAC/D,eAAe,CAAC,mBAAmB,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,QAAQ,EAAE,MAAM,IAAI,EAAE,CAAC,CAAC;IACxE,CAAC,CACF,CAAC;IAEF,IAAI,WAAW,EAAE;QACf,WAAW,CAAC,IAAI,EAAE,CAAC;KACpB;IAED,IAAI,MAAM,EAAE;QACV,KAAK,CAAC,IAAI,CACR,GAAG,eAAe,CAAC,GAAG,CAAC,CAAC,MAAM,EAAE,KAAK,EAAE,EAAE,CAAC;YACxC,OAAO,CAAC,KAAK,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,SAAS,OAAO,CAAC,KAAK,CAAC,CAAC,KAAK,EAAE;YACjE,GAAG,MAAM;YACT,GAAG,MAAM,CAAC,MAAM,CAAC,0BAA0B,CAAC,KAAK,CAAC,CAAC;SACpD,CAAC,CACH,CAAC;KACH;SAAM;QACL,KAAK,CAAC,IAAI,CACR,GAAG,eAAe,CAAC,GAAG,CAAC,CAAC,MAAM,EAAE,KAAK,EAAE,EAAE,CAAC,CAAC,GAAG,MAAM,EAAE,GAAG,MAAM,CAAC,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CACtF,CAAC;KACH;IAED,OAAO,EAAE,OAAO,EAAE,KAAK,EAAE,KAAK,EAAE,CAAC;AACnC,CAAC"}

package/dist/index.d.ts

@@ -0,0 +1,7 @@
+import type { ApiProvider, EvaluateOptions, EvaluateSummary } from './types.js';
+declare function evaluate(providers: (string | ApiProvider)[] | (string | ApiProvider), options: Omit<EvaluateOptions, 'providers'>): Promise<EvaluateSummary>;
+declare const _default: {
+    evaluate: typeof evaluate;
+};
+export default _default;
+//# sourceMappingURL=index.d.ts.map

package/dist/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAGA,OAAO,KAAK,EAAE,WAAW,EAAE,eAAe,EAAE,eAAe,EAAE,MAAM,YAAY,CAAC;AAEhF,iBAAe,QAAQ,CACrB,SAAS,EAAE,CAAC,MAAM,GAAG,WAAW,CAAC,EAAE,GAAG,CAAC,MAAM,GAAG,WAAW,CAAC,EAC5D,OAAO,EAAE,IAAI,CAAC,eAAe,EAAE,WAAW,CAAC,GAC1C,OAAO,CAAC,eAAe,CAAC,CAsB1B;;;;AAED,wBAEE"}

package/dist/index.js

@@ -0,0 +1,29 @@
+import { evaluate as doEvaluate } from './evaluator.js';
+import { loadApiProvider } from './providers.js';
+async function evaluate(providers, options) {
+    let apiProviders = [];
+    const addProvider = async (provider) => {
+        if (typeof provider === 'string') {
+            apiProviders.push(await loadApiProvider(provider));
+        }
+        else {
+            apiProviders.push(provider);
+        }
+    };
+    if (Array.isArray(providers)) {
+        for (const provider of providers) {
+            await addProvider(provider);
+        }
+    }
+    else {
+        await addProvider(providers);
+    }
+    return doEvaluate({
+        ...options,
+        providers: apiProviders,
+    });
+}
+export default {
+    evaluate,
+};
+//# sourceMappingURL=index.js.map

package/dist/index.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,IAAI,UAAU,EAAE,MAAM,gBAAgB,CAAC;AACxD,OAAO,EAAE,eAAe,EAAE,MAAM,gBAAgB,CAAC;AAIjD,KAAK,UAAU,QAAQ,CACrB,SAA4D,EAC5D,OAA2C;IAE3C,IAAI,YAAY,GAAkB,EAAE,CAAC;IACrC,MAAM,WAAW,GAAG,KAAK,EAAE,QAA8B,EAAE,EAAE;QAC3D,IAAI,OAAO,QAAQ,KAAK,QAAQ,EAAE;YAChC,YAAY,CAAC,IAAI,CAAC,MAAM,eAAe,CAAC,QAAQ,CAAC,CAAC,CAAC;SACpD;aAAM;YACL,YAAY,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC;SAC7B;IACH,CAAC,CAAC;IAEF,IAAI,KAAK,CAAC,OAAO,CAAC,SAAS,CAAC,EAAE;QAC5B,KAAK,MAAM,QAAQ,IAAI,SAAS,EAAE;YAChC,MAAM,WAAW,CAAC,QAAQ,CAAC,CAAC;SAC7B;KACF;SAAM;QACL,MAAM,WAAW,CAAC,SAAS,CAAC,CAAC;KAC9B;IAED,OAAO,UAAU,CAAC;QAChB,GAAG,OAAO;QACV,SAAS,EAAE,YAAY;KACxB,CAAC,CAAC;AACL,CAAC;AAED,eAAe;IACb,QAAQ;CACT,CAAC"}

package/dist/logger.d.ts

@@ -0,0 +1,11 @@
+import winston from 'winston';
+declare const logLevels: {
+    error: number;
+    warn: number;
+    info: number;
+    debug: number;
+};
+declare const logger: winston.Logger;
+export declare function setLogLevel(level: keyof typeof logLevels): void;
+export default logger;
+//# sourceMappingURL=logger.d.ts.map

package/dist/logger.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"logger.d.ts","sourceRoot":"","sources":["../src/logger.ts"],"names":[],"mappings":"AACA,OAAO,OAAO,MAAM,SAAS,CAAC;AAE9B,QAAA,MAAM,SAAS;;;;;CAKd,CAAC;AAeF,QAAA,MAAM,MAAM,gBAIV,CAAC;AAEH,wBAAgB,WAAW,CAAC,KAAK,EAAE,MAAM,OAAO,SAAS,QAMxD;AAED,eAAe,MAAM,CAAC"}