promptfoo 0.5.1 → 0.7.0

package/README.md

@@ -9,22 +9,35 @@ With promptfoo, you can:
 
 - **Test multiple prompts** against predefined test cases
 - **Evaluate quality and catch regressions** by comparing LLM outputs side-by-side
-- **Speed up evaluations** by running tests concurrently
+- **Speed up evaluations** with caching and concurrent tests
 - **Flag bad outputs automatically** by setting "expectations"
 - Use as a command line tool, or integrate into your workflow as a library
 - Use OpenAI models, open-source models like Llama and Vicuna, or integrate custom API providers for any LLM API
 
-**» [View docs on website](https://promptfoo.dev/docs/intro) «**
+The goal: **test-driven prompt engineering**, rather than trial-and-error.
 
-promptfoo produces matrix views that allow you to quickly review prompt outputs across many inputs. The goal: tune prompts systematically across all relevant test cases, instead of testing prompts by trial and error.
+# [» View full documentation «](https://promptfoo.dev/docs/intro)
+
+promptfoo produces matrix views that let you quickly evaluate outputs across many prompts.
 
 Here's an example of a side-by-side comparison of multiple prompts and inputs:
 
 ![Prompt evaluation matrix - web viewer](https://github.com/typpo/promptfoo/assets/310310/ddcd77df-2783-425e-ade9-1a20dd0b6cd2)
 
 It works on the command line too:
+
 ![Prompt evaluation](https://user-images.githubusercontent.com/310310/235529431-f4d5c395-d569-448e-9697-cd637e0372a5.gif)
 
+## Workflow
+
+Start by establishing a handful of test cases - core use cases and failure cases that you want to ensure your prompt can handle.
+
+As you explore modifications to the prompt, use `promptfoo eval` to rate all outputs.  This ensures the prompt is actually improving overall.
+
+As you collect more examples and establish a user feedback loop, continue to build the pool of test cases.
+
+<img width="772" alt="LLM ops" src="https://github.com/typpo/promptfoo/assets/310310/cf0461a7-2832-4362-9fbb-4ebd911d06ff">
+
 ## Usage (command line & web viewer)
 
 To get started, run the following command:
@@ -41,19 +54,20 @@ After editing the prompts and variables to your liking, run the eval command to
 npx promptfoo eval
 ```
 
-If you're looking to customize your usage, you have the full set of parameters at your disposal:
-
-```bash
-npx promptfoo eval -p <prompt_paths...> -o <output_path> -r <providers> [-v <vars_path>] [-j <max_concurrency] [-c <config_path>] [--grader <grading_provider>]
-```
+If you're looking to customize your usage, you have a wide set of parameters at your disposal. See the [Configuration docs](https://www.promptfoo.dev/docs/configuration/parameters) for more detail:
 
-- `<prompt_paths...>`: Paths to prompt file(s)
-- `<output_path>`: Path to output CSV, JSON, YAML, or HTML file. Defaults to terminal output
-- `<providers>`: One or more of: `openai:<model_name>`, or filesystem path to custom API caller module
-- `<vars_path>` (optional): Path to CSV, JSON, or YAML file with prompt variables
-- `<max_concurrency>` (optional): Number of simultaneous API requests. Defaults to 4
-- `<config_path>` (optional): Path to configuration file
-- `<grading_provider>`: A provider that handles the grading process, if you are using [LLM grading](#expected-outputs)
+| Option                              | Description                                                                                                                                                                                    |
+| ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `-p, --prompts <paths...>`          | Paths to prompt files, directory, or glob                                                                                                                                                      |
+| `-r, --providers <name or path...>` | One of: openai:chat, openai:completion, openai:model-name, localai:chat:model-name, localai:completion:model-name. See [API providers](https://www.promptfoo.dev/docs/configuration/providers) |
+| `-o, --output <path>`               | Path to output file (csv, json, yaml, html)                                                                                                                                                    |
+| `-v, --vars <path>`                 | Path to file with prompt variables (csv, json, yaml)                                                                                                                                           |
+| `-c, --config <path>`               | Path to configuration file. `promptfooconfig.js[on]` is automatically loaded if present                                                                                                        |
+| `-j, --max-concurrency <number>`    | Maximum number of concurrent API calls                                                                                                                                                         |
+| `--table-cell-max-length <number>`  | Truncate console table cells to this length                                                                                                                                                    |
+| `--prompt-prefix <path>`            | This prefix is prepended to every prompt                                                                                                                                                       |
+| `--prompt-suffix <path>`            | This suffix is append to every prompt                                                                                                                                                          |
+| `--grader`                          | Provider that will grade outputs, if you are using [LLM grading](https://www.promptfoo.dev/docs/configuration/expected-outputs)                                                                |
 
 After running an eval, you may optionally use the `view` command to open the web viewer:
 
@@ -174,247 +188,16 @@ This code imports the `promptfoo` library, defines the evaluation options, and t
 
 ## Configuration
 
-### Prompt Files
-
-Prompt files are plain text files that contain the prompts you want to test. If you have only one file, you can include multiple prompts in the file, separated by the delimiter `---`. If you have multiple files, each prompt should be in a separate file.
-
-You can use [Nunjucks](https://mozilla.github.io/nunjucks/) templating syntax to include variables in your prompts, which will be replaced with actual values from the `vars` CSV file during evaluation.
-
-Example of a single prompt file with multiple prompts (`prompts.txt`):
-
-```
-Translate the following text to French: "{{name}}: {{text}}"
----
-Translate the following text to German: "{{name}}: {{text}}"
-```
-
-Example of multiple prompt files:
-
-- `prompt1.txt`:
-
-  ```
-  Translate the following text to French: "{{name}}: {{text}}"
-  ```
-
-- `prompt2.txt`:
-
-  ```
-  Translate the following text to German: "{{name}}: {{text}}"
-  ```
-
-### Vars File
-
-The Vars file is a CSV, JSON, or YAML file that contains the values for the variables used in the prompts. The first row of the CSV file should contain the variable names, and each subsequent row should contain the corresponding values for each test case.
-
-Vars are substituted by [Nunjucks](https://mozilla.github.io/nunjucks/) templating syntax into prompts.
-
-Example of a vars file (`vars.csv`):
-
-```
-"name","text"
-"Bob","Hello, world!"
-"Joe","Goodbye, everyone!"
-```
-
-Example of a vars file (`vars.json`):
-
-```json
-[
-  { "name": "Bob", "text": "Hello, world!" },
-  { "name": "Joe", "text": "Goodbye, everyone!" }
-]
-```
-
-### Expected Outputs
-
-You can specify an expected value for each test case to evaluate the success or failure of the model's output. To do this, add a special field called `__expected` in the `vars` file. The `__expected` field supports these types of value comparisons:
-
-1. If the expected value starts with `eval:`, it will evaluate the contents as the body of a JavaScript function defined like: `function(output) { <eval> }`. The function should return a boolean value, where `true` indicates success and `false` indicates failure.
-
-1. If the expected value starts with `similar:`, it will compare the semantic similarity of the expected and output values. For example, `similar: greetings, world!` is semantically similar to "Hello world" even though it's not an exact match.
-
-   The `similar` directive uses cosine similarity, where 1.0 is the most similar and 0.0 is the least similar. Tune the similarity threshold by specifying `similar(0.8): ...` (passes only if similarity >= 0.8).
-
-   The embedding model currently supported is OpenAI's `text-embedding-ada-002`. As a result, the `similar` directive requires the OPENAI_API_KEY environment variable to be set.
-
-1. If the expected value starts with `grade:`, it will ask an LLM to evaluate whether the output meets the condition. For example, `grade: don't mention being an AI`. This option requires a provider name to be supplied to promptfoo via the `--grader` argument: `promptfoo --grader openai:gpt-4 ...`.
-
-1. Otherwise, it attempts an exact string match comparison between the expected value and the model's output.
-
-Example of a vars file with the `__expected` field (`vars.csv`):
-
-```
-text,__expected
-"Hello, world!","Bonjour le monde"
-"Goodbye, everyone!","eval:output.includes('Au revoir');"
-"I am a pineapple","grade:doesn't reference any fruits besides pineapple"
-```
-
-Example of a vars file with the `__expected` field (`vars.json`):
-
-```json
-[
-  { "text": "Hello, world!", "__expected": "Bonjour le monde" },
-  { "text": "Goodbye, everyone!", "__expected": "eval:output.includes('Au revoir');" }
-  { "text": "I am a pineapple", "__expected": "grade:doesn't reference any fruits besides pineapple" }
-]
-```
-
-When the `__expected` field is provided, the success and failure statistics in the evaluation summary will be based on whether the expected criteria are met.
-
-For more advanced test cases, we recommend using a testing framework like [Jest](https://jestjs.io/) or [Mocha](https://mochajs.org/) and using promptfoo as a library.
-
-### Output File
-
-The results of the evaluation are written to this file. Each record in the output file corresponds to a test case and includes the original prompt, the output generated by the LLM, and the values of the variables used in the test case.
-
-For example outputs, see the [examples/](https://github.com/typpo/promptfoo/tree/main/examples) directory.
-
-### Configuration File
-
-You can specify any option in a configuration file (e.g., `.promptfoorc`, `promptfoo.config.json`). This can help you avoid repetitive command-line options and simplify the CLI invocation.
-
-Example of a configuration file (`promptfoo.config.json`):
-
-```json
-{
-  "provider": "openai:chat",
-  "vars": "/path/to/vars.csv"
-}
-```
+- **[Setting up an eval](https://promptfoo.dev/docs/configuration/parameters)**: Learn more about how to set up prompt files, vars file, output, etc.
+- **[Configuring test cases](https://promptfoo.dev/docs/configuration/expected-outputs)**: Learn more about how to configure expected outputs and test assertions.
 
 ## Installation
 
-1. Clone the repository:
-
-```bash
-git clone https://github.com/typpo/promptfoo.git
-```
-
-2. Install the dependencies:
-
-```bash
-npm install
-```
-
-3. Link the CLI tool:
-
-```bash
-npm link
-```
-
-4. Build:
-
-```bash
-npm run build
-```
-
-5. Make the entrypoint executable:
-
-```bash
-chmod +x dist/main.js
-```
+See **[installation docs](https://promptfoo.dev/docs/installation)**
 
 ## API Providers
 
-`promptfoo` supports OpenAI API models out of the box. To use a custom API provider, create a custom module that implements the `ApiProvider` interface and pass the path to the module as the `provider` option.
-
-### OpenAI API
-
-To use the OpenAI API, set the `OPENAI_API_KEY` environment variable or pass the API key as an argument to the constructor.
-
-Example:
-
-```bash
-export OPENAI_API_KEY=your_api_key_here
-```
-
-Other OpenAI-related environment variables are supported:
-
-- `OPENAI_TEMPERATURE` - temperature model parameter, defaults to 0
-- `OPENAI_MAX_TOKENS` - max_tokens model parameter, defaults to 1024
-- `OPENAI_STOP` - stopwords in JSON format, defaults to []
-- `OPENAI_API_HOST` - override the hostname for the API request. Useful for proxies like Helicone.
-- `REQUEST_TIMEOUT_MS` - maximum request time, in milliseconds. Defaults to 60000.
-
-The OpenAI provider supports the following model formats:
-
-- `openai:chat` - defaults to gpt-3.5-turbo
-- `openai:completion` - defaults to `text-davinci-003`
-- `openai:<model name>` - uses a specific model name (mapped automatically to chat or completion endpoint)
-- `openai:chat:<model name>` - uses any model name against the chat endpoint
-- `openai:completion:<model name>` - uses any model name against the completion endpoint
-
-The `openai:<endpoint>:<model>` construction is useful if OpenAI releases a new model, or if you have a custom model. For example, if OpenAI releases gpt-5 chat completion, you could begin using it immediately with `openai:chat:gpt-5`.
-
-### LocalAI
-
-LocalAI is an API wrapper for open-source LLMs that is compatible with OpenAI. You can run LocalAI for compatibility with Llama, Alpaca, Vicuna, GPT4All, RedPajama, and many other models compatible with the ggml format.
-
-View all compatible models [here](https://github.com/go-skynet/LocalAI#model-compatibility-table).
-
-Once you have LocalAI up and running, specify one of the following based on the model you have selected:
-
-- `localai:chat:<model name>`
-- `localai:completion:<model name>`
-- `localai:<model name>` - defaults to chat-type model
-
-The model name is typically the filename of the .bin file that you downloaded to set up the model in LocalAI. For example, `ggml-vic13b-uncensored-q5_1.bin`
-
-Supported environment variables:
-
-- `LOCALAI_BASE_URL` - defaults to `http://localhost:8080/v1`
-- `REQUEST_TIMEOUT_MS` - maximum request time, in milliseconds. Defaults to 60000.
-
-### Custom API Provider
-
-To create a custom API provider, implement the `ApiProvider` interface in a separate module. Here is the interface:
-
-```js
-export interface ApiProvider {
-  id: () => string;
-  callApi: (prompt: string) => Promise<ProviderResult>;
-}
-```
-
-Below is an example of a custom API provider that returns a predefined output and token usage:
-
-```javascript
-// customApiProvider.js
-import fetch from 'node-fetch';
-
-class CustomApiProvider {
-  id() {
-    return 'my-custom-api';
-  }
-
-  async callApi(prompt) {
-    // Add your custom API logic here
-
-    return {
-      // Required
-      output: 'Model output',
-
-      // Optional
-      tokenUsage: {
-        total: 10,
-        prompt: 5,
-        completion: 5,
-      },
-    };
-  }
-}
-
-export default CustomApiProvider;
-```
-
-To use the custom API provider with `promptfoo`, pass the path to the module as the `provider` option in the CLI invocation:
-
-```bash
-promptfoo eval -p prompt1.txt prompt2.txt -o results.csv  -v vars.csv -r ./customApiProvider.js
-```
-
-This command will evaluate the prompts using the custom API provider and save the results to the specified CSV file.
+We support OpenAI's API as well as a number of open-source models. It's also to set up your own custom API provider. **[See Provider documentation](https://promptfoo.dev/docs/configuration/providers)** for more details.
 
 ## Development
 
@@ -428,3 +211,5 @@ Here are some of the available scripts:
 - `build:watch`: Continuously watch and transpile TypeScript files on changes
 - `test`: Run test suite
 - `test:watch`: Continuously run test suite on changes
+
+# [» View full documentation «](https://promptfoo.dev/docs/intro)

package/dist/__mocks__/esm.js

@@ -1,4 +1,8 @@
-export function getDirectory() {
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.getDirectory = void 0;
+function getDirectory() {
     return '/test/dir';
 }
+exports.getDirectory = getDirectory;
 //# sourceMappingURL=esm.js.map

package/dist/__mocks__/esm.js.map

@@ -1 +1 @@
-{"version":3,"file":"esm.js","sourceRoot":"","sources":["../../src/__mocks__/esm.ts"],"names":[],"mappings":"AAAA,MAAM,UAAU,YAAY;IAC1B,OAAO,WAAW,CAAC;AACrB,CAAC"}
+{"version":3,"file":"esm.js","sourceRoot":"","sources":["../../src/__mocks__/esm.ts"],"names":[],"mappings":";;;AAAA,SAAgB,YAAY;IAC1B,OAAO,WAAW,CAAC;AACrB,CAAC;AAFD,oCAEC"}

package/dist/assertions.d.ts

@@ -0,0 +1,18 @@
+import type { EvaluateOptions, GradingConfig, TokenUsage } from './types.js';
+interface GradingResult {
+    pass: boolean;
+    reason: string;
+    tokensUsed: TokenUsage;
+}
+export declare function matchesExpectedValue(expected: string, output: string, options: EvaluateOptions): Promise<{
+    pass: boolean;
+    reason?: string;
+}>;
+export declare function matchesSimilarity(expected: string, output: string, threshold: number): Promise<GradingResult>;
+export declare function matchesLlmRubric(expected: string, output: string, options?: GradingConfig): Promise<GradingResult>;
+declare const _default: {
+    matchesSimilarity: typeof matchesSimilarity;
+    matchesLlmRubric: typeof matchesLlmRubric;
+};
+export default _default;
+//# sourceMappingURL=assertions.d.ts.map

package/dist/assertions.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"assertions.d.ts","sourceRoot":"","sources":["../src/assertions.ts"],"names":[],"mappings":"AAOA,OAAO,KAAK,EAAE,eAAe,EAAE,aAAa,EAAE,UAAU,EAAE,MAAM,YAAY,CAAC;AAE7E,UAAU,aAAa;IACrB,IAAI,EAAE,OAAO,CAAC;IACd,MAAM,EAAE,MAAM,CAAC;IACf,UAAU,EAAE,UAAU,CAAC;CACxB;AAMD,wBAAsB,oBAAoB,CACxC,QAAQ,EAAE,MAAM,EAChB,MAAM,EAAE,MAAM,EACd,OAAO,EAAE,eAAe,GACvB,OAAO,CAAC;IAAE,IAAI,EAAE,OAAO,CAAC;IAAC,MAAM,CAAC,EAAE,MAAM,CAAA;CAAE,CAAC,CAuB7C;AAED,wBAAsB,iBAAiB,CACrC,QAAQ,EAAE,MAAM,EAChB,MAAM,EAAE,MAAM,EACd,SAAS,EAAE,MAAM,GAChB,OAAO,CAAC,aAAa,CAAC,CA0CxB;AAED,wBAAsB,gBAAgB,CACpC,QAAQ,EAAE,MAAM,EAChB,MAAM,EAAE,MAAM,EACd,OAAO,CAAC,EAAE,aAAa,GACtB,OAAO,CAAC,aAAa,CAAC,CAgDxB;;;;;AAED,wBAGE"}

package/dist/assertions.js

@@ -0,0 +1,128 @@
+"use strict";
+var __importDefault = (this && this.__importDefault) || function (mod) {
+    return (mod && mod.__esModule) ? mod : { "default": mod };
+};
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.matchesLlmRubric = exports.matchesSimilarity = exports.matchesExpectedValue = void 0;
+const nunjucks_1 = __importDefault(require("nunjucks"));
+const openai_js_1 = require("./providers/openai.js");
+const util_js_1 = require("./util.js");
+const providers_js_1 = require("./providers.js");
+const prompts_js_1 = require("./prompts.js");
+const SIMILAR_REGEX = /similar(?::|\((\d+(\.\d+)?)\):)/;
+const DEFAULT_SEMANTIC_SIMILARITY_THRESHOLD = 0.8;
+async function matchesExpectedValue(expected, output, options) {
+    const match = expected.match(SIMILAR_REGEX);
+    if (match) {
+        const threshold = parseFloat(match[1]) || DEFAULT_SEMANTIC_SIMILARITY_THRESHOLD;
+        const rest = expected.replace(SIMILAR_REGEX, '').trim();
+        return matchesSimilarity(rest, output, threshold);
+    }
+    else if (expected.startsWith('fn:') || expected.startsWith('eval:')) {
+        // TODO(1.0): delete eval: legacy option
+        const sliceLength = expected.startsWith('fn:') ? 'fn:'.length : 'eval:'.length;
+        const functionBody = expected.slice(sliceLength);
+        const customFunction = new Function('output', `return ${functionBody}`);
+        return { pass: customFunction(output) };
+    }
+    else if (expected.startsWith('grade:')) {
+        return matchesLlmRubric(expected.slice(6), output, options.grading);
+    }
+    else {
+        const pass = expected === output;
+        return {
+            pass,
+            reason: pass ? undefined : `Expected: ${expected}, Output: ${output}`,
+        };
+    }
+}
+exports.matchesExpectedValue = matchesExpectedValue;
+async function matchesSimilarity(expected, output, threshold) {
+    const expectedEmbedding = await openai_js_1.DefaultEmbeddingProvider.callEmbeddingApi(expected);
+    const outputEmbedding = await openai_js_1.DefaultEmbeddingProvider.callEmbeddingApi(output);
+    const tokensUsed = {
+        total: (expectedEmbedding.tokenUsage?.total || 0) + (outputEmbedding.tokenUsage?.total || 0),
+        prompt: (expectedEmbedding.tokenUsage?.prompt || 0) + (outputEmbedding.tokenUsage?.prompt || 0),
+        completion: (expectedEmbedding.tokenUsage?.completion || 0) +
+            (outputEmbedding.tokenUsage?.completion || 0),
+    };
+    if (expectedEmbedding.error || outputEmbedding.error) {
+        return {
+            pass: false,
+            reason: expectedEmbedding.error || outputEmbedding.error || 'Unknown error fetching embeddings',
+            tokensUsed,
+        };
+    }
+    if (!expectedEmbedding.embedding || !outputEmbedding.embedding) {
+        return {
+            pass: false,
+            reason: 'Embedding not found',
+            tokensUsed,
+        };
+    }
+    const similarity = (0, util_js_1.cosineSimilarity)(expectedEmbedding.embedding, outputEmbedding.embedding);
+    if (similarity < threshold) {
+        return {
+            pass: false,
+            reason: `Similarity ${similarity} is less than threshold ${threshold}`,
+            tokensUsed,
+        };
+    }
+    return {
+        pass: true,
+        reason: `Similarity ${similarity} is greater than threshold ${threshold}`,
+        tokensUsed,
+    };
+}
+exports.matchesSimilarity = matchesSimilarity;
+async function matchesLlmRubric(expected, output, options) {
+    if (!options) {
+        throw new Error('Cannot grade output without grading config. Specify --grader option or grading config.');
+    }
+    const prompt = nunjucks_1.default.renderString(options.prompt || prompts_js_1.DEFAULT_GRADING_PROMPT, {
+        content: output,
+        rubric: expected,
+    });
+    let provider = options.provider || openai_js_1.DefaultGradingProvider;
+    if (typeof provider === 'string') {
+        provider = await (0, providers_js_1.loadApiProvider)(provider);
+    }
+    const resp = await provider.callApi(prompt);
+    if (resp.error || !resp.output) {
+        return {
+            pass: false,
+            reason: resp.error || 'No output',
+            tokensUsed: {
+                total: resp.tokenUsage?.total || 0,
+                prompt: resp.tokenUsage?.prompt || 0,
+                completion: resp.tokenUsage?.completion || 0,
+            },
+        };
+    }
+    try {
+        const parsed = JSON.parse(resp.output);
+        parsed.tokensUsed = {
+            total: resp.tokenUsage?.total || 0,
+            prompt: resp.tokenUsage?.prompt || 0,
+            completion: resp.tokenUsage?.completion || 0,
+        };
+        return parsed;
+    }
+    catch (err) {
+        return {
+            pass: false,
+            reason: `Output is not valid JSON: ${resp.output}`,
+            tokensUsed: {
+                total: resp.tokenUsage?.total || 0,
+                prompt: resp.tokenUsage?.prompt || 0,
+                completion: resp.tokenUsage?.completion || 0,
+            },
+        };
+    }
+}
+exports.matchesLlmRubric = matchesLlmRubric;
+exports.default = {
+    matchesSimilarity,
+    matchesLlmRubric,
+};
+//# sourceMappingURL=assertions.js.map

package/dist/assertions.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"assertions.js","sourceRoot":"","sources":["../src/assertions.ts"],"names":[],"mappings":";;;;;;AAAA,wDAAgC;AAEhC,qDAAyF;AACzF,uCAA6C;AAC7C,iDAAiD;AACjD,6CAAsD;AAUtD,MAAM,aAAa,GAAG,iCAAiC,CAAC;AAExD,MAAM,qCAAqC,GAAG,GAAG,CAAC;AAE3C,KAAK,UAAU,oBAAoB,CACxC,QAAgB,EAChB,MAAc,EACd,OAAwB;IAExB,MAAM,KAAK,GAAG,QAAQ,CAAC,KAAK,CAAC,aAAa,CAAC,CAAC;IAE5C,IAAI,KAAK,EAAE;QACT,MAAM,SAAS,GAAG,UAAU,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,IAAI,qCAAqC,CAAC;QAChF,MAAM,IAAI,GAAG,QAAQ,CAAC,OAAO,CAAC,aAAa,EAAE,EAAE,CAAC,CAAC,IAAI,EAAE,CAAC;QACxD,OAAO,iBAAiB,CAAC,IAAI,EAAE,MAAM,EAAE,SAAS,CAAC,CAAC;KACnD;SAAM,IAAI,QAAQ,CAAC,UAAU,CAAC,KAAK,CAAC,IAAI,QAAQ,CAAC,UAAU,CAAC,OAAO,CAAC,EAAE;QACrE,wCAAwC;QACxC,MAAM,WAAW,GAAG,QAAQ,CAAC,UAAU,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,CAAC,OAAO,CAAC,MAAM,CAAC;QAC/E,MAAM,YAAY,GAAG,QAAQ,CAAC,KAAK,CAAC,WAAW,CAAC,CAAC;QAEjD,MAAM,cAAc,GAAG,IAAI,QAAQ,CAAC,QAAQ,EAAE,UAAU,YAAY,EAAE,CAAC,CAAC;QACxE,OAAO,EAAE,IAAI,EAAE,cAAc,CAAC,MAAM,CAAC,EAAE,CAAC;KACzC;SAAM,IAAI,QAAQ,CAAC,UAAU,CAAC,QAAQ,CAAC,EAAE;QACxC,OAAO,gBAAgB,CAAC,QAAQ,CAAC,KAAK,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,OAAO,CAAC,OAAO,CAAC,CAAC;KACrE;SAAM;QACL,MAAM,IAAI,GAAG,QAAQ,KAAK,MAAM,CAAC;QACjC,OAAO;YACL,IAAI;YACJ,MAAM,EAAE,IAAI,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,aAAa,QAAQ,aAAa,MAAM,EAAE;SACtE,CAAC;KACH;AACH,CAAC;AA3BD,oDA2BC;AAEM,KAAK,UAAU,iBAAiB,CACrC,QAAgB,EAChB,MAAc,EACd,SAAiB;IAEjB,MAAM,iBAAiB,GAAG,MAAM,oCAAwB,CAAC,gBAAgB,CAAC,QAAQ,CAAC,CAAC;IACpF,MAAM,eAAe,GAAG,MAAM,oCAAwB,CAAC,gBAAgB,CAAC,MAAM,CAAC,CAAC;IAEhF,MAAM,UAAU,GAAG;QACjB,KAAK,EAAE,CAAC,iBAAiB,CAAC,UAAU,EAAE,KAAK,IAAI,CAAC,CAAC,GAAG,CAAC,eAAe,CAAC,UAAU,EAAE,KAAK,IAAI,CAAC,CAAC;QAC5F,MAAM,EAAE,CAAC,iBAAiB,CAAC,UAAU,EAAE,MAAM,IAAI,CAAC,CAAC,GAAG,CAAC,eAAe,CAAC,UAAU,EAAE,MAAM,IAAI,CAAC,CAAC;QAC/F,UAAU,EACR,CAAC,iBAAiB,CAAC,UAAU,EAAE,UAAU,IAAI,CAAC,CAAC;YAC/C,CAAC,eAAe,CAAC,UAAU,EAAE,UAAU,IAAI,CAAC,CAAC;KAChD,CAAC;IAEF,IAAI,iBAAiB,CAAC,KAAK,IAAI,eAAe,CAAC,KAAK,EAAE;QACpD,OAAO;YACL,IAAI,EAAE,KAAK;YACX,MAAM,EACJ,iBAAiB,CAAC,KAAK,IAAI,eAAe,CAAC,KAAK,IAAI,mCAAmC;YACzF,UAAU;SACX,CAAC;KACH;IAED,IAAI,CAAC,iBAAiB,CAAC,SAAS,IAAI,CAAC,eAAe,CAAC,SAAS,EAAE;QAC9D,OAAO;YACL,IAAI,EAAE,KAAK;YACX,MAAM,EAAE,qBAAqB;YAC7B,UAAU;SACX,CAAC;KACH;IAED,MAAM,UAAU,GAAG,IAAA,0BAAgB,EAAC,iBAAiB,CAAC,SAAS,EAAE,eAAe,CAAC,SAAS,CAAC,CAAC;IAC5F,IAAI,UAAU,GAAG,SAAS,EAAE;QAC1B,OAAO;YACL,IAAI,EAAE,KAAK;YACX,MAAM,EAAE,cAAc,UAAU,2BAA2B,SAAS,EAAE;YACtE,UAAU;SACX,CAAC;KACH;IACD,OAAO;QACL,IAAI,EAAE,IAAI;QACV,MAAM,EAAE,cAAc,UAAU,8BAA8B,SAAS,EAAE;QACzE,UAAU;KACX,CAAC;AACJ,CAAC;AA9CD,8CA8CC;AAEM,KAAK,UAAU,gBAAgB,CACpC,QAAgB,EAChB,MAAc,EACd,OAAuB;IAEvB,IAAI,CAAC,OAAO,EAAE;QACZ,MAAM,IAAI,KAAK,CACb,wFAAwF,CACzF,CAAC;KACH;IAED,MAAM,MAAM,GAAG,kBAAQ,CAAC,YAAY,CAAC,OAAO,CAAC,MAAM,IAAI,mCAAsB,EAAE;QAC7E,OAAO,EAAE,MAAM;QACf,MAAM,EAAE,QAAQ;KACjB,CAAC,CAAC;IAEH,IAAI,QAAQ,GAAG,OAAO,CAAC,QAAQ,IAAI,kCAAsB,CAAC;IAC1D,IAAI,OAAO,QAAQ,KAAK,QAAQ,EAAE;QAChC,QAAQ,GAAG,MAAM,IAAA,8BAAe,EAAC,QAAQ,CAAC,CAAC;KAC5C;IACD,MAAM,IAAI,GAAG,MAAM,QAAQ,CAAC,OAAO,CAAC,MAAM,CAAC,CAAC;IAC5C,IAAI,IAAI,CAAC,KAAK,IAAI,CAAC,IAAI,CAAC,MAAM,EAAE;QAC9B,OAAO;YACL,IAAI,EAAE,KAAK;YACX,MAAM,EAAE,IAAI,CAAC,KAAK,IAAI,WAAW;YACjC,UAAU,EAAE;gBACV,KAAK,EAAE,IAAI,CAAC,UAAU,EAAE,KAAK,IAAI,CAAC;gBAClC,MAAM,EAAE,IAAI,CAAC,UAAU,EAAE,MAAM,IAAI,CAAC;gBACpC,UAAU,EAAE,IAAI,CAAC,UAAU,EAAE,UAAU,IAAI,CAAC;aAC7C;SACF,CAAC;KACH;IAED,IAAI;QACF,MAAM,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,MAAM,CAAkB,CAAC;QACxD,MAAM,CAAC,UAAU,GAAG;YAClB,KAAK,EAAE,IAAI,CAAC,UAAU,EAAE,KAAK,IAAI,CAAC;YAClC,MAAM,EAAE,IAAI,CAAC,UAAU,EAAE,MAAM,IAAI,CAAC;YACpC,UAAU,EAAE,IAAI,CAAC,UAAU,EAAE,UAAU,IAAI,CAAC;SAC7C,CAAC;QACF,OAAO,MAAM,CAAC;KACf;IAAC,OAAO,GAAG,EAAE;QACZ,OAAO;YACL,IAAI,EAAE,KAAK;YACX,MAAM,EAAE,6BAA6B,IAAI,CAAC,MAAM,EAAE;YAClD,UAAU,EAAE;gBACV,KAAK,EAAE,IAAI,CAAC,UAAU,EAAE,KAAK,IAAI,CAAC;gBAClC,MAAM,EAAE,IAAI,CAAC,UAAU,EAAE,MAAM,IAAI,CAAC;gBACpC,UAAU,EAAE,IAAI,CAAC,UAAU,EAAE,UAAU,IAAI,CAAC;aAC7C;SACF,CAAC;KACH;AACH,CAAC;AApDD,4CAoDC;AAED,kBAAe;IACb,iBAAiB;IACjB,gBAAgB;CACjB,CAAC"}

package/dist/cache.d.ts

@@ -0,0 +1,8 @@
+import type { RequestInfo, RequestInit } from 'node-fetch';
+export declare function fetchJsonWithCache(url: RequestInfo, options: RequestInit | undefined, timeout: number): Promise<{
+    data: any;
+    cached: boolean;
+}>;
+export declare function enableCache(): void;
+export declare function disableCache(): void;
+//# sourceMappingURL=cache.d.ts.map

package/dist/cache.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"cache.d.ts","sourceRoot":"","sources":["../src/cache.ts"],"names":[],"mappings":"AAUA,OAAO,KAAK,EAAE,WAAW,EAAE,WAAW,EAAE,MAAM,YAAY,CAAC;AA4B3D,wBAAsB,kBAAkB,CACtC,GAAG,EAAE,WAAW,EAChB,OAAO,yBAAkB,EACzB,OAAO,EAAE,MAAM,GACd,OAAO,CAAC;IAAE,IAAI,EAAE,GAAG,CAAC;IAAC,MAAM,EAAE,OAAO,CAAA;CAAE,CAAC,CAuCzC;AAED,wBAAgB,WAAW,SAE1B;AAED,wBAAgB,YAAY,SAG3B"}

package/dist/cache.js

@@ -0,0 +1,78 @@
+"use strict";
+var __importDefault = (this && this.__importDefault) || function (mod) {
+    return (mod && mod.__esModule) ? mod : { "default": mod };
+};
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.disableCache = exports.enableCache = exports.fetchJsonWithCache = void 0;
+const node_path_1 = __importDefault(require("node:path"));
+const cache_manager_1 = __importDefault(require("cache-manager"));
+const cache_manager_fs_hash_1 = __importDefault(require("cache-manager-fs-hash"));
+const logger_js_1 = __importDefault(require("./logger.js"));
+const util_js_1 = require("./util.js");
+let cacheInstance;
+let enabled = typeof process.env.PROMPTFOO_CACHE_ENABLED === 'undefined'
+    ? true
+    : Boolean(process.env.PROMPTFOO_CACHE_ENABLED);
+const cacheType = process.env.PROMPTFOO_CACHE_TYPE || (process.env.NODE_ENV === 'test' ? 'memory' : 'disk');
+function getCache() {
+    if (!cacheInstance) {
+        cacheInstance = cache_manager_1.default.caching({
+            store: cacheType === 'disk' ? cache_manager_fs_hash_1.default : 'memory',
+            options: {
+                max: process.env.PROMPTFOO_CACHE_MAX_FILE_COUNT || 10000,
+                path: process.env.PROMPTFOO_CACHE_PATH || node_path_1.default.join((0, util_js_1.getConfigDirectoryPath)(), 'cache'),
+                ttl: process.env.PROMPTFOO_CACHE_TTL || 60 * 60 * 24 * 14,
+                maxsize: process.env.PROMPTFOO_CACHE_MAX_SIZE || 1e7, // in bytes, 10mb
+                //zip: true, // whether to use gzip compression
+            },
+        });
+    }
+    return cacheInstance;
+}
+async function fetchJsonWithCache(url, options = {}, timeout) {
+    if (!enabled) {
+        const resp = await (0, util_js_1.fetchWithTimeout)(url, options, timeout);
+        return {
+            cached: false,
+            data: await resp.json(),
+        };
+    }
+    const cache = await getCache();
+    const copy = Object.assign({}, options);
+    delete copy.headers;
+    const cacheKey = `fetch:${url}:${JSON.stringify(copy)}`;
+    // Try to get the cached response
+    const cachedResponse = await cache.get(cacheKey);
+    if (cachedResponse) {
+        logger_js_1.default.debug(`Returning cached response for ${url}: ${cachedResponse}`);
+        return {
+            cached: true,
+            data: JSON.parse(cachedResponse),
+        };
+    }
+    // Fetch the actual data and store it in the cache
+    const response = await (0, util_js_1.fetchWithTimeout)(url, options, timeout);
+    try {
+        const data = await response.json();
+        logger_js_1.default.debug(`Storing ${url} response in cache: ${data}`);
+        await cache.set(cacheKey, JSON.stringify(data));
+        return {
+            cached: false,
+            data,
+        };
+    }
+    catch (err) {
+        throw new Error(`Error parsing response from ${url}: ${err}`);
+    }
+}
+exports.fetchJsonWithCache = fetchJsonWithCache;
+function enableCache() {
+    enabled = true;
+}
+exports.enableCache = enableCache;
+function disableCache() {
+    logger_js_1.default.info('Cache is disabled.');
+    enabled = false;
+}
+exports.disableCache = disableCache;
+//# sourceMappingURL=cache.js.map

package/dist/cache.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"cache.js","sourceRoot":"","sources":["../src/cache.ts"],"names":[],"mappings":";;;;;;AAAA,0DAA6B;AAE7B,kEAAyC;AACzC,kFAA4C;AAG5C,4DAAiC;AACjC,uCAAqE;AAKrE,IAAI,aAAgC,CAAC;AAErC,IAAI,OAAO,GACT,OAAO,OAAO,CAAC,GAAG,CAAC,uBAAuB,KAAK,WAAW;IACxD,CAAC,CAAC,IAAI;IACN,CAAC,CAAC,OAAO,CAAC,OAAO,CAAC,GAAG,CAAC,uBAAuB,CAAC,CAAC;AAEnD,MAAM,SAAS,GACb,OAAO,CAAC,GAAG,CAAC,oBAAoB,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,QAAQ,KAAK,MAAM,CAAC,CAAC,CAAC,QAAQ,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC;AAE5F,SAAS,QAAQ;IACf,IAAI,CAAC,aAAa,EAAE;QAClB,aAAa,GAAG,uBAAY,CAAC,OAAO,CAAC;YACnC,KAAK,EAAE,SAAS,KAAK,MAAM,CAAC,CAAC,CAAC,+BAAO,CAAC,CAAC,CAAC,QAAQ;YAChD,OAAO,EAAE;gBACP,GAAG,EAAE,OAAO,CAAC,GAAG,CAAC,8BAA8B,IAAI,KAAM;gBACzD,IAAI,EAAE,OAAO,CAAC,GAAG,CAAC,oBAAoB,IAAI,mBAAI,CAAC,IAAI,CAAC,IAAA,gCAAsB,GAAE,EAAE,OAAO,CAAC;gBACtF,GAAG,EAAE,OAAO,CAAC,GAAG,CAAC,mBAAmB,IAAI,EAAE,GAAG,EAAE,GAAG,EAAE,GAAG,EAAE;gBACzD,OAAO,EAAE,OAAO,CAAC,GAAG,CAAC,wBAAwB,IAAI,GAAG,EAAE,iBAAiB;gBACvE,+CAA+C;aAChD;SACF,CAAC,CAAC;KACJ;IACD,OAAO,aAAa,CAAC;AACvB,CAAC;AAEM,KAAK,UAAU,kBAAkB,CACtC,GAAgB,EAChB,UAAuB,EAAE,EACzB,OAAe;IAEf,IAAI,CAAC,OAAO,EAAE;QACZ,MAAM,IAAI,GAAG,MAAM,IAAA,0BAAgB,EAAC,GAAG,EAAE,OAAO,EAAE,OAAO,CAAC,CAAC;QAC3D,OAAO;YACL,MAAM,EAAE,KAAK;YACb,IAAI,EAAE,MAAM,IAAI,CAAC,IAAI,EAAE;SACxB,CAAC;KACH;IAED,MAAM,KAAK,GAAG,MAAM,QAAQ,EAAE,CAAC;IAE/B,MAAM,IAAI,GAAG,MAAM,CAAC,MAAM,CAAC,EAAE,EAAE,OAAO,CAAC,CAAC;IACxC,OAAO,IAAI,CAAC,OAAO,CAAC;IACpB,MAAM,QAAQ,GAAG,SAAS,GAAG,IAAI,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,EAAE,CAAC;IAExD,iCAAiC;IACjC,MAAM,cAAc,GAAG,MAAM,KAAK,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC;IAEjD,IAAI,cAAc,EAAE;QAClB,mBAAM,CAAC,KAAK,CAAC,iCAAiC,GAAG,KAAK,cAAc,EAAE,CAAC,CAAC;QACxE,OAAO;YACL,MAAM,EAAE,IAAI;YACZ,IAAI,EAAE,IAAI,CAAC,KAAK,CAAC,cAAwB,CAAC;SAC3C,CAAC;KACH;IAED,kDAAkD;IAClD,MAAM,QAAQ,GAAG,MAAM,IAAA,0BAAgB,EAAC,GAAG,EAAE,OAAO,EAAE,OAAO,CAAC,CAAC;IAC/D,IAAI;QACF,MAAM,IAAI,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,CAAC;QACnC,mBAAM,CAAC,KAAK,CAAC,WAAW,GAAG,uBAAuB,IAAI,EAAE,CAAC,CAAC;QAC1D,MAAM,KAAK,CAAC,GAAG,CAAC,QAAQ,EAAE,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC,CAAC;QAChD,OAAO;YACL,MAAM,EAAE,KAAK;YACb,IAAI;SACL,CAAC;KACH;IAAC,OAAO,GAAG,EAAE;QACZ,MAAM,IAAI,KAAK,CAAC,+BAA+B,GAAG,KAAK,GAAG,EAAE,CAAC,CAAC;KAC/D;AACH,CAAC;AA3CD,gDA2CC;AAED,SAAgB,WAAW;IACzB,OAAO,GAAG,IAAI,CAAC;AACjB,CAAC;AAFD,kCAEC;AAED,SAAgB,YAAY;IAC1B,mBAAM,CAAC,IAAI,CAAC,oBAAoB,CAAC,CAAC;IAClC,OAAO,GAAG,KAAK,CAAC;AAClB,CAAC;AAHD,oCAGC"}

package/dist/esm.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"esm.d.ts","sourceRoot":"","sources":["../src/esm.ts"],"names":[],"mappings":"AAKA,wBAAgB,YAAY,IAAI,MAAM,CAIrC"}
+{"version":3,"file":"esm.d.ts","sourceRoot":"","sources":["../src/esm.ts"],"names":[],"mappings":"AAKA,wBAAgB,YAAY,IAAI,MAAM,CAOrC"}

package/dist/esm.js

@@ -1,9 +1,16 @@
+"use strict";
 // esm-specific crap that needs to get mocked out in tests
-import path from 'path';
-import { fileURLToPath } from 'url';
-export function getDirectory() {
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.getDirectory = void 0;
+//import path from 'path';
+//import { fileURLToPath } from 'url';
+function getDirectory() {
+    /*
     // @ts-ignore: Jest chokes on this
     const __filename = fileURLToPath(import.meta.url);
     return path.dirname(__filename);
+   */
+    return __dirname;
 }
+exports.getDirectory = getDirectory;
 //# sourceMappingURL=esm.js.map

package/dist/esm.js.map

@@ -1 +1 @@
-{"version":3,"file":"esm.js","sourceRoot":"","sources":["../src/esm.ts"],"names":[],"mappings":"AAAA,0DAA0D;AAE1D,OAAO,IAAI,MAAM,MAAM,CAAC;AACxB,OAAO,EAAE,aAAa,EAAE,MAAM,KAAK,CAAC;AAEpC,MAAM,UAAU,YAAY;IAC1B,kCAAkC;IAClC,MAAM,UAAU,GAAG,aAAa,CAAC,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;IAClD,OAAO,IAAI,CAAC,OAAO,CAAC,UAAU,CAAC,CAAC;AAClC,CAAC"}
+{"version":3,"file":"esm.js","sourceRoot":"","sources":["../src/esm.ts"],"names":[],"mappings":";AAAA,0DAA0D;;;AAE1D,0BAA0B;AAC1B,sCAAsC;AAEtC,SAAgB,YAAY;IAC1B;;;;KAIC;IACD,OAAO,SAAS,CAAC;AACnB,CAAC;AAPD,oCAOC"}

package/dist/evaluator.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"evaluator.d.ts","sourceRoot":"","sources":["../src/evaluator.ts"],"names":[],"mappings":"AAOA,OAAO,KAAK,EAEV,eAAe,EAGf,eAAe,EAIhB,MAAM,YAAY,CAAC;AAiWpB,wBAAgB,QAAQ,CAAC,OAAO,EAAE,eAAe,4BAGhD"}
+{"version":3,"file":"evaluator.d.ts","sourceRoot":"","sources":["../src/evaluator.ts"],"names":[],"mappings":"AAUA,OAAO,KAAK,EAEV,eAAe,EAGf,eAAe,EAGhB,MAAM,YAAY,CAAC;AAmRpB,wBAAgB,QAAQ,CAAC,OAAO,EAAE,eAAe,4BAGhD"}