promptfoo 0.66.0 → 0.68.0

package/README.md

@@ -2,8 +2,8 @@
 
 [![npm](https://img.shields.io/npm/v/promptfoo)](https://npmjs.com/package/promptfoo)
 [![npm](https://img.shields.io/npm/dm/promptfoo)](https://npmjs.com/package/promptfoo)
-[![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/typpo/promptfoo/main.yml)](https://github.com/typpo/promptfoo/actions/workflows/main.yml)
-![MIT license](https://img.shields.io/github/license/typpo/promptfoo)
+[![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/typpo/promptfoo/main.yml)](https://github.com/promptfoo/promptfoo/actions/workflows/main.yml)
+![MIT license](https://img.shields.io/github/license/promptfoo/promptfoo)
 [![Discord](https://dcbadge.vercel.app/api/server/gHPS9jjfbs?style=flat&compact=true)](https://discord.gg/gHPS9jjfbs)
 
 `promptfoo` is a tool for testing and evaluating LLM apps.
@@ -30,7 +30,7 @@ promptfoo produces matrix views that let you quickly evaluate outputs across man
 
 It works on the command line too:
 
-![Prompt evaluation](https://github.com/typpo/promptfoo/assets/310310/480e1114-d049-40b9-bd5f-f81c15060284)
+![Prompt evaluation](https://github.com/promptfoo/promptfoo/assets/310310/480e1114-d049-40b9-bd5f-f81c15060284)
 
 ## Why choose promptfoo?
 
@@ -52,7 +52,7 @@ As you explore modifications to the prompt, use `promptfoo eval` to rate all out
 
 As you collect more examples and establish a user feedback loop, continue to build the pool of test cases.
 
-<img width="772" alt="LLM ops" src="https://github.com/typpo/promptfoo/assets/310310/cf0461a7-2832-4362-9fbb-4ebd911d06ff">
+<img width="772" alt="LLM ops" src="https://github.com/promptfoo/promptfoo/assets/310310/cf0461a7-2832-4362-9fbb-4ebd911d06ff">
 
 ## Usage
 
@@ -161,7 +161,7 @@ providers: [openai:gpt-3.5-turbo]
 tests: tests.csv
 ```
 
-See [example CSV](https://github.com/typpo/promptfoo/blob/main/examples/simple-test/tests.csv).
+See [example CSV](https://github.com/promptfoo/promptfoo/blob/main/examples/simple-test/tests.csv).
 
 ### Command-line
 
@@ -169,7 +169,7 @@ If you're looking to customize your usage, you have a wide set of parameters at
 
 | Option                              | Description                                                                                                                                                                            |
 | ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `-p, --prompts <paths...>`          | Paths to [prompt files](https://www.promptfoo.dev/docs/configuration/parameters#prompt-files), directory, or glob                                                                      |
+| `-p, --prompts <paths...>`          | Paths to [prompt files](https://www.promptfoo.dev/docs/configuration/parameters#prompts), directory, or glob                                                                           |
 | `-r, --providers <name or path...>` | One of: openai:chat, openai:completion, openai:model-name, localai:chat:model-name, localai:completion:model-name. See [API providers][providers-docs]                                 |
 | `-o, --output <path>`               | Path to [output file](https://www.promptfoo.dev/docs/configuration/parameters#output-file) (csv, json, yaml, html)                                                                     |
 | `--tests <path>`                    | Path to [external test file](https://www.promptfoo.dev/docs/configurationexpected-outputsassertions#load-an-external-tests-file)                                                       |
@@ -190,7 +190,7 @@ npx promptfoo view
 
 #### Prompt quality
 
-In [this example](https://github.com/typpo/promptfoo/tree/main/examples/assistant-cli), we evaluate whether adding adjectives to the personality of an assistant bot affects the responses:
+In [this example](https://github.com/promptfoo/promptfoo/tree/main/examples/assistant-cli), we evaluate whether adding adjectives to the personality of an assistant bot affects the responses:
 
 ```
 npx promptfoo eval -p prompts.txt -r openai:gpt-3.5-turbo -t tests.csv
@@ -204,13 +204,13 @@ npx promptfoo eval -p prompts.txt -r openai:gpt-3.5-turbo -t tests.csv
 
 This command will evaluate the prompts in `prompts.txt`, substituting the variable values from `vars.csv`, and output results in your terminal.
 
-You can also output a nice [spreadsheet](https://docs.google.com/spreadsheets/d/1nanoj3_TniWrDl1Sj-qYqIMD6jwm5FBy15xPFdUTsmI/edit?usp=sharing), [JSON](https://github.com/typpo/promptfoo/blob/main/examples/simple-cli/output.json), YAML, or an HTML file:
+You can also output a nice [spreadsheet](https://docs.google.com/spreadsheets/d/1nanoj3_TniWrDl1Sj-qYqIMD6jwm5FBy15xPFdUTsmI/edit?usp=sharing), [JSON](https://github.com/promptfoo/promptfoo/blob/main/examples/simple-cli/output.json), YAML, or an HTML file:
 
 ![Table output](https://user-images.githubusercontent.com/310310/235483444-4ddb832d-e103-4b9c-a862-b0d6cc11cdc0.png)
 
 #### Model quality
 
-In the [next example](https://github.com/typpo/promptfoo/tree/main/examples/gpt-3.5-vs-4), we evaluate the difference between GPT 3 and GPT 4 outputs for a given prompt:
+In the [next example](https://github.com/promptfoo/promptfoo/tree/main/examples/gpt-3.5-vs-4), we evaluate the difference between GPT 3 and GPT 4 outputs for a given prompt:
 
 ```
 npx promptfoo eval -p prompts.txt -r openai:gpt-3.5-turbo openai:gpt-4 -o output.html
@@ -302,7 +302,7 @@ const results = await promptfoo.evaluate({
 
 This code imports the `promptfoo` library, defines the evaluation options, and then calls the `evaluate` function with these options.
 
-See the full example [here](https://github.com/typpo/promptfoo/tree/main/examples/simple-import), which includes an example results object.
+See the full example [here](https://github.com/promptfoo/promptfoo/tree/main/examples/simple-import), which includes an example results object.
 
 ## Configuration
 

package/dist/package.json

@@ -2,7 +2,7 @@
     "name": "promptfoo",
     "description": "LLM eval & testing toolkit",
     "author": "Ian Webster",
-    "version": "0.66.0",
+    "version": "0.68.0",
     "license": "MIT",
     "type": "commonjs",
     "repository": "promptfoo/promptfoo",
@@ -23,7 +23,7 @@
         "dist"
     ],
     "engines": {
-        "node": ">=18"
+        "node": ">=18.0.0"
     },
     "bin": {
         "promptfoo": "dist/src/main.js"
@@ -49,12 +49,12 @@
         "prepublishOnly": "npm run build:clean && npm run build"
     },
     "peerDependencies": {
-        "@aws-sdk/client-bedrock-runtime": "^3.458.0",
+        "@aws-sdk/client-bedrock-runtime": "^3.602.0",
         "@azure/identity": "^4.0.0",
         "@azure/openai-assistants": "^1.0.0-beta.5",
         "@ibm-generative-ai/node-sdk": "^2.0.6",
+        "@smithy/node-http-handler": "^3.1.1",
         "google-auth-library": "^9.7.0",
-        "googleapis": "^134.0.0",
         "langfuse": "^3.7.0",
         "node-sql-parser": "^5.2.0"
     },
@@ -65,6 +65,7 @@
         "@swc/cli": "^0.3.12",
         "@swc/core": "^1.6.1",
         "@swc/jest": "^0.2.36",
+        "@trivago/prettier-plugin-sort-imports": "^4.3.0",
         "@types/async": "^3.2.24",
         "@types/better-sqlite3": "^7.6.10",
         "@types/cache-manager": "^4.0.6",
@@ -103,6 +104,7 @@
     "dependencies": {
         "@anthropic-ai/sdk": "^0.24.0",
         "@apidevtools/json-schema-ref-parser": "^11.6.4",
+        "@googleapis/sheets": "^8.0.0",
         "ajv": "^8.16.0",
         "ajv-formats": "^2.1.1",
         "async": "^3.2.5",

package/dist/src/assertions/validateAssertions.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"validateAssertions.d.ts","sourceRoot":"","sources":["../../../src/assertions/validateAssertions.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,MAAM,UAAU,CAAC;AAEpC,qBAAa,oBAAqB,SAAQ,KAAK;gBACjC,OAAO,EAAE,MAAM,EAAE,QAAQ,EAAE,QAAQ;CAMhD;AAED,wBAAgB,kBAAkB,CAAC,KAAK,EAAE,QAAQ,CAAC,MAAM,CAAC,MAAM,EAAE,MAAM,GAAG,MAAM,GAAG,MAAM,EAAE,CAAC,CAAC,EAAE,QAU/F"}
+{"version":3,"file":"validateAssertions.d.ts","sourceRoot":"","sources":["../../../src/assertions/validateAssertions.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,MAAM,UAAU,CAAC;AAEpC,qBAAa,oBAAqB,SAAQ,KAAK;gBACjC,OAAO,EAAE,MAAM,EAAE,QAAQ,EAAE,QAAQ;CAMhD;AAgBD,wBAAgB,kBAAkB,CAAC,KAAK,EAAE,QAAQ,CAAC,MAAM,CAAC,MAAM,EAAE,MAAM,GAAG,MAAM,GAAG,MAAM,EAAE,CAAC,CAAC,EAAE,QAU/F"}

package/dist/src/assertions/validateAssertions.js

@@ -9,6 +9,17 @@ class AssertValiationError extends Error {
     }
 }
 exports.AssertValiationError = AssertValiationError;
+function validateAssertSet(assertion, test) {
+    if (!('assert' in assertion)) {
+        throw new AssertValiationError('assert-set must have an `assert` property', test);
+    }
+    if (!Array.isArray(assertion.assert)) {
+        throw new AssertValiationError('assert-set `assert` must be an array of assertions', test);
+    }
+    if (assertion.assert.some((assertion) => assertion.type === 'assert-set')) {
+        throw new AssertValiationError('assert-set must not have child assert-sets', test);
+    }
+}
 function validateAssertions(tests) {
     for (const test of tests) {
         if (test.assert) {
@@ -21,15 +32,4 @@ function validateAssertions(tests) {
     }
 }
 exports.validateAssertions = validateAssertions;
-function validateAssertSet(assertion, test) {
-    if (!('assert' in assertion)) {
-        throw new AssertValiationError('assert-set must have an `assert` property', test);
-    }
-    if (!Array.isArray(assertion.assert)) {
-        throw new AssertValiationError('assert-set `assert` must be an array of assertions', test);
-    }
-    if (assertion.assert.some((assertion) => assertion.type === 'assert-set')) {
-        throw new AssertValiationError('assert-set must not have child assert-sets', test);
-    }
-}
 //# sourceMappingURL=validateAssertions.js.map

package/dist/src/assertions/validateAssertions.js.map

@@ -1 +1 @@
-{"version":3,"file":"validateAssertions.js","sourceRoot":"","sources":["../../../src/assertions/validateAssertions.ts"],"names":[],"mappings":";;;AAEA,MAAa,oBAAqB,SAAQ,KAAK;IAC7C,YAAY,OAAe,EAAE,QAAkB;QAC7C,MAAM,mBAAmB,GAAG,QAAQ,CAAC,WAAW,IAAI,IAAI,CAAC,SAAS,CAAC,QAAQ,CAAC,CAAC;QAE7E,KAAK,CAAC,GAAG,OAAO,SAAS,mBAAmB,EAAE,CAAC,CAAC;QAChD,IAAI,CAAC,IAAI,GAAG,sBAAsB,CAAC;IACrC,CAAC;CACF;AAPD,oDAOC;AAED,SAAgB,kBAAkB,CAAC,KAA6D;IAC9F,KAAK,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC;QACzB,IAAI,IAAI,CAAC,MAAM,EAAE,CAAC;YAChB,KAAK,MAAM,SAAS,IAAI,IAAI,CAAC,MAAM,EAAE,CAAC;gBACpC,IAAI,SAAS,CAAC,IAAI,KAAK,YAAY,EAAE,CAAC;oBACpC,iBAAiB,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;gBACrC,CAAC;YACH,CAAC;QACH,CAAC;IACH,CAAC;AACH,CAAC;AAVD,gDAUC;AAED,SAAS,iBAAiB,CAAC,SAAiB,EAAE,IAAc;IAC1D,IAAI,CAAC,CAAC,QAAQ,IAAI,SAAS,CAAC,EAAE,CAAC;QAC7B,MAAM,IAAI,oBAAoB,CAAC,2CAA2C,EAAE,IAAI,CAAC,CAAC;IACpF,CAAC;IAED,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,SAAS,CAAC,MAAM,CAAC,EAAE,CAAC;QACrC,MAAM,IAAI,oBAAoB,CAAC,oDAAoD,EAAE,IAAI,CAAC,CAAC;IAC7F,CAAC;IAED,IAAI,SAAS,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,SAAS,EAAE,EAAE,CAAC,SAAS,CAAC,IAAI,KAAK,YAAY,CAAC,EAAE,CAAC;QAC1E,MAAM,IAAI,oBAAoB,CAAC,4CAA4C,EAAE,IAAI,CAAC,CAAC;IACrF,CAAC;AACH,CAAC"}
+{"version":3,"file":"validateAssertions.js","sourceRoot":"","sources":["../../../src/assertions/validateAssertions.ts"],"names":[],"mappings":";;;AAEA,MAAa,oBAAqB,SAAQ,KAAK;IAC7C,YAAY,OAAe,EAAE,QAAkB;QAC7C,MAAM,mBAAmB,GAAG,QAAQ,CAAC,WAAW,IAAI,IAAI,CAAC,SAAS,CAAC,QAAQ,CAAC,CAAC;QAE7E,KAAK,CAAC,GAAG,OAAO,SAAS,mBAAmB,EAAE,CAAC,CAAC;QAChD,IAAI,CAAC,IAAI,GAAG,sBAAsB,CAAC;IACrC,CAAC;CACF;AAPD,oDAOC;AAED,SAAS,iBAAiB,CAAC,SAAiB,EAAE,IAAc;IAC1D,IAAI,CAAC,CAAC,QAAQ,IAAI,SAAS,CAAC,EAAE,CAAC;QAC7B,MAAM,IAAI,oBAAoB,CAAC,2CAA2C,EAAE,IAAI,CAAC,CAAC;IACpF,CAAC;IAED,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,SAAS,CAAC,MAAM,CAAC,EAAE,CAAC;QACrC,MAAM,IAAI,oBAAoB,CAAC,oDAAoD,EAAE,IAAI,CAAC,CAAC;IAC7F,CAAC;IAED,IAAI,SAAS,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC,SAAS,EAAE,EAAE,CAAC,SAAS,CAAC,IAAI,KAAK,YAAY,CAAC,EAAE,CAAC;QAC1E,MAAM,IAAI,oBAAoB,CAAC,4CAA4C,EAAE,IAAI,CAAC,CAAC;IACrF,CAAC;AACH,CAAC;AAED,SAAgB,kBAAkB,CAAC,KAA6D;IAC9F,KAAK,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC;QACzB,IAAI,IAAI,CAAC,MAAM,EAAE,CAAC;YAChB,KAAK,MAAM,SAAS,IAAI,IAAI,CAAC,MAAM,EAAE,CAAC;gBACpC,IAAI,SAAS,CAAC,IAAI,KAAK,YAAY,EAAE,CAAC;oBACpC,iBAAiB,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;gBACrC,CAAC;YACH,CAAC;QACH,CAAC;IACH,CAAC;AACH,CAAC;AAVD,gDAUC"}

package/dist/src/assertions.d.ts

@@ -1,29 +1,31 @@
 import { matchesSimilarity, matchesLlmRubric, matchesFactuality, matchesClosedQa, matchesClassification, matchesAnswerRelevance, matchesContextRecall, matchesContextRelevance, matchesContextFaithfulness, matchesSelectBest, matchesModeration } from './matchers';
 import { type ApiProvider, type Assertion, type AssertionType, type AtomicTestCase, type GradingResult, AssertionValue } from './types';
 export declare const MODEL_GRADED_ASSERTION_TYPES: Set<AssertionType>;
-export declare function runAssertions({ prompt, provider, test, output, latencyMs, logProbs, cost, }: {
+export declare function isSql(outputString: string, renderedValue: AssertionValue | undefined, inverse: boolean, assertion: Assertion): Promise<GradingResult>;
+export declare function runAssertion({ prompt, provider, assertion, test, output, latencyMs, logProbs, cost, }: {
     prompt?: string;
     provider?: ApiProvider;
+    assertion: Assertion;
     test: AtomicTestCase;
     output: string | object;
     latencyMs?: number;
     logProbs?: number[];
     cost?: number;
 }): Promise<GradingResult>;
-export declare function runAssertion({ prompt, provider, assertion, test, output, latencyMs, logProbs, cost, }: {
+export declare function runAssertions({ prompt, provider, test, output, latencyMs, logProbs, cost, }: {
     prompt?: string;
     provider?: ApiProvider;
-    assertion: Assertion;
     test: AtomicTestCase;
     output: string | object;
     latencyMs?: number;
     logProbs?: number[];
     cost?: number;
 }): Promise<GradingResult>;
-export declare function isSql(outputString: string, renderedValue: AssertionValue | undefined, inverse: boolean, assertion: Assertion): Promise<GradingResult>;
 export declare function runCompareAssertion(test: AtomicTestCase, assertion: Assertion, outputs: string[]): Promise<GradingResult[]>;
 export declare function readAssertions(filePath: string): Promise<Assertion[]>;
 declare const _default: {
+    runAssertion: typeof runAssertion;
+    runAssertions: typeof runAssertions;
     matchesSimilarity: typeof matchesSimilarity;
     matchesClassification: typeof matchesClassification;
     matchesLlmRubric: typeof matchesLlmRubric;

package/dist/src/assertions.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"assertions.d.ts","sourceRoot":"","sources":["../../src/assertions.ts"],"names":[],"mappings":"AAkBA,OAAO,EACL,iBAAiB,EACjB,gBAAgB,EAChB,iBAAiB,EACjB,eAAe,EACf,qBAAqB,EACrB,sBAAsB,EACtB,oBAAoB,EACpB,uBAAuB,EACvB,0BAA0B,EAC1B,iBAAiB,EACjB,iBAAiB,EAClB,MAAM,YAAY,CAAC;AAMpB,OAAO,EACL,KAAK,WAAW,EAChB,KAAK,SAAS,EACd,KAAK,aAAa,EAClB,KAAK,cAAc,EACnB,KAAK,aAAa,EAGlB,cAAc,EACf,MAAM,SAAS,CAAC;AASjB,eAAO,MAAM,4BAA4B,oBASvC,CAAC;AAqDH,wBAAsB,aAAa,CAAC,EAClC,MAAM,EACN,QAAQ,EACR,IAAI,EACJ,MAAM,EACN,SAAS,EACT,QAAQ,EACR,IAAI,GACL,EAAE;IACD,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,QAAQ,CAAC,EAAE,WAAW,CAAC;IACvB,IAAI,EAAE,cAAc,CAAC;IACrB,MAAM,EAAE,MAAM,GAAG,MAAM,CAAC;IACxB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,QAAQ,CAAC,EAAE,MAAM,EAAE,CAAC;IACpB,IAAI,CAAC,EAAE,MAAM,CAAC;CACf,GAAG,OAAO,CAAC,aAAa,CAAC,CAoFzB;AAED,wBAAsB,YAAY,CAAC,EACjC,MAAM,EACN,QAAQ,EACR,SAAS,EACT,IAAI,EACJ,MAAM,EACN,SAAS,EACT,QAAQ,EACR,IAAI,GACL,EAAE;IACD,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,QAAQ,CAAC,EAAE,WAAW,CAAC;IACvB,SAAS,EAAE,SAAS,CAAC;IACrB,IAAI,EAAE,cAAc,CAAC;IACrB,MAAM,EAAE,MAAM,GAAG,MAAM,CAAC;IACxB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,QAAQ,CAAC,EAAE,MAAM,EAAE,CAAC;IACpB,IAAI,CAAC,EAAE,MAAM,CAAC;CACf,GAAG,OAAO,CAAC,aAAa,CAAC,CAqhCzB;AAgCD,wBAAsB,KAAK,CACzB,YAAY,EAAE,MAAM,EACpB,aAAa,EAAE,cAAc,GAAG,SAAS,EACzC,OAAO,EAAE,OAAO,EAChB,SAAS,EAAE,SAAS,GACnB,OAAO,CAAC,aAAa,CAAC,CA2ExB;AAED,wBAAsB,mBAAmB,CACvC,IAAI,EAAE,cAAc,EACpB,SAAS,EAAE,SAAS,EACpB,OAAO,EAAE,MAAM,EAAE,GAChB,OAAO,CAAC,aAAa,EAAE,CAAC,CAe1B;AAED,wBAAsB,cAAc,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,EAAE,CAAC,CAU3E;;;;;;;;;;;;;;AAGD,wBAYE"}
+{"version":3,"file":"assertions.d.ts","sourceRoot":"","sources":["../../src/assertions.ts"],"names":[],"mappings":"AAiBA,OAAO,EACL,iBAAiB,EACjB,gBAAgB,EAChB,iBAAiB,EACjB,eAAe,EACf,qBAAqB,EACrB,sBAAsB,EACtB,oBAAoB,EACpB,uBAAuB,EACvB,0BAA0B,EAC1B,iBAAiB,EACjB,iBAAiB,EAClB,MAAM,YAAY,CAAC;AAMpB,OAAO,EACL,KAAK,WAAW,EAChB,KAAK,SAAS,EACd,KAAK,aAAa,EAClB,KAAK,cAAc,EACnB,KAAK,aAAa,EAGlB,cAAc,EACf,MAAM,SAAS,CAAC;AAOjB,eAAO,MAAM,4BAA4B,oBASvC,CAAC;AAqDH,wBAAsB,KAAK,CACzB,YAAY,EAAE,MAAM,EACpB,aAAa,EAAE,cAAc,GAAG,SAAS,EACzC,OAAO,EAAE,OAAO,EAChB,SAAS,EAAE,SAAS,GACnB,OAAO,CAAC,aAAa,CAAC,CA2ExB;AAED,wBAAsB,YAAY,CAAC,EACjC,MAAM,EACN,QAAQ,EACR,SAAS,EACT,IAAI,EACJ,MAAM,EACN,SAAS,EACT,QAAQ,EACR,IAAI,GACL,EAAE;IACD,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,QAAQ,CAAC,EAAE,WAAW,CAAC;IACvB,SAAS,EAAE,SAAS,CAAC;IACrB,IAAI,EAAE,cAAc,CAAC;IACrB,MAAM,EAAE,MAAM,GAAG,MAAM,CAAC;IACxB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,QAAQ,CAAC,EAAE,MAAM,EAAE,CAAC;IACpB,IAAI,CAAC,EAAE,MAAM,CAAC;CACf,GAAG,OAAO,CAAC,aAAa,CAAC,CAqhCzB;AAED,wBAAsB,aAAa,CAAC,EAClC,MAAM,EACN,QAAQ,EACR,IAAI,EACJ,MAAM,EACN,SAAS,EACT,QAAQ,EACR,IAAI,GACL,EAAE;IACD,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,QAAQ,CAAC,EAAE,WAAW,CAAC;IACvB,IAAI,EAAE,cAAc,CAAC;IACrB,MAAM,EAAE,MAAM,GAAG,MAAM,CAAC;IACxB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,QAAQ,CAAC,EAAE,MAAM,EAAE,CAAC;IACpB,IAAI,CAAC,EAAE,MAAM,CAAC;CACf,GAAG,OAAO,CAAC,aAAa,CAAC,CAoFzB;AAED,wBAAsB,mBAAmB,CACvC,IAAI,EAAE,cAAc,EACpB,SAAS,EAAE,SAAS,EACpB,OAAO,EAAE,MAAM,EAAE,GAChB,OAAO,CAAC,aAAa,EAAE,CAAC,CAe1B;AAED,wBAAsB,cAAc,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,SAAS,EAAE,CAAC,CAU3E;;;;;;;;;;;;;;;;AAGD,wBAcE"}

package/dist/src/assertions.js

@@ -26,30 +26,30 @@ var __importDefault = (this && this.__importDefault) || function (mod) {
     return (mod && mod.__esModule) ? mod : { "default": mod };
 };
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.readAssertions = exports.runCompareAssertion = exports.isSql = exports.runAssertion = exports.runAssertions = exports.MODEL_GRADED_ASSERTION_TYPES = void 0;
+exports.readAssertions = exports.runCompareAssertion = exports.runAssertions = exports.runAssertion = exports.isSql = exports.MODEL_GRADED_ASSERTION_TYPES = void 0;
+const ajv_1 = __importDefault(require("ajv"));
+const ajv_formats_1 = __importDefault(require("ajv-formats"));
+const async_1 = __importDefault(require("async"));
+const fastest_levenshtein_1 = require("fastest-levenshtein");
 const fs_1 = __importDefault(require("fs"));
-const path_1 = __importDefault(require("path"));
+const js_yaml_1 = __importDefault(require("js-yaml"));
 const node_util_1 = __importDefault(require("node:util"));
-const async_1 = __importDefault(require("async"));
+const path_1 = __importDefault(require("path"));
+const rfdc_1 = __importDefault(require("rfdc"));
 const rouge_1 = __importDefault(require("rouge"));
 const tiny_invariant_1 = __importDefault(require("tiny-invariant"));
-const js_yaml_1 = __importDefault(require("js-yaml"));
-const ajv_1 = __importDefault(require("ajv"));
-const ajv_formats_1 = __importDefault(require("ajv-formats"));
-const rfdc_1 = __importDefault(require("rfdc"));
-const fastest_levenshtein_1 = require("fastest-levenshtein");
+const AssertionsResult_1 = require("./assertions/AssertionsResult");
 const cliState_1 = __importDefault(require("./cliState"));
-const telemetry_1 = __importDefault(require("./telemetry"));
-const logger_1 = __importDefault(require("./logger"));
+const esm_1 = require("./esm");
 const fetch_1 = require("./fetch");
-const util_1 = require("./util");
+const logger_1 = __importDefault(require("./logger"));
 const matchers_1 = require("./matchers");
 const openaiUtil_1 = require("./providers/openaiUtil");
+const shared_1 = require("./providers/shared");
 const wrapper_1 = require("./python/wrapper");
-const esm_1 = require("./esm");
+const telemetry_1 = __importDefault(require("./telemetry"));
 const types_1 = require("./types");
-const AssertionsResult_1 = require("./assertions/AssertionsResult");
-const shared_1 = require("./providers/shared");
+const util_1 = require("./util");
 const ASSERTIONS_MAX_CONCURRENCY = process.env.PROMPTFOO_ASSERTIONS_MAX_CONCURRENCY
     ? parseInt(process.env.PROMPTFOO_ASSERTIONS_MAX_CONCURRENCY, 10)
     : 3;
@@ -96,71 +96,68 @@ function handleRougeScore(baseType, assertion, expected, output, inverted) {
         assertion,
     };
 }
-async function runAssertions({ prompt, provider, test, output, latencyMs, logProbs, cost, }) {
-    if (!test.assert || test.assert.length < 1) {
-        return AssertionsResult_1.AssertionsResult.noAssertsResult();
+async function isSql(outputString, renderedValue, inverse, assertion) {
+    let pass = false;
+    let parsedSql;
+    let databaseType = 'MySQL';
+    let whiteTableList;
+    let whiteColumnList;
+    if (renderedValue && typeof renderedValue === 'object') {
+        const value = renderedValue;
+        databaseType = value.database || 'MySQL';
+        whiteTableList = value.allowedTables;
+        whiteColumnList = value.allowedColumns;
     }
-    const mainAssertResult = new AssertionsResult_1.AssertionsResult({
-        threshold: test.threshold,
+    if (renderedValue && typeof renderedValue !== 'object') {
+        throw new Error('is-sql assertion must have a object value.');
+    }
+    const { Parser: SqlParser } = await Promise.resolve().then(() => __importStar(require('node-sql-parser'))).catch(() => {
+        throw new Error('node-sql-parser is not installed. Please install it first');
     });
-    const subAssertResults = [];
-    const asserts = test.assert
-        .map((assertion, i) => {
-        if (assertion.type === 'assert-set') {
-            const subAssertResult = new AssertionsResult_1.AssertionsResult({
-                threshold: assertion.threshold,
-                parentAssertionSet: {
-                    assertionSet: assertion,
-                    index: i,
-                },
-            });
-            subAssertResults.push(subAssertResult);
-            return assertion.assert.map((subAssert, j) => {
-                return {
-                    assertion: subAssert,
-                    assertResult: subAssertResult,
-                    index: j,
-                };
-            });
+    const sqlParser = new SqlParser();
+    const opt = { database: databaseType };
+    const failureReasons = [];
+    try {
+        parsedSql = sqlParser.astify(outputString, opt);
+        pass = !inverse;
+    }
+    catch (err) {
+        pass = inverse;
+        failureReasons.push(`SQL statement does not conform to the provided ${databaseType} database syntax.`);
+    }
+    if (whiteTableList) {
+        opt.type = 'table';
+        try {
+            sqlParser.whiteListCheck(outputString, whiteTableList, opt);
         }
-        return { assertion, assertResult: mainAssertResult, index: i };
-    })
-        .flat();
-    await async_1.default.forEachOfLimit(asserts, ASSERTIONS_MAX_CONCURRENCY, async ({ assertion, assertResult, index }) => {
-        if (assertion.type.startsWith('select-')) {
-            // Select-type assertions are handled separately because they depend on multiple outputs.
-            return;
+        catch (err) {
+            pass = inverse;
+            const error = err;
+            failureReasons.push(`SQL validation failed: ${error.message}.`);
         }
-        const result = await runAssertion({
-            prompt,
-            provider,
-            assertion,
-            test,
-            output,
-            latencyMs,
-            logProbs,
-            cost,
-        });
-        assertResult.addResult({
-            index,
-            result,
-            metric: assertion.metric,
-            weight: assertion.weight,
-        });
-    });
-    subAssertResults.forEach((subAssertResult) => {
-        const result = subAssertResult.testResult();
-        const { index, assertionSet: { metric, weight }, } = subAssertResult.parentAssertionSet;
-        mainAssertResult.addResult({
-            index,
-            result,
-            metric,
-            weight,
-        });
-    });
-    return mainAssertResult.testResult();
+    }
+    if (whiteColumnList) {
+        opt.type = 'column';
+        try {
+            sqlParser.whiteListCheck(outputString, whiteColumnList, opt);
+        }
+        catch (err) {
+            pass = inverse;
+            const error = err;
+            failureReasons.push(`SQL validation failed: ${error.message}.`);
+        }
+    }
+    if (inverse && pass === false && failureReasons.length === 0) {
+        failureReasons.push('The output SQL statement is valid');
+    }
+    return {
+        pass,
+        score: pass ? 1 : 0,
+        reason: pass ? 'Assertion passed' : failureReasons.join(' '),
+        assertion,
+    };
 }
-exports.runAssertions = runAssertions;
+exports.isSql = isSql;
 async function runAssertion({ prompt, provider, assertion, test, output, latencyMs, logProbs, cost, }) {
     let pass = false;
     let score = 0.0;
@@ -441,10 +438,10 @@ async function runAssertion({ prompt, provider, assertion, test, output, latency
     }
     if (baseType === 'contains-json') {
         let errorMessage = 'Expected output to contain valid JSON';
-        const jsonOutputs = containsJSON(outputString);
-        for (const jsonMatch of jsonOutputs) {
-            pass = jsonMatch !== inverse;
-            if (pass && renderedValue) {
+        const jsonObjects = (0, util_1.extractJsonObjects)(outputString);
+        pass = inverse ? jsonObjects.length === 0 : jsonObjects.length > 0;
+        for (const jsonObject of jsonObjects) {
+            if (renderedValue) {
                 let validate;
                 if (typeof renderedValue === 'string') {
                     if (renderedValue.startsWith('file://')) {
@@ -465,7 +462,7 @@ async function runAssertion({ prompt, provider, assertion, test, output, latency
                 else {
                     throw new Error('contains-json assertion must have a string or object value');
                 }
-                pass = validate(jsonMatch);
+                pass = validate(jsonObject);
                 if (pass) {
                     break;
                 }
@@ -996,97 +993,71 @@ ${isMultiline
     throw new Error('Unknown assertion type: ' + assertion.type);
 }
 exports.runAssertion = runAssertion;
-function containsJSON(str) {
-    // This will extract all json objects from a string
-    const jsonObjects = [];
-    let openBracket = str.indexOf('{');
-    let closeBracket = str.indexOf('}', openBracket);
-    // Iterate over the string until we find a valid JSON-like pattern
-    // Iterate over all trailing } until the contents parse as json
-    while (openBracket !== -1) {
-        const jsonStr = str.slice(openBracket, closeBracket + 1);
-        try {
-            jsonObjects.push(JSON.parse(jsonStr));
-            // This is a valid JSON object, so start looking for
-            // an opening bracket after the last closing bracket
-            openBracket = str.indexOf('{', closeBracket + 1);
-            closeBracket = str.indexOf('}', openBracket);
-        }
-        catch (err) {
-            // Not a valid object, move on to the next closing bracket
-            closeBracket = str.indexOf('}', closeBracket + 1);
-            while (closeBracket === -1) {
-                // No closing brackets made a valid json object, so
-                // start looking with the next opening bracket
-                openBracket = str.indexOf('{', openBracket + 1);
-                closeBracket = str.indexOf('}', openBracket);
-            }
-        }
-    }
-    return jsonObjects;
-}
-async function isSql(outputString, renderedValue, inverse, assertion) {
-    let pass = false;
-    let parsedSql;
-    let databaseType = 'MySQL';
-    let whiteTableList;
-    let whiteColumnList;
-    if (renderedValue && typeof renderedValue === 'object') {
-        const value = renderedValue;
-        databaseType = value.database || 'MySQL';
-        whiteTableList = value.allowedTables;
-        whiteColumnList = value.allowedColumns;
-    }
-    if (renderedValue && typeof renderedValue !== 'object') {
-        throw new Error('is-sql assertion must have a object value.');
+async function runAssertions({ prompt, provider, test, output, latencyMs, logProbs, cost, }) {
+    if (!test.assert || test.assert.length < 1) {
+        return AssertionsResult_1.AssertionsResult.noAssertsResult();
     }
-    const { Parser: SqlParser } = await Promise.resolve().then(() => __importStar(require('node-sql-parser'))).catch(() => {
-        throw new Error('node-sql-parser is not installed. Please install it first');
+    const mainAssertResult = new AssertionsResult_1.AssertionsResult({
+        threshold: test.threshold,
     });
-    const sqlParser = new SqlParser();
-    const opt = { database: databaseType };
-    const failureReasons = [];
-    try {
-        parsedSql = sqlParser.astify(outputString, opt);
-        pass = !inverse;
-    }
-    catch (err) {
-        pass = inverse;
-        failureReasons.push(`SQL statement does not conform to the provided ${databaseType} database syntax.`);
-    }
-    if (whiteTableList) {
-        opt.type = 'table';
-        try {
-            sqlParser.whiteListCheck(outputString, whiteTableList, opt);
-        }
-        catch (err) {
-            pass = inverse;
-            const error = err;
-            failureReasons.push(`SQL validation failed: ${error.message}.`);
-        }
-    }
-    if (whiteColumnList) {
-        opt.type = 'column';
-        try {
-            sqlParser.whiteListCheck(outputString, whiteColumnList, opt);
+    const subAssertResults = [];
+    const asserts = test.assert
+        .map((assertion, i) => {
+        if (assertion.type === 'assert-set') {
+            const subAssertResult = new AssertionsResult_1.AssertionsResult({
+                threshold: assertion.threshold,
+                parentAssertionSet: {
+                    assertionSet: assertion,
+                    index: i,
+                },
+            });
+            subAssertResults.push(subAssertResult);
+            return assertion.assert.map((subAssert, j) => {
+                return {
+                    assertion: subAssert,
+                    assertResult: subAssertResult,
+                    index: j,
+                };
+            });
         }
-        catch (err) {
-            pass = inverse;
-            const error = err;
-            failureReasons.push(`SQL validation failed: ${error.message}.`);
+        return { assertion, assertResult: mainAssertResult, index: i };
+    })
+        .flat();
+    await async_1.default.forEachOfLimit(asserts, ASSERTIONS_MAX_CONCURRENCY, async ({ assertion, assertResult, index }) => {
+        if (assertion.type.startsWith('select-')) {
+            // Select-type assertions are handled separately because they depend on multiple outputs.
+            return;
         }
-    }
-    if (inverse && pass === false && failureReasons.length === 0) {
-        failureReasons.push('The output SQL statement is valid');
-    }
-    return {
-        pass,
-        score: pass ? 1 : 0,
-        reason: pass ? 'Assertion passed' : failureReasons.join(' '),
-        assertion,
-    };
+        const result = await runAssertion({
+            prompt,
+            provider,
+            assertion,
+            test,
+            output,
+            latencyMs,
+            logProbs,
+            cost,
+        });
+        assertResult.addResult({
+            index,
+            result,
+            metric: assertion.metric,
+            weight: assertion.weight,
+        });
+    });
+    subAssertResults.forEach((subAssertResult) => {
+        const result = subAssertResult.testResult();
+        const { index, assertionSet: { metric, weight }, } = subAssertResult.parentAssertionSet;
+        mainAssertResult.addResult({
+            index,
+            result,
+            metric,
+            weight,
+        });
+    });
+    return mainAssertResult.testResult();
 }
-exports.isSql = isSql;
+exports.runAssertions = runAssertions;
 async function runCompareAssertion(test, assertion, outputs) {
     (0, tiny_invariant_1.default)(typeof assertion.value === 'string', 'select-best must have a string value');
     test.options = test.options || {};
@@ -1114,6 +1085,8 @@ async function readAssertions(filePath) {
 exports.readAssertions = readAssertions;
 // These exports are used by the node.js package (index.ts)
 exports.default = {
+    runAssertion,
+    runAssertions,
     matchesSimilarity: matchers_1.matchesSimilarity,
     matchesClassification: matchers_1.matchesClassification,
     matchesLlmRubric: matchers_1.matchesLlmRubric,