langium-ai-tools 0.0.1

package/README.md

@@ -0,0 +1,83 @@
+# Langium AI Tools
+
+## Overview
+
+This project provides core tools that make it easier to build AI applications for Langium DSLs. These core tools help to solve the following problems around building AI applications by making it easier to:
+
+- Determine which models work well for your DSL
+- Evaluate which changes to your tooling actually improve your generation results
+- How to process DSL documents in a way that makes sense for your DSL & target application
+
+To solve these problems this package provides:
+
+- Splitting Support: Using your DSL's parser to make it easier to pre-process documents before ingest (such as into a vector DB)
+- Training & Evaluation Support: Assess the output of your model + RAG + whatever else you have in your stack with regards to a structured input/output evaluation phase.
+- Constraint Support: Synthesize BNF-style grammars from your Langium grammar, which can be used to control the token output from an LLM to conform to your DSL's expected structure (this feature has been added directly into the **langium-cli** itself, as it has wider general applications).
+
+What's also important is what is not provided:
+- *We don't choose your model for you.* We believe this is your choice, and we don't want to presume we know best or lock you in. All we assume is that you have a model (or stack) that we can use. For tooling that leverages models directly, we'll be providing a separate package under Langium AI that will be separate from the core here.
+- *We don't choose your stack for you.* There are many excellent choices for hosting providers, databases, caches, and other supporting services (local & remote). There's so many, and they change so often, that we decided it was best to not assume what works here, and rather support preparing information for whatever stack you choose.
+
+LLMs (and transformers in general), are evolving quite rapidly. With this approach, these tools help you build your own specific approach, whilst letting you keep up with the latest and greatest in model developments.
+
+## Installation
+
+You can install Langium AI Tools by running:
+
+```sh
+npm i --save langium-ai-tools
+```
+
+## Usage
+
+### Splitting
+
+Langium AI Tools presents various supporting behaviors for splitting.
+
+The simplest approach is to, of course, not split at all. For smaller DSL programs this may be perfectly viable, but in all likelihood you're reading this to handle medium to large programs -- or a large quantity of smaller programs with overlapping constructs.
+
+In most cases you can split by specific AST nodes. This will map directly to those types that are generated by your Langium grammar rules, and makes it easy to mark how you want to delineate.
+
+### Evaluation
+
+Regardless of how you've sourced your model, you'll need a metric for determining the quality of your output.
+
+For Langium DSLs, we provide an series of *evaluator* utilities to help in assessing the correctness of DSL output.
+
+It's important to point out that evaluations are *not* tests, instead this is more similar to [OpenAI's evals framework](https://github.com/openai/evals). The idea is that we're grading or scoring outputs with regards to an expected output from a known input. This is a simple but effective approach to determining if your model is generally doing what you expect it to in a structured way, and *not* doing something else as well.
+
+Take the following evaluator for example. Let's assume you have [Ollama](https://ollama.com/) running locally, and the [ollama-js](https://github.com/ollama/ollama-js) package installed. From a given base model you can define evaluatiosn like so.
+
+```ts
+import { Evaluator, EvaluatorScore } from 'langium-ai-tools/evaluator';
+import ollama from 'ollama';
+
+// get your language's services
+const services = createMyDSLServices(EmptyFileSystem).MyDSL;
+
+// define an evaluator using your language's services
+// this effectively uses your existing parser & validations to 'grade' the response
+const evaluator = new LangiumEvaluator(services);
+
+// make some prompt
+const response = await ollama.chat({
+    'llama3.2',
+    [{
+        role: 'user',
+        content: 'Write me a hello world program written in MyDSL.'
+    }]
+});
+
+const es: EvaluatorScore = evaluator.evaluate(response.message.content);
+
+// print out your score!
+console.log(es);
+```
+
+You can also define custom evaluators that are more tuned to the needs of your DSL. This could be handling diagnostics in a very specific fashion, extracting code out of the response itself to check, using an evaluation model to grade the response, or using a combination of techniques to get a more accurate score for your model's output.
+
+In general we stick to focusing on what Langium can do to help with evaluation, but leave the opportunity open for you to extend, supplement, or modify evaluation logic as you see fit.
+
+## Contributing
+
+If you want to help feel free to open an issue or a PR. As a general note we're open to accept changes that focus on improving how we can support AI application development for Langium DSLs. But we don't want to provide explicit bindings to actual services/providers at this time, such as LLamaIndex, Ollama, LangChain, or others. Similarly this package doesn't provide direct bindings for AI providers such as OpenAI and Anthropic here. Instead these changes will go into a separate package under Langium AI that is intended for this purpose.

package/dist/evaluator/chart.d.ts

@@ -0,0 +1,32 @@
+/**
+ * Generates & exports an HTML radar chart report using plotly JS
+ */
+import { EvaluatorResult, EvaluatorResultData } from "./evaluator.js";
+/**
+ * Generates an HTML radar chart from the provided data
+ * @param evalResults Evaluator results to chart
+ * @param dest Output file to write the chart to
+ * @param rFunc polar r function, used to extract the r values from the data
+ * @param theta theta values, i.e. the property names to use for the radar chart
+ */
+export declare function generateRadarChart<T extends EvaluatorResultData>(chartName: string, evalResults: EvaluatorResult[], dest: string, rFunc: (d: T, metadata: Record<string, unknown>) => Record<string, unknown>, preprocess?: (arr: EvaluatorResult[]) => EvaluatorResult[]): void;
+export declare function generateHistogram<T extends EvaluatorResultData>(chartName: string, evalResults: EvaluatorResult[], dest: string, dataFunc: (d: T, metadata: Record<string, unknown>) => Record<string, unknown>, preprocess?: (arr: EvaluatorResult[]) => EvaluatorResult[]): void;
+/**
+ * Normalizes all numeric data entries in results (while also retaining non-numeric entries)
+ */
+export declare function normalizeData(data: EvaluatorResult[]): EvaluatorResult[];
+/**
+ * Generates a historical chart from the provided data, showing runners along the X, and their performance over time along the X axis
+ * @param chartName
+ * @param folder
+ * @param dest
+ * @param dataFunc
+ * @param options
+ */
+export declare function generateHistoricalChart<T extends EvaluatorResultData>(chartName: string, folder: string, dest: string, dataFunc: (d: T, metadata: Record<string, unknown>) => number, options?: {
+    preprocess?: (arr: EvaluatorResult[]) => EvaluatorResult[];
+    filter?: (r: EvaluatorResult) => boolean;
+    take?: number;
+    chartType?: string;
+}): void;
+//# sourceMappingURL=chart.d.ts.map

package/dist/evaluator/chart.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"chart.d.ts","sourceRoot":"","sources":["../../src/evaluator/chart.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH,OAAO,EAAE,eAAe,EAAE,mBAAmB,EAAoC,MAAM,gBAAgB,CAAC;AAIxG;;;;;;GAMG;AACH,wBAAgB,kBAAkB,CAAC,CAAC,SAAS,mBAAmB,EAC5D,SAAS,EAAE,MAAM,EACjB,WAAW,EAAE,eAAe,EAAE,EAC9B,IAAI,EAAE,MAAM,EACZ,KAAK,EAAE,CAAC,CAAC,EAAE,CAAC,EAAE,QAAQ,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,KAAK,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,EAC3E,UAAU,CAAC,EAAE,CAAC,GAAG,EAAE,eAAe,EAAE,KAAK,eAAe,EAAE,GAC3D,IAAI,CAsDN;AAED,wBAAgB,iBAAiB,CAAC,CAAC,SAAS,mBAAmB,EAC3D,SAAS,EAAE,MAAM,EACjB,WAAW,EAAE,eAAe,EAAE,EAC9B,IAAI,EAAE,MAAM,EACZ,QAAQ,EAAE,CAAC,CAAC,EAAE,CAAC,EAAE,QAAQ,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,KAAK,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,EAC9E,UAAU,CAAC,EAAE,CAAC,GAAG,EAAE,eAAe,EAAE,KAAK,eAAe,EAAE,QAgD7D;AAID;;GAEG;AACH,wBAAgB,aAAa,CAAC,IAAI,EAAE,eAAe,EAAE,GAAG,eAAe,EAAE,CA2BxE;AAED;;;;;;;GAOG;AACH,wBAAgB,uBAAuB,CAAC,CAAC,SAAS,mBAAmB,EACjE,SAAS,EAAE,MAAM,EACjB,MAAM,EAAE,MAAM,EACd,IAAI,EAAE,MAAM,EACZ,QAAQ,EAAE,CAAC,CAAC,EAAE,CAAC,EAAE,QAAQ,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,KAAK,MAAM,EAC7D,OAAO,CAAC,EAAE;IACN,UAAU,CAAC,EAAE,CAAC,GAAG,EAAE,eAAe,EAAE,KAAK,eAAe,EAAE,CAAC;IAC3D,MAAM,CAAC,EAAE,CAAC,CAAC,EAAE,eAAe,KAAK,OAAO,CAAC;IACzC,IAAI,CAAC,EAAE,MAAM,CAAC;IACd,SAAS,CAAC,EAAE,MAAM,CAAA;CACrB,QA4FJ"}

package/dist/evaluator/chart.js

@@ -0,0 +1,218 @@
+/**
+ * Generates & exports an HTML radar chart report using plotly JS
+ */
+import { averageAcrossRunners, loadReport } from "./evaluator.js";
+import { writeFileSync, readdirSync } from 'fs';
+import * as path from 'path';
+/**
+ * Generates an HTML radar chart from the provided data
+ * @param evalResults Evaluator results to chart
+ * @param dest Output file to write the chart to
+ * @param rFunc polar r function, used to extract the r values from the data
+ * @param theta theta values, i.e. the property names to use for the radar chart
+ */
+export function generateRadarChart(chartName, evalResults, dest, rFunc, preprocess) {
+    // process results first to average out data (either using the user supplied function, or defaulting to average across runners)
+    const processedResults = preprocess ? preprocess(evalResults) : averageAcrossRunners(evalResults);
+    const data = processedResults.map((result) => {
+        const resultData = result.data;
+        const rfuncResult = rFunc(resultData, result.metadata);
+        const theta = Object.keys(rfuncResult);
+        const r = Object.values(rfuncResult);
+        return {
+            type: 'scatterpolar',
+            r,
+            theta,
+            fill: 'toself',
+            name: result.name
+        };
+    });
+    const layout = {
+        title: chartName,
+        name: chartName,
+        polar: {
+            radialaxis: {
+                visible: true,
+                range: [0, 1]
+            }
+        },
+        showlegend: true,
+        width: 1000,
+        height: 800
+    };
+    const html = `
+<!DOCTYPE html>
+<html>
+    <head>
+    <title>${chartName}</title>
+    <script src="https://cdn.plot.ly/plotly-2.35.2.min.js" charset="utf-8"></script>
+    </head>
+    <body>
+    <div id="langium-ai-chart" style="width:1000px;height:1000px;margin:8px auto;"></div>
+    <script>
+        data = ${JSON.stringify(data)};
+        layout = ${JSON.stringify(layout)};
+        Plotly.newPlot("langium-ai-chart", data, layout);
+    </script>
+    </body>
+</html>
+        `;
+    writeFileSync(dest, html);
+    console.log(`Radar chart report written to: ${dest}`);
+}
+export function generateHistogram(chartName, evalResults, dest, dataFunc, preprocess) {
+    // process results first to average out data (either using the user supplied function, or defaulting to average across runners)
+    const processedResults = preprocess ? preprocess(evalResults) : averageAcrossRunners(evalResults);
+    const data = processedResults.map((result) => {
+        const data = result.data;
+        const dd = dataFunc(data, result.metadata);
+        const yLabels = Object.keys(dd);
+        const xData = Object.values(dd);
+        return {
+            type: 'bar',
+            x: xData,
+            y: yLabels,
+            orientation: 'h',
+            name: result.name
+        };
+    });
+    const layout = {
+        title: chartName,
+        barmode: 'group',
+        showlegend: true,
+        width: 1000,
+        height: 800
+    };
+    const html = `
+<!DOCTYPE html>
+<html>
+    <head>
+    <title>${chartName}</title>
+    <script src="https://cdn.plot.ly/plotly-2.35.2.min.js" charset="utf-8"></script>
+    </head>
+    <body>
+    <div id="langium-ai-chart" style="width:1000px;height:1000px;margin:8px auto;"></div>
+    <script>
+        data = ${JSON.stringify(data)};
+        layout = ${JSON.stringify(layout)};
+        Plotly.newPlot("langium-ai-chart", data, layout);
+    </script>
+    </body>
+</html>
+        `;
+    writeFileSync(dest, html);
+    console.log(`Histogram report written to: ${dest}`);
+}
+/**
+ * Normalizes all numeric data entries in results (while also retaining non-numeric entries)
+ */
+export function normalizeData(data) {
+    const maxValues = new Map();
+    for (const result of data) {
+        const d = result.data;
+        for (const [key, value] of Object.entries(d)) {
+            if (typeof value !== 'number') {
+                continue;
+            }
+            const existingMax = maxValues.get(key) ?? 0;
+            if (value > existingMax) {
+                maxValues.set(key, value);
+            }
+        }
+    }
+    for (const result of data) {
+        const d = result.data;
+        for (const [key, value] of Object.entries(d)) {
+            if (typeof value === 'number') {
+                const max = maxValues.get(key) ?? 1;
+                d[key] = value / max;
+            }
+        }
+    }
+    return data;
+}
+/**
+ * Generates a historical chart from the provided data, showing runners along the X, and their performance over time along the X axis
+ * @param chartName
+ * @param folder
+ * @param dest
+ * @param dataFunc
+ * @param options
+ */
+export function generateHistoricalChart(chartName, folder, dest, dataFunc, options) {
+    // generate a historical chart by calculating the average for runners in all previous reports, and organizing them in ascending date order
+    let files = readdirSync(folder).filter(f => f.endsWith('.json'));
+    // array of results, where each array of results is presumed to be a stream of results from a collection of historical runs
+    const runnerResultsMap = new Map();
+    // take the most recent files if take is set
+    if (options?.take) {
+        files = files.sort().slice(0, options.take);
+    }
+    for (const file of files) {
+        // retrieve results from this file
+        const report = loadReport(path.join(folder, file));
+        const results = report.results;
+        const date = report.date;
+        console.log(`Processing historical results from: ${date}`);
+        // process results first
+        let processedResults = options?.preprocess ? options.preprocess(results) : averageAcrossRunners(results);
+        // normalize
+        processedResults = normalizeData(processedResults);
+        // add to the map based by runner name
+        for (const result of processedResults) {
+            if (options?.filter && !options.filter(result)) {
+                // skip
+                continue;
+            }
+            const name = result.metadata.runner;
+            const existingResults = runnerResultsMap.get(name) ?? [];
+            const rc = {
+                ...result
+            };
+            rc.metadata.date = new Date(date).toISOString();
+            existingResults.push(result);
+            runnerResultsMap.set(name, existingResults);
+        }
+    }
+    const allData = [];
+    // organize by date in ascending order
+    for (let [name, results] of runnerResultsMap) {
+        results.sort((a, b) => {
+            return new Date(a.metadata.date).getTime() - new Date(b.metadata.date).getTime();
+        });
+        const runners = results.map(r => r.metadata.runner);
+        const data = results.map(r => dataFunc(r.data, r.metadata)).sort();
+        allData.push({
+            type: options?.chartType ? options.chartType : 'scatter',
+            x: runners,
+            y: data,
+            name
+        });
+    }
+    const layout = {
+        title: chartName,
+        showlegend: true,
+        width: 1000,
+        height: 1000
+    };
+    const html = `
+<!DOCTYPE html>
+<html>
+    <head>
+    <title>${chartName}</title>
+    <script src="https://cdn.plot.ly/plotly-2.35.2.min.js" charset="utf-8"></script>
+    </head>
+    <body>
+    <div id="langium-ai-chart" style="width:1000px;height:1000px;margin:8px auto;"></div>
+    <script>
+        data = ${JSON.stringify(allData)};
+        layout = ${JSON.stringify(layout)};
+        Plotly.newPlot("langium-ai-chart", data, layout);
+    </script>
+    </body>
+</html>
+        `;
+    writeFileSync(dest, html);
+    console.log(`Historical report written to: ${dest}`);
+}
+//# sourceMappingURL=chart.js.map

package/dist/evaluator/chart.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"chart.js","sourceRoot":"","sources":["../../src/evaluator/chart.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH,OAAO,EAAwC,oBAAoB,EAAE,UAAU,EAAE,MAAM,gBAAgB,CAAC;AACxG,OAAO,EAAE,aAAa,EAAE,WAAW,EAAgB,MAAM,IAAI,CAAC;AAC9D,OAAO,KAAK,IAAI,MAAM,MAAM,CAAC;AAE7B;;;;;;GAMG;AACH,MAAM,UAAU,kBAAkB,CAC9B,SAAiB,EACjB,WAA8B,EAC9B,IAAY,EACZ,KAA2E,EAC3E,UAA0D;IAG1D,+HAA+H;IAC/H,MAAM,gBAAgB,GAAG,UAAU,CAAC,CAAC,CAAC,UAAU,CAAC,WAAW,CAAC,CAAC,CAAC,CAAC,oBAAoB,CAAC,WAAW,CAAC,CAAC;IAElG,MAAM,IAAI,GAAG,gBAAgB,CAAC,GAAG,CAAC,CAAC,MAAM,EAAE,EAAE;QACzC,MAAM,UAAU,GAAG,MAAM,CAAC,IAAS,CAAC;QACpC,MAAM,WAAW,GAAG,KAAK,CAAC,UAAU,EAAE,MAAM,CAAC,QAAQ,CAAC,CAAC;QACvD,MAAM,KAAK,GAAG,MAAM,CAAC,IAAI,CAAC,WAAW,CAAC,CAAC;QACvC,MAAM,CAAC,GAAG,MAAM,CAAC,MAAM,CAAC,WAAW,CAAC,CAAC;QAErC,OAAO;YACH,IAAI,EAAE,cAAc;YACpB,CAAC;YACD,KAAK;YACL,IAAI,EAAE,QAAQ;YACd,IAAI,EAAE,MAAM,CAAC,IAAI;SACpB,CAAC;IACN,CAAC,CAAC,CAAC;IAEH,MAAM,MAAM,GAAG;QACX,KAAK,EAAE,SAAS;QAChB,IAAI,EAAE,SAAS;QACf,KAAK,EAAE;YACH,UAAU,EAAE;gBACR,OAAO,EAAE,IAAI;gBACb,KAAK,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC;aAChB;SACJ;QACD,UAAU,EAAE,IAAI;QAChB,KAAK,EAAE,IAAI;QACX,MAAM,EAAE,GAAG;KACd,CAAC;IAEF,MAAM,IAAI,GAAG;;;;aAIJ,SAAS;;;;;;iBAML,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC;mBAClB,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC;;;;;SAKhC,CAAC;IAEN,aAAa,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC;IAC1B,OAAO,CAAC,GAAG,CAAC,kCAAkC,IAAI,EAAE,CAAC,CAAC;AAC1D,CAAC;AAED,MAAM,UAAU,iBAAiB,CAC7B,SAAiB,EACjB,WAA8B,EAC9B,IAAY,EACZ,QAA8E,EAC9E,UAA0D;IAG1D,+HAA+H;IAC/H,MAAM,gBAAgB,GAAG,UAAU,CAAC,CAAC,CAAC,UAAU,CAAC,WAAW,CAAC,CAAC,CAAC,CAAC,oBAAoB,CAAC,WAAW,CAAC,CAAC;IAElG,MAAM,IAAI,GAAG,gBAAgB,CAAC,GAAG,CAAC,CAAC,MAAM,EAAE,EAAE;QACzC,MAAM,IAAI,GAAG,MAAM,CAAC,IAAS,CAAC;QAC9B,MAAM,EAAE,GAAG,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC,QAAQ,CAAC,CAAC;QAC3C,MAAM,OAAO,GAAG,MAAM,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;QAChC,MAAM,KAAK,GAAG,MAAM,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC;QAChC,OAAO;YACH,IAAI,EAAE,KAAK;YACX,CAAC,EAAE,KAAK;YACR,CAAC,EAAE,OAAO;YACV,WAAW,EAAE,GAAG;YAChB,IAAI,EAAE,MAAM,CAAC,IAAI;SACpB,CAAC;IACN,CAAC,CAAC,CAAC;IAEH,MAAM,MAAM,GAAG;QACX,KAAK,EAAE,SAAS;QAChB,OAAO,EAAE,OAAO;QAChB,UAAU,EAAE,IAAI;QAChB,KAAK,EAAE,IAAI;QACX,MAAM,EAAE,GAAG;KACd,CAAC;IAEF,MAAM,IAAI,GAAG;;;;aAIJ,SAAS;;;;;;iBAML,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC;mBAClB,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC;;;;;SAKhC,CAAC;IAEN,aAAa,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC;IAC1B,OAAO,CAAC,GAAG,CAAC,gCAAgC,IAAI,EAAE,CAAC,CAAC;AACxD,CAAC;AAID;;GAEG;AACH,MAAM,UAAU,aAAa,CAAC,IAAuB;IACjD,MAAM,SAAS,GAAG,IAAI,GAAG,EAAkB,CAAC;IAE5C,KAAK,MAAM,MAAM,IAAI,IAAI,EAAE,CAAC;QACxB,MAAM,CAAC,GAAG,MAAM,CAAC,IAA2B,CAAC;QAC7C,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC;YAC3C,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;gBAC5B,SAAS;YACb,CAAC;YACD,MAAM,WAAW,GAAG,SAAS,CAAC,GAAG,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;YAC5C,IAAI,KAAK,GAAG,WAAW,EAAE,CAAC;gBACtB,SAAS,CAAC,GAAG,CAAC,GAAG,EAAE,KAAK,CAAC,CAAC;YAC9B,CAAC;QACL,CAAC;IACL,CAAC;IAED,KAAK,MAAM,MAAM,IAAI,IAAI,EAAE,CAAC;QACxB,MAAM,CAAC,GAAG,MAAM,CAAC,IAA2B,CAAC;QAC7C,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC;YAC3C,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;gBAC5B,MAAM,GAAG,GAAG,SAAS,CAAC,GAAG,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;gBACpC,CAAC,CAAC,GAAG,CAAC,GAAG,KAAK,GAAG,GAAG,CAAC;YACzB,CAAC;QACL,CAAC;IACL,CAAC;IAED,OAAO,IAAI,CAAC;AAChB,CAAC;AAED;;;;;;;GAOG;AACH,MAAM,UAAU,uBAAuB,CACnC,SAAiB,EACjB,MAAc,EACd,IAAY,EACZ,QAA6D,EAC7D,OAKC;IAED,0IAA0I;IAC1I,IAAI,KAAK,GAAG,WAAW,CAAC,MAAM,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC,CAAC;IAEjE,2HAA2H;IAC3H,MAAM,gBAAgB,GAAuC,IAAI,GAAG,EAAE,CAAC;IAEvE,4CAA4C;IAC5C,IAAI,OAAO,EAAE,IAAI,EAAE,CAAC;QAChB,KAAK,GAAG,KAAK,CAAC,IAAI,EAAE,CAAC,KAAK,CAAC,CAAC,EAAE,OAAO,CAAC,IAAI,CAAC,CAAC;IAChD,CAAC;IAED,KAAK,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC;QACvB,kCAAkC;QAClC,MAAM,MAAM,GAAG,UAAU,CAAC,IAAI,CAAC,IAAI,CAAC,MAAM,EAAE,IAAI,CAAC,CAAC,CAAC;QACnD,MAAM,OAAO,GAAG,MAAM,CAAC,OAAO,CAAC;QAC/B,MAAM,IAAI,GAAW,MAAM,CAAC,IAAI,CAAC;QACjC,OAAO,CAAC,GAAG,CAAC,uCAAuC,IAAI,EAAE,CAAC,CAAC;QAE3D,wBAAwB;QACxB,IAAI,gBAAgB,GAAG,OAAO,EAAE,UAAU,CAAC,CAAC,CAAC,OAAO,CAAC,UAAU,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,oBAAoB,CAAC,OAAO,CAAC,CAAC;QACzG,YAAY;QACZ,gBAAgB,GAAG,aAAa,CAAC,gBAAgB,CAAC,CAAC;QAEnD,sCAAsC;QACtC,KAAK,MAAM,MAAM,IAAI,gBAAgB,EAAE,CAAC;YACpC,IAAI,OAAO,EAAE,MAAM,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,MAAM,CAAC,EAAE,CAAC;gBAC7C,OAAO;gBACP,SAAS;YACb,CAAC;YAED,MAAM,IAAI,GAAG,MAAM,CAAC,QAAQ,CAAC,MAAM,CAAC;YACpC,MAAM,eAAe,GAAG,gBAAgB,CAAC,GAAG,CAAC,IAAI,CAAC,IAAI,EAAE,CAAC;YAEzD,MAAM,EAAE,GAAG;gBACP,GAAG,MAAM;aACZ,CAAC;YACF,EAAE,CAAC,QAAQ,CAAC,IAAI,GAAG,IAAI,IAAI,CAAC,IAAI,CAAC,CAAC,WAAW,EAAE,CAAC;YAEhD,eAAe,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;YAC7B,gBAAgB,CAAC,GAAG,CAAC,IAAI,EAAE,eAAe,CAAC,CAAC;QAChD,CAAC;IACL,CAAC;IAED,MAAM,OAAO,GAAc,EAAE,CAAC;IAE9B,sCAAsC;IACtC,KAAK,IAAI,CAAC,IAAI,EAAE,OAAO,CAAC,IAAI,gBAAgB,EAAE,CAAC;QAC3C,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE;YAClB,OAAO,IAAI,IAAI,CAAC,CAAC,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,OAAO,EAAE,GAAG,IAAI,IAAI,CAAC,CAAC,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,OAAO,EAAE,CAAC;QACrF,CAAC,CAAC,CAAC;QAEH,MAAM,OAAO,GAAG,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC;QACpD,MAAM,IAAI,GAAG,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAS,EAAE,CAAC,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;QAExE,OAAO,CAAC,IAAI,CAAC;YACT,IAAI,EAAE,OAAO,EAAE,SAAS,CAAC,CAAC,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC,CAAC,SAAS;YACxD,CAAC,EAAE,OAAO;YACV,CAAC,EAAE,IAAI;YACP,IAAI;SACP,CAAC,CAAC;IACP,CAAC;IAED,MAAM,MAAM,GAAG;QACX,KAAK,EAAE,SAAS;QAChB,UAAU,EAAE,IAAI;QAChB,KAAK,EAAE,IAAI;QACX,MAAM,EAAE,IAAI;KACf,CAAC;IAEF,MAAM,IAAI,GAAG;;;;aAIJ,SAAS;;;;;;iBAML,IAAI,CAAC,SAAS,CAAC,OAAO,CAAC;mBACrB,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC;;;;;SAKhC,CAAC;IAEN,aAAa,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC;IAC1B,OAAO,CAAC,GAAG,CAAC,iCAAiC,IAAI,EAAE,CAAC,CAAC;AAEzD,CAAC"}

package/dist/evaluator/edit-distance-evaluator.d.ts

@@ -0,0 +1,8 @@
+import { Evaluator, EvaluatorResult, EvaluatorResultData } from './evaluator.js';
+export interface EditDistanceEvaluatorResultData extends EvaluatorResultData {
+    edit_distance: number;
+}
+export declare class EditDistanceEvaluator extends Evaluator {
+    evaluate(response: string, expected_response: string): Promise<Partial<EvaluatorResult>>;
+}
+//# sourceMappingURL=edit-distance-evaluator.d.ts.map

package/dist/evaluator/edit-distance-evaluator.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"edit-distance-evaluator.d.ts","sourceRoot":"","sources":["../../src/evaluator/edit-distance-evaluator.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,SAAS,EAAE,eAAe,EAAE,mBAAmB,EAAE,MAAM,gBAAgB,CAAC;AAEjF,MAAM,WAAW,+BAAgC,SAAQ,mBAAmB;IACxE,aAAa,EAAE,MAAM,CAAC;CACzB;AAED,qBAAa,qBAAsB,SAAQ,SAAS;IAC1C,QAAQ,CAAC,QAAQ,EAAE,MAAM,EAAE,iBAAiB,EAAE,MAAM,GAAG,OAAO,CAAC,OAAO,CAAC,eAAe,CAAC,CAAC;CAQjG"}

package/dist/evaluator/edit-distance-evaluator.js

@@ -0,0 +1,13 @@
+import { levenshteinEditDistance } from 'levenshtein-edit-distance';
+import { Evaluator } from './evaluator.js';
+export class EditDistanceEvaluator extends Evaluator {
+    async evaluate(response, expected_response) {
+        const distance = levenshteinEditDistance(response, expected_response);
+        return {
+            data: {
+                edit_distance: distance
+            }
+        };
+    }
+}
+//# sourceMappingURL=edit-distance-evaluator.js.map

package/dist/evaluator/edit-distance-evaluator.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"edit-distance-evaluator.js","sourceRoot":"","sources":["../../src/evaluator/edit-distance-evaluator.ts"],"names":[],"mappings":"AAAA,OAAO,EAAC,uBAAuB,EAAC,MAAM,2BAA2B,CAAC;AAClE,OAAO,EAAE,SAAS,EAAwC,MAAM,gBAAgB,CAAC;AAMjF,MAAM,OAAO,qBAAsB,SAAQ,SAAS;IAChD,KAAK,CAAC,QAAQ,CAAC,QAAgB,EAAE,iBAAyB;QACtD,MAAM,QAAQ,GAAG,uBAAuB,CAAC,QAAQ,EAAE,iBAAiB,CAAC,CAAC;QACtE,OAAO;YACH,IAAI,EAAE;gBACF,aAAa,EAAE,QAAQ;aAC1B;SACJ,CAAC;IACN,CAAC;CACJ"}

package/dist/evaluator/eval-matrix.d.ts

@@ -0,0 +1,95 @@
+import { Evaluator, EvaluatorResult } from "./evaluator.js";
+/**
+ * Configuration for the evaluation matrix
+ */
+export interface EvalMatrixConfig {
+    config: {
+        /**
+         * Name of the evaluation matrix
+         */
+        name: string;
+        /**
+         * Helpful description of the evaluation matrix
+         */
+        description: string;
+        /**
+         * Where to store run history
+         */
+        history_folder: string;
+        /**
+         * The number of runs to perform for each case
+         * Note this will trigger evaluation for all registered evaluators for each run
+         */
+        num_runs: number;
+    };
+    /**
+     * Runners to evaluate
+     */
+    runners: Runner[];
+    /**
+     * Evaluators to evaluate with
+     */
+    evaluators: NamedEvaluator[];
+    /**
+     * Cases to evaluate
+     */
+    cases: Case[];
+}
+/**
+ * Evaluation matrix for running multiple runners on multiple cases with multiple evaluators
+ */
+export declare class EvalMatrix {
+    private config;
+    constructor(config: EvalMatrixConfig);
+    /**
+     * Run the evaluation matrix, getting all results back
+     */
+    run(): Promise<EvaluatorResult[]>;
+}
+/**
+ * General format for histories when prompting
+ */
+export interface Message {
+    role: 'user' | 'system' | 'assistant';
+    content: string;
+}
+/**
+ * Runner interface for running a prompt against a mode, a service, or something else that provides a response
+ */
+export interface Runner {
+    name: string;
+    runner: (prompt: string, messages: Message[]) => Promise<string>;
+}
+/**
+ * Generic evaluator interface w/ a name to identify it
+ */
+export interface NamedEvaluator {
+    name: string;
+    eval: Evaluator;
+}
+/**
+ * Case interface for defining an evaluation case
+ */
+export interface Case {
+    /**
+     * Name of the case
+     */
+    name: string;
+    /**
+     * Options Message history, used for system, user & assistant messages
+     */
+    history?: Message[];
+    /**
+     * Core prompt to run with
+     */
+    prompt: string;
+    /**
+     * Context for the prompt, used for RAG applications
+     */
+    context: string[];
+    /**
+     * Expected response
+     */
+    expected_response: string;
+}
+//# sourceMappingURL=eval-matrix.d.ts.map

package/dist/evaluator/eval-matrix.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"eval-matrix.d.ts","sourceRoot":"","sources":["../../src/evaluator/eval-matrix.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,eAAe,EAAE,MAAM,gBAAgB,CAAC;AAI5D;;GAEG;AACH,MAAM,WAAW,gBAAgB;IAC7B,MAAM,EAAE;QACJ;;WAEG;QACH,IAAI,EAAE,MAAM,CAAC;QAEb;;WAEG;QACH,WAAW,EAAE,MAAM,CAAC;QAEpB;;WAEG;QACH,cAAc,EAAE,MAAM,CAAC;QAEvB;;;WAGG;QACH,QAAQ,EAAE,MAAM,CAAC;KACpB,CAAC;IAEF;;OAEG;IACH,OAAO,EAAE,MAAM,EAAE,CAAC;IAElB;;OAEG;IACH,UAAU,EAAE,cAAc,EAAE,CAAC;IAE7B;;OAEG;IACH,KAAK,EAAE,IAAI,EAAE,CAAC;CACjB;AAED;;GAEG;AACH,qBAAa,UAAU;IACnB,OAAO,CAAC,MAAM,CAAmB;gBAErB,MAAM,EAAE,gBAAgB;IAIpC;;OAEG;IACG,GAAG,IAAI,OAAO,CAAC,eAAe,EAAE,CAAC;CA2F1C;AAED;;GAEG;AACH,MAAM,WAAW,OAAO;IACpB,IAAI,EAAE,MAAM,GAAG,QAAQ,GAAG,WAAW,CAAC;IACtC,OAAO,EAAE,MAAM,CAAC;CACnB;AAED;;GAEG;AACH,MAAM,WAAW,MAAM;IACnB,IAAI,EAAE,MAAM,CAAC;IACb,MAAM,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,QAAQ,EAAE,OAAO,EAAE,KAAK,OAAO,CAAC,MAAM,CAAC,CAAC;CACpE;AAED;;GAEG;AACH,MAAM,WAAW,cAAc;IAC3B,IAAI,EAAE,MAAM,CAAC;IACb,IAAI,EAAE,SAAS,CAAC;CACnB;AAED;;GAEG;AACH,MAAM,WAAW,IAAI;IACjB;;OAEG;IACH,IAAI,EAAE,MAAM,CAAC;IAEb;;OAEG;IACH,OAAO,CAAC,EAAE,OAAO,EAAE,CAAC;IAEpB;;OAEG;IACH,MAAM,EAAE,MAAM,CAAC;IAEf;;OAEG;IACH,OAAO,EAAE,MAAM,EAAE,CAAC;IAElB;;OAEG;IACH,iBAAiB,EAAE,MAAM,CAAC;CAC7B"}

package/dist/evaluator/eval-matrix.js

@@ -0,0 +1,87 @@
+import fs from 'fs';
+import * as path from 'path';
+/**
+ * Evaluation matrix for running multiple runners on multiple cases with multiple evaluators
+ */
+export class EvalMatrix {
+    constructor(config) {
+        this.config = config;
+    }
+    /**
+     * Run the evaluation matrix, getting all results back
+     */
+    async run() {
+        // get the current timestamp
+        const start = new Date();
+        const results = [];
+        // verify that all runners have unique names first
+        const runnerNames = this.config.runners.map(r => r.name);
+        const uniqueRunnerNames = new Set();
+        for (const name of runnerNames) {
+            if (uniqueRunnerNames.has(name)) {
+                throw new Error(`Runner names must be unique, found duplicate: ${name}`);
+            }
+            uniqueRunnerNames.add(name);
+        }
+        console.log(`Running evaluation matrix: ${this.config.config.name}`);
+        console.log(`Found ${this.config.runners.length * this.config.cases.length * this.config.evaluators.length} runner-evaluator-case combinations to handle`);
+        // run all runners
+        for (const runner of this.config.runners) {
+            console.log(`* Runner: ${runner.name}`);
+            // run all cases for this runner
+            for (const testCase of this.config.cases) {
+                console.log(`  * Case: ${testCase.name}`);
+                const runCount = this.config.config.num_runs ?? 1;
+                for (let iteration = 0; iteration < runCount; iteration++) {
+                    const runnerStartTime = new Date();
+                    const response = await runner.runner(testCase.prompt, testCase.history ?? []);
+                    const runnerEndTime = new Date();
+                    // run all evaluators on this response
+                    for (const evaluator of this.config.evaluators) {
+                        console.log(`    * Evaluator: ${evaluator.name} (run ${iteration + 1})`);
+                        const result = await evaluator.eval.evaluate(response, testCase.expected_response);
+                        if (!result.name) {
+                            result.name = `${runner.name} - ${testCase.name} - ${evaluator.name}`;
+                        }
+                        // add runtime there too, so we have access to it
+                        result.data._runtime = (runnerEndTime.getTime() - runnerStartTime.getTime()) / 1000.0; // in seconds
+                        result.metadata = {
+                            runner: runner.name,
+                            evaluator: evaluator.name,
+                            testCase: { ...testCase },
+                            actual_response: response,
+                            duration: (runnerEndTime.getTime() - runnerStartTime.getTime()) / 1000.0, // in seconds
+                            run_count: iteration + 1
+                        };
+                        results.push(result);
+                    }
+                }
+            }
+        }
+        // check if the folder exists first
+        if (!fs.existsSync(this.config.config.history_folder)) {
+            fs.mkdirSync(this.config.config.history_folder);
+        }
+        const dateStr = new Date().toISOString();
+        const sanitizedDateStr = dateStr.replace(/:/g, '-').replace(/\./g, '-');
+        let fileName = `${sanitizedDateStr}-${this.config.config.name.toLowerCase().replace(/\s+/g, '-')}.json`;
+        // escape any slashes too
+        fileName = fileName.replace(/\//g, '-');
+        console.log(`Writing results to file: ${path.join(this.config.config.history_folder, fileName)}`);
+        // run time in seconds
+        const runTime = (new Date().getTime() - start.getTime()) / 1000;
+        console.log(`Evaluation matrix completed in ${runTime} seconds (${runTime / 60} minutes)`);
+        // prepare & write results to file
+        const report = {
+            config: this.config.config,
+            date: dateStr,
+            runTime: `${runTime}s`,
+            results
+        };
+        fs.writeFileSync(path.join(this.config.config.history_folder, fileName), JSON.stringify(report, null, 2));
+        // write the name of this last report into last.txt
+        fs.writeFileSync(path.join(this.config.config.history_folder, 'last.txt'), fileName);
+        return results;
+    }
+}
+//# sourceMappingURL=eval-matrix.js.map

package/dist/evaluator/eval-matrix.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"eval-matrix.js","sourceRoot":"","sources":["../../src/evaluator/eval-matrix.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,MAAM,IAAI,CAAC;AACpB,OAAO,KAAK,IAAI,MAAM,MAAM,CAAC;AA6C7B;;GAEG;AACH,MAAM,OAAO,UAAU;IAGnB,YAAY,MAAwB;QAChC,IAAI,CAAC,MAAM,GAAG,MAAM,CAAC;IACzB,CAAC;IAED;;OAEG;IACH,KAAK,CAAC,GAAG;QAEL,4BAA4B;QAC5B,MAAM,KAAK,GAAG,IAAI,IAAI,EAAE,CAAC;QAEzB,MAAM,OAAO,GAAsB,EAAE,CAAC;QAEtC,kDAAkD;QAClD,MAAM,WAAW,GAAG,IAAI,CAAC,MAAM,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC;QACzD,MAAM,iBAAiB,GAAG,IAAI,GAAG,EAAE,CAAC;QACpC,KAAK,MAAM,IAAI,IAAI,WAAW,EAAE,CAAC;YAC7B,IAAI,iBAAiB,CAAC,GAAG,CAAC,IAAI,CAAC,EAAE,CAAC;gBAC9B,MAAM,IAAI,KAAK,CAAC,iDAAiD,IAAI,EAAE,CAAC,CAAC;YAC7E,CAAC;YACD,iBAAiB,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;QAChC,CAAC;QAED,OAAO,CAAC,GAAG,CAAC,8BAA8B,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,IAAI,EAAE,CAAC,CAAC;QACrE,OAAO,CAAC,GAAG,CAAC,SAAS,IAAI,CAAC,MAAM,CAAC,OAAO,CAAC,MAAM,GAAG,IAAI,CAAC,MAAM,CAAC,KAAK,CAAC,MAAM,GAAG,IAAI,CAAC,MAAM,CAAC,UAAU,CAAC,MAAM,+CAA+C,CAAC,CAAC;QAE3J,kBAAkB;QAClB,KAAK,MAAM,MAAM,IAAI,IAAI,CAAC,MAAM,CAAC,OAAO,EAAE,CAAC;YAEvC,OAAO,CAAC,GAAG,CAAC,aAAa,MAAM,CAAC,IAAI,EAAE,CAAC,CAAC;YAExC,gCAAgC;YAChC,KAAK,MAAM,QAAQ,IAAI,IAAI,CAAC,MAAM,CAAC,KAAK,EAAE,CAAC;gBACvC,OAAO,CAAC,GAAG,CAAC,aAAa,QAAQ,CAAC,IAAI,EAAE,CAAC,CAAC;gBAE1C,MAAM,QAAQ,GAAG,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,QAAQ,IAAI,CAAC,CAAC;gBAClD,KAAK,IAAI,SAAS,GAAG,CAAC,EAAE,SAAS,GAAG,QAAQ,EAAE,SAAS,EAAE,EAAE,CAAC;oBACxD,MAAM,eAAe,GAAG,IAAI,IAAI,EAAE,CAAC;oBACnC,MAAM,QAAQ,GAAG,MAAM,MAAM,CAAC,MAAM,CAAC,QAAQ,CAAC,MAAM,EAAE,QAAQ,CAAC,OAAO,IAAI,EAAE,CAAC,CAAC;oBAC9E,MAAM,aAAa,GAAG,IAAI,IAAI,EAAE,CAAC;oBAEjC,sCAAsC;oBACtC,KAAK,MAAM,SAAS,IAAI,IAAI,CAAC,MAAM,CAAC,UAAU,EAAE,CAAC;wBAC7C,OAAO,CAAC,GAAG,CAAC,oBAAoB,SAAS,CAAC,IAAI,SAAS,SAAS,GAAG,CAAC,GAAG,CAAC,CAAC;wBACzE,MAAM,MAAM,GAAG,MAAM,SAAS,CAAC,IAAI,CAAC,QAAQ,CAAC,QAAQ,EAAE,QAAQ,CAAC,iBAAiB,CAAC,CAAC;wBACnF,IAAI,CAAC,MAAM,CAAC,IAAI,EAAE,CAAC;4BACf,MAAM,CAAC,IAAI,GAAG,GAAG,MAAM,CAAC,IAAI,MAAM,QAAQ,CAAC,IAAI,MAAM,SAAS,CAAC,IAAI,EAAE,CAAC;wBAC1E,CAAC;wBACD,iDAAiD;wBACjD,MAAM,CAAC,IAAK,CAAC,QAAQ,GAAG,CAAC,aAAa,CAAC,OAAO,EAAE,GAAG,eAAe,CAAC,OAAO,EAAE,CAAC,GAAG,MAAM,CAAC,CAAC,aAAa;wBAErG,MAAM,CAAC,QAAQ,GAAG;4BACd,MAAM,EAAE,MAAM,CAAC,IAAI;4BACnB,SAAS,EAAE,SAAS,CAAC,IAAI;4BACzB,QAAQ,EAAE,EAAE,GAAG,QAAQ,EAAE;4BACzB,eAAe,EAAE,QAAQ;4BACzB,QAAQ,EAAE,CAAC,aAAa,CAAC,OAAO,EAAE,GAAG,eAAe,CAAC,OAAO,EAAE,CAAC,GAAG,MAAM,EAAE,aAAa;4BACvF,SAAS,EAAE,SAAS,GAAG,CAAC;yBAC3B,CAAC;wBAEF,OAAO,CAAC,IAAI,CAAC,MAAyB,CAAC,CAAC;oBAC5C,CAAC;gBACL,CAAC;YACL,CAAC;QACL,CAAC;QAED,mCAAmC;QACnC,IAAI,CAAC,EAAE,CAAC,UAAU,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,cAAc,CAAC,EAAE,CAAC;YACpD,EAAE,CAAC,SAAS,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,cAAc,CAAC,CAAC;QACpD,CAAC;QAED,MAAM,OAAO,GAAG,IAAI,IAAI,EAAE,CAAC,WAAW,EAAE,CAAC;QACzC,MAAM,gBAAgB,GAAG,OAAO,CAAC,OAAO,CAAC,IAAI,EAAE,GAAG,CAAC,CAAC,OAAO,CAAC,KAAK,EAAE,GAAG,CAAC,CAAC;QACxE,IAAI,QAAQ,GAAG,GAAG,gBAAgB,IAAI,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,IAAI,CAAC,WAAW,EAAE,CAAC,OAAO,CAAC,MAAM,EAAE,GAAG,CAAC,OAAO,CAAC;QACxG,yBAAyB;QACzB,QAAQ,GAAG,QAAQ,CAAC,OAAO,CAAC,KAAK,EAAE,GAAG,CAAC,CAAC;QAExC,OAAO,CAAC,GAAG,CAAC,4BAA4B,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,cAAc,EAAE,QAAQ,CAAC,EAAE,CAAC,CAAC;QAElG,sBAAsB;QACtB,MAAM,OAAO,GAAG,CAAC,IAAI,IAAI,EAAE,CAAC,OAAO,EAAE,GAAG,KAAK,CAAC,OAAO,EAAE,CAAC,GAAG,IAAI,CAAC;QAChE,OAAO,CAAC,GAAG,CAAC,kCAAkC,OAAO,aAAa,OAAO,GAAG,EAAE,WAAW,CAAC,CAAC;QAE3F,kCAAkC;QAClC,MAAM,MAAM,GAAG;YACX,MAAM,EAAE,IAAI,CAAC,MAAM,CAAC,MAAM;YAC1B,IAAI,EAAE,OAAO;YACb,OAAO,EAAE,GAAG,OAAO,GAAG;YACtB,OAAO;SACV,CAAC;QACF,EAAE,CAAC,aAAa,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,cAAc,EAAE,QAAQ,CAAC,EAAE,IAAI,CAAC,SAAS,CAAC,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,CAAC;QAE1G,mDAAmD;QACnD,EAAE,CAAC,aAAa,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,cAAc,EAAE,UAAU,CAAC,EAAE,QAAQ,CAAC,CAAC;QAErF,OAAO,OAAO,CAAC;IACnB,CAAC;CACJ"}

package/dist/evaluator/evaluator.d.ts

@@ -0,0 +1,64 @@
+/**
+ * Baseline Validator Class
+ */
+export type EvaluatorResultData = Record<string, unknown> & {
+    _runtime?: number;
+};
+/**
+ * Evaluator result type
+ */
+export type EvaluatorResult = {
+    /**
+     * Name of this evaluation
+     */
+    name: string;
+    /**
+     * Optional metadata, can be used to store additional information
+     */
+    metadata: Record<string, any>;
+    /**
+     * Data for this evaluation
+     */
+    data: EvaluatorResultData;
+};
+/**
+ * Helper to process a set of results, averaging all runs of each runner-evaluator-case combination
+ */
+export declare function averageAcrossCases(results: EvaluatorResult[]): EvaluatorResult[];
+/**
+ * Averages all results across runners at the highest level, to get a single result for each runner
+ */
+export declare function averageAcrossRunners(results: EvaluatorResult[]): EvaluatorResult[];
+/**
+ * Report
+ */
+export interface Report {
+    config: {
+        name: string;
+        description: string;
+        history_folder: string;
+        num_runs: number;
+    };
+    date: string;
+    runTime: string;
+    results: EvaluatorResult[];
+}
+/**
+ * Loads a specific report, containing evaluator results from a file & returns it
+ */
+export declare function loadReport(file: string): Report;
+/**
+ * Attempts to load the most recent evaluator results from the given file
+ */
+export declare function loadLastResults(dir: string, take?: number): EvaluatorResult[];
+/**
+ * Evaluator class for evaluating agent responses
+ */
+export declare abstract class Evaluator {
+    /**
+     * Validate some agent response
+     */
+    abstract evaluate(response: string, expected_response: string): Promise<Partial<EvaluatorResult>>;
+}
+export declare function mergeEvaluators(...evaluators: Evaluator[]): Evaluator;
+//# sourceMappingURL=evaluator.d.ts.map

package/dist/evaluator/evaluator.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"evaluator.d.ts","sourceRoot":"","sources":["../../src/evaluator/evaluator.ts"],"names":[],"mappings":"AAAA;;GAEG;AAKH,MAAM,MAAM,mBAAmB,GAAG,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,GAAG;IACxD,QAAQ,CAAC,EAAE,MAAM,CAAC;CACrB,CAAC;AAEF;;GAEG;AACH,MAAM,MAAM,eAAe,GAAG;IAC1B;;OAEG;IACH,IAAI,EAAE,MAAM,CAAC;IAEb;;OAEG;IACH,QAAQ,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC;IAE9B;;OAEG;IACH,IAAI,EAAE,mBAAmB,CAAC;CAE7B,CAAC;AAEF;;GAEG;AACH,wBAAgB,kBAAkB,CAAC,OAAO,EAAE,eAAe,EAAE,GAAG,eAAe,EAAE,CA4ChF;AAED;;GAEG;AACH,wBAAgB,oBAAoB,CAAC,OAAO,EAAE,eAAe,EAAE,GAAG,eAAe,EAAE,CAiDlF;AAED;;GAEG;AACH,MAAM,WAAW,MAAM;IACnB,MAAM,EAAE;QACJ,IAAI,EAAE,MAAM,CAAC;QACb,WAAW,EAAE,MAAM,CAAC;QACpB,cAAc,EAAE,MAAM,CAAC;QACvB,QAAQ,EAAE,MAAM,CAAC;KACpB,CAAC;IACF,IAAI,EAAE,MAAM,CAAC;IACb,OAAO,EAAE,MAAM,CAAC;IAChB,OAAO,EAAE,eAAe,EAAE,CAAC;CAC9B;AAED;;GAEG;AACH,wBAAgB,UAAU,CAAC,IAAI,EAAE,MAAM,GAAG,MAAM,CAE/C;AAED;;GAEG;AACH,wBAAgB,eAAe,CAAC,GAAG,EAAE,MAAM,EAAE,IAAI,CAAC,EAAE,MAAM,GAAG,eAAe,EAAE,CAmC7E;AAED;;GAEG;AACH,8BAAsB,SAAS;IAC3B;;OAEG;IACH,QAAQ,CAAC,QAAQ,CAAC,QAAQ,EAAE,MAAM,EAAE,iBAAiB,EAAE,MAAM,GAAG,OAAO,CAAC,OAAO,CAAC,eAAe,CAAC,CAAC;CAEpG;AAED,wBAAgB,eAAe,CAAC,GAAG,UAAU,EAAE,SAAS,EAAE,GAAG,SAAS,CAGrE"}

package/dist/evaluator/evaluator.js

@@ -0,0 +1,162 @@
+/**
+ * Baseline Validator Class
+ */
+import { readFileSync, existsSync, readdirSync } from 'fs';
+import * as path from 'path';
+/**
+ * Helper to process a set of results, averaging all runs of each runner-evaluator-case combination
+ */
+export function averageAcrossCases(results) {
+    const mappedResults = new Map();
+    const averagedResults = [];
+    // collect like-results
+    for (const result of results) {
+        // add this result to the map (grouping by runner & case)
+        const name = result.name;
+        const existingResult = mappedResults.get(name) ?? [];
+        existingResult.push(result);
+        mappedResults.set(name, existingResult);
+    }
+    // average the results
+    for (const [_key, groupedResults] of mappedResults) {
+        const avgData = groupedResults[0].data;
+        // sum all results except the first
+        for (const result of groupedResults.slice(1)) {
+            const resultData = result.data;
+            for (const [key, value] of Object.entries(resultData)) {
+                if (typeof value === 'number') {
+                    avgData[key] = (avgData[key] ?? 0) + value;
+                }
+            }
+        }
+        // lastly, divide each entry by the number of 'groupedResults'
+        for (const [key, value] of Object.entries(avgData)) {
+            if (typeof value === 'number') {
+                avgData[key] = value / groupedResults.length;
+                // round to 2 decimal places
+                avgData[key] = Math.round(avgData[key] * 100) / 100;
+            }
+        }
+        averagedResults.push({
+            name: groupedResults[0].name,
+            metadata: groupedResults[0].metadata,
+            data: avgData
+        });
+    }
+    return averagedResults;
+}
+/**
+ * Averages all results across runners at the highest level, to get a single result for each runner
+ */
+export function averageAcrossRunners(results) {
+    // first average across runs
+    const processedResults = averageAcrossCases(results);
+    // now average across runners
+    const mappedResults = new Map();
+    const averagedResults = [];
+    // collect like-results
+    for (const result of processedResults) {
+        // add this result to the map (grouping by runner)
+        const name = result.metadata.runner;
+        const existingResult = mappedResults.get(name) ?? [];
+        existingResult.push(result);
+        mappedResults.set(name, existingResult);
+    }
+    // average the results
+    for (const [_key, groupedResults] of mappedResults) {
+        const avgData = groupedResults[0].data;
+        // sum all results except the first
+        for (const result of groupedResults.slice(1)) {
+            const resultData = result.data;
+            for (const [key, value] of Object.entries(resultData)) {
+                if (typeof value === 'number') {
+                    avgData[key] = (avgData[key] ?? 0) + value;
+                }
+            }
+        }
+        // lastly, divide each entry by the number of 'groupedResults'
+        for (const [key, value] of Object.entries(avgData)) {
+            if (typeof value === 'number') {
+                avgData[key] = value / groupedResults.length;
+                // round to 2 decimal places
+                avgData[key] = Math.round(avgData[key] * 100) / 100;
+            }
+        }
+        averagedResults.push({
+            name: groupedResults[0].metadata.runner,
+            metadata: groupedResults[0].metadata,
+            data: avgData
+        });
+    }
+    return averagedResults;
+}
+/**
+ * Loads a specific report, containing evaluator results from a file & returns it
+ */
+export function loadReport(file) {
+    return JSON.parse(readFileSync(file, 'utf-8'));
+}
+/**
+ * Attempts to load the most recent evaluator results from the given file
+ */
+export function loadLastResults(dir, take) {
+    if (!existsSync(dir)) {
+        throw new Error(`Directory does not exist: ${dir}`);
+    }
+    let files = readdirSync(dir).filter(f => f.endsWith('.json'));
+    if (!take) {
+        const lastFile = path.join(dir, 'last.txt');
+        if (!existsSync(lastFile)) {
+            throw new Error(`Last file does not exist in directory: ${dir}. Try running an evaluation matrix first.`);
+        }
+        // read name from last file
+        const lastFileName = readFileSync(lastFile).toString();
+        files.push(lastFileName);
+    }
+    else {
+        // read the most recent files
+        files = files.sort().reverse().slice(0, take);
+    }
+    const results = [];
+    for (const file of files) {
+        const report = loadReport(path.join(dir, file));
+        results.push(...report.results);
+    }
+    // find the most recently created file in the path & read it
+    // const lastFileName = readFileSync(lastFile).toString();
+    // return loadReport(path.join(dir, lastFileName)).results;
+    return results;
+}
+/**
+ * Evaluator class for evaluating agent responses
+ */
+export class Evaluator {
+}
+export function mergeEvaluators(...evaluators) {
+    // merge evaluators in sequence
+    return evaluators.reduce((acc, val) => mergeEvaluatorsInternal(acc, val));
+}
+/**
+ * Merges two evaluators together in sequence, such that results of a are combined with b (b takes precedence in key overrides)
+ * @param a First evaluator to merge
+ * @param b Second evaluator to merge
+ */
+function mergeEvaluatorsInternal(a, b) {
+    return {
+        async evaluate(response, expected_response) {
+            const r1 = await a.evaluate(response, expected_response);
+            const r2 = await b.evaluate(response, expected_response);
+            return {
+                metadata: {
+                    ...r1.metadata,
+                    ...r2.metadata
+                },
+                data: {
+                    ...r1.data,
+                    ...r2.data
+                }
+            };
+        }
+    };
+}
+//# sourceMappingURL=evaluator.js.map

package/dist/evaluator/evaluator.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"evaluator.js","sourceRoot":"","sources":["../../src/evaluator/evaluator.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH,OAAO,EAAE,YAAY,EAAE,UAAU,EAAE,WAAW,EAAE,MAAM,IAAI,CAAC;AAC3D,OAAO,KAAK,IAAI,MAAM,MAAM,CAAC;AA2B7B;;GAEG;AACH,MAAM,UAAU,kBAAkB,CAAC,OAA0B;IACzD,MAAM,aAAa,GAAmC,IAAI,GAAG,EAAE,CAAC;IAEhE,MAAM,eAAe,GAAsB,EAAE,CAAC;IAE9C,uBAAuB;IACvB,KAAK,MAAM,MAAM,IAAI,OAAO,EAAE,CAAC;QAC3B,yDAAyD;QACzD,MAAM,IAAI,GAAG,MAAM,CAAC,IAAI,CAAC;QACzB,MAAM,cAAc,GAAG,aAAa,CAAC,GAAG,CAAC,IAAI,CAAC,IAAI,EAAE,CAAC;QACrD,cAAc,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;QAC5B,aAAa,CAAC,GAAG,CAAC,IAAI,EAAE,cAAc,CAAC,CAAC;IAC5C,CAAC;IAED,sBAAsB;IACtB,KAAK,MAAM,CAAC,IAAI,EAAE,cAAc,CAAC,IAAI,aAAa,EAAE,CAAC;QACjD,MAAM,OAAO,GAAG,cAAc,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;QAEvC,mCAAmC;QACnC,KAAK,MAAM,MAAM,IAAI,cAAc,CAAC,KAAK,CAAC,CAAC,CAAC,EAAE,CAAC;YAC3C,MAAM,UAAU,GAAG,MAAM,CAAC,IAAI,CAAC;YAC/B,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,UAAU,CAAC,EAAE,CAAC;gBACpD,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;oBAC5B,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,OAAO,CAAC,GAAG,CAAW,IAAI,CAAC,CAAC,GAAG,KAAK,CAAC;gBACzD,CAAC;YACL,CAAC;QACL,CAAC;QAED,8DAA8D;QAC9D,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,OAAO,CAAC,EAAE,CAAC;YACjD,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;gBAC5B,OAAO,CAAC,GAAG,CAAC,GAAG,KAAK,GAAG,cAAc,CAAC,MAAM,CAAC;gBAC7C,4BAA4B;gBAC5B,OAAO,CAAC,GAAG,CAAC,GAAG,IAAI,CAAC,KAAK,CAAE,OAAO,CAAC,GAAG,CAAY,GAAG,GAAG,CAAC,GAAG,GAAG,CAAC;YACpE,CAAC;QACL,CAAC;QAED,eAAe,CAAC,IAAI,CAAC;YACjB,IAAI,EAAE,cAAc,CAAC,CAAC,CAAC,CAAC,IAAI;YAC5B,QAAQ,EAAE,cAAc,CAAC,CAAC,CAAC,CAAC,QAAQ;YACpC,IAAI,EAAE,OAAO;SAChB,CAAC,CAAC;IACP,CAAC;IACD,OAAO,eAAe,CAAC;AAC3B,CAAC;AAED;;GAEG;AACH,MAAM,UAAU,oBAAoB,CAAC,OAA0B;IAC3D,4BAA4B;IAC5B,MAAM,gBAAgB,GAAG,kBAAkB,CAAC,OAAO,CAAC,CAAC;IAErD,6BAA6B;IAC7B,MAAM,aAAa,GAAmC,IAAI,GAAG,EAAE,CAAC;IAEhE,MAAM,eAAe,GAAsB,EAAE,CAAC;IAE9C,uBAAuB;IACvB,KAAK,MAAM,MAAM,IAAI,gBAAgB,EAAE,CAAC;QACpC,kDAAkD;QAClD,MAAM,IAAI,GAAG,MAAM,CAAC,QAAQ,CAAC,MAAM,CAAC;QACpC,MAAM,cAAc,GAAG,aAAa,CAAC,GAAG,CAAC,IAAI,CAAC,IAAI,EAAE,CAAC;QACrD,cAAc,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;QAC5B,aAAa,CAAC,GAAG,CAAC,IAAI,EAAE,cAAc,CAAC,CAAC;IAC5C,CAAC;IAED,sBAAsB;IACtB,KAAK,MAAM,CAAC,IAAI,EAAE,cAAc,CAAC,IAAI,aAAa,EAAE,CAAC;QACjD,MAAM,OAAO,GAAG,cAAc,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;QAEvC,mCAAmC;QACnC,KAAK,MAAM,MAAM,IAAI,cAAc,CAAC,KAAK,CAAC,CAAC,CAAC,EAAE,CAAC;YAC3C,MAAM,UAAU,GAAG,MAAM,CAAC,IAAI,CAAC;YAC/B,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,UAAU,CAAC,EAAE,CAAC;gBACpD,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;oBAC5B,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,OAAO,CAAC,GAAG,CAAW,IAAI,CAAC,CAAC,GAAG,KAAK,CAAC;gBACzD,CAAC;YACL,CAAC;QACL,CAAC;QAED,8DAA8D;QAC9D,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,OAAO,CAAC,EAAE,CAAC;YACjD,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE,CAAC;gBAC5B,OAAO,CAAC,GAAG,CAAC,GAAG,KAAK,GAAG,cAAc,CAAC,MAAM,CAAC;gBAC7C,4BAA4B;gBAC5B,OAAO,CAAC,GAAG,CAAC,GAAG,IAAI,CAAC,KAAK,CAAE,OAAO,CAAC,GAAG,CAAY,GAAG,GAAG,CAAC,GAAG,GAAG,CAAC;YACpE,CAAC;QACL,CAAC;QAED,eAAe,CAAC,IAAI,CAAC;YACjB,IAAI,EAAE,cAAc,CAAC,CAAC,CAAC,CAAC,QAAQ,CAAC,MAAM;YACvC,QAAQ,EAAE,cAAc,CAAC,CAAC,CAAC,CAAC,QAAQ;YACpC,IAAI,EAAE,OAAO;SAChB,CAAC,CAAC;IACP,CAAC;IAED,OAAO,eAAe,CAAC;AAC3B,CAAC;AAiBD;;GAEG;AACH,MAAM,UAAU,UAAU,CAAC,IAAY;IACnC,OAAO,IAAI,CAAC,KAAK,CAAC,YAAY,CAAC,IAAI,EAAE,OAAO,CAAC,CAAW,CAAC;AAC7D,CAAC;AAED;;GAEG;AACH,MAAM,UAAU,eAAe,CAAC,GAAW,EAAE,IAAa;IACtD,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,EAAE,CAAC;QACnB,MAAM,IAAI,KAAK,CAAC,6BAA6B,GAAG,EAAE,CAAC,CAAC;IACxD,CAAC;IAED,IAAI,KAAK,GAAG,WAAW,CAAC,GAAG,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC,CAAC;IAE9D,IAAI,CAAC,IAAI,EAAE,CAAC;QACR,MAAM,QAAQ,GAAG,IAAI,CAAC,IAAI,CAAC,GAAG,EAAE,UAAU,CAAC,CAAC;QAE5C,IAAI,CAAC,UAAU,CAAC,QAAQ,CAAC,EAAE,CAAC;YACxB,MAAM,IAAI,KAAK,CAAC,0CAA0C,GAAG,2CAA2C,CAAC,CAAC;QAC9G,CAAC;QACD,2BAA2B;QAC3B,MAAM,YAAY,GAAG,YAAY,CAAC,QAAQ,CAAC,CAAC,QAAQ,EAAE,CAAC;QAEvD,KAAK,CAAC,IAAI,CAAC,YAAY,CAAC,CAAC;IAE7B,CAAC;SAAM,CAAC;QACJ,6BAA6B;QAC7B,KAAK,GAAG,KAAK,CAAC,IAAI,EAAE,CAAC,OAAO,EAAE,CAAC,KAAK,CAAC,CAAC,EAAE,IAAI,CAAC,CAAC;IAElD,CAAC;IAED,MAAM,OAAO,GAAsB,EAAE,CAAC;IAEtC,KAAK,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC;QACvB,MAAM,MAAM,GAAG,UAAU,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,EAAE,IAAI,CAAC,CAAC,CAAC;QAChD,OAAO,CAAC,IAAI,CAAC,GAAG,MAAM,CAAC,OAAO,CAAC,CAAC;IACpC,CAAC;IAED,4DAA4D;IAC5D,0DAA0D;IAC1D,2DAA2D;IAC3D,OAAO,OAAO,CAAC;AACnB,CAAC;AAED;;GAEG;AACH,MAAM,OAAgB,SAAS;CAM9B;AAED,MAAM,UAAU,eAAe,CAAC,GAAG,UAAuB;IACtD,+BAA+B;IAC/B,OAAO,UAAU,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,GAAG,EAAE,EAAE,CAAC,uBAAuB,CAAC,GAAG,EAAE,GAAG,CAAC,CAAC,CAAC;AAC9E,CAAC;AAED;;;;GAIG;AACH,SAAS,uBAAuB,CAAC,CAAY,EAAE,CAAY;IACvD,OAAO;QACH,KAAK,CAAC,QAAQ,CAAC,QAAgB,EAAE,iBAAyB;YACtD,MAAM,EAAE,GAAG,MAAM,CAAC,CAAC,QAAQ,CAAC,QAAQ,EAAE,iBAAiB,CAAC,CAAC;YACzD,MAAM,EAAE,GAAG,MAAM,CAAC,CAAC,QAAQ,CAAC,QAAQ,EAAE,iBAAiB,CAAC,CAAC;YACzD,OAAO;gBACH,QAAQ,EAAE;oBACN,GAAG,EAAE,CAAC,QAAQ;oBACd,GAAG,EAAE,CAAC,QAAQ;iBACjB;gBACD,IAAI,EAAE;oBACF,GAAG,EAAE,CAAC,IAAI;oBACV,GAAG,EAAE,CAAC,IAAI;iBACb;aACJ,CAAC;QACN,CAAC;KACJ,CAAC;AACN,CAAC"}

package/dist/evaluator/index.d.ts

@@ -0,0 +1,6 @@
+export * from './evaluator.js';
+export * from './langium-evaluator.js';
+export * from './edit-distance-evaluator.js';
+export * from './eval-matrix.js';
+export * from './chart.js';
+//# sourceMappingURL=index.d.ts.map

package/dist/evaluator/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/evaluator/index.ts"],"names":[],"mappings":"AAAA,cAAc,gBAAgB,CAAC;AAC/B,cAAc,wBAAwB,CAAC;AACvC,cAAc,8BAA8B,CAAC;AAC7C,cAAc,kBAAkB,CAAC;AACjC,cAAc,YAAY,CAAC"}

package/dist/evaluator/index.js

@@ -0,0 +1,6 @@
+export * from './evaluator.js';
+export * from './langium-evaluator.js';
+export * from './edit-distance-evaluator.js';
+export * from './eval-matrix.js';
+export * from './chart.js';
+//# sourceMappingURL=index.js.map

package/dist/evaluator/index.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/evaluator/index.ts"],"names":[],"mappings":"AAAA,cAAc,gBAAgB,CAAC;AAC/B,cAAc,wBAAwB,CAAC;AACvC,cAAc,8BAA8B,CAAC;AAC7C,cAAc,kBAAkB,CAAC;AACjC,cAAc,YAAY,CAAC"}

package/dist/evaluator/langium-evaluator.d.ts

@@ -0,0 +1,55 @@
+/**
+ * Base Langium DSL validator (taps into Langium's validator messages to provide better results)
+ */
+import { LangiumServices } from "langium/lsp";
+import { Diagnostic } from "vscode-languageserver-types";
+import { Evaluator, EvaluatorResult, EvaluatorResultData } from "./evaluator.js";
+/**
+ * Langium-specific evaluator result data
+ */
+export interface LangiumEvaluatorResultData extends EvaluatorResultData {
+    /**
+     * Number of validation failures
+     */
+    failures: number;
+    /**
+     * Number of errors
+     */
+    errors: number;
+    /**
+     * Number of warnings
+     */
+    warnings: number;
+    /**
+     * Number of infos
+     */
+    infos: number;
+    /**
+     * Number of hints
+     */
+    hints: number;
+    /**
+     * Number of unassigned diagnostics
+     */
+    unassigned: number;
+    /**
+     * Length of the response in chars
+     */
+    response_length: number;
+    /**
+     * Raw diagnostic data, same which is used to compute the other values above
+     */
+    diagnostics: Diagnostic[];
+}
+export declare class LangiumEvaluator<T extends LangiumServices> extends Evaluator {
+    /**
+     * Services to use for evaluation
+     */
+    protected services: T;
+    constructor(services: T);
+    /**
+     * Validate an agent response as if it's a langium program. If we can parse it, we attempt to validate it.
+     */
+    evaluate(response: string): Promise<Partial<EvaluatorResult>>;
+}
+//# sourceMappingURL=langium-evaluator.d.ts.map

package/dist/evaluator/langium-evaluator.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"langium-evaluator.d.ts","sourceRoot":"","sources":["../../src/evaluator/langium-evaluator.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH,OAAO,EAAE,eAAe,EAAE,MAAM,aAAa,CAAC;AAC9C,OAAO,EAAE,UAAU,EAAE,MAAM,6BAA6B,CAAC;AACzD,OAAO,EAAE,SAAS,EAAE,eAAe,EAAE,mBAAmB,EAAE,MAAM,gBAAgB,CAAC;AAGjF;;GAEG;AACH,MAAM,WAAW,0BAA2B,SAAQ,mBAAmB;IAEnE;;OAEG;IACH,QAAQ,EAAE,MAAM,CAAC;IAEjB;;OAEG;IACH,MAAM,EAAE,MAAM,CAAC;IAEf;;OAEG;IACH,QAAQ,EAAE,MAAM,CAAC;IAEjB;;OAEG;IACH,KAAK,EAAE,MAAM,CAAC;IAEd;;OAEG;IACH,KAAK,EAAE,MAAM,CAAC;IAEd;;OAEG;IACH,UAAU,EAAE,MAAM,CAAC;IAEnB;;OAEG;IACH,eAAe,EAAE,MAAM,CAAC;IAExB;;OAEG;IACH,WAAW,EAAE,UAAU,EAAE,CAAC;CAC7B;AAED,qBAAa,gBAAgB,CAAC,CAAC,SAAS,eAAe,CAAE,SAAQ,SAAS;IAEtE;;OAEG;IACH,SAAS,CAAC,QAAQ,EAAE,CAAC,CAAC;gBAEV,QAAQ,EAAE,CAAC;IAKvB;;OAEG;IACG,QAAQ,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,OAAO,CAAC,eAAe,CAAC,CAAC;CAqEtE"}

package/dist/evaluator/langium-evaluator.js

@@ -0,0 +1,78 @@
+/**
+ * Base Langium DSL validator (taps into Langium's validator messages to provide better results)
+ */
+import { Evaluator } from "./evaluator.js";
+import { URI } from "langium";
+export class LangiumEvaluator extends Evaluator {
+    constructor(services) {
+        super();
+        this.services = services;
+    }
+    /**
+     * Validate an agent response as if it's a langium program. If we can parse it, we attempt to validate it.
+     */
+    async evaluate(response) {
+        if (response.includes('```')) {
+            // take the first code block instead, if present (assuming it's a langium grammar)
+            const codeBlock = response.split(/```[a-z-]*/)[1];
+            response = codeBlock;
+        }
+        const doc = this.services.shared.workspace.LangiumDocumentFactory.fromString(response, URI.parse('memory://test.langium'));
+        try {
+            await this.services.shared.workspace.DocumentBuilder.build([doc], { validation: true });
+            const validationResults = doc.diagnostics ?? [];
+            // count the number of each type of diagnostic
+            let evalData = {
+                failures: 0,
+                errors: 0,
+                warnings: 0,
+                infos: 0,
+                hints: 0,
+                unassigned: 0,
+                // include length of the response for checking
+                response_length: response.length,
+                // include the diagnostics for debugging if desired
+                diagnostics: validationResults
+            };
+            for (const diagnostic of validationResults) {
+                if (diagnostic.severity) {
+                    switch (diagnostic.severity) {
+                        case 1:
+                            evalData.errors++;
+                            break;
+                        case 2:
+                            evalData.warnings++;
+                            break;
+                        case 3:
+                            evalData.infos++;
+                            break;
+                        case 4:
+                            evalData.hints++;
+                            break;
+                        default:
+                            evalData.unassigned++;
+                            break;
+                    }
+                }
+            }
+            return {
+                data: evalData
+            };
+        }
+        catch (e) {
+            console.error('Error during evaluation: ', e);
+            return {
+                data: {
+                    failures: 1,
+                    errors: 0,
+                    warnings: 0,
+                    infos: 0,
+                    hints: 0,
+                    unassigned: 0,
+                    response_length: response.length
+                }
+            };
+        }
+    }
+}
+//# sourceMappingURL=langium-evaluator.js.map

package/dist/evaluator/langium-evaluator.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"langium-evaluator.js","sourceRoot":"","sources":["../../src/evaluator/langium-evaluator.ts"],"names":[],"mappings":"AAAA;;GAEG;AAIH,OAAO,EAAE,SAAS,EAAwC,MAAM,gBAAgB,CAAC;AACjF,OAAO,EAAE,GAAG,EAAE,MAAM,SAAS,CAAC;AAgD9B,MAAM,OAAO,gBAA4C,SAAQ,SAAS;IAOtE,YAAY,QAAW;QACnB,KAAK,EAAE,CAAC;QACR,IAAI,CAAC,QAAQ,GAAG,QAAQ,CAAC;IAC7B,CAAC;IAED;;OAEG;IACH,KAAK,CAAC,QAAQ,CAAC,QAAgB;QAE3B,IAAI,QAAQ,CAAC,QAAQ,CAAC,KAAK,CAAC,EAAE,CAAC;YAC3B,kFAAkF;YAClF,MAAM,SAAS,GAAG,QAAQ,CAAC,KAAK,CAAC,YAAY,CAAC,CAAC,CAAC,CAAC,CAAC;YAClD,QAAQ,GAAG,SAAS,CAAC;QACzB,CAAC;QAED,MAAM,GAAG,GAAG,IAAI,CAAC,QAAQ,CAAC,MAAM,CAAC,SAAS,CAAC,sBAAsB,CAAC,UAAU,CAAC,QAAQ,EAAE,GAAG,CAAC,KAAK,CAAC,uBAAuB,CAAC,CAAC,CAAC;QAE3H,IAAI,CAAC;YACD,MAAM,IAAI,CAAC,QAAQ,CAAC,MAAM,CAAC,SAAS,CAAC,eAAe,CAAC,KAAK,CAAC,CAAC,GAAG,CAAC,EAAE,EAAE,UAAU,EAAE,IAAI,EAAE,CAAC,CAAC;YACxF,MAAM,iBAAiB,GAAG,GAAG,CAAC,WAAW,IAAI,EAAE,CAAC;YAEhD,8CAA8C;YAC9C,IAAI,QAAQ,GAA+B;gBACvC,QAAQ,EAAE,CAAC;gBACX,MAAM,EAAE,CAAC;gBACT,QAAQ,EAAE,CAAC;gBACX,KAAK,EAAE,CAAC;gBACR,KAAK,EAAE,CAAC;gBACR,UAAU,EAAE,CAAC;gBACb,8CAA8C;gBAC9C,eAAe,EAAE,QAAQ,CAAC,MAAM;gBAChC,mDAAmD;gBACnD,WAAW,EAAE,iBAAiB;aACjC,CAAC;YAEF,KAAK,MAAM,UAAU,IAAI,iBAAiB,EAAE,CAAC;gBACzC,IAAI,UAAU,CAAC,QAAQ,EAAE,CAAC;oBACtB,QAAQ,UAAU,CAAC,QAAQ,EAAE,CAAC;wBAC1B,KAAK,CAAC;4BACF,QAAQ,CAAC,MAAM,EAAE,CAAC;4BAClB,MAAM;wBACV,KAAK,CAAC;4BACF,QAAQ,CAAC,QAAQ,EAAE,CAAC;4BACpB,MAAM;wBACV,KAAK,CAAC;4BACF,QAAQ,CAAC,KAAK,EAAE,CAAC;4BACjB,MAAM;wBACV,KAAK,CAAC;4BACF,QAAQ,CAAC,KAAK,EAAE,CAAC;4BACjB,MAAM;wBACV;4BACI,QAAQ,CAAC,UAAU,EAAE,CAAC;4BACtB,MAAM;oBACd,CAAC;gBACL,CAAC;YACL,CAAC;YAED,OAAO;gBACH,IAAI,EAAE,QAAQ;aACjB,CAAC;QAEN,CAAC;QAAC,OAAO,CAAC,EAAE,CAAC;YACT,OAAO,CAAC,KAAK,CAAC,2BAA2B,EAAE,CAAC,CAAC,CAAC;YAC9C,OAAO;gBACH,IAAI,EAAE;oBACF,QAAQ,EAAE,CAAC;oBACX,MAAM,EAAE,CAAC;oBACT,QAAQ,EAAE,CAAC;oBACX,KAAK,EAAE,CAAC;oBACR,KAAK,EAAE,CAAC;oBACR,UAAU,EAAE,CAAC;oBACb,eAAe,EAAE,QAAQ,CAAC,MAAM;iBACL;aAClC,CAAC;QACN,CAAC;IACL,CAAC;CACJ"}

package/dist/index.d.ts

@@ -0,0 +1,3 @@
+export * from './evaluator/index.js';
+export * from './splitter/index.js';
+//# sourceMappingURL=index.d.ts.map

package/dist/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,cAAc,sBAAsB,CAAC;AACrC,cAAc,qBAAqB,CAAC"}

package/dist/index.js

@@ -0,0 +1,3 @@
+export * from './evaluator/index.js';
+export * from './splitter/index.js';
+//# sourceMappingURL=index.js.map

package/dist/index.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,cAAc,sBAAsB,CAAC;AACrC,cAAc,qBAAqB,CAAC"}

package/dist/splitter/index.d.ts

@@ -0,0 +1,2 @@
+export * from './splitter.js';
+//# sourceMappingURL=index.d.ts.map

package/dist/splitter/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/splitter/index.ts"],"names":[],"mappings":"AAAA,cAAc,eAAe,CAAC"}

package/dist/splitter/index.js

@@ -0,0 +1,2 @@
+export * from './splitter.js';
+//# sourceMappingURL=index.js.map

package/dist/splitter/index.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/splitter/index.ts"],"names":[],"mappings":"AAAA,cAAc,eAAe,CAAC"}

package/dist/splitter/splitter.d.ts

@@ -0,0 +1,21 @@
+import { AstNode } from "langium";
+import { LangiumServices } from "langium/lsp";
+interface SplitterOptions {
+    /**
+     * List of comment rule names to include in the chunk.
+     * If not provided comments are ignored.
+     * Default: ['ML_COMMENT', 'SL_COMMENT']
+    */
+    commentRuleNames?: string[];
+}
+/**
+ * Splitter function that splits a single text document into 1 or more chunks based on a splitting strategy
+ * @param document - The text document to be split.
+ * @param nodePredicates - The predicates to determine the nodes for splitting.
+ * @param services - The Langium services used for parsing the document.
+ * @param options - The splitter configuration. See {@link SplitterOptions}.
+ * @returns The chunks of the split document.
+ */
+export declare function splitByNode(document: string, nodePredicates: Array<(node: AstNode) => boolean> | ((node: AstNode) => boolean), services: LangiumServices, options?: SplitterOptions): string[];
+export {};
+//# sourceMappingURL=splitter.d.ts.map

package/dist/splitter/splitter.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"splitter.d.ts","sourceRoot":"","sources":["../../src/splitter/splitter.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,OAAO,EAAiB,MAAM,SAAS,CAAC;AACjD,OAAO,EAAE,eAAe,EAAE,MAAM,aAAa,CAAC;AAG9C,UAAU,eAAe;IACrB;;;;MAIE;IACF,gBAAgB,CAAC,EAAE,MAAM,EAAE,CAAA;CAC9B;AAED;;;;;;;GAOG;AACH,wBAAgB,WAAW,CACvB,QAAQ,EAAE,MAAM,EAChB,cAAc,EAAE,KAAK,CAAC,CAAC,IAAI,EAAE,OAAO,KAAK,OAAO,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,OAAO,KAAK,OAAO,CAAC,EAChF,QAAQ,EAAE,eAAe,EACzB,OAAO,GAAE,eAAoE,GAAG,MAAM,EAAE,CA2D3F"}

package/dist/splitter/splitter.js

@@ -0,0 +1,59 @@
+import { CstUtils, URI } from "langium";
+import { AstUtils } from 'langium';
+/**
+ * Splitter function that splits a single text document into 1 or more chunks based on a splitting strategy
+ * @param document - The text document to be split.
+ * @param nodePredicates - The predicates to determine the nodes for splitting.
+ * @param services - The Langium services used for parsing the document.
+ * @param options - The splitter configuration. See {@link SplitterOptions}.
+ * @returns The chunks of the split document.
+ */
+export function splitByNode(document, nodePredicates, services, options = { commentRuleNames: ['ML_COMMENT', 'SL_COMMENT'] }) {
+    // 1. parse the document into an AST
+    // 2. verify that we parsed the document correctly
+    // 3. split the document into chunks based on the node
+    // 4. using the corresponding CST offsets from those nodes, split the document into chunks
+    // 5. return the chunks
+    if (document.trim() === '') {
+        return [];
+    }
+    const langiumDoc = services.shared.workspace.LangiumDocumentFactory.fromString(document, URI.parse('memory://document.langium'));
+    // not checking for lexer or parser errors here...
+    const txtDoc = langiumDoc.textDocument;
+    const chunks = [];
+    const predicates = Array.isArray(nodePredicates) ? nodePredicates : [nodePredicates];
+    // selectively stream nodes from the ast in langium
+    const stream = AstUtils.streamAst(langiumDoc.parseResult.value);
+    for (const node of stream) {
+        if (predicates.some(p => p(node))) {
+            // get the starting point of this node
+            let start = node.$cstNode?.range.start;
+            if (options?.commentRuleNames) {
+                // include comments in the chunk
+                const cstNode = node.$cstNode;
+                const commentNode = CstUtils.findCommentNode(cstNode, options.commentRuleNames);
+                if (commentNode) {
+                    // adjust start to include the comment
+                    start = commentNode.range.start;
+                }
+            }
+            const end = node.$cstNode?.range.end;
+            // add a chunk from the last offset to the start of this node
+            const chunk = txtDoc.getText({
+                start: {
+                    line: start?.line || 0,
+                    character: start?.character || 0
+                },
+                end: {
+                    line: end?.line || 0,
+                    character: end?.character || 0
+                }
+            });
+            if (chunk.trim().length > 0) {
+                chunks.push(chunk);
+            }
+        }
+    }
+    return chunks;
+}
+//# sourceMappingURL=splitter.js.map

package/dist/splitter/splitter.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"splitter.js","sourceRoot":"","sources":["../../src/splitter/splitter.ts"],"names":[],"mappings":"AAAA,OAAO,EAAW,QAAQ,EAAE,GAAG,EAAE,MAAM,SAAS,CAAC;AAEjD,OAAO,EAAE,QAAQ,EAAE,MAAM,SAAS,CAAC;AAWnC;;;;;;;GAOG;AACH,MAAM,UAAU,WAAW,CACvB,QAAgB,EAChB,cAAgF,EAChF,QAAyB,EACzB,UAA2B,EAAE,gBAAgB,EAAE,CAAC,YAAY,EAAE,YAAY,CAAC,EAAE;IAC7E,oCAAoC;IACpC,kDAAkD;IAClD,sDAAsD;IACtD,0FAA0F;IAC1F,uBAAuB;IAEvB,IAAI,QAAQ,CAAC,IAAI,EAAE,KAAK,EAAE,EAAE,CAAC;QACzB,OAAO,EAAE,CAAC;IACd,CAAC;IAED,MAAM,UAAU,GAAG,QAAQ,CAAC,MAAM,CAAC,SAAS,CAAC,sBAAsB,CAAC,UAAU,CAAC,QAAQ,EAAE,GAAG,CAAC,KAAK,CAAC,2BAA2B,CAAC,CAAC,CAAC;IAEjI,kDAAkD;IAElD,MAAM,MAAM,GAAG,UAAU,CAAC,YAAY,CAAC;IAEvC,MAAM,MAAM,GAAa,EAAE,CAAC;IAE5B,MAAM,UAAU,GAAG,KAAK,CAAC,OAAO,CAAC,cAAc,CAAC,CAAC,CAAC,CAAC,cAAc,CAAC,CAAC,CAAC,CAAC,cAAc,CAAC,CAAC;IAErF,mDAAmD;IACnD,MAAM,MAAM,GAAG,QAAQ,CAAC,SAAS,CAAC,UAAU,CAAC,WAAW,CAAC,KAAK,CAAC,CAAC;IAChE,KAAK,MAAM,IAAI,IAAI,MAAM,EAAE,CAAC;QACxB,IAAI,UAAU,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,EAAE,CAAC;YAChC,sCAAsC;YACtC,IAAI,KAAK,GAAG,IAAI,CAAC,QAAQ,EAAE,KAAK,CAAC,KAAK,CAAC;YAEvC,IAAI,OAAO,EAAE,gBAAgB,EAAE,CAAC;gBAC5B,gCAAgC;gBAChC,MAAM,OAAO,GAAG,IAAI,CAAC,QAAQ,CAAC;gBAC9B,MAAM,WAAW,GAAG,QAAQ,CAAC,eAAe,CAAC,OAAO,EAAE,OAAO,CAAC,gBAAgB,CAAC,CAAC;gBAChF,IAAI,WAAW,EAAE,CAAC;oBACd,sCAAsC;oBACtC,KAAK,GAAG,WAAW,CAAC,KAAK,CAAC,KAAK,CAAC;gBACpC,CAAC;YACL,CAAC;YAED,MAAM,GAAG,GAAG,IAAI,CAAC,QAAQ,EAAE,KAAK,CAAC,GAAG,CAAC;YACrC,6DAA6D;YAC7D,MAAM,KAAK,GAAG,MAAM,CAAC,OAAO,CAAC;gBACzB,KAAK,EAAE;oBACH,IAAI,EAAE,KAAK,EAAE,IAAI,IAAI,CAAC;oBACtB,SAAS,EAAE,KAAK,EAAE,SAAS,IAAI,CAAC;iBACnC;gBACD,GAAG,EAAE;oBACD,IAAI,EAAE,GAAG,EAAE,IAAI,IAAI,CAAC;oBACpB,SAAS,EAAE,GAAG,EAAE,SAAS,IAAI,CAAC;iBACjC;aACJ,CAAC,CAAC;YAEH,IAAI,KAAK,CAAC,IAAI,EAAE,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;gBAC1B,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;YACvB,CAAC;QACL,CAAC;IACL,CAAC;IAED,OAAO,MAAM,CAAC;AAElB,CAAC"}

package/package.json

@@ -0,0 +1,61 @@
+{
+    "name": "langium-ai-tools",
+    "version": "0.0.1",
+    "description": "Tooling for building AI Applications that leverage Langium DSLs",
+    "repository": {
+        "type": "git",
+        "url": "git+https://github.com/eclipse-langium/langium-ai.git",
+        "directory": "packages/langium-ai-tools"
+    },
+    "bugs": "https://github.com/eclipse-langium/langium-ai/issues",
+    "type": "module",
+    "main": "dist/index.js",
+    "private": false,
+    "files": [
+        "dist"
+    ],
+    "exports": {
+        ".": {
+            "import": "./dist/index.js",
+            "types": "./dist/index.d.ts"
+        },
+        "./splitter": {
+            "import": "./dist/splitter/index.js",
+            "types": "./dist/splitter/index.d.ts"
+        },
+        "./evaluator": {
+            "import": "./dist/evaluator/index.js",
+            "types": "./dist/evaluator/index.d.ts"
+        }
+    },
+    "scripts": {
+        "build": "tsc",
+        "watch": "tsc -w",
+        "test": "vitest run",
+        "clean": "rimraf ./dist"
+    },
+    "author": {
+        "name": "TypeFox",
+        "url": "https://www.typefox.io"
+    },
+    "keywords": [
+        "langium",
+        "ai",
+        "tools",
+        "llm"
+    ],
+    "license": "MIT",
+    "dependencies": {
+        "langium": "~3.4.0",
+        "levenshtein-edit-distance": "^3.0.1"
+    },
+    "volta": {
+        "node": "20.10.0",
+        "npm": "10.2.3"
+    },
+    "devDependencies": {
+        "typescript": "^5.4.5",
+        "vitest": "^3.0.9",
+        "rimraf": "^6.0.1"
+    }
+}