langium-ai-tools 4.2.0 → 5.0.0

package/README.md

@@ -22,17 +22,13 @@ LLMs (and transformers in general), are evolving quite rapidly. With this approa
 
 ## Installation
 
-Langium AI tools tracks tightly with Langium releases. If you're using Langium 3.X or 4.X in your project, you'll want to install the corresponding version of Langium AI Tools that matches it.
+Langium AI Tools treats Langium as a **peer dependency**, supporting Langium 4.x and up. Your project provides its own Langium version, and `langium-ai-tools` works alongside it.
 
 ```bash
-# if you're using Langium 4.1.X
-npm i --save langium-ai-tools@^4.1.0
-
-# or 3.5.X
-npm i --save langium-ai-tools@^3.5.0
+npm i --save langium-ai-tools
 ```
 
-We don't actively support Langium 2.X or earlier.
+Make sure your project already has a compatible version of `langium` installed (4.x or later).
 
 ## Usage
 
@@ -151,6 +147,59 @@ You can also define custom evaluators that are more tuned to the needs of your D
 
 In general we stick to focusing on what Langium can do to help with evaluation, but leave the opportunity open for you to extend, supplement, or modify evaluation logic as you see fit.
 
+### Testing API
+
+Langium AI Tools provides a vitest-style testing API for writing programmatic evaluation test suites. This allows you to define test cases in TypeScript with familiar features like:
+
+- **Test suites** with `describe()`, `describe.skip()`, and `describe.only()`
+- **Lifecycle hooks**: `beforeAll()`, `afterAll()`, `beforeEach()`, `afterEach()`
+- **Parametrized tests** with `evaluation.each()` for testing multiple data sets
+- **Test filtering** with `.skip()` and `.only()` modifiers
+
+```typescript
+import { describe, evaluation, beforeAll, afterAll, beforeEach } from 'langium-ai-tools/testing';
+
+describe('DSL Generation Tests', () => {
+  let model;
+
+  beforeAll(async () => {
+    // runs once before all tests
+    model = await setupModel();
+  });
+
+  beforeEach(() => {
+    // runs before each test
+    clearCache();
+  });
+
+  evaluation('generates valid syntax', async (ctx) => {
+    const result = await generateCode(model, ctx.systemPrompt);
+    return {
+      passed: validateSyntax(result),
+      error: !validateSyntax(result) ? 'Invalid syntax' : undefined
+    };
+  });
+
+  // parametrized tests
+  evaluation.each([
+    { input: 'person Alice', expected: 'Alice' },
+    { input: 'person Bob', expected: 'Bob' }
+  ])('extracts name $expected', (data) => async (ctx) => {
+    const parsed = parse(data.input);
+    return {
+      passed: parsed.name === data.expected
+    };
+  });
+
+  afterAll(() => {
+    // cleanup after all tests
+    cleanupModel();
+  });
+});
+```
+
+**For detailed documentation on the Testing API**, see the [Testing API Reference](../cli/docs/testing-api.md).
+
 ### Evaluation Matrix
 
 The Evaluation Matrix provides a framework for testing multiple model configurations against a set of test cases using Langium AI evaluators. This is particularly helpful when comparing across models, prompt strategies, RAG setups, or other variations in your AI stack.
@@ -443,6 +492,73 @@ console.table(averaged.map(r => ({
 
 For more complete examples, see the [example-dsl-evaluator](../examples/example-dsl-evaluator) project.
 
+### Analysis
+
+The `LangiumDocumentAnalyzer` extends the `LangiumEvaluator` with syntax usage analysis. It helps you collects statistics about which Langium grammar rules are used in documents, and it computes diversity metrics. Generally, this is useful for determining the coverage of a set of DSL programs for evaluation or training in regards to what your Langium grammar can express. It won't indicate the exact way rules are expressed, but it will give you quantitative values you can use to determine the breadth & quality of your dataset.
+
+The following is a quick example of how you can leverage the analyzer:
+
+```ts
+import { LangiumDocumentAnalyzer, AnalysisMode } from 'langium-ai-tools/analyzer';
+import { createMyDSLServices } from './my-dsl';
+import { EmptyFileSystem } from 'langium';
+
+const services = createMyDSLServices(EmptyFileSystem).MyDSL;
+
+// create an analyzer
+const analyzer = new LangiumDocumentAnalyzer(services, {
+    analysisMode: AnalysisMode.ALL,
+    // rules to exclude from analysis (WS is always excluded)
+    excludeRules: ['DeprecatedRule'],
+    // include rules from imported grammars (defaults to true)
+    includeImportedRules: true,
+    // include hidden tokens, like comments (defaults to true)
+    includeHiddenRules: true,
+    // compute diversity metrics (defaults to true)
+    computeDiversity: true
+});
+
+// evaluate per usual
+const result = analyzer.evaluate(someGeneratedDSLCode);
+
+// extract syntax statistics from the result
+const stats = analyzer.extractStatisticsFromResult(result);
+if (stats) {
+    // { RuleName: count, ... }
+    console.log('Rule usage:', stats.ruleUsage);
+    // percentage of grammar rules used
+    console.log('Coverage:', stats.coverage);
+    // { entropy, giniCoefficient, simpsonIndex }
+    console.log('Diversity:', stats.diversity);
+}
+```
+
+The analyzer computes the following metrics when parsing is successful:
+
+- **Rule usage**: A map of grammar rule names to the number of times each rule was matched in the document.
+- **Coverage**: The percentage of available grammar rules that are present 1 or more times.
+- **Diversity metrics**:
+  - **Shannon entropy** — higher values indicate more diverse rule usage patterns
+  - **Gini coefficient** — 0 suggests perfectly equal usage, 1 means maximally unequal
+  - **Simpson's diversity index** — higher values suggest more diversity
+
+The `LangiumDocumentAnalyzer` can be used as a drop-in replacement anywhere a regular `LangiumEvaluator` is expected, such as in an `EvalMatrix`:
+
+```ts
+import { EvalMatrix } from 'langium-ai-tools/evaluator';
+import { LangiumDocumentAnalyzer } from 'langium-ai-tools/analyzer';
+
+const matrix = new EvalMatrix({
+    config: { name: 'Analysis Run', history_folder: '.eval-history', num_runs: 1 },
+    runners: [myRunner],
+    evaluators: [{
+        name: 'Langium Analyzer',
+        eval: new LangiumDocumentAnalyzer(services)
+    }],
+    cases: testCases
+});
+```
+
 ## Contributing
 
 If you want to help feel free to open an issue or a PR. As a general note we're open to accept changes that focus on improving how we can support AI application development for Langium DSLs. But we don't want to provide explicit bindings to actual services/providers at this time, such as LLamaIndex, Ollama, LangChain, or others. Similarly this package doesn't provide direct bindings for AI providers such as OpenAI and Anthropic here. Instead these changes will go into a separate package under Langium AI that is intended for this purpose.

package/dist/analyzer/document-analyzer.d.ts

@@ -5,22 +5,37 @@
  *
  * @author Dennis Hübner
  ******************************************************************************/
-import { type Grammar, GrammarAST, type LangiumDocument } from "langium";
-import { type LangiumServices } from "langium/lsp";
-import { type EvaluationContext } from "../evaluator/document-evaluator.js";
-import { type EvaluatorResult } from "../evaluator/evaluator.js";
-import { LangiumEvaluator, type LangiumEvaluatorResultData } from "../evaluator/langium-evaluator.js";
-import { EvaluatorResultMsg, SyntaxStatistic } from "../gen/interface.js";
+import { type Grammar, GrammarAST, type LangiumDocument } from 'langium';
+import type { LangiumServicesLike } from '../types.js';
+import { type EvaluationContext } from '../evaluator/document-evaluator.js';
+import { type EvaluatorResult } from '../evaluator/evaluator.js';
+import { LangiumEvaluator, type LangiumEvaluatorResultData } from '../evaluator/langium-evaluator.js';
+import { EvaluatorResultMsg, SyntaxStatistic } from '../gen/interface.js';
+/**
+ * Options for resolving grammar imports when collecting rules.
+ * If provided, the analyzer can follow grammar imports to include
+ * rules from transitively imported grammars.
+ */
+export interface GrammarImportResolver {
+    /**
+     * Resolve all transitively imported grammars and return their grammar objects.
+     * This mirrors the behavior of langium's `resolveTransitiveImports`, which should be passed here
+     */
+    resolveImports(grammar: Grammar): Grammar[];
+}
 /**
  * Extends LangiumEvaluator and adds analysis capabilities.
  */
-export declare class LangiumDocumentAnalyzer<T extends LangiumServices> extends LangiumEvaluator<T> {
+export declare class LangiumDocumentAnalyzer<T extends LangiumServicesLike> extends LangiumEvaluator<T> {
     static readonly METADATA_KEY = "syntax_statistics";
     private readonly analysisOptions;
+    private readonly importResolver?;
     /**
      * Creates an instance of LangiumDocumentAnalyzer.
      * @param services Langium services
      * @param analysisOptions Analysis options
+     * @param importResolver Optional resolver for grammar imports. If not provided,
+     *        imported grammar rules will not be included even if `includeImportedRules` is true.
      * @example
      * ```typescript
      * const analyzer = new LangiumDocumentAnalyzer(services, {
@@ -30,7 +45,7 @@ export declare class LangiumDocumentAnalyzer<T extends LangiumServices> extends
      * });
      * ```
      */
-    constructor(services: T, analysisOptions?: Partial<AnalysisOptions>);
+    constructor(services: T, analysisOptions?: Partial<AnalysisOptions>, importResolver?: GrammarImportResolver);
     /**
      * Evaluates a Langium document.
      * Here we return protocol compatible object EvaluatorResultMsg.

package/dist/analyzer/document-analyzer.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"document-analyzer.d.ts","sourceRoot":"","sources":["../../src/analyzer/document-analyzer.ts"],"names":[],"mappings":"AAAA;;;;;;gFAMgF;AAEhF,OAAO,EAAY,KAAK,OAAO,EAAE,UAAU,EAAE,KAAK,eAAe,EAAiB,MAAM,SAAS,CAAC;AAElG,OAAO,EAAE,KAAK,eAAe,EAAE,MAAM,aAAa,CAAC;AACnD,OAAO,EAAE,KAAK,iBAAiB,EAAE,MAAM,oCAAoC,CAAC;AAC5E,OAAO,EAAE,KAAK,eAAe,EAAE,MAAM,2BAA2B,CAAC;AACjE,OAAO,EAAE,gBAAgB,EAAE,KAAK,0BAA0B,EAAE,MAAM,mCAAmC,CAAC;AACtG,OAAO,EAAE,kBAAkB,EAAE,eAAe,EAAE,MAAM,qBAAqB,CAAC;AAI1E;;GAEG;AACH,qBAAa,uBAAuB,CAAC,CAAC,SAAS,eAAe,CAAE,SAAQ,gBAAgB,CAAC,CAAC,CAAC;IAEvF,gBAAuB,YAAY,uBAAuB;IAE1D,OAAO,CAAC,QAAQ,CAAC,eAAe,CAAkB;IAElD;;;;;;;;;;;;OAYG;gBACS,QAAQ,EAAE,CAAC,EAAE,eAAe,GAAE,OAAO,CAAC,eAAe,CAAM;IAKvE;;;;;;;OAOG;IACH,gBAAgB,CAAC,GAAG,EAAE,eAAe,EAAE,GAAG,EAAE,iBAAiB,GAAG,eAAe,CAAC,0BAA0B,CAAC,GAAG,kBAAkB;IA4BhI,4BAA4B,CAAC,GAAG,EAAE,eAAe,EAAE,OAAO,EAAE,OAAO,GAAG,eAAe;IAqDrF;;OAEG;IACH,eAAe,CAAC,SAAS,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,GAAG,MAAM;IAK1D;;;OAGG;IACH,cAAc,CAAC,SAAS,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,GAAG,MAAM;IAgBzD;;;OAGG;IACH,sBAAsB,CAAC,SAAS,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,GAAG,MAAM;IAmBjE;;;OAGG;IACH,mBAAmB,CAAC,SAAS,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,GAAG,MAAM;IAe9D;;;;OAIG;IACH,2BAA2B,CAAC,MAAM,EAAE,OAAO,CAAC,eAAe,CAAC,GAAG,SAAS,GAAG,eAAe,GAAG,SAAS;IAYtG,SAAS,CAAC,eAAe,CAAC,OAAO,EAAE,OAAO,GAAG,UAAU,CAAC,YAAY,EAAE;IAWtE,SAAS,CAAC,0BAA0B,IAAI,eAAe;CAW1D;AAED;;GAEG;AACH,oBAAY,YAAY;IACpB,GAAG,QAAQ;IACX,YAAY,iBAAiB;CAChC;AAED,UAAU,eAAe;IACrB,YAAY,EAAE,YAAY,CAAC;IAC3B;;;OAGG;IACH,YAAY,EAAE,MAAM,EAAE,CAAC;IACvB;;OAEG;IACH,oBAAoB,EAAE,OAAO,CAAC;IAC9B;;;OAGG;IACH,kBAAkB,EAAE,OAAO,CAAC;IAC5B;;OAEG;IACH,gBAAgB,EAAE,OAAO,CAAC;CAC7B"}
+{"version":3,"file":"document-analyzer.d.ts","sourceRoot":"","sources":["../../src/analyzer/document-analyzer.ts"],"names":[],"mappings":"AAAA;;;;;;gFAMgF;AAEhF,OAAO,EAAY,KAAK,OAAO,EAAE,UAAU,EAAE,KAAK,eAAe,EAAiB,MAAM,SAAS,CAAC;AAClG,OAAO,KAAK,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AACvD,OAAO,EAAE,KAAK,iBAAiB,EAAE,MAAM,oCAAoC,CAAC;AAC5E,OAAO,EAAE,KAAK,eAAe,EAAE,MAAM,2BAA2B,CAAC;AACjE,OAAO,EAAE,gBAAgB,EAAE,KAAK,0BAA0B,EAAE,MAAM,mCAAmC,CAAC;AACtG,OAAO,EAAE,kBAAkB,EAAE,eAAe,EAAE,MAAM,qBAAqB,CAAC;AAE1E;;;;GAIG;AACH,MAAM,WAAW,qBAAqB;IAClC;;;OAGG;IACH,cAAc,CAAC,OAAO,EAAE,OAAO,GAAG,OAAO,EAAE,CAAC;CAC/C;AAED;;GAEG;AACH,qBAAa,uBAAuB,CAAC,CAAC,SAAS,mBAAmB,CAAE,SAAQ,gBAAgB,CAAC,CAAC,CAAC;IAC3F,gBAAuB,YAAY,uBAAuB;IAE1D,OAAO,CAAC,QAAQ,CAAC,eAAe,CAAkB;IAClD,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAC,CAAwB;IAExD;;;;;;;;;;;;;;OAcG;gBACS,QAAQ,EAAE,CAAC,EAAE,eAAe,GAAE,OAAO,CAAC,eAAe,CAAM,EAAE,cAAc,CAAC,EAAE,qBAAqB;IAM/G;;;;;;;OAOG;IACH,gBAAgB,CACZ,GAAG,EAAE,eAAe,EACpB,GAAG,EAAE,iBAAiB,GACvB,eAAe,CAAC,0BAA0B,CAAC,GAAG,kBAAkB;IAgCnE,4BAA4B,CAAC,GAAG,EAAE,eAAe,EAAE,OAAO,EAAE,OAAO,GAAG,eAAe;IAqDrF;;OAEG;IACH,eAAe,CAAC,SAAS,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,GAAG,MAAM;IAK1D;;;OAGG;IACH,cAAc,CAAC,SAAS,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,GAAG,MAAM;IAgBzD;;;OAGG;IACH,sBAAsB,CAAC,SAAS,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,GAAG,MAAM;IAmBjE;;;OAGG;IACH,mBAAmB,CAAC,SAAS,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,GAAG,MAAM;IAe9D;;;;OAIG;IACH,2BAA2B,CAAC,MAAM,EAAE,OAAO,CAAC,eAAe,CAAC,GAAG,SAAS,GAAG,eAAe,GAAG,SAAS;IAgBtG,SAAS,CAAC,eAAe,CAAC,OAAO,EAAE,OAAO,GAAG,UAAU,CAAC,YAAY,EAAE;IAiBtE,SAAS,CAAC,0BAA0B,IAAI,eAAe;CAW1D;AAED;;GAEG;AACH,oBAAY,YAAY;IACpB,GAAG,QAAQ;IACX,YAAY,iBAAiB;CAChC;AAED,UAAU,eAAe;IACrB,YAAY,EAAE,YAAY,CAAC;IAC3B;;;OAGG;IACH,YAAY,EAAE,MAAM,EAAE,CAAC;IACvB;;OAEG;IACH,oBAAoB,EAAE,OAAO,CAAC;IAC9B;;;OAGG;IACH,kBAAkB,EAAE,OAAO,CAAC;IAC5B;;OAEG;IACH,gBAAgB,EAAE,OAAO,CAAC;CAC7B"}

package/dist/analyzer/document-analyzer.js

@@ -5,13 +5,11 @@
  *
  * @author Dennis Hübner
  ******************************************************************************/
-import { CstUtils, GrammarAST, isLeafCstNode } from "langium";
-import { resolveTransitiveImports, } from 'langium/grammar';
-import {} from "langium/lsp";
-import {} from "../evaluator/document-evaluator.js";
-import {} from "../evaluator/evaluator.js";
-import { LangiumEvaluator } from "../evaluator/langium-evaluator.js";
-import { EvaluatorResultMsg, SyntaxStatistic } from "../gen/interface.js";
+import { CstUtils, GrammarAST, isLeafCstNode } from 'langium';
+import {} from '../evaluator/document-evaluator.js';
+import {} from '../evaluator/evaluator.js';
+import { LangiumEvaluator } from '../evaluator/langium-evaluator.js';
+import { EvaluatorResultMsg, SyntaxStatistic } from '../gen/interface.js';
 /**
  * Extends LangiumEvaluator and adds analysis capabilities.
  */
@@ -20,6 +18,8 @@ export class LangiumDocumentAnalyzer extends LangiumEvaluator {
      * Creates an instance of LangiumDocumentAnalyzer.
      * @param services Langium services
      * @param analysisOptions Analysis options
+     * @param importResolver Optional resolver for grammar imports. If not provided,
+     *        imported grammar rules will not be included even if `includeImportedRules` is true.
      * @example
      * ```typescript
      * const analyzer = new LangiumDocumentAnalyzer(services, {
@@ -29,9 +29,10 @@ export class LangiumDocumentAnalyzer extends LangiumEvaluator {
      * });
      * ```
      */
-    constructor(services, analysisOptions = {}) {
+    constructor(services, analysisOptions = {}, importResolver) {
         super(services);
         this.analysisOptions = { ...DEFAULT_OPTIONS, ...analysisOptions };
+        this.importResolver = importResolver;
     }
     /**
      * Evaluates a Langium document.
@@ -43,29 +44,31 @@ export class LangiumDocumentAnalyzer extends LangiumEvaluator {
      */
     evaluateDocument(doc, ctx) {
         const validationResult = super.evaluateDocument(doc, ctx);
-        if (this.analysisOptions.analysisMode !== AnalysisMode.NO_STATISTIC && validationResult.data && validationResult.data.failures === 0) {
-            // Add syntax usage statistics only if build was successful
+        if (this.analysisOptions.analysisMode !== AnalysisMode.NO_STATISTIC &&
+            validationResult.data &&
+            validationResult.data.failures === 0) {
+            // add syntax usage statistics only if build was successful
             const statistics = this.collectSyntaxUsageStatistics(doc, this.services.Grammar);
             validationResult.metadata[LangiumDocumentAnalyzer.METADATA_KEY] = {
                 value: {
                     oneofKind: 'syntaxStatisticValue',
-                    syntaxStatisticValue: statistics
-                }
+                    syntaxStatisticValue: statistics,
+                },
             };
         }
-        // make sure we fulfill the EvaluatorResultMsg interface 
+        // make sure we fulfill the EvaluatorResultMsg interface
         return {
             ...validationResult,
             data: {
                 ...validationResult.data,
-                diagnostics: validationResult.data.diagnostics.map(diagnostic => {
+                diagnostics: validationResult.data.diagnostics.map((diagnostic) => {
                     const code = typeof diagnostic.code === 'number' ? String(diagnostic.code) : diagnostic.code;
                     return {
                         ...diagnostic,
-                        code
+                        code,
                     };
-                })
-            }
+                }),
+            },
         };
     }
     collectSyntaxUsageStatistics(doc, grammar) {
@@ -78,11 +81,11 @@ export class LangiumDocumentAnalyzer extends LangiumEvaluator {
         const isRuleExcluded = (ruleName) => ruleName === 'WS' || excludedRules.has(ruleName);
         const allRules = includeImportedRules ? this.collectAllRules(grammar) : grammar.rules;
         const ruleUsage = {};
-        // Initialize rule usage map, excluding rules specified in excludeRules. Also skip entry rule.
+        // initialize rule usage map, excluding rules specified in excludeRules. Also skip entry rule.
         for (const rule of allRules) {
             if (!isRuleExcluded(rule.name)) {
-                if ((GrammarAST.isParserRule(rule) && rule.entry)
-                    || (GrammarAST.isTerminalRule(rule) && rule.hidden && !includeHiddenRules)) {
+                if ((GrammarAST.isParserRule(rule) && rule.entry) ||
+                    (GrammarAST.isTerminalRule(rule) && rule.hidden && !includeHiddenRules)) {
                     continue;
                 }
                 ruleUsage[rule.name] = 0;
@@ -96,7 +99,7 @@ export class LangiumDocumentAnalyzer extends LangiumEvaluator {
                 }
             };
             if (grammarSource && GrammarAST.isRuleCall(grammarSource)) {
-                // For now handle only RuleCalls
+                // for now handle only RuleCalls
                 addIfNotExcluded(grammarSource.rule.ref?.name ?? 'unknown');
             }
             else if (includeHiddenRules && cstNode.hidden && isLeafCstNode(cstNode)) {
@@ -108,7 +111,7 @@ export class LangiumDocumentAnalyzer extends LangiumEvaluator {
             diversity = {
                 entropy: this.computeEntropy(ruleUsage),
                 giniCoefficient: this.computeGiniCoefficient(ruleUsage),
-                simpsonIndex: this.computeSimpsonIndex(ruleUsage)
+                simpsonIndex: this.computeSimpsonIndex(ruleUsage),
             };
         }
         const coverage = this.computeCoverage(ruleUsage);
@@ -118,7 +121,7 @@ export class LangiumDocumentAnalyzer extends LangiumEvaluator {
      * Computes coverage as percentage of used rules over all available rules
      */
     computeCoverage(ruleUsage) {
-        const usedRules = Object.values(ruleUsage).filter(count => count > 0).length;
+        const usedRules = Object.values(ruleUsage).filter((count) => count > 0).length;
         return usedRules > 0 ? (usedRules / Object.keys(ruleUsage).length) * 100 : 0;
     }
     /**
@@ -192,8 +195,14 @@ export class LangiumDocumentAnalyzer extends LangiumEvaluator {
         return undefined;
     }
     collectAllRules(grammar) {
+        if (!this.importResolver) {
+            return grammar.rules;
+        }
         try {
-            return grammar.rules.concat(resolveTransitiveImports(this.services.shared.workspace.LangiumDocuments, grammar).map(g => g.rules).flat());
+            return grammar.rules.concat(this.importResolver
+                .resolveImports(grammar)
+                .map((g) => g.rules)
+                .flat());
         }
         catch (e) {
             console.error('Error resolving imports: ', e);
@@ -207,8 +216,8 @@ export class LangiumDocumentAnalyzer extends LangiumEvaluator {
             diversity: {
                 entropy: 0,
                 giniCoefficient: 0,
-                simpsonIndex: 0
-            }
+                simpsonIndex: 0,
+            },
         };
     }
 }
@@ -226,6 +235,6 @@ const DEFAULT_OPTIONS = {
     excludeRules: [],
     includeImportedRules: true,
     includeHiddenRules: true,
-    computeDiversity: true
+    computeDiversity: true,
 };
 //# sourceMappingURL=document-analyzer.js.map

package/dist/analyzer/document-analyzer.js.map

@@ -1 +1 @@
-{"version":3,"file":"document-analyzer.js","sourceRoot":"","sources":["../../src/analyzer/document-analyzer.ts"],"names":[],"mappings":"AAAA;;;;;;gFAMgF;AAEhF,OAAO,EAAE,QAAQ,EAAgB,UAAU,EAAwB,aAAa,EAAE,MAAM,SAAS,CAAC;AAClG,OAAO,EAAE,wBAAwB,GAAG,MAAM,iBAAiB,CAAC;AAC5D,OAAO,EAAwB,MAAM,aAAa,CAAC;AACnD,OAAO,EAA0B,MAAM,oCAAoC,CAAC;AAC5E,OAAO,EAAwB,MAAM,2BAA2B,CAAC;AACjE,OAAO,EAAE,gBAAgB,EAAmC,MAAM,mCAAmC,CAAC;AACtG,OAAO,EAAE,kBAAkB,EAAE,eAAe,EAAE,MAAM,qBAAqB,CAAC;AAI1E;;GAEG;AACH,MAAM,OAAO,uBAAmD,SAAQ,gBAAmB;IAMvF;;;;;;;;;;;;OAYG;IACH,YAAY,QAAW,EAAE,kBAA4C,EAAE;QACnE,KAAK,CAAC,QAAQ,CAAC,CAAC;QAChB,IAAI,CAAC,eAAe,GAAG,EAAE,GAAG,eAAe,EAAE,GAAG,eAAe,EAAE,CAAC;IACtE,CAAC;IAED;;;;;;;OAOG;IACH,gBAAgB,CAAC,GAAoB,EAAE,GAAsB;QACzD,MAAM,gBAAgB,GAAG,KAAK,CAAC,gBAAgB,CAAC,GAAG,EAAE,GAAG,CAAC,CAAC;QAC1D,IAAI,IAAI,CAAC,eAAe,CAAC,YAAY,KAAK,YAAY,CAAC,YAAY,IAAI,gBAAgB,CAAC,IAAI,IAAI,gBAAgB,CAAC,IAAI,CAAC,QAAQ,KAAK,CAAC,EAAE,CAAC;YACnI,2DAA2D;YAC3D,MAAM,UAAU,GAAG,IAAI,CAAC,4BAA4B,CAAC,GAAG,EAAE,IAAI,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC;YACjF,gBAAgB,CAAC,QAAQ,CAAC,uBAAuB,CAAC,YAAY,CAAC,GAAG;gBAC9D,KAAK,EAAE;oBACH,SAAS,EAAE,sBAAsB;oBACjC,oBAAoB,EAAE,UAAU;iBACnC;aACJ,CAAC;QACN,CAAC;QACD,yDAAyD;QACzD,OAAO;YACH,GAAG,gBAAgB;YACnB,IAAI,EAAE;gBACF,GAAG,gBAAgB,CAAC,IAAI;gBACxB,WAAW,EAAE,gBAAgB,CAAC,IAAI,CAAC,WAAW,CAAC,GAAG,CAAC,UAAU,CAAC,EAAE;oBAC5D,MAAM,IAAI,GAAG,OAAO,UAAU,CAAC,IAAI,KAAK,QAAQ,CAAC,CAAC,CAAC,MAAM,CAAC,UAAU,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,UAAU,CAAC,IAAI,CAAC;oBAC7F,OAAO;wBACH,GAAG,UAAU;wBACb,IAAI;qBACP,CAAC;gBACN,CAAC,CAAC;aACL;SACgE,CAAC;IAC1E,CAAC;IAED,4BAA4B,CAAC,GAAoB,EAAE,OAAgB;QAC/D,MAAM,WAAW,GAAG,GAAG,CAAC,WAAW,CAAC,KAAK,CAAC,QAAQ,CAAC;QACnD,IAAI,CAAC,WAAW,EAAE,CAAC;YACf,OAAO,IAAI,CAAC,0BAA0B,EAAE,CAAC;QAC7C,CAAC;QACD,MAAM,EAAE,oBAAoB,EAAE,YAAY,EAAE,gBAAgB,EAAE,kBAAkB,EAAE,GAAG,IAAI,CAAC,eAAe,CAAC;QAC1G,MAAM,aAAa,GAAG,IAAI,GAAG,CAAC,YAAY,CAAC,CAAC;QAC5C,MAAM,cAAc,GAAG,CAAC,QAAgB,EAAE,EAAE,CAAC,QAAQ,KAAK,IAAI,IAAI,aAAa,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC;QAE9F,MAAM,QAAQ,GAAG,oBAAoB,CAAC,CAAC,CAAC,IAAI,CAAC,eAAe,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,KAAK,CAAC;QACtF,MAAM,SAAS,GAA2B,EAAE,CAAC;QAC7C,8FAA8F;QAC9F,KAAK,MAAM,IAAI,IAAI,QAAQ,EAAE,CAAC;YAC1B,IAAI,CAAC,cAAc,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC;gBAC7B,IACI,CAAC,UAAU,CAAC,YAAY,CAAC,IAAI,CAAC,IAAI,IAAI,CAAC,KAAK,CAAC;uBAC1C,CAAC,UAAU,CAAC,cAAc,CAAC,IAAI,CAAC,IAAI,IAAI,CAAC,MAAM,IAAI,CAAC,kBAAkB,CAAC,EAC5E,CAAC;oBACC,SAAS;gBACb,CAAC;gBACD,SAAS,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;YAC7B,CAAC;QACL,CAAC;QAED,KAAK,MAAM,OAAO,IAAI,QAAQ,CAAC,SAAS,CAAC,WAAW,CAAC,EAAE,CAAC;YACpD,MAAM,aAAa,GAAG,OAAO,CAAC,aAAa,CAAC;YAE5C,MAAM,gBAAgB,GAAG,CAAC,QAAgB,EAAE,EAAE;gBAC1C,IAAI,CAAC,cAAc,CAAC,QAAQ,CAAC,EAAE,CAAC;oBAC5B,SAAS,CAAC,QAAQ,CAAC,GAAG,CAAC,SAAS,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;gBACzD,CAAC;YACL,CAAC,CAAC;YAEF,IAAI,aAAa,IAAI,UAAU,CAAC,UAAU,CAAC,aAAa,CAAC,EAAE,CAAC;gBACxD,gCAAgC;gBAChC,gBAAgB,CAAC,aAAa,CAAC,IAAI,CAAC,GAAG,EAAE,IAAI,IAAI,SAAS,CAAC,CAAC;YAChE,CAAC;iBAAM,IAAI,kBAAkB,IAAI,OAAO,CAAC,MAAM,IAAI,aAAa,CAAC,OAAO,CAAC,EAAE,CAAC;gBACxE,gBAAgB,CAAC,OAAO,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;YAC7C,CAAC;QACL,CAAC;QAED,IAAI,SAAS,GAAG,EAAE,OAAO,EAAE,CAAC,EAAE,eAAe,EAAE,CAAC,EAAE,YAAY,EAAE,CAAC,EAAE,CAAC;QACpE,IAAI,gBAAgB,EAAE,CAAC;YACnB,SAAS,GAAG;gBACR,OAAO,EAAE,IAAI,CAAC,cAAc,CAAC,SAAS,CAAC;gBACvC,eAAe,EAAE,IAAI,CAAC,sBAAsB,CAAC,SAAS,CAAC;gBACvD,YAAY,EAAE,IAAI,CAAC,mBAAmB,CAAC,SAAS,CAAC;aACpD,CAAC;QACN,CAAC;QACD,MAAM,QAAQ,GAAG,IAAI,CAAC,eAAe,CAAC,SAAS,CAAC,CAAC;QACjD,OAAO,EAAE,SAAS,EAAE,QAAQ,EAAE,SAAS,EAAE,CAAC;IAC9C,CAAC;IAED;;OAEG;IACH,eAAe,CAAC,SAAiC;QAC7C,MAAM,SAAS,GAAG,MAAM,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,EAAE,CAAC,KAAK,GAAG,CAAC,CAAC,CAAC,MAAM,CAAC;QAC7E,OAAO,SAAS,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,SAAS,GAAG,MAAM,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC,MAAM,CAAC,GAAG,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC;IACjF,CAAC;IAED;;;OAGG;IACH,cAAc,CAAC,SAAiC;QAC5C,MAAM,UAAU,GAAG,MAAM,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,KAAK,EAAE,EAAE,CAAC,GAAG,GAAG,KAAK,EAAE,CAAC,CAAC,CAAC;QACnF,IAAI,UAAU,KAAK,CAAC,EAAE,CAAC;YACnB,OAAO,CAAC,CAAC;QACb,CAAC;QAED,IAAI,OAAO,GAAG,CAAC,CAAC;QAChB,KAAK,MAAM,KAAK,IAAI,MAAM,CAAC,MAAM,CAAC,SAAS,CAAC,EAAE,CAAC;YAC3C,IAAI,KAAK,GAAG,CAAC,EAAE,CAAC;gBACZ,MAAM,WAAW,GAAG,KAAK,GAAG,UAAU,CAAC;gBACvC,OAAO,IAAI,WAAW,GAAG,IAAI,CAAC,IAAI,CAAC,WAAW,CAAC,CAAC;YACpD,CAAC;QACL,CAAC;QACD,OAAO,OAAO,CAAC;IACnB,CAAC;IAED;;;OAGG;IACH,sBAAsB,CAAC,SAAiC;QACpD,MAAM,MAAM,GAAG,MAAM,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;QAC9D,MAAM,CAAC,GAAG,MAAM,CAAC,MAAM,CAAC;QACxB,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC;YACV,OAAO,CAAC,CAAC;QACb,CAAC;QAED,MAAM,GAAG,GAAG,MAAM,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,GAAG,EAAE,EAAE,CAAC,GAAG,GAAG,GAAG,EAAE,CAAC,CAAC,CAAC;QACtD,IAAI,GAAG,KAAK,CAAC,EAAE,CAAC;YACZ,OAAO,CAAC,CAAC;QACb,CAAC;QAED,IAAI,SAAS,GAAG,CAAC,CAAC;QAClB,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC;YACzB,SAAS,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,GAAG,MAAM,CAAC,CAAC,CAAC,CAAC;QACnD,CAAC;QACD,OAAO,SAAS,GAAG,CAAC,CAAC,GAAG,GAAG,CAAC,CAAC;IACjC,CAAC;IAED;;;OAGG;IACH,mBAAmB,CAAC,SAAiC;QACjD,MAAM,UAAU,GAAG,MAAM,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,KAAK,EAAE,EAAE,CAAC,GAAG,GAAG,KAAK,EAAE,CAAC,CAAC,CAAC;QACnF,IAAI,UAAU,KAAK,CAAC,EAAE,CAAC;YACnB,OAAO,CAAC,CAAC;QACb,CAAC;QAED,IAAI,GAAG,GAAG,CAAC,CAAC;QACZ,KAAK,MAAM,KAAK,IAAI,MAAM,CAAC,MAAM,CAAC,SAAS,CAAC,EAAE,CAAC;YAC3C,MAAM,WAAW,GAAG,KAAK,GAAG,UAAU,CAAC;YACvC,GAAG,IAAI,WAAW,GAAG,WAAW,CAAC;QACrC,CAAC;QAED,OAAO,CAAC,GAAG,GAAG,CAAC,CAAC,kCAAkC;IACtD,CAAC;IAED;;;;OAIG;IACH,2BAA2B,CAAC,MAA4C;QACpE,MAAM,QAAQ,GAAG,MAAM,EAAE,QAAQ,CAAC;QAClC,IAAI,QAAQ,IAAI,QAAQ,CAAC,uBAAuB,CAAC,YAAY,CAAC,EAAE,CAAC;YAC7D,MAAM,KAAK,GAAI,QAAQ,CAAC,uBAAuB,CAAC,YAAY,CAA4E,CAAC,KAAK,CAAC;YAC/I,IAAI,KAAK,CAAC,SAAS,KAAK,sBAAsB,EAAE,CAAC;gBAC7C,OAAO,KAAK,CAAC,oBAAoB,CAAC;YACtC,CAAC;YACD,OAAO,SAAS,CAAC;QACrB,CAAC;QACD,OAAO,SAAS,CAAC;IACrB,CAAC;IAES,eAAe,CAAC,OAAgB;QACtC,IAAI,CAAC;YACD,OAAO,OAAO,CAAC,KAAK,CAAC,MAAM,CACvB,wBAAwB,CAAC,IAAI,CAAC,QAAQ,CAAC,MAAM,CAAC,SAAS,CAAC,gBAAgB,EAAE,OAAO,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,IAAI,EAAE,CAC9G,CAAC;QACN,CAAC;QAAC,OAAO,CAAC,EAAE,CAAC;YACT,OAAO,CAAC,KAAK,CAAC,2BAA2B,EAAE,CAAC,CAAC,CAAC;YAC9C,OAAO,EAAE,CAAC;QACd,CAAC;IACL,CAAC;IAES,0BAA0B;QAChC,OAAO;YACH,SAAS,EAAE,EAAE;YACb,QAAQ,EAAE,CAAC;YACX,SAAS,EAAE;gBACP,OAAO,EAAE,CAAC;gBACV,eAAe,EAAE,CAAC;gBAClB,YAAY,EAAE,CAAC;aAClB;SACJ,CAAC;IACN,CAAC;;AA3NsB,oCAAY,GAAG,mBAAmB,CAAC;AA8N9D;;GAEG;AACH,MAAM,CAAN,IAAY,YAGX;AAHD,WAAY,YAAY;IACpB,2BAAW,CAAA;IACX,6CAA6B,CAAA;AACjC,CAAC,EAHW,YAAY,KAAZ,YAAY,QAGvB;AAwBD,MAAM,eAAe,GAAoB;IACrC,YAAY,EAAE,YAAY,CAAC,GAAG;IAC9B,YAAY,EAAE,EAAE;IAChB,oBAAoB,EAAE,IAAI;IAC1B,kBAAkB,EAAE,IAAI;IACxB,gBAAgB,EAAE,IAAI;CACzB,CAAC"}
+{"version":3,"file":"document-analyzer.js","sourceRoot":"","sources":["../../src/analyzer/document-analyzer.ts"],"names":[],"mappings":"AAAA;;;;;;gFAMgF;AAEhF,OAAO,EAAE,QAAQ,EAAgB,UAAU,EAAwB,aAAa,EAAE,MAAM,SAAS,CAAC;AAElG,OAAO,EAA0B,MAAM,oCAAoC,CAAC;AAC5E,OAAO,EAAwB,MAAM,2BAA2B,CAAC;AACjE,OAAO,EAAE,gBAAgB,EAAmC,MAAM,mCAAmC,CAAC;AACtG,OAAO,EAAE,kBAAkB,EAAE,eAAe,EAAE,MAAM,qBAAqB,CAAC;AAe1E;;GAEG;AACH,MAAM,OAAO,uBAAuD,SAAQ,gBAAmB;IAM3F;;;;;;;;;;;;;;OAcG;IACH,YAAY,QAAW,EAAE,kBAA4C,EAAE,EAAE,cAAsC;QAC3G,KAAK,CAAC,QAAQ,CAAC,CAAC;QAChB,IAAI,CAAC,eAAe,GAAG,EAAE,GAAG,eAAe,EAAE,GAAG,eAAe,EAAE,CAAC;QAClE,IAAI,CAAC,cAAc,GAAG,cAAc,CAAC;IACzC,CAAC;IAED;;;;;;;OAOG;IACH,gBAAgB,CACZ,GAAoB,EACpB,GAAsB;QAEtB,MAAM,gBAAgB,GAAG,KAAK,CAAC,gBAAgB,CAAC,GAAG,EAAE,GAAG,CAAC,CAAC;QAC1D,IACI,IAAI,CAAC,eAAe,CAAC,YAAY,KAAK,YAAY,CAAC,YAAY;YAC/D,gBAAgB,CAAC,IAAI;YACrB,gBAAgB,CAAC,IAAI,CAAC,QAAQ,KAAK,CAAC,EACtC,CAAC;YACC,2DAA2D;YAC3D,MAAM,UAAU,GAAG,IAAI,CAAC,4BAA4B,CAAC,GAAG,EAAE,IAAI,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC;YACjF,gBAAgB,CAAC,QAAQ,CAAC,uBAAuB,CAAC,YAAY,CAAC,GAAG;gBAC9D,KAAK,EAAE;oBACH,SAAS,EAAE,sBAAsB;oBACjC,oBAAoB,EAAE,UAAU;iBACnC;aACJ,CAAC;QACN,CAAC;QACD,wDAAwD;QACxD,OAAO;YACH,GAAG,gBAAgB;YACnB,IAAI,EAAE;gBACF,GAAG,gBAAgB,CAAC,IAAI;gBACxB,WAAW,EAAE,gBAAgB,CAAC,IAAI,CAAC,WAAW,CAAC,GAAG,CAAC,CAAC,UAAU,EAAE,EAAE;oBAC9D,MAAM,IAAI,GAAG,OAAO,UAAU,CAAC,IAAI,KAAK,QAAQ,CAAC,CAAC,CAAC,MAAM,CAAC,UAAU,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,UAAU,CAAC,IAAI,CAAC;oBAC7F,OAAO;wBACH,GAAG,UAAU;wBACb,IAAI;qBACP,CAAC;gBACN,CAAC,CAAC;aACL;SACgE,CAAC;IAC1E,CAAC;IAED,4BAA4B,CAAC,GAAoB,EAAE,OAAgB;QAC/D,MAAM,WAAW,GAAG,GAAG,CAAC,WAAW,CAAC,KAAK,CAAC,QAAQ,CAAC;QACnD,IAAI,CAAC,WAAW,EAAE,CAAC;YACf,OAAO,IAAI,CAAC,0BAA0B,EAAE,CAAC;QAC7C,CAAC;QACD,MAAM,EAAE,oBAAoB,EAAE,YAAY,EAAE,gBAAgB,EAAE,kBAAkB,EAAE,GAAG,IAAI,CAAC,eAAe,CAAC;QAC1G,MAAM,aAAa,GAAG,IAAI,GAAG,CAAC,YAAY,CAAC,CAAC;QAC5C,MAAM,cAAc,GAAG,CAAC,QAAgB,EAAE,EAAE,CAAC,QAAQ,KAAK,IAAI,IAAI,aAAa,CAAC,GAAG,CAAC,QAAQ,CAAC,CAAC;QAE9F,MAAM,QAAQ,GAAG,oBAAoB,CAAC,CAAC,CAAC,IAAI,CAAC,eAAe,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,KAAK,CAAC;QACtF,MAAM,SAAS,GAA2B,EAAE,CAAC;QAC7C,8FAA8F;QAC9F,KAAK,MAAM,IAAI,IAAI,QAAQ,EAAE,CAAC;YAC1B,IAAI,CAAC,cAAc,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC;gBAC7B,IACI,CAAC,UAAU,CAAC,YAAY,CAAC,IAAI,CAAC,IAAI,IAAI,CAAC,KAAK,CAAC;oBAC7C,CAAC,UAAU,CAAC,cAAc,CAAC,IAAI,CAAC,IAAI,IAAI,CAAC,MAAM,IAAI,CAAC,kBAAkB,CAAC,EACzE,CAAC;oBACC,SAAS;gBACb,CAAC;gBACD,SAAS,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;YAC7B,CAAC;QACL,CAAC;QAED,KAAK,MAAM,OAAO,IAAI,QAAQ,CAAC,SAAS,CAAC,WAAW,CAAC,EAAE,CAAC;YACpD,MAAM,aAAa,GAAG,OAAO,CAAC,aAAa,CAAC;YAE5C,MAAM,gBAAgB,GAAG,CAAC,QAAgB,EAAE,EAAE;gBAC1C,IAAI,CAAC,cAAc,CAAC,QAAQ,CAAC,EAAE,CAAC;oBAC5B,SAAS,CAAC,QAAQ,CAAC,GAAG,CAAC,SAAS,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;gBACzD,CAAC;YACL,CAAC,CAAC;YAEF,IAAI,aAAa,IAAI,UAAU,CAAC,UAAU,CAAC,aAAa,CAAC,EAAE,CAAC;gBACxD,gCAAgC;gBAChC,gBAAgB,CAAC,aAAa,CAAC,IAAI,CAAC,GAAG,EAAE,IAAI,IAAI,SAAS,CAAC,CAAC;YAChE,CAAC;iBAAM,IAAI,kBAAkB,IAAI,OAAO,CAAC,MAAM,IAAI,aAAa,CAAC,OAAO,CAAC,EAAE,CAAC;gBACxE,gBAAgB,CAAC,OAAO,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;YAC7C,CAAC;QACL,CAAC;QAED,IAAI,SAAS,GAAG,EAAE,OAAO,EAAE,CAAC,EAAE,eAAe,EAAE,CAAC,EAAE,YAAY,EAAE,CAAC,EAAE,CAAC;QACpE,IAAI,gBAAgB,EAAE,CAAC;YACnB,SAAS,GAAG;gBACR,OAAO,EAAE,IAAI,CAAC,cAAc,CAAC,SAAS,CAAC;gBACvC,eAAe,EAAE,IAAI,CAAC,sBAAsB,CAAC,SAAS,CAAC;gBACvD,YAAY,EAAE,IAAI,CAAC,mBAAmB,CAAC,SAAS,CAAC;aACpD,CAAC;QACN,CAAC;QACD,MAAM,QAAQ,GAAG,IAAI,CAAC,eAAe,CAAC,SAAS,CAAC,CAAC;QACjD,OAAO,EAAE,SAAS,EAAE,QAAQ,EAAE,SAAS,EAAE,CAAC;IAC9C,CAAC;IAED;;OAEG;IACH,eAAe,CAAC,SAAiC;QAC7C,MAAM,SAAS,GAAG,MAAM,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC,MAAM,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,KAAK,GAAG,CAAC,CAAC,CAAC,MAAM,CAAC;QAC/E,OAAO,SAAS,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,SAAS,GAAG,MAAM,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC,MAAM,CAAC,GAAG,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC;IACjF,CAAC;IAED;;;OAGG;IACH,cAAc,CAAC,SAAiC;QAC5C,MAAM,UAAU,GAAG,MAAM,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,KAAK,EAAE,EAAE,CAAC,GAAG,GAAG,KAAK,EAAE,CAAC,CAAC,CAAC;QACnF,IAAI,UAAU,KAAK,CAAC,EAAE,CAAC;YACnB,OAAO,CAAC,CAAC;QACb,CAAC;QAED,IAAI,OAAO,GAAG,CAAC,CAAC;QAChB,KAAK,MAAM,KAAK,IAAI,MAAM,CAAC,MAAM,CAAC,SAAS,CAAC,EAAE,CAAC;YAC3C,IAAI,KAAK,GAAG,CAAC,EAAE,CAAC;gBACZ,MAAM,WAAW,GAAG,KAAK,GAAG,UAAU,CAAC;gBACvC,OAAO,IAAI,WAAW,GAAG,IAAI,CAAC,IAAI,CAAC,WAAW,CAAC,CAAC;YACpD,CAAC;QACL,CAAC;QACD,OAAO,OAAO,CAAC;IACnB,CAAC;IAED;;;OAGG;IACH,sBAAsB,CAAC,SAAiC;QACpD,MAAM,MAAM,GAAG,MAAM,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;QAC9D,MAAM,CAAC,GAAG,MAAM,CAAC,MAAM,CAAC;QACxB,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC;YACV,OAAO,CAAC,CAAC;QACb,CAAC;QAED,MAAM,GAAG,GAAG,MAAM,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,GAAG,EAAE,EAAE,CAAC,GAAG,GAAG,GAAG,EAAE,CAAC,CAAC,CAAC;QACtD,IAAI,GAAG,KAAK,CAAC,EAAE,CAAC;YACZ,OAAO,CAAC,CAAC;QACb,CAAC;QAED,IAAI,SAAS,GAAG,CAAC,CAAC;QAClB,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC;YACzB,SAAS,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,GAAG,MAAM,CAAC,CAAC,CAAC,CAAC;QACnD,CAAC;QACD,OAAO,SAAS,GAAG,CAAC,CAAC,GAAG,GAAG,CAAC,CAAC;IACjC,CAAC;IAED;;;OAGG;IACH,mBAAmB,CAAC,SAAiC;QACjD,MAAM,UAAU,GAAG,MAAM,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,KAAK,EAAE,EAAE,CAAC,GAAG,GAAG,KAAK,EAAE,CAAC,CAAC,CAAC;QACnF,IAAI,UAAU,KAAK,CAAC,EAAE,CAAC;YACnB,OAAO,CAAC,CAAC;QACb,CAAC;QAED,IAAI,GAAG,GAAG,CAAC,CAAC;QACZ,KAAK,MAAM,KAAK,IAAI,MAAM,CAAC,MAAM,CAAC,SAAS,CAAC,EAAE,CAAC;YAC3C,MAAM,WAAW,GAAG,KAAK,GAAG,UAAU,CAAC;YACvC,GAAG,IAAI,WAAW,GAAG,WAAW,CAAC;QACrC,CAAC;QAED,OAAO,CAAC,GAAG,GAAG,CAAC,CAAC,kCAAkC;IACtD,CAAC;IAED;;;;OAIG;IACH,2BAA2B,CAAC,MAA4C;QACpE,MAAM,QAAQ,GAAG,MAAM,EAAE,QAAQ,CAAC;QAClC,IAAI,QAAQ,IAAI,QAAQ,CAAC,uBAAuB,CAAC,YAAY,CAAC,EAAE,CAAC;YAC7D,MAAM,KAAK,GACP,QAAQ,CAAC,uBAAuB,CAAC,YAAY,CAGhD,CAAC,KAAK,CAAC;YACR,IAAI,KAAK,CAAC,SAAS,KAAK,sBAAsB,EAAE,CAAC;gBAC7C,OAAO,KAAK,CAAC,oBAAoB,CAAC;YACtC,CAAC;YACD,OAAO,SAAS,CAAC;QACrB,CAAC;QACD,OAAO,SAAS,CAAC;IACrB,CAAC;IAES,eAAe,CAAC,OAAgB;QACtC,IAAI,CAAC,IAAI,CAAC,cAAc,EAAE,CAAC;YACvB,OAAO,OAAO,CAAC,KAAK,CAAC;QACzB,CAAC;QACD,IAAI,CAAC;YACD,OAAO,OAAO,CAAC,KAAK,CAAC,MAAM,CACvB,IAAI,CAAC,cAAc;iBACd,cAAc,CAAC,OAAO,CAAC;iBACvB,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,CAAC;iBACnB,IAAI,EAAE,CACd,CAAC;QACN,CAAC;QAAC,OAAO,CAAC,EAAE,CAAC;YACT,OAAO,CAAC,KAAK,CAAC,2BAA2B,EAAE,CAAC,CAAC,CAAC;YAC9C,OAAO,EAAE,CAAC;QACd,CAAC;IACL,CAAC;IAES,0BAA0B;QAChC,OAAO;YACH,SAAS,EAAE,EAAE;YACb,QAAQ,EAAE,CAAC;YACX,SAAS,EAAE;gBACP,OAAO,EAAE,CAAC;gBACV,eAAe,EAAE,CAAC;gBAClB,YAAY,EAAE,CAAC;aAClB;SACJ,CAAC;IACN,CAAC;;AAhPsB,oCAAY,GAAG,mBAAmB,CAAC;AAmP9D;;GAEG;AACH,MAAM,CAAN,IAAY,YAGX;AAHD,WAAY,YAAY;IACpB,2BAAW,CAAA;IACX,6CAA6B,CAAA;AACjC,CAAC,EAHW,YAAY,KAAZ,YAAY,QAGvB;AAwBD,MAAM,eAAe,GAAoB;IACrC,YAAY,EAAE,YAAY,CAAC,GAAG;IAC9B,YAAY,EAAE,EAAE;IAChB,oBAAoB,EAAE,IAAI;IAC1B,kBAAkB,EAAE,IAAI;IACxB,gBAAgB,EAAE,IAAI;CACzB,CAAC"}

package/dist/evals/index.d.ts

@@ -0,0 +1,118 @@
+/**
+ * Testing API for langium-ai evaluations
+ *
+ * Provides a vitest-style API for defining evaluation test cases.
+ */
+import type { EvaluatorResult } from '../evaluator/evaluator.js';
+/**
+ * Standard data that can be expected from a case
+ */
+export type EvaluationData = {
+    /**
+     * Normalized score between 0 and 1 (0 = complete failure, 1 = full pass)
+     */
+    score: number;
+    /**
+     * Whether this case was skipped (via .skip() or .only() filtering)
+     */
+    skipped?: boolean;
+    /**
+     * When score is 0, expect an error here
+     */
+    error?: Error | string;
+};
+/**
+ * Evaluation case result from running the lai cli
+ *
+ * A modified version of the baseline evaluator result, to accomodate for additional
+ * requisite properties (certain metadat for cases + suites & a 'pass' result)
+ */
+export type EvaluationCaseResult = EvaluatorResult<EvaluationData> & {
+    metadata: {
+        /**
+         * Containing evaluation file for the associated case
+         */
+        evalFile: string;
+        /**
+         * Name of the suite that we ran under
+         */
+        suiteName: string;
+        /**
+         * Evaluation case name
+         */
+        caseName: string;
+        /**
+         * Duration of evaluation case from start to finish
+         */
+        duration: number;
+    };
+};
+export interface EvalContext {
+    systemPrompt: string;
+    project: {
+        name: string;
+    };
+}
+/**
+ * Evalutor function definition, takes some context & produce evaluation data
+ */
+export type EvaluatorFunction = (ctx: EvalContext) => Promise<EvaluationData>;
+interface EvalDefinition {
+    name: string;
+    fn: EvaluatorFunction;
+    skip?: boolean;
+    only?: boolean;
+}
+interface EvalSuite {
+    name: string;
+    evaluations: EvalDefinition[];
+    beforeAllHook?: () => void | Promise<void>;
+    afterAllHook?: () => void | Promise<void>;
+    beforeEachHook?: () => void | Promise<void>;
+    afterEachHook?: () => void | Promise<void>;
+    skip?: boolean;
+    only?: boolean;
+}
+/**
+ * Define a test suite
+ */
+export declare function describe(name: string, fn: () => void): void;
+export declare namespace describe {
+    var skip: (name: string, fn: () => void) => void;
+    var only: (name: string, fn: () => void) => void;
+}
+/**
+ * Define an evaluation case
+ */
+export declare function evaluation(name: string, fn: EvaluatorFunction): void;
+export declare namespace evaluation {
+    var skip: (name: string, fn: EvaluatorFunction) => void;
+    var only: (name: string, fn: EvaluatorFunction) => void;
+    var each: <T>(cases: T[]) => (name: string, fn: (data: T) => EvaluatorFunction) => void;
+}
+/**
+ * Define a hook that runs before each evaluation in the suite
+ */
+export declare function beforeEach(fn: () => void | Promise<void>): void;
+/**
+ * Define a hook that runs after each evaluation in the suite
+ */
+export declare function afterEach(fn: () => void | Promise<void>): void;
+/**
+ * Define a hook that runs once before all evaluations in the suite
+ */
+export declare function beforeAll(fn: () => void | Promise<void>): void;
+/**
+ * Define a hook that runs once after all evaluations in the suite
+ */
+export declare function afterAll(fn: () => void | Promise<void>): void;
+/**
+ * Export collected suites for runner
+ */
+export declare function getCollectedSuites(): EvalSuite[];
+/**
+ * Clear all collected suites
+ */
+export declare function clearSuites(): void;
+export { runEvalFile, type EvalProgressCallback } from './runner.js';
+//# sourceMappingURL=index.d.ts.map

package/dist/evals/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/evals/index.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAEH,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,2BAA2B,CAAC;AAEjE;;GAEG;AACH,MAAM,MAAM,cAAc,GAAG;IACzB;;OAEG;IACH,KAAK,EAAE,MAAM,CAAC;IAEd;;OAEG;IACH,OAAO,CAAC,EAAE,OAAO,CAAC;IAElB;;OAEG;IACH,KAAK,CAAC,EAAE,KAAK,GAAG,MAAM,CAAC;CAC1B,CAAC;AAEF;;;;;GAKG;AACH,MAAM,MAAM,oBAAoB,GAAG,eAAe,CAAC,cAAc,CAAC,GAAG;IACjE,QAAQ,EAAE;QACN;;WAEG;QACH,QAAQ,EAAE,MAAM,CAAC;QAEjB;;WAEG;QACH,SAAS,EAAE,MAAM,CAAC;QAElB;;WAEG;QACH,QAAQ,EAAE,MAAM,CAAC;QAEjB;;WAEG;QACH,QAAQ,EAAE,MAAM,CAAC;KACpB,CAAC;CACL,CAAC;AAEF,MAAM,WAAW,WAAW;IACxB,YAAY,EAAE,MAAM,CAAC;IACrB,OAAO,EAAE;QAAE,IAAI,EAAE,MAAM,CAAA;KAAE,CAAC;CAC7B;AAED;;GAEG;AACH,MAAM,MAAM,iBAAiB,GAAG,CAAC,GAAG,EAAE,WAAW,KAAK,OAAO,CAAC,cAAc,CAAC,CAAC;AAG9E,UAAU,cAAc;IACpB,IAAI,EAAE,MAAM,CAAC;IACb,EAAE,EAAE,iBAAiB,CAAC;IACtB,IAAI,CAAC,EAAE,OAAO,CAAC;IACf,IAAI,CAAC,EAAE,OAAO,CAAC;CAClB;AAED,UAAU,SAAS;IACf,IAAI,EAAE,MAAM,CAAC;IACb,WAAW,EAAE,cAAc,EAAE,CAAC;IAC9B,aAAa,CAAC,EAAE,MAAM,IAAI,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC;IAC3C,YAAY,CAAC,EAAE,MAAM,IAAI,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC;IAC1C,cAAc,CAAC,EAAE,MAAM,IAAI,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC;IAC5C,aAAa,CAAC,EAAE,MAAM,IAAI,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC;IAC3C,IAAI,CAAC,EAAE,OAAO,CAAC;IACf,IAAI,CAAC,EAAE,OAAO,CAAC;CAClB;AAKD;;GAEG;AACH,wBAAgB,QAAQ,CAAC,IAAI,EAAE,MAAM,EAAE,EAAE,EAAE,MAAM,IAAI,GAAG,IAAI,CAM3D;yBANe,QAAQ;qBAWQ,MAAM,MAAM,MAAM,IAAI,KAAG,IAAI;qBAW7B,MAAM,MAAM,MAAM,IAAI,KAAG,IAAI;;AAQ7D;;GAEG;AACH,wBAAgB,UAAU,CAAC,IAAI,EAAE,MAAM,EAAE,EAAE,EAAE,iBAAiB,GAAG,IAAI,CAKpE;yBALe,UAAU;qBAUQ,MAAM,MAAM,iBAAiB,KAAG,IAAI;qBAUpC,MAAM,MAAM,iBAAiB,KAAG,IAAI;eAkB1C,CAAC,SAAS,CAAC,EAAE,KAAG,CAAC,IAAI,EAAE,MAAM,EAAE,EAAE,EAAE,CAAC,IAAI,EAAE,CAAC,KAAK,iBAAiB,KAAK,IAAI;;AAwCtG;;GAEG;AACH,wBAAgB,UAAU,CAAC,EAAE,EAAE,MAAM,IAAI,GAAG,OAAO,CAAC,IAAI,CAAC,GAAG,IAAI,CAK/D;AAED;;GAEG;AACH,wBAAgB,SAAS,CAAC,EAAE,EAAE,MAAM,IAAI,GAAG,OAAO,CAAC,IAAI,CAAC,GAAG,IAAI,CAK9D;AAED;;GAEG;AACH,wBAAgB,SAAS,CAAC,EAAE,EAAE,MAAM,IAAI,GAAG,OAAO,CAAC,IAAI,CAAC,GAAG,IAAI,CAK9D;AAED;;GAEG;AACH,wBAAgB,QAAQ,CAAC,EAAE,EAAE,MAAM,IAAI,GAAG,OAAO,CAAC,IAAI,CAAC,GAAG,IAAI,CAK7D;AAED;;GAEG;AACH,wBAAgB,kBAAkB,IAAI,SAAS,EAAE,CAIhD;AAED;;GAEG;AACH,wBAAgB,WAAW,IAAI,IAAI,CAGlC;AAGD,OAAO,EAAE,WAAW,EAAE,KAAK,oBAAoB,EAAE,MAAM,aAAa,CAAC"}