@pauly4010/evalai-sdk 1.3.0 → 1.4.1

package/CHANGELOG.md

@@ -5,6 +5,18 @@ All notable changes to the @pauly4010/evalai-sdk package will be documented in t
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.4.1] - 2026-02-18
+
+### ✨ Added
+
+- **evalai check `--baseline production`** — Compare against latest run tagged with `environment=prod`
+- **Baseline missing handling** — Clear failure when baseline not found and comparison requested
+
+### 🔧 Changed
+
+- **Package hardening** — `files`, `module`, `sideEffects: false` for leaner npm publish
+- **CLI** — Passes `baseline` param to quality API for deterministic CI gates
+
 ## [1.3.0] - 2025-10-21
 
 ### ✨ Added

package/README.md

@@ -1,5 +1,8 @@
 # @pauly4010/evalai-sdk
 
+[![npm version](https://img.shields.io/npm/v/@pauly4010/evalai-sdk.svg)](https://www.npmjs.com/package/@pauly4010/evalai-sdk)
+[![npm downloads](https://img.shields.io/npm/dm/@pauly4010/evalai-sdk.svg)](https://www.npmjs.com/package/@pauly4010/evalai-sdk)
+
 Official TypeScript/JavaScript SDK for the AI Evaluation Platform. Build confidence in your AI systems with comprehensive evaluation tools.
 
 ## Installation
@@ -498,9 +501,83 @@ console.log("Plan:", org.plan);
 console.log("Status:", org.status);
 ```
 
+## evalai CLI (v1.4.1)
+
+The SDK includes a CLI for CI/CD evaluation gates. Install globally or use via `npx`:
+
+```bash
+# Via npx (no global install)
+npx @pauly4010/evalai-sdk check --minScore 92 --evaluationId 42 --apiKey $EVALAI_API_KEY
+
+# Or install globally
+npm install -g @pauly4010/evalai-sdk
+evalai check --minScore 92 --evaluationId 42
+```
+
+### evalai check
+
+Gate deployments on quality scores, regression, and compliance:
+
+| Option | Description |
+|--------|-------------|
+| `--evaluationId <id>` | **Required.** Evaluation to gate on |
+| `--apiKey <key>` | API key (or `EVALAI_API_KEY` env) |
+| `--minScore <n>` | Fail if score &lt; n (0–100) |
+| `--maxDrop <n>` | Fail if score dropped &gt; n from baseline |
+| `--minN <n>` | Fail if total test cases &lt; n |
+| `--allowWeakEvidence` | Permit weak evidence level |
+| `--policy <name>` | Enforce HIPAA, SOC2, GDPR, PCI_DSS, FINRA_4511 |
+| `--baseline <mode>` | `published`, `previous`, or `production` |
+| `--baseUrl <url>` | API base URL |
+
+**Exit codes:** 0=pass, 1=score below, 2=regression, 3=policy violation, 4=API error, 5=bad args, 6=low N, 7=weak evidence
+
 ## Changelog
 
-### v1.2.1 (Latest - Bug Fixes)
+### v1.4.1 (Latest)
+
+- **evalai check `--baseline production`** — Compare against latest prod-tagged run
+- **Package hardening** — Leaner npm publish with `files`, `sideEffects: false`
+
+### v1.4.0
+
+- **evalai CLI** — Command-line tool for CI/CD evaluation gates
+  - `evalai check` — Gate deployments on quality scores, regression, and compliance
+  - `--minScore <n>` — Fail if quality score &lt; n (0–100)
+  - `--maxDrop <n>` — Fail if score dropped &gt; n points from baseline
+  - `--minN <n>` — Fail if total test cases &lt; n
+  - `--allowWeakEvidence` — Permit weak evidence level (default: fail)
+  - `--policy <name>` — Enforce compliance (HIPAA, SOC2, GDPR, PCI_DSS, FINRA_4511)
+  - `--baseline <mode>` — Compare to `published` or `previous` run
+  - `--evaluationId <id>` — Required. Evaluation to gate on
+  - Environment: `EVALAI_API_KEY`, `EVALAI_BASE_URL`
+  - Exit codes: 0=pass, 1=score below, 2=regression, 3=policy violation, 4=API error, 5=bad args, 6=low N, 7=weak evidence
+- **CLI Exports** — `parseArgs`, `runCheck`, `EXIT` from `@pauly4010/evalai-sdk` for programmatic use
+
+### v1.3.0
+
+- **Workflow Tracing** — Multi-agent orchestration with full lifecycle instrumentation
+  - `WorkflowTracer` class with `startWorkflow`, `endWorkflow`, `startAgentSpan`, `endAgentSpan`
+  - `createWorkflowTracer` convenience factory
+  - `traceWorkflowStep` generic wrapper for any async function
+  - Agent handoff recording (`delegation`, `escalation`, `parallel`, `fallback`)
+  - Decision auditing with alternatives, confidence scores, reasoning, and context factors
+  - Cost tracking per span/workflow with automatic pricing (16+ models)
+  - Cost breakdown by category (`llm`, `tool`, `embedding`, `other`)
+- **Framework Integrations** — Wrap popular multi-agent frameworks:
+  - `traceLangChainAgent` — wraps `.invoke()` and `.call()` with auto-tracing
+  - `traceCrewAI` — wraps `.kickoff()` with workflow start/end
+  - `traceAutoGen` — wraps `.initiate_chat()` with workflow start/end
+- **Performance Utilities**
+  - `RequestCache` with configurable TTL (`CacheTTL` presets)
+  - `PaginatedIterator` / `createPaginatedIterator` / `autoPaginate` for cursor-based pagination
+  - `RequestBatcher` for batching API calls
+  - `RateLimiter` client-side rate limit handling
+- **Cost Tracking Types** — `CostRecord`, `CostBreakdown`, `ProviderPricing` interfaces
+- **Agent Decision Auditing Types** — `AgentDecision`, `DecisionAlternative`, `RecordDecisionParams` interfaces
+- **Benchmark Types** — `Benchmark`, `BenchmarkResult`, `AgentConfig` interfaces
+
+### v1.2.1 (Bug Fixes)
 
 - 🐛 **Critical Fixes**
   - Fixed CLI import paths for proper npm package distribution
@@ -560,6 +637,5 @@ MIT
 
 ## Support
 
-- Documentation: https://docs.evalai.com
-- Issues: https://github.com/evalai/sdk/issues
-- Discord: https://discord.gg/evalai
+- Documentation: https://v0-ai-evaluation-platform-nu.vercel.app/documentation
+- Issues: https://github.com/pauly7610/ai-evaluation-platform/issues

package/dist/__tests__/assertions.test.d.ts

@@ -0,0 +1 @@
+export {};

package/dist/__tests__/assertions.test.js

@@ -0,0 +1,288 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+const vitest_1 = require("vitest");
+const assertions_1 = require("../assertions");
+(0, vitest_1.describe)('Expectation fluent API', () => {
+    (0, vitest_1.describe)('toEqual', () => {
+        (0, vitest_1.it)('should pass when values are equal', () => {
+            const result = (0, assertions_1.expect)('hello').toEqual('hello');
+            (0, vitest_1.expect)(result.passed).toBe(true);
+            (0, vitest_1.expect)(result.name).toBe('toEqual');
+        });
+        (0, vitest_1.it)('should fail when values differ', () => {
+            const result = (0, assertions_1.expect)('hello').toEqual('world');
+            (0, vitest_1.expect)(result.passed).toBe(false);
+        });
+        (0, vitest_1.it)('should handle objects', () => {
+            const result = (0, assertions_1.expect)({ a: 1 }).toEqual({ a: 1 });
+            (0, vitest_1.expect)(result.passed).toBe(true);
+        });
+    });
+    (0, vitest_1.describe)('toContain', () => {
+        (0, vitest_1.it)('should pass when substring is found', () => {
+            const result = (0, assertions_1.expect)('Hello, world!').toContain('world');
+            (0, vitest_1.expect)(result.passed).toBe(true);
+        });
+        (0, vitest_1.it)('should fail when substring is missing', () => {
+            const result = (0, assertions_1.expect)('Hello, world!').toContain('foo');
+            (0, vitest_1.expect)(result.passed).toBe(false);
+        });
+    });
+    (0, vitest_1.describe)('toContainKeywords', () => {
+        (0, vitest_1.it)('should pass when all keywords are present (case insensitive)', () => {
+            const result = (0, assertions_1.expect)('The quick Brown fox').toContainKeywords(['quick', 'brown']);
+            (0, vitest_1.expect)(result.passed).toBe(true);
+        });
+        (0, vitest_1.it)('should fail when keywords are missing', () => {
+            const result = (0, assertions_1.expect)('The quick fox').toContainKeywords(['quick', 'brown']);
+            (0, vitest_1.expect)(result.passed).toBe(false);
+            (0, vitest_1.expect)(result.message).toContain('brown');
+        });
+    });
+    (0, vitest_1.describe)('toNotContain', () => {
+        (0, vitest_1.it)('should pass when substring is absent', () => {
+            const result = (0, assertions_1.expect)('safe text').toNotContain('danger');
+            (0, vitest_1.expect)(result.passed).toBe(true);
+        });
+        (0, vitest_1.it)('should fail when substring is present', () => {
+            const result = (0, assertions_1.expect)('some danger ahead').toNotContain('danger');
+            (0, vitest_1.expect)(result.passed).toBe(false);
+        });
+    });
+    (0, vitest_1.describe)('toNotContainPII', () => {
+        (0, vitest_1.it)('should pass with no PII', () => {
+            const result = (0, assertions_1.expect)('Just a normal sentence').toNotContainPII();
+            (0, vitest_1.expect)(result.passed).toBe(true);
+        });
+        (0, vitest_1.it)('should fail with an email', () => {
+            const result = (0, assertions_1.expect)('Contact me at user@example.com').toNotContainPII();
+            (0, vitest_1.expect)(result.passed).toBe(false);
+        });
+        (0, vitest_1.it)('should fail with a phone number', () => {
+            const result = (0, assertions_1.expect)('Call 555-123-4567').toNotContainPII();
+            (0, vitest_1.expect)(result.passed).toBe(false);
+        });
+        (0, vitest_1.it)('should fail with an SSN', () => {
+            const result = (0, assertions_1.expect)('SSN is 123-45-6789').toNotContainPII();
+            (0, vitest_1.expect)(result.passed).toBe(false);
+        });
+    });
+    (0, vitest_1.describe)('toMatchPattern', () => {
+        (0, vitest_1.it)('should pass when pattern matches', () => {
+            const result = (0, assertions_1.expect)('Order #12345').toMatchPattern(/Order #\d+/);
+            (0, vitest_1.expect)(result.passed).toBe(true);
+        });
+        (0, vitest_1.it)('should fail when pattern does not match', () => {
+            const result = (0, assertions_1.expect)('No order here').toMatchPattern(/Order #\d+/);
+            (0, vitest_1.expect)(result.passed).toBe(false);
+        });
+    });
+    (0, vitest_1.describe)('toBeValidJSON', () => {
+        (0, vitest_1.it)('should pass for valid JSON', () => {
+            const result = (0, assertions_1.expect)('{"key": "value"}').toBeValidJSON();
+            (0, vitest_1.expect)(result.passed).toBe(true);
+        });
+        (0, vitest_1.it)('should fail for invalid JSON', () => {
+            const result = (0, assertions_1.expect)('not json').toBeValidJSON();
+            (0, vitest_1.expect)(result.passed).toBe(false);
+        });
+    });
+    (0, vitest_1.describe)('toMatchJSON', () => {
+        (0, vitest_1.it)('should pass when all schema keys exist', () => {
+            const result = (0, assertions_1.expect)('{"status":"ok","data":1}').toMatchJSON({ status: '', data: '' });
+            (0, vitest_1.expect)(result.passed).toBe(true);
+        });
+        (0, vitest_1.it)('should fail when schema keys are missing', () => {
+            const result = (0, assertions_1.expect)('{"status":"ok"}').toMatchJSON({ status: '', missing: '' });
+            (0, vitest_1.expect)(result.passed).toBe(false);
+        });
+    });
+    (0, vitest_1.describe)('toHaveSentiment', () => {
+        (0, vitest_1.it)('should detect positive sentiment', () => {
+            const result = (0, assertions_1.expect)('This is great and amazing!').toHaveSentiment('positive');
+            (0, vitest_1.expect)(result.passed).toBe(true);
+        });
+        (0, vitest_1.it)('should detect negative sentiment', () => {
+            const result = (0, assertions_1.expect)('This is terrible and awful').toHaveSentiment('negative');
+            (0, vitest_1.expect)(result.passed).toBe(true);
+        });
+        (0, vitest_1.it)('should detect neutral sentiment', () => {
+            const result = (0, assertions_1.expect)('The sky is blue').toHaveSentiment('neutral');
+            (0, vitest_1.expect)(result.passed).toBe(true);
+        });
+    });
+    (0, vitest_1.describe)('toHaveLength', () => {
+        (0, vitest_1.it)('should pass when within range', () => {
+            const result = (0, assertions_1.expect)('hello').toHaveLength({ min: 3, max: 10 });
+            (0, vitest_1.expect)(result.passed).toBe(true);
+        });
+        (0, vitest_1.it)('should fail when too short', () => {
+            const result = (0, assertions_1.expect)('hi').toHaveLength({ min: 5 });
+            (0, vitest_1.expect)(result.passed).toBe(false);
+        });
+        (0, vitest_1.it)('should fail when too long', () => {
+            const result = (0, assertions_1.expect)('a very long string').toHaveLength({ max: 5 });
+            (0, vitest_1.expect)(result.passed).toBe(false);
+        });
+    });
+    (0, vitest_1.describe)('toNotHallucinate', () => {
+        (0, vitest_1.it)('should pass when all facts are present', () => {
+            const result = (0, assertions_1.expect)('Paris is the capital of France').toNotHallucinate(['paris', 'france']);
+            (0, vitest_1.expect)(result.passed).toBe(true);
+        });
+        (0, vitest_1.it)('should fail when facts are missing', () => {
+            const result = (0, assertions_1.expect)('Berlin is great').toNotHallucinate(['paris', 'france']);
+            (0, vitest_1.expect)(result.passed).toBe(false);
+        });
+    });
+    (0, vitest_1.describe)('toBeFasterThan', () => {
+        (0, vitest_1.it)('should pass when value is under threshold', () => {
+            const result = (0, assertions_1.expect)(500).toBeFasterThan(1000);
+            (0, vitest_1.expect)(result.passed).toBe(true);
+        });
+        (0, vitest_1.it)('should fail when value exceeds threshold', () => {
+            const result = (0, assertions_1.expect)(1500).toBeFasterThan(1000);
+            (0, vitest_1.expect)(result.passed).toBe(false);
+        });
+    });
+    (0, vitest_1.describe)('toBeTruthy / toBeFalsy', () => {
+        (0, vitest_1.it)('should pass for truthy values', () => {
+            (0, vitest_1.expect)((0, assertions_1.expect)('hello').toBeTruthy().passed).toBe(true);
+            (0, vitest_1.expect)((0, assertions_1.expect)(1).toBeTruthy().passed).toBe(true);
+        });
+        (0, vitest_1.it)('should pass for falsy values', () => {
+            (0, vitest_1.expect)((0, assertions_1.expect)('').toBeFalsy().passed).toBe(true);
+            (0, vitest_1.expect)((0, assertions_1.expect)(0).toBeFalsy().passed).toBe(true);
+        });
+    });
+    (0, vitest_1.describe)('toBeGreaterThan / toBeLessThan / toBeBetween', () => {
+        (0, vitest_1.it)('should work for greater than', () => {
+            (0, vitest_1.expect)((0, assertions_1.expect)(10).toBeGreaterThan(5).passed).toBe(true);
+            (0, vitest_1.expect)((0, assertions_1.expect)(3).toBeGreaterThan(5).passed).toBe(false);
+        });
+        (0, vitest_1.it)('should work for less than', () => {
+            (0, vitest_1.expect)((0, assertions_1.expect)(3).toBeLessThan(5).passed).toBe(true);
+            (0, vitest_1.expect)((0, assertions_1.expect)(10).toBeLessThan(5).passed).toBe(false);
+        });
+        (0, vitest_1.it)('should work for between', () => {
+            (0, vitest_1.expect)((0, assertions_1.expect)(5).toBeBetween(1, 10).passed).toBe(true);
+            (0, vitest_1.expect)((0, assertions_1.expect)(15).toBeBetween(1, 10).passed).toBe(false);
+        });
+    });
+    (0, vitest_1.describe)('toContainCode', () => {
+        (0, vitest_1.it)('should detect code blocks', () => {
+            const result = (0, assertions_1.expect)('Here is code:\n```js\nconsole.log("hi")\n```').toContainCode();
+            (0, vitest_1.expect)(result.passed).toBe(true);
+        });
+        (0, vitest_1.it)('should fail without code blocks', () => {
+            const result = (0, assertions_1.expect)('No code here').toContainCode();
+            (0, vitest_1.expect)(result.passed).toBe(false);
+        });
+    });
+    (0, vitest_1.describe)('toBeProfessional', () => {
+        (0, vitest_1.it)('should pass for professional text', () => {
+            const result = (0, assertions_1.expect)('Thank you for your inquiry.').toBeProfessional();
+            (0, vitest_1.expect)(result.passed).toBe(true);
+        });
+        (0, vitest_1.it)('should fail for unprofessional text', () => {
+            const result = (0, assertions_1.expect)('This is damn stupid').toBeProfessional();
+            (0, vitest_1.expect)(result.passed).toBe(false);
+        });
+    });
+    (0, vitest_1.describe)('toHaveProperGrammar', () => {
+        (0, vitest_1.it)('should pass for properly formatted text', () => {
+            const result = (0, assertions_1.expect)('This is a sentence.').toHaveProperGrammar();
+            (0, vitest_1.expect)(result.passed).toBe(true);
+        });
+        (0, vitest_1.it)('should fail for double spaces', () => {
+            const result = (0, assertions_1.expect)('This  has double spaces.').toHaveProperGrammar();
+            (0, vitest_1.expect)(result.passed).toBe(false);
+        });
+    });
+});
+(0, vitest_1.describe)('runAssertions', () => {
+    (0, vitest_1.it)('should collect all results', () => {
+        const results = (0, assertions_1.runAssertions)([
+            () => (0, assertions_1.expect)('hello').toContain('hello'),
+            () => (0, assertions_1.expect)('hello').toContain('missing'),
+        ]);
+        (0, vitest_1.expect)(results).toHaveLength(2);
+        (0, vitest_1.expect)(results[0].passed).toBe(true);
+        (0, vitest_1.expect)(results[1].passed).toBe(false);
+    });
+    (0, vitest_1.it)('should catch thrown errors', () => {
+        const results = (0, assertions_1.runAssertions)([
+            () => { throw new Error('boom'); },
+        ]);
+        (0, vitest_1.expect)(results[0].passed).toBe(false);
+        (0, vitest_1.expect)(results[0].message).toBe('boom');
+    });
+});
+(0, vitest_1.describe)('Standalone assertion functions', () => {
+    (0, vitest_1.it)('containsKeywords', () => {
+        (0, vitest_1.expect)((0, assertions_1.containsKeywords)('The quick brown fox', ['quick', 'brown'])).toBe(true);
+        (0, vitest_1.expect)((0, assertions_1.containsKeywords)('The quick fox', ['quick', 'brown'])).toBe(false);
+    });
+    (0, vitest_1.it)('matchesPattern', () => {
+        (0, vitest_1.expect)((0, assertions_1.matchesPattern)('abc123', /\d+/)).toBe(true);
+        (0, vitest_1.expect)((0, assertions_1.matchesPattern)('abc', /\d+/)).toBe(false);
+    });
+    (0, vitest_1.it)('hasLength', () => {
+        (0, vitest_1.expect)((0, assertions_1.hasLength)('hello', { min: 3, max: 10 })).toBe(true);
+        (0, vitest_1.expect)((0, assertions_1.hasLength)('hi', { min: 5 })).toBe(false);
+    });
+    (0, vitest_1.it)('containsJSON', () => {
+        (0, vitest_1.expect)((0, assertions_1.containsJSON)('{"a":1}')).toBe(true);
+        (0, vitest_1.expect)((0, assertions_1.containsJSON)('not json')).toBe(false);
+    });
+    (0, vitest_1.it)('notContainsPII', () => {
+        (0, vitest_1.expect)((0, assertions_1.notContainsPII)('Just a normal text')).toBe(true);
+        (0, vitest_1.expect)((0, assertions_1.notContainsPII)('Email: user@example.com')).toBe(false);
+    });
+    (0, vitest_1.it)('hasSentiment', () => {
+        (0, vitest_1.expect)((0, assertions_1.hasSentiment)('This is great', 'positive')).toBe(true);
+        (0, vitest_1.expect)((0, assertions_1.hasSentiment)('This is terrible', 'negative')).toBe(true);
+    });
+    (0, vitest_1.it)('similarTo', () => {
+        (0, vitest_1.expect)((0, assertions_1.similarTo)('the quick brown fox', 'the quick brown dog', 0.5)).toBe(true);
+        (0, vitest_1.expect)((0, assertions_1.similarTo)('hello world', 'completely different', 0.8)).toBe(false);
+    });
+    (0, vitest_1.it)('withinRange', () => {
+        (0, vitest_1.expect)((0, assertions_1.withinRange)(5, 1, 10)).toBe(true);
+        (0, vitest_1.expect)((0, assertions_1.withinRange)(15, 1, 10)).toBe(false);
+    });
+    (0, vitest_1.it)('isValidEmail', () => {
+        (0, vitest_1.expect)((0, assertions_1.isValidEmail)('user@example.com')).toBe(true);
+        (0, vitest_1.expect)((0, assertions_1.isValidEmail)('not-an-email')).toBe(false);
+    });
+    (0, vitest_1.it)('isValidURL', () => {
+        (0, vitest_1.expect)((0, assertions_1.isValidURL)('https://example.com')).toBe(true);
+        (0, vitest_1.expect)((0, assertions_1.isValidURL)('not a url')).toBe(false);
+    });
+    (0, vitest_1.it)('hasNoHallucinations', () => {
+        (0, vitest_1.expect)((0, assertions_1.hasNoHallucinations)('Paris is in France', ['Paris', 'France'])).toBe(true);
+        (0, vitest_1.expect)((0, assertions_1.hasNoHallucinations)('Berlin is great', ['Paris'])).toBe(false);
+    });
+    (0, vitest_1.it)('matchesSchema', () => {
+        (0, vitest_1.expect)((0, assertions_1.matchesSchema)({ name: 'test', value: 1 }, { name: '', value: '' })).toBe(true);
+        (0, vitest_1.expect)((0, assertions_1.matchesSchema)({ name: 'test' }, { name: '', missing: '' })).toBe(false);
+        (0, vitest_1.expect)((0, assertions_1.matchesSchema)('not an object', { key: '' })).toBe(false);
+    });
+    (0, vitest_1.it)('hasNoToxicity', () => {
+        (0, vitest_1.expect)((0, assertions_1.hasNoToxicity)('Have a nice day')).toBe(true);
+        (0, vitest_1.expect)((0, assertions_1.hasNoToxicity)('You are an idiot')).toBe(false);
+    });
+    (0, vitest_1.it)('followsInstructions', () => {
+        (0, vitest_1.expect)((0, assertions_1.followsInstructions)('Hello world', ['Hello'])).toBe(true);
+        (0, vitest_1.expect)((0, assertions_1.followsInstructions)('Hello world', ['!goodbye'])).toBe(true);
+        (0, vitest_1.expect)((0, assertions_1.followsInstructions)('Hello world', ['missing'])).toBe(false);
+    });
+    (0, vitest_1.it)('containsAllRequiredFields', () => {
+        (0, vitest_1.expect)((0, assertions_1.containsAllRequiredFields)({ a: 1, b: 2 }, ['a', 'b'])).toBe(true);
+        (0, vitest_1.expect)((0, assertions_1.containsAllRequiredFields)({ a: 1 }, ['a', 'b'])).toBe(false);
+    });
+    (0, vitest_1.it)('hasValidCodeSyntax', () => {
+        (0, vitest_1.expect)((0, assertions_1.hasValidCodeSyntax)('{"valid": true}', 'json')).toBe(true);
+        (0, vitest_1.expect)((0, assertions_1.hasValidCodeSyntax)('{invalid}', 'json')).toBe(false);
+    });
+});

package/dist/__tests__/client.test.d.ts

@@ -0,0 +1 @@
+export {};

package/dist/__tests__/client.test.js

@@ -0,0 +1,185 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+const vitest_1 = require("vitest");
+const client_1 = require("../client");
+// Mock fetch globally
+const mockFetch = vitest_1.vi.fn();
+(0, vitest_1.describe)('AIEvalClient', () => {
+    (0, vitest_1.beforeEach)(() => {
+        vitest_1.vi.stubGlobal('fetch', mockFetch);
+        mockFetch.mockReset();
+    });
+    (0, vitest_1.afterEach)(() => {
+        vitest_1.vi.unstubAllGlobals();
+    });
+    (0, vitest_1.describe)('constructor', () => {
+        (0, vitest_1.it)('should throw if no API key is provided', () => {
+            (0, vitest_1.expect)(() => new client_1.AIEvalClient({ apiKey: '' })).toThrow('API key is required');
+        });
+        (0, vitest_1.it)('should initialize with explicit config', () => {
+            const client = new client_1.AIEvalClient({
+                apiKey: 'test-key',
+                baseUrl: 'https://api.test.com',
+                organizationId: 42,
+            });
+            (0, vitest_1.expect)(client).toBeDefined();
+            (0, vitest_1.expect)(client.getOrganizationId()).toBe(42);
+        });
+        (0, vitest_1.it)('should have all API modules', () => {
+            const client = new client_1.AIEvalClient({ apiKey: 'test-key' });
+            (0, vitest_1.expect)(client.traces).toBeDefined();
+            (0, vitest_1.expect)(client.evaluations).toBeDefined();
+            (0, vitest_1.expect)(client.llmJudge).toBeDefined();
+            (0, vitest_1.expect)(client.annotations).toBeDefined();
+            (0, vitest_1.expect)(client.developer).toBeDefined();
+            (0, vitest_1.expect)(client.organizations).toBeDefined();
+        });
+    });
+    (0, vitest_1.describe)('request method', () => {
+        (0, vitest_1.it)('should send auth header', async () => {
+            mockFetch.mockResolvedValue({
+                ok: true,
+                json: async () => ({ data: 'test' }),
+                status: 200,
+            });
+            const client = new client_1.AIEvalClient({ apiKey: 'my-secret-key', baseUrl: 'http://localhost:3000' });
+            await client.request('/api/test');
+            (0, vitest_1.expect)(mockFetch).toHaveBeenCalledTimes(1);
+            const [url, options] = mockFetch.mock.calls[0];
+            (0, vitest_1.expect)(url).toBe('http://localhost:3000/api/test');
+            (0, vitest_1.expect)(options.headers['Authorization']).toBe('Bearer my-secret-key');
+            (0, vitest_1.expect)(options.headers['Content-Type']).toBe('application/json');
+        });
+        (0, vitest_1.it)('should return parsed JSON on success', async () => {
+            mockFetch.mockResolvedValue({
+                ok: true,
+                json: async () => ({ result: 'ok' }),
+                status: 200,
+            });
+            const client = new client_1.AIEvalClient({ apiKey: 'key', baseUrl: 'http://localhost:3000' });
+            const data = await client.request('/api/test');
+            (0, vitest_1.expect)(data).toEqual({ result: 'ok' });
+        });
+        (0, vitest_1.it)('should throw on non-ok response', async () => {
+            mockFetch.mockResolvedValue({
+                ok: false,
+                json: async () => ({ error: 'Not found', code: 'NOT_FOUND' }),
+                status: 404,
+            });
+            const client = new client_1.AIEvalClient({ apiKey: 'key', baseUrl: 'http://localhost:3000', retry: { maxAttempts: 1 } });
+            await (0, vitest_1.expect)(client.request('/api/test')).rejects.toThrow();
+        });
+        (0, vitest_1.it)('should retry on rate limit errors', async () => {
+            mockFetch
+                .mockResolvedValueOnce({
+                ok: false,
+                json: async () => ({ error: 'Rate limited', code: 'RATE_LIMIT_EXCEEDED' }),
+                status: 429,
+            })
+                .mockResolvedValueOnce({
+                ok: true,
+                json: async () => ({ result: 'success' }),
+                status: 200,
+            });
+            const client = new client_1.AIEvalClient({
+                apiKey: 'key',
+                baseUrl: 'http://localhost:3000',
+                retry: { maxAttempts: 3, backoff: 'fixed' },
+            });
+            const data = await client.request('/api/test');
+            (0, vitest_1.expect)(data).toEqual({ result: 'success' });
+            (0, vitest_1.expect)(mockFetch).toHaveBeenCalledTimes(2);
+        });
+        (0, vitest_1.it)('should handle timeout', async () => {
+            mockFetch.mockImplementation(() => new Promise((_, reject) => {
+                const abortError = new Error('The operation was aborted');
+                abortError.name = 'AbortError';
+                setTimeout(() => reject(abortError), 50);
+            }));
+            const client = new client_1.AIEvalClient({
+                apiKey: 'key',
+                baseUrl: 'http://localhost:3000',
+                timeout: 10,
+                retry: { maxAttempts: 1 },
+            });
+            await (0, vitest_1.expect)(client.request('/api/slow')).rejects.toThrow();
+        });
+    });
+    (0, vitest_1.describe)('TraceAPI', () => {
+        (0, vitest_1.it)('should call correct endpoint for traces.create', async () => {
+            mockFetch.mockResolvedValue({
+                ok: true,
+                json: async () => ({ id: 1, name: 'Test Trace', traceId: 'trace-1' }),
+                status: 200,
+            });
+            const client = new client_1.AIEvalClient({ apiKey: 'key', baseUrl: 'http://localhost:3000', organizationId: 1 });
+            const result = await client.traces.create({ name: 'Test Trace', traceId: 'trace-1' });
+            (0, vitest_1.expect)(result.name).toBe('Test Trace');
+            const [url, options] = mockFetch.mock.calls[0];
+            (0, vitest_1.expect)(url).toBe('http://localhost:3000/api/traces');
+            (0, vitest_1.expect)(options.method).toBe('POST');
+        });
+        (0, vitest_1.it)('should call correct endpoint for traces.list', async () => {
+            mockFetch.mockResolvedValue({
+                ok: true,
+                json: async () => [],
+                status: 200,
+            });
+            const client = new client_1.AIEvalClient({ apiKey: 'key', baseUrl: 'http://localhost:3000' });
+            await client.traces.list({ limit: 10 });
+            const [url] = mockFetch.mock.calls[0];
+            (0, vitest_1.expect)(url).toContain('/api/traces');
+            (0, vitest_1.expect)(url).toContain('limit=10');
+        });
+    });
+    (0, vitest_1.describe)('EvaluationAPI', () => {
+        (0, vitest_1.it)('should call correct endpoint for evaluations.create', async () => {
+            mockFetch.mockResolvedValue({
+                ok: true,
+                json: async () => ({ id: 1, name: 'Eval' }),
+                status: 200,
+            });
+            const client = new client_1.AIEvalClient({ apiKey: 'key', baseUrl: 'http://localhost:3000', organizationId: 1 });
+            await client.evaluations.create({
+                name: 'Eval',
+                type: 'unit_test',
+                organizationId: 1,
+                createdBy: 1,
+            });
+            const [url, options] = mockFetch.mock.calls[0];
+            (0, vitest_1.expect)(url).toBe('http://localhost:3000/api/evaluations');
+            (0, vitest_1.expect)(options.method).toBe('POST');
+        });
+        (0, vitest_1.it)('should call correct endpoint for evaluations.createRun', async () => {
+            mockFetch.mockResolvedValue({
+                ok: true,
+                json: async () => ({ id: 1, status: 'running' }),
+                status: 200,
+            });
+            const client = new client_1.AIEvalClient({ apiKey: 'key', baseUrl: 'http://localhost:3000' });
+            await client.evaluations.createRun(42, { status: 'running' });
+            const [url, options] = mockFetch.mock.calls[0];
+            (0, vitest_1.expect)(url).toBe('http://localhost:3000/api/evaluations/42/runs');
+            (0, vitest_1.expect)(options.method).toBe('POST');
+        });
+    });
+    (0, vitest_1.describe)('LLMJudgeAPI', () => {
+        (0, vitest_1.it)('should call correct endpoint for llmJudge.evaluate', async () => {
+            mockFetch.mockResolvedValue({
+                ok: true,
+                json: async () => ({ result: { score: 85 }, config: {} }),
+                status: 200,
+            });
+            const client = new client_1.AIEvalClient({ apiKey: 'key', baseUrl: 'http://localhost:3000' });
+            const result = await client.llmJudge.evaluate({
+                configId: 1,
+                input: 'test input',
+                output: 'test output',
+            });
+            (0, vitest_1.expect)(result.result.score).toBe(85);
+            const [url, options] = mockFetch.mock.calls[0];
+            (0, vitest_1.expect)(url).toBe('http://localhost:3000/api/llm-judge/evaluate');
+            (0, vitest_1.expect)(options.method).toBe('POST');
+        });
+    });
+});

package/dist/__tests__/testing.test.d.ts

@@ -0,0 +1 @@
+export {};