@evalgate/sdk 2.2.1 → 2.2.3

package/CHANGELOG.md

@@ -5,6 +5,67 @@ All notable changes to the @evalgate/sdk package will be documented in this file
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.2.3] - 2026-03-03
+
+### Fixed
+
+- **`RequestCache.set` missing default TTL** — entries stored without an explicit TTL were immediately stale on next read. Default is now `CacheTTL.MEDIUM`; callers that omit `ttl` get a live cache entry instead of a cache miss every time.
+- **`EvalGateError` subclass prototype chain** — `ValidationError.name` was silently overwritten by the base class constructor, surfacing as `"EvalGateError"` in stack traces and `instanceof` checks. All four subclasses (`ValidationError`, `RateLimitError`, `AuthenticationError`, `NetworkError`) now call `Object.setPrototypeOf(this, Subclass.prototype)` and set `this.name` after `super()`.
+- **`RateLimitError.retryAfter` not a direct property** — the value was only stored inside `details.retryAfter` and not accessible as `err.retryAfter`. It is now assigned directly on the instance when provided.
+- **`autoPaginate` returned `AsyncGenerator` instead of `Promise<T[]>`** — calling `await autoPaginate(fetcher)` was resolving to an unexhausted generator. It now collects all pages and returns a flat `Promise<T[]>`. The original streaming behaviour is available via the new `autoPaginateGenerator` export.
+- **`createEvalRuntime` string-only overload** — passing `{ name, projectRoot }` config objects was ignored (treated as `process.cwd()`). The function now accepts `string | { name?: string; projectRoot?: string }` and extracts `projectRoot` correctly.
+- **`defaultLocalExecutor` was an instance, not a factory** — importing `defaultLocalExecutor` returned a pre-constructed executor rather than a callable factory. It is now re-exported as `createLocalExecutor` so each import site can call it to get a fresh instance.
+- **`SnapshotManager.save` crash on `undefined`/`null` output** — passing `undefined` or `null` to `snapshot(name, output)` threw `TypeError: Cannot convert undefined to string`. Both values are now serialized to the strings `"undefined"` and `"null"` respectively, matching the existing `null`-safe coercion already present for objects.
+- **`compareSnapshots` loaded raw string instead of disk snapshot** — the old `compareWithSnapshot` alias passed its second argument as literal content rather than a snapshot name, producing meaningless diffs. The new `compareSnapshots(nameA, nameB, dir?)` loads both snapshots from disk before diffing.
+- **`AIEvalClient` default `baseUrl`** — the no-arg constructor defaulted to `http://localhost:3000`, causing silent failures in production environments. Default is now `https://api.evalgate.com`.
+- **`importData` unguarded `client.traces` / `client.evaluations` access** — calling `importData(data)` with a partial or undefined client could throw `TypeError: Cannot read properties of undefined`. Both property accesses now use optional chaining (`client?.traces`, `client?.evaluations`).
+- **`toContainCode` required a fenced code block** — raw function definitions, `const` assignments, class declarations, arrow functions, `import`/`export` statements, and `return` expressions now satisfy the assertion without needing triple-backtick fencing.
+- **`hasReadabilityScore` ignored `{min}` object form** — passing `{ min: 40 }` instead of a plain number was coerced to `NaN` threshold, making every call return `true`. The function now unwraps `{ min?, max? }` objects and applies both bounds.
+
+### Added
+
+- **`autoPaginateGenerator`** — new export for streaming pagination as an `AsyncGenerator<T[]>` (one chunk per page). Use when you want to process pages incrementally rather than wait for all pages to load.
+- **`compareSnapshots(nameA, nameB, dir?)`** — loads both named snapshots from disk and returns a `SnapshotComparison`. Replaces the incorrectly aliased `compareWithSnapshot`.
+- **141 new regression tests** across 9 test files covering all fixes above: `RequestCache` TTL defaults, error class prototype chains, `autoPaginate` flat-array return, `createEvalRuntime` config-object overload, `defaultLocalExecutor` callable factory, `SnapshotManager` null/undefined handling, `compareSnapshots` disk-load path, `AIEvalClient` default `baseUrl`, `importData` guards, `toContainCode` raw-code detection, and `hasReadabilityScore` object form.
+- **`upgrade --full` post-upgrade warning** — CLI now prints a reminder to run `npx evalgate baseline update` after a full upgrade to avoid a false regression on the next CI run.
+- **Optional chaining on OpenAI / Anthropic integration `traces.create`** — `evalClient.traces?.create(...)` prevents crashes when the `traces` resource is unavailable on the client (e.g. minimal config or testing without a full API key).
+
+---
+
+## [2.2.2] - 2026-03-03
+
+### Fixed
+
+- **8 stub assertions replaced with real implementations:**
+  - `hasSentiment` — substring matching + expanded 34/31-word positive/negative lexicon (was exact-match, 4 words each)
+  - `hasNoHallucinations` — case-insensitive fact matching (was case-sensitive)
+  - `hasFactualAccuracy` — case-insensitive fact matching (was case-sensitive)
+  - `containsLanguage` — expanded from 3 languages (en/es/fr) to 12 (+ de/it/pt/nl/ru/zh/ja/ko/ar) with BCP-47 subtag support (`zh-CN` → `zh`)
+  - `hasValidCodeSyntax` — real bracket/brace/parenthesis balance checker with string literal and comment awareness (handles JS `//`/`/* */`, Python `#`, template literals, single/double quotes); JSON fast-path via `JSON.parse`
+  - `hasNoToxicity` — expanded from 4 words to ~80 terms across 9 categories: insults, degradation, violence/threats, self-harm directed at others, dehumanization, hate/rejection, harassment, profanity-as-attacks, bullying/appearance/mental-health weaponization
+  - `hasReadabilityScore` — fixed Flesch-Kincaid syllable counting to be per-word (was treating entire text as one word)
+  - `matchesSchema` — now dispatches on schema format: JSON Schema `required` array (`{ required: ['name'] }` → checks required keys exist), JSON Schema `properties` object (`{ properties: { name: {} } }` → checks property keys exist), or simple key-presence template (existing behavior preserved for backward compat). Fixes regression: `matchesSchema({ name: 'test', score: 95 }, { type: 'object', required: ['name'] })` was returning `false`
+- **`importData` crash** — `options: ImportOptions` parameter now defaults to `{}` to prevent `Cannot read properties of undefined (reading 'dryRun')` when called as `importData(client, data)`
+- **`compareWithSnapshot` / `SnapshotManager.compare` object coercion** — both now accept `unknown` input and coerce non-string values via `JSON.stringify` before comparison, matching the existing behavior of `SnapshotManager.save()`
+- **`WorkflowTracer` constructor crash** — defensive guard: `typeof client?.getOrganizationId === "function"` before calling it; prevents `TypeError: client.getOrganizationId is not a function` when using partial clients or initializing without an API key
+
+### Added
+
+- **LLM-backed async assertion variants** — 6 new exported functions:
+  - `hasSentimentAsync(text, expected, config?)` — LLM classifies sentiment with full context awareness
+  - `hasNoToxicityAsync(text, config?)` — LLM detects sarcastic, implicit, and culturally specific toxic content that blocklists miss
+  - `containsLanguageAsync(text, language, config?)` — LLM language detection for any language
+  - `hasValidCodeSyntaxAsync(code, language, config?)` — LLM deep syntax analysis beyond bracket balance
+  - `hasFactualAccuracyAsync(text, facts, config?)` — LLM checks facts semantically, catches paraphrased inaccuracies
+  - `hasNoHallucinationsAsync(text, groundTruth, config?)` — LLM detects fabricated claims even when paraphrased
+- **`configureAssertions(config: AssertionLLMConfig)`** — set global LLM provider/apiKey/model/baseUrl once; all `*Async` functions use it automatically; per-call `config` overrides it
+- **`getAssertionConfig()`** — retrieve current global assertion LLM config
+- **`AssertionLLMConfig` type** — exported interface: `{ provider: "openai" | "anthropic"; apiKey: string; model?: string; baseUrl?: string }`
+- **JSDoc `**Fast and approximate**` / `**Slow and accurate**` markers** on all sync/async assertion pairs with `{@link xAsync}` cross-references that appear in IDE tooltips
+- **115 new tests** in `assertions.test.ts` covering all improved sync assertions (expanded lexicons, JSON Schema formats, bracket balance edge cases, 12-language detection, BCP-47) and all 6 async variants (OpenAI path, Anthropic path, global config, error cases, HTTP 4xx handling)
+
+---
+
 ## [2.2.1] - 2026-03-03
 
 ### Fixed
@@ -50,6 +111,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Low:** Explain no longer shows "unnamed" for builtin gate failures
 - **Docs:** Added missing `discover --manifest` step to local quickstart
 
+---
+
 ## [2.1.2] - 2026-03-02
 
 ### Fixed
@@ -57,12 +120,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Type safety** — aligned with platform 2.1.2; zero TypeScript errors across all integration points
 - **CI gate** — all SDK tests, lint, and build checks passing
 
+---
+
 ## [2.1.1] - 2026-03-02
 
 ### Fixed
 
 - Version alignment with platform 2.1.1
 
+---
+
 ## [2.0.0] - 2026-03-01
 
 ### Breaking — EvalGate Rebrand
@@ -364,7 +431,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - error catalog stability + graceful handling of unknown codes
   - exports contract (retention visibility, 410 semantics)
 
---
+---
 
 ## [1.5.0] - 2026-02-18
 
@@ -412,6 +479,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Package hardening** — `files`, `module`, `sideEffects: false` for leaner npm publish
 - **CLI** — Passes `baseline` param to quality API for deterministic CI gates
 
+---
+
 ## [1.3.0] - 2025-10-21
 
 ### ✨ Added

package/README.md

@@ -3,7 +3,7 @@
 [![npm version](https://img.shields.io/npm/v/@evalgate/sdk.svg)](https://www.npmjs.com/package/@evalgate/sdk)
 [![npm downloads](https://img.shields.io/npm/dm/@evalgate/sdk.svg)](https://www.npmjs.com/package/@evalgate/sdk)
 [![TypeScript](https://img.shields.io/badge/TypeScript-strict-blue.svg)](https://www.typescriptlang.org/)
-[![SDK Tests](https://img.shields.io/badge/tests-172%20passed-brightgreen.svg)](#)
+[![SDK Tests](https://img.shields.io/badge/tests-159%20passed-brightgreen.svg)](#)
 [![Contract Version](https://img.shields.io/badge/report%20schema-v1-blue.svg)](#)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
@@ -366,6 +366,33 @@ import type {
 } from "@evalgate/sdk/regression";
 ```
 
+### Assertions — Sync (fast, heuristic) and Async (slow, LLM-backed)
+
+```typescript
+import {
+  // Sync — fast and approximate (no API key needed)
+  hasSentiment, hasNoToxicity, hasValidCodeSyntax,
+  containsLanguage, hasFactualAccuracy, hasNoHallucinations,
+  matchesSchema,
+  // Async — slow and accurate (requires API key)
+  configureAssertions, hasSentimentAsync, hasNoToxicityAsync,
+  hasValidCodeSyntaxAsync, containsLanguageAsync,
+  hasFactualAccuracyAsync, hasNoHallucinationsAsync,
+} from "@evalgate/sdk";
+
+// Configure once (or pass per-call)
+configureAssertions({ provider: "openai", apiKey: process.env.OPENAI_API_KEY });
+
+// Sync — fast, no network
+console.log(hasSentiment("I love this!", "positive"));   // true
+console.log(hasNoToxicity("Have a great day!"));          // true
+console.log(hasValidCodeSyntax("function f() {}", "js")); // true
+
+// Async — LLM-backed, context-aware
+console.log(await hasSentimentAsync("subtle irony...", "negative")); // true
+console.log(await hasNoToxicityAsync("sarcastic attack text"));       // false
+```
+
 ### Platform Client
 
 ```typescript
@@ -423,17 +450,19 @@ Your local `openAIChatEval` runs continue to work. No account cancellation. No d
 
 See [CHANGELOG.md](CHANGELOG.md) for the full release history.
 
-**v1.8.0** — `evalgate doctor` rewrite (9-check checklist), `evalgate explain` command, guided failure flow, CI template with doctor preflight
+**v2.2.3** — Bug-fix release. `RequestCache` default TTL, `EvalGateError` subclass prototype chain and `retryAfter` direct property, `autoPaginate` now returns `Promise<T[]>` (new `autoPaginateGenerator` for streaming), `createEvalRuntime` config-object overload, `defaultLocalExecutor` callable factory, `SnapshotManager.save` null/undefined safety, `compareSnapshots` loads both sides from disk, `AIEvalClient` default baseUrl → `https://api.evalgate.com`, `importData` optional-chaining guards, `toContainCode` raw-code detection, `hasReadabilityScore` `{min,max}` object form. 141 new regression tests.
 
-**v1.7.0** — `evalgate init` scaffolder, `evalgate upgrade --full`, `detectRunner()`, machine-readable gate output, init test matrix
+**v2.2.2** — 8 stub assertions replaced with real implementations (`hasSentiment` expanded lexicon, `hasNoToxicity` ~80-term blocklist, `hasValidCodeSyntax` real bracket balance, `containsLanguage` 12 languages + BCP-47, `hasFactualAccuracy`/`hasNoHallucinations` case-insensitive, `hasReadabilityScore` per-word syllable fix, `matchesSchema` JSON Schema support). Added LLM-backed `*Async` variants + `configureAssertions`. Fixed `importData` crash, `compareWithSnapshot` object coercion, `WorkflowTracer` defensive guard. 115 new tests.
 
-**v1.6.0** — `evalgate gate`, `evalgate baseline`, regression gate constants & types
+**v2.2.1** — `snapshot(name, output)` accepts objects; auto-serialized via `JSON.stringify`
 
-**v1.5.8** — secureRoute fix, test infra fixes, 304 handling fix
+**v2.2.0** — `expect().not` modifier, `hasPII()`, `defineSuite` object form, `snapshot` parameter order fix, `specId` collision fix
 
-**v1.5.5** — PASS/WARN/FAIL semantics, flake intelligence, golden regression suite
+**v1.8.0** — `evalgate doctor` rewrite (9-check checklist), `evalgate explain` command, guided failure flow, CI template with doctor preflight
+
+**v1.7.0** — `evalgate init` scaffolder, `evalgate upgrade --full`, `detectRunner()`, machine-readable gate output, init test matrix
 
-**v1.5.0** — GitHub annotations, `--onFail import`, `evalgate doctor`
+**v1.6.0** — `evalgate gate`, `evalgate baseline`, regression gate constants & types
 
 ## License
 

package/dist/assertions.d.ts

@@ -126,10 +126,11 @@ export declare class Expectation {
      */
     toBeBetween(min: number, max: number, message?: string): AssertionResult;
     /**
-     * Assert value contains code block
+     * Assert value contains code block or raw code
      * @example expect(output).toContainCode()
+     * @example expect(output).toContainCode('typescript')
      */
-    toContainCode(message?: string): AssertionResult;
+    toContainCode(language?: string, message?: string): AssertionResult;
     /**
      * Assert value is professional tone (no profanity)
      * @example expect(output).toBeProfessional()
@@ -193,18 +194,79 @@ export declare function notContainsPII(text: string): boolean;
  * if (hasPII(response)) throw new Error("PII leak");
  */
 export declare function hasPII(text: string): boolean;
+/**
+ * Lexicon-based sentiment check. **Fast and approximate** — suitable for
+ * low-stakes filtering or CI smoke tests. For production safety gates use
+ * {@link hasSentimentAsync} with an LLM provider for context-aware accuracy.
+ */
 export declare function hasSentiment(text: string, expected: "positive" | "negative" | "neutral"): boolean;
 export declare function similarTo(text1: string, text2: string, threshold?: number): boolean;
 export declare function withinRange(value: number, min: number, max: number): boolean;
 export declare function isValidEmail(email: string): boolean;
 export declare function isValidURL(url: string): boolean;
-export declare function hasNoHallucinations(text: string, groundTruth: string[]): boolean;
+/**
+ * Substring-based hallucination check — verifies each ground-truth fact
+ * appears verbatim in the text. **Fast and approximate**: catches missing
+ * facts but cannot detect paraphrased fabrications. Use
+ * {@link hasNoHallucinationsAsync} for semantic accuracy.
+ */
+export declare function hasNoHallucinations(text: string, groundTruth?: string[]): boolean;
 export declare function matchesSchema(value: unknown, schema: Record<string, unknown>): boolean;
-export declare function hasReadabilityScore(text: string, minScore: number): boolean;
+export declare function hasReadabilityScore(text: string, minScore: number | {
+    min?: number;
+    max?: number;
+}): boolean;
+/**
+ * Keyword-frequency language detector supporting 12 languages.
+ * **Fast and approximate** — detects the most common languages reliably
+ * but may struggle with short texts or closely related languages.
+ * Use {@link containsLanguageAsync} for reliable detection of any language.
+ */
 export declare function containsLanguage(text: string, language: string): boolean;
+/**
+ * Substring-based factual accuracy check. **Fast and approximate** — verifies
+ * each fact string appears in the text but cannot reason about meaning or
+ * paraphrasing. Use {@link hasFactualAccuracyAsync} for semantic accuracy.
+ */
 export declare function hasFactualAccuracy(text: string, facts: string[]): boolean;
 export declare function respondedWithinTime(startTime: number, maxMs: number): boolean;
+/**
+ * Blocklist-based toxicity check (~80 terms across 9 categories).
+ * **Fast and approximate** — catches explicit harmful language but has
+ * inherent gaps and context-blind false positives. Do NOT rely on this
+ * alone for production content safety gates; use {@link hasNoToxicityAsync}
+ * with an LLM for context-aware moderation.
+ */
 export declare function hasNoToxicity(text: string): boolean;
-export declare function followsInstructions(text: string, instructions: string[]): boolean;
+export declare function followsInstructions(text: string, instructions: string | string[]): boolean;
 export declare function containsAllRequiredFields(obj: unknown, requiredFields: string[]): boolean;
+export interface AssertionLLMConfig {
+    provider: "openai" | "anthropic";
+    apiKey: string;
+    model?: string;
+    baseUrl?: string;
+}
+export declare function configureAssertions(config: AssertionLLMConfig): void;
+export declare function getAssertionConfig(): AssertionLLMConfig | null;
+/**
+ * LLM-backed sentiment check. **Slow and accurate** — uses an LLM to
+ * classify sentiment with full context awareness. Requires
+ * {@link configureAssertions} or an inline `config` argument.
+ * Falls back gracefully with a clear error if no API key is configured.
+ */
+export declare function hasSentimentAsync(text: string, expected: "positive" | "negative" | "neutral", config?: AssertionLLMConfig): Promise<boolean>;
+/**
+ * LLM-backed toxicity check. **Slow and accurate** — context-aware, handles
+ * sarcasm, implicit threats, and culturally specific harmful content that
+ * blocklists miss. Recommended for production content safety gates.
+ */
+export declare function hasNoToxicityAsync(text: string, config?: AssertionLLMConfig): Promise<boolean>;
+export declare function containsLanguageAsync(text: string, language: string, config?: AssertionLLMConfig): Promise<boolean>;
+export declare function hasValidCodeSyntaxAsync(code: string, language: string, config?: AssertionLLMConfig): Promise<boolean>;
+export declare function hasFactualAccuracyAsync(text: string, facts: string[], config?: AssertionLLMConfig): Promise<boolean>;
+/**
+ * LLM-backed hallucination check. **Slow and accurate** — detects fabricated
+ * claims even when they are paraphrased or contradict facts indirectly.
+ */
+export declare function hasNoHallucinationsAsync(text: string, groundTruth: string[], config?: AssertionLLMConfig): Promise<boolean>;
 export declare function hasValidCodeSyntax(code: string, language: string): boolean;