nexus-ai-pro 0.6.0 → 0.7.0

package/EXPLANATION.md

@@ -269,6 +269,7 @@ Evals:
 - operational metrics
 - RAG metrics
 - safety heuristics
+- optional LLM-as-judge scoring with rubrics and thresholds
 
 ## Voice Integrations
 
@@ -285,6 +286,7 @@ The basic operations are:
 - `ai.transcribe(...)` for speech-to-text
 - `ai.speak(...)` for text-to-speech
 - `ai.voice(...)` for the full voice pipeline
+- `ai.createVoiceSession(...)` for multi-turn call sessions with tools
 - `registerVoiceProvider(...)` for private, local, or hosted services
 
 The full voice turn is:
@@ -299,7 +301,17 @@ audio/transcript
 
 Because the completion step calls `ai.complete(...)`, it reuses routing, security, context-window compaction, token optimization, cache, retries, metrics, and tracing.
 
-Realtime sessions are still future work. That should be added as a separate `VoiceSession` abstraction instead of forcing realtime state into normal completion streams.
+`VoiceSession` adds state across caller turns. It accepts transcript or audio, builds prompts from any combination of `prompt`, `instructions`, and conditional `taskPrompts`, executes registered tools when the model requests app data, and can synthesize the final answer.
+
+Low-level realtime transport is still future work. That should be added as a bridge from telephony/WebSocket/SIP media events into `VoiceSession` instead of forcing socket state into normal completion streams.
+
+Phone-number telephony is a separate optional layer:
+
+- `nexus-ai-pro/telephony` provides provider-neutral call, webhook, and media-stream types.
+- `nexus-ai-pro/telephony/twilio` provides a Twilio adapter without adding the Twilio SDK as a required dependency.
+- `NexusAI` exposes helper methods for outbound calls, TwiML webhook responses, webhook validation, and media event normalization.
+
+This keeps a small transcription/TTS app small, while larger phone-agent apps can add Twilio or another carrier as needed.
 
 ## Choosing a Setup
 

package/NEXUS.md

@@ -83,6 +83,7 @@ Supporting modules:
 - Chain of Verification
 - self-consistency sampling
 - eval runner and metrics
+- optional LLM-as-judge eval scoring
 - workflow templates
 - web/search connector tools
 - provider conformance fixtures
@@ -95,6 +96,8 @@ Voice is implemented as an optional layer. Import size is controlled through sub
 
 - `nexus-ai-pro/voice` for provider-neutral types, errors, and `VoiceManager`
 - `nexus-ai-pro/voice/openai` for the OpenAI voice adapter
+- `nexus-ai-pro/telephony` for provider-neutral phone-call types and `TelephonyManager`
+- `nexus-ai-pro/telephony/twilio` for the Twilio phone-number adapter
 
 Current support:
 
@@ -102,21 +105,30 @@ Current support:
 - `ai.speak(...)`
 - `ai.voice(...)`
 - `ai.voiceTurn(...)`
+- `ai.createVoiceSession(...)`
 - `registerVoiceProvider(...)`
 - configurable `voice.providers`
 - custom voice providers
 - OpenAI voice provider adapter
 - full completion pipeline reuse after transcription
+- stateful `VoiceSession` turns with prompts, task prompts, tool calls, history, and optional speech
+- optional telephony providers for calls, webhook responses, webhook validation, and media-stream event parsing
 
 Useful test cases:
 
 - registered voice provider can transcribe
 - registered voice provider can synthesize speech
 - `ai.voice(...)` transcribes, appends transcript, calls `ai.complete(...)`, and optionally speaks
+- `VoiceSession` can accept transcript or audio, select matching task prompts, execute tools, and speak
 - `nexus-ai-pro/voice` imports without provider adapters
 - `nexus-ai-pro/voice/openai` imports separately
+- `nexus-ai-pro/voice/session` imports separately
 - type consumer can configure `VoiceConfig`
 - missing voice provider errors clearly
+- registered telephony provider can create calls
+- Twilio adapter can create inline-TwiML/media-stream calls
+- Twilio webhook signatures can be validated
+- Twilio media-stream messages can be normalized
 
 Example:
 
@@ -144,9 +156,16 @@ const result = await ai.voice({
 
 Still future work:
 
-- realtime `VoiceSession`
 - audio chunks in `NexusStream`
-- WebRTC/WebSocket/SIP session helpers
+- WebRTC/SIP session helpers
+- fully managed Twilio/OpenAI realtime bridge
+
+Telephony is intentionally separate from voice:
+
+- `voice` handles file/buffer/stream audio turns.
+- `VoiceSession` handles multi-turn call state and app/tool requests during a conversation.
+- `telephony` handles phone numbers, TwiML, provider webhooks, and media-stream protocol messages.
+- apps can use one, both, or neither.
 
 ## Context Window Testing
 

package/README.md

@@ -343,6 +343,62 @@ console.log(result.response.content);
 console.log(result.speech?.audio.data);
 ```
 
+### Voice Session With App Requests
+
+Use `VoiceSession` when a call has multiple turns, should keep history, and may need to call your app while talking.
+
+The prompt setup is flexible:
+
+- `prompt` describes the assistant or business role.
+- `instructions` describe behavior.
+- `taskPrompts` add conditional instructions when the caller asks about something specific.
+- `tools` let the model call your app, such as booking, CRM, billing, or inventory APIs.
+
+```ts
+import { tool } from 'nexus-ai-pro';
+
+const session = ai.createVoiceSession({
+  model: 'auto',
+  prompt: 'You are a phone booking assistant for a small clinic.',
+  instructions: [
+    'Keep answers short and natural for a phone call.',
+    'Ask one follow-up question at a time.',
+  ],
+  taskPrompts: [
+    {
+      name: 'booking',
+      when: ['book', 'appointment', 'free slot'],
+      instructions: 'When the caller asks about availability, call check_free_slots before offering times.',
+      tools: ['check_free_slots'],
+    },
+  ],
+  tools: [
+    tool({
+      name: 'check_free_slots',
+      description: 'Check available booking slots for a date.',
+      parameters: {
+        type: 'object',
+        properties: { date: { type: 'string' } },
+        required: ['date'],
+      },
+      execute: async ({ date }) => bookingApi.freeSlots(String(date)),
+    }),
+  ],
+  toolSelection: 'task',
+  speech: { provider: 'openai', voice: 'alloy', format: 'mp3' },
+});
+
+const turn = await session.handleTurn({
+  transcript: 'Do you have any free slots tomorrow?',
+});
+
+console.log(turn.response.content);
+console.log(turn.toolSteps);
+console.log(turn.speech?.audio.data);
+```
+
+You can use only `prompt`, only `instructions`, only `taskPrompts`, or all of them together. For phone calls through Twilio media streams, parse incoming audio/media events in `telephony`, feed complete caller turns into `session.handleTurn(...)`, then send the returned speech audio back through your call transport.
+
 ### Custom Voice Provider
 
 Bring any private, local, or hosted voice service:
@@ -369,7 +425,62 @@ ai.registerVoiceProvider('private-voice', {
 });
 ```
 
-Realtime voice sessions and audio chunks inside `NexusStream` are still future work; the current voice layer covers transcription, speech generation, and full request/response voice turns.
+Low-level realtime transport adapters are still future work; the current voice layer covers transcription, speech generation, full request/response voice turns, and stateful voice sessions with tool calls.
+
+## Phone Number Telephony
+
+Phone-number support is optional and separate from the voice layer. Use `voice` for audio transcription/TTS, and use `telephony` when calls, webhooks, TwiML, or media-stream events are involved.
+
+```ts
+import { TelephonyManager } from 'nexus-ai-pro/telephony';
+import { TwilioTelephonyProvider } from 'nexus-ai-pro/telephony/twilio';
+```
+
+Configure only the provider you need:
+
+```ts
+const ai = new NexusAI({
+  providers: { openai: { apiKey: process.env.OPENAI_API_KEY! } },
+  telephony: {
+    defaultProvider: 'twilio',
+    providers: {
+      twilio: new TwilioTelephonyProvider({
+        accountSid: process.env.TWILIO_ACCOUNT_SID!,
+        authToken: process.env.TWILIO_AUTH_TOKEN!,
+      }),
+    },
+  },
+});
+```
+
+Generate a Twilio webhook response for an inbound phone number:
+
+```ts
+const response = await ai.createTelephonyResponse({
+  say: 'Thanks for calling. Connecting you now.',
+  stream: {
+    url: 'wss://voice.example.com/twilio',
+    mode: 'bidirectional',
+    parameters: { tenant: 'acme' },
+  },
+});
+
+return new Response(response.body, {
+  headers: { 'content-type': response.contentType },
+});
+```
+
+Start an outbound call with either a webhook, inline TwiML, an application SID, or a media-stream URL:
+
+```ts
+const call = await ai.createCall({
+  to: '+15551230000',
+  from: '+15557650000',
+  mediaStreamUrl: 'wss://voice.example.com/twilio',
+});
+```
+
+For smaller apps, skip Twilio entirely and register your own `TelephonyProvider`.
 
 ## Security
 
@@ -435,9 +546,35 @@ Available runtime helpers:
 - batch completion with concurrency control
 - in-memory job queue
 - Redis and BullMQ queue adapters
-- eval runner with quality, operations, RAG, and safety metrics
+- eval runner with quality, operations, RAG, safety metrics, and optional LLM-as-judge scoring
+- voice sessions for multi-turn calls with conditional task prompts and app tools
 - workflow templates for RAG answers, extraction, classification, comparison, support, sales, legal review, and code review
 
+Use a plain assertion for small evals, or add an LLM judge when a rubric is more useful than exact matching:
+
+```ts
+import { LLMJudge } from 'nexus-ai-pro/evals';
+
+const judge = new LLMJudge({
+  client: ai,
+  model: 'openai/gpt-4.1-mini',
+  rubric: 'Score whether the answer is correct, concise, and grounded in the supplied context.',
+  passThreshold: 0.7,
+});
+
+const run = await ai.runEvals([
+  {
+    name: 'support answer quality',
+    request: {
+      model: 'auto',
+      messages: [{ role: 'user', content: 'Explain our refund policy.' }],
+    },
+    expected: 'Refunds are available within 30 days.',
+    judge: judge.asEvalJudge(),
+  },
+]);
+```
+
 ## Observability and Reliability
 
 ```ts

package/dist/core/nexus.d.ts

@@ -3,9 +3,11 @@ import type { CompletionRequest } from '../types/messages.js';
 import type { NexusPlan } from '../types/planning.js';
 import type { NexusResponse, NexusStream } from '../types/response.js';
 import type { AgentConfig, AgentResult } from '../types/agent.js';
-import type { SpeechRequest, SpeechResponse, TranscriptionRequest, TranscriptionResponse, VoiceProvider, VoiceTurnRequest, VoiceTurnResponse } from '../types/voice.js';
+import type { SpeechRequest, SpeechResponse, TranscriptionRequest, TranscriptionResponse, VoiceProvider, VoiceSessionConfig, VoiceTurnRequest, VoiceTurnResponse } from '../types/voice.js';
+import type { CreateCallRequest, CreateCallResponse, TelephonyMediaStreamEvent, TelephonyOutboundAudioMessage, TelephonyProvider, TelephonyResponseRequest, TelephonyWebhookResponse, TelephonyWebhookValidationRequest } from '../types/telephony.js';
 import type { PipelineStep } from '../pipeline/types.js';
 import { BaseProvider } from '../providers/base.js';
+import { VoiceSession } from '../voice/index.js';
 import { type VerificationOptions } from '../hallucination/verification.js';
 import { type SelfConsistencyOptions } from '../hallucination/consistency.js';
 import { type BatchOptions, type BatchItemResult } from '../jobs/batch.js';
@@ -21,6 +23,7 @@ export declare class NexusAI {
     private security;
     private contextWindow;
     private voiceManager;
+    private telephonyManager;
     private optimizer;
     private cache;
     private auditLogger;
@@ -40,12 +43,24 @@ export declare class NexusAI {
     speak(request: SpeechRequest): Promise<SpeechResponse>;
     voice(request: VoiceTurnRequest): Promise<VoiceTurnResponse>;
     voiceTurn(request: VoiceTurnRequest): Promise<VoiceTurnResponse>;
+    createVoiceSession(config: VoiceSessionConfig): VoiceSession;
+    createCall(request: CreateCallRequest): Promise<CreateCallResponse>;
+    createTelephonyResponse(request: TelephonyResponseRequest): Promise<TelephonyWebhookResponse>;
+    validateTelephonyWebhook(request: TelephonyWebhookValidationRequest): Promise<boolean>;
+    parseTelephonyMediaEvent(providerName: string, message: string | Record<string, unknown>): TelephonyMediaStreamEvent | undefined;
+    formatTelephonyAudioMessage(providerName: string, streamId: string, payload: string, options?: {
+        event?: 'media' | 'mark' | 'clear';
+        markName?: string;
+    }): TelephonyOutboundAudioMessage;
     registerProvider(name: string, provider: BaseProvider): this;
     registerVoiceProvider(name: string, provider: VoiceProvider): this;
+    registerTelephonyProvider(name: string, provider: TelephonyProvider): this;
     hasProvider(name: string): boolean;
     hasVoiceProvider(name: string): boolean;
+    hasTelephonyProvider(name: string): boolean;
     listProviders(): string[];
     listVoiceProviders(): string[];
+    listTelephonyProviders(): string[];
     use(step: PipelineStep): this;
     batchComplete(requests: CompletionRequest[], options?: BatchOptions): Promise<Array<BatchItemResult<NexusResponse>>>;
     createQueue(options?: QueueOptions): JobQueue<CompletionRequest, NexusResponse>;

package/dist/core/nexus.js

@@ -11,6 +11,7 @@ import { Logger } from '../utils/logger.js';
 import { SecurityPipeline } from '../security/index.js';
 import { ContextWindowManager } from '../context/index.js';
 import { VoiceManager } from '../voice/index.js';
+import { TelephonyManager } from '../telephony/index.js';
 import { TokenOptimizer } from '../optimizer/index.js';
 import { AgentLoop } from '../agent/loop.js';
 import { resolveModel } from '../models/registry.js';
@@ -39,6 +40,7 @@ export class NexusAI {
     security;
     contextWindow;
     voiceManager;
+    telephonyManager;
     optimizer;
     cache;
     auditLogger;
@@ -57,6 +59,7 @@ export class NexusAI {
         this.security = new SecurityPipeline(this.config.security || 'standard');
         this.contextWindow = new ContextWindowManager(this.config.contextWindow || {});
         this.voiceManager = new VoiceManager(this.config.voice || {});
+        this.telephonyManager = new TelephonyManager(this.config.telephony || {});
         this.optimizer = new TokenOptimizer(this.config.tokenOptimizer || {});
         this.cache = new MemoryCache(this.config.cache?.maxEntries || 500);
         this.auditLogger = new AuditLogger(this.config.auditLog);
@@ -320,6 +323,24 @@ export class NexusAI {
     async voiceTurn(request) {
         return this.voiceManager.runTurn(request, this);
     }
+    createVoiceSession(config) {
+        return this.voiceManager.createSession(config, this);
+    }
+    async createCall(request) {
+        return this.telephonyManager.createCall(request);
+    }
+    async createTelephonyResponse(request) {
+        return this.telephonyManager.createWebhookResponse(request);
+    }
+    async validateTelephonyWebhook(request) {
+        return this.telephonyManager.validateWebhook(request);
+    }
+    parseTelephonyMediaEvent(providerName, message) {
+        return this.telephonyManager.parseMediaStreamEvent(providerName, message);
+    }
+    formatTelephonyAudioMessage(providerName, streamId, payload, options) {
+        return this.telephonyManager.formatAudioMessage(providerName, streamId, payload, options);
+    }
     registerProvider(name, provider) {
         this.providers.set(name, provider);
         return this;
@@ -328,18 +349,28 @@ export class NexusAI {
         this.voiceManager.registerProvider(name, provider);
         return this;
     }
+    registerTelephonyProvider(name, provider) {
+        this.telephonyManager.registerProvider(name, provider);
+        return this;
+    }
     hasProvider(name) {
         return this.providers.has(name);
     }
     hasVoiceProvider(name) {
         return this.voiceManager.hasProvider(name);
     }
+    hasTelephonyProvider(name) {
+        return this.telephonyManager.hasProvider(name);
+    }
     listProviders() {
         return [...this.providers.keys()];
     }
     listVoiceProviders() {
         return this.voiceManager.listProviders();
     }
+    listTelephonyProviders() {
+        return this.telephonyManager.listProviders();
+    }
     use(step) {
         this.pipeline.use(step);
         return this;

package/dist/evals/index.d.ts

@@ -0,0 +1,3 @@
+export { EvalRunner, type EvalCase, type EvalClient, type EvalJudge, type EvalJudgment, type EvalResult, type EvalRunResult, } from './runner.js';
+export { LLMJudge, createLLMJudgeEval, parseJudgeResponse, type JudgeClient, type LLMJudgeInput, type LLMJudgeInputMapper, type LLMJudgeOptions, type LLMJudgeResult, } from './judge.js';
+export { calculateEvalMetrics, exactMatch, f1Score, semanticSimilarity, passAtK, perplexity, tokensPerSecond, faithfulness, contextualPrecision, contextualRecall, toxicityScore, biasScore, policyAdherence, refusalRate, type EvalMetrics, type MetricInputs, type OperationalMetrics, type QualityMetrics, type RagMetrics, type SafetyMetrics, } from './metrics.js';

package/dist/evals/index.js

@@ -0,0 +1,3 @@
+export { EvalRunner, } from './runner.js';
+export { LLMJudge, createLLMJudgeEval, parseJudgeResponse, } from './judge.js';
+export { calculateEvalMetrics, exactMatch, f1Score, semanticSimilarity, passAtK, perplexity, tokensPerSecond, faithfulness, contextualPrecision, contextualRecall, toxicityScore, biasScore, policyAdherence, refusalRate, } from './metrics.js';

package/dist/evals/judge.d.ts

@@ -0,0 +1,50 @@
+import type { CompletionRequest } from '../types/messages.js';
+import type { NexusResponse } from '../types/response.js';
+import type { EvalCase, EvalJudge } from './runner.js';
+export interface JudgeClient<Response = NexusResponse> {
+    complete(request: CompletionRequest): Promise<Response>;
+}
+export interface LLMJudgeInput {
+    actual: string;
+    expected?: string;
+    query?: string;
+    rubric?: string;
+    contexts?: string[];
+    metadata?: Record<string, unknown>;
+}
+export interface LLMJudgeOptions<Response = NexusResponse> {
+    client: JudgeClient<Response>;
+    model: string;
+    rubric?: string;
+    systemPrompt?: string;
+    passThreshold?: number;
+    range?: [number, number];
+    temperature?: number;
+    maxTokens?: number;
+    metadata?: Record<string, unknown>;
+}
+export interface LLMJudgeResult {
+    score: number;
+    passed: boolean;
+    rationale?: string;
+    labels?: string[];
+    raw: unknown;
+    providerUsed?: string;
+    modelUsed?: string;
+}
+export type LLMJudgeInputMapper<Response> = (response: Response, testCase: EvalCase<Response>) => LLMJudgeInput | Promise<LLMJudgeInput>;
+export declare class LLMJudge<Response = NexusResponse> {
+    private options;
+    constructor(options: LLMJudgeOptions<Response>);
+    evaluate(input: LLMJudgeInput): Promise<LLMJudgeResult>;
+    asEvalJudge(mapper?: LLMJudgeInputMapper<Response>): EvalJudge<Response>;
+    private messages;
+}
+export declare function createLLMJudgeEval<Response = NexusResponse>(judge: LLMJudge<Response>, mapper?: LLMJudgeInputMapper<Response>): EvalJudge<Response>;
+export declare function parseJudgeResponse(content: string, options?: {
+    threshold?: number;
+    range?: [number, number];
+    raw?: unknown;
+    providerUsed?: string;
+    modelUsed?: string;
+}): LLMJudgeResult;

package/dist/evals/judge.js

@@ -0,0 +1,154 @@
+const DEFAULT_SYSTEM_PROMPT = [
+    'You are a careful evaluation judge.',
+    'Return only JSON with these fields: score, passed, rationale, labels.',
+    'Score must be numeric within the configured range.',
+].join(' ');
+export class LLMJudge {
+    options;
+    constructor(options) {
+        this.options = options;
+    }
+    async evaluate(input) {
+        const range = this.options.range || [0, 1];
+        const threshold = this.options.passThreshold ?? midpoint(range);
+        const response = await this.options.client.complete({
+            model: this.options.model,
+            messages: this.messages(input, range, threshold),
+            temperature: this.options.temperature ?? 0,
+            maxTokens: this.options.maxTokens ?? 512,
+            responseFormat: {
+                type: 'json_schema',
+                schema: {
+                    type: 'object',
+                    additionalProperties: false,
+                    properties: {
+                        score: { type: 'number' },
+                        passed: { type: 'boolean' },
+                        rationale: { type: 'string' },
+                        labels: { type: 'array', items: { type: 'string' } },
+                    },
+                    required: ['score', 'passed'],
+                },
+            },
+            metadata: {
+                ...this.options.metadata,
+                llmJudge: true,
+            },
+        });
+        return parseJudgeResponse(contentOf(response), {
+            threshold,
+            range,
+            raw: response,
+            providerUsed: providerOf(response),
+            modelUsed: modelOf(response),
+        });
+    }
+    asEvalJudge(mapper) {
+        return async (response, testCase) => {
+            const input = mapper
+                ? await mapper(response, testCase)
+                : {
+                    actual: contentOf(response),
+                    expected: testCase.expected,
+                    rubric: this.options.rubric,
+                };
+            return this.evaluate(input);
+        };
+    }
+    messages(input, range, threshold) {
+        const rubric = input.rubric || this.options.rubric || 'Judge whether the answer satisfies the request.';
+        const sections = [
+            `Rubric:\n${rubric}`,
+            `Score range: ${range[0]} to ${range[1]}`,
+            `Pass threshold: ${threshold}`,
+            input.query ? `User query:\n${input.query}` : undefined,
+            input.expected ? `Reference answer:\n${input.expected}` : undefined,
+            input.contexts?.length ? `Context:\n${input.contexts.join('\n\n')}` : undefined,
+            `Model answer:\n${input.actual}`,
+        ].filter(Boolean);
+        return [
+            { role: 'system', content: this.options.systemPrompt || DEFAULT_SYSTEM_PROMPT },
+            { role: 'user', content: sections.join('\n\n') },
+        ];
+    }
+}
+export function createLLMJudgeEval(judge, mapper) {
+    return judge.asEvalJudge(mapper);
+}
+export function parseJudgeResponse(content, options = {}) {
+    const range = options.range || [0, 1];
+    const threshold = options.threshold ?? midpoint(range);
+    const parsed = parseJsonObject(content);
+    const score = clamp(numberField(parsed.score) ?? numberFromText(content) ?? range[0], range);
+    const passed = typeof parsed.passed === 'boolean' ? parsed.passed : score >= threshold;
+    const labels = Array.isArray(parsed.labels)
+        ? parsed.labels.filter((label) => typeof label === 'string')
+        : undefined;
+    return {
+        score,
+        passed,
+        rationale: typeof parsed.rationale === 'string' ? parsed.rationale : undefined,
+        labels,
+        raw: options.raw ?? parsed,
+        providerUsed: options.providerUsed,
+        modelUsed: options.modelUsed,
+    };
+}
+function contentOf(response) {
+    if (typeof response === 'string')
+        return response;
+    if (response && typeof response === 'object' && 'content' in response) {
+        const content = response.content;
+        if (typeof content === 'string')
+            return content;
+    }
+    return JSON.stringify(response);
+}
+function providerOf(response) {
+    return response && typeof response === 'object'
+        ? response.meta?.providerUsed
+        : undefined;
+}
+function modelOf(response) {
+    return response && typeof response === 'object'
+        ? response.meta?.modelUsed
+        : undefined;
+}
+function parseJsonObject(content) {
+    const trimmed = content.trim();
+    const unfenced = trimmed
+        .replace(/^```(?:json)?\s*/i, '')
+        .replace(/\s*```$/, '')
+        .trim();
+    try {
+        const parsed = JSON.parse(unfenced);
+        return parsed && typeof parsed === 'object' && !Array.isArray(parsed)
+            ? parsed
+            : {};
+    }
+    catch {
+        return {};
+    }
+}
+function numberField(value) {
+    if (typeof value === 'number' && Number.isFinite(value))
+        return value;
+    if (typeof value === 'string') {
+        const parsed = Number(value);
+        return Number.isFinite(parsed) ? parsed : undefined;
+    }
+    return undefined;
+}
+function numberFromText(value) {
+    const match = value.match(/-?\d+(?:\.\d+)?/);
+    if (!match)
+        return undefined;
+    const parsed = Number(match[0]);
+    return Number.isFinite(parsed) ? parsed : undefined;
+}
+function clamp(value, [min, max]) {
+    return Math.min(Math.max(value, min), max);
+}
+function midpoint([min, max]) {
+    return min + (max - min) / 2;
+}

package/dist/evals/runner.d.ts

@@ -3,10 +3,21 @@ import type { EvalMetrics, MetricInputs } from './metrics.js';
 export interface EvalClient<Response = unknown> {
     complete(request: CompletionRequest): Promise<Response>;
 }
+export interface EvalJudgment {
+    score: number;
+    passed: boolean;
+    rationale?: string;
+    labels?: string[];
+    raw?: unknown;
+    providerUsed?: string;
+    modelUsed?: string;
+}
+export type EvalJudge<Response = unknown> = (response: Response, testCase: EvalCase<Response>) => boolean | EvalJudgment | Promise<boolean | EvalJudgment>;
 export interface EvalCase<Response = unknown> {
     name: string;
     request: CompletionRequest;
-    assert: (response: Response) => boolean | Promise<boolean>;
+    assert?: (response: Response) => boolean | Promise<boolean>;
+    judge?: EvalJudge<Response>;
     expected?: string;
     metrics?: (response: Response) => MetricInputs | Promise<MetricInputs>;
     tags?: string[];
@@ -17,6 +28,7 @@ export interface EvalResult<Response = unknown> {
     durationMs: number;
     response?: Response;
     metrics?: EvalMetrics;
+    judgment?: EvalJudgment;
     error?: string;
     tags?: string[];
 }

package/dist/evals/runner.js

@@ -11,7 +11,14 @@ export class EvalRunner {
             const caseStarted = Date.now();
             try {
                 const response = await this.client.complete(testCase.request);
-                const passed = await testCase.assert(response);
+                if (!testCase.assert && !testCase.judge) {
+                    throw new Error(`Eval case "${testCase.name}" requires assert or judge`);
+                }
+                const assertPassed = testCase.assert ? await testCase.assert(response) : true;
+                const judgment = testCase.judge
+                    ? normalizeJudgment(await testCase.judge(response, testCase))
+                    : undefined;
+                const passed = assertPassed && (judgment?.passed ?? true);
                 const metrics = testCase.metrics
                     ? await calculateEvalMetrics(await testCase.metrics(response))
                     : undefined;
@@ -21,6 +28,7 @@ export class EvalRunner {
                     durationMs: Date.now() - caseStarted,
                     response,
                     metrics,
+                    judgment,
                     tags: testCase.tags,
                 });
             }
@@ -45,3 +53,12 @@ export class EvalRunner {
         };
     }
 }
+function normalizeJudgment(value) {
+    if (typeof value === 'boolean') {
+        return {
+            score: value ? 1 : 0,
+            passed: value,
+        };
+    }
+    return value;
+}