qualifire 1.2.2 → 1.4.0

package/README.md

@@ -1,4 +1,4 @@
-Qualifire
+# Qualifire SDK
 
 [![CodeQL](https://github.com/qualifire-dev/qualifire-typescript-sdk/actions/workflows/codeql-analysis.yml/badge.svg)](https://github.com/qualifire-dev/qualifire-typescript-sdk/actions/workflows/codeql-analysis.yml)
 [![Release](https://github.com/qualifire-dev/qualifire-typescript-sdk/actions/workflows/release.yml/badge.svg)](https://github.com/qualifire-dev/qualifire-typescript-sdk/actions/workflows/release.yml)
@@ -7,9 +7,7 @@ Qualifire
 [![Commitizen Friendly][commitizen-img]][commitizen-url]
 [![Semantic Release][semantic-release-img]][semantic-release-url]
 
-# Qualifire SDK
-
-This is the official SDK for interacting with the Qualifire API.
+The official TypeScript SDK for evaluating LLM outputs with [Qualifire](https://qualifire.ai). Detect hallucinations, prompt injections, PII leakage, content policy violations, and more.
 
 ## Installation
 
@@ -17,79 +15,238 @@ This is the official SDK for interacting with the Qualifire API.
 npm install qualifire
 ```
 
-## usage
+## Quick Start
+
+```typescript
+import { Qualifire } from 'qualifire';
+import OpenAI from 'openai';
+
+const qualifire = new Qualifire({ apiKey: 'your-api-key' });
+const openai = new OpenAI();
+
+// Make your LLM call
+const request = {
+  model: 'gpt-4o',
+  messages: [
+    { role: 'system', content: 'You are a helpful assistant.' },
+    { role: 'user', content: 'What is the capital of France?' },
+  ],
+};
+
+const response = await openai.chat.completions.create(request);
 
-First, import the `Qualifire` class from the SDK:
+// Evaluate the response
+const evaluation = await qualifire.evaluate({
+  framework: 'openai',
+  request,
+  response,
+  hallucinationsCheck: true,
+  groundingCheck: true,
+});
 
-```javascript
-import { Qualifire } from 'qualifire-sdk';
+console.log(evaluation);
+// {
+//   status: 'passed',
+//   score: 100,
+//   evaluationResults: [...]
+// }
 ```
 
-Then, create a new instance of the Qualifire class, passing your API key and the base URL of the Qualifire API:
+## Supported Frameworks
 
-```javascript
-const qualifire = new Qualifire({
-  apiKey: 'your-api-key',
+| Framework | Value | SDK |
+|-----------|-------|-----|
+| OpenAI | `openai` | `openai` (Chat Completions & Responses API) |
+| Anthropic Claude | `claude` | `@anthropic-ai/sdk` |
+| Google Gemini | `gemini` | `@google/genai` |
+| Vercel AI SDK | `vercelai` | `ai` |
+
+All frameworks support both streaming and non-streaming responses.
+
+## Available Evaluation Checks
+
+| Check | Parameter | Description |
+|-------|-----------|-------------|
+| Hallucinations | `hallucinationsCheck` | Detect fabricated information |
+| Grounding | `groundingCheck` | Verify responses are grounded in context |
+| Prompt Injections | `promptInjections` | Detect prompt injection attempts |
+| PII Detection | `piiCheck` | Identify personally identifiable information |
+| Content Moderation | `contentModerationCheck` | Flag harmful content |
+| Instructions Following | `instructionsFollowingCheck` | Verify adherence to system instructions |
+| Tool Selection Quality | `toolSelectionQualityCheck` | Evaluate tool/function call accuracy |
+| Custom Assertions | `assertions` | Array of custom assertion strings |
+
+## Framework Examples
+
+### OpenAI
+
+```typescript
+// Chat Completions API
+const request = {
+  model: 'gpt-4o',
+  messages: [{ role: 'user', content: 'Hello!' }],
+};
+const response = await openai.chat.completions.create(request);
+
+await qualifire.evaluate({
+  framework: 'openai',
+  request,
+  response,
+  hallucinationsCheck: true,
+});
+
+// Streaming
+const streamRequest = { ...request, stream: true };
+const stream = await openai.chat.completions.create(streamRequest);
+
+const chunks = [];
+for await (const chunk of stream) {
+  chunks.push(chunk);
+}
+
+await qualifire.evaluate({
+  framework: 'openai',
+  request: streamRequest,
+  response: chunks,
+  hallucinationsCheck: true,
 });
 ```
 
-ℹ️ There are default environment variables if you prefer to set it that way `QUALIFIRE_API_KEY`
+### Anthropic Claude
 
-You can now use the `evaluate` method to evaluate input and output data:
+```typescript
+import Anthropic from '@anthropic-ai/sdk';
 
-```javascript
-const input = {
-  model: 'gpt-3.5-turbo',
-  messages: [
-    {
-      role: 'user',
-      content: 'this is my awesome request',
-    },
-  ],
+const anthropic = new Anthropic();
+
+const request = {
+  model: 'claude-sonnet-4-20250514',
+  max_tokens: 1024,
+  messages: [{ role: 'user', content: 'Hello!' }],
 };
+const response = await anthropic.messages.create(request);
 
-const output = await openai.chat.completions.create(input);
+await qualifire.evaluate({
+  framework: 'claude',
+  request,
+  response,
+  promptInjections: true,
+});
+```
+
+### Google Gemini
+
+```typescript
+import { GoogleGenAI } from '@google/genai';
+
+const genai = new GoogleGenAI({ apiKey: 'your-key' });
+
+const request = {
+  model: 'gemini-2.0-flash',
+  contents: [{ role: 'user', parts: [{ text: 'Hello!' }] }],
+};
+const response = await genai.models.generateContent(request);
+
+await qualifire.evaluate({
+  framework: 'gemini',
+  request,
+  response,
+  contentModerationCheck: true,
+});
+```
+
+### Vercel AI SDK
+
+```typescript
+import { generateText } from 'ai';
+import { openai } from '@ai-sdk/openai';
+
+const request = {
+  model: openai('gpt-4o'),
+  prompt: 'Hello!',
+};
+const response = await generateText(request);
 
-const evaluationResponse = await qualifire.evaluate(input, output); // This will block until the evaluation is done
-console.log(evaluationResponse);
+await qualifire.evaluate({
+  framework: 'vercelai',
+  request,
+  response,
+  piiCheck: true,
+});
 ```
 
-### Non-blocking execution
+## Direct Message Mode
 
-In case you want to trigger a completely async evaluation (to view in qualifire's UI) simply add the `{async: true}` option to your call.
+For cases where you don't use a supported framework, pass messages directly:
 
-```javascript
-const input = {
-  model: 'gpt-3.5-turbo',
+```typescript
+await qualifire.evaluate({
   messages: [
-    {
-      role: 'user',
-      content: 'this is my awesome request',
-    },
+    { role: 'user', content: 'What is 2+2?' },
+    { role: 'assistant', content: 'The answer is 4.' },
   ],
-};
+  hallucinationsCheck: true,
+  groundingCheck: true,
+});
+```
+
+## Invoke Pre-configured Evaluations
 
-const output = await openai.chat.completions.create(input);
+Run evaluations configured in the Qualifire dashboard:
 
-const evaluationResponse = await qualifire.evaluate(input, output, {
-  async: true,
-}); // This will block until the evaluation is done
-console.log(evaluationResponse);
+```typescript
+const result = await qualifire.invokeEvaluation({
+  input: 'What is the capital of France?',
+  output: 'Paris is the capital of France.',
+  evaluationId: 'eval-123',
+});
 ```
 
-Evaluates the input and output using the Qualifire API. Returns a promise that resolves to the evaluation response, or undefined if async is true.
+## Configuration
 
+### Constructor Options
+
+```typescript
+const qualifire = new Qualifire({
+  apiKey: 'your-api-key',      // Required (or set QUALIFIRE_API_KEY env var)
+  baseUrl: 'https://...',      // Optional, defaults to https://proxy.qualifire.ai
+});
+```
+
+### Environment Variables
+
+| Variable | Description |
+|----------|-------------|
+| `QUALIFIRE_API_KEY` | API key for authentication |
+| `QUALIFIRE_BASE_URL` | Override the API base URL |
+
+## Response Format
+
+```typescript
+interface EvaluationResponse {
+  status: 'passed' | 'failed';
+  score: number;  // 0-100
+  evaluationResults: Array<{
+    type: string;
+    results: Array<{
+      name: string;
+      score: number;
+      label: string;
+      confidence_score: number;
+      reason: string;
+    }>;
+  }>;
+}
+```
 
+## License
 
+MIT
 
-[build-img]: https://github.com/qualifire-dev/develop/qualifire-typescript-sdk/actions/workflows/release.yml/badge.svg
-[build-url]: https://github.com/qualifire-dev/qualifire-typescript-sdk/actions/workflows/release.yml
-[downloads-img]: https://img.shields.io/npm/dt/main/qualifire
-[npm-url]: https://www.npmjs.com/package/qualifire
-[issues-img]: https://img.shields.io/github/issues/qualifire-dev/develop/qualifire-typescript-sdk
+[issues-img]: https://img.shields.io/github/issues/qualifire-dev/qualifire-typescript-sdk
 [issues-url]: https://github.com/qualifire-dev/qualifire-typescript-sdk/issues
-[codecov-img]: https://codecov.io/gh/qualifire-dev/develop/qualifire-typescript-sdk/branch/main/graph/badge.svg
-[codecov-url]: https://codecov.io/gh/qualifire-dev/develop/qualifire-typescript-sdk
+[codecov-img]: https://codecov.io/gh/qualifire-dev/qualifire-typescript-sdk/branch/main/graph/badge.svg
+[codecov-url]: https://codecov.io/gh/qualifire-dev/qualifire-typescript-sdk
 [semantic-release-img]: https://img.shields.io/badge/%20%20%F0%9F%93%A6%F0%9F%9A%80-semantic--release-e10079.svg
 [semantic-release-url]: https://github.com/semantic-release/semantic-release
 [commitizen-img]: https://img.shields.io/badge/commitizen-friendly-brightgreen.svg

package/lib/frameworks/canonical.d.ts

@@ -0,0 +1,4 @@
+import { EvaluationProxyAPIRequest } from '../types';
+export interface CanonicalEvaluationStrategy<RequestType, ResponseType> {
+    convertToQualifireEvaluationRequest(request: RequestType, response: ResponseType): Promise<EvaluationProxyAPIRequest>;
+}

package/lib/frameworks/canonical.js

@@ -0,0 +1,2 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });

package/lib/frameworks/claude/claude-converter.d.ts

@@ -0,0 +1,15 @@
+import { Message, type MessageCreateParams, type MessageStreamParams } from '@anthropic-ai/sdk/resources';
+import { RawMessageStreamEvent } from '@anthropic-ai/sdk/resources/messages';
+import { EvaluationProxyAPIRequest } from '../../types';
+import { CanonicalEvaluationStrategy } from '../canonical';
+type AnthropicCreateAPIResponsesType = Message | RawMessageStreamEvent;
+type AnthropicAPIRequestsType = MessageCreateParams;
+type AnthropicAPIResponsesType = AnthropicCreateAPIResponsesType | MessageStreamParams;
+export declare class ClaudeCanonicalEvaluationStrategy implements CanonicalEvaluationStrategy<AnthropicAPIRequestsType, AnthropicAPIResponsesType> {
+    convertToQualifireEvaluationRequest(request: AnthropicAPIRequestsType, response: AnthropicAPIResponsesType): Promise<EvaluationProxyAPIRequest>;
+    convertRequest(request: any): EvaluationProxyAPIRequest;
+    private handleStreaming;
+    private handleNonStreamingResponse;
+    private convertClaudeMessagesToLLMMessages;
+}
+export {};

package/lib/frameworks/claude/claude-converter.js

@@ -0,0 +1,229 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.ClaudeCanonicalEvaluationStrategy = void 0;
+class ClaudeCanonicalEvaluationStrategy {
+    async convertToQualifireEvaluationRequest(request, response) {
+        const { messages: requestMessages, available_tools: requestAvailableTools, } = this.convertRequest(request);
+        const messages = requestMessages || [];
+        const availableTools = requestAvailableTools || [];
+        // Avoid undefined response
+        if (!response) {
+            return {
+                messages,
+                available_tools: availableTools,
+            };
+        }
+        // Check if response is streaming or non-streaming
+        if (Array.isArray(response)) {
+            const streamingResultMessages = await this.handleStreaming(response);
+            messages.push(...streamingResultMessages);
+        }
+        else {
+            const nonStreamingResultMessages = await this.handleNonStreamingResponse(response);
+            messages.push(...nonStreamingResultMessages);
+        }
+        return {
+            messages,
+            available_tools: availableTools,
+        };
+    }
+    convertRequest(request) {
+        const messages = [];
+        const availableTools = [];
+        // Handle Claude system message first (if present)
+        if (request?.system) {
+            messages.push({
+                role: 'system',
+                content: request.system,
+            });
+        }
+        // Handle Claude request messages
+        if (request?.messages) {
+            messages.push(...this.convertClaudeMessagesToLLMMessages(request.messages));
+        }
+        // Handle tools
+        if (request?.tools) {
+            for (const tool of request.tools) {
+                availableTools.push({
+                    name: tool.name,
+                    description: tool.description,
+                    parameters: tool.input_schema?.properties || {},
+                });
+            }
+        }
+        return {
+            messages,
+            available_tools: availableTools,
+        };
+    }
+    async handleStreaming(response) {
+        const messages = [];
+        let role;
+        let accumulatedContent = [];
+        let accumulatedToolName;
+        let accumulatedToolId;
+        let accumulatedToolInput = [];
+        for (const responseEvent of response) {
+            switch (responseEvent.type) {
+                case 'message_start':
+                    const rawMessageStartEvent = responseEvent;
+                    role = rawMessageStartEvent.message.role;
+                    accumulatedContent = [];
+                    accumulatedToolName = undefined;
+                    accumulatedToolId = undefined;
+                    accumulatedToolInput = [];
+                    break;
+                case 'content_block_start':
+                    const rawContentBlockStartEvent = responseEvent;
+                    switch (rawContentBlockStartEvent.content_block.type) {
+                        case 'text':
+                            const textBlock = rawContentBlockStartEvent.content_block;
+                            accumulatedContent.push(textBlock.text);
+                            break;
+                        case 'tool_use':
+                            const toolUseBlock = rawContentBlockStartEvent.content_block;
+                            accumulatedToolId = toolUseBlock.id;
+                            accumulatedToolName = toolUseBlock.name;
+                            accumulatedToolInput = [];
+                            break;
+                        case 'thinking':
+                            const thinkingBlock = rawContentBlockStartEvent.content_block;
+                            accumulatedContent.push(thinkingBlock.thinking);
+                            break;
+                        default:
+                            console.debug(`Invalid content block type: ${responseEvent}`);
+                    }
+                    break;
+                case 'content_block_delta':
+                    const rawContentBlockDeltaEvent = responseEvent;
+                    switch (rawContentBlockDeltaEvent.delta.type) {
+                        case 'text_delta':
+                            const textDelta = rawContentBlockDeltaEvent.delta;
+                            accumulatedContent.push(textDelta.text);
+                            break;
+                        case 'input_json_delta':
+                            const inputJsonDelta = rawContentBlockDeltaEvent.delta;
+                            accumulatedToolInput.push(inputJsonDelta.partial_json);
+                            break;
+                        default:
+                            console.debug(`Invalid delta type: ${rawContentBlockDeltaEvent}`);
+                    }
+                    break;
+                case 'message_stop':
+                    let finalContent;
+                    if (accumulatedContent.length > 0) {
+                        finalContent = accumulatedContent.join('').trim();
+                    }
+                    let finalTool;
+                    if (accumulatedToolName) {
+                        finalTool = {
+                            id: accumulatedToolId,
+                            name: accumulatedToolName,
+                            arguments: JSON.parse(accumulatedToolInput.join('')),
+                        };
+                    }
+                    ;
+                    if (!role) {
+                        console.debug('role was not set');
+                        continue;
+                    }
+                    messages.push({
+                        role: role == 'model' ? 'assistant' : role,
+                        content: finalContent ?? undefined,
+                        tool_calls: finalTool ? [finalTool] : undefined,
+                    });
+                    role = undefined;
+                    accumulatedContent = [];
+                    accumulatedToolName = undefined;
+                    accumulatedToolId = undefined;
+                    accumulatedToolInput = [];
+                    break;
+                case 'content_block_stop':
+                case 'message_delta':
+                    break;
+                default:
+                    console.debug(`Invalid event: ${responseEvent}`);
+            }
+        }
+        return messages;
+    }
+    async handleNonStreamingResponse(response) {
+        const messages = [];
+        if (response.role !== 'assistant') {
+            throw new Error(`Response role must be 'assistant'. Make sure to use response 
+        from anthropic.messages.create() when not using streaming.`);
+        }
+        messages.push(...this.convertClaudeMessagesToLLMMessages([response]));
+        return messages;
+    }
+    // Claude-specific function to convert Response API messages to LLM messages
+    convertClaudeMessagesToLLMMessages(messages) {
+        const extractedMessages = [];
+        for (const message of messages) {
+            if (typeof message.content === 'string') {
+                const llmMessage = {
+                    role: message.role,
+                    content: message.content,
+                };
+                extractedMessages.push(llmMessage);
+                continue;
+            }
+            const aggregatedContent = [];
+            const aggregatedToolCalls = [];
+            let role = message.role;
+            if (!message.content) {
+                continue;
+            }
+            for (const part of message.content) {
+                switch (part.type) {
+                    case 'tool_use':
+                        const toolUseBlock = part;
+                        aggregatedToolCalls.push({
+                            name: toolUseBlock.name,
+                            arguments: toolUseBlock.input,
+                            id: toolUseBlock.id,
+                        });
+                        break;
+                    case 'tool_result':
+                        role = 'tool'; // Claude expects 'user' role for tool results. But Qualifire treats tool as results as it is sent from 'tool'
+                        const toolResultBlock = part;
+                        if (typeof toolResultBlock.content === 'string') {
+                            aggregatedContent.push(toolResultBlock.content);
+                        }
+                        else {
+                            toolResultBlock.content.filter(part => part.type === 'text').forEach(part => {
+                                const textPart = part;
+                                aggregatedContent.push(textPart.text);
+                            });
+                        }
+                        break;
+                    case 'text':
+                        const textBlock = part;
+                        aggregatedContent.push(textBlock.text);
+                        break;
+                    default:
+                        console.debug('Invalid Claude output: message - ' +
+                            JSON.stringify(message) +
+                            ' part - ' +
+                            JSON.stringify(part));
+                }
+            }
+            // If we accumulated aggregatedContent or aggregatedToolCalls, add the message
+            if (aggregatedContent.length > 0 || aggregatedToolCalls.length > 0) {
+                const accumulatedMessage = {
+                    role,
+                };
+                if (aggregatedContent.length > 0) {
+                    accumulatedMessage.content = aggregatedContent.join('');
+                }
+                // Only add aggregatedToolCalls property for assistant messages
+                if (aggregatedToolCalls.length > 0) {
+                    accumulatedMessage.tool_calls = aggregatedToolCalls;
+                }
+                extractedMessages.push(accumulatedMessage);
+            }
+        }
+        return extractedMessages;
+    }
+}
+exports.ClaudeCanonicalEvaluationStrategy = ClaudeCanonicalEvaluationStrategy;

package/lib/frameworks/gemini/gemini-converter.d.ts

@@ -0,0 +1,11 @@
+import { EvaluationProxyAPIRequest } from '../../types';
+import { CanonicalEvaluationStrategy } from '../canonical';
+type GeminiAICanonicalEvaluationStrategyResponse = any;
+type GeminiAICanonicalEvaluationStrategyRequest = any;
+export declare class GeminiAICanonicalEvaluationStrategy implements CanonicalEvaluationStrategy<GeminiAICanonicalEvaluationStrategyRequest, GeminiAICanonicalEvaluationStrategyResponse> {
+    convertToQualifireEvaluationRequest(request: GeminiAICanonicalEvaluationStrategyRequest, response: GeminiAICanonicalEvaluationStrategyResponse): Promise<EvaluationProxyAPIRequest>;
+    convertRequest(request: GeminiAICanonicalEvaluationStrategyRequest): Promise<EvaluationProxyAPIRequest>;
+    private handleNonStreamingResponse;
+    private handleStreaming;
+}
+export {};