@inference-gateway/sdk 0.2.0 → 0.3.2

package/CHANGELOG.md

@@ -1,3 +1,35 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+## [0.3.2](https://github.com/inference-gateway/typescript-sdk/compare/v0.3.1...v0.3.2) (2025-03-31)
+
+### 👷 CI
+
+* Add npm ci step to install project dependencies in release workflow ([84791b1](https://github.com/inference-gateway/typescript-sdk/commit/84791b1e4c319f91798c456c783ded6e22da8f81))
+
+## [0.3.1](https://github.com/inference-gateway/typescript-sdk/compare/v0.3.0...v0.3.1) (2025-03-31)
+
+### ♻️ Improvements
+
+* Make the SDK OpenAI compatible ([#2](https://github.com/inference-gateway/typescript-sdk/issues/2)) ([31657b3](https://github.com/inference-gateway/typescript-sdk/commit/31657b358f34ccc39acc5994248a95127f1ea46a))
+
+### 👷 CI
+
+* Update GitHub Actions release workflow to use GitHub App token and improve release handling ([14835e8](https://github.com/inference-gateway/typescript-sdk/commit/14835e8f9289314f34e711c02faf865ad9af6d66))
+* Update release configuration for semantic-release plugins and rules to be consistent with other repos ([20bd3f8](https://github.com/inference-gateway/typescript-sdk/commit/20bd3f82c68d0b1ee1d07b4fa75eb67524db4fb8))
+
+## [0.3.0](https://github.com/inference-gateway/typescript-sdk/compare/v0.2.0...v0.3.0) (2025-02-02)
+
+### ✨ Features
+
+* add streaming content functionality to InferenceGatewayClient and update README ([ba41d2d](https://github.com/inference-gateway/typescript-sdk/commit/ba41d2dc136b83372820af2aefa63969932e16f0))
+
+### 📚 Documentation
+
+* **fix:** Update examples in README.md ([4e972fc](https://github.com/inference-gateway/typescript-sdk/commit/4e972fc2c577f41b0b443f1c87cde7561717b577))
+* Update OpenAPI spec - download it from Inference-gateway ([9816b15](https://github.com/inference-gateway/typescript-sdk/commit/9816b151db6b48b04723f93b988daf83239a09df))
+
 ## [0.2.0](https://github.com/inference-gateway/typescript-sdk/compare/v0.1.6...v0.2.0) (2025-01-28)
 
 ### ✨ Features

package/README.md

@@ -1,15 +1,18 @@
-# Inference Gateway Typescript SDK
+# Inference Gateway TypeScript SDK
 
-An SDK written in Typescript for the [Inference Gateway](https://github.com/edenreich/inference-gateway).
+An SDK written in TypeScript for the [Inference Gateway](https://github.com/edenreich/inference-gateway).
 
-- [Inference Gateway Typescript SDK](#inference-gateway-typescript-sdk)
+- [Inference Gateway TypeScript SDK](#inference-gateway-typescript-sdk)
   - [Installation](#installation)
   - [Usage](#usage)
     - [Creating a Client](#creating-a-client)
-    - [Listing All Models](#listing-all-models)
-    - [List Models by Provider](#list-models-by-provider)
-    - [Generating Content](#generating-content)
+    - [Listing Models](#listing-models)
+    - [Creating Chat Completions](#creating-chat-completions)
+    - [Streaming Chat Completions](#streaming-chat-completions)
+    - [Tool Calls](#tool-calls)
+    - [Proxying Requests](#proxying-requests)
     - [Health Check](#health-check)
+    - [Creating a Client with Custom Options](#creating-a-client-with-custom-options)
   - [Contributing](#contributing)
   - [License](#license)
 
@@ -21,116 +24,199 @@ Run `npm i @inference-gateway/sdk`.
 
 ### Creating a Client
 
+```typescript
+import { InferenceGatewayClient } from '@inference-gateway/sdk';
+
+// Create a client with default options
+const client = new InferenceGatewayClient({
+  baseURL: 'http://localhost:8080/v1',
+  apiKey: 'your-api-key', // Optional
+});
+```
+
+### Listing Models
+
+To list all available models:
+
+```typescript
+import { InferenceGatewayClient, Provider } from '@inference-gateway/sdk';
+
+const client = new InferenceGatewayClient({
+  baseURL: 'http://localhost:8080/v1',
+});
+
+try {
+  // List all models
+  const models = await client.listModels();
+  console.log('All models:', models);
+
+  // List models from a specific provider
+  const openaiModels = await client.listModels(Provider.OpenAI);
+  console.log('OpenAI models:', openaiModels);
+} catch (error) {
+  console.error('Error:', error);
+}
+```
+
+### Creating Chat Completions
+
+To generate content using a model:
+
 ```typescript
 import {
   InferenceGatewayClient,
-  Message,
+  MessageRole,
   Provider,
 } from '@inference-gateway/sdk';
 
-async function main() {
-  const client = new InferenceGatewayClient('http://localhost:8080');
-
-  try {
-    // List available models
-    const models = await client.listModels();
-    models.forEach((providerModels) => {
-      console.log(`Provider: ${providerModels.provider}`);
-      providerModels.models.forEach((model) => {
-        console.log(`Model: ${model.id}`);
-      });
-    });
-
-    // Generate content
-    const response = await client.generateContent({
-      provider: Provider.Ollama,
-      model: 'llama2',
+const client = new InferenceGatewayClient({
+  baseURL: 'http://localhost:8080/v1',
+});
+
+try {
+  const response = await client.createChatCompletion(
+    {
+      model: 'gpt-4o',
       messages: [
         {
           role: MessageRole.System,
-          content: 'You are a helpful llama',
+          content: 'You are a helpful assistant',
         },
         {
           role: MessageRole.User,
           content: 'Tell me a joke',
         },
       ],
-    });
+    },
+    Provider.OpenAI
+  ); // Provider is optional
 
-    console.log('Response:', response);
-  } catch (error) {
-    console.error('Error:', error);
-  }
+  console.log('Response:', response.choices[0].message.content);
+} catch (error) {
+  console.error('Error:', error);
 }
-
-main();
 ```
 
-### Listing All Models
+### Streaming Chat Completions
 
-To list all available models from all providers, use the `listModels` method:
+To stream content from a model:
 
 ```typescript
+import {
+  InferenceGatewayClient,
+  MessageRole,
+  Provider,
+} from '@inference-gateway/sdk';
+
+const client = new InferenceGatewayClient({
+  baseURL: 'http://localhost:8080/v1',
+});
+
 try {
-  const models = await client.listModels();
-  models.forEach((providerModels) => {
-    console.log(`Provider: ${providerModels.provider}`);
-    providerModels.models.forEach((model) => {
-      console.log(`Model: ${model.id}`);
-    });
-  });
+  await client.streamChatCompletion(
+    {
+      model: 'llama-3.3-70b-versatile',
+      messages: [
+        {
+          role: MessageRole.User,
+          content: 'Tell me a story',
+        },
+      ],
+    },
+    {
+      onOpen: () => console.log('Stream opened'),
+      onContent: (content) => process.stdout.write(content),
+      onChunk: (chunk) => console.log('Received chunk:', chunk.id),
+      onFinish: () => console.log('\nStream completed'),
+      onError: (error) => console.error('Stream error:', error),
+    },
+    Provider.Groq // Provider is optional
+  );
 } catch (error) {
   console.error('Error:', error);
 }
 ```
 
-### List Models by Provider
+### Tool Calls
 
-To list all available models from a specific provider, use the `listModelsByProvider` method:
+To use tool calls with models that support them:
 
 ```typescript
+import {
+  InferenceGatewayClient,
+  MessageRole,
+  Provider,
+} from '@inference-gateway/sdk';
+
+const client = new InferenceGatewayClient({
+  baseURL: 'http://localhost:8080/v1',
+});
+
 try {
-  const providerModels = await client.listModelsByProvider(Provider.OpenAI);
-  console.log(`Provider: ${providerModels.provider}`);
-  providerModels.models.forEach((model) => {
-    console.log(`Model: ${model.name}`);
-  });
+  await client.streamChatCompletion(
+    {
+      model: 'gpt-4o',
+      messages: [
+        {
+          role: MessageRole.User,
+          content: 'What's the weather in San Francisco?',
+        },
+      ],
+      tools: [
+        {
+          type: 'function',
+          function: {
+            name: 'get_weather',
+            parameters: {
+              type: 'object',
+              properties: {
+                location: {
+                  type: 'string',
+                  description: 'The city and state, e.g. San Francisco, CA',
+                },
+              },
+              required: ['location'],
+            },
+          },
+        },
+      ],
+    },
+    {
+      onTool: (toolCall) => {
+        console.log('Tool call:', toolCall.function.name);
+        console.log('Arguments:', toolCall.function.arguments);
+      },
+      onContent: (content) => process.stdout.write(content),
+      onFinish: () => console.log('\nStream completed'),
+    },
+    Provider.OpenAI
+  );
 } catch (error) {
   console.error('Error:', error);
 }
 ```
 
-### Generating Content
+### Proxying Requests
 
-To generate content using a model, use the `generateContent` method:
+To proxy requests directly to a provider:
 
 ```typescript
-import {
-  InferenceGatewayClient,
-  Message,
-  MessageRole,
-  Provider,
-} from '@inference-gateway/sdk';
+import { InferenceGatewayClient, Provider } from '@inference-gateway/sdk';
 
-const client = new InferenceGatewayClient('http://localhost:8080');
+const client = new InferenceGatewayClient({
+  baseURL: 'http://localhost:8080/v1',
+});
 
-  const response = await client.generateContent({
-    provider: Provider.Ollama,
-    model: 'llama2',
-    messages: [
-      {
-        role: MessageRole.System,
-        content: 'You are a helpful llama',
-      },
-      {
-        role: MessageRole.User,
-        content: 'Tell me a joke',
-      },
-    ],
+try {
+  const response = await client.proxy(Provider.OpenAI, 'embeddings', {
+    method: 'POST',
+    body: JSON.stringify({
+      model: 'text-embedding-ada-002',
+      input: 'Hello world',
+    }),
   });
 
-  console.log('Provider:', response.provider);
-  console.log('Response:', response.response);
+  console.log('Embeddings:', response);
 } catch (error) {
   console.error('Error:', error);
 }
@@ -138,9 +224,15 @@ const client = new InferenceGatewayClient('http://localhost:8080');
 
 ### Health Check
 
-To check if the Inference Gateway is running, use the `healthCheck` method:
+To check if the Inference Gateway is running:
 
 ```typescript
+import { InferenceGatewayClient } from '@inference-gateway/sdk';
+
+const client = new InferenceGatewayClient({
+  baseURL: 'http://localhost:8080/v1',
+});
+
 try {
   const isHealthy = await client.healthCheck();
   console.log('API is healthy:', isHealthy);
@@ -149,6 +241,26 @@ try {
 }
 ```
 
+### Creating a Client with Custom Options
+
+You can create a new client with custom options using the `withOptions` method:
+
+```typescript
+import { InferenceGatewayClient } from '@inference-gateway/sdk';
+
+const client = new InferenceGatewayClient({
+  baseURL: 'http://localhost:8080/v1',
+});
+
+// Create a new client with custom headers
+const clientWithHeaders = client.withOptions({
+  defaultHeaders: {
+    'X-Custom-Header': 'value',
+  },
+  timeout: 60000, // 60 seconds
+});
+```
+
 ## Contributing
 
 Please refer to the [CONTRIBUTING.md](CONTRIBUTING.md) file for information about how to get involved. We welcome issues, questions, and pull requests.

package/dist/src/client.d.ts

@@ -1,11 +1,46 @@
-import { GenerateContentRequest, GenerateContentResponse, Provider, ProviderModels } from './types';
+import { ChatCompletionRequest, ChatCompletionResponse, ChatCompletionStreamCallbacks, ListModelsResponse, Provider } from './types';
+export interface ClientOptions {
+    baseURL?: string;
+    apiKey?: string;
+    defaultHeaders?: Record<string, string>;
+    defaultQuery?: Record<string, string>;
+    timeout?: number;
+    fetch?: typeof globalThis.fetch;
+}
 export declare class InferenceGatewayClient {
-    private baseUrl;
-    private authToken?;
-    constructor(baseUrl: string, authToken?: string);
+    private baseURL;
+    private apiKey?;
+    private defaultHeaders;
+    private defaultQuery;
+    private timeout;
+    private fetchFn;
+    constructor(options?: ClientOptions);
+    /**
+     * Creates a new instance of the client with the given options merged with the existing options.
+     */
+    withOptions(options: ClientOptions): InferenceGatewayClient;
+    /**
+     * Makes a request to the API.
+     */
     private request;
-    listModels(): Promise<ProviderModels[]>;
-    listModelsByProvider(provider: Provider): Promise<ProviderModels>;
-    generateContent(params: GenerateContentRequest): Promise<GenerateContentResponse>;
+    /**
+     * Lists the currently available models.
+     */
+    listModels(provider?: Provider): Promise<ListModelsResponse>;
+    /**
+     * Creates a chat completion.
+     */
+    createChatCompletion(request: ChatCompletionRequest, provider?: Provider): Promise<ChatCompletionResponse>;
+    /**
+     * Creates a streaming chat completion.
+     */
+    streamChatCompletion(request: ChatCompletionRequest, callbacks: ChatCompletionStreamCallbacks, provider?: Provider): Promise<void>;
+    /**
+     * Proxy a request to a specific provider.
+     */
+    proxy<T = unknown>(provider: Provider, path: string, options?: RequestInit): Promise<T>;
+    /**
+     * Health check endpoint.
+     */
     healthCheck(): Promise<boolean>;
 }

package/dist/src/client.js

@@ -2,48 +2,201 @@
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.InferenceGatewayClient = void 0;
 class InferenceGatewayClient {
-    baseUrl;
-    authToken;
-    constructor(baseUrl, authToken) {
-        this.baseUrl = baseUrl.replace(/\/$/, '');
-        this.authToken = authToken;
+    baseURL;
+    apiKey;
+    defaultHeaders;
+    defaultQuery;
+    timeout;
+    fetchFn;
+    constructor(options = {}) {
+        this.baseURL = options.baseURL || 'http://localhost:8080/v1';
+        this.apiKey = options.apiKey;
+        this.defaultHeaders = options.defaultHeaders || {};
+        this.defaultQuery = options.defaultQuery || {};
+        this.timeout = options.timeout || 30000;
+        this.fetchFn = options.fetch || globalThis.fetch;
     }
-    async request(path, options = {}) {
+    /**
+     * Creates a new instance of the client with the given options merged with the existing options.
+     */
+    withOptions(options) {
+        return new InferenceGatewayClient({
+            baseURL: options.baseURL || this.baseURL,
+            apiKey: options.apiKey || this.apiKey,
+            defaultHeaders: { ...this.defaultHeaders, ...options.defaultHeaders },
+            defaultQuery: { ...this.defaultQuery, ...options.defaultQuery },
+            timeout: options.timeout || this.timeout,
+            fetch: options.fetch || this.fetchFn,
+        });
+    }
+    /**
+     * Makes a request to the API.
+     */
+    async request(path, options = {}, query = {}) {
         const headers = new Headers({
             'Content-Type': 'application/json',
+            ...this.defaultHeaders,
             ...options.headers,
         });
-        if (this.authToken) {
-            headers.set('Authorization', `Bearer ${this.authToken}`);
+        if (this.apiKey) {
+            headers.set('Authorization', `Bearer ${this.apiKey}`);
         }
-        const response = await fetch(`${this.baseUrl}${path}`, {
-            ...options,
-            headers,
+        // Combine default query parameters with provided ones
+        const queryParams = new URLSearchParams({
+            ...this.defaultQuery,
+            ...query,
         });
-        if (!response.ok) {
-            const error = await response.json();
-            throw new Error(error.error || `HTTP error! status: ${response.status}`);
+        const queryString = queryParams.toString();
+        const url = `${this.baseURL}${path}${queryString ? `?${queryString}` : ''}`;
+        const controller = new AbortController();
+        const timeoutId = globalThis.setTimeout(() => controller.abort(), this.timeout);
+        try {
+            const response = await this.fetchFn(url, {
+                ...options,
+                headers,
+                signal: controller.signal,
+            });
+            if (!response.ok) {
+                const error = (await response.json());
+                throw new Error(error.error || `HTTP error! status: ${response.status}`);
+            }
+            return response.json();
+        }
+        finally {
+            globalThis.clearTimeout(timeoutId);
         }
-        return response.json();
-    }
-    async listModels() {
-        return this.request('/llms');
     }
-    async listModelsByProvider(provider) {
-        return this.request(`/llms/${provider}`);
+    /**
+     * Lists the currently available models.
+     */
+    async listModels(provider) {
+        const query = {};
+        if (provider) {
+            query.provider = provider;
+        }
+        return this.request('/models', { method: 'GET' }, query);
     }
-    async generateContent(params) {
-        return this.request(`/llms/${params.provider}/generate`, {
+    /**
+     * Creates a chat completion.
+     */
+    async createChatCompletion(request, provider) {
+        const query = {};
+        if (provider) {
+            query.provider = provider;
+        }
+        return this.request('/chat/completions', {
             method: 'POST',
-            body: JSON.stringify({
-                model: params.model,
-                messages: params.messages,
-            }),
+            body: JSON.stringify(request),
+        }, query);
+    }
+    /**
+     * Creates a streaming chat completion.
+     */
+    async streamChatCompletion(request, callbacks, provider) {
+        const query = {};
+        if (provider) {
+            query.provider = provider;
+        }
+        const queryParams = new URLSearchParams({
+            ...this.defaultQuery,
+            ...query,
         });
+        const queryString = queryParams.toString();
+        const url = `${this.baseURL}/chat/completions${queryString ? `?${queryString}` : ''}`;
+        const headers = new Headers({
+            'Content-Type': 'application/json',
+            ...this.defaultHeaders,
+        });
+        if (this.apiKey) {
+            headers.set('Authorization', `Bearer ${this.apiKey}`);
+        }
+        const controller = new AbortController();
+        const timeoutId = globalThis.setTimeout(() => controller.abort(), this.timeout);
+        try {
+            const response = await this.fetchFn(url, {
+                method: 'POST',
+                headers,
+                body: JSON.stringify({
+                    ...request,
+                    stream: true,
+                }),
+                signal: controller.signal,
+            });
+            if (!response.ok) {
+                const error = (await response.json());
+                throw new Error(error.error || `HTTP error! status: ${response.status}`);
+            }
+            if (!response.body) {
+                throw new Error('Response body is not readable');
+            }
+            callbacks.onOpen?.();
+            const reader = response.body.getReader();
+            const decoder = new TextDecoder();
+            let buffer = '';
+            while (true) {
+                const { done, value } = await reader.read();
+                if (done)
+                    break;
+                buffer += decoder.decode(value, { stream: true });
+                const lines = buffer.split('\n');
+                buffer = lines.pop() || '';
+                for (const line of lines) {
+                    if (line.startsWith('data: ')) {
+                        const data = line.slice(5).trim();
+                        if (data === '[DONE]') {
+                            callbacks.onFinish?.(null);
+                            return;
+                        }
+                        try {
+                            const chunk = JSON.parse(data);
+                            callbacks.onChunk?.(chunk);
+                            const content = chunk.choices[0]?.delta?.content;
+                            if (content) {
+                                callbacks.onContent?.(content);
+                            }
+                            const toolCalls = chunk.choices[0]?.delta?.tool_calls;
+                            if (toolCalls && toolCalls.length > 0) {
+                                const toolCall = {
+                                    id: toolCalls[0].id || '',
+                                    type: 'function',
+                                    function: {
+                                        name: toolCalls[0].function?.name || '',
+                                        arguments: toolCalls[0].function?.arguments || '',
+                                    },
+                                };
+                                callbacks.onTool?.(toolCall);
+                            }
+                        }
+                        catch (e) {
+                            globalThis.console.error('Error parsing SSE data:', e);
+                        }
+                    }
+                }
+            }
+        }
+        catch (error) {
+            const apiError = {
+                error: error.message || 'Unknown error',
+            };
+            callbacks.onError?.(apiError);
+            throw error;
+        }
+        finally {
+            globalThis.clearTimeout(timeoutId);
+        }
+    }
+    /**
+     * Proxy a request to a specific provider.
+     */
+    async proxy(provider, path, options = {}) {
+        return this.request(`/proxy/${provider}/${path}`, options);
     }
+    /**
+     * Health check endpoint.
+     */
     async healthCheck() {
         try {
-            await this.request('/health');
+            await this.fetchFn(`${this.baseURL.replace('/v1', '')}/health`);
             return true;
         }
         catch {

package/dist/src/types/index.d.ts

@@ -2,37 +2,127 @@ export declare enum Provider {
     Ollama = "ollama",
     Groq = "groq",
     OpenAI = "openai",
-    Google = "google",
     Cloudflare = "cloudflare",
     Cohere = "cohere",
-    Anthropic = "anthropic"
+    Anthropic = "anthropic",
+    DeepSeek = "deepseek"
 }
 export declare enum MessageRole {
     System = "system",
     User = "user",
-    Assistant = "assistant"
+    Assistant = "assistant",
+    Tool = "tool"
 }
 export interface Message {
     role: MessageRole;
     content: string;
+    tool_calls?: ChatCompletionMessageToolCall[];
+    tool_call_id?: string;
 }
 export interface Model {
+    id: string;
+    object: string;
+    created: number;
+    owned_by: string;
+}
+export interface ListModelsResponse {
+    object: string;
+    data: Model[];
+}
+export interface ChatCompletionMessageToolCallFunction {
+    name: string;
+    arguments: string;
+}
+export interface ChatCompletionMessageToolCall {
+    id: string;
+    type: 'function';
+    function: ChatCompletionMessageToolCallFunction;
+}
+export interface ChatCompletionMessageToolCallChunk {
+    index: number;
+    id?: string;
+    type?: string;
+    function?: {
+        name?: string;
+        arguments?: string;
+    };
+}
+export interface FunctionParameters {
+    type: string;
+    properties?: Record<string, unknown>;
+    required?: string[];
+}
+export interface FunctionObject {
+    description?: string;
     name: string;
+    parameters: FunctionParameters;
+    strict?: boolean;
 }
-export interface ProviderModels {
-    provider: Provider;
-    models: Model[];
+export interface ChatCompletionTool {
+    type: 'function';
+    function: FunctionObject;
 }
-export interface GenerateContentRequest {
-    provider: Provider;
+export interface ChatCompletionRequest {
     model: string;
     messages: Message[];
+    max_tokens?: number;
+    stream?: boolean;
+    stream_options?: ChatCompletionStreamOptions;
+    tools?: ChatCompletionTool[];
+    temperature?: number;
+    top_p?: number;
+    top_k?: number;
 }
-export interface GenerateContentResponse {
-    provider: string;
-    response: {
-        role: 'assistant';
-        model: string;
-        content: string;
-    };
+export interface ChatCompletionStreamOptions {
+    include_usage?: boolean;
+}
+export interface ChatCompletionChoice {
+    finish_reason: 'stop' | 'length' | 'tool_calls' | 'content_filter' | 'function_call';
+    index: number;
+    message: Message;
+    logprobs?: Record<string, unknown>;
+}
+export interface CompletionUsage {
+    prompt_tokens: number;
+    completion_tokens: number;
+    total_tokens: number;
+}
+export interface ChatCompletionResponse {
+    id: string;
+    choices: ChatCompletionChoice[];
+    created: number;
+    model: string;
+    object: string;
+    usage?: CompletionUsage;
+}
+export interface ChatCompletionStreamChoice {
+    delta: ChatCompletionStreamResponseDelta;
+    finish_reason: 'stop' | 'length' | 'tool_calls' | 'content_filter' | 'function_call' | null;
+    index: number;
+    logprobs?: Record<string, unknown>;
+}
+export interface ChatCompletionStreamResponseDelta {
+    content?: string;
+    tool_calls?: ChatCompletionMessageToolCallChunk[];
+    role?: MessageRole;
+    refusal?: string;
+}
+export interface ChatCompletionStreamResponse {
+    id: string;
+    choices: ChatCompletionStreamChoice[];
+    created: number;
+    model: string;
+    object: string;
+    usage?: CompletionUsage;
+}
+export interface ChatCompletionStreamCallbacks {
+    onOpen?: () => void;
+    onChunk?: (chunk: ChatCompletionStreamResponse) => void;
+    onContent?: (content: string) => void;
+    onTool?: (toolCall: ChatCompletionMessageToolCall) => void;
+    onFinish?: (response: ChatCompletionStreamResponse) => void;
+    onError?: (error: Error) => void;
+}
+export interface Error {
+    error: string;
 }

package/dist/src/types/index.js

@@ -6,14 +6,15 @@ var Provider;
     Provider["Ollama"] = "ollama";
     Provider["Groq"] = "groq";
     Provider["OpenAI"] = "openai";
-    Provider["Google"] = "google";
     Provider["Cloudflare"] = "cloudflare";
     Provider["Cohere"] = "cohere";
     Provider["Anthropic"] = "anthropic";
+    Provider["DeepSeek"] = "deepseek";
 })(Provider || (exports.Provider = Provider = {}));
 var MessageRole;
 (function (MessageRole) {
     MessageRole["System"] = "system";
     MessageRole["User"] = "user";
     MessageRole["Assistant"] = "assistant";
+    MessageRole["Tool"] = "tool";
 })(MessageRole || (exports.MessageRole = MessageRole = {}));

package/dist/tests/client.test.js

@@ -2,124 +2,338 @@
 Object.defineProperty(exports, "__esModule", { value: true });
 const client_1 = require("@/client");
 const types_1 = require("@/types");
+const web_1 = require("node:stream/web");
+const node_util_1 = require("node:util");
 describe('InferenceGatewayClient', () => {
     let client;
-    const mockBaseUrl = 'http://localhost:8080';
+    const mockFetch = jest.fn();
     beforeEach(() => {
-        client = new client_1.InferenceGatewayClient(mockBaseUrl);
-        global.fetch = jest.fn();
+        client = new client_1.InferenceGatewayClient({
+            baseURL: 'http://localhost:8080/v1',
+            fetch: mockFetch,
+        });
+    });
+    afterEach(() => {
+        jest.clearAllMocks();
     });
     describe('listModels', () => {
         it('should fetch available models', async () => {
-            const mockResponse = [
-                {
-                    provider: types_1.Provider.Ollama,
-                    models: [
-                        {
-                            name: 'llama2',
-                        },
-                    ],
-                },
-            ];
-            global.fetch.mockResolvedValueOnce({
+            const mockResponse = {
+                object: 'list',
+                data: [
+                    {
+                        id: 'gpt-4o',
+                        object: 'model',
+                        created: 1686935002,
+                        owned_by: 'openai',
+                    },
+                    {
+                        id: 'llama-3.3-70b-versatile',
+                        object: 'model',
+                        created: 1723651281,
+                        owned_by: 'groq',
+                    },
+                ],
+            };
+            mockFetch.mockResolvedValueOnce({
                 ok: true,
                 json: () => Promise.resolve(mockResponse),
             });
             const result = await client.listModels();
             expect(result).toEqual(mockResponse);
-            expect(global.fetch).toHaveBeenCalledWith(`${mockBaseUrl}/llms`, expect.objectContaining({
+            expect(mockFetch).toHaveBeenCalledWith('http://localhost:8080/v1/models', expect.objectContaining({
+                method: 'GET',
                 headers: expect.any(Headers),
             }));
         });
-    });
-    describe('listModelsByProvider', () => {
         it('should fetch models for a specific provider', async () => {
             const mockResponse = {
-                provider: types_1.Provider.OpenAI,
-                models: [
+                object: 'list',
+                data: [
                     {
-                        name: 'gpt-4',
+                        id: 'gpt-4o',
+                        object: 'model',
+                        created: 1686935002,
+                        owned_by: 'openai',
                     },
                 ],
             };
-            global.fetch.mockResolvedValueOnce({
+            mockFetch.mockResolvedValueOnce({
                 ok: true,
                 json: () => Promise.resolve(mockResponse),
             });
-            const result = await client.listModelsByProvider(types_1.Provider.OpenAI);
+            const result = await client.listModels(types_1.Provider.OpenAI);
             expect(result).toEqual(mockResponse);
-            expect(global.fetch).toHaveBeenCalledWith(`${mockBaseUrl}/llms/${types_1.Provider.OpenAI}`, expect.objectContaining({
+            expect(mockFetch).toHaveBeenCalledWith('http://localhost:8080/v1/models?provider=openai', expect.objectContaining({
+                method: 'GET',
                 headers: expect.any(Headers),
             }));
         });
-        it('should throw error when provider request fails', async () => {
+        it('should throw error when request fails', async () => {
             const errorMessage = 'Provider not found';
-            global.fetch.mockResolvedValueOnce({
+            mockFetch.mockResolvedValueOnce({
                 ok: false,
                 status: 404,
                 json: () => Promise.resolve({ error: errorMessage }),
             });
-            await expect(client.listModelsByProvider(types_1.Provider.OpenAI)).rejects.toThrow(errorMessage);
+            await expect(client.listModels(types_1.Provider.OpenAI)).rejects.toThrow(errorMessage);
         });
     });
-    describe('generateContent', () => {
-        it('should generate content with the specified provider', async () => {
+    describe('createChatCompletion', () => {
+        it('should create a chat completion', async () => {
             const mockRequest = {
-                provider: types_1.Provider.Ollama,
-                model: 'llama2',
+                model: 'gpt-4o',
                 messages: [
                     { role: types_1.MessageRole.System, content: 'You are a helpful assistant' },
                     { role: types_1.MessageRole.User, content: 'Hello' },
                 ],
             };
             const mockResponse = {
-                provider: types_1.Provider.Ollama,
-                response: {
-                    role: types_1.MessageRole.Assistant,
-                    model: 'llama2',
-                    content: 'Hi there!',
+                id: 'chatcmpl-123',
+                object: 'chat.completion',
+                created: 1677652288,
+                model: 'gpt-4o',
+                choices: [
+                    {
+                        index: 0,
+                        message: {
+                            role: types_1.MessageRole.Assistant,
+                            content: 'Hello! How can I help you today?',
+                        },
+                        finish_reason: 'stop',
+                    },
+                ],
+                usage: {
+                    prompt_tokens: 10,
+                    completion_tokens: 8,
+                    total_tokens: 18,
                 },
             };
-            global.fetch.mockResolvedValueOnce({
+            mockFetch.mockResolvedValueOnce({
+                ok: true,
+                json: () => Promise.resolve(mockResponse),
+            });
+            const result = await client.createChatCompletion(mockRequest);
+            expect(result).toEqual(mockResponse);
+            expect(mockFetch).toHaveBeenCalledWith('http://localhost:8080/v1/chat/completions', expect.objectContaining({
+                method: 'POST',
+                body: JSON.stringify(mockRequest),
+            }));
+        });
+        it('should create a chat completion with a specific provider', async () => {
+            const mockRequest = {
+                model: 'claude-3-opus-20240229',
+                messages: [{ role: types_1.MessageRole.User, content: 'Hello' }],
+            };
+            const mockResponse = {
+                id: 'chatcmpl-456',
+                object: 'chat.completion',
+                created: 1677652288,
+                model: 'claude-3-opus-20240229',
+                choices: [
+                    {
+                        index: 0,
+                        message: {
+                            role: types_1.MessageRole.Assistant,
+                            content: 'Hello! How can I assist you today?',
+                        },
+                        finish_reason: 'stop',
+                    },
+                ],
+                usage: {
+                    prompt_tokens: 5,
+                    completion_tokens: 8,
+                    total_tokens: 13,
+                },
+            };
+            mockFetch.mockResolvedValueOnce({
+                ok: true,
+                json: () => Promise.resolve(mockResponse),
+            });
+            const result = await client.createChatCompletion(mockRequest, types_1.Provider.Anthropic);
+            expect(result).toEqual(mockResponse);
+            expect(mockFetch).toHaveBeenCalledWith('http://localhost:8080/v1/chat/completions?provider=anthropic', expect.objectContaining({
+                method: 'POST',
+                body: JSON.stringify(mockRequest),
+            }));
+        });
+    });
+    describe('streamChatCompletion', () => {
+        it('should handle streaming chat completions', async () => {
+            const mockRequest = {
+                model: 'gpt-4o',
+                messages: [{ role: types_1.MessageRole.User, content: 'Hello' }],
+            };
+            const mockStream = new web_1.TransformStream();
+            const writer = mockStream.writable.getWriter();
+            const encoder = new node_util_1.TextEncoder();
+            mockFetch.mockResolvedValueOnce({
+                ok: true,
+                body: mockStream.readable,
+            });
+            const callbacks = {
+                onOpen: jest.fn(),
+                onChunk: jest.fn(),
+                onContent: jest.fn(),
+                onFinish: jest.fn(),
+                onError: jest.fn(),
+            };
+            const streamPromise = client.streamChatCompletion(mockRequest, callbacks);
+            // Simulate SSE events
+            await writer.write(encoder.encode('data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1677652288,"model":"gpt-4o","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}\n\n' +
+                'data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1677652288,"model":"gpt-4o","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}\n\n' +
+                'data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1677652288,"model":"gpt-4o","choices":[{"index":0,"delta":{"content":"!"},"finish_reason":null}]}\n\n' +
+                'data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1677652288,"model":"gpt-4o","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}\n\n' +
+                'data: [DONE]\n\n'));
+            await writer.close();
+            await streamPromise;
+            expect(callbacks.onOpen).toHaveBeenCalledTimes(1);
+            expect(callbacks.onChunk).toHaveBeenCalledTimes(4);
+            expect(callbacks.onContent).toHaveBeenCalledWith('Hello');
+            expect(callbacks.onContent).toHaveBeenCalledWith('!');
+            expect(callbacks.onFinish).toHaveBeenCalledTimes(1);
+            expect(mockFetch).toHaveBeenCalledWith('http://localhost:8080/v1/chat/completions', expect.objectContaining({
+                method: 'POST',
+                body: JSON.stringify({
+                    ...mockRequest,
+                    stream: true,
+                }),
+            }));
+        });
+        it('should handle tool calls in streaming chat completions', async () => {
+            const mockRequest = {
+                model: 'gpt-4o',
+                messages: [
+                    {
+                        role: types_1.MessageRole.User,
+                        content: 'What is the weather in San Francisco?',
+                    },
+                ],
+                tools: [
+                    {
+                        type: 'function',
+                        function: {
+                            name: 'get_weather',
+                            parameters: {
+                                type: 'object',
+                                properties: {
+                                    location: {
+                                        type: 'string',
+                                        description: 'The city and state, e.g. San Francisco, CA',
+                                    },
+                                },
+                                required: ['location'],
+                            },
+                        },
+                    },
+                ],
+            };
+            const mockStream = new web_1.TransformStream();
+            const writer = mockStream.writable.getWriter();
+            const encoder = new node_util_1.TextEncoder();
+            mockFetch.mockResolvedValueOnce({
+                ok: true,
+                body: mockStream.readable,
+            });
+            const callbacks = {
+                onOpen: jest.fn(),
+                onChunk: jest.fn(),
+                onTool: jest.fn(),
+                onFinish: jest.fn(),
+            };
+            const streamPromise = client.streamChatCompletion(mockRequest, callbacks);
+            // Simulate SSE events with tool calls
+            await writer.write(encoder.encode('data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1677652288,"model":"gpt-4o","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}\n\n' +
+                'data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1677652288,"model":"gpt-4o","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_123","type":"function","function":{"name":"get_weather"}}]},"finish_reason":null}]}\n\n' +
+                'data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1677652288,"model":"gpt-4o","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\\"location\\""}}]},"finish_reason":null}]}\n\n' +
+                'data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1677652288,"model":"gpt-4o","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":":\\"San Francisco, CA\\""}}]},"finish_reason":null}]}\n\n' +
+                'data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1677652288,"model":"gpt-4o","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"}"}}]},"finish_reason":null}]}\n\n' +
+                'data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1677652288,"model":"gpt-4o","choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}]}\n\n' +
+                'data: [DONE]\n\n'));
+            await writer.close();
+            await streamPromise;
+            expect(callbacks.onOpen).toHaveBeenCalledTimes(1);
+            expect(callbacks.onChunk).toHaveBeenCalledTimes(6);
+            expect(callbacks.onTool).toHaveBeenCalledTimes(4); // Called for each chunk with tool_calls
+            expect(callbacks.onFinish).toHaveBeenCalledTimes(1);
+        });
+        it('should handle errors in streaming chat completions', async () => {
+            const mockRequest = {
+                model: 'gpt-4o',
+                messages: [{ role: types_1.MessageRole.User, content: 'Hello' }],
+            };
+            mockFetch.mockResolvedValueOnce({
+                ok: false,
+                status: 400,
+                json: () => Promise.resolve({ error: 'Bad Request' }),
+            });
+            const callbacks = {
+                onError: jest.fn(),
+            };
+            await expect(client.streamChatCompletion(mockRequest, callbacks)).rejects.toThrow('Bad Request');
+            expect(callbacks.onError).toHaveBeenCalledTimes(1);
+        });
+    });
+    describe('proxy', () => {
+        it('should proxy requests to a specific provider', async () => {
+            const mockResponse = { result: 'success' };
+            mockFetch.mockResolvedValueOnce({
                 ok: true,
                 json: () => Promise.resolve(mockResponse),
             });
-            const result = await client.generateContent(mockRequest);
+            const result = await client.proxy(types_1.Provider.OpenAI, 'embeddings', {
+                method: 'POST',
+                body: JSON.stringify({
+                    model: 'text-embedding-ada-002',
+                    input: 'Hello world',
+                }),
+            });
             expect(result).toEqual(mockResponse);
-            expect(global.fetch).toHaveBeenCalledWith(`${mockBaseUrl}/llms/${mockRequest.provider}/generate`, expect.objectContaining({
+            expect(mockFetch).toHaveBeenCalledWith('http://localhost:8080/v1/proxy/openai/embeddings', expect.objectContaining({
                 method: 'POST',
                 body: JSON.stringify({
-                    model: mockRequest.model,
-                    messages: mockRequest.messages,
+                    model: 'text-embedding-ada-002',
+                    input: 'Hello world',
                 }),
             }));
         });
     });
     describe('healthCheck', () => {
         it('should return true when API is healthy', async () => {
-            global.fetch.mockResolvedValueOnce({
+            mockFetch.mockResolvedValueOnce({
                 ok: true,
-                json: () => Promise.resolve({}),
             });
             const result = await client.healthCheck();
             expect(result).toBe(true);
-            expect(global.fetch).toHaveBeenCalledWith(`${mockBaseUrl}/health`, expect.any(Object));
+            expect(mockFetch).toHaveBeenCalledWith('http://localhost:8080/health');
         });
         it('should return false when API is unhealthy', async () => {
-            global.fetch.mockRejectedValueOnce(new Error('API error'));
+            mockFetch.mockRejectedValueOnce(new Error('API error'));
             const result = await client.healthCheck();
             expect(result).toBe(false);
         });
     });
-    describe('error handling', () => {
-        it('should throw error when API request fails', async () => {
-            const errorMessage = 'Bad Request';
-            global.fetch.mockResolvedValueOnce({
-                ok: false,
-                status: 400,
-                json: () => Promise.resolve({ error: errorMessage }),
+    describe('withOptions', () => {
+        it('should create a new client with merged options', () => {
+            const originalClient = new client_1.InferenceGatewayClient({
+                baseURL: 'http://localhost:8080/v1',
+                apiKey: 'test-key',
+                fetch: mockFetch,
+            });
+            const newClient = originalClient.withOptions({
+                defaultHeaders: { 'X-Custom-Header': 'value' },
             });
-            await expect(client.listModels()).rejects.toThrow(errorMessage);
+            expect(newClient).toBeInstanceOf(client_1.InferenceGatewayClient);
+            expect(newClient).not.toBe(originalClient);
+            // We can't directly test private properties, but we can test behavior
+            mockFetch.mockResolvedValueOnce({
+                ok: true,
+                json: () => Promise.resolve({}),
+            });
+            newClient.listModels();
+            expect(mockFetch).toHaveBeenCalledWith('http://localhost:8080/v1/models', expect.objectContaining({
+                headers: expect.any(Headers),
+            }));
         });
     });
 });

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@inference-gateway/sdk",
-  "version": "0.2.0",
+  "version": "0.3.2",
   "description": "An SDK written in Typescript for the [Inference Gateway](https://github.com/inference-gateway/inference-gateway).",
   "main": "dist/src/index.js",
   "types": "dist/src/index.d.ts",
@@ -18,7 +18,8 @@
     "ollama",
     "cloudflare",
     "cohere",
-    "typescript"
+    "typescript",
+    "deepseek"
   ],
   "author": "Eden Reich <eden.reich@gmail.com>",
   "license": "MIT",