inference-server 1.0.0-beta.19

package/README.md

@@ -0,0 +1,216 @@
+## inference-server
+
+Libraries and server to build AI applications doing local inference in node. Use it within your application, or as a microservice. Adapters to [llama.cpp](https://github.com/ggerganov/llama.cpp/) via [node-llama-cpp](https://github.com/withcatai/node-llama-cpp) and [gpt4all](https://github.com/nomic-ai/gpt4all). And [transformers.js](https://github.com/xenova/transformers.js/) using [ONNX](https://github.com/microsoft/onnxruntime/tree/main/js#onnxruntime-node)!
+
+The project includes a model resource pool, an inference queue and a HTTP API server. Model file management is abstracted away as much as possible - configure a URL and go. This package is useful for quick model evaluations and experiments (in JavaScript), small-scale chatbots, resource efficient assistants on edge devices, or any applications where private & offline are interesting criteria. For other - not node-based solutions - check out the [related solutions](#related-solutions) section.
+
+⚠️ This package is currently in beta. APIs may change. Things may break. [Help is welcome](#contributing).
+
+### Features
+
+- Configure as many models as you want, they will be downloaded and cached to disk. You may provide them as abs file paths if you already have models downloaded.
+- Adjust the pool `concurrency`, and the models `maxInstances`, `ttl` and `contextSize` to fit your usecase. Combine multiple pools for more complex setups.
+- Can be tuned to either use no resources when idle or to always keep a model ready with context preloaded.
+- A chat session cache that will effectively reuse context across multiple turns or stateless requests.
+- OpenAI spec API endpoints. See [HTTP API docs](./docs/http-api.md) for details. A "native" HTTP API is not yet implemented.
+- BYO web server or use the provided express server and middleware. Incoming requests are queued - stall, if needed - and processed as soon as resources are available.
+- Have as many ModelServers running as you want, they can share the same cache directory. (Multiple processes can as well)
+- Use the [ModelPool](./examples/pool.js) class directly for a lowerlevel transaction-like API to aquire/release model instances.
+- Use [custom engines](./docs/engines.md#custom-engines) to combine multiple models (or do RAG) behind the scenes.
+
+### Usage
+
+Example with minimal configuration:
+
+```ts basic.ts
+import { ModelServer } from 'inference-server'
+
+const modelServer = new ModelServer({
+  log: 'info', // default is 'warn'
+  models: {
+    'my-model': { // Identifiers can use a-zA-Z0-9_:\-\.
+      // Required are `task`, `engine`, `url` and/or `file`.
+      task: 'text-completion', // text-completion models can be used for chat and text generation tasks
+      engine: 'node-llama-cpp', // each engine comes with a peer dep. `npm install node-llama-cpp@3`
+      url: 'https://huggingface.co/HuggingFaceTB/smollm-135M-instruct-v0.2-Q8_0-GGUF/blob/main/smollm-135m-instruct-add-basics-q8_0.gguf',
+    },
+  },
+})
+await modelServer.start()
+const result = await modelServer.processChatCompletionTask({
+  model: 'my-model',
+  messages: [
+    {
+      role: 'user',
+      content: 'Why are bananas rather blue than bread at night?',
+    },
+  ],
+})
+console.debug(result)
+modelServer.stop()
+```
+
+Or, to start an OAI compatible HTTP server with two concurrent instances of the same model:
+
+```ts http-api.ts
+import { startHTTPServer } from 'inference-server'
+import OpenAI from 'openai'
+
+const server = await startHTTPServer({
+  listen: { port: 3000 }, // apart from `listen` options are identical to ModelServer
+  concurrency: 2, // two inference processes may run at the same time
+  models: {
+    'smollm': {
+      task: 'text-completion',
+      engine: 'node-llama-cpp',
+      url: 'https://huggingface.co/HuggingFaceTB/smollm-135M-instruct-v0.2-Q8_0-GGUF/blob/main/smollm-135m-instruct-add-basics-q8_0.gguf',
+      maxInstances: 2, // two instances of this model may be loaded into memory
+      device: {
+        cpuThreads: 4, // limit cpu threads so we dont occupy all cores
+      }
+    },
+  },
+})
+
+const client = new OpenAI({
+  baseURL: 'http://localhost:3000/openai/v1/',
+  apiKey: 'yes',
+})
+const completion = await client.beta.chat.completions.stream({
+  stream_options: { include_usage: true },
+  model: 'smollm',
+  messages: [
+    { role: 'user', content: 'lets count to 10, but only whisper every second number' },
+  ],
+})
+for await (const chunk of completion) {
+  if (chunk.choices[0]?.delta?.content) {
+    process.stdout.write(chunk.choices[0].delta.content)
+  }
+}
+server.stop()
+```
+
+More usage examples:
+- Using all available options / model options API doc [./examples/all-options](./examples/all-options.js).
+- Custom engines [./tests/engines/experiments.test.ts](./tests/engines/experiments.test.ts).
+- A chat cli [./examples/chat-cli](./examples/chat-cli.js).
+- `concurrency` behavior [./examples/concurrency](./examples/concurrency.js).
+- Using the ModelPool directly [./examples/pool](./examples/pool.js).
+- Using the express middleware [./examples/express](./examples/express.js).
+
+Currently supported inference engines are:
+
+| Engine | Peer Dependency |
+| --- | --- |
+| node-llama-cpp | `node-llama-cpp >= 3.0.0` |
+| gpt4all | `gpt4all >= 4.0.0` |
+| transformers-js | `@huggingface/transformers >= 3.0.0` |
+
+See [engine docs](./docs/engines.md) for more information on each.
+
+#### Limitations and Known Issues
+
+##### Only one model can run on GPU at a time
+Llama.cpp bindings currently do not support running multiple models on gpu at the same time. This can/will likely be improved in the future. See [GPU docs](./docs/gpu.md) for more information on how to work around that.
+
+##### System Messages
+System role messages are supported only as the first message in a chat completion session. All other system messages will be ignored. This is only for simplicity reasons and might change in the future.
+
+##### Chat Context Cache / Reusing the correct instance on stateless requests
+Note that the current context cache implementation only works if (apart from the final user message) the _same messages_ are resent in the _same order_. This is because the messages will be hashed to be compared during follow up turns, to match requests to the correct session. If no hash matches everything will still work, but slower. Because a fresh context will be used and the whole input conversation will be reingested, instead of just the new user message.
+
+##### Function Calling
+Only available when using node-llama-cpp and a model that supports function calling. `tool_choice` can currently not be controlled and will always be `auto`. GBNF grammars cannot be used together with function calling.
+
+##### Huge node_modules when installing all engines
+CUDA binaries are distributed with each engine seperately, which leads to an extra 0.5-1GB of disk use. Unfortunately there is nothing I can do about that.
+
+#### TODO / Roadmap
+
+Not in any particular order:
+
+- [x] Automatic download of GGUF's with ipull
+- [x] Engine abstraction
+- [x] Model instance pool and queue
+- [x] Basic OpenAI API compatibility
+- [x] POC of chat context reuse across requests
+- [x] Tests for context reuse and context leaking
+- [x] Logging Interface
+- [x] Better Examples
+- [x] GPU support
+- [x] node-llama-cpp context reuse
+- [x] Instance TTL
+- [x] Allow configuring model hashes / verification
+- [x] Improve template code / stop trigger support
+- [x] Support configuring a timeout on completion processing
+- [x] Logit bias / Token bias support
+- [x] Improve tests for longer conversations / context window shifting
+- [x] Embeddings APIs
+- [x] Improve node-llama-cpp token usage counts / TokenMeter
+- [x] Reuse download logic from node-llama-cpp to support split ggufs.
+- [x] Support preloading instances with context, like a long system message or few shot examples
+- [x] transformers.js engine
+- [x] Support custom engine implementations
+- [x] Make sure nothing clashes if multiple servers/stores are using the same cache directory
+- [x] See if we can install supported engines as peer deps
+- [x] Improve types, simpler node-llama-cpp grammar integration
+- [x] Restructure docs, add function calling & grammar usage docs
+- [x] TTL=0 should immediately dispose of instances instead of waiting (currently on avg 30s) for the next TTL check
+- [x] Expose node-llama-cpp context shift strategy, lora, allow json schema as input for `grammar`
+- [x] Improve types for tool definitions / json schema
+- [x] Make pool dispose / stop more robust
+- [x] Tests for cancellation and timeouts
+- [x] transformer.js text embeddings
+- [x] transformer.js image embeddings
+- [x] transformer.js multimodal image/text embeddings (see [jina-clip-v1](https://github.com/xenova/transformers.js/issues/793) and [nomic-embed-vision](https://github.com/xenova/transformers.js/issues/848) issues.)
+- [x] Allow "prefilling" (partial) assistant responses like outlined [here](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/prefill-claudes-response#how-to-prefill-claudes-response)
+- [x] non-chat text completions: Allow reuse of context
+- [x] non-chat text completions: Support preloading of prefixes
+- [ ] Add some light jsdoc for server/pool/store methods
+- [ ] utilize node-llama-cpp's support to reuse LlamaModel instances with multiple contexts
+- [ ] Support transformer.js for text-completion tasks ([not yet supported in Node.js](https://github.com/huggingface/transformers.js/blob/e129c47c65a049173f35e6263fd8d9f660dfc1a7/src/models.js#L240-L242))
+- [ ] Implement more transformer.js tasks (`imageToImage`, `textToImage`, `textToSpeech`?)
+- [ ] Infill completion support https://github.com/withcatai/node-llama-cpp/blob/beta/src/evaluator/LlamaCompletion.ts#L322-L336
+- [ ] Find a way to type available custom engines (and their options?)
+- [ ] Rework GPU+device usage / lock (Support multiple models on gpu in cases where its possible)
+- [ ] Add engine interfaces for resource use (and estimates, see https://github.com/ggerganov/llama.cpp/issues/4315 and https://github.com/withcatai/node-llama-cpp/blob/beta/src/gguf/insights/utils/resolveContextContextSizeOption.ts)
+- [ ] Allow configuring a pools max memory usage
+- [ ] Test deno/bun support
+- [ ] Add image generation endpoint in oai api
+- [ ] Add transcript endpoint in oai api
+- [ ] Add `n` parameter support to node-llama-cpp chat completions
+- [ ] [CLI](https://github.com/iimez/inference-server/discussions/7)
+- [ ] Replace express with tinyhttp
+- [ ] Add stable-diffusion engine
+
+### Contributing
+
+If you are using this package - let me know where [you would like this to go](https://github.com/iimez/inference-server/discussions). Code also welcome. You find things im planning to do (eventually) above, and the wishlist below.
+
+#### Possible Future Goals
+
+- See if it would make sense to implement engines for [leejet/stable-diffusion.cpp](https://github.com/leejet/stable-diffusion.cpp) and [onnxruntime-node](https://www.npmjs.com/package/onnxruntime-node)
+- Create a separate HTTP API thats independent of the OpenAI spec and stateful. See [discussion](https://github.com/iimez/inference-server/discussions/8).
+- Add a clientside library (React hooks?) for use of above API.
+- Provide a Docker image. And maybe a Prometheus endpoint.
+- Logprobs support for node-llama-cpp.
+
+#### Currently not the Goals
+
+- A facade to LLM cloud hoster HTTP API's. The strengths here are local/private/offline use.
+- Worry too much about authentication or rate limiting or misuse. Host this with caution, it's likely DDoS-able.
+- Some kind of distributed or multi-node setup. That should probably be something designed for this purpose from the ground up.
+- Other common related tooling like vector stores, Chat GUIs, etc. Scope would probably get out of hand.
+
+### Related Solutions
+
+If you look at this package, you might also want to take a look at these other solutions:
+
+- [ollama API](https://github.com/ollama/ollama/blob/main/docs/api.md) - Uses llama.cpp and provides a HTTP API. Also has experimental OpenAI API compatibility.
+- [llama.cpp Server](https://github.com/ggerganov/llama.cpp/tree/master/examples/server#llamacpp-http-server) - The official llama.cpp HTTP API.
+- [VLLM](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html) - A more production ready solution for hosting large language models.
+- [LM Studio](https://lmstudio.ai/docs/local-server) - Also has a local server.
+- [LocalAI](https://github.com/mudler/LocalAI) - Similar project in go.
+- [Petals](https://github.com/bigscience-workshop/petals) - Local (and distributed!) inference in python.
+- [cortex.cpp](https://github.com/janhq/cortex.cpp/)

package/dist/api/openai/enums.d.ts

@@ -0,0 +1,4 @@
+import { CompletionFinishReason, ChatMessage } from '../../types/index.js';
+import OpenAI from 'openai';
+export declare const finishReasonMap: Record<CompletionFinishReason, OpenAI.ChatCompletion.Choice['finish_reason']>;
+export declare const messageRoleMap: Record<OpenAI.ChatCompletionMessageParam['role'], ChatMessage['role']>;

package/dist/api/openai/enums.js

@@ -0,0 +1,17 @@
+export const finishReasonMap = {
+    maxTokens: 'length',
+    toolCalls: 'tool_calls',
+    eogToken: 'stop',
+    stopTrigger: 'stop',
+    timeout: 'stop',
+    cancel: 'stop',
+    abort: 'stop',
+};
+export const messageRoleMap = {
+    user: 'user',
+    system: 'system',
+    assistant: 'assistant',
+    tool: 'tool',
+    function: 'tool',
+};
+//# sourceMappingURL=enums.js.map

package/dist/api/openai/enums.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"enums.js","sourceRoot":"","sources":["../../../src/api/openai/enums.ts"],"names":[],"mappings":"AAGA,MAAM,CAAC,MAAM,eAAe,GAAkF;IAC7G,SAAS,EAAE,QAAQ;IACnB,SAAS,EAAE,YAAY;IACvB,QAAQ,EAAE,MAAM;IAChB,WAAW,EAAE,MAAM;IACnB,OAAO,EAAE,MAAM;IACf,MAAM,EAAE,MAAM;IACd,KAAK,EAAE,MAAM;CACJ,CAAA;AAEV,MAAM,CAAC,MAAM,cAAc,GAA2E;IACrG,IAAI,EAAE,MAAM;IACZ,MAAM,EAAE,QAAQ;IAChB,SAAS,EAAE,WAAW;IACtB,IAAI,EAAE,MAAM;IACZ,QAAQ,EAAE,MAAM;CAChB,CAAA"}

package/dist/api/openai/handlers/chat.d.ts

@@ -0,0 +1,3 @@
+import type { IncomingMessage, ServerResponse } from 'node:http';
+import type { ModelServer } from '../../../server.js';
+export declare function createChatCompletionHandler(modelServer: ModelServer): (req: IncomingMessage, res: ServerResponse) => Promise<void>;

package/dist/api/openai/handlers/chat.js

@@ -0,0 +1,358 @@
+import { parseJSONRequestBody } from '../../../api/parseJSONRequestBody.js';
+import { omitEmptyValues } from '../../../lib/util.js';
+import { loadImageFromUrl } from '../../../lib/loadImage.js';
+import { finishReasonMap, messageRoleMap } from '../enums.js';
+async function prepareIncomingMessages(messages) {
+    const downloadPromises = {};
+    const resultMessages = [];
+    for (const message of messages) {
+        const role = messageRoleMap[message.role];
+        const resultMessage = {
+            role,
+            content: [],
+        };
+        if (role === 'tool' && 'tool_call_id' in message) {
+            resultMessage.callId = message.tool_call_id;
+        }
+        if (typeof message.content === 'string') {
+            resultMessage.content.push({
+                type: 'text',
+                text: message.content,
+            });
+        }
+        else if (Array.isArray(message.content)) {
+            for (const part of message.content) {
+                if (typeof part === 'string') {
+                    resultMessage.content.push({
+                        type: 'text',
+                        text: part,
+                    });
+                }
+                else if (part.type === 'text') {
+                    resultMessage.content.push({
+                        type: 'text',
+                        text: part.text,
+                    });
+                }
+                else if (part.type === 'image_url') {
+                    if (!downloadPromises[part.image_url.url]) {
+                        downloadPromises[part.image_url.url] = loadImageFromUrl(part.image_url.url);
+                    }
+                    const content = {
+                        type: 'image',
+                    };
+                    resultMessage.content.push(content);
+                    downloadPromises[part.image_url.url].then((image) => {
+                        content.image = image;
+                    });
+                }
+                else if (part.type === 'input_audio') {
+                    resultMessage.content.push({
+                        type: 'audio',
+                        audio: part.input_audio,
+                    });
+                }
+                else if (part.type === 'refusal') {
+                    resultMessage.content.push({
+                        type: 'text',
+                        text: part.refusal,
+                    });
+                }
+            }
+        }
+        else {
+            throw new Error('Invalid message content');
+        }
+        resultMessages.push(resultMessage);
+    }
+    await Promise.all(Object.values(downloadPromises));
+    return resultMessages;
+}
+function createResponseMessageContent(content) {
+    if (!content) {
+        return null;
+    }
+    if (typeof content === 'string') {
+        return content;
+    }
+    if (!Array.isArray(content)) {
+        throw new Error('Invalid response message content');
+    }
+    let text = '';
+    for (const part of content) {
+        if (part.type === 'text') {
+            text += part.text;
+        }
+        // assistant may only respond with text in openai chat completions
+    }
+    return text;
+}
+// v1/chat/completions
+// https://platform.openai.com/docs/api-reference/chat/create
+export function createChatCompletionHandler(modelServer) {
+    return async (req, res) => {
+        let args;
+        try {
+            const body = await parseJSONRequestBody(req);
+            args = body;
+        }
+        catch (e) {
+            console.error(e);
+            res.writeHead(400, { 'Content-Type': 'application/json' });
+            res.end(JSON.stringify({ error: 'Invalid request' }));
+            return;
+        }
+        // TODO ajv schema validation?
+        if (!args.model || !args.messages) {
+            res.writeHead(400, { 'Content-Type': 'application/json' });
+            res.end(JSON.stringify({ error: 'Invalid request (need at least model and messages)' }));
+            return;
+        }
+        if (!modelServer.modelExists(args.model)) {
+            res.writeHead(400, { 'Content-Type': 'application/json' });
+            res.end(JSON.stringify({ error: 'Model does not exist' }));
+            return;
+        }
+        const controller = new AbortController();
+        req.on('close', () => {
+            console.debug('Client closed connection');
+            controller.abort();
+        });
+        req.on('end', () => {
+            console.debug('Client ended connection');
+            controller.abort();
+        });
+        req.on('aborted', () => {
+            console.debug('Client aborted connection');
+            controller.abort();
+        });
+        req.on('error', () => {
+            console.debug('Client error');
+            controller.abort();
+        });
+        try {
+            let ssePing;
+            if (args.stream) {
+                res.writeHead(200, {
+                    'Content-Type': 'text/event-stream',
+                    'Cache-Control': 'no-cache',
+                    Connection: 'keep-alive',
+                });
+                res.flushHeaders();
+                ssePing = setInterval(() => {
+                    res.write(':ping\n\n');
+                }, 30000);
+            }
+            let stop = args.stop ? args.stop : undefined;
+            if (typeof stop === 'string') {
+                stop = [stop];
+            }
+            let completionGrammar;
+            if (args.response_format) {
+                if (args.response_format.type === 'json_object') {
+                    completionGrammar = 'json';
+                }
+            }
+            let completionTools;
+            if (args.tools) {
+                const functionTools = args.tools
+                    .filter((tool) => tool.type === 'function')
+                    .map((tool) => {
+                    return {
+                        name: tool.function.name,
+                        description: tool.function.description,
+                        parameters: tool.function.parameters,
+                    };
+                });
+                if (functionTools.length) {
+                    if (!completionTools) {
+                        completionTools = {};
+                    }
+                    for (const tool of functionTools) {
+                        completionTools[tool.name] = {
+                            description: tool.description,
+                            parameters: tool.parameters,
+                        };
+                    }
+                }
+            }
+            const messages = await prepareIncomingMessages(args.messages);
+            const completionReq = omitEmptyValues({
+                model: args.model,
+                messages,
+                temperature: args.temperature ? args.temperature : undefined,
+                stream: args.stream ? Boolean(args.stream) : false,
+                maxTokens: args.max_tokens ? args.max_tokens : undefined,
+                seed: args.seed ? args.seed : undefined,
+                stop,
+                frequencyPenalty: args.frequency_penalty
+                    ? args.frequency_penalty
+                    : undefined,
+                presencePenalty: args.presence_penalty
+                    ? args.presence_penalty
+                    : undefined,
+                topP: args.top_p ? args.top_p : undefined,
+                tokenBias: args.logit_bias ? args.logit_bias : undefined,
+                grammar: completionGrammar,
+                tools: completionTools,
+                // additional non-spec params
+                repeatPenaltyNum: args.repeat_penalty_num
+                    ? args.repeat_penalty_num
+                    : undefined,
+                minP: args.min_p ? args.min_p : undefined,
+                topK: args.top_k ? args.top_k : undefined,
+            });
+            const { instance, release } = await modelServer.requestInstance(completionReq, controller.signal);
+            if (ssePing) {
+                clearInterval(ssePing);
+            }
+            const task = instance.processChatCompletionTask(completionReq, {
+                signal: controller.signal,
+                onChunk: (chunk) => {
+                    if (args.stream) {
+                        const chunkData = {
+                            id: task.id,
+                            object: 'chat.completion.chunk',
+                            model: task.model,
+                            created: Math.floor(task.createdAt.getTime() / 1000),
+                            choices: [
+                                {
+                                    index: 0,
+                                    delta: {
+                                        role: 'assistant',
+                                        content: chunk.text,
+                                    },
+                                    logprobs: null,
+                                    finish_reason: null,
+                                },
+                            ],
+                        };
+                        res.write(`data: ${JSON.stringify(chunkData)}\n\n`);
+                    }
+                },
+            });
+            const result = await task.result;
+            release();
+            if (args.stream) {
+                if (result.finishReason === 'toolCalls') {
+                    // currently not possible to stream function calls
+                    // imitating a stream here by sending two chunks. makes it work with the openai client
+                    const streamedToolCallChunk = {
+                        id: task.id,
+                        object: 'chat.completion.chunk',
+                        model: task.model,
+                        created: Math.floor(task.createdAt.getTime() / 1000),
+                        choices: [
+                            {
+                                index: 0,
+                                delta: {
+                                    role: 'assistant',
+                                    content: null,
+                                },
+                                logprobs: null,
+                                finish_reason: result.finishReason
+                                    ? finishReasonMap[result.finishReason]
+                                    : 'stop',
+                            },
+                        ],
+                    };
+                    const toolCalls = result.message.toolCalls.map((call, index) => {
+                        return {
+                            index,
+                            id: call.id,
+                            type: 'function',
+                            function: {
+                                name: call.name,
+                                arguments: JSON.stringify(call.parameters),
+                            },
+                        };
+                    });
+                    streamedToolCallChunk.choices[0].delta.tool_calls = toolCalls;
+                    res.write(`data: ${JSON.stringify(streamedToolCallChunk)}\n\n`);
+                }
+                if (args.stream_options?.include_usage) {
+                    const finalChunk = {
+                        id: task.id,
+                        object: 'chat.completion.chunk',
+                        model: task.model,
+                        created: Math.floor(task.createdAt.getTime() / 1000),
+                        system_fingerprint: instance.fingerprint,
+                        choices: [
+                            {
+                                index: 0,
+                                delta: {},
+                                logprobs: null,
+                                finish_reason: result.finishReason
+                                    ? finishReasonMap[result.finishReason]
+                                    : 'stop',
+                            },
+                        ],
+                        usage: {
+                            prompt_tokens: result.promptTokens,
+                            completion_tokens: result.completionTokens,
+                            total_tokens: result.contextTokens,
+                        },
+                    };
+                    res.write(`data: ${JSON.stringify(finalChunk)}\n\n`);
+                }
+                res.write('data: [DONE]');
+                res.end();
+            }
+            else {
+                const response = {
+                    id: task.id,
+                    model: task.model,
+                    object: 'chat.completion',
+                    created: Math.floor(task.createdAt.getTime() / 1000),
+                    system_fingerprint: instance.fingerprint,
+                    choices: [
+                        {
+                            index: 0,
+                            message: {
+                                role: 'assistant',
+                                content: createResponseMessageContent(result.message.content),
+                                refusal: null,
+                            },
+                            logprobs: null,
+                            finish_reason: result.finishReason
+                                ? finishReasonMap[result.finishReason]
+                                : 'stop',
+                        },
+                    ],
+                    usage: {
+                        prompt_tokens: result.promptTokens,
+                        completion_tokens: result.completionTokens,
+                        total_tokens: result.contextTokens,
+                    },
+                };
+                if ('toolCalls' in result.message &&
+                    result.message.toolCalls?.length) {
+                    response.choices[0].message.tool_calls =
+                        result.message.toolCalls.map((call) => {
+                            return {
+                                id: call.id,
+                                type: 'function',
+                                function: {
+                                    name: call.name,
+                                    arguments: JSON.stringify(call.parameters),
+                                },
+                            };
+                        });
+                }
+                res.writeHead(200, { 'Content-Type': 'application/json' });
+                res.end(JSON.stringify(response, null, 2));
+            }
+        }
+        catch (e) {
+            console.error(e);
+            if (args.stream) {
+                res.write('data: [ERROR]');
+            }
+            else {
+                res.writeHead(500, { 'Content-Type': 'application/json' });
+                res.end(JSON.stringify({ error: 'Internal server error' }));
+            }
+        }
+    };
+}
+//# sourceMappingURL=chat.js.map

package/dist/api/openai/handlers/chat.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"chat.js","sourceRoot":"","sources":["../../../../src/api/openai/handlers/chat.ts"],"names":[],"mappings":"AAWA,OAAO,EAAE,oBAAoB,EAAE,MAAM,sCAAsC,CAAA;AAC3E,OAAO,EAAE,eAAe,EAAE,MAAM,sBAAsB,CAAA;AACtD,OAAO,EAAE,gBAAgB,EAAE,MAAM,2BAA2B,CAAA;AAC5D,OAAO,EAAE,eAAe,EAAE,cAAc,EAAE,MAAM,aAAa,CAAA;AAc7D,KAAK,UAAU,uBAAuB,CACrC,QAAsC;IAEtC,MAAM,gBAAgB,GAAmC,EAAE,CAAA;IAC3D,MAAM,cAAc,GAAkB,EAAE,CAAA;IAExC,KAAK,MAAM,OAAO,IAAI,QAAQ,EAAE,CAAC;QAChC,MAAM,IAAI,GAAG,cAAc,CAAC,OAAO,CAAC,IAAI,CAAC,CAAA;QACzC,MAAM,aAAa,GAAQ;YAC1B,IAAI;YACJ,OAAO,EAAE,EAAE;SACX,CAAA;QACD,IAAI,IAAI,KAAK,MAAM,IAAI,cAAc,IAAI,OAAO,EAAE,CAAC;YAClD,aAAa,CAAC,MAAM,GAAG,OAAO,CAAC,YAAY,CAAA;QAC5C,CAAC;QAED,IAAI,OAAO,OAAO,CAAC,OAAO,KAAK,QAAQ,EAAE,CAAC;YACzC,aAAa,CAAC,OAAO,CAAC,IAAI,CAAC;gBAC1B,IAAI,EAAE,MAAM;gBACZ,IAAI,EAAE,OAAO,CAAC,OAAO;aACrB,CAAC,CAAA;QACH,CAAC;aAAM,IAAI,KAAK,CAAC,OAAO,CAAC,OAAO,CAAC,OAAO,CAAC,EAAE,CAAC;YAC3C,KAAK,MAAM,IAAI,IAAI,OAAO,CAAC,OAAO,EAAE,CAAC;gBAEpC,IAAI,OAAO,IAAI,KAAK,QAAQ,EAAE,CAAC;oBAC9B,aAAa,CAAC,OAAO,CAAC,IAAI,CAAC;wBAC1B,IAAI,EAAE,MAAM;wBACZ,IAAI,EAAE,IAAI;qBACV,CAAC,CAAA;gBACH,CAAC;qBAAM,IAAI,IAAI,CAAC,IAAI,KAAK,MAAM,EAAE,CAAC;oBACjC,aAAa,CAAC,OAAO,CAAC,IAAI,CAAC;wBAC1B,IAAI,EAAE,MAAM;wBACZ,IAAI,EAAE,IAAI,CAAC,IAAI;qBACf,CAAC,CAAA;gBACH,CAAC;qBAAM,IAAI,IAAI,CAAC,IAAI,KAAK,WAAW,EAAE,CAAC;oBACtC,IAAI,CAAC,gBAAgB,CAAC,IAAI,CAAC,SAAS,CAAC,GAAG,CAAC,EAAE,CAAC;wBAC3C,gBAAgB,CAAC,IAAI,CAAC,SAAS,CAAC,GAAG,CAAC,GAAG,gBAAgB,CAAC,IAAI,CAAC,SAAS,CAAC,GAAG,CAAC,CAAA;oBAC5E,CAAC;oBACD,MAAM,OAAO,GAAgC;wBAC5C,IAAI,EAAE,OAAO;qBACb,CAAA;oBACD,aAAa,CAAC,OAAO,CAAC,IAAI,CAAC,OAAO,CAAC,CAAA;oBACnC,gBAAgB,CAAC,IAAI,CAAC,SAAS,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC,CAAC,KAAK,EAAE,EAAE;wBACnD,OAAO,CAAC,KAAK,GAAG,KAAK,CAAA;oBACtB,CAAC,CAAC,CAAA;gBACH,CAAC;qBAAM,IAAI,IAAI,CAAC,IAAI,KAAK,aAAa,EAAE,CAAC;oBACxC,aAAa,CAAC,OAAO,CAAC,IAAI,CAAC;wBAC1B,IAAI,EAAE,OAAO;wBACb,KAAK,EAAE,IAAI,CAAC,WAAW;qBACvB,CAAC,CAAA;gBACH,CAAC;qBAAM,IAAI,IAAI,CAAC,IAAI,KAAK,SAAS,EAAE,CAAC;oBACpC,aAAa,CAAC,OAAO,CAAC,IAAI,CAAC;wBAC1B,IAAI,EAAE,MAAM;wBACZ,IAAI,EAAE,IAAI,CAAC,OAAO;qBAClB,CAAC,CAAA;gBACH,CAAC;YACF,CAAC;QACF,CAAC;aAAM,CAAC;YACP,MAAM,IAAI,KAAK,CAAC,yBAAyB,CAAC,CAAA;QAC3C,CAAC;QAED,cAAc,CAAC,IAAI,CAAC,aAAa,CAAC,CAAA;IACnC,CAAC;IAED,MAAM,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,MAAM,CAAC,gBAAgB,CAAC,CAAC,CAAA;IAElD,OAAO,cAAc,CAAA;AAEtB,CAAC;AAED,SAAS,4BAA4B,CACpC,OAAsC;IAEtC,IAAI,CAAC,OAAO,EAAE,CAAC;QACd,OAAO,IAAI,CAAA;IACZ,CAAC;IACD,IAAI,OAAO,OAAO,KAAK,QAAQ,EAAE,CAAC;QACjC,OAAO,OAAO,CAAA;IACf,CAAC;IACD,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,OAAO,CAAC,EAAE,CAAC;QAC7B,MAAM,IAAI,KAAK,CAAC,kCAAkC,CAAC,CAAA;IACpD,CAAC;IAED,IAAI,IAAI,GAAG,EAAE,CAAA;IACb,KAAK,MAAM,IAAI,IAAI,OAAO,EAAE,CAAC;QAC5B,IAAI,IAAI,CAAC,IAAI,KAAK,MAAM,EAAE,CAAC;YAC1B,IAAI,IAAI,IAAI,CAAC,IAAI,CAAA;QAClB,CAAC;QACD,kEAAkE;IACnE,CAAC;IACD,OAAO,IAAI,CAAA;AACZ,CAAC;AAED,sBAAsB;AACtB,6DAA6D;AAC7D,MAAM,UAAU,2BAA2B,CAAC,WAAwB;IACnE,OAAO,KAAK,EAAE,GAAoB,EAAE,GAAmB,EAAE,EAAE;QAC1D,IAAI,IAAgC,CAAA;QAEpC,IAAI,CAAC;YACJ,MAAM,IAAI,GAAG,MAAM,oBAAoB,CAAC,GAAG,CAAC,CAAA;YAC5C,IAAI,GAAG,IAAI,CAAA;QACZ,CAAC;QAAC,OAAO,CAAC,EAAE,CAAC;YACZ,OAAO,CAAC,KAAK,CAAC,CAAC,CAAC,CAAA;YAChB,GAAG,CAAC,SAAS,CAAC,GAAG,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE,CAAC,CAAA;YAC1D,GAAG,CAAC,GAAG,CAAC,IAAI,CAAC,SAAS,CAAC,EAAE,KAAK,EAAE,iBAAiB,EAAE,CAAC,CAAC,CAAA;YACrD,OAAM;QACP,CAAC;QAED,8BAA8B;QAC9B,IAAI,CAAC,IAAI,CAAC,KAAK,IAAI,CAAC,IAAI,CAAC,QAAQ,EAAE,CAAC;YACnC,GAAG,CAAC,SAAS,CAAC,GAAG,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE,CAAC,CAAA;YAC1D,GAAG,CAAC,GAAG,CAAC,IAAI,CAAC,SAAS,CAAC,EAAE,KAAK,EAAE,oDAAoD,EAAE,CAAC,CAAC,CAAA;YACxF,OAAM;QACP,CAAC;QAED,IAAI,CAAC,WAAW,CAAC,WAAW,CAAC,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC;YAC1C,GAAG,CAAC,SAAS,CAAC,GAAG,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE,CAAC,CAAA;YAC1D,GAAG,CAAC,GAAG,CAAC,IAAI,CAAC,SAAS,CAAC,EAAE,KAAK,EAAE,sBAAsB,EAAE,CAAC,CAAC,CAAA;YAC1D,OAAM;QACP,CAAC;QAED,MAAM,UAAU,GAAG,IAAI,eAAe,EAAE,CAAA;QACxC,GAAG,CAAC,EAAE,CAAC,OAAO,EAAE,GAAG,EAAE;YACpB,OAAO,CAAC,KAAK,CAAC,0BAA0B,CAAC,CAAA;YACzC,UAAU,CAAC,KAAK,EAAE,CAAA;QACnB,CAAC,CAAC,CAAA;QACF,GAAG,CAAC,EAAE,CAAC,KAAK,EAAE,GAAG,EAAE;YAClB,OAAO,CAAC,KAAK,CAAC,yBAAyB,CAAC,CAAA;YACxC,UAAU,CAAC,KAAK,EAAE,CAAA;QACnB,CAAC,CAAC,CAAA;QACF,GAAG,CAAC,EAAE,CAAC,SAAS,EAAE,GAAG,EAAE;YACtB,OAAO,CAAC,KAAK,CAAC,2BAA2B,CAAC,CAAA;YAC1C,UAAU,CAAC,KAAK,EAAE,CAAA;QACnB,CAAC,CAAC,CAAA;QACF,GAAG,CAAC,EAAE,CAAC,OAAO,EAAE,GAAG,EAAE;YACpB,OAAO,CAAC,KAAK,CAAC,cAAc,CAAC,CAAA;YAC7B,UAAU,CAAC,KAAK,EAAE,CAAA;QACnB,CAAC,CAAC,CAAA;QAEF,IAAI,CAAC;YACJ,IAAI,OAAmC,CAAA;YACvC,IAAI,IAAI,CAAC,MAAM,EAAE,CAAC;gBACjB,GAAG,CAAC,SAAS,CAAC,GAAG,EAAE;oBAClB,cAAc,EAAE,mBAAmB;oBACnC,eAAe,EAAE,UAAU;oBAC3B,UAAU,EAAE,YAAY;iBACxB,CAAC,CAAA;gBACF,GAAG,CAAC,YAAY,EAAE,CAAA;gBAClB,OAAO,GAAG,WAAW,CAAC,GAAG,EAAE;oBAC1B,GAAG,CAAC,KAAK,CAAC,WAAW,CAAC,CAAA;gBACvB,CAAC,EAAE,KAAK,CAAC,CAAA;YACV,CAAC;YAED,IAAI,IAAI,GAAG,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,SAAS,CAAA;YAC5C,IAAI,OAAO,IAAI,KAAK,QAAQ,EAAE,CAAC;gBAC9B,IAAI,GAAG,CAAC,IAAI,CAAC,CAAA;YACd,CAAC;YAED,IAAI,iBAAqC,CAAA;YACzC,IAAI,IAAI,CAAC,eAAe,EAAE,CAAC;gBAC1B,IAAI,IAAI,CAAC,eAAe,CAAC,IAAI,KAAK,aAAa,EAAE,CAAC;oBACjD,iBAAiB,GAAG,MAAM,CAAA;gBAC3B,CAAC;YACF,CAAC;YAED,IAAI,eAEQ,CAAA;YAEZ,IAAI,IAAI,CAAC,KAAK,EAAE,CAAC;gBAChB,MAAM,aAAa,GAAG,IAAI,CAAC,KAAK;qBAC9B,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,IAAI,KAAK,UAAU,CAAC;qBAC1C,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE;oBACb,OAAO;wBACN,IAAI,EAAE,IAAI,CAAC,QAAQ,CAAC,IAAI;wBACxB,WAAW,EAAE,IAAI,CAAC,QAAQ,CAAC,WAAW;wBACtC,UAAU,EAAE,IAAI,CAAC,QAAQ,CAAC,UAAU;qBACpC,CAAA;gBACF,CAAC,CAAC,CAAA;gBACH,IAAI,aAAa,CAAC,MAAM,EAAE,CAAC;oBAC1B,IAAI,CAAC,eAAe,EAAE,CAAC;wBACtB,eAAe,GAAG,EAAE,CAAA;oBACrB,CAAC;oBACD,KAAK,MAAM,IAAI,IAAI,aAAa,EAAE,CAAC;wBAClC,eAAe,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG;4BAC5B,WAAW,EAAE,IAAI,CAAC,WAAW;4BAC7B,UAAU,EAAE,IAAI,CAAC,UAAU;yBACT,CAAA;oBACpB,CAAC;gBACF,CAAC;YACF,CAAC;YAED,MAAM,QAAQ,GAAG,MAAM,uBAAuB,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAA;YAC7D,MAAM,aAAa,GAAG,eAAe,CAAwB;gBAC5D,KAAK,EAAE,IAAI,CAAC,KAAK;gBACjB,QAAQ;gBACR,WAAW,EAAE,IAAI,CAAC,WAAW,CAAC,CAAC,CAAC,IAAI,CAAC,WAAW,CAAC,CAAC,CAAC,SAAS;gBAC5D,MAAM,EAAE,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC,OAAO,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,KAAK;gBAClD,SAAS,EAAE,IAAI,CAAC,UAAU,CAAC,CAAC,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC,CAAC,SAAS;gBACxD,IAAI,EAAE,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,SAAS;gBACvC,IAAI;gBACJ,gBAAgB,EAAE,IAAI,CAAC,iBAAiB;oBACvC,CAAC,CAAC,IAAI,CAAC,iBAAiB;oBACxB,CAAC,CAAC,SAAS;gBACZ,eAAe,EAAE,IAAI,CAAC,gBAAgB;oBACrC,CAAC,CAAC,IAAI,CAAC,gBAAgB;oBACvB,CAAC,CAAC,SAAS;gBACZ,IAAI,EAAE,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,SAAS;gBACzC,SAAS,EAAE,IAAI,CAAC,UAAU,CAAC,CAAC,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC,CAAC,SAAS;gBACxD,OAAO,EAAE,iBAAiB;gBAC1B,KAAK,EAAE,eAAe;gBACtB,6BAA6B;gBAC7B,gBAAgB,EAAE,IAAI,CAAC,kBAAkB;oBACxC,CAAC,CAAC,IAAI,CAAC,kBAAkB;oBACzB,CAAC,CAAC,SAAS;gBACZ,IAAI,EAAE,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,SAAS;gBACzC,IAAI,EAAE,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,SAAS;aACzC,CAAC,CAAA;YACF,MAAM,EAAE,QAAQ,EAAE,OAAO,EAAE,GAAG,MAAM,WAAW,CAAC,eAAe,CAC9D,aAAa,EACb,UAAU,CAAC,MAAM,CACjB,CAAA;YAED,IAAI,OAAO,EAAE,CAAC;gBACb,aAAa,CAAC,OAAO,CAAC,CAAA;YACvB,CAAC;YACD,MAAM,IAAI,GAAG,QAAQ,CAAC,yBAAyB,CAAC,aAAa,EAAE;gBAC9D,MAAM,EAAE,UAAU,CAAC,MAAM;gBACzB,OAAO,EAAE,CAAC,KAAK,EAAE,EAAE;oBAClB,IAAI,IAAI,CAAC,MAAM,EAAE,CAAC;wBACjB,MAAM,SAAS,GAA8B;4BAC5C,EAAE,EAAE,IAAI,CAAC,EAAE;4BACX,MAAM,EAAE,uBAAuB;4BAC/B,KAAK,EAAE,IAAI,CAAC,KAAK;4BACjB,OAAO,EAAE,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,SAAS,CAAC,OAAO,EAAE,GAAG,IAAI,CAAC;4BACpD,OAAO,EAAE;gCACR;oCACC,KAAK,EAAE,CAAC;oCACR,KAAK,EAAE;wCACN,IAAI,EAAE,WAAW;wCACjB,OAAO,EAAE,KAAK,CAAC,IAAI;qCACnB;oCACD,QAAQ,EAAE,IAAI;oCACd,aAAa,EAAE,IAAI;iCACnB;6BACD;yBACD,CAAA;wBACD,GAAG,CAAC,KAAK,CAAC,SAAS,IAAI,CAAC,SAAS,CAAC,SAAS,CAAC,MAAM,CAAC,CAAA;oBACpD,CAAC;gBACF,CAAC;aACD,CAAC,CAAA;YAEF,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,MAAM,CAAA;YAEhC,OAAO,EAAE,CAAA;YAET,IAAI,IAAI,CAAC,MAAM,EAAE,CAAC;gBACjB,IAAI,MAAM,CAAC,YAAY,KAAK,WAAW,EAAE,CAAC;oBACzC,kDAAkD;oBAClD,sFAAsF;oBACtF,MAAM,qBAAqB,GAA8B;wBACxD,EAAE,EAAE,IAAI,CAAC,EAAE;wBACX,MAAM,EAAE,uBAAuB;wBAC/B,KAAK,EAAE,IAAI,CAAC,KAAK;wBACjB,OAAO,EAAE,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,SAAS,CAAC,OAAO,EAAE,GAAG,IAAI,CAAC;wBACpD,OAAO,EAAE;4BACR;gCACC,KAAK,EAAE,CAAC;gCACR,KAAK,EAAE;oCACN,IAAI,EAAE,WAAW;oCACjB,OAAO,EAAE,IAAI;iCACb;gCACD,QAAQ,EAAE,IAAI;gCACd,aAAa,EAAE,MAAM,CAAC,YAAY;oCACjC,CAAC,CAAC,eAAe,CAAC,MAAM,CAAC,YAAY,CAAC;oCACtC,CAAC,CAAC,MAAM;6BACT;yBACD;qBACD,CAAA;oBAED,MAAM,SAAS,GACd,MAAM,CAAC,OAAO,CAAC,SAAU,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,KAAK,EAAE,EAAE;wBAC7C,OAAO;4BACN,KAAK;4BACL,EAAE,EAAE,IAAI,CAAC,EAAE;4BACX,IAAI,EAAE,UAAU;4BAChB,QAAQ,EAAE;gCACT,IAAI,EAAE,IAAI,CAAC,IAAI;gCACf,SAAS,EAAE,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,UAAU,CAAC;6BAC1C;yBACD,CAAA;oBACF,CAAC,CAAC,CAAA;oBACH,qBAAqB,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,UAAU,GAAG,SAAS,CAAA;oBAC7D,GAAG,CAAC,KAAK,CAAC,SAAS,IAAI,CAAC,SAAS,CAAC,qBAAqB,CAAC,MAAM,CAAC,CAAA;gBAChE,CAAC;gBACD,IAAI,IAAI,CAAC,cAAc,EAAE,aAAa,EAAE,CAAC;oBACxC,MAAM,UAAU,GAA8B;wBAC7C,EAAE,EAAE,IAAI,CAAC,EAAE;wBACX,MAAM,EAAE,uBAAuB;wBAC/B,KAAK,EAAE,IAAI,CAAC,KAAK;wBACjB,OAAO,EAAE,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,SAAS,CAAC,OAAO,EAAE,GAAG,IAAI,CAAC;wBACpD,kBAAkB,EAAE,QAAQ,CAAC,WAAW;wBACxC,OAAO,EAAE;4BACR;gCACC,KAAK,EAAE,CAAC;gCACR,KAAK,EAAE,EAAE;gCACT,QAAQ,EAAE,IAAI;gCACd,aAAa,EAAE,MAAM,CAAC,YAAY;oCACjC,CAAC,CAAC,eAAe,CAAC,MAAM,CAAC,YAAY,CAAC;oCACtC,CAAC,CAAC,MAAM;6BACT;yBACD;wBACD,KAAK,EAAE;4BACN,aAAa,EAAE,MAAM,CAAC,YAAY;4BAClC,iBAAiB,EAAE,MAAM,CAAC,gBAAgB;4BAC1C,YAAY,EAAE,MAAM,CAAC,aAAa;yBAClC;qBACD,CAAA;oBACD,GAAG,CAAC,KAAK,CAAC,SAAS,IAAI,CAAC,SAAS,CAAC,UAAU,CAAC,MAAM,CAAC,CAAA;gBACrD,CAAC;gBACD,GAAG,CAAC,KAAK,CAAC,cAAc,CAAC,CAAA;gBACzB,GAAG,CAAC,GAAG,EAAE,CAAA;YACV,CAAC;iBAAM,CAAC;gBACP,MAAM,QAAQ,GAA0B;oBACvC,EAAE,EAAE,IAAI,CAAC,EAAE;oBACX,KAAK,EAAE,IAAI,CAAC,KAAK;oBACjB,MAAM,EAAE,iBAAiB;oBACzB,OAAO,EAAE,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,SAAS,CAAC,OAAO,EAAE,GAAG,IAAI,CAAC;oBACpD,kBAAkB,EAAE,QAAQ,CAAC,WAAW;oBACxC,OAAO,EAAE;wBACR;4BACC,KAAK,EAAE,CAAC;4BACR,OAAO,EAAE;gCACR,IAAI,EAAE,WAAW;gCACjB,OAAO,EAAE,4BAA4B,CAAC,MAAM,CAAC,OAAO,CAAC,OAAO,CAAC;gCAC7D,OAAO,EAAE,IAAI;6BACb;4BACD,QAAQ,EAAE,IAAI;4BACd,aAAa,EAAE,MAAM,CAAC,YAAY;gCACjC,CAAC,CAAC,eAAe,CAAC,MAAM,CAAC,YAAY,CAAC;gCACtC,CAAC,CAAC,MAAM;yBACT;qBACD;oBACD,KAAK,EAAE;wBACN,aAAa,EAAE,MAAM,CAAC,YAAY;wBAClC,iBAAiB,EAAE,MAAM,CAAC,gBAAgB;wBAC1C,YAAY,EAAE,MAAM,CAAC,aAAa;qBAClC;iBACD,CAAA;gBACD,IACC,WAAW,IAAI,MAAM,CAAC,OAAO;oBAC7B,MAAM,CAAC,OAAO,CAAC,SAAS,EAAE,MAAM,EAC/B,CAAC;oBACF,QAAQ,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,UAAU;wBACrC,MAAM,CAAC,OAAO,CAAC,SAAS,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE;4BACrC,OAAO;gCACN,EAAE,EAAE,IAAI,CAAC,EAAE;gCACX,IAAI,EAAE,UAAU;gCAChB,QAAQ,EAAE;oCACT,IAAI,EAAE,IAAI,CAAC,IAAI;oCACf,SAAS,EAAE,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,UAAU,CAAC;iCAC1C;6BACD,CAAA;wBACF,CAAC,CAAC,CAAA;gBACJ,CAAC;gBACD,GAAG,CAAC,SAAS,CAAC,GAAG,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE,CAAC,CAAA;gBAC1D,GAAG,CAAC,GAAG,CAAC,IAAI,CAAC,SAAS,CAAC,QAAQ,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,CAAA;YAC3C,CAAC;QACF,CAAC;QAAC,OAAO,CAAC,EAAE,CAAC;YACZ,OAAO,CAAC,KAAK,CAAC,CAAC,CAAC,CAAA;YAChB,IAAI,IAAI,CAAC,MAAM,EAAE,CAAC;gBACjB,GAAG,CAAC,KAAK,CAAC,eAAe,CAAC,CAAA;YAC3B,CAAC;iBAAM,CAAC;gBACP,GAAG,CAAC,SAAS,CAAC,GAAG,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE,CAAC,CAAA;gBAC1D,GAAG,CAAC,GAAG,CAAC,IAAI,CAAC,SAAS,CAAC,EAAE,KAAK,EAAE,uBAAuB,EAAE,CAAC,CAAC,CAAA;YAC5D,CAAC;QACF,CAAC;IACF,CAAC,CAAA;AACF,CAAC"}

package/dist/api/openai/handlers/completions.d.ts

@@ -0,0 +1,3 @@
+import type { IncomingMessage, ServerResponse } from 'node:http';
+import type { ModelServer } from '../../../server.js';
+export declare function createCompletionHandler(modelServer: ModelServer): (req: IncomingMessage, res: ServerResponse) => Promise<void>;