modelfusion 0.53.2 → 0.55.0

package/README.md

@@ -20,6 +20,7 @@
 - **Type inference and validation**: ModelFusion infers TypeScript types wherever possible and to validates model responses.
 - **Observability and logging**: ModelFusion provides an observer framework and out-of-the-box logging support.
 - **Resilience and Robustness**: ModelFusion ensures seamless operation through automatic retries, throttling, and error handling mechanisms.
+- **Server**: ModelFusion provides a Fastify plugin that exposes a ModelFusion flow as a REST endpoint that uses server-sent events.
 
 ## Quick Install
 
@@ -53,7 +54,7 @@ const text = await generateText(
 );
 ```
 
-Providers: [OpenAI](https://modelfusion.dev/integration/model-provider/openai), [Anthropic](https://modelfusion.dev/integration/model-provider/anthropic), [Cohere](https://modelfusion.dev/integration/model-provider/cohere), [Llama.cpp](https://modelfusion.dev/integration/model-provider/llamacpp), [Hugging Face](https://modelfusion.dev/integration/model-provider/huggingface)
+Providers: [OpenAI](https://modelfusion.dev/integration/model-provider/openai), [Anthropic](https://modelfusion.dev/integration/model-provider/anthropic), [Cohere](https://modelfusion.dev/integration/model-provider/cohere), [Llama.cpp](https://modelfusion.dev/integration/model-provider/llamacpp), [Ollama](https://modelfusion.dev/integration/model-provider/ollama), [Hugging Face](https://modelfusion.dev/integration/model-provider/huggingface)
 
 #### streamText
 
@@ -70,7 +71,7 @@ for await (const textPart of textStream) {
 }
 ```
 
-Providers: [OpenAI](https://modelfusion.dev/integration/model-provider/openai), [Anthropic](https://modelfusion.dev/integration/model-provider/anthropic), [Cohere](https://modelfusion.dev/integration/model-provider/cohere), [Llama.cpp](https://modelfusion.dev/integration/model-provider/llamacpp)
+Providers: [OpenAI](https://modelfusion.dev/integration/model-provider/openai), [Anthropic](https://modelfusion.dev/integration/model-provider/anthropic), [Cohere](https://modelfusion.dev/integration/model-provider/cohere), [Llama.cpp](https://modelfusion.dev/integration/model-provider/llamacpp), [Ollama](https://modelfusion.dev/integration/model-provider/ollama)
 
 ### [Generate Image](https://modelfusion.dev/guide/function/generate-image)
 
@@ -543,6 +544,71 @@ ModelFusion provides an [observer framework](https://modelfusion.dev/guide/util/
 setGlobalFunctionLogging("detailed-object"); // log full events
 ```
 
+### [Server](https://modelfusion.dev/guide/server/)
+
+> [!WARNING]
+> ModelFusion Server is in its initial development phase and not feature-complete. The API is experimental and breaking changes are likely. Feedback and suggestions are welcome.
+
+ModelFusion Server is desigend for running multi-modal generative AI flows that take up to several minutes to complete. It provides the following benefits:
+
+- 🔄 Real-time progress updates via custom server-sent events
+- 🔒Type-safety with Zod-schema for inputs/events
+- 📦 Efficient handling of dynamically created binary assets (images, audio)
+- 📜 Auto-logging for AI model interactions within flows
+
+ModelFusion provides a [Fastify](https://fastify.dev/) plugin that allows you to set up a server that exposes your ModelFusion flows as REST endpoints using server-sent events.
+
+```ts
+import {
+  FileSystemAssetStorage,
+  FileSystemLogger,
+  modelFusionFastifyPlugin,
+} from "modelfusion/fastify-server"; // '/fastify-server' import path
+
+// configurable logging for all runs using ModelFusion observability:
+const logger = new FileSystemLogger({
+  path: (run) => path.join(fsBasePath, run.runId, "logs"),
+});
+
+// configurable storage for large files like images and audio files:
+const assetStorage = new FileSystemAssetStorage({
+  path: (run) => path.join(fsBasePath, run.runId, "assets"),
+  logger,
+});
+
+fastify.register(modelFusionFastifyPlugin, {
+  baseUrl,
+  basePath: "/myFlow",
+  logger,
+  assetStorage,
+  flow: exampleFlow,
+});
+```
+
+Using `invokeFlow`, you can easily connect your client to a ModelFusion flow endpoint:
+
+```ts
+import { invokeFlow } from "modelfusion/browser"; // '/browser' import path
+
+invokeFlow({
+  url: `${BASE_URL}/myFlow`,
+  schema: myFlowSchema,
+  input: { prompt },
+  onEvent(event) {
+    switch (event.type) {
+      case "my-event": {
+        // do something with the event
+        break;
+      }
+      // more events...
+    }
+  },
+  onStop() {
+    // flow finished
+  },
+});
+```
+
 ## Documentation
 
 ### [Guide](https://modelfusion.dev/guide)
@@ -563,6 +629,7 @@ setGlobalFunctionLogging("detailed-object"); // log full events
   - [Retrieve](https://modelfusion.dev/guide/vector-index/retrieve)
 - [Text Chunks](https://modelfusion.dev/guide/text-chunk/)
   - [Split Text](https://modelfusion.dev/guide/text-chunk/split)
+- [Server](https://modelfusion.dev/guide/server/)
 - [Utilities](https://modelfusion.dev/guide/util/)
   - [API Configuration](https://modelfusion.dev/guide/util/api-configuration)
     - [Retry strategies](https://modelfusion.dev/guide/util/api-configuration/retry)
@@ -623,9 +690,9 @@ Create an 19th century painting image for your input.
 
 Record audio with push-to-talk and transcribe it using Whisper, implemented as a Next.js app. The app shows a list of the transcriptions.
 
-### [Speech Streaming (Vite(React) + Fastify))](https://github.com/lgrammel/modelfusion/tree/main/examples/speech-streaming-vite-react-fastify)
+### [Duplex Speech Streaming (using Vite/React & ModelFusion Server/Fastify)](https://github.com/lgrammel/modelfusion/tree/main/examples/speech-streaming-vite-react-fastify)
 
-> _Speech Streaming_, _OpenAI_, _Elevenlabs_ _streaming_, _Vite_, _Fastify_
+> _Speech Streaming_, _OpenAI_, _Elevenlabs_ _streaming_, _Vite_, _Fastify_, _ModelFusion Server_
 
 Given a prompt, the server returns both a text and a speech stream response.
 

package/browser/convertAudioChunksToBase64.cjs

@@ -0,0 +1,8 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.convertAudioChunksToBase64 = void 0;
+const convertBlobToBase64_js_1 = require("./convertBlobToBase64.cjs");
+function convertAudioChunksToBase64({ audioChunks, mimeType, }) {
+    return (0, convertBlobToBase64_js_1.convertBlobToBase64)(new Blob(audioChunks, { type: mimeType }));
+}
+exports.convertAudioChunksToBase64 = convertAudioChunksToBase64;

package/browser/convertAudioChunksToBase64.d.ts

@@ -0,0 +1,4 @@
+export declare function convertAudioChunksToBase64({ audioChunks, mimeType, }: {
+    audioChunks: Blob[];
+    mimeType: string;
+}): Promise<string>;

package/browser/convertAudioChunksToBase64.js

@@ -0,0 +1,4 @@
+import { convertBlobToBase64 } from "./convertBlobToBase64.js";
+export function convertAudioChunksToBase64({ audioChunks, mimeType, }) {
+    return convertBlobToBase64(new Blob(audioChunks, { type: mimeType }));
+}

package/browser/convertBlobToBase64.cjs

@@ -0,0 +1,23 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.convertBlobToBase64 = void 0;
+async function convertBlobToBase64(blob) {
+    return new Promise((resolve, reject) => {
+        const reader = new FileReader();
+        reader.onloadend = () => {
+            if (reader.result) {
+                const base64String = btoa(new Uint8Array(reader.result).reduce((data, byte) => data + String.fromCharCode(byte), ""));
+                resolve(base64String);
+            }
+            else {
+                reject(new Error("Failed to read blob."));
+            }
+        };
+        reader.onerror = () => {
+            reader.abort();
+            reject(new DOMException("Problem parsing input blob."));
+        };
+        reader.readAsArrayBuffer(blob);
+    });
+}
+exports.convertBlobToBase64 = convertBlobToBase64;

package/browser/convertBlobToBase64.d.ts

@@ -0,0 +1 @@
+export declare function convertBlobToBase64(blob: Blob): Promise<string>;

package/browser/convertBlobToBase64.js

@@ -0,0 +1,19 @@
+export async function convertBlobToBase64(blob) {
+    return new Promise((resolve, reject) => {
+        const reader = new FileReader();
+        reader.onloadend = () => {
+            if (reader.result) {
+                const base64String = btoa(new Uint8Array(reader.result).reduce((data, byte) => data + String.fromCharCode(byte), ""));
+                resolve(base64String);
+            }
+            else {
+                reject(new Error("Failed to read blob."));
+            }
+        };
+        reader.onerror = () => {
+            reader.abort();
+            reject(new DOMException("Problem parsing input blob."));
+        };
+        reader.readAsArrayBuffer(blob);
+    });
+}

package/{event-source → browser}/index.cjs

@@ -14,6 +14,9 @@ var __exportStar = (this && this.__exportStar) || function(m, exports) {
     for (var p in m) if (p !== "default" && !Object.prototype.hasOwnProperty.call(exports, p)) __createBinding(exports, m, p);
 };
 Object.defineProperty(exports, "__esModule", { value: true });
-__exportStar(require("./createEventSourceStream.cjs"), exports);
+__exportStar(require("./MediaSourceAppender.cjs"), exports);
+__exportStar(require("./convertAudioChunksToBase64.cjs"), exports);
+__exportStar(require("./convertBlobToBase64.cjs"), exports);
+__exportStar(require("./invokeFlow.cjs"), exports);
 __exportStar(require("./readEventSource.cjs"), exports);
 __exportStar(require("./readEventSourceStream.cjs"), exports);

package/browser/index.d.ts

@@ -0,0 +1,6 @@
+export * from "./MediaSourceAppender.js";
+export * from "./convertAudioChunksToBase64.js";
+export * from "./convertBlobToBase64.js";
+export * from "./invokeFlow.js";
+export * from "./readEventSource.js";
+export * from "./readEventSourceStream.js";

package/browser/index.js

@@ -0,0 +1,6 @@
+export * from "./MediaSourceAppender.js";
+export * from "./convertAudioChunksToBase64.js";
+export * from "./convertBlobToBase64.js";
+export * from "./invokeFlow.js";
+export * from "./readEventSource.js";
+export * from "./readEventSourceStream.js";

package/browser/invokeFlow.cjs

@@ -0,0 +1,23 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.invokeFlow = void 0;
+const ZodSchema_1 = require("../core/structure/ZodSchema");
+const readEventSource_1 = require("./readEventSource");
+async function invokeFlow({ url, input, schema, onEvent, onStop, }) {
+    const response = await fetch(url, {
+        method: "POST",
+        headers: { "Content-Type": "application/json" },
+        body: JSON.stringify(input),
+    });
+    const eventSourceUrl = (await response.json()).url;
+    (0, readEventSource_1.readEventSource)({
+        url: eventSourceUrl,
+        schema: new ZodSchema_1.ZodSchema(schema.events),
+        isStopEvent(event) {
+            return event.data === "[DONE]";
+        },
+        onEvent,
+        onStop,
+    });
+}
+exports.invokeFlow = invokeFlow;

package/browser/invokeFlow.d.ts

@@ -0,0 +1,8 @@
+import { FlowSchema } from "../server/fastify/FlowSchema";
+export declare function invokeFlow<INPUT, EVENT>({ url, input, schema, onEvent, onStop, }: {
+    url: string;
+    input: INPUT;
+    schema: FlowSchema<INPUT, EVENT>;
+    onEvent: (event: EVENT, eventSource: EventSource) => void;
+    onStop?: (eventSource: EventSource) => void;
+}): Promise<void>;

package/browser/invokeFlow.js

@@ -0,0 +1,19 @@
+import { ZodSchema } from "../core/structure/ZodSchema";
+import { readEventSource } from "./readEventSource";
+export async function invokeFlow({ url, input, schema, onEvent, onStop, }) {
+    const response = await fetch(url, {
+        method: "POST",
+        headers: { "Content-Type": "application/json" },
+        body: JSON.stringify(input),
+    });
+    const eventSourceUrl = (await response.json()).url;
+    readEventSource({
+        url: eventSourceUrl,
+        schema: new ZodSchema(schema.events),
+        isStopEvent(event) {
+            return event.data === "[DONE]";
+        },
+        onEvent,
+        onStop,
+    });
+}

package/{event-source → browser}/readEventSource.cjs

@@ -2,11 +2,16 @@
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.readEventSource = void 0;
 const parseJSON_js_1 = require("../util/parseJSON.cjs");
-function readEventSource({ url, schema, onEvent, onError = console.error, }) {
+function readEventSource({ url, schema, onEvent, onError = console.error, onStop, isStopEvent, }) {
     const eventSource = new EventSource(url);
-    eventSource.onmessage = (e) => {
+    eventSource.onmessage = (event) => {
         try {
-            const parseResult = (0, parseJSON_js_1.safeParseJsonWithSchema)(e.data, schema);
+            if (isStopEvent?.(event)) {
+                eventSource.close();
+                onStop?.(eventSource);
+                return;
+            }
+            const parseResult = (0, parseJSON_js_1.safeParseJsonWithSchema)(event.data, schema);
             if (!parseResult.success) {
                 onError(parseResult.error, eventSource);
                 return;

package/{event-source → browser}/readEventSource.d.ts

@@ -1,7 +1,9 @@
 import { Schema } from "../core/structure/Schema.js";
-export declare function readEventSource<T>({ url, schema, onEvent, onError, }: {
+export declare function readEventSource<T>({ url, schema, onEvent, onError, onStop, isStopEvent, }: {
     url: string;
     schema: Schema<T>;
     onEvent: (event: T, eventSource: EventSource) => void;
     onError?: (error: unknown, eventSource: EventSource) => void;
+    onStop?: (eventSource: EventSource) => void;
+    isStopEvent?: (event: MessageEvent<unknown>) => boolean;
 }): void;

package/{event-source → browser}/readEventSource.js

@@ -1,9 +1,14 @@
 import { safeParseJsonWithSchema } from "../util/parseJSON.js";
-export function readEventSource({ url, schema, onEvent, onError = console.error, }) {
+export function readEventSource({ url, schema, onEvent, onError = console.error, onStop, isStopEvent, }) {
     const eventSource = new EventSource(url);
-    eventSource.onmessage = (e) => {
+    eventSource.onmessage = (event) => {
         try {
-            const parseResult = safeParseJsonWithSchema(e.data, schema);
+            if (isStopEvent?.(event)) {
+                eventSource.close();
+                onStop?.(eventSource);
+                return;
+            }
+            const parseResult = safeParseJsonWithSchema(event.data, schema);
             if (!parseResult.success) {
                 onError(parseResult.error, eventSource);
                 return;

package/{event-source → browser}/readEventSourceStream.cjs

@@ -3,7 +3,7 @@ Object.defineProperty(exports, "__esModule", { value: true });
 exports.readEventSourceStream = void 0;
 const parseJSON_js_1 = require("../util/parseJSON.cjs");
 const AsyncQueue_js_1 = require("../util/AsyncQueue.cjs");
-const parseEventSourceStream_js_1 = require("./parseEventSourceStream.cjs");
+const parseEventSourceStream_js_1 = require("../util/streaming/parseEventSourceStream.cjs");
 function readEventSourceStream({ stream, schema, errorHandler, }) {
     const queue = new AsyncQueue_js_1.AsyncQueue();
     // run async (no await on purpose):

package/{event-source → browser}/readEventSourceStream.js

@@ -1,6 +1,6 @@
 import { safeParseJsonWithSchema } from "../util/parseJSON.js";
 import { AsyncQueue } from "../util/AsyncQueue.js";
-import { parseEventSourceStream } from "./parseEventSourceStream.js";
+import { parseEventSourceStream } from "../util/streaming/parseEventSourceStream.js";
 export function readEventSourceStream({ stream, schema, errorHandler, }) {
     const queue = new AsyncQueue();
     // run async (no await on purpose):

package/index.cjs

@@ -17,7 +17,6 @@ Object.defineProperty(exports, "__esModule", { value: true });
 __exportStar(require("./composed-function/index.cjs"), exports);
 __exportStar(require("./core/index.cjs"), exports);
 __exportStar(require("./cost/index.cjs"), exports);
-__exportStar(require("./event-source/index.cjs"), exports);
 __exportStar(require("./guard/index.cjs"), exports);
 __exportStar(require("./model-function/index.cjs"), exports);
 __exportStar(require("./model-provider/index.cjs"), exports);
@@ -25,6 +24,5 @@ __exportStar(require("./observability/index.cjs"), exports);
 __exportStar(require("./retriever/index.cjs"), exports);
 __exportStar(require("./text-chunk/index.cjs"), exports);
 __exportStar(require("./tool/index.cjs"), exports);
-__exportStar(require("./ui/index.cjs"), exports);
 __exportStar(require("./util/index.cjs"), exports);
 __exportStar(require("./vector-index/index.cjs"), exports);

package/index.d.ts

@@ -1,7 +1,6 @@
 export * from "./composed-function/index.js";
 export * from "./core/index.js";
 export * from "./cost/index.js";
-export * from "./event-source/index.js";
 export * from "./guard/index.js";
 export * from "./model-function/index.js";
 export * from "./model-provider/index.js";
@@ -9,6 +8,5 @@ export * from "./observability/index.js";
 export * from "./retriever/index.js";
 export * from "./text-chunk/index.js";
 export * from "./tool/index.js";
-export * from "./ui/index.js";
 export * from "./util/index.js";
 export * from "./vector-index/index.js";

package/index.js

@@ -1,7 +1,6 @@
 export * from "./composed-function/index.js";
 export * from "./core/index.js";
 export * from "./cost/index.js";
-export * from "./event-source/index.js";
 export * from "./guard/index.js";
 export * from "./model-function/index.js";
 export * from "./model-provider/index.js";
@@ -9,6 +8,5 @@ export * from "./observability/index.js";
 export * from "./retriever/index.js";
 export * from "./text-chunk/index.js";
 export * from "./tool/index.js";
-export * from "./ui/index.js";
 export * from "./util/index.js";
 export * from "./vector-index/index.js";

package/model-provider/anthropic/AnthropicTextGenerationModel.cjs

@@ -4,7 +4,7 @@ exports.AnthropicTextGenerationResponseFormat = exports.AnthropicTextGenerationM
 const zod_1 = require("zod");
 const callWithRetryAndThrottle_js_1 = require("../../core/api/callWithRetryAndThrottle.cjs");
 const postToApi_js_1 = require("../../core/api/postToApi.cjs");
-const parseEventSourceStream_js_1 = require("../../event-source/parseEventSourceStream.cjs");
+const parseEventSourceStream_js_1 = require("../../util/streaming/parseEventSourceStream.cjs");
 const AbstractModel_js_1 = require("../../model-function/AbstractModel.cjs");
 const PromptFormatTextStreamingModel_js_1 = require("../../model-function/generate-text/PromptFormatTextStreamingModel.cjs");
 const AsyncQueue_js_1 = require("../../util/AsyncQueue.cjs");

package/model-provider/anthropic/AnthropicTextGenerationModel.js

@@ -1,7 +1,7 @@
 import { z } from "zod";
 import { callWithRetryAndThrottle } from "../../core/api/callWithRetryAndThrottle.js";
 import { createJsonResponseHandler, postJsonToApi, } from "../../core/api/postToApi.js";
-import { parseEventSourceStream } from "../../event-source/parseEventSourceStream.js";
+import { parseEventSourceStream } from "../../util/streaming/parseEventSourceStream.js";
 import { AbstractModel } from "../../model-function/AbstractModel.js";
 import { PromptFormatTextStreamingModel } from "../../model-function/generate-text/PromptFormatTextStreamingModel.js";
 import { AsyncQueue } from "../../util/AsyncQueue.js";

package/model-provider/cohere/CohereTextGenerationModel.cjs

@@ -4,12 +4,12 @@ exports.CohereTextGenerationResponseFormat = exports.CohereTextGenerationModel =
 const zod_1 = require("zod");
 const callWithRetryAndThrottle_js_1 = require("../../core/api/callWithRetryAndThrottle.cjs");
 const postToApi_js_1 = require("../../core/api/postToApi.cjs");
-const AsyncQueue_js_1 = require("../../util/AsyncQueue.cjs");
 const AbstractModel_js_1 = require("../../model-function/AbstractModel.cjs");
 const PromptFormatTextStreamingModel_js_1 = require("../../model-function/generate-text/PromptFormatTextStreamingModel.cjs");
 const TextPromptFormat_js_1 = require("../../model-function/generate-text/TextPromptFormat.cjs");
 const countTokens_js_1 = require("../../model-function/tokenize-text/countTokens.cjs");
-const parseJSON_js_1 = require("../../util/parseJSON.cjs");
+const AsyncQueue_js_1 = require("../../util/AsyncQueue.cjs");
+const parseJsonStream_js_1 = require("../../util/streaming/parseJsonStream.cjs");
 const CohereApiConfiguration_js_1 = require("./CohereApiConfiguration.cjs");
 const CohereError_js_1 = require("./CohereError.cjs");
 const CohereTokenizer_js_1 = require("./CohereTokenizer.cjs");
@@ -216,58 +216,39 @@ const cohereTextStreamingResponseSchema = zod_1.z.discriminatedUnion("is_finishe
 async function createCohereTextGenerationFullDeltaIterableQueue(stream) {
     const queue = new AsyncQueue_js_1.AsyncQueue();
     let accumulatedText = "";
-    function processLine(line) {
-        const event = (0, parseJSON_js_1.parseJsonWithZod)(line, cohereTextStreamingResponseSchema);
-        if (event.is_finished === true) {
-            queue.push({
-                type: "delta",
-                fullDelta: {
-                    content: accumulatedText,
-                    isComplete: true,
-                    delta: "",
-                },
-                valueDelta: "",
-            });
-        }
-        else {
-            accumulatedText += event.text;
-            queue.push({
-                type: "delta",
-                fullDelta: {
-                    content: accumulatedText,
-                    isComplete: false,
-                    delta: event.text,
-                },
-                valueDelta: event.text,
-            });
-        }
-    }
     // process the stream asynchonously (no 'await' on purpose):
-    (async () => {
-        try {
-            let unprocessedText = "";
-            const reader = new ReadableStreamDefaultReader(stream);
-            const utf8Decoder = new TextDecoder("utf-8");
-            // eslint-disable-next-line no-constant-condition
-            while (true) {
-                const { value: chunk, done } = await reader.read();
-                if (done) {
-                    break;
-                }
-                unprocessedText += utf8Decoder.decode(chunk, { stream: true });
-                const processableLines = unprocessedText.split(/\r\n|\n|\r/g);
-                unprocessedText = processableLines.pop() || "";
-                processableLines.forEach(processLine);
+    (0, parseJsonStream_js_1.parseJsonStream)({
+        stream,
+        schema: cohereTextStreamingResponseSchema,
+        process(event) {
+            if (event.is_finished === true) {
+                queue.push({
+                    type: "delta",
+                    fullDelta: {
+                        content: accumulatedText,
+                        isComplete: true,
+                        delta: "",
+                    },
+                    valueDelta: "",
+                });
             }
-            // processing remaining text:
-            if (unprocessedText) {
-                processLine(unprocessedText);
+            else {
+                accumulatedText += event.text;
+                queue.push({
+                    type: "delta",
+                    fullDelta: {
+                        content: accumulatedText,
+                        isComplete: false,
+                        delta: event.text,
+                    },
+                    valueDelta: event.text,
+                });
             }
-        }
-        finally {
+        },
+        onDone() {
             queue.close();
-        }
-    })();
+        },
+    });
     return queue;
 }
 exports.CohereTextGenerationResponseFormat = {

package/model-provider/cohere/CohereTextGenerationModel.js

@@ -1,12 +1,12 @@
 import { z } from "zod";
 import { callWithRetryAndThrottle } from "../../core/api/callWithRetryAndThrottle.js";
 import { createJsonResponseHandler, postJsonToApi, } from "../../core/api/postToApi.js";
-import { AsyncQueue } from "../../util/AsyncQueue.js";
 import { AbstractModel } from "../../model-function/AbstractModel.js";
 import { PromptFormatTextStreamingModel } from "../../model-function/generate-text/PromptFormatTextStreamingModel.js";
 import { mapChatPromptToTextFormat, mapInstructionPromptToTextFormat, } from "../../model-function/generate-text/TextPromptFormat.js";
 import { countTokens } from "../../model-function/tokenize-text/countTokens.js";
-import { parseJsonWithZod } from "../../util/parseJSON.js";
+import { AsyncQueue } from "../../util/AsyncQueue.js";
+import { parseJsonStream } from "../../util/streaming/parseJsonStream.js";
 import { CohereApiConfiguration } from "./CohereApiConfiguration.js";
 import { failedCohereCallResponseHandler } from "./CohereError.js";
 import { CohereTokenizer } from "./CohereTokenizer.js";
@@ -212,58 +212,39 @@ const cohereTextStreamingResponseSchema = z.discriminatedUnion("is_finished", [
 async function createCohereTextGenerationFullDeltaIterableQueue(stream) {
     const queue = new AsyncQueue();
     let accumulatedText = "";
-    function processLine(line) {
-        const event = parseJsonWithZod(line, cohereTextStreamingResponseSchema);
-        if (event.is_finished === true) {
-            queue.push({
-                type: "delta",
-                fullDelta: {
-                    content: accumulatedText,
-                    isComplete: true,
-                    delta: "",
-                },
-                valueDelta: "",
-            });
-        }
-        else {
-            accumulatedText += event.text;
-            queue.push({
-                type: "delta",
-                fullDelta: {
-                    content: accumulatedText,
-                    isComplete: false,
-                    delta: event.text,
-                },
-                valueDelta: event.text,
-            });
-        }
-    }
     // process the stream asynchonously (no 'await' on purpose):
-    (async () => {
-        try {
-            let unprocessedText = "";
-            const reader = new ReadableStreamDefaultReader(stream);
-            const utf8Decoder = new TextDecoder("utf-8");
-            // eslint-disable-next-line no-constant-condition
-            while (true) {
-                const { value: chunk, done } = await reader.read();
-                if (done) {
-                    break;
-                }
-                unprocessedText += utf8Decoder.decode(chunk, { stream: true });
-                const processableLines = unprocessedText.split(/\r\n|\n|\r/g);
-                unprocessedText = processableLines.pop() || "";
-                processableLines.forEach(processLine);
+    parseJsonStream({
+        stream,
+        schema: cohereTextStreamingResponseSchema,
+        process(event) {
+            if (event.is_finished === true) {
+                queue.push({
+                    type: "delta",
+                    fullDelta: {
+                        content: accumulatedText,
+                        isComplete: true,
+                        delta: "",
+                    },
+                    valueDelta: "",
+                });
             }
-            // processing remaining text:
-            if (unprocessedText) {
-                processLine(unprocessedText);
+            else {
+                accumulatedText += event.text;
+                queue.push({
+                    type: "delta",
+                    fullDelta: {
+                        content: accumulatedText,
+                        isComplete: false,
+                        delta: event.text,
+                    },
+                    valueDelta: event.text,
+                });
             }
-        }
-        finally {
+        },
+        onDone() {
             queue.close();
-        }
-    })();
+        },
+    });
     return queue;
 }
 export const CohereTextGenerationResponseFormat = {

package/model-provider/index.cjs

@@ -21,5 +21,6 @@ __exportStar(require("./elevenlabs/index.cjs"), exports);
 __exportStar(require("./huggingface/index.cjs"), exports);
 __exportStar(require("./llamacpp/index.cjs"), exports);
 __exportStar(require("./lmnt/index.cjs"), exports);
+__exportStar(require("./ollama/index.cjs"), exports);
 __exportStar(require("./openai/index.cjs"), exports);
 __exportStar(require("./stability/index.cjs"), exports);

package/model-provider/index.d.ts

@@ -5,5 +5,6 @@ export * from "./elevenlabs/index.js";
 export * from "./huggingface/index.js";
 export * from "./llamacpp/index.js";
 export * from "./lmnt/index.js";
+export * from "./ollama/index.js";
 export * from "./openai/index.js";
 export * from "./stability/index.js";

package/model-provider/index.js

@@ -5,5 +5,6 @@ export * from "./elevenlabs/index.js";
 export * from "./huggingface/index.js";
 export * from "./llamacpp/index.js";
 export * from "./lmnt/index.js";
+export * from "./ollama/index.js";
 export * from "./openai/index.js";
 export * from "./stability/index.js";