modelfusion 0.129.1 → 0.130.1

package/CHANGELOG.md

@@ -1,5 +1,30 @@
 # Changelog
 
+## v0.130.1 - 2024-01-22
+
+### Fixed
+
+- Duplex speech streaming works in Vercel Edge Functions.
+
+## v0.130.0 - 2024-01-21
+
+### Changed
+
+- **breaking change**: updated `generateTranscription` interface. The function now takes a `mimeType` and `audioData` (base64-encoded string, `Uint8Array`, `Buffer` or `ArrayBuffer`). Example:
+
+  ```ts
+  import { generateTranscription, openai } from "modelfusion";
+  import fs from "node:fs";
+
+  const transcription = await generateTranscription({
+    model: openai.Transcriber({ model: "whisper-1" }),
+    mimeType: "audio/mp3",
+    audioData: await fs.promises.readFile("data/test.mp3"),
+  });
+  ```
+
+- Images in instruction and chat prompts can be `Buffer` or `ArrayBuffer` instances (in addition to base64-encoded strings and `Uint8Array` instances).
+
 ## v0.129.0 - 2024-01-20
 
 ### Changed

package/README.md

@@ -17,7 +17,7 @@
 - **Vendor-neutral**: ModelFusion is a non-commercial open source project that is community-driven. You can use it with any supported provider.
 - **Multi-modal**: ModelFusion supports a wide range of models including text generation, image generation, vision, text-to-speech, speech-to-text, and embedding models.
 - **Type inference and validation**: ModelFusion infers TypeScript types wherever possible and validates model responses.
-- **Observability and logging**: ModelFusion provides an observer framework and out-of-the-box logging support.
+- **Observability and logging**: ModelFusion provides an observer framework and logging support.
 - **Resilience and robustness**: ModelFusion ensures seamless operation through automatic retries, throttling, and error handling mechanisms.
 - **Built for production**: ModelFusion is fully tree-shakeable, can be used in serverless environments, and only uses a minimal set of dependencies.
 
@@ -89,13 +89,16 @@ import { readFileSync } from "fs";
 const image = readFileSync("./image.png");
 
 const textStream = await streamText({
-  model: openai.ChatTextGenerator({ model: "gpt-4-vision-preview" }),
-  prompt: [
-    openai.ChatMessage.user([
-      { type: "text", text: "Describe the image in detail:" },
+  model: openai
+    .ChatTextGenerator({ model: "gpt-4-vision-preview" })
+    .withInstructionPrompt(),
+
+  prompt: {
+    instruction: [
+      { type: "text", text: "Describe the image in detail." },
       { type: "image", image, mimeType: "image/png" },
-    ]),
-  ],
+    ],
+  },
 });
 
 for await (const textPart of textStream) {
@@ -269,13 +272,12 @@ Transcribe speech (audio) data into text. Also called speech-to-text (STT).
 
 ```ts
 import { generateTranscription, openai } from "modelfusion";
+import fs from "node:fs";
 
 const transcription = await generateTranscription({
   model: openai.Transcriber({ model: "whisper-1" }),
-  data: {
-    type: "mp3",
-    data: await fs.promises.readFile("data/test.mp3"),
-  },
+  mimeType: "audio/mp3",
+  audioData: await fs.promises.readFile("data/test.mp3"),
 });
 ```
 
@@ -510,14 +512,16 @@ const textStream = await streamText({
 You an use prompt templates with image models as well, e.g. to use a basic text prompt. It is available as a shorthand method:
 
 ```ts
-const image = await generateImage(
-  stability
+const image = await generateImage({
+  model: stability
     .ImageGenerator({
       //...
     })
     .withTextPrompt(),
-  "the wicked witch of the west in the style of early 19th century painting"
-);
+
+  prompt:
+    "the wicked witch of the west in the style of early 19th century painting",
+});
 ```
 
 | Prompt Template | Text Prompt |
@@ -551,14 +555,18 @@ for (const choice of (rawResponse as OpenAICompletionResponse).choices) {
 
 ### Logging and Observability
 
-ModelFusion provides an [observer framework](https://modelfusion.dev/guide/util/observer) and [out-of-the-box logging support](https://modelfusion.dev/guide/util/logging). You can easily trace runs and call hierarchies, and you can add your own observers.
+ModelFusion provides an [observer framework](https://modelfusion.dev/guide/util/observer) and [logging support](https://modelfusion.dev/guide/util/logging). You can easily trace runs and call hierarchies, and you can add your own observers.
 
-#### Global Logging Example
+#### Enabling Logging on a Function Call
 
 ```ts
-import { modelfusion } from "modelfusion";
+import { generateText, openai } from "modelfusion";
 
-modelfusion.setLogFormat("detailed-object"); // log full events
+const text = await generateText({
+  model: openai.CompletionTextGenerator({ model: "gpt-3.5-turbo-instruct" }),
+  prompt: "Write a short story about a robot learning to love:\n\n",
+  logging: "detailed-object",
+});
 ```
 
 ## Documentation
@@ -637,12 +645,13 @@ Ask questions about a PDF document and get answers from the document.
 
 ### [Next.js / ModelFusion Demos](https://github.com/lgrammel/modelfusion/tree/main/examples/nextjs)
 
-> _Next.js app_, _image generation_, _transcription_, _OpenAI_, _Stability AI_
+> _Next.js app_, _image generation_, _transcription_, _structure streaming_, _OpenAI_, _Stability AI_, _Ollama_
 
 Examples of using ModelFusion with Next.js 14 (App Router):
 
 - image generation
 - voice recording & transcription
+- structure streaming (client-side)
 
 ### [Duplex Speech Streaming (using Vite/React & ModelFusion Server/Fastify)](https://github.com/lgrammel/modelfusion/tree/main/examples/speech-streaming-vite-react-fastify)
 

package/core/api/postToApi.cjs

@@ -1,7 +1,7 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.postToApi = exports.postJsonToApi = exports.createAudioMpegResponseHandler = exports.createTextResponseHandler = exports.createJsonResponseHandler = exports.createTextErrorResponseHandler = exports.createJsonErrorResponseHandler = void 0;
-const UInt8Utils_js_1 = require("../../util/UInt8Utils.cjs");
+const DataContent_js_1 = require("../../util/format/DataContent.cjs");
 const parseJSON_js_1 = require("../schema/parseJSON.cjs");
 const ApiCallError_js_1 = require("./ApiCallError.cjs");
 const createJsonErrorResponseHandler = ({ errorSchema, errorToMessage, isRetryable, }) => async ({ response, url, requestBodyValues }) => {
@@ -87,7 +87,7 @@ const createAudioMpegResponseHandler = () => async ({ response, url, requestBody
             requestBodyValues,
         });
     }
-    return (0, UInt8Utils_js_1.toUint8Array)(await response.arrayBuffer());
+    return (0, DataContent_js_1.convertDataContentToUint8Array)(await response.arrayBuffer());
 };
 exports.createAudioMpegResponseHandler = createAudioMpegResponseHandler;
 const postJsonToApi = async ({ url, headers, body, failedResponseHandler, successfulResponseHandler, abortSignal, }) => (0, exports.postToApi)({

package/core/api/postToApi.js

@@ -1,4 +1,4 @@
-import { toUint8Array } from "../../util/UInt8Utils.js";
+import { convertDataContentToUint8Array } from "../../util/format/DataContent.js";
 import { parseJSON, safeParseJSON } from "../schema/parseJSON.js";
 import { ApiCallError } from "./ApiCallError.js";
 export const createJsonErrorResponseHandler = ({ errorSchema, errorToMessage, isRetryable, }) => async ({ response, url, requestBodyValues }) => {
@@ -80,7 +80,7 @@ export const createAudioMpegResponseHandler = () => async ({ response, url, requ
             requestBodyValues,
         });
     }
-    return toUint8Array(await response.arrayBuffer());
+    return convertDataContentToUint8Array(await response.arrayBuffer());
 };
 export const postJsonToApi = async ({ url, headers, body, failedResponseHandler, successfulResponseHandler, abortSignal, }) => postToApi({
     url,

package/model-function/generate-image/generateImage.cjs

@@ -1,7 +1,7 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.generateImage = void 0;
-const UInt8Utils_js_1 = require("../../util/UInt8Utils.cjs");
+const UInt8Utils_js_1 = require("../../util/format/UInt8Utils.cjs");
 const executeStandardCall_js_1 = require("../executeStandardCall.cjs");
 async function generateImage({ model, prompt, fullResponse, ...options }) {
     const callResponse = await (0, executeStandardCall_js_1.executeStandardCall)({

package/model-function/generate-image/generateImage.js

@@ -1,4 +1,4 @@
-import { base64ToUint8Array } from "../../util/UInt8Utils.js";
+import { base64ToUint8Array } from "../../util/format/UInt8Utils.js";
 import { executeStandardCall } from "../executeStandardCall.js";
 export async function generateImage({ model, prompt, fullResponse, ...options }) {
     const callResponse = await executeStandardCall({

package/model-function/generate-text/prompt-template/ContentPart.cjs

@@ -1,15 +1,7 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.validateContentIsString = exports.getImageAsBase64 = void 0;
-const UInt8Utils_js_1 = require("../../../util/UInt8Utils.cjs");
+exports.validateContentIsString = void 0;
 const InvalidPromptError_js_1 = require("./InvalidPromptError.cjs");
-function getImageAsBase64(image) {
-    if (typeof image === "string") {
-        return image;
-    }
-    return (0, UInt8Utils_js_1.uint8ArrayToBase64)(image);
-}
-exports.getImageAsBase64 = getImageAsBase64;
 function validateContentIsString(content, prompt) {
     if (typeof content !== "string") {
         throw new InvalidPromptError_js_1.InvalidPromptError("Only text prompts are are supported by this prompt template.", prompt);

package/model-function/generate-text/prompt-template/ContentPart.d.ts

@@ -1,4 +1,4 @@
-/// <reference types="node" />
+import { DataContent } from "../../../util/format/DataContent.js";
 export interface TextPart {
     type: "text";
     /**
@@ -9,9 +9,9 @@ export interface TextPart {
 export interface ImagePart {
     type: "image";
     /**
-     * Image data. Can either be a base64-encoded string, a Uint8Array, or a Buffer.
+     * Image data. Can either be a base64-encoded string, a Uint8Array, an ArrayBuffer, or a Buffer.
      */
-    image: string | Uint8Array | Buffer;
+    image: DataContent;
     /**
      * Optional mime type of the image.
      */
@@ -28,5 +28,4 @@ export interface ToolResponsePart {
     id: string;
     response: unknown;
 }
-export declare function getImageAsBase64(image: string | Uint8Array | Buffer): string;
 export declare function validateContentIsString(content: string | unknown, prompt: unknown): string;

package/model-function/generate-text/prompt-template/ContentPart.js

@@ -1,11 +1,4 @@
-import { uint8ArrayToBase64 } from "../../../util/UInt8Utils.js";
 import { InvalidPromptError } from "./InvalidPromptError.js";
-export function getImageAsBase64(image) {
-    if (typeof image === "string") {
-        return image;
-    }
-    return uint8ArrayToBase64(image);
-}
 export function validateContentIsString(content, prompt) {
     if (typeof content !== "string") {
         throw new InvalidPromptError("Only text prompts are are supported by this prompt template.", prompt);

package/model-function/generate-transcription/TranscriptionModel.d.ts

@@ -1,9 +1,13 @@
+import { DataContent } from "../../util/format/DataContent.js";
 import { FunctionCallOptions } from "../../core/FunctionOptions.js";
 import { Model, ModelSettings } from "../Model.js";
 export interface TranscriptionModelSettings extends ModelSettings {
 }
-export interface TranscriptionModel<DATA, SETTINGS extends TranscriptionModelSettings = TranscriptionModelSettings> extends Model<SETTINGS> {
-    doTranscribe: (data: DATA, options: FunctionCallOptions) => PromiseLike<{
+export interface TranscriptionModel<SETTINGS extends TranscriptionModelSettings = TranscriptionModelSettings> extends Model<SETTINGS> {
+    doTranscribe: (input: {
+        mimeType: string;
+        audioData: DataContent;
+    }, options: FunctionCallOptions) => PromiseLike<{
         rawResponse: unknown;
         transcription: string;
     }>;

package/model-function/generate-transcription/generateTranscription.cjs

@@ -2,14 +2,15 @@
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.generateTranscription = void 0;
 const executeStandardCall_js_1 = require("../executeStandardCall.cjs");
-async function generateTranscription({ model, data, fullResponse, ...options }) {
+async function generateTranscription({ model, audioData, mimeType, fullResponse, ...options }) {
+    const input = { mimeType, audioData };
     const callResponse = await (0, executeStandardCall_js_1.executeStandardCall)({
         functionType: "generate-transcription",
-        input: data,
+        input,
         model,
         options,
         generateResponse: async (options) => {
-            const result = await model.doTranscribe(data, options);
+            const result = await model.doTranscribe(input, options);
             return {
                 rawResponse: result.rawResponse,
                 extractedValue: result.transcription,

package/model-function/generate-transcription/generateTranscription.d.ts

@@ -1,4 +1,6 @@
 import { FunctionOptions } from "../../core/FunctionOptions.js";
+import { AudioMimeType } from "../../util/audio/AudioMimeType.js";
+import { DataContent } from "../../util/format/DataContent.js";
 import { ModelCallMetadata } from "../ModelCallMetadata.js";
 import { TranscriptionModel, TranscriptionModelSettings } from "./TranscriptionModel.js";
 /**
@@ -7,26 +9,30 @@ import { TranscriptionModel, TranscriptionModelSettings } from "./TranscriptionM
  * @see https://modelfusion.dev/guide/function/generate-transcription
  *
  * @example
- * const data = await fs.promises.readFile("data/test.mp3");
+ * const audioData = await fs.promises.readFile("data/test.mp3");
  *
  * const transcription = await generateTranscription({
  *   model: openai.Transcriber({ model: "whisper-1" }),
- *   data: { type: "mp3", data }
+ *   mimeType: "audio/mp3",
+ *   audioData,
  * });
  *
- * @param {TranscriptionModel<DATA, TranscriptionModelSettings>} model - The model to use for transcription.
- * @param {DATA} data - The data to transcribe.
+ * @param {TranscriptionModel<DATA, TranscriptionModelSettings>} options.model - The model to use for transcription.
+ * @param {AudioMimeType} options.model - The MIME type of the audio data.
+ * @param {DataContent} options.model - The audio data to transcribe. Can be a base64-encoded string, a Uint8Array, or a Buffer.
  *
  * @returns {Promise<string>} A promise that resolves to the transcribed text.
  */
-export declare function generateTranscription<DATA>(args: {
-    model: TranscriptionModel<DATA, TranscriptionModelSettings>;
-    data: DATA;
+export declare function generateTranscription(args: {
+    model: TranscriptionModel<TranscriptionModelSettings>;
+    mimeType: AudioMimeType | (string & {});
+    audioData: DataContent;
     fullResponse?: false;
 } & FunctionOptions): Promise<string>;
-export declare function generateTranscription<DATA>(args: {
-    model: TranscriptionModel<DATA, TranscriptionModelSettings>;
-    data: DATA;
+export declare function generateTranscription(args: {
+    model: TranscriptionModel<TranscriptionModelSettings>;
+    mimeType: AudioMimeType | (string & {});
+    audioData: DataContent;
     fullResponse: true;
 } & FunctionOptions): Promise<{
     value: string;

package/model-function/generate-transcription/generateTranscription.js

@@ -1,12 +1,13 @@
 import { executeStandardCall } from "../executeStandardCall.js";
-export async function generateTranscription({ model, data, fullResponse, ...options }) {
+export async function generateTranscription({ model, audioData, mimeType, fullResponse, ...options }) {
+    const input = { mimeType, audioData };
     const callResponse = await executeStandardCall({
         functionType: "generate-transcription",
-        input: data,
+        input,
         model,
         options,
         generateResponse: async (options) => {
-            const result = await model.doTranscribe(data, options);
+            const result = await model.doTranscribe(input, options);
             return {
                 rawResponse: result.rawResponse,
                 extractedValue: result.transcription,

package/model-provider/elevenlabs/ElevenLabsSpeechModel.cjs

@@ -9,7 +9,7 @@ const parseJSON_js_1 = require("../../core/schema/parseJSON.cjs");
 const AbstractModel_js_1 = require("../../model-function/AbstractModel.cjs");
 const AsyncQueue_js_1 = require("../../util/AsyncQueue.cjs");
 const SimpleWebSocket_js_1 = require("../../util/SimpleWebSocket.cjs");
-const UInt8Utils_js_1 = require("../../util/UInt8Utils.cjs");
+const UInt8Utils_js_1 = require("../../util/format/UInt8Utils.cjs");
 const ElevenLabsApiConfiguration_js_1 = require("./ElevenLabsApiConfiguration.cjs");
 const elevenLabsModels = [
     "eleven_multilingual_v2",

package/model-provider/elevenlabs/ElevenLabsSpeechModel.js

@@ -6,7 +6,7 @@ import { safeParseJSON } from "../../core/schema/parseJSON.js";
 import { AbstractModel } from "../../model-function/AbstractModel.js";
 import { AsyncQueue } from "../../util/AsyncQueue.js";
 import { createSimpleWebSocket } from "../../util/SimpleWebSocket.js";
-import { base64ToUint8Array } from "../../util/UInt8Utils.js";
+import { base64ToUint8Array } from "../../util/format/UInt8Utils.js";
 import { ElevenLabsApiConfiguration } from "./ElevenLabsApiConfiguration.js";
 const elevenLabsModels = [
     "eleven_multilingual_v2",

package/model-provider/llamacpp/LlamaCppBakLLaVA1PromptTemplate.cjs

@@ -4,6 +4,7 @@ exports.chat = exports.instruction = exports.text = void 0;
 const ContentPart_js_1 = require("../../model-function/generate-text/prompt-template/ContentPart.cjs");
 const InvalidPromptError_js_1 = require("../../model-function/generate-text/prompt-template/InvalidPromptError.cjs");
 const TextPromptTemplate_js_1 = require("../../model-function/generate-text/prompt-template/TextPromptTemplate.cjs");
+const DataContent_js_1 = require("../../util/format/DataContent.cjs");
 // default Vicuna 1 system message
 const DEFAULT_SYSTEM_MESSAGE = "A chat between a curious user and an artificial intelligence assistant. " +
     "The assistant gives helpful, detailed, and polite answers to the user's questions.";
@@ -46,7 +47,8 @@ function instruction() {
                         }
                         case "image": {
                             text += `[img-${imageCounter}]`;
-                            images[imageCounter.toString()] = (0, ContentPart_js_1.getImageAsBase64)(content.image);
+                            images[imageCounter.toString()] =
+                                (0, DataContent_js_1.convertDataContentToBase64String)(content.image);
                             imageCounter++;
                             break;
                         }
@@ -85,7 +87,8 @@ function chat() {
                                 }
                                 case "image": {
                                     text += `[img-${imageCounter}]`;
-                                    images[imageCounter.toString()] = (0, ContentPart_js_1.getImageAsBase64)(part.image);
+                                    images[imageCounter.toString()] =
+                                        (0, DataContent_js_1.convertDataContentToBase64String)(part.image);
                                     imageCounter++;
                                     break;
                                 }

package/model-provider/llamacpp/LlamaCppBakLLaVA1PromptTemplate.js

@@ -1,6 +1,7 @@
-import { getImageAsBase64, validateContentIsString, } from "../../model-function/generate-text/prompt-template/ContentPart.js";
+import { validateContentIsString } from "../../model-function/generate-text/prompt-template/ContentPart.js";
 import { InvalidPromptError } from "../../model-function/generate-text/prompt-template/InvalidPromptError.js";
 import { text as vicunaText } from "../../model-function/generate-text/prompt-template/TextPromptTemplate.js";
+import { convertDataContentToBase64String } from "../../util/format/DataContent.js";
 // default Vicuna 1 system message
 const DEFAULT_SYSTEM_MESSAGE = "A chat between a curious user and an artificial intelligence assistant. " +
     "The assistant gives helpful, detailed, and polite answers to the user's questions.";
@@ -42,7 +43,8 @@ export function instruction() {
                         }
                         case "image": {
                             text += `[img-${imageCounter}]`;
-                            images[imageCounter.toString()] = getImageAsBase64(content.image);
+                            images[imageCounter.toString()] =
+                                convertDataContentToBase64String(content.image);
                             imageCounter++;
                             break;
                         }
@@ -80,7 +82,8 @@ export function chat() {
                                 }
                                 case "image": {
                                     text += `[img-${imageCounter}]`;
-                                    images[imageCounter.toString()] = getImageAsBase64(part.image);
+                                    images[imageCounter.toString()] =
+                                        convertDataContentToBase64String(part.image);
                                     imageCounter++;
                                     break;
                                 }

package/model-provider/lmnt/LmntSpeechModel.cjs

@@ -6,7 +6,7 @@ const callWithRetryAndThrottle_js_1 = require("../../core/api/callWithRetryAndTh
 const postToApi_js_1 = require("../../core/api/postToApi.cjs");
 const ZodSchema_js_1 = require("../../core/schema/ZodSchema.cjs");
 const AbstractModel_js_1 = require("../../model-function/AbstractModel.cjs");
-const UInt8Utils_js_1 = require("../../util/UInt8Utils.cjs");
+const UInt8Utils_js_1 = require("../../util/format/UInt8Utils.cjs");
 const LmntApiConfiguration_js_1 = require("./LmntApiConfiguration.cjs");
 /**
  * Synthesize speech using the LMNT API.

package/model-provider/lmnt/LmntSpeechModel.js

@@ -3,7 +3,7 @@ import { callWithRetryAndThrottle } from "../../core/api/callWithRetryAndThrottl
 import { createJsonResponseHandler, createTextErrorResponseHandler, postToApi, } from "../../core/api/postToApi.js";
 import { zodSchema } from "../../core/schema/ZodSchema.js";
 import { AbstractModel } from "../../model-function/AbstractModel.js";
-import { base64ToUint8Array } from "../../util/UInt8Utils.js";
+import { base64ToUint8Array } from "../../util/format/UInt8Utils.js";
 import { LmntApiConfiguration } from "./LmntApiConfiguration.js";
 /**
  * Synthesize speech using the LMNT API.

package/model-provider/ollama/OllamaChatPromptTemplate.cjs

@@ -3,6 +3,7 @@ Object.defineProperty(exports, "__esModule", { value: true });
 exports.chat = exports.instruction = exports.text = exports.identity = void 0;
 const ContentPart_js_1 = require("../../model-function/generate-text/prompt-template/ContentPart.cjs");
 const InvalidPromptError_js_1 = require("../../model-function/generate-text/prompt-template/InvalidPromptError.cjs");
+const DataContent_js_1 = require("../../util/format/DataContent.cjs");
 /**
  * OllamaChatPrompt identity chat format.
  */
@@ -95,7 +96,7 @@ function extractUserContent(input) {
             content += part.text;
         }
         else {
-            images.push((0, ContentPart_js_1.getImageAsBase64)(part.image));
+            images.push((0, DataContent_js_1.convertDataContentToBase64String)(part.image));
         }
     }
     return { content, images };

package/model-provider/ollama/OllamaChatPromptTemplate.js

@@ -1,5 +1,6 @@
-import { getImageAsBase64, validateContentIsString, } from "../../model-function/generate-text/prompt-template/ContentPart.js";
+import { validateContentIsString } from "../../model-function/generate-text/prompt-template/ContentPart.js";
 import { InvalidPromptError } from "../../model-function/generate-text/prompt-template/InvalidPromptError.js";
+import { convertDataContentToBase64String } from "../../util/format/DataContent.js";
 /**
  * OllamaChatPrompt identity chat format.
  */
@@ -88,7 +89,7 @@ function extractUserContent(input) {
             content += part.text;
         }
         else {
-            images.push(getImageAsBase64(part.image));
+            images.push(convertDataContentToBase64String(part.image));
         }
     }
     return { content, images };

package/model-provider/openai/OpenAIChatMessage.cjs

@@ -1,7 +1,7 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.OpenAIChatMessage = void 0;
-const ContentPart_js_1 = require("../../model-function/generate-text/prompt-template/ContentPart.cjs");
+const DataContent_js_1 = require("../../util/format/DataContent.cjs");
 exports.OpenAIChatMessage = {
     /**
      * Creates a system chat message.
@@ -25,7 +25,7 @@ exports.OpenAIChatMessage = {
                         case "image": {
                             return {
                                 type: "image_url",
-                                image_url: `data:${part.mimeType ?? "image/jpeg"};base64,${(0, ContentPart_js_1.getImageAsBase64)(part.image)}`,
+                                image_url: `data:${part.mimeType ?? "image/jpeg"};base64,${(0, DataContent_js_1.convertDataContentToBase64String)(part.image)}`,
                             };
                         }
                     }

package/model-provider/openai/OpenAIChatMessage.js

@@ -1,4 +1,4 @@
-import { getImageAsBase64, } from "../../model-function/generate-text/prompt-template/ContentPart.js";
+import { convertDataContentToBase64String } from "../../util/format/DataContent.js";
 export const OpenAIChatMessage = {
     /**
      * Creates a system chat message.
@@ -22,7 +22,7 @@ export const OpenAIChatMessage = {
                         case "image": {
                             return {
                                 type: "image_url",
-                                image_url: `data:${part.mimeType ?? "image/jpeg"};base64,${getImageAsBase64(part.image)}`,
+                                image_url: `data:${part.mimeType ?? "image/jpeg"};base64,${convertDataContentToBase64String(part.image)}`,
                             };
                         }
                     }

package/model-provider/openai/OpenAITranscriptionModel.cjs

@@ -6,6 +6,8 @@ const callWithRetryAndThrottle_js_1 = require("../../core/api/callWithRetryAndTh
 const postToApi_js_1 = require("../../core/api/postToApi.cjs");
 const ZodSchema_js_1 = require("../../core/schema/ZodSchema.cjs");
 const AbstractModel_js_1 = require("../../model-function/AbstractModel.cjs");
+const getAudioFileExtension_js_1 = require("../../util/audio/getAudioFileExtension.cjs");
+const DataContent_js_1 = require("../../util/format/DataContent.cjs");
 const OpenAIApiConfiguration_js_1 = require("./OpenAIApiConfiguration.cjs");
 const OpenAIError_js_1 = require("./OpenAIError.cjs");
 /**
@@ -54,25 +56,26 @@ class OpenAITranscriptionModel extends AbstractModel_js_1.AbstractModel {
     get modelName() {
         return this.settings.model;
     }
-    async doTranscribe(data, options) {
-        const rawResponse = await this.callAPI(data, options, {
-            responseFormat: exports.OpenAITranscriptionResponseFormat.verboseJson,
-        });
+    async doTranscribe({ audioData, mimeType, }, options) {
+        const rawResponse = await this.callAPI({
+            fileExtension: (0, getAudioFileExtension_js_1.getAudioFileExtension)(mimeType),
+            audioData: (0, DataContent_js_1.convertDataContentToUint8Array)(audioData),
+        }, options, { responseFormat: exports.OpenAITranscriptionResponseFormat.verboseJson });
         return {
             rawResponse,
             transcription: rawResponse.text,
         };
     }
-    async callAPI(data, callOptions, options) {
+    async callAPI(input, callOptions, options) {
         const api = this.settings.api ?? new OpenAIApiConfiguration_js_1.OpenAIApiConfiguration();
         const abortSignal = callOptions?.run?.abortSignal;
         return (0, callWithRetryAndThrottle_js_1.callWithRetryAndThrottle)({
             retry: api.retry,
             throttle: api.throttle,
             call: async () => {
-                const fileName = `audio.${data.type}`;
+                const fileName = `audio.${input.fileExtension}`;
                 const formData = new FormData();
-                formData.append("file", new Blob([data.data]), fileName);
+                formData.append("file", new Blob([input.audioData]), fileName);
                 formData.append("model", this.settings.model);
                 if (this.settings.prompt != null) {
                     formData.append("prompt", this.settings.prompt);

package/model-provider/openai/OpenAITranscriptionModel.d.ts

@@ -4,6 +4,7 @@ import { ApiConfiguration } from "../../core/api/ApiConfiguration.js";
 import { ResponseHandler } from "../../core/api/postToApi.js";
 import { AbstractModel } from "../../model-function/AbstractModel.js";
 import { TranscriptionModel, TranscriptionModelSettings } from "../../model-function/generate-transcription/TranscriptionModel.js";
+import { DataContent } from "../../util/format/DataContent.js";
 /**
  * @see https://openai.com/pricing
  */
@@ -40,10 +41,6 @@ export interface OpenAITranscriptionModelSettings extends TranscriptionModelSett
      */
     prompt?: string;
 }
-export type OpenAITranscriptionInput = {
-    type: "flac" | "m4a" | "mp3" | "mp4" | "mpeg" | "mpga" | "ogg" | "wav" | "webm";
-    data: Uint8Array;
-};
 /**
  * Create a transcription model that calls the OpenAI transcription API.
  *
@@ -60,11 +57,14 @@ export type OpenAITranscriptionInput = {
  *   }
  * );
  */
-export declare class OpenAITranscriptionModel extends AbstractModel<OpenAITranscriptionModelSettings> implements TranscriptionModel<OpenAITranscriptionInput, OpenAITranscriptionModelSettings> {
+export declare class OpenAITranscriptionModel extends AbstractModel<OpenAITranscriptionModelSettings> implements TranscriptionModel<OpenAITranscriptionModelSettings> {
     constructor(settings: OpenAITranscriptionModelSettings);
     readonly provider: "openai";
     get modelName(): "whisper-1";
-    doTranscribe(data: OpenAITranscriptionInput, options: FunctionCallOptions): Promise<{
+    doTranscribe({ audioData, mimeType, }: {
+        audioData: DataContent;
+        mimeType: string;
+    }, options: FunctionCallOptions): Promise<{
         rawResponse: {
             text: string;
             duration: number;
@@ -86,7 +86,10 @@ export declare class OpenAITranscriptionModel extends AbstractModel<OpenAITransc
         };
         transcription: string;
     }>;
-    callAPI<RESULT>(data: OpenAITranscriptionInput, callOptions: FunctionCallOptions, options: {
+    callAPI<RESULT>(input: {
+        fileExtension: string;
+        audioData: Uint8Array;
+    }, callOptions: FunctionCallOptions, options: {
         responseFormat: OpenAITranscriptionResponseFormatType<RESULT>;
     }): Promise<RESULT>;
     get settingsForEvent(): Partial<OpenAITranscriptionModelSettings>;

package/model-provider/openai/OpenAITranscriptionModel.js

@@ -3,6 +3,8 @@ import { callWithRetryAndThrottle } from "../../core/api/callWithRetryAndThrottl
 import { createJsonResponseHandler, createTextResponseHandler, postToApi, } from "../../core/api/postToApi.js";
 import { zodSchema } from "../../core/schema/ZodSchema.js";
 import { AbstractModel } from "../../model-function/AbstractModel.js";
+import { getAudioFileExtension } from "../../util/audio/getAudioFileExtension.js";
+import { convertDataContentToUint8Array, } from "../../util/format/DataContent.js";
 import { OpenAIApiConfiguration } from "./OpenAIApiConfiguration.js";
 import { failedOpenAICallResponseHandler } from "./OpenAIError.js";
 /**
@@ -50,25 +52,26 @@ export class OpenAITranscriptionModel extends AbstractModel {
     get modelName() {
         return this.settings.model;
     }
-    async doTranscribe(data, options) {
-        const rawResponse = await this.callAPI(data, options, {
-            responseFormat: OpenAITranscriptionResponseFormat.verboseJson,
-        });
+    async doTranscribe({ audioData, mimeType, }, options) {
+        const rawResponse = await this.callAPI({
+            fileExtension: getAudioFileExtension(mimeType),
+            audioData: convertDataContentToUint8Array(audioData),
+        }, options, { responseFormat: OpenAITranscriptionResponseFormat.verboseJson });
         return {
             rawResponse,
             transcription: rawResponse.text,
         };
     }
-    async callAPI(data, callOptions, options) {
+    async callAPI(input, callOptions, options) {
         const api = this.settings.api ?? new OpenAIApiConfiguration();
         const abortSignal = callOptions?.run?.abortSignal;
         return callWithRetryAndThrottle({
             retry: api.retry,
             throttle: api.throttle,
             call: async () => {
-                const fileName = `audio.${data.type}`;
+                const fileName = `audio.${input.fileExtension}`;
                 const formData = new FormData();
-                formData.append("file", new Blob([data.data]), fileName);
+                formData.append("file", new Blob([input.audioData]), fileName);
                 formData.append("model", this.settings.model);
                 if (this.settings.prompt != null) {
                     formData.append("prompt", this.settings.prompt);

package/model-provider/whispercpp/WhisperCppTranscriptionModel.cjs

@@ -8,6 +8,8 @@ const postToApi_js_1 = require("../../core/api/postToApi.cjs");
 const ZodSchema_js_1 = require("../../core/schema/ZodSchema.cjs");
 const parseJSON_js_1 = require("../../core/schema/parseJSON.cjs");
 const AbstractModel_js_1 = require("../../model-function/AbstractModel.cjs");
+const getAudioFileExtension_js_1 = require("../../util/audio/getAudioFileExtension.cjs");
+const DataContent_js_1 = require("../../util/format/DataContent.cjs");
 const WhisperCppApiConfiguration_js_1 = require("./WhisperCppApiConfiguration.cjs");
 class WhisperCppTranscriptionModel extends AbstractModel_js_1.AbstractModel {
     constructor(settings) {
@@ -25,14 +27,17 @@ class WhisperCppTranscriptionModel extends AbstractModel_js_1.AbstractModel {
             value: null
         });
     }
-    async doTranscribe(data, options) {
-        const rawResponse = await this.callAPI(data, options);
+    async doTranscribe({ audioData, mimeType, }, options) {
+        const rawResponse = await this.callAPI({
+            fileExtension: (0, getAudioFileExtension_js_1.getAudioFileExtension)(mimeType),
+            audioData: (0, DataContent_js_1.convertDataContentToUint8Array)(audioData),
+        }, options);
         return {
             rawResponse,
             transcription: rawResponse.text,
         };
     }
-    async callAPI(data, callOptions) {
+    async callAPI(input, callOptions) {
         const { temperature } = this.settings;
         const api = this.settings.api ?? new WhisperCppApiConfiguration_js_1.WhisperCppApiConfiguration();
         const abortSignal = callOptions.run?.abortSignal;
@@ -41,7 +46,7 @@ class WhisperCppTranscriptionModel extends AbstractModel_js_1.AbstractModel {
             throttle: api.throttle,
             call: async () => {
                 const formData = new FormData();
-                formData.append("file", new Blob([data.data]), `audio.${data.type}`);
+                formData.append("file", new Blob([input.audioData]), `audio.${input.fileExtension}`);
                 formData.append("response_format", "json");
                 if (temperature != null) {
                     formData.append("temperature", temperature.toString());

package/model-provider/whispercpp/WhisperCppTranscriptionModel.d.ts

@@ -2,25 +2,28 @@ import { FunctionCallOptions } from "../../core/FunctionOptions.js";
 import { ApiConfiguration } from "../../core/api/ApiConfiguration.js";
 import { AbstractModel } from "../../model-function/AbstractModel.js";
 import { TranscriptionModel, TranscriptionModelSettings } from "../../model-function/generate-transcription/TranscriptionModel.js";
+import { DataContent } from "../../util/format/DataContent.js";
 export interface WhisperCppTranscriptionModelSettings extends TranscriptionModelSettings {
     api?: ApiConfiguration;
     temperature?: number;
 }
-export type WhisperCppTranscriptionInput = {
-    type: "wav";
-    data: Uint8Array;
-};
-export declare class WhisperCppTranscriptionModel extends AbstractModel<WhisperCppTranscriptionModelSettings> implements TranscriptionModel<WhisperCppTranscriptionInput, WhisperCppTranscriptionModelSettings> {
+export declare class WhisperCppTranscriptionModel extends AbstractModel<WhisperCppTranscriptionModelSettings> implements TranscriptionModel<WhisperCppTranscriptionModelSettings> {
     constructor(settings: WhisperCppTranscriptionModelSettings);
     readonly provider: "whispercpp";
     readonly modelName: null;
-    doTranscribe(data: WhisperCppTranscriptionInput, options: FunctionCallOptions): Promise<{
+    doTranscribe({ audioData, mimeType, }: {
+        audioData: DataContent;
+        mimeType: string;
+    }, options: FunctionCallOptions): Promise<{
         rawResponse: {
             text: string;
         };
         transcription: string;
     }>;
-    callAPI(data: WhisperCppTranscriptionInput, callOptions: FunctionCallOptions): Promise<{
+    callAPI(input: {
+        fileExtension: string;
+        audioData: Uint8Array;
+    }, callOptions: FunctionCallOptions): Promise<{
         text: string;
     }>;
     get settingsForEvent(): Partial<WhisperCppTranscriptionModelSettings>;

package/model-provider/whispercpp/WhisperCppTranscriptionModel.js

@@ -5,6 +5,8 @@ import { postToApi } from "../../core/api/postToApi.js";
 import { zodSchema } from "../../core/schema/ZodSchema.js";
 import { safeParseJSON } from "../../core/schema/parseJSON.js";
 import { AbstractModel } from "../../model-function/AbstractModel.js";
+import { getAudioFileExtension } from "../../util/audio/getAudioFileExtension.js";
+import { convertDataContentToUint8Array, } from "../../util/format/DataContent.js";
 import { WhisperCppApiConfiguration } from "./WhisperCppApiConfiguration.js";
 export class WhisperCppTranscriptionModel extends AbstractModel {
     constructor(settings) {
@@ -22,14 +24,17 @@ export class WhisperCppTranscriptionModel extends AbstractModel {
             value: null
         });
     }
-    async doTranscribe(data, options) {
-        const rawResponse = await this.callAPI(data, options);
+    async doTranscribe({ audioData, mimeType, }, options) {
+        const rawResponse = await this.callAPI({
+            fileExtension: getAudioFileExtension(mimeType),
+            audioData: convertDataContentToUint8Array(audioData),
+        }, options);
         return {
             rawResponse,
             transcription: rawResponse.text,
         };
     }
-    async callAPI(data, callOptions) {
+    async callAPI(input, callOptions) {
         const { temperature } = this.settings;
         const api = this.settings.api ?? new WhisperCppApiConfiguration();
         const abortSignal = callOptions.run?.abortSignal;
@@ -38,7 +43,7 @@ export class WhisperCppTranscriptionModel extends AbstractModel {
             throttle: api.throttle,
             call: async () => {
                 const formData = new FormData();
-                formData.append("file", new Blob([data.data]), `audio.${data.type}`);
+                formData.append("file", new Blob([input.audioData]), `audio.${input.fileExtension}`);
                 formData.append("response_format", "json");
                 if (temperature != null) {
                     formData.append("temperature", temperature.toString());

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "modelfusion",
   "description": "The TypeScript library for building AI applications.",
-  "version": "0.129.1",
+  "version": "0.130.1",
   "author": "Lars Grammel",
   "license": "MIT",
   "keywords": [

package/util/SimpleWebSocket.cjs

@@ -24,18 +24,25 @@ var __importStar = (this && this.__importStar) || function (mod) {
 };
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.createSimpleWebSocket = void 0;
+const Runtime = __importStar(require("./detectRuntime.cjs"));
 /**
  * Creates a simplified websocket connection. This function works in both Node.js and browser.
  */
 async function createSimpleWebSocket(url) {
-    if (typeof window === "undefined") {
-        // Use ws library in Node.js:
-        const { default: WebSocket } = await Promise.resolve().then(() => __importStar(require("ws")));
-        return new WebSocket(url);
-    }
-    else {
-        // Use native WebSocket in browser:
-        return new WebSocket(url);
+    switch (Runtime.detectRuntime()) {
+        case "vercel-edge":
+        case "cloudflare-workers":
+        case "browser": {
+            return new WebSocket(url);
+        }
+        case "node": {
+            // Use ws library (for Node.js):
+            const { default: WebSocket } = await Promise.resolve().then(() => __importStar(require("ws")));
+            return new WebSocket(url);
+        }
+        default: {
+            throw new Error("Unknown runtime");
+        }
     }
 }
 exports.createSimpleWebSocket = createSimpleWebSocket;

package/util/SimpleWebSocket.js

@@ -1,14 +1,21 @@
+import * as Runtime from "./detectRuntime.js";
 /**
  * Creates a simplified websocket connection. This function works in both Node.js and browser.
  */
 export async function createSimpleWebSocket(url) {
-    if (typeof window === "undefined") {
-        // Use ws library in Node.js:
-        const { default: WebSocket } = await import("ws");
-        return new WebSocket(url);
-    }
-    else {
-        // Use native WebSocket in browser:
-        return new WebSocket(url);
+    switch (Runtime.detectRuntime()) {
+        case "vercel-edge":
+        case "cloudflare-workers":
+        case "browser": {
+            return new WebSocket(url);
+        }
+        case "node": {
+            // Use ws library (for Node.js):
+            const { default: WebSocket } = await import("ws");
+            return new WebSocket(url);
+        }
+        default: {
+            throw new Error("Unknown runtime");
+        }
     }
 }

package/util/audio/AudioMimeType.cjs

@@ -0,0 +1,2 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });

package/util/audio/AudioMimeType.d.ts

@@ -0,0 +1 @@
+export type AudioMimeType = "audio/webm" | "audio/mp3" | "audio/wav" | "audio/mp4" | "audio/mpeg" | "audio/mpga" | "audio/ogg" | "audio/oga" | "audio/flac" | "audio/m4a";

package/util/audio/AudioMimeType.js

@@ -0,0 +1 @@
+export {};

package/util/audio/getAudioFileExtension.cjs

@@ -0,0 +1,29 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.getAudioFileExtension = void 0;
+function getAudioFileExtension(mimeType) {
+    const normalizedMimeType = mimeType.split(";")[0].toLowerCase();
+    switch (normalizedMimeType) {
+        case "audio/webm":
+            return "webm";
+        case "audio/mp3":
+            return "mp3";
+        case "audio/wav":
+            return "wav";
+        case "audio/mp4":
+            return "mp4";
+        case "audio/mpeg":
+        case "audio/mpga":
+            return "mpeg";
+        case "audio/ogg":
+        case "audio/oga":
+            return "ogg";
+        case "audio/flac":
+            return "flac";
+        case "audio/m4a":
+            return "m4a";
+        default:
+            throw new Error(`Unsupported audio format: ${mimeType}`);
+    }
+}
+exports.getAudioFileExtension = getAudioFileExtension;

package/util/audio/getAudioFileExtension.d.ts

@@ -0,0 +1 @@
+export declare function getAudioFileExtension(mimeType: string): "mp3" | "flac" | "webm" | "wav" | "mp4" | "mpeg" | "ogg" | "m4a";

package/util/audio/getAudioFileExtension.js

@@ -0,0 +1,25 @@
+export function getAudioFileExtension(mimeType) {
+    const normalizedMimeType = mimeType.split(";")[0].toLowerCase();
+    switch (normalizedMimeType) {
+        case "audio/webm":
+            return "webm";
+        case "audio/mp3":
+            return "mp3";
+        case "audio/wav":
+            return "wav";
+        case "audio/mp4":
+            return "mp4";
+        case "audio/mpeg":
+        case "audio/mpga":
+            return "mpeg";
+        case "audio/ogg":
+        case "audio/oga":
+            return "ogg";
+        case "audio/flac":
+            return "flac";
+        case "audio/m4a":
+            return "m4a";
+        default:
+            throw new Error(`Unsupported audio format: ${mimeType}`);
+    }
+}

package/util/audio/index.cjs

@@ -0,0 +1,18 @@
+"use strict";
+var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    var desc = Object.getOwnPropertyDescriptor(m, k);
+    if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
+      desc = { enumerable: true, get: function() { return m[k]; } };
+    }
+    Object.defineProperty(o, k2, desc);
+}) : (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    o[k2] = m[k];
+}));
+var __exportStar = (this && this.__exportStar) || function(m, exports) {
+    for (var p in m) if (p !== "default" && !Object.prototype.hasOwnProperty.call(exports, p)) __createBinding(exports, m, p);
+};
+Object.defineProperty(exports, "__esModule", { value: true });
+__exportStar(require("./AudioMimeType.cjs"), exports);
+__exportStar(require("./getAudioFileExtension.cjs"), exports);

package/util/audio/index.d.ts

@@ -0,0 +1,2 @@
+export * from "./AudioMimeType.js";
+export * from "./getAudioFileExtension.js";

package/util/audio/index.js

@@ -0,0 +1,2 @@
+export * from "./AudioMimeType.js";
+export * from "./getAudioFileExtension.js";

package/util/detectRuntime.cjs

@@ -0,0 +1,21 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.detectRuntime = void 0;
+function detectRuntime() {
+    // eslint-disable-next-line @typescript-eslint/no-explicit-any
+    const globalThisAny = globalThis;
+    if (globalThisAny.EdgeRuntime) {
+        return "vercel-edge";
+    }
+    if (globalThis.navigator?.userAgent === "Cloudflare-Workers") {
+        return "cloudflare-workers";
+    }
+    if (globalThis.process?.release?.name === "node") {
+        return "node";
+    }
+    if (globalThis.window) {
+        return "browser";
+    }
+    return null;
+}
+exports.detectRuntime = detectRuntime;

package/util/detectRuntime.d.ts

@@ -0,0 +1 @@
+export declare function detectRuntime(): "vercel-edge" | "cloudflare-workers" | "node" | "browser" | null;

package/util/detectRuntime.js

@@ -0,0 +1,17 @@
+export function detectRuntime() {
+    // eslint-disable-next-line @typescript-eslint/no-explicit-any
+    const globalThisAny = globalThis;
+    if (globalThisAny.EdgeRuntime) {
+        return "vercel-edge";
+    }
+    if (globalThis.navigator?.userAgent === "Cloudflare-Workers") {
+        return "cloudflare-workers";
+    }
+    if (globalThis.process?.release?.name === "node") {
+        return "node";
+    }
+    if (globalThis.window) {
+        return "browser";
+    }
+    return null;
+}

package/util/format/DataContent.cjs

@@ -0,0 +1,27 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.convertDataContentToUint8Array = exports.convertDataContentToBase64String = void 0;
+const UInt8Utils_js_1 = require("./UInt8Utils.cjs");
+function convertDataContentToBase64String(content) {
+    if (typeof content === "string") {
+        return content;
+    }
+    if (content instanceof ArrayBuffer) {
+        return (0, UInt8Utils_js_1.uint8ArrayToBase64)(new Uint8Array(content));
+    }
+    return (0, UInt8Utils_js_1.uint8ArrayToBase64)(content);
+}
+exports.convertDataContentToBase64String = convertDataContentToBase64String;
+function convertDataContentToUint8Array(content) {
+    if (content instanceof Uint8Array) {
+        return content;
+    }
+    if (typeof content === "string") {
+        return (0, UInt8Utils_js_1.base64ToUint8Array)(content);
+    }
+    if (content instanceof ArrayBuffer) {
+        return new Uint8Array(content);
+    }
+    throw new Error(`Invalid data content. Expected a string, Uint8Array, ArrayBuffer, or Buffer, but got ${typeof content}.`);
+}
+exports.convertDataContentToUint8Array = convertDataContentToUint8Array;

package/util/format/DataContent.d.ts

@@ -0,0 +1,7 @@
+/// <reference types="node" />
+/**
+ * Data content. Can either be a base64-encoded string, a Uint8Array, an ArrayBuffer, or a Buffer.
+ */
+export type DataContent = string | Uint8Array | ArrayBuffer | Buffer;
+export declare function convertDataContentToBase64String(content: DataContent): string;
+export declare function convertDataContentToUint8Array(content: DataContent): Uint8Array;

package/util/format/DataContent.js

@@ -0,0 +1,22 @@
+import { base64ToUint8Array, uint8ArrayToBase64 } from "./UInt8Utils.js";
+export function convertDataContentToBase64String(content) {
+    if (typeof content === "string") {
+        return content;
+    }
+    if (content instanceof ArrayBuffer) {
+        return uint8ArrayToBase64(new Uint8Array(content));
+    }
+    return uint8ArrayToBase64(content);
+}
+export function convertDataContentToUint8Array(content) {
+    if (content instanceof Uint8Array) {
+        return content;
+    }
+    if (typeof content === "string") {
+        return base64ToUint8Array(content);
+    }
+    if (content instanceof ArrayBuffer) {
+        return new Uint8Array(content);
+    }
+    throw new Error(`Invalid data content. Expected a string, Uint8Array, ArrayBuffer, or Buffer, but got ${typeof content}.`);
+}

package/util/{UInt8Utils.cjs → format/UInt8Utils.cjs}

@@ -11,18 +11,8 @@
  * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
  */
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.uint8ArrayToBase64 = exports.base64ToUint8Array = exports.toUint8Array = void 0;
+exports.uint8ArrayToBase64 = exports.base64ToUint8Array = void 0;
 // copied from: https://github.com/sindresorhus/uint8array-extras/blob/main/index.js
-function toUint8Array(value) {
-    if (value instanceof ArrayBuffer) {
-        return new Uint8Array(value);
-    }
-    if (ArrayBuffer.isView(value)) {
-        return new Uint8Array(value.buffer, value.byteOffset, value.byteLength);
-    }
-    throw new TypeError(`Unsupported value, got \`${typeof value}\`.`);
-}
-exports.toUint8Array = toUint8Array;
 function base64ToUint8Array(base64String) {
     return Uint8Array.from(globalThis.atob(base64UrlToBase64(base64String)), (x) => x.codePointAt(0));
 }

package/util/{UInt8Utils.d.ts → format/UInt8Utils.d.ts}

@@ -1,3 +1,2 @@
-export declare function toUint8Array(value: unknown): Uint8Array;
 export declare function base64ToUint8Array(base64String: string): Uint8Array;
 export declare function uint8ArrayToBase64(array: Uint8Array): string;

package/util/{UInt8Utils.js → format/UInt8Utils.js}

@@ -10,15 +10,6 @@
  * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
  */
 // copied from: https://github.com/sindresorhus/uint8array-extras/blob/main/index.js
-export function toUint8Array(value) {
-    if (value instanceof ArrayBuffer) {
-        return new Uint8Array(value);
-    }
-    if (ArrayBuffer.isView(value)) {
-        return new Uint8Array(value.buffer, value.byteOffset, value.byteLength);
-    }
-    throw new TypeError(`Unsupported value, got \`${typeof value}\`.`);
-}
 export function base64ToUint8Array(base64String) {
     return Uint8Array.from(globalThis.atob(base64UrlToBase64(base64String)), (x) => x.codePointAt(0));
 }

package/util/format/index.cjs

@@ -0,0 +1,17 @@
+"use strict";
+var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    var desc = Object.getOwnPropertyDescriptor(m, k);
+    if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
+      desc = { enumerable: true, get: function() { return m[k]; } };
+    }
+    Object.defineProperty(o, k2, desc);
+}) : (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    o[k2] = m[k];
+}));
+var __exportStar = (this && this.__exportStar) || function(m, exports) {
+    for (var p in m) if (p !== "default" && !Object.prototype.hasOwnProperty.call(exports, p)) __createBinding(exports, m, p);
+};
+Object.defineProperty(exports, "__esModule", { value: true });
+__exportStar(require("./DataContent.cjs"), exports);

package/util/format/index.d.ts

@@ -0,0 +1 @@
+export * from "./DataContent.js";

package/util/format/index.js

@@ -0,0 +1 @@
+export * from "./DataContent.js";

package/util/index.cjs

@@ -15,6 +15,8 @@ var __exportStar = (this && this.__exportStar) || function(m, exports) {
 };
 Object.defineProperty(exports, "__esModule", { value: true });
 __exportStar(require("./AsyncQueue.cjs"), exports);
+__exportStar(require("./audio/index.cjs"), exports);
 __exportStar(require("./cosineSimilarity.cjs"), exports);
 __exportStar(require("./delay.cjs"), exports);
+__exportStar(require("./format/index.cjs"), exports);
 __exportStar(require("./streaming/index.cjs"), exports);

package/util/index.d.ts

@@ -1,4 +1,6 @@
 export * from "./AsyncQueue.js";
+export * from "./audio/index.js";
 export * from "./cosineSimilarity.js";
 export * from "./delay.js";
+export * from "./format/index.js";
 export * from "./streaming/index.js";

package/util/index.js

@@ -1,4 +1,6 @@
 export * from "./AsyncQueue.js";
+export * from "./audio/index.js";
 export * from "./cosineSimilarity.js";
 export * from "./delay.js";
+export * from "./format/index.js";
 export * from "./streaming/index.js";