@mux/ai 0.30.0 → 0.32.0

package/README.md

@@ -3,7 +3,7 @@
 [![npm version](https://badge.fury.io/js/@mux%2Fai.svg)](https://www.npmjs.com/package/@mux/ai)
 [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
 
-Easy to use, purpose-driven, cost effective, configurable **_workflow functions_** in a TypeScript SDK for building AI-powered video and audio workflows on the server, powered by [Mux](https://www.mux.com), with support for popular AI/LLM providers (OpenAI, Anthropic, Google).
+Easy to use, purpose-driven, cost effective, configurable **_workflow functions_** in a TypeScript SDK for building AI-powered video and audio workflows on the server, powered by [Mux](https://www.mux.com), with support for popular AI/LLM providers (OpenAI, Anthropic, Google, Baseten, and any OpenAI-compatible endpoint).
 
 - **Examples:** [`getSummaryAndTags`](#video-summarization), [`getModerationScores`](#content-moderation), [`hasBurnedInCaptions`](#burned-in-caption-detection), [`generateChapters`](#chapter-generation), [`generateEmbeddings`](#search-with-embeddings), [`generateEngagementInsights`](#engagement-insights), [`translateCaptions`](#caption-translation), [`editCaptions`](#caption-editing), [`translateAudio`](#audio-dubbing)
 - Workflows automatically ship with `"use workflow"` [compatability with Workflow DevKit](#compatability-with-workflow-devkit)
@@ -25,7 +25,7 @@ Add your credentials to a `.env` file (we support [dotenv](https://www.npmjs.com
 ```bash
 MUX_TOKEN_ID=your_mux_token_id
 MUX_TOKEN_SECRET=your_mux_token_secret
-OPENAI_API_KEY=your_openai_api_key          # or ANTHROPIC_API_KEY, GOOGLE_GENERATIVE_AI_API_KEY
+OPENAI_API_KEY=your_openai_api_key          # or ANTHROPIC_API_KEY, GOOGLE_GENERATIVE_AI_API_KEY, BASETEN_API_KEY
 ```
 
 You only need credentials for the AI provider you're using. See the [Credentials guide](./docs/CREDENTIALS.md) for full setup details including signed playback, S3 storage, and all supported providers.
@@ -54,9 +54,9 @@ console.log(result.tags);         // ["typescript", "tutorial", "programming"]
 
 - **Pre-built workflows for media AI tasks.** Common multi-step operations (transcript access, frame analysis, LLM calls, and structured parsing) are available as high-level functions.
 - **Support for video and audio assets.** The same workflows work with video and [audio-only assets](./docs/AUDIO-ONLY.md), including summarization, moderation, chaptering, and more.
-- **Provider-flexible API.** Choose OpenAI, Anthropic, or Google through workflow options while keeping the same workflow interface.
+- **Provider-flexible API.** Choose OpenAI, Anthropic, Google, Baseten, or any OpenAI-compatible endpoint through workflow options while keeping the same workflow interface.
 - **Published evaluation coverage.** Workflows include [evals](./docs/EVALS.md) for quality, latency, and cost, with results [published publicly](https://evaluating-mux-ai.vercel.app/) on pushes to `main`.
-- **Sensible default models.** Defaults (`gpt-5.6-luna` at medium reasoning, `claude-sonnet-4-5`, `gemini-3-flash-preview`) are selected to balance output quality and runtime cost.
+- **Sensible default models.** Defaults (`gpt-5.6-luna` at medium reasoning, `claude-sonnet-4-5`, `gemini-3-flash-preview`) are selected to balance output quality and runtime cost. Baseten and OpenAI-compatible endpoints serve user-deployed models, so they use an explicit `model` or the `BASETEN_MODEL`/`OPENAI_COMPATIBLE_MODEL` environment variables.
 - **Typed end-to-end.** Workflow inputs, options, and outputs are fully typed in TypeScript.
 - **Operational defaults included.** Retry handling, error handling, signed playback support, and [Workflow DevKit](https://useworkflow.dev) compatibility are built in.
 - **Prompt customization support.** Use `promptOverrides` to adjust sections of workflow prompts for your domain or product requirements.
@@ -68,14 +68,14 @@ Workflows are high-level functions that handle complete media AI tasks end-to-en
 
 | Workflow | What it does | Providers | Audio-only |
 | --- | --- | --- | :---: |
-| [`getSummaryAndTags`](./docs/WORKFLOWS.md#video-summarization) | Generate titles, descriptions, and tags | OpenAI, Anthropic, Google | Yes |
+| [`getSummaryAndTags`](./docs/WORKFLOWS.md#video-summarization) | Generate titles, descriptions, and tags | OpenAI, Anthropic, Google, Baseten, OpenAI-compatible | Yes |
 | [`getModerationScores`](./docs/WORKFLOWS.md#content-moderation) | Detect inappropriate content | OpenAI, Hive, Google Vision | Yes |
-| [`hasBurnedInCaptions`](./docs/WORKFLOWS.md#burned-in-caption-detection) | Detect hardcoded subtitles in video frames | OpenAI, Anthropic, Google | — |
-| [`askQuestions`](./docs/WORKFLOWS.md#ask-questions) | Answer yes/no questions about asset content | OpenAI, Anthropic, Google | Yes |
-| [`generateChapters`](./docs/WORKFLOWS.md#chapter-generation) | Create chapter markers from transcripts | OpenAI, Anthropic, Google | Yes |
-| [`generateEmbeddings`](./docs/WORKFLOWS.md#embeddings) | Generate vector embeddings for semantic search | OpenAI, Google | Yes |
-| [`translateCaptions`](./docs/WORKFLOWS.md#caption-translation) | Translate captions into other languages | OpenAI, Anthropic, Google | Yes |
-| [`editCaptions`](./docs/WORKFLOWS.md#caption-editing) | Edit captions: censor profanity and/or apply static replacements | OpenAI, Anthropic, Google | Yes |
+| [`hasBurnedInCaptions`](./docs/WORKFLOWS.md#burned-in-caption-detection) | Detect hardcoded subtitles in video frames | OpenAI, Anthropic, Google, Baseten, OpenAI-compatible | — |
+| [`askQuestions`](./docs/WORKFLOWS.md#ask-questions) | Answer yes/no questions about asset content | OpenAI, Anthropic, Google, Baseten, OpenAI-compatible | Yes |
+| [`generateChapters`](./docs/WORKFLOWS.md#chapter-generation) | Create chapter markers from transcripts | OpenAI, Anthropic, Google, Baseten, OpenAI-compatible | Yes |
+| [`generateEmbeddings`](./docs/WORKFLOWS.md#embeddings) | Generate vector embeddings for semantic search | OpenAI, Google, Baseten, OpenAI-compatible | Yes |
+| [`translateCaptions`](./docs/WORKFLOWS.md#caption-translation) | Translate captions into other languages | OpenAI, Anthropic, Google, Baseten, OpenAI-compatible | Yes |
+| [`editCaptions`](./docs/WORKFLOWS.md#caption-editing) | Edit captions: censor profanity and/or apply static replacements | OpenAI, Anthropic, Google, Baseten, OpenAI-compatible | Yes |
 | [`translateAudio`](./docs/WORKFLOWS.md#audio-dubbing) | Create AI-dubbed audio tracks | ElevenLabs | Yes |
 
 See the [Workflows guide](./docs/WORKFLOWS.md) for detailed documentation, options, and examples for each workflow. See the [API Reference](./docs/API.md) for complete parameter and return type details.
@@ -145,6 +145,31 @@ for (const chunk of result.chunks) {
 }
 ```
 
+**Baseten provider:**
+
+```bash
+BASETEN_API_KEY=your_baseten_api_key
+BASETEN_MODEL=your-language-model-id
+BASETEN_URL=https://model-id.api.baseten.co/environments/production/sync/v1
+BASETEN_EMBEDDING_MODEL=your-embedding-model-id
+BASETEN_EMBEDDING_URL=https://model-id.api.baseten.co/environments/production/sync
+```
+
+Omit `BASETEN_URL` to use Baseten's shared [Model APIs](https://docs.baseten.co/development/model-apis/overview); for a dedicated deployment, set it to the deployment's `/sync/v1` URL. Embeddings always run against a dedicated deployment, so `BASETEN_EMBEDDING_URL` (its `/sync` or `/sync/v1` URL) is required.
+
+**OpenAI-compatible provider:**
+
+```bash
+OPENAI_COMPATIBLE_BASE_URL=https://my-endpoint.example.com/v1
+OPENAI_COMPATIBLE_API_KEY=optional-api-key
+OPENAI_COMPATIBLE_MODEL=your-language-model-id
+OPENAI_COMPATIBLE_EMBEDDING_MODEL=your-embedding-model-id
+```
+
+Use `provider: "openai-compatible"` with any endpoint that speaks the OpenAI API protocol — vLLM, Ollama, Together AI, Fireworks, OpenRouter, or a self-hosted gateway. `OPENAI_COMPATIBLE_API_KEY` is optional for endpoints that don't require authentication, and `OPENAI_COMPATIBLE_EMBEDDING_BASE_URL` can point embeddings at a separate endpoint (defaults to the shared base URL). Workflows request schema-constrained output (`response_format: json_schema`), so the endpoint must support structured outputs.
+
+For both providers, workflows that analyze storyboards or frames (such as `getSummaryAndTags`, `hasBurnedInCaptions`, and `askQuestions`) require the deployed model to be vision-capable. See [Vision-Capable Models](./docs/VISION-MODELS.md) for known-good models and how to verify one.
+
 ## Prompt Customization
 
 Every workflow prompt is built from a structured template of named sections. The `promptOverrides` option lets you swap out individual sections with your own instructions while keeping the battle-tested defaults for everything else — no need to rewrite entire prompts.
@@ -164,7 +189,7 @@ This works with `getSummaryAndTags`, `generateChapters`, and `hasBurnedInCaption
 
 ## Evaluations
 
-Choosing between OpenAI, Anthropic, and Google for a given workflow isn't guesswork. Every workflow in `@mux/ai` ships with eval coverage that benchmarks providers and models against three dimensions:
+Choosing between OpenAI, Anthropic, and Google for a given workflow isn't guesswork. Every workflow in `@mux/ai` ships with eval coverage that benchmarks those providers and models against three dimensions. Baseten and generic OpenAI-compatible endpoints serve user-deployed models, so they are supported at runtime but intentionally skipped by eval runs.
 
 - **Efficacy** — Does it produce accurate, high-quality results?
 - **Efficiency** — How fast is it and how many tokens does it consume?

package/dist/{index-D3cB2ltG.d.ts → index-D_nj_Nr4.d.ts}

@@ -2,7 +2,7 @@ import { z } from 'zod';
 import { createAnthropic } from '@ai-sdk/anthropic';
 import { createGoogleGenerativeAI } from '@ai-sdk/google';
 import { createOpenAI } from '@ai-sdk/openai';
-import { f as ScopedMuxAIOptions, I as ImageSubmissionMode, i as TokenUsage, M as MuxAIOptions, b as ChunkingStrategy, k as VideoEmbeddingsResult, H as Hotspot, v as Shot, j as ToneType, g as StorageAdapter } from './shots-Bh0GrvLx.js';
+import { f as ScopedMuxAIOptions, I as ImageSubmissionMode, i as TokenUsage, M as MuxAIOptions, b as ChunkingStrategy, k as VideoEmbeddingsResult, H as Hotspot, v as Shot, j as ToneType, g as StorageAdapter } from './shots-BMbViFbZ.js';
 
 /**
  * Language Code Conversion Utilities
@@ -232,21 +232,25 @@ interface SafetyReport {
     scrubbedFields: ScrubbedFieldReport[];
 }
 
-type SupportedProvider = "openai" | "anthropic" | "google";
-type SupportedEmbeddingProvider = "openai" | "google";
+type SupportedProvider = "openai" | "anthropic" | "google" | "baseten" | "openai-compatible";
+type SupportedEmbeddingProvider = "openai" | "google" | "baseten" | "openai-compatible";
 type OpenAIModelId = Parameters<ReturnType<typeof createOpenAI>["chat"]>[0];
 type AnthropicModelId = Parameters<ReturnType<typeof createAnthropic>["chat"]>[0];
 type GoogleModelId = Parameters<ReturnType<typeof createGoogleGenerativeAI>["chat"]>[0];
 type OpenAIEmbeddingModelId = Parameters<ReturnType<typeof createOpenAI>["embedding"]>[0];
 type GoogleEmbeddingModelId = Parameters<ReturnType<typeof createGoogleGenerativeAI>["textEmbeddingModel"]>[0];
 interface ModelIdByProvider {
-    openai: OpenAIModelId;
-    anthropic: AnthropicModelId;
-    google: GoogleModelId;
+    "openai": OpenAIModelId;
+    "anthropic": AnthropicModelId;
+    "google": GoogleModelId;
+    "baseten": string;
+    "openai-compatible": string;
 }
 interface EmbeddingModelIdByProvider {
-    openai: OpenAIEmbeddingModelId;
-    google: GoogleEmbeddingModelId;
+    "openai": OpenAIEmbeddingModelId;
+    "google": GoogleEmbeddingModelId;
+    "baseten": string;
+    "openai-compatible": string;
 }
 
 /**
@@ -395,16 +399,16 @@ declare const questionAnswerSchema: z.ZodObject<{
     reasoning: z.ZodString;
     skipped: z.ZodBoolean;
 }, "strip", z.ZodTypeAny, {
+    reasoning: string;
     question: string;
     answer: string | null;
     confidence: number;
-    reasoning: string;
     skipped: boolean;
 }, {
+    reasoning: string;
     question: string;
     answer: string | null;
     confidence: number;
-    reasoning: string;
     skipped: boolean;
 }>;
 type QuestionAnswerType = z.infer<typeof questionAnswerSchema>;
@@ -422,32 +426,32 @@ declare function createAskQuestionsSchema(answerSchema: z.ZodType<string>): z.Zo
     } & {
         answer: z.ZodType<string, z.ZodTypeDef, string>;
     }, "strip", z.ZodTypeAny, {
+        reasoning: string;
         question: string;
         answer: string;
         confidence: number;
-        reasoning: string;
         skipped: boolean;
     }, {
+        reasoning: string;
         question: string;
         answer: string;
         confidence: number;
-        reasoning: string;
         skipped: boolean;
     }>, "many">;
 }, "strip", z.ZodTypeAny, {
     answers: {
+        reasoning: string;
         question: string;
         answer: string;
         confidence: number;
-        reasoning: string;
         skipped: boolean;
     }[];
 }, {
     answers: {
+        reasoning: string;
         question: string;
         answer: string;
         confidence: number;
-        reasoning: string;
         skipped: boolean;
     }[];
 }>;
@@ -1255,6 +1259,10 @@ interface AudioTranslationResult {
     dubbingId: string;
     uploadedTrackId?: string;
     presignedUrl?: string;
+    /** Mux text track ID for the dubbed captions, present when `uploadCaptionsToMux` is true and the upload succeeded. */
+    captionsTrackId?: string;
+    /** Presigned URL for the dub's translated transcript (WebVTT) staged to S3. */
+    captionsPresignedUrl?: string;
     /** Workflow usage metadata (asset duration, thumbnails, etc.). */
     usage?: TokenUsage;
 }
@@ -1290,6 +1298,13 @@ interface AudioTranslationOptions extends MuxAIOptions {
      * required for track creation.
      */
     uploadToMux?: boolean;
+    /**
+     * When true the dub's translated transcript (the same translation that was
+     * voiced) is attached as a subtitles text track on the Mux asset. Implies
+     * `uploadToS3: true`. Defaults to false. The SDK does not check for existing
+     * text tracks — callers decide conflict semantics before enabling this.
+     */
+    uploadCaptionsToMux?: boolean;
     /** Optional storage adapter override for upload + presign operations. */
     storageAdapter?: StorageAdapter;
     /** Expiry duration in seconds for S3 presigned GET URLs. Defaults to 86400 (24 hours). */

package/dist/{index-CJtQiGj7.d.ts → index-SjpXUdXl.d.ts}

@@ -1,4 +1,4 @@
-import { l as WorkflowCredentialsInput, n as WorkflowScope, A as AssetTextTrack, d as MuxAsset, T as TextChunk, b as ChunkingStrategy, q as CompletedShotsResult, r as ErroredShotsResult, H as Hotspot, s as HotspotOptions, t as HotspotResponse, u as PendingShotsResult, v as Shot, w as ShotRequestOptions, x as ShotsResult, y as WaitForShotsOptions, z as getHotspotsForAsset, B as getHotspotsForPlaybackId, D as getHotspotsForVideo, F as getShotsForAsset, G as requestShotsForAsset, J as waitForShotsForAsset } from './shots-Bh0GrvLx.js';
+import { l as WorkflowCredentialsInput, n as WorkflowScope, A as AssetTextTrack, d as MuxAsset, T as TextChunk, b as ChunkingStrategy, q as CompletedShotsResult, r as ErroredShotsResult, H as Hotspot, s as HotspotOptions, t as HotspotResponse, u as PendingShotsResult, v as Shot, w as ShotRequestOptions, x as ShotsResult, y as WaitForShotsOptions, z as getHotspotsForAsset, B as getHotspotsForPlaybackId, D as getHotspotsForVideo, F as getShotsForAsset, G as requestShotsForAsset, J as waitForShotsForAsset } from './shots-BMbViFbZ.js';
 
 interface HeatmapOptions {
     /** Time window for results, e.g., '7:days' (default: '7:days') */

package/dist/index.d.ts

@@ -1,15 +1,15 @@
-export { I as ISO639_1, a as ISO639_3, L as LanguageCodePair, S as SupportedISO639_1, b as SupportedISO639_3, g as getLanguageCodePair, c as getLanguageName, i as isUndeterminedLanguageCode, d as isValidISO639_1, e as isValidISO639_3, t as toISO639_1, f as toISO639_3, h as workflows } from './index-D3cB2ltG.js';
-import { W as WorkflowCredentials, S as StoragePutObjectInput, a as StoragePresignGetObjectInput } from './shots-Bh0GrvLx.js';
-export { A as AssetTextTrack, C as ChunkEmbedding, b as ChunkingStrategy, E as Encrypted, c as EncryptedPayload, I as ImageSubmissionMode, M as MuxAIOptions, d as MuxAsset, P as PlaybackAsset, e as PlaybackPolicy, f as ScopedMuxAIOptions, g as StorageAdapter, T as TextChunk, h as TokenChunkingConfig, i as TokenUsage, j as ToneType, U as UsageMetadata, V as VTTChunkingConfig, k as VideoEmbeddingsResult, l as WorkflowCredentialsInput, m as WorkflowMuxClient, n as WorkflowScope, o as decryptFromWorkflow, p as encryptForWorkflow } from './shots-Bh0GrvLx.js';
+export { I as ISO639_1, a as ISO639_3, L as LanguageCodePair, S as SupportedISO639_1, b as SupportedISO639_3, g as getLanguageCodePair, c as getLanguageName, i as isUndeterminedLanguageCode, d as isValidISO639_1, e as isValidISO639_3, t as toISO639_1, f as toISO639_3, h as workflows } from './index-D_nj_Nr4.js';
+import { W as WorkflowCredentials, S as StoragePutObjectInput, a as StoragePresignGetObjectInput } from './shots-BMbViFbZ.js';
+export { A as AssetTextTrack, C as ChunkEmbedding, b as ChunkingStrategy, E as Encrypted, c as EncryptedPayload, I as ImageSubmissionMode, M as MuxAIOptions, d as MuxAsset, P as PlaybackAsset, e as PlaybackPolicy, f as ScopedMuxAIOptions, g as StorageAdapter, T as TextChunk, h as TokenChunkingConfig, i as TokenUsage, j as ToneType, U as UsageMetadata, V as VTTChunkingConfig, k as VideoEmbeddingsResult, l as WorkflowCredentialsInput, m as WorkflowMuxClient, n as WorkflowScope, o as decryptFromWorkflow, p as encryptForWorkflow } from './shots-BMbViFbZ.js';
 import { WORKFLOW_SERIALIZE, WORKFLOW_DESERIALIZE } from '@workflow/serde';
-export { i as primitives } from './index-CJtQiGj7.js';
+export { i as primitives } from './index-SjpXUdXl.js';
 import 'zod';
 import '@ai-sdk/anthropic';
 import '@ai-sdk/google';
 import '@ai-sdk/openai';
 import '@mux/mux-node';
 
-var version = "0.30.0";
+var version = "0.32.0";
 
 type MuxAiErrorType = "validation_error" | "processing_error" | "timeout_error" | "content_policy_error";
 /**