@ai-sdk/openai 4.0.66 → 4.0.68

package/docs/03-openai.mdx

@@ -39,7 +39,8 @@ You can use the following optional settings to customize the OpenAI provider ins
 - **baseURL** _string_
 
   Use a different URL prefix for API calls, e.g. to use proxy servers.
-  The default prefix is `https://api.openai.com/v1`.
+  It defaults to the `OPENAI_BASE_URL` environment variable, and then to
+  `https://api.openai.com/v1`.
   The default OpenAI model factory (`openai('model-id')`) uses the Responses
   API. If your custom base URL only supports the Chat Completions API, create
   chat models with `openai.chat('model-id')` instead, or use the
@@ -2850,17 +2851,38 @@ The following optional provider options are available for OpenAI completion mode
 
 <Note type="warning">Realtime is an experimental feature.</Note>
 
-You can create models that call the [OpenAI Realtime API](https://platform.openai.com/docs/guides/realtime)
-using the `.experimental_realtime()` factory method.
+Use `.experimental_realtime()` for both [OpenAI Realtime](https://platform.openai.com/docs/guides/realtime)
+and [OpenAI Live](https://developers.openai.com/api/docs/guides/live) models.
 
 ```ts
 import { openai } from '@ai-sdk/openai';
 
 const model = openai.experimental_realtime('gpt-realtime');
+const liveModel = openai.experimental_realtime('gpt-live-1');
+
+// Select the API explicitly for an early-access or otherwise unknown model ID.
+const earlyAccessModel = openai.experimental_realtime('not-yet-released', {
+  api: 'live',
+});
 ```
 
-Realtime sessions run in the browser and require a short-lived token created on
-your server with `openai.experimental_realtime.getToken()`:
+The OpenAI-only `api` option accepts `'live'` or `'realtime'` and takes precedence
+over model-ID selection. Without it, the known `gpt-live-1` ID uses Live and other
+IDs retain the Realtime default. The provider does not switch APIs after a failed
+request. Both model implementations use the existing experimental realtime interface.
+
+Live supports **browser WSS sessions through an application-owned relay with client
+delegation**, using `experimental_useRealtime` for the browser runtime, and optional
+browser-direct WebRTC. Your application owns the agent and tool execution; Live
+delegations do not invoke `onToolCall`. For a low-level server adapter, obtain server-only connection
+settings with `liveModel.getServerWebSocketConfig()` and serialize commands with
+`liveModel.serializeClientEvent()`. Create one parser per connection with
+`liveModel.createServerEventParser()` and pass incoming JSON to the returned
+function. It returns arrays of normalized events and retains each original event
+in `raw`.
+
+For a token-based **Realtime API** session, create a short-lived token on your
+server with `openai.experimental_realtime.getToken()`:
 
 ```ts
 const token = await openai.experimental_realtime.getToken({
@@ -2868,8 +2890,81 @@ const token = await openai.experimental_realtime.getToken({
 });
 ```
 
-See [Realtime](/docs/ai-sdk-core/realtime) for the complete setup and tool
-calling pattern.
+Live does not use Realtime client-secret minting. Calling `getToken` for a known
+Live model or with `api: 'live'` rejects before a mint request. Live's WebSocket
+connection authenticates on a trusted server; keep the provider API key out of
+browser code. The API selector chooses a protocol, not a different authentication
+capability for the same model.
+
+Browser authentication to an application-owned relay is separate from OpenAI
+authentication. Protect that relay with your application's session or short-lived
+credential; an origin check alone is not user authentication. The relay uses its
+server-held OpenAI key upstream. Selecting a Live model does not make the Realtime
+client-secret endpoint compatible with the Live API.
+
+For browser-direct Live WebRTC, an authenticated application endpoint performs the
+SDP exchange through `liveModel.doCreateWebRTCSession({ sdp, sessionConfig, abortSignal })`
+using the server's project key. The browser receives the
+session connection answer, not the project key or a Realtime client secret. This
+setup endpoint still needs application authentication. See the
+[`experimental_useRealtime` reference](/docs/reference/ai-sdk-ui/use-realtime)
+for the `api.session` setup contract and the runnable example.
+
+### Live session options
+
+Configure Live-specific settings through `sessionConfig.providerOptions.openai`,
+typed with `Experimental_OpenAIRealtimeModelLiveOptions`. For example:
+
+```ts
+import type { Experimental_OpenAIRealtimeModelLiveOptions as OpenAIRealtimeModelLiveOptions } from '@ai-sdk/openai';
+
+const sessionConfig = {
+  instructions: 'Be concise. Delegate questions that need research or tools.',
+  providerOptions: {
+    openai: {
+      delegation: { type: 'client' },
+    } satisfies OpenAIRealtimeModelLiveOptions,
+  },
+};
+```
+
+- `delegation`: use `{ type: 'client' }` for application-owned work. Omission or
+  `null` also selects client mode. Other delegation modes are unsupported.
+- `input`: prior text messages with developer, user or assistant roles.
+- `voice`: an authorized custom voice reference `{ id }`, mutually exclusive with
+  the ordinary `sessionConfig.voice` string.
+- `store`: requests provider-side session storage; availability depends on the
+  OpenAI project configuration.
+- `client.dataChannel`: WebRTC startup permission policy with
+  `allowedClientEvents` and `allowedServerEvents`. Set it on the application server,
+  not from trusted-looking browser input. Omission retains the provider default;
+  `[]` allows none and `'all'` allows all. Response event selectors use
+  `{ type: 'response.event', responseEvent: 'response.completed' }` to select
+  native nested events, which remain raw `custom` events rather than managed SDK
+  responses. Keep readiness, errors, and terminal usage observable for
+  the session features your application enables.
+
+Send the initial WebSocket configuration with `session-start`; WebRTC starts the
+session during SDP setup and negotiates audio formats through SDP. Live startup settings are
+immutable: `session-update` is rejected. Use `input-audio-mute` and
+`input-audio-unmute` for remote input processing; these commands do not release the
+local microphone. Return application context through
+`context-append`, using the opaque `delegationId` from `delegation-created`, or
+`null` for context without a delegation. Its `providerOptions.openai.channel`
+accepts `instructions`, `thinking` (the default), or `commentary`.
+
+For graceful shutdown, install a `session-closed` listener before sending `session-close`;
+that event confirms final voice usage. `session-usage` contains cumulative seconds,
+so replace earlier snapshots rather than adding them.
+
+For the browser integration, see the
+[`experimental_useRealtime` reference](/docs/reference/ai-sdk-ui/use-realtime#continuous-conversations).
+It covers the JSON/PCM16 WebSocket relay runtime, optional WebRTC setup and permission
+policy, microphone and playback ownership, `session.delegations`, `sendEvent` for
+application context, and `close()` for bounded finalization. Use `wss:` for production
+relays. Continuous conversation semantics do not imply browser support for every
+provider codec or transport. The [Realtime guide](/docs/ai-sdk-core/realtime) covers
+legacy token-based, turn-based sessions.
 
 ## Embedding Models
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ai-sdk/openai",
-  "version": "4.0.66",
+  "version": "4.0.68",
   "type": "module",
   "license": "Apache-2.0",
   "sideEffects": false,
@@ -35,8 +35,8 @@
     }
   },
   "dependencies": {
-    "@ai-sdk/provider": "4.0.14",
-    "@ai-sdk/provider-utils": "5.0.40"
+    "@ai-sdk/provider": "4.0.16",
+    "@ai-sdk/provider-utils": "5.0.42"
   },
   "devDependencies": {
     "@ai-sdk/test-server": "2.0.1",

package/src/index.ts

@@ -1,5 +1,15 @@
 export { createOpenAI, openai } from './openai-provider';
 export type { OpenAIProvider, OpenAIProviderSettings } from './openai-provider';
+export type {
+  OpenAIRealtimeFactory as Experimental_OpenAIRealtimeFactory,
+  OpenAIRealtimeOptions as Experimental_OpenAIRealtimeOptions,
+} from './realtime/openai-realtime-factory';
+export { OpenAIRealtimeModelLive as Experimental_OpenAIRealtimeModelLive } from './live/openai-realtime-model-live';
+export type { OpenAIRealtimeModelLiveConfig as Experimental_OpenAIRealtimeModelLiveConfig } from './live/openai-realtime-model-live';
+export type {
+  OpenAIRealtimeModelLiveId as Experimental_OpenAIRealtimeModelLiveId,
+  OpenAIRealtimeModelLiveOptions as Experimental_OpenAIRealtimeModelLiveOptions,
+} from './live/openai-realtime-model-live-options';
 export { OpenAIRealtimeModel as Experimental_OpenAIRealtimeModel } from './realtime/openai-realtime-model';
 export type { OpenAIRealtimeModelConfig as Experimental_OpenAIRealtimeModelConfig } from './realtime/openai-realtime-model';
 export type {

package/src/live/openai-live-event-mapper.ts

@@ -0,0 +1,286 @@
+import {
+  UnsupportedFunctionalityError,
+  type Experimental_RealtimeModelV4ClientEvent as RealtimeModelV4ClientEvent,
+  type Experimental_RealtimeModelV4ServerEvent as RealtimeModelV4ServerEvent,
+} from '@ai-sdk/provider';
+import { z } from 'zod/v4';
+import { buildOpenAILiveSessionConfig } from './openai-live-session-config';
+
+const sessionSchema = z.object({ id: z.string().min(1) });
+const startedSessionSchema = sessionSchema.extend({
+  delegation: z.object({ type: z.enum(['client', 'responses']) }).nullish(),
+});
+const usageSchema = z.object({ seconds: z.number().nonnegative() });
+const transcriptFields = {
+  delta: z.string(),
+  start_ms: z.number().nonnegative(),
+  end_ms: z.number().nonnegative(),
+};
+const acknowledgmentFields = { client_event_id: z.string().nullish() };
+const appendAcknowledgmentFields = {
+  ...acknowledgmentFields,
+  start_ms: z.number().nonnegative(),
+  end_ms: z.number().nonnegative(),
+};
+const serverEventSchema = z.discriminatedUnion('type', [
+  z.object({
+    type: z.literal('session.started'),
+    session: startedSessionSchema,
+  }),
+  z.object({
+    type: z.literal('session.closed'),
+    session: sessionSchema.nullish(),
+    usage: usageSchema,
+    reason: z.string(),
+  }),
+  z.object({
+    type: z.literal('session.usage.updated'),
+    usage: usageSchema,
+    context_window: z
+      .object({ usage_ratio: z.number().min(0).max(1) })
+      .nullish(),
+  }),
+  z.object({
+    type: z.literal('session.output_audio.delta'),
+    delta: z.string(),
+  }),
+  z.object({
+    type: z.literal('session.input_transcript.delta'),
+    ...transcriptFields,
+  }),
+  z.object({
+    type: z.literal('session.output_transcript.delta'),
+    ...transcriptFields,
+  }),
+  z.object({
+    type: z.literal('session.delegation.created'),
+    offset_ms: z.number().nonnegative().nullish(),
+    delegation: z.object({
+      id: z.string().min(1),
+      target: z.enum(['client', 'responses']).nullish(),
+      response_id: z.string().min(1).nullish(),
+    }),
+  }),
+  z.object({
+    type: z.literal('session.updated'),
+    session: sessionSchema,
+    ...acknowledgmentFields,
+  }),
+  z.object({
+    type: z.literal('session.input_audio.muted'),
+    ...acknowledgmentFields,
+  }),
+  z.object({
+    type: z.literal('session.input_audio.unmuted'),
+    ...acknowledgmentFields,
+  }),
+  z.object({
+    type: z.literal('session.instructions.appended'),
+    ...appendAcknowledgmentFields,
+  }),
+  z.object({
+    type: z.literal('session.thinking.appended'),
+    ...appendAcknowledgmentFields,
+  }),
+  z.object({
+    type: z.literal('session.commentary.appended'),
+    ...appendAcknowledgmentFields,
+  }),
+  z.object({
+    type: z.literal('error'),
+    error: z.object({
+      message: z.string(),
+      code: z.string().nullish(),
+      client_event_id: z.string().nullish(),
+    }),
+  }),
+]);
+const knownTypes = new Set<string>(
+  serverEventSchema.options.map(schema => schema.shape.type.value),
+);
+const envelopeSchema = z.object({ type: z.string() });
+
+export function createOpenAILiveServerEventParser(): (
+  raw: unknown,
+) => RealtimeModelV4ServerEvent[] {
+  return raw => parseOpenAILiveServerEvent(raw);
+}
+
+export function parseOpenAILiveServerEvent(
+  raw: unknown,
+): RealtimeModelV4ServerEvent[] {
+  return [parseServerEvent(raw)];
+}
+
+function parseServerEvent(raw: unknown): RealtimeModelV4ServerEvent {
+  const envelope = envelopeSchema.safeParse(raw);
+  if (envelope.success && !knownTypes.has(envelope.data.type)) {
+    return { type: 'custom', rawType: envelope.data.type, raw };
+  }
+  const parsed = serverEventSchema.safeParse(raw);
+  if (!parsed.success) {
+    return {
+      type: 'error',
+      code: 'invalid_server_event',
+      message: 'Invalid OpenAI Live server event.',
+      raw,
+    };
+  }
+  const event = parsed.data;
+  if ('start_ms' in event && event.end_ms < event.start_ms) {
+    return {
+      type: 'error',
+      code: 'invalid_server_event',
+      message: 'Invalid OpenAI Live event time interval.',
+      raw,
+    };
+  }
+  switch (event.type) {
+    case 'session.started':
+      return {
+        type: 'session-started',
+        sessionId: event.session.id,
+        delegationMode:
+          event.session.delegation?.type === 'responses'
+            ? 'provider'
+            : 'client',
+        raw,
+      };
+    case 'session.closed':
+      return {
+        type: 'session-closed',
+        sessionId: event.session?.id,
+        usage: event.usage,
+        reason: event.reason,
+        raw,
+      };
+    case 'session.usage.updated':
+      return {
+        type: 'session-usage',
+        usage: event.usage,
+        contextWindowUsageRatio: event.context_window?.usage_ratio,
+        raw,
+      };
+    case 'session.output_audio.delta':
+      return { type: 'audio-chunk', delta: event.delta, raw };
+    case 'session.input_transcript.delta':
+    case 'session.output_transcript.delta':
+      return {
+        type: 'transcript-fragment',
+        speaker:
+          event.type === 'session.input_transcript.delta'
+            ? 'user'
+            : 'assistant',
+        delta: event.delta,
+        startMs: event.start_ms,
+        endMs: event.end_ms,
+        raw,
+      };
+    case 'session.delegation.created':
+      return {
+        type: 'delegation-created',
+        delegationId: event.delegation.id,
+        target:
+          event.delegation.target === 'responses'
+            ? 'provider'
+            : (event.delegation.target ?? undefined),
+        offsetMs: event.offset_ms ?? undefined,
+        ...(event.delegation.response_id != null
+          ? { responseId: event.delegation.response_id }
+          : {}),
+        raw,
+      };
+    case 'error':
+      return {
+        type: 'error',
+        message: event.error.message,
+        code: event.error.code ?? undefined,
+        clientEventId: event.error.client_event_id ?? undefined,
+        raw,
+      };
+    case 'session.updated':
+      return {
+        type: 'command-acknowledged',
+        command: 'session.update',
+        clientEventId: event.client_event_id ?? undefined,
+        raw,
+      };
+    case 'session.input_audio.muted':
+    case 'session.input_audio.unmuted':
+      return {
+        type: 'command-acknowledged',
+        command: event.type.slice(0, -1),
+        clientEventId: event.client_event_id ?? undefined,
+        raw,
+      };
+    case 'session.instructions.appended':
+    case 'session.thinking.appended':
+    case 'session.commentary.appended':
+      return {
+        type: 'command-acknowledged',
+        command: event.type.slice(0, -2),
+        clientEventId: event.client_event_id ?? undefined,
+        raw,
+      };
+  }
+}
+
+export function serializeOpenAILiveClientEvent(
+  event: RealtimeModelV4ClientEvent,
+  modelId: string,
+): unknown {
+  const eventId =
+    'eventId' in event && event.eventId !== undefined
+      ? { event_id: event.eventId }
+      : {};
+  switch (event.type) {
+    case 'session-start':
+      return {
+        type: 'session.start',
+        session: buildOpenAILiveSessionConfig(event.config, modelId),
+        ...eventId,
+      };
+    case 'session-update':
+      throw new UnsupportedFunctionalityError({
+        functionality:
+          'OpenAI Live session-update; startup settings are immutable; use context-append or input-audio-mute/input-audio-unmute',
+      });
+    case 'session-close':
+      return { type: 'session.close', ...eventId };
+    case 'input-audio-append':
+      return {
+        type: 'session.input_audio.append',
+        audio: event.audio,
+        ...eventId,
+      };
+    case 'input-audio-mute':
+      return { type: 'session.input_audio.mute', ...eventId };
+    case 'input-audio-unmute':
+      return { type: 'session.input_audio.unmute', ...eventId };
+    case 'context-append': {
+      const context = z
+        .object({
+          content: z.string(),
+          delegationId: z.string().min(1).nullable(),
+        })
+        .parse(event);
+      const options = z
+        .strictObject({
+          channel: z
+            .enum(['instructions', 'thinking', 'commentary'])
+            .optional(),
+        })
+        .parse(event.providerOptions?.openai ?? {});
+      return {
+        type: `session.${options.channel ?? 'thinking'}.append`,
+        content: context.content,
+        delegation_id: context.delegationId,
+        ...eventId,
+      };
+    }
+    default:
+      throw new UnsupportedFunctionalityError({
+        functionality: `OpenAI Live command: ${event.type}; use continuous audio and context-append instead of voice-turn commands`,
+      });
+  }
+}

package/src/live/openai-live-session-config.ts

@@ -0,0 +1,145 @@
+import {
+  InvalidArgumentError,
+  UnsupportedFunctionalityError,
+  type Experimental_RealtimeModelV4SessionConfig as RealtimeModelV4SessionConfig,
+} from '@ai-sdk/provider';
+import { z } from 'zod/v4';
+import { openaiRealtimeModelLiveOptionsSchema } from './openai-realtime-model-live-options';
+
+const audioFormatSchema = z.union([
+  z.strictObject({
+    type: z.literal('audio/pcm'),
+    rate: z.union([z.literal(16000), z.literal(24000)]),
+  }),
+  z.strictObject({
+    type: z.enum(['audio/pcma', 'audio/pcmu']),
+    rate: z.literal(8000),
+  }),
+]);
+
+export function buildOpenAILiveSessionConfig(
+  config: RealtimeModelV4SessionConfig,
+  modelId: string,
+  transport: 'websocket' | 'webrtc' = 'websocket',
+): Record<string, unknown> {
+  for (const key of Object.keys(config)) {
+    if (
+      ![
+        'instructions',
+        'voice',
+        'inputAudioFormat',
+        'outputAudioFormat',
+        'providerOptions',
+      ].includes(key)
+    ) {
+      throw new UnsupportedFunctionalityError({
+        functionality: `OpenAI Live session setting: ${key}`,
+      });
+    }
+  }
+
+  if (
+    z
+      .object({ delegation: z.object({ type: z.literal('responses') }) })
+      .safeParse(config.providerOptions?.openai).success
+  ) {
+    throw new UnsupportedFunctionalityError({
+      functionality:
+        'OpenAI Live Responses delegation; only client delegation is supported',
+    });
+  }
+  const options = openaiRealtimeModelLiveOptionsSchema.parse(
+    config.providerOptions?.openai ?? {},
+  );
+  if (options.client !== undefined && transport !== 'webrtc') {
+    throw new UnsupportedFunctionalityError({
+      functionality: 'OpenAI Live client permissions outside WebRTC startup',
+    });
+  }
+  if (options.voice != null && config.voice != null) {
+    throw new InvalidArgumentError({
+      argument: 'voice',
+      message: 'Choose either voice or providerOptions.openai.voice.',
+    });
+  }
+  if (
+    transport === 'webrtc' &&
+    (config.inputAudioFormat !== undefined ||
+      config.outputAudioFormat !== undefined)
+  ) {
+    throw new UnsupportedFunctionalityError({
+      functionality:
+        'Fixed audio formats for OpenAI Live WebRTC; audio is negotiated through SDP',
+    });
+  }
+
+  const inputFormat =
+    config.inputAudioFormat == null
+      ? undefined
+      : audioFormatSchema.parse(config.inputAudioFormat);
+  const outputFormat =
+    config.outputAudioFormat == null
+      ? undefined
+      : audioFormatSchema.parse(config.outputAudioFormat);
+  if (
+    inputFormat != null &&
+    outputFormat != null &&
+    (inputFormat.type !== outputFormat.type ||
+      inputFormat.rate !== outputFormat.rate)
+  ) {
+    throw new InvalidArgumentError({
+      argument: 'outputAudioFormat',
+      message: 'OpenAI Live requires the same input and output audio format.',
+    });
+  }
+
+  return {
+    model: modelId,
+    ...(config.instructions !== undefined
+      ? { instructions: config.instructions }
+      : {}),
+    audio: {
+      ...(transport === 'websocket'
+        ? {
+            format: inputFormat ??
+              outputFormat ?? { type: 'audio/pcm', rate: 24000 },
+          }
+        : {}),
+      output: { voice: options.voice ?? config.voice ?? 'marin' },
+    },
+    ...(options.delegation !== undefined
+      ? { delegation: options.delegation }
+      : {}),
+    ...(options.client !== undefined
+      ? {
+          client: {
+            data_channel: {
+              ...(options.client.dataChannel.allowedClientEvents !== undefined
+                ? {
+                    allowed_client_events:
+                      options.client.dataChannel.allowedClientEvents,
+                  }
+                : {}),
+              ...(options.client.dataChannel.allowedServerEvents !== undefined
+                ? {
+                    allowed_server_events:
+                      options.client.dataChannel.allowedServerEvents === 'all'
+                        ? 'all'
+                        : options.client.dataChannel.allowedServerEvents.map(
+                            selector => ({
+                              type: selector.type,
+                              ...(selector.responseEvent !== undefined
+                                ? { response_event: selector.responseEvent }
+                                : {}),
+                            }),
+                          ),
+                  }
+                : {}),
+            },
+          },
+        }
+      : {}),
+    ...(options.input !== undefined ? { input: options.input } : {}),
+    ...(options.store !== undefined ? { store: options.store } : {}),
+  };
+}

package/src/live/openai-realtime-model-live-options.ts

@@ -0,0 +1,65 @@
+import { z } from 'zod/v4';
+
+export type OpenAIRealtimeModelLiveId = 'gpt-live-1' | (string & {});
+
+const serverEventSelectorSchema = z
+  .strictObject({
+    type: z.string(),
+    responseEvent: z.string().optional(),
+  })
+  .refine(
+    selector =>
+      (selector.type === 'response.event') ===
+      (selector.responseEvent !== undefined),
+    'responseEvent is required for response.event and forbidden for other event types.',
+  );
+
+export const openaiRealtimeModelLiveOptionsSchema = z.strictObject({
+  client: z
+    .strictObject({
+      dataChannel: z.strictObject({
+        allowedClientEvents: z
+          .union([z.literal('all'), z.array(z.string())])
+          .optional(),
+        allowedServerEvents: z
+          .union([z.literal('all'), z.array(serverEventSelectorSchema)])
+          .optional(),
+      }),
+    })
+    .optional(),
+  delegation: z
+    .strictObject({ type: z.literal('client') })
+    .nullable()
+    .optional(),
+  input: z
+    .array(
+      z.discriminatedUnion('role', [
+        z.strictObject({
+          type: z.literal('message'),
+          role: z.enum(['developer', 'user']),
+          content: z.tuple([
+            z.strictObject({ type: z.literal('input_text'), text: z.string() }),
+          ]),
+        }),
+        z.strictObject({
+          type: z.literal('message'),
+          role: z.literal('assistant'),
+          content: z.tuple([
+            z.strictObject({
+              type: z.enum(['text', 'output_text']),
+              text: z.string(),
+            }),
+          ]),
+        }),
+      ]),
+    )
+    .max(128)
+    .optional(),
+  store: z.boolean().optional(),
+  voice: z.strictObject({ id: z.string().min(1) }).optional(),
+});
+
+/** Experimental Live options under sessionConfig.providerOptions.openai. */
+export type OpenAIRealtimeModelLiveOptions = z.infer<
+  typeof openaiRealtimeModelLiveOptionsSchema
+>;