@ai-sdk/provider 4.0.34 → 4.0.36

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,40 @@
1
1
  # @ai-sdk/provider
2
2
 
3
+ ## 4.0.36
4
+
5
+ ### Patch Changes
6
+
7
+ - 36802d4: Support native URL transcription with an optional provider capability, keeping existing file-generation contracts unchanged. Groq sends HTTP(S) audio URLs through its `url` parameter without downloading and re-uploading them, avoiding the attachment upload limit. Supplying a custom `download` function retains local download behavior.
8
+
9
+ For telemetry integrations, `TranscriptionEndEvent.audio.byteLength` and `audio.mediaType` can now be `undefined` when a provider fetches the audio URL natively. Check for missing values before using them as a number or string. Downloaded and inline audio continue to report their metadata; streaming transcription metadata is unchanged.
10
+
11
+ Strip media fragments before sending URLs to Groq. If Groq rejects a redirect, fall back to the existing secure local downloader and file upload rather than failing previously working URL inputs. Other provider errors are preserved.
12
+
13
+ - e0143b0: feat(ai): support provider-owned realtime transports
14
+
15
+ Add an optional `createTransport` interface and `provider` connection capability.
16
+
17
+ - 934100c: Report the actual OpenAI Chat HTTP endpoint (also used by Azure Chat and Bedrock Mantle Chat) through optional request metadata and model-call end events. Record its host and port on OpenTelemetry chat spans without exposing URL credentials, paths, queries, or fragments. Expose only the sanitized HTTP origin in text and object generation results and step/finish callbacks. Providers that do not report an endpoint leave these attributes unset.
18
+ - 892af4d: feat (provider): add server-side realtime extensions to the experimental RealtimeModelV4 spec
19
+
20
+ Extends the existing optional `getServerWebSocketConfig()` method and a normalized `usage` field so a trusted server/proxy can drive realtime providers the way it drives HTTP language models:
21
+
22
+ - `getServerWebSocketConfig({ intent? })` returns the upstream WebSocket `{ url, headers, protocols? }` for a server-initiated connection (header auth with the long-lived credential). Providers fail fast for unsupported endpoint intents instead of silently falling back. It is the server-side sibling of the browser-only `getWebSocketConfig`.
23
+ - `serializeClientEvent` and `buildSessionConfig` now accept the same optional `intent` so server-side callers can keep endpoint selection and session serialization aligned.
24
+ - New `RealtimeModelV4Usage` type and optional `usage` on `response-done`, `input-transcription-completed`, and provider `custom` snapshot events. Gross totals and known modality buckets remain separate, and duration is reported as `seconds`. Counts are gross (cache-inclusive) with cached portions reported separately so consumers can apply their own billing split without re-parsing `raw`.
25
+ - New `RealtimeModelV4ServerConnection` and `RealtimeModelV4SessionIntent` types, exported with the `Experimental_` prefix.
26
+
27
+ - 41465fd: Expose optional transcription word timings separately from segments and accept shared timestampGranularities. Explicit provider timing settings take precedence. OpenAI now sends timestamp_granularities only with verbose_json and warns when a requested timing is unavailable or excluded.
28
+ - 9754014: Add typed streaming heartbeat parts. Anthropic emits them only with `includePings: true` (disabled by default), without requiring raw chunks. Core forwards pings through stream retries without storing them as generated content or sending them to UI message streams. Consumers with exhaustive stream-part switches must handle the new `ping` union variant when upgrading, even when the opt-in is disabled. Typed pings do not flush partial words during stream smoothing.
29
+ - 4bb73b0: Add optional video, reranking and decision sections to model capability catalogs, including support for declaring provider options for these operations.
30
+ - 2e44144: Represent videos delivered directly to caller-managed storage with an explicit uploaded outcome. Video generation reports `uploadedVideos` without downloading or synthesizing video bytes; access `video` only when `videos` is nonempty.
31
+
32
+ ## 4.0.35
33
+
34
+ ### Patch Changes
35
+
36
+ - dcd2f8d: Introduce a model entry contract with named capability sections and request configuration. Consolidate the OpenAI and Anthropic model catalogs so existing request behavior and generated capability documentation use the same source of truth. Catalogs also declare the provider options each provider accepts and which models support them.
37
+
3
38
  ## 4.0.34
4
39
 
5
40
  ### Patch Changes
package/dist/index.d.ts CHANGED
@@ -776,6 +776,8 @@ export type LanguageModelV4GenerateResult = {
776
776
  * Optional request information for telemetry and debugging purposes.
777
777
  */
778
778
  request?: {
779
+ /** The actual HTTP endpoint used for this request, when known. */
780
+ url?: string;
779
781
  /**
780
782
  * Request HTTP body that was sent to the provider API.
781
783
  */
@@ -3409,6 +3411,8 @@ export type LanguageModelV4StreamPart = {
3409
3411
  usage: LanguageModelV4Usage;
3410
3412
  finishReason: LanguageModelV4FinishReason;
3411
3413
  providerMetadata?: SharedV4ProviderMetadata;
3414
+ } | {
3415
+ type: 'ping';
3412
3416
  } | {
3413
3417
  type: 'raw';
3414
3418
  rawValue: unknown;
@@ -3430,6 +3434,8 @@ export type LanguageModelV4StreamResult = {
3430
3434
  * Optional request information for telemetry and debugging purposes.
3431
3435
  */
3432
3436
  request?: {
3437
+ /** The actual HTTP endpoint used for this request, when known. */
3438
+ url?: string;
3433
3439
  /**
3434
3440
  * Request HTTP body that was sent to the provider API.
3435
3441
  */
@@ -5669,6 +5675,156 @@ export type EmbeddingModelV3Middleware = {
5669
5675
  }) => Promise<Awaited<ReturnType<EmbeddingModelV3['doEmbed']>>>;
5670
5676
  };
5671
5677
  //#endregion
5678
+ //#region src/model-capabilities.d.ts
5679
+ /**
5680
+ * Support declared by a provider's model catalog.
5681
+ *
5682
+ * `unknown` is distinct from unsupported. A condition describes when support is
5683
+ * available; it is documentation, not an expression to execute. Providers remain
5684
+ * responsible for evaluating their typed request policies.
5685
+ */
5686
+ type ModelCapability = boolean | 'unknown' | {
5687
+ readonly condition: string;
5688
+ };
5689
+ /**
5690
+ * Model metadata shared by provider implementations and generated documentation.
5691
+ *
5692
+ * These declarations describe the capabilities exposed by the provider. They do
5693
+ * not introduce validation or change request behavior. Provider catalogs keep
5694
+ * protocol details and request configuration alongside this metadata, and may
5695
+ * resolve different entries for different API endpoints.
5696
+ *
5697
+ * This contract is intentionally independent of the versioned model interfaces.
5698
+ * Providers can adopt it without requiring changes to existing model objects.
5699
+ */
5700
+ type ModelCapabilities = {
5701
+ /** Omitted sections are undeclared, not explicitly unsupported. */
5702
+ readonly language?: {
5703
+ readonly inputs: {
5704
+ readonly image: ModelCapability;
5705
+ readonly audio: ModelCapability;
5706
+ };
5707
+ readonly objectGeneration: ModelCapability;
5708
+ readonly tools: {
5709
+ readonly usage: ModelCapability;
5710
+ readonly streaming: ModelCapability;
5711
+ };
5712
+ readonly reasoning: {
5713
+ readonly support: ModelCapability;
5714
+ /** Omission means the provider does not declare an exhaustive list. */
5715
+ readonly efforts?: readonly string[];
5716
+ };
5717
+ readonly sampling: {
5718
+ readonly temperature: ModelCapability;
5719
+ readonly topP: ModelCapability;
5720
+ readonly logprobs: ModelCapability;
5721
+ };
5722
+ };
5723
+ readonly embedding?: {
5724
+ readonly defaultDimensions: number;
5725
+ readonly customDimensions: ModelCapability;
5726
+ };
5727
+ readonly image?: {
5728
+ readonly sizes: readonly string[] | {
5729
+ readonly description: string;
5730
+ };
5731
+ readonly fileInput: ModelCapability;
5732
+ readonly maskInput: ModelCapability;
5733
+ readonly maxImagesPerCall: number | undefined;
5734
+ };
5735
+ readonly video?: {
5736
+ readonly inputs: {
5737
+ readonly text: ModelCapability;
5738
+ /** The single starting image passed through the `image` option. */
5739
+ readonly image: ModelCapability;
5740
+ /** Role-tagged images passed through the `frameImages` option. */
5741
+ readonly firstFrame: ModelCapability;
5742
+ readonly lastFrame: ModelCapability;
5743
+ /** Media passed through the `inputReferences` option. */
5744
+ readonly referenceImage: ModelCapability;
5745
+ readonly referenceVideo: ModelCapability;
5746
+ };
5747
+ readonly generateAudio: ModelCapability;
5748
+ readonly aspectRatio: ModelCapability;
5749
+ readonly resolution: ModelCapability;
5750
+ readonly duration: ModelCapability;
5751
+ readonly fps: ModelCapability;
5752
+ readonly seed: ModelCapability;
5753
+ /** Undefined means the catalog does not declare a fixed limit. */
5754
+ readonly maxVideosPerCall: number | undefined;
5755
+ };
5756
+ readonly reranking?: {
5757
+ /** Input support exposed by the provider, including compatibility conversions. */
5758
+ readonly documents: {
5759
+ readonly text: ModelCapability;
5760
+ readonly object: ModelCapability;
5761
+ };
5762
+ readonly topN: ModelCapability;
5763
+ };
5764
+ readonly decision?: {
5765
+ readonly questionTypes: {
5766
+ readonly boolean: ModelCapability;
5767
+ readonly choice: ModelCapability;
5768
+ readonly score: ModelCapability;
5769
+ readonly box: ModelCapability;
5770
+ };
5771
+ readonly inputs: {
5772
+ readonly image: ModelCapability;
5773
+ };
5774
+ readonly reasoning: ModelCapability;
5775
+ readonly maxOutputTokens: ModelCapability;
5776
+ };
5777
+ readonly transcription?: {
5778
+ readonly transcription: ModelCapability;
5779
+ readonly streaming: ModelCapability;
5780
+ readonly duration: ModelCapability;
5781
+ readonly segments: ModelCapability;
5782
+ readonly language: ModelCapability;
5783
+ readonly streamingDelay?: ModelCapability;
5784
+ };
5785
+ readonly speechTranslation?: {
5786
+ readonly translatedAudio: ModelCapability;
5787
+ readonly translatedText: ModelCapability;
5788
+ readonly sourceTranscript: ModelCapability;
5789
+ };
5790
+ readonly speech?: {
5791
+ readonly instructions: ModelCapability;
5792
+ };
5793
+ readonly realtime?: {
5794
+ readonly conversation: 'continuous' | 'turn-based';
5795
+ readonly transports: readonly ('websocket' | 'webrtc')[];
5796
+ readonly connections: readonly ('client-secret-websocket' | 'server-websocket' | 'webrtc')[];
5797
+ };
5798
+ };
5799
+ /**
5800
+ * A provider option, declared once at the provider level.
5801
+ */
5802
+ type ProviderOptionDefinition = {
5803
+ readonly description: string;
5804
+ };
5805
+ /**
5806
+ * The provider options a provider accepts, grouped by the operation they
5807
+ * configure. Section names match `ModelCapabilities`; option names match the
5808
+ * keys accepted under `providerOptions.<provider>` for that operation.
5809
+ */
5810
+ type ProviderOptionDefinitions = { readonly [Section in keyof ModelCapabilities]?: Readonly<Record<string, ProviderOptionDefinition>>; };
5811
+ /**
5812
+ * Per-model support for the options in the provider's
5813
+ * `ProviderOptionDefinitions`, keyed by the same sections and option names.
5814
+ */
5815
+ type ModelProviderOptions = { readonly [Section in keyof ModelCapabilities]?: Readonly<Record<string, ModelCapability>>; };
5816
+ /**
5817
+ * One provider-owned model record, which can declare several operations.
5818
+ * RequestConfig is a provider-defined type for request encoding and defaults.
5819
+ * Feature support belongs in capabilities, including provider-specific features.
5820
+ */
5821
+ type ModelEntry<RequestConfig = never> = {
5822
+ readonly capabilities: ModelCapabilities;
5823
+ /** Omitted sections or options are undeclared, not explicitly unsupported. */
5824
+ readonly providerOptions?: ModelProviderOptions;
5825
+ readonly requestConfig?: RequestConfig;
5826
+ };
5827
+ //#endregion
5672
5828
  //#region src/reranking-model/v4/reranking-model-v4-call-options.d.ts
5673
5829
  type RerankingModelV4CallOptions = {
5674
5830
  /**
@@ -5928,6 +6084,11 @@ type SpeechModelV4 = {
5928
6084
  //#endregion
5929
6085
  //#region src/transcription-model/v4/transcription-model-v4-call-options.d.ts
5930
6086
  type TranscriptionModelV4ProviderOptions$1 = Record<string, JSONObject>;
6087
+ /**
6088
+ * Timing detail to request from a transcription model.
6089
+ * `word` is for word highlighting. `segment` is for caption-sized phrases.
6090
+ */
6091
+ type TranscriptionModelV4TimestampGranularity = 'word' | 'segment';
5931
6092
  type TranscriptionModelV4CallOptions = {
5932
6093
  /**
5933
6094
  * Audio data to transcribe.
@@ -5940,6 +6101,16 @@ type TranscriptionModelV4CallOptions = {
5940
6101
  * @see https://www.iana.org/assignments/media-types/media-types.xhtml
5941
6102
  */
5942
6103
  mediaType: string;
6104
+ /**
6105
+ * Word and segment timings to request.
6106
+ *
6107
+ * Supporting adapters map this onto the provider's native control when that
6108
+ * provider option is omitted. An explicit provider option wins. Supporting
6109
+ * adapters warn when they cannot provide a requested granularity.
6110
+ * An empty array requests no additional timing detail; it does not suppress
6111
+ * timings returned by default.
6112
+ */
6113
+ timestampGranularities?: Array<TranscriptionModelV4TimestampGranularity>;
5943
6114
  /**
5944
6115
  * Additional provider-specific options that are passed through to the provider
5945
6116
  * as body parameters.
@@ -5993,6 +6164,25 @@ type TranscriptionModelV4Result = {
5993
6164
  */
5994
6165
  endSecond: number;
5995
6166
  }>;
6167
+ /**
6168
+ * Word-level timings reported by the provider, whether requested or returned
6169
+ * by default. Existing `segments` behavior is preserved; some providers also
6170
+ * put word timings in `segments`. Omitted when word timings are unavailable.
6171
+ */
6172
+ words?: Array<{
6173
+ /**
6174
+ * The text of this word.
6175
+ */
6176
+ text: string;
6177
+ /**
6178
+ * The start time of this word in seconds.
6179
+ */
6180
+ startSecond: number;
6181
+ /**
6182
+ * The end time of this word in seconds.
6183
+ */
6184
+ endSecond: number;
6185
+ }>;
5996
6186
  /**
5997
6187
  * The detected language of the audio content, as an ISO-639-1 code (e.g., 'en' for English).
5998
6188
  * May be undefined if the language couldn't be detected.
@@ -6232,6 +6422,20 @@ type TranscriptionModelV4 = {
6232
6422
  * Generates a transcript.
6233
6423
  */
6234
6424
  doGenerate(options: TranscriptionModelV4CallOptions): PromiseLike<TranscriptionModelV4Result>;
6425
+ /**
6426
+ * Optional native URL transcription. Its presence signals that HTTP(S)
6427
+ * audio URLs can be sent to the provider without downloading them locally.
6428
+ * The provider is responsible for fetching the audio; request headers are
6429
+ * for the provider API and are not forwarded to the audio host.
6430
+ *
6431
+ * Return `undefined` if the provider cannot fetch this URL and the SDK
6432
+ * should fall back to downloading it locally and calling `doGenerate`.
6433
+ * Throw for other errors (such as authentication or invalid audio).
6434
+ * Existing providers can continue implementing only `doGenerate`.
6435
+ */
6436
+ doGenerateFromUrl?(options: Omit<TranscriptionModelV4CallOptions, 'audio' | 'mediaType'> & {
6437
+ audio: URL;
6438
+ }): PromiseLike<TranscriptionModelV4Result | undefined>;
6235
6439
  /**
6236
6440
  * Streams a transcript for live audio.
6237
6441
  *
@@ -7505,6 +7709,95 @@ type RealtimeModelV4ClientEvent = {
7505
7709
  type: 'response-cancel';
7506
7710
  };
7507
7711
  //#endregion
7712
+ //#region src/realtime-model/v4/realtime-model-v4-server-connection.d.ts
7713
+ /**
7714
+ * The realtime endpoint variant to open.
7715
+ *
7716
+ * Most providers expose a single conversational ("conversation") voice
7717
+ * endpoint; some additionally expose dedicated transcription
7718
+ * endpoints that live at a different URL. `intent` selects which one a
7719
+ * server-side connection should target. Providers should fail fast when an
7720
+ * intent is not supported by their endpoint and codec.
7721
+ */
7722
+ type RealtimeModelV4SessionIntent = 'conversation' | 'transcription';
7723
+ /**
7724
+ * Descriptor for a server-initiated upstream WebSocket connection: the URL to
7725
+ * dial plus the headers (and optional subprotocols) to send on the handshake.
7726
+ *
7727
+ * Unlike `getWebSocketConfig` — which is browser-side and authenticates with a
7728
+ * short-lived token carried in a subprotocol — this carries the long-lived
7729
+ * provider credential in `headers` and is intended for a trusted server or
7730
+ * proxy that owns the socket itself.
7731
+ */
7732
+ type RealtimeModelV4ServerConnection = {
7733
+ /**
7734
+ * The `wss://` (or `ws://`) URL to connect to, including any provider-specific
7735
+ * query parameters (e.g. model id, transcription intent).
7736
+ */
7737
+ url: string;
7738
+ /**
7739
+ * Headers to send on the WebSocket handshake, including provider
7740
+ * authentication (e.g. `Authorization: Bearer <key>`).
7741
+ */
7742
+ headers: Record<string, string>;
7743
+ /**
7744
+ * Optional WebSocket subprotocols to negotiate on the handshake.
7745
+ */
7746
+ protocols?: string[];
7747
+ };
7748
+ //#endregion
7749
+ //#region src/realtime-model/v4/realtime-model-v4-usage.d.ts
7750
+ /**
7751
+ * Normalized realtime usage on completion events or provider custom snapshots.
7752
+ * Snapshots without a response association must not be summed as increments.
7753
+ *
7754
+ * Counts are the provider's raw counts: the input buckets INCLUDE any cached
7755
+ * tokens, and the cached portions are reported separately in
7756
+ * `cachedInput*Tokens`. This mirrors how providers report realtime usage on the
7757
+ * wire and lets a consumer apply its own cached-vs-uncached billing split
7758
+ * without re-parsing the event's `raw` payload.
7759
+ *
7760
+ * Every field is optional: providers only populate what they actually report,
7761
+ * and the whole `usage` object is omitted when nothing is observable. Gross
7762
+ * totals can include modalities without a dedicated text/audio bucket.
7763
+ */
7764
+ type RealtimeModelV4Usage = {
7765
+ /** Gross input tokens across all reported modalities; includes cached tokens. */
7766
+ inputTokens?: number;
7767
+ /** Gross output tokens across all reported modalities. */
7768
+ outputTokens?: number;
7769
+ /** Cached portion of gross input tokens, when the provider reports a total. */
7770
+ cachedInputTokens?: number;
7771
+ /**
7772
+ * Input text tokens (includes any cached text tokens).
7773
+ */
7774
+ inputTextTokens?: number;
7775
+ /**
7776
+ * Input audio tokens (includes any cached audio tokens).
7777
+ */
7778
+ inputAudioTokens?: number;
7779
+ /**
7780
+ * Output text tokens.
7781
+ */
7782
+ outputTextTokens?: number;
7783
+ /**
7784
+ * Output audio tokens.
7785
+ */
7786
+ outputAudioTokens?: number;
7787
+ /**
7788
+ * Cached portion of the input text tokens (subset of `inputTextTokens`).
7789
+ */
7790
+ cachedInputTextTokens?: number;
7791
+ /**
7792
+ * Cached portion of the input audio tokens (subset of `inputAudioTokens`).
7793
+ */
7794
+ cachedInputAudioTokens?: number;
7795
+ /**
7796
+ * Audio duration in seconds. Populated for duration-billed transcription.
7797
+ */
7798
+ seconds?: number;
7799
+ };
7800
+ //#endregion
7508
7801
  //#region src/realtime-model/v4/realtime-model-v4-server-event.d.ts
7509
7802
  /**
7510
7803
  * Normalized events emitted by the realtime model (model → browser).
@@ -7587,6 +7880,7 @@ type RealtimeModelV4ServerEvent = {
7587
7880
  type: 'input-transcription-completed';
7588
7881
  itemId: string;
7589
7882
  transcript: string;
7883
+ usage?: RealtimeModelV4Usage;
7590
7884
  raw: unknown;
7591
7885
  } | {
7592
7886
  type: 'response-created';
@@ -7596,6 +7890,7 @@ type RealtimeModelV4ServerEvent = {
7596
7890
  type: 'response-done';
7597
7891
  responseId: string;
7598
7892
  status: string;
7893
+ usage?: RealtimeModelV4Usage;
7599
7894
  raw: unknown;
7600
7895
  } | {
7601
7896
  type: 'output-item-added';
@@ -7697,9 +7992,54 @@ type RealtimeModelV4ServerEvent = {
7697
7992
  * The original event type string from the provider.
7698
7993
  */
7699
7994
  rawType: string;
7995
+ /** Provider usage snapshot without an inferred response association. */
7996
+ usage?: RealtimeModelV4Usage;
7700
7997
  raw: unknown;
7701
7998
  };
7702
7999
  //#endregion
8000
+ //#region src/realtime-model/v4/realtime-model-v4-transport.d.ts
8001
+ type RealtimeModelV4MediaStream = typeof globalThis extends {
8002
+ MediaStream: {
8003
+ prototype: infer T;
8004
+ };
8005
+ } ? T : never;
8006
+ /** Callbacks for a provider-owned browser transport. */
8007
+ type RealtimeModelV4TransportOptions = {
8008
+ onEvent: (event: RealtimeModelV4ServerEvent) => void | Promise<void>;
8009
+ onError: (error: Error) => void;
8010
+ onFatalError: (error: Error, drain?: Promise<void>) => void;
8011
+ onClose: (error?: Error) => void;
8012
+ onCapturing: (value: boolean) => void;
8013
+ onPlaying: (value: boolean) => void;
8014
+ };
8015
+ /**
8016
+ * Browser transport for providers whose media protocol requires its own client.
8017
+ * Construction must not acquire media or open a connection. Provider-specific
8018
+ * dependencies and session setup/cleanup belong to the implementation.
8019
+ */
8020
+ type RealtimeModelV4Transport = {
8021
+ connect(options: {
8022
+ /** Application endpoint for creating and closing provider sessions. */
8023
+ api: string;
8024
+ sessionConfig?: RealtimeModelV4SessionConfig;
8025
+ /** Caller-owned tracks must never be stopped by the transport. */
8026
+ stream?: RealtimeModelV4MediaStream;
8027
+ capture?: boolean;
8028
+ abortSignal: AbortSignal;
8029
+ }): Promise<void>;
8030
+ sendEvent(event: RealtimeModelV4ClientEvent, shouldSend?: () => boolean): Promise<void>;
8031
+ startCapture(stream?: RealtimeModelV4MediaStream): Promise<void>;
8032
+ stopCapture(): Promise<void>;
8033
+ stopPlayback(): void;
8034
+ resumePlayback(): Promise<void>;
8035
+ /** Drain already accepted incoming events. */
8036
+ finish(): Promise<void>;
8037
+ /** Release local media and await remote session cleanup. Must be idempotent. */
8038
+ close(): Promise<void>;
8039
+ /** Release local resources immediately and initiate remote cleanup. */
8040
+ dispose(): void;
8041
+ };
8042
+ //#endregion
7703
8043
  //#region src/realtime-model/v4/realtime-model-v4.d.ts
7704
8044
  /**
7705
8045
  * Specification for a realtime model that supports bidirectional
@@ -7726,20 +8066,14 @@ type RealtimeModelV4 = {
7726
8066
  conversation: 'continuous' | 'turn-based';
7727
8067
  transports: readonly ('websocket' | 'webrtc')[];
7728
8068
  /** Omission preserves the legacy client-secret WebSocket connection. */
7729
- connections?: readonly ('client-secret-websocket' | 'server-websocket' | 'webrtc')[];
8069
+ connections?: readonly ('client-secret-websocket' | 'server-websocket' | 'webrtc' | 'provider')[];
7730
8070
  /** Omission preserves session-update startup. WebRTC setup may start the session. */
7731
8071
  startup?: 'session-start' | 'session-update';
7732
8072
  /** Omission preserves transport-close finalization. */
7733
8073
  finalization?: 'session-close' | 'transport-close';
7734
8074
  };
7735
- /** Server-only connection settings. Headers can contain long-lived credentials. */
7736
- getServerWebSocketConfig?(): {
7737
- url: string;
7738
- headers: Record<string, string>;
7739
- } | PromiseLike<{
7740
- url: string;
7741
- headers: Record<string, string>;
7742
- }>;
8075
+ /** Create a provider-owned browser transport without opening a connection. */
8076
+ createTransport?(options: RealtimeModelV4TransportOptions): RealtimeModelV4Transport;
7743
8077
  /** Provider-specific setup for the WebRTC event channel. */
7744
8078
  getWebRTCConfig?(): {
7745
8079
  dataChannelLabel: string;
@@ -7786,16 +8120,37 @@ type RealtimeModelV4 = {
7786
8120
  /** Create a raw-event parser per connection and discard it on disconnect. */
7787
8121
  createServerEventParser?(): (raw: unknown) => RealtimeModelV4ServerEvent | RealtimeModelV4ServerEvent[];
7788
8122
  /**
7789
- * Browser-side: Serializes a normalized client event into the
7790
- * provider's native JSON format for sending over the WebSocket.
8123
+ * Browser-side: Serializes a normalized client event into the provider's
8124
+ * native WebSocket format. Server-side callers should pass the same `intent`
8125
+ * used for `getServerWebSocketConfig`.
8126
+ */
8127
+ serializeClientEvent(event: RealtimeModelV4ClientEvent, options?: {
8128
+ intent?: RealtimeModelV4SessionIntent;
8129
+ }): unknown | PromiseLike<unknown>;
8130
+ /**
8131
+ * Browser-side: Builds the provider-specific session configuration payload.
8132
+ * Server-side callers should pass the same `intent` used for
8133
+ * `getServerWebSocketConfig`.
7791
8134
  */
7792
- serializeClientEvent(event: RealtimeModelV4ClientEvent): unknown | PromiseLike<unknown>;
8135
+ buildSessionConfig(config: RealtimeModelV4SessionConfig, options?: {
8136
+ intent?: RealtimeModelV4SessionIntent;
8137
+ }): unknown;
7793
8138
  /**
7794
- * Browser-side: Builds the provider-specific session configuration
7795
- * payload from a normalized session config. Used to construct the
7796
- * session.update event sent after WebSocket connection.
8139
+ * Server-side: Returns the upstream WebSocket URL and auth headers for a
8140
+ * server-initiated connection — e.g. a proxy or gateway that holds the
8141
+ * long-lived provider credential and owns the socket itself.
8142
+ *
8143
+ * Unlike `getWebSocketConfig` (browser-side, authenticates with a short-lived
8144
+ * token carried in a subprotocol), this uses the provider credential the
8145
+ * model was constructed with (header auth) and selects supported provider
8146
+ * endpoint variants via `intent` (e.g. transcription).
8147
+ *
8148
+ * May return a promise for providers whose server connection requires a
8149
+ * round-trip (e.g. minting an ephemeral connection token).
7797
8150
  */
7798
- buildSessionConfig(config: RealtimeModelV4SessionConfig): unknown;
8151
+ getServerWebSocketConfig?(options?: {
8152
+ intent?: RealtimeModelV4SessionIntent;
8153
+ }): RealtimeModelV4ServerConnection | PromiseLike<RealtimeModelV4ServerConnection>;
7799
8154
  /**
7800
8155
  * Browser-side: Returns a message to auto-send back over the WebSocket
7801
8156
  * in response to a raw incoming message, or null if no response is needed.
@@ -8502,9 +8857,17 @@ type VideoModelV4CallOptions = {
8502
8857
  //#endregion
8503
8858
  //#region src/video-model/v4/video-model-v4-result.d.ts
8504
8859
  /**
8505
- * Generated video data. Can be a URL, base64-encoded string, or binary data.
8860
+ * Generated video data, or confirmation of delivery to caller-managed storage.
8506
8861
  */
8507
8862
  type VideoModelV4VideoData = {
8863
+ /**
8864
+ * The provider completed delivery to caller-managed storage.
8865
+ * No downloadable URL or video bytes are available to the SDK.
8866
+ * Never include a signed upload URL in this result.
8867
+ */
8868
+ type: 'uploaded';
8869
+ mediaType: string;
8870
+ } | {
8508
8871
  /**
8509
8872
  * Video available as a URL (most common for video providers).
8510
8873
  */
@@ -8531,7 +8894,7 @@ type VideoModelV4VideoData = {
8531
8894
  */
8532
8895
  type VideoModelV4Result = {
8533
8896
  /**
8534
- * Generated videos as URLs, base64 strings, or binary data.
8897
+ * Generated videos as URLs, base64 strings, binary data, or upload confirmations.
8535
8898
  *
8536
8899
  * Most providers return URLs to video files (MP4, WebM) due to large file sizes.
8537
8900
  * Use the discriminated union to indicate the type of video data being returned.
@@ -9076,5 +9439,5 @@ type VideoModelV3 = {
9076
9439
  }>;
9077
9440
  };
9078
9441
  //#endregion
9079
- export { type BatchV4 as Experimental_BatchV4, type BatchV4CancelResult as Experimental_BatchV4CancelResult, type BatchV4Error as Experimental_BatchV4Error, type BatchV4ItemResult as Experimental_BatchV4ItemResult, type BatchV4ListItem as Experimental_BatchV4ListItem, type BatchV4ListOptions as Experimental_BatchV4ListOptions, type BatchV4ListResult as Experimental_BatchV4ListResult, type BatchV4ModelIds as Experimental_BatchV4ModelIds, type BatchV4OperationOptions as Experimental_BatchV4OperationOptions, type BatchV4Request as Experimental_BatchV4Request, type BatchV4RequestBase as Experimental_BatchV4RequestBase, type BatchV4StartOptions as Experimental_BatchV4StartOptions, type BatchV4StartResult as Experimental_BatchV4StartResult, type BatchV4Status as Experimental_BatchV4Status, type DecisionModelV4 as Experimental_DecisionModelV4, type DecisionModelV4Answer as Experimental_DecisionModelV4Answer, type DecisionModelV4CallOptions as Experimental_DecisionModelV4CallOptions, type DecisionModelV4Input as Experimental_DecisionModelV4Input, type DecisionModelV4Question as Experimental_DecisionModelV4Question, type DecisionModelV4Result as Experimental_DecisionModelV4Result, type DecisionModelV4State as Experimental_DecisionModelV4State, type DecisionModelV4StatePart as Experimental_DecisionModelV4StatePart, DecisionRefusalError as Experimental_DecisionRefusalError, DecisionUnsupportedQuestionTypeError as Experimental_DecisionUnsupportedQuestionTypeError, type ImageBatchV4ItemResult as Experimental_ImageBatchV4ItemResult, type ImageBatchV4Request as Experimental_ImageBatchV4Request, type RealtimeFactoryV4 as Experimental_RealtimeFactoryV4, type RealtimeFactoryV4GetTokenOptions as Experimental_RealtimeFactoryV4GetTokenOptions, type RealtimeFactoryV4GetTokenResult as Experimental_RealtimeFactoryV4GetTokenResult, type RealtimeModelV4 as Experimental_RealtimeModelV4, type RealtimeModelV4AudioMessage as Experimental_RealtimeModelV4AudioMessage, type RealtimeModelV4ClientEvent as Experimental_RealtimeModelV4ClientEvent, type RealtimeModelV4ClientSecretOptions as Experimental_RealtimeModelV4ClientSecretOptions, type RealtimeModelV4ClientSecretResult as Experimental_RealtimeModelV4ClientSecretResult, type RealtimeModelV4ConversationItem as Experimental_RealtimeModelV4ConversationItem, type RealtimeModelV4FunctionCallOutput as Experimental_RealtimeModelV4FunctionCallOutput, type RealtimeModelV4ServerEvent as Experimental_RealtimeModelV4ServerEvent, type RealtimeModelV4SessionConfig as Experimental_RealtimeModelV4SessionConfig, type RealtimeModelV4TextMessage as Experimental_RealtimeModelV4TextMessage, type RealtimeModelV4ToolDefinition as Experimental_RealtimeModelV4ToolDefinition, type SpeechModelV4StreamOptions as Experimental_SpeechModelV4StreamOptions, type SpeechModelV4StreamPart as Experimental_SpeechModelV4StreamPart, type SpeechModelV4StreamResult as Experimental_SpeechModelV4StreamResult, type SpeechModelV4StreamTextPart as Experimental_SpeechModelV4StreamTextPart, type SpeechModelV4StreamUsage as Experimental_SpeechModelV4StreamUsage, type SpeechTranslationModelV4 as Experimental_SpeechTranslationModelV4, type SpeechTranslationModelV4StreamOptions as Experimental_SpeechTranslationModelV4StreamOptions, type SpeechTranslationModelV4StreamPart as Experimental_SpeechTranslationModelV4StreamPart, type SpeechTranslationModelV4StreamResult as Experimental_SpeechTranslationModelV4StreamResult, type SpeechTranslationModelV4Usage as Experimental_SpeechTranslationModelV4Usage, type StreamingSpeechModelV4 as Experimental_StreamingSpeechModelV4, type TextBatchV4ItemResult as Experimental_TextBatchV4ItemResult, type TextBatchV4Request as Experimental_TextBatchV4Request, type TranscriptionModelV4StreamOptions as Experimental_TranscriptionModelV4StreamOptions, type TranscriptionModelV4StreamPart as Experimental_TranscriptionModelV4StreamPart, type TranscriptionModelV4StreamResult as Experimental_TranscriptionModelV4StreamResult, type VideoModelV3 as Experimental_VideoModelV3, type VideoModelV3CallOptions as Experimental_VideoModelV3CallOptions, type VideoModelV3File as Experimental_VideoModelV3File, type VideoModelV3FrameImage as Experimental_VideoModelV3FrameImage, type VideoModelV3FrameType as Experimental_VideoModelV3FrameType, type VideoModelV3VideoData as Experimental_VideoModelV3VideoData, type VideoModelV4 as Experimental_VideoModelV4, type VideoModelV4CallOptions as Experimental_VideoModelV4CallOptions, type VideoModelV4File as Experimental_VideoModelV4File, type VideoModelV4FrameImage as Experimental_VideoModelV4FrameImage, type VideoModelV4FrameType as Experimental_VideoModelV4FrameType, type VideoModelV4OperationStartResult as Experimental_VideoModelV4OperationStartResult, type VideoModelV4OperationStatusResult as Experimental_VideoModelV4OperationStatusResult, type VideoModelV4OperationWebhook as Experimental_VideoModelV4OperationWebhook, type VideoModelV4Result as Experimental_VideoModelV4Result, type VideoModelV4VideoData as Experimental_VideoModelV4VideoData, type FilesV4, type FilesV4DeleteFileCallOptions, type FilesV4DeleteFileResult, type FilesV4DownloadFileCallOptions, type FilesV4DownloadFileResult, type FilesV4GetFileMetadataCallOptions, type FilesV4GetFileMetadataResult, type FilesV4UploadFileCallOptions, type FilesV4UploadFileResult, type FilesV4UploadFileStreamData, type ImageModelV2, type ImageModelV2CallOptions, type ImageModelV2CallWarning, type ImageModelV2ProviderMetadata, type ImageModelV3, type ImageModelV3CallOptions, type ImageModelV3File, type ImageModelV3ProviderMetadata, type ImageModelV3Usage, type ImageModelV4, type ImageModelV4CallOptions, type ImageModelV4File, type ImageModelV4ProviderMetadata, type ImageModelV4Result, type ImageModelV4Usage, type JSONArray, type JSONObject, type JSONSchema7, type JSONSchema7Definition, type JSONValue, type ProviderV2, type ProviderV3, type ProviderV4, type RerankingModelV3, type RerankingModelV3CallOptions, type RerankingModelV4, type RerankingModelV4CallOptions, type RerankingModelV4Result, type SkillsV4, type SkillsV4File, type SkillsV4UploadSkillCallOptions, type SkillsV4UploadSkillResult, type SpeechModelV2, type SpeechModelV2CallOptions, type SpeechModelV2CallWarning, type SpeechModelV3, type SpeechModelV3CallOptions, type SpeechModelV4, type SpeechModelV4CallOptions, type SpeechModelV4Result, type TranscriptionModelV2, type TranscriptionModelV2CallOptions, type TranscriptionModelV2CallWarning, type TranscriptionModelV3, type TranscriptionModelV3CallOptions, type TranscriptionModelV4, type TranscriptionModelV4CallOptions, type TranscriptionModelV4Result, type TypeValidationContext };
9442
+ export { type BatchV4 as Experimental_BatchV4, type BatchV4CancelResult as Experimental_BatchV4CancelResult, type BatchV4Error as Experimental_BatchV4Error, type BatchV4ItemResult as Experimental_BatchV4ItemResult, type BatchV4ListItem as Experimental_BatchV4ListItem, type BatchV4ListOptions as Experimental_BatchV4ListOptions, type BatchV4ListResult as Experimental_BatchV4ListResult, type BatchV4ModelIds as Experimental_BatchV4ModelIds, type BatchV4OperationOptions as Experimental_BatchV4OperationOptions, type BatchV4Request as Experimental_BatchV4Request, type BatchV4RequestBase as Experimental_BatchV4RequestBase, type BatchV4StartOptions as Experimental_BatchV4StartOptions, type BatchV4StartResult as Experimental_BatchV4StartResult, type BatchV4Status as Experimental_BatchV4Status, type DecisionModelV4 as Experimental_DecisionModelV4, type DecisionModelV4Answer as Experimental_DecisionModelV4Answer, type DecisionModelV4CallOptions as Experimental_DecisionModelV4CallOptions, type DecisionModelV4Input as Experimental_DecisionModelV4Input, type DecisionModelV4Question as Experimental_DecisionModelV4Question, type DecisionModelV4Result as Experimental_DecisionModelV4Result, type DecisionModelV4State as Experimental_DecisionModelV4State, type DecisionModelV4StatePart as Experimental_DecisionModelV4StatePart, DecisionRefusalError as Experimental_DecisionRefusalError, DecisionUnsupportedQuestionTypeError as Experimental_DecisionUnsupportedQuestionTypeError, type ImageBatchV4ItemResult as Experimental_ImageBatchV4ItemResult, type ImageBatchV4Request as Experimental_ImageBatchV4Request, type RealtimeFactoryV4 as Experimental_RealtimeFactoryV4, type RealtimeFactoryV4GetTokenOptions as Experimental_RealtimeFactoryV4GetTokenOptions, type RealtimeFactoryV4GetTokenResult as Experimental_RealtimeFactoryV4GetTokenResult, type RealtimeModelV4 as Experimental_RealtimeModelV4, type RealtimeModelV4AudioMessage as Experimental_RealtimeModelV4AudioMessage, type RealtimeModelV4ClientEvent as Experimental_RealtimeModelV4ClientEvent, type RealtimeModelV4ClientSecretOptions as Experimental_RealtimeModelV4ClientSecretOptions, type RealtimeModelV4ClientSecretResult as Experimental_RealtimeModelV4ClientSecretResult, type RealtimeModelV4ConversationItem as Experimental_RealtimeModelV4ConversationItem, type RealtimeModelV4FunctionCallOutput as Experimental_RealtimeModelV4FunctionCallOutput, type RealtimeModelV4ServerConnection as Experimental_RealtimeModelV4ServerConnection, type RealtimeModelV4ServerEvent as Experimental_RealtimeModelV4ServerEvent, type RealtimeModelV4SessionConfig as Experimental_RealtimeModelV4SessionConfig, type RealtimeModelV4SessionIntent as Experimental_RealtimeModelV4SessionIntent, type RealtimeModelV4TextMessage as Experimental_RealtimeModelV4TextMessage, type RealtimeModelV4ToolDefinition as Experimental_RealtimeModelV4ToolDefinition, type RealtimeModelV4Transport as Experimental_RealtimeModelV4Transport, type RealtimeModelV4TransportOptions as Experimental_RealtimeModelV4TransportOptions, type RealtimeModelV4Usage as Experimental_RealtimeModelV4Usage, type SpeechModelV4StreamOptions as Experimental_SpeechModelV4StreamOptions, type SpeechModelV4StreamPart as Experimental_SpeechModelV4StreamPart, type SpeechModelV4StreamResult as Experimental_SpeechModelV4StreamResult, type SpeechModelV4StreamTextPart as Experimental_SpeechModelV4StreamTextPart, type SpeechModelV4StreamUsage as Experimental_SpeechModelV4StreamUsage, type SpeechTranslationModelV4 as Experimental_SpeechTranslationModelV4, type SpeechTranslationModelV4StreamOptions as Experimental_SpeechTranslationModelV4StreamOptions, type SpeechTranslationModelV4StreamPart as Experimental_SpeechTranslationModelV4StreamPart, type SpeechTranslationModelV4StreamResult as Experimental_SpeechTranslationModelV4StreamResult, type SpeechTranslationModelV4Usage as Experimental_SpeechTranslationModelV4Usage, type StreamingSpeechModelV4 as Experimental_StreamingSpeechModelV4, type TextBatchV4ItemResult as Experimental_TextBatchV4ItemResult, type TextBatchV4Request as Experimental_TextBatchV4Request, type TranscriptionModelV4StreamOptions as Experimental_TranscriptionModelV4StreamOptions, type TranscriptionModelV4StreamPart as Experimental_TranscriptionModelV4StreamPart, type TranscriptionModelV4StreamResult as Experimental_TranscriptionModelV4StreamResult, type VideoModelV3 as Experimental_VideoModelV3, type VideoModelV3CallOptions as Experimental_VideoModelV3CallOptions, type VideoModelV3File as Experimental_VideoModelV3File, type VideoModelV3FrameImage as Experimental_VideoModelV3FrameImage, type VideoModelV3FrameType as Experimental_VideoModelV3FrameType, type VideoModelV3VideoData as Experimental_VideoModelV3VideoData, type VideoModelV4 as Experimental_VideoModelV4, type VideoModelV4CallOptions as Experimental_VideoModelV4CallOptions, type VideoModelV4File as Experimental_VideoModelV4File, type VideoModelV4FrameImage as Experimental_VideoModelV4FrameImage, type VideoModelV4FrameType as Experimental_VideoModelV4FrameType, type VideoModelV4OperationStartResult as Experimental_VideoModelV4OperationStartResult, type VideoModelV4OperationStatusResult as Experimental_VideoModelV4OperationStatusResult, type VideoModelV4OperationWebhook as Experimental_VideoModelV4OperationWebhook, type VideoModelV4Result as Experimental_VideoModelV4Result, type VideoModelV4VideoData as Experimental_VideoModelV4VideoData, type FilesV4, type FilesV4DeleteFileCallOptions, type FilesV4DeleteFileResult, type FilesV4DownloadFileCallOptions, type FilesV4DownloadFileResult, type FilesV4GetFileMetadataCallOptions, type FilesV4GetFileMetadataResult, type FilesV4UploadFileCallOptions, type FilesV4UploadFileResult, type FilesV4UploadFileStreamData, type ImageModelV2, type ImageModelV2CallOptions, type ImageModelV2CallWarning, type ImageModelV2ProviderMetadata, type ImageModelV3, type ImageModelV3CallOptions, type ImageModelV3File, type ImageModelV3ProviderMetadata, type ImageModelV3Usage, type ImageModelV4, type ImageModelV4CallOptions, type ImageModelV4File, type ImageModelV4ProviderMetadata, type ImageModelV4Result, type ImageModelV4Usage, type JSONArray, type JSONObject, type JSONSchema7, type JSONSchema7Definition, type JSONValue, type ModelCapabilities, type ModelCapability, type ModelEntry, type ModelProviderOptions, type ProviderOptionDefinition, type ProviderOptionDefinitions, type ProviderV2, type ProviderV3, type ProviderV4, type RerankingModelV3, type RerankingModelV3CallOptions, type RerankingModelV4, type RerankingModelV4CallOptions, type RerankingModelV4Result, type SkillsV4, type SkillsV4File, type SkillsV4UploadSkillCallOptions, type SkillsV4UploadSkillResult, type SpeechModelV2, type SpeechModelV2CallOptions, type SpeechModelV2CallWarning, type SpeechModelV3, type SpeechModelV3CallOptions, type SpeechModelV4, type SpeechModelV4CallOptions, type SpeechModelV4Result, type TranscriptionModelV2, type TranscriptionModelV2CallOptions, type TranscriptionModelV2CallWarning, type TranscriptionModelV3, type TranscriptionModelV3CallOptions, type TranscriptionModelV4, type TranscriptionModelV4CallOptions, type TranscriptionModelV4Result, type TranscriptionModelV4TimestampGranularity, type TypeValidationContext };
9080
9443
  //# sourceMappingURL=index.d.ts.map