@xmaxai/web-sdk 1.0.4 → 1.0.6

package/README.md

@@ -214,10 +214,8 @@ The following JavaScript snippet creates a camera stream, starts real-time gener
 and binds the output to a video view. Run this within an async function in the browser,
 after a user action such as clicking a Start button.
 
-Choose a model with `RealtimeModel.x2_0` (`x2.0`) or
-`RealtimeModel.x2_0_trtc` (`x2.0-trtc`). For Agora, use
-`RealtimeModel.x2_0_agora` (`x2.0-agora`) with `RtcProvider.agora`. For VeRTC, use
-`RealtimeModel.x2_1_preview` (`x2.1-preview`) with `RtcProvider.vertc`.
+For Agora, use `RealtimeModel.x2_1_preview` (`x2.1-preview`). For TRTC, use
+`RealtimeModel.x2_0_trtc` (`x2.0-trtc`). `RealtimeModel.x2_0` (`x2.0`) uses VeRTC.
 
 ```javascript
 import {
@@ -272,11 +270,11 @@ videoView.remoteTrack = remoteStream.videoTrack;
 Add a `video-container` element with an explicit width and height to your page.
 The view displays a local camera preview until the first generated frame arrives.
 
-Use `updateVideoFormat` after creating the local camera stream, including during
+Use `updateUplinkVideoFormat` after creating the local camera stream, including during
 generation, to change upstream encoding without reconnecting or restarting the task:
 
 ```javascript
-await realtime.updateVideoFormat(new RealtimeVideoFormat({
+await realtime.updateUplinkVideoFormat(new RealtimeVideoFormat({
   width: 1024,
   height: 1920,
   fps: 20,
@@ -292,39 +290,110 @@ resolution, frame rate, and bitrate depend on the device, browser, and network;
 use the statistics callbacks to observe them. Await each operation before starting
 another configuration or lifecycle operation.
 
-TRTC is the default RTC provider. To use Agora, replace the manager configuration
-above; camera capture, generation, rendering, and cleanup use the same APIs:
+For a frame-rate-only experiment after connecting a local camera or file stream:
 
 ```javascript
-import { RtcProvider } from "@xmaxai/web-sdk";
+await realtime.updateUplinkFrameRate(24);
+```
+
+This preserves the last successfully applied width, height, resolved bitrate limits,
+and encoding preference. It does not recalculate default bitrates for the new frame
+rate or copy measured send statistics into the configuration. The complete encoder
+configuration is still resubmitted, so preservation of RTC's internal adaptation
+state is not guaranteed. Validate actual send statistics under weak networks.
+
+**Agora camera sender-only experiment:** camera capture stays at its initial settings.
+Camera updates use `RTCRtpSender.setParameters()` (`scaleResolutionDownBy`,
+`maxFramerate`, `maxBitrate` in bps), not Agora's `setEncoderConfiguration()`.
+Only the maximum bitrate is applied; the standard sender API cannot enforce the
+configured minimum bitrate. Pre-publish updates are queued and applied immediately
+after publication, so the very first transmitted frames may use the initial profile.
+Scaling uses the actual captured dimensions, preserves aspect ratio, and does not upscale.
+Sender replacement/rejoin reapplies the latest target. Unsupported or silently ignored
+parameters are reported as errors, without falling back to camera constraints.
+Read-back confirms accepted parameters, not actual frame dimensions or persistent
+ownership against Agora's later adjustments; validate Chrome/Safari using RTC statistics.
+Search `Agora Sender Parameters` for accepted limits and `Agora RTC Statistics` for
+capture/send dimensions. File-video encoding and other RTC providers are unchanged.
+
+Agora uplink and downlink adaptation are always enabled; no configuration flag or manual
+selector is needed. Other RTC providers are not yet integrated. After generation
+starts, three consecutive poor uplink quality samples lower one tier, and five good
+samples probe an upgrade. Five tiers jointly change resolution and frame rate:
+1024×1920/30, 768×1440/24, 640×1200/24, 512×960/20, and 384×720/16.
+Landscape dimensions are swapped; custom sources scale proportionally from their
+initial or most recently manually applied format, with even dimensions and FPS
+capped by that baseline. Automatic updates preserve this baseline across reconnects.
+Default bitrate limits are recalculated for each tier; explicit limits and encoding
+preference are preserved. Uplink changes do not alter generation or result-stream sizes.
+Downlink independently uses the same quality thresholds and five resolution tiers,
+relative to the generation's original dimensions. It sends only `change_target_size`
+with `[width, height]`, without changing FPS, bitrate limits, or model inference.
+The protocol has no application ACK: a successful send records a requested tier,
+not a confirmed output size; use remote RTC statistics to verify actual dimensions.
+New Agora local tracks read the last applied uplink tier before publishing
+(the camera sender-only experiment applies it immediately after publication);
+new generation tasks restore the last requested downlink tier via `start.params.target_size`.
+Each direction falls back to L2 when no valid history is available. History is stored
+in `localStorage`, separately by model and direction, with no expiry or stability window
+in this first version. Storage failures never block generation. The original L1 baseline
+is retained for recovery; reconnecting the same local track keeps its in-memory uplink tier.
+Downlink changes only resolution, not FPS. Interpolation follows the requested dimensions,
+showing raw video while frame sizes do not match. See
+[the strategy and weak-network test steps](docs/realtime-adaptive-quality.md).
+
+Each model declares its supported RTC providers. The SDK selects the model's
+default provider when `provider` is omitted. To use Agora, replace the manager
+configuration above; camera capture, generation, rendering, and cleanup use the same APIs:
 
+```javascript
 const realtime = client.createRealtimeManager(
   new RealtimeConfiguration({
-    provider: RtcProvider.agora,
-    model: RealtimeModel.x2_0_agora,
+    model: RealtimeModel.x2_1_preview,
   })
 );
 ```
 
-To use VeRTC, set both values explicitly:
+To use VeRTC, select `x2.0`:
 
 ```javascript
 const realtime = client.createRealtimeManager(
   new RealtimeConfiguration({
-    provider: RtcProvider.vertc,
-    model: RealtimeModel.x2_1_preview,
+    model: RealtimeModel.x2_0,
   })
 );
 ```
 
-The provider is fixed for each manager, defaults to TRTC, and is chosen only by
-`RealtimeConfiguration.provider`. The SDK does not infer it from the model or read
-`modelExtra.provider` from the backend. The selected provider determines how RTC
-credentials are parsed.
+The provider is fixed for each manager: `x2.0-trtc` uses `RtcProvider.trtc`,
+`x2.1-preview` uses `RtcProvider.agora`, and `x2.0` uses `RtcProvider.vertc`.
+Import `RtcProvider` to specify it explicitly; unsupported known-model/provider
+combinations are rejected during configuration. Use the exported
+`supportedRtcProviders(model)` to inspect model capabilities. The SDK does not read
+`modelExtra.provider` from the backend; the resolved configuration determines how
+RTC credentials are parsed.
+
+The `x2.1-preview` model uses `https://cloud.xmax.22duck.cn/open/api/v1` in both environments.
+The other built-in models and file uploads use the configured
+environment's production API endpoint: `https://cloud.xmax.22duck.cn/open/api/v1`
+for China and `https://api.xmax.cloud/open/api/v1` for Global.
 
-The `x2.0-agora` and `x2.1-preview` models use
-`https://dev.xmaxai.com/open/api/v1` in both environments.
-Other models and file uploads use the configured environment's API endpoint.
+Custom realtime model names are also accepted without an SDK update:
+
+```javascript
+const realtime = client.createRealtimeManager(
+  new RealtimeConfiguration({ model: "your-new-model" })
+);
+```
+
+The model name is sent unchanged; the server determines whether it is available.
+Unknown models inherit the `x2.1-preview` defaults: Agora, the production session API above
+(in both client environments), a 1024 × 1920 camera at 30 fps, and the same
+1024 × 1920 / 1920 × 1024 input resolution buckets and media rules.
+You may explicitly choose another supported `RtcProvider` for a custom model;
+`supportedRtcProviders` lists all SDK adapters for custom models, with Agora first,
+but this does not guarantee server-side support. `modelDisplayName` returns the
+custom name unchanged. Empty/blank or non-string model names, invalid RTC providers,
+and invalid video parameters are still rejected locally.
 
 VeRTC uses `@volcengine/rtc` 4.69.3 and the fixed AppID
 `69a177e226e9b90176a86b96`. Session credentials must contain this `rtc_app_id`,
@@ -403,6 +472,14 @@ For camera input, bind the returned video track to a preview view. The SDK enter
 `ready` after it has received a valid frame and the preview view is bound; observe
 this through `setStateListener`.
 
+At generation start, the SDK sends `params.mirror: true` for the front camera
+and `false` for the rear camera, local files, and network videos. The backend
+mirrors the input; generated results are not automatically mirrored in the
+browser. The front-camera local preview remains mirrored independently.
+This flag is sent only with `start`: switching cameras during an active generation
+does not update it. Disconnect and start generation again to apply the new camera
+position to the backend.
+
 <br>
 
 ### Resource Cleanup
@@ -437,6 +514,156 @@ this through `setStateListener`.
 
 <br>
 
+### Generate from a local video
+
+Use a manager configured for TRTC, Agora or VeRTC. The SDK decodes a local `File` or `Blob`, sends its video through Canvas and its audio through Web Audio, and receives the generated result over RTC. It does not upload the file to storage or request camera/microphone access.
+
+```ts
+// Call directly from a file-selection or click handler to unlock browser playback.
+const localStream = await realtime.createLocalVideoStream({
+  file,
+  videoFormat: new RealtimeVideoFormat({ width: 1920, height: 1024, fps: 30 }),
+  loop: true,
+  includeAudio: true,
+});
+
+// Optionally bind localStream.videoTrack to an SDK video view.
+const remoteStream = await realtime.startGeneration({
+  localStream,
+  context: new RealtimeContext({ prompt: "Transform the scene into an animation" }),
+});
+```
+
+Preparation stops at the first frame. After the connection is established and the start signal is sent, the file plays from the beginning; playback does not wait for a remote frame. `loop` defaults to `true`, looping both file video and audio; set it to `false` to play once. The current protocol does not acknowledge server input readiness, so frame-perfect processing of the very beginning is not guaranteed.
+
+For local files, requested dimensions automatically select the model resolution with the closest aspect ratio, preferring landscape on ties; exact matches are preserved. For example, `1280×720` selects `1920×1024`. Models without fixed resolutions retain their existing pixel-budget sizing rules. The file is scaled proportionally into the resulting Canvas, with black bars where needed, and the returned track's `videoFormat` reflects that target size. This does not change camera or network-video size validation. Local preview is always silent; `localAudioVolume` does not alter the file's uplink audio. File audio is enabled by default; a file without an audio track produces silence. Set `includeAudio: false` to omit the audio track entirely. Use `setRemoteAudioVolume` to hear the generated result (muted by default).
+
+`disconnect()` pauses playback and retains the source. A new generation after reconnecting starts from the beginning; updating conditions during generation does not restart playback. With looping disabled, file completion holds the final frame and leaves the RTC connection open so remote tail frames can play. `close()` releases all file resources and tracks. `updateUplinkVideoFormat()` can adjust uplink encoding without changing the original model input format.
+
+File codecs must be supported by the browser, and Canvas capture and Web Audio must be available. Keep the page foregrounded for stable frame delivery; browser background throttling can reduce the upload frame rate. Playback restrictions, decode errors and preparation timeouts are reported rather than silently sending an empty stream.
+
+### Generate from a network video
+
+Pass an HTTP or HTTPS video URL to let the server read the source directly. The SDK joins the RTC room and receives generated video and audio without capturing or publishing local media.
+
+```ts
+import { RealtimeContext, RealtimeVideoFormat, RealtimeVideoSampleMethod } from "@xmaxai/web-sdk";
+
+const localStream = await realtime.createNetworkVideoStream({
+  url: "https://example.com/source.mp4",
+  videoFormat: new RealtimeVideoFormat({ width: 1920, height: 1024, fps: 30 }),
+  sampleMethod: RealtimeVideoSampleMethod.time,
+  onFinish: () => {
+    console.log("Server finished processing the video");
+  },
+});
+
+const remoteStream = await realtime.startGeneration({
+  localStream,
+  context: new RealtimeContext({ prompt: "Transform the scene into an animation" }),
+});
+```
+
+Use a video format supported by the selected model. Bind the returned tracks to SDK video views as usual. The optional local preview plays once, muted, independently of server processing; no Canvas or local upload is involved. Creating the source does not start generation. Sampling defaults to `time`; `fps` is also available.
+
+The URL must be directly accessible to the server. Browser cookies and custom authorization headers are not forwarded. Local preview playback depends on browser codec support and page security policies; a preview failure does not stop server processing. The selected model's backend must support network video input.
+
+`onFinish` is delivered once for the current task after the server's `video_stopped` message and generation readiness. It does not mean the last remote frame has finished playing, so the SDK does not automatically disconnect. Local preview completion does not trigger this callback. Remote audio follows `setRemoteAudioVolume` (muted by default).
+
+`disconnect()` retains the source and preview for reconnection; `close()` releases them. Close the current source before creating another. `updateUplinkVideoFormat()` adjusts camera uplink encoding and is not available for a network video source.
+
+### Non-realtime video tasks
+
+Use `client.createNonRealtimeManager()` to submit video-file processing tasks.
+This is cloud processing, not on-device or offline execution. It uses the client's
+API key and selects a dedicated task endpoint from `XmaxEnvironment`:
+
+- `china`: `https://api.xmaxai.com/open/api/v1`
+- `global`: `https://api.xmax.ai/open/api/v1`
+
+Task requests append `/offline-task` and its subpaths to this base URL.
+Realtime model endpoint overrides do not affect task requests; storage credentials
+continue to use the environment's default API endpoint.
+
+```typescript
+import { NonRealtimeQuality, NonRealtimeTaskError } from "@xmaxai/web-sdk";
+
+const manager = client.createNonRealtimeManager();
+const controller = new AbortController();
+
+// videoFile is a File selected by the user.
+const storedVideo = await client.createStorageService().uploadVideo({
+  data: videoFile,
+  fileName: videoFile.name,
+  onProgress: (loaded, total) => console.log(loaded, total),
+});
+
+const task = await manager.submitTask({
+  videoPath: storedVideo.url,
+  referencePath: referenceImageURL, // Optional; obtain via uploadImage().
+  prompt: "Change the character's clothing while preserving the motion.",
+  quality: NonRealtimeQuality.hd,
+  // Omit fps to retain the source frame rate, including fractional rates.
+});
+
+// Persist task.uid before waiting, so the task can be queried after a reload.
+try {
+  const completed = await manager.waitForCompletion(task.uid, {
+    intervalMs: 2000,
+    timeoutMs: 30 * 60 * 1000, // Optional client-side deadline.
+    signal: controller.signal,
+    onTaskUpdated: (snapshot) => console.log(snapshot.status),
+  });
+  console.log(completed.result?.url);
+} catch (error) {
+  if (error instanceof NonRealtimeTaskError) {
+    console.error(error.message, error.task);
+  } else {
+    throw error;
+  }
+}
+```
+
+- `submitTask()` returns immediately after acceptance and never automatically
+  retries. A timeout or invalid response may still mean the server created and
+  charged for the task; do not blindly resubmit.
+- `waitForCompletion()` polls serially and retries transient query failures up to
+  three consecutive times by default, with backoff. A successful query resets the
+  retry count. Without `timeoutMs`, waiting continues until completion, failure, or
+  cancellation. `controller.abort()` stops local waiting and the in-flight query;
+  it does **not** cancel or refund the server task.
+- `getTask(uid)` returns any task status, including `error`. Waiting for an `error`
+  task, or a completed task without a usable result, throws `NonRealtimeTaskError`
+  with the task snapshot attached.
+- `getTasks([uid, ...])` queries up to 100 IDs. `listTasks({ pageNumber, pageSize,
+  status })` retrieves one page, newest first. Saved IDs can be queried or waited
+  on using a new manager instance.
+- Upload local videos with `client.createStorageService().uploadVideo()` before
+  submitting a task. It shares the image upload options and progress callback,
+  uploads the original bytes via COS `putObject`, and returns a `StoredFile`.
+  It does not transcode or use multipart upload; COS simple upload supports files
+  up to [5GB](https://cloud.tencent.com/document/product/436/14113).
+  MIME inference supports MP4/M4V, MOV, WebM, AVI, MKV, and OGV; successful storage
+  does not guarantee that the task processor supports the file's codec or container.
+
+### Downloading videos
+
+Call this from a user-initiated action, such as a download button:
+
+```typescript
+await client.createStorageService().downloadVideo({
+  url: videoURL,
+  fileName: "video.mp4",
+  // signal: controller.signal, // Optional cancellation.
+});
+```
+
+The method fetches the complete file as a Blob and then requests browser saving.
+The URL must be directly accessible and allow CORS for cross-origin requests;
+the SDK does not send API credentials or cookies to it. This requires a browser
+document and retains the complete file until the browser takes over. Resolution
+means saving was requested, not that the user saved the file to disk.
+
 ## Example Project
 
 A complete example application featuring React is available in

package/dist/{VideoContentMode-BM97_H8V.d.cts → VideoRenderStatistics-C7zqzfuS.d.cts}

@@ -190,4 +190,26 @@ declare enum VideoContentMode {
     fill = "fill"
 }
 
-export { CameraPosition as C, RealtimeVideoFormat as R, VideoContentMode as V, RealtimeVideoTrack as a, RealtimeVideoEncoderPreference as b, type RealtimeVideoFormatInit as c };
+/**
+ * 单个视频视图最近采样窗口的呈现帧率，不代表屏幕实际刷新帧率。
+ */
+interface VideoRenderStatistics {
+    /**
+     * video 按浏览器 presentedFrames 增量计数；canvas 按实际呈现提交计数。
+     */
+    readonly source: "video" | "canvas";
+    /**
+     * 原帧与插值帧的总呈现帧率，单位 fps。
+     */
+    readonly frameRate: number;
+    /**
+     * 原帧呈现帧率，单位 fps。
+     */
+    readonly originalFrameRate: number;
+    /**
+     * 插值帧呈现帧率，单位 fps；未开启插帧时为 0。
+     */
+    readonly interpolatedFrameRate: number;
+}
+
+export { CameraPosition as C, RealtimeVideoFormat as R, VideoContentMode as V, RealtimeVideoTrack as a, type VideoRenderStatistics as b, RealtimeVideoEncoderPreference as c, type RealtimeVideoFormatInit as d };

package/dist/{VideoContentMode-BM97_H8V.d.ts → VideoRenderStatistics-C7zqzfuS.d.ts}

@@ -190,4 +190,26 @@ declare enum VideoContentMode {
     fill = "fill"
 }
 
-export { CameraPosition as C, RealtimeVideoFormat as R, VideoContentMode as V, RealtimeVideoTrack as a, RealtimeVideoEncoderPreference as b, type RealtimeVideoFormatInit as c };
+/**
+ * 单个视频视图最近采样窗口的呈现帧率，不代表屏幕实际刷新帧率。
+ */
+interface VideoRenderStatistics {
+    /**
+     * video 按浏览器 presentedFrames 增量计数；canvas 按实际呈现提交计数。
+     */
+    readonly source: "video" | "canvas";
+    /**
+     * 原帧与插值帧的总呈现帧率，单位 fps。
+     */
+    readonly frameRate: number;
+    /**
+     * 原帧呈现帧率，单位 fps。
+     */
+    readonly originalFrameRate: number;
+    /**
+     * 插值帧呈现帧率，单位 fps；未开启插帧时为 0。
+     */
+    readonly interpolatedFrameRate: number;
+}
+
+export { CameraPosition as C, RealtimeVideoFormat as R, VideoContentMode as V, RealtimeVideoTrack as a, type VideoRenderStatistics as b, RealtimeVideoEncoderPreference as c, type RealtimeVideoFormatInit as d };