@xmaxai/web-sdk 1.0.4 → 1.0.5

package/README.md

@@ -292,39 +292,58 @@ resolution, frame rate, and bitrate depend on the device, browser, and network;
 use the statistics callbacks to observe them. Await each operation before starting
 another configuration or lifecycle operation.
 
-TRTC is the default RTC provider. To use Agora, replace the manager configuration
-above; camera capture, generation, rendering, and cleanup use the same APIs:
+Each model declares its supported RTC providers. The SDK selects the model's
+default provider when `provider` is omitted. To use Agora, replace the manager
+configuration above; camera capture, generation, rendering, and cleanup use the same APIs:
 
 ```javascript
-import { RtcProvider } from "@xmaxai/web-sdk";
-
 const realtime = client.createRealtimeManager(
   new RealtimeConfiguration({
-    provider: RtcProvider.agora,
     model: RealtimeModel.x2_0_agora,
   })
 );
 ```
 
-To use VeRTC, set both values explicitly:
+To use VeRTC, select the preview model:
 
 ```javascript
 const realtime = client.createRealtimeManager(
   new RealtimeConfiguration({
-    provider: RtcProvider.vertc,
     model: RealtimeModel.x2_1_preview,
   })
 );
 ```
 
-The provider is fixed for each manager, defaults to TRTC, and is chosen only by
-`RealtimeConfiguration.provider`. The SDK does not infer it from the model or read
-`modelExtra.provider` from the backend. The selected provider determines how RTC
-credentials are parsed.
+The provider is fixed for each manager: `x2.0` and `x2.0-trtc` use
+`RtcProvider.trtc`, `x2.0-agora` uses `RtcProvider.agora`, and
+`x2.1-preview` uses `RtcProvider.vertc`.
+Import `RtcProvider` to specify it explicitly; unsupported known-model/provider
+combinations are rejected during configuration. Use the exported
+`supportedRtcProviders(model)` to inspect model capabilities. The SDK does not read
+`modelExtra.provider` from the backend; the resolved configuration determines how
+RTC credentials are parsed.
 
 The `x2.0-agora` and `x2.1-preview` models use
 `https://dev.xmaxai.com/open/api/v1` in both environments.
-Other models and file uploads use the configured environment's API endpoint.
+The other built-in models and file uploads use the configured environment's API endpoint.
+
+Custom realtime model names are also accepted without an SDK update:
+
+```javascript
+const realtime = client.createRealtimeManager(
+  new RealtimeConfiguration({ model: "your-new-model" })
+);
+```
+
+The model name is sent unchanged; the server determines whether it is available.
+Unknown models inherit the `x2.1-preview` defaults: VeRTC, the dev session API above
+(in both client environments), a 1024 × 1920 camera at 30 fps, and the same
+1024 × 1920 / 1920 × 1024 input resolution buckets and media rules.
+You may explicitly choose another supported `RtcProvider` for a custom model;
+`supportedRtcProviders` lists all SDK adapters for custom models, with VeRTC first,
+but this does not guarantee server-side support. `modelDisplayName` returns the
+custom name unchanged. Empty/blank or non-string model names, invalid RTC providers,
+and invalid video parameters are still rejected locally.
 
 VeRTC uses `@volcengine/rtc` 4.69.3 and the fixed AppID
 `69a177e226e9b90176a86b96`. Session credentials must contain this `rtc_app_id`,
@@ -437,6 +456,156 @@ this through `setStateListener`.
 
 <br>
 
+### Generate from a local video
+
+Use a manager configured for TRTC, Agora or VeRTC. The SDK decodes a local `File` or `Blob`, sends its video through Canvas and its audio through Web Audio, and receives the generated result over RTC. It does not upload the file to storage or request camera/microphone access.
+
+```ts
+// Call directly from a file-selection or click handler to unlock browser playback.
+const localStream = await realtime.createLocalVideoStream({
+  file,
+  videoFormat: new RealtimeVideoFormat({ width: 1920, height: 1024, fps: 30 }),
+  loop: true,
+  includeAudio: true,
+});
+
+// Optionally bind localStream.videoTrack to an SDK video view.
+const remoteStream = await realtime.startGeneration({
+  localStream,
+  context: new RealtimeContext({ prompt: "Transform the scene into an animation" }),
+});
+```
+
+Preparation stops at the first frame. After the connection is established and the start signal is sent, the file plays from the beginning; playback does not wait for a remote frame. `loop` defaults to `true`, looping both file video and audio; set it to `false` to play once. The current protocol does not acknowledge server input readiness, so frame-perfect processing of the very beginning is not guaranteed.
+
+For local files, requested dimensions automatically select the model resolution with the closest aspect ratio, preferring landscape on ties; exact matches are preserved. For example, `1280×720` selects `1920×1024`. Models without fixed resolutions retain their existing pixel-budget sizing rules. The file is scaled proportionally into the resulting Canvas, with black bars where needed, and the returned track's `videoFormat` reflects that target size. This does not change camera or network-video size validation. Local preview is always silent; `localAudioVolume` does not alter the file's uplink audio. File audio is enabled by default; a file without an audio track produces silence. Set `includeAudio: false` to omit the audio track entirely. Use `setRemoteAudioVolume` to hear the generated result (muted by default).
+
+`disconnect()` pauses playback and retains the source. A new generation after reconnecting starts from the beginning; updating conditions during generation does not restart playback. With looping disabled, file completion holds the final frame and leaves the RTC connection open so remote tail frames can play. `close()` releases all file resources and tracks. `updateVideoFormat()` can adjust uplink encoding without changing the original model input format.
+
+File codecs must be supported by the browser, and Canvas capture and Web Audio must be available. Keep the page foregrounded for stable frame delivery; browser background throttling can reduce the upload frame rate. Playback restrictions, decode errors and preparation timeouts are reported rather than silently sending an empty stream.
+
+### Generate from a network video
+
+Pass an HTTP or HTTPS video URL to let the server read the source directly. The SDK joins the RTC room and receives generated video and audio without capturing or publishing local media.
+
+```ts
+import { RealtimeContext, RealtimeVideoFormat, RealtimeVideoSampleMethod } from "@xmaxai/web-sdk";
+
+const localStream = await realtime.createNetworkVideoStream({
+  url: "https://example.com/source.mp4",
+  videoFormat: new RealtimeVideoFormat({ width: 1920, height: 1024, fps: 30 }),
+  sampleMethod: RealtimeVideoSampleMethod.time,
+  onFinish: () => {
+    console.log("Server finished processing the video");
+  },
+});
+
+const remoteStream = await realtime.startGeneration({
+  localStream,
+  context: new RealtimeContext({ prompt: "Transform the scene into an animation" }),
+});
+```
+
+Use a video format supported by the selected model. Bind the returned tracks to SDK video views as usual. The optional local preview plays once, muted, independently of server processing; no Canvas or local upload is involved. Creating the source does not start generation. Sampling defaults to `time`; `fps` is also available.
+
+The URL must be directly accessible to the server. Browser cookies and custom authorization headers are not forwarded. Local preview playback depends on browser codec support and page security policies; a preview failure does not stop server processing. The selected model's backend must support network video input.
+
+`onFinish` is delivered once for the current task after the server's `video_stopped` message and generation readiness. It does not mean the last remote frame has finished playing, so the SDK does not automatically disconnect. Local preview completion does not trigger this callback. Remote audio follows `setRemoteAudioVolume` (muted by default).
+
+`disconnect()` retains the source and preview for reconnection; `close()` releases them. Close the current source before creating another. `updateVideoFormat()` adjusts camera uplink encoding and is not available for a network video source.
+
+### Non-realtime video tasks
+
+Use `client.createNonRealtimeManager()` to submit video-file processing tasks.
+This is cloud processing, not on-device or offline execution. It uses the client's
+API key and selects a dedicated task endpoint from `XmaxEnvironment`:
+
+- `china`: `https://api.xmaxai.com/open/api/v1`
+- `global`: `https://api.xmax.ai/open/api/v1`
+
+Task requests append `/offline-task` and its subpaths to this base URL.
+Realtime model endpoint overrides do not affect task requests; storage credentials
+continue to use the environment's default API endpoint.
+
+```typescript
+import { NonRealtimeQuality, NonRealtimeTaskError } from "@xmaxai/web-sdk";
+
+const manager = client.createNonRealtimeManager();
+const controller = new AbortController();
+
+// videoFile is a File selected by the user.
+const storedVideo = await client.createStorageService().uploadVideo({
+  data: videoFile,
+  fileName: videoFile.name,
+  onProgress: (loaded, total) => console.log(loaded, total),
+});
+
+const task = await manager.submitTask({
+  videoPath: storedVideo.url,
+  referencePath: referenceImageURL, // Optional; obtain via uploadImage().
+  prompt: "Change the character's clothing while preserving the motion.",
+  quality: NonRealtimeQuality.hd,
+  // Omit fps to retain the source frame rate, including fractional rates.
+});
+
+// Persist task.uid before waiting, so the task can be queried after a reload.
+try {
+  const completed = await manager.waitForCompletion(task.uid, {
+    intervalMs: 2000,
+    timeoutMs: 30 * 60 * 1000, // Optional client-side deadline.
+    signal: controller.signal,
+    onTaskUpdated: (snapshot) => console.log(snapshot.status),
+  });
+  console.log(completed.result?.url);
+} catch (error) {
+  if (error instanceof NonRealtimeTaskError) {
+    console.error(error.message, error.task);
+  } else {
+    throw error;
+  }
+}
+```
+
+- `submitTask()` returns immediately after acceptance and never automatically
+  retries. A timeout or invalid response may still mean the server created and
+  charged for the task; do not blindly resubmit.
+- `waitForCompletion()` polls serially and retries transient query failures up to
+  three consecutive times by default, with backoff. A successful query resets the
+  retry count. Without `timeoutMs`, waiting continues until completion, failure, or
+  cancellation. `controller.abort()` stops local waiting and the in-flight query;
+  it does **not** cancel or refund the server task.
+- `getTask(uid)` returns any task status, including `error`. Waiting for an `error`
+  task, or a completed task without a usable result, throws `NonRealtimeTaskError`
+  with the task snapshot attached.
+- `getTasks([uid, ...])` queries up to 100 IDs. `listTasks({ pageNumber, pageSize,
+  status })` retrieves one page, newest first. Saved IDs can be queried or waited
+  on using a new manager instance.
+- Upload local videos with `client.createStorageService().uploadVideo()` before
+  submitting a task. It shares the image upload options and progress callback,
+  uploads the original bytes via COS `putObject`, and returns a `StoredFile`.
+  It does not transcode or use multipart upload; COS simple upload supports files
+  up to [5GB](https://cloud.tencent.com/document/product/436/14113).
+  MIME inference supports MP4/M4V, MOV, WebM, AVI, MKV, and OGV; successful storage
+  does not guarantee that the task processor supports the file's codec or container.
+
+### Downloading videos
+
+Call this from a user-initiated action, such as a download button:
+
+```typescript
+await client.createStorageService().downloadVideo({
+  url: videoURL,
+  fileName: "video.mp4",
+  // signal: controller.signal, // Optional cancellation.
+});
+```
+
+The method fetches the complete file as a Blob and then requests browser saving.
+The URL must be directly accessible and allow CORS for cross-origin requests;
+the SDK does not send API credentials or cookies to it. This requires a browser
+document and retains the complete file until the browser takes over. Resolution
+means saving was requested, not that the user saved the file to disk.
+
 ## Example Project
 
 A complete example application featuring React is available in