@unisphere/models-types 1.2.0 → 1.3.0

package/README.md

@@ -1,134 +1,504 @@
-# Unisphere - Kaltura Models
+# Kaltura Avatar SDK
 
-## Overview
+A TypeScript SDK for integrating with Kaltura's speech-to-video (STV) avatar service.
 
-This repository is a **Unisphere workspace** for an experience named **Models**.
+## Features
 
-It contains everything you need to:
-- Create and publish npm / JFrog packages  
-- Bundle and version runtime artifacts  
-- Build and deploy applications  
-- Develop locally and promote changes to production  
+- 🎭 **Simple API** - Create sessions and control avatars with just a few lines of code
+- 🎥 **WebRTC Streaming** - Real-time avatar video via WHEP protocol
+- 🗣️ **Text-to-Speech** - Automatic text chunking for optimal processing
+- 🎵 **Audio Support** - Send audio files for avatar to speak
+- 🔄 **Auto-Retry** - Built-in retry logic with exponential backoff
+- 💓 **Keep-Alive** - Automatic session maintenance every 10 seconds
+- 🎬 **Auto-Attach** - Optional automatic video element creation and attachment
+- 🛠️ **TypeScript** - Full type safety and IntelliSense support
+- 📦 **Modular** - Clean SDK-of-SDKs architecture
 
-The Unisphere workspace acts as the single entry point for the entire lifecycle — from local development, through CI/CD, and all the way to production deployment.
+## Installation
 
-To better understand the ideas behind Unisphere and how it works, read more at:  
-https://unisphere.kaltura.com/
+```bash
+npm install @unisphere/models-sdk-js
+```
 
----
+## Quick Start
 
-## Prerequisites
+### Option 1: Auto-Attach Video (Recommended)
 
-At the moment, **deploying Unisphere experiences is available only to Kaltura employees**.
+```typescript
+import { KalturaAvatarSession } from '@unisphere/models-sdk-js';
 
-Before doing anything else, make sure your machine is properly configured to access:
-- Kaltura GitHub repositories  
-- Kaltura JFrog Artifactory registries  
+// 1. Create SDK instance
+const session = new KalturaAvatarSession('your-api-key', {
+  baseUrl: 'https://api.avatar.example.com/v1/avatar-session'
+});
 
-Follow the setup guide here:  
-https://unisphere.kaltura.com/docs/create/kaltura-employees/setup-machine
+// 2. Create session with auto-attach
+// The SDK will automatically create a video element inside the container
+await session.createSession({
+  avatarId: 'avatar-123',
+  voiceId: 'voice-456', // optional
+  videoContainerId: 'avatar-container' // ID of div element
+});
 
----
+// 3. Make avatar speak
+await session.sayText('Hello from Kaltura Avatar!');
 
-## Add the Experience to Kaltura Github Organization
+// 4. End session
+await session.endSession();
+```
 
-Before you can deploy or activate this experience, it must be registered in Github under Kaltura Organization.
+### Option 2: Manual Attach
 
-This step connects your Unisphere workspace to the relevant Kaltura infrastructure and enables CI/CD, deployment, and activation flows.
+```typescript
+import { KalturaAvatarSession } from '@unisphere/models-sdk-js';
 
-Follow the guide here:  
-https://unisphere.kaltura.com/docs/create/devops/create-github-repository
+// 1. Create SDK instance
+const session = new KalturaAvatarSession('your-api-key', {
+  baseUrl: 'https://api.avatar.example.com/v1/avatar-session'
+});
 
----
+// 2. Create session
+await session.createSession({
+  avatarId: 'avatar-123',
+  voiceId: 'voice-456' // optional
+});
 
-## Create Artifacts
+// 3. Attach video to container
+// The SDK will create a video element inside the specified div
+session.attachAvatar('avatar-container');
 
-Unisphere allows you to create and manage different types of artifacts, including:
-- Packages  
-- Runtimes  
-- Applications  
+// 4. Make avatar speak
+await session.sayText('Hello from Kaltura Avatar!');
 
-Each artifact type has its own role in the ecosystem, but they all live and evolve together inside this workspace.
+// 5. End session
+await session.endSession();
+```
 
-Read more about creating artifacts here:  
-https://unisphere.kaltura.com/docs/create/overview
+**HTML Setup:**
 
----
+```html
+<div id="avatar-container" style="width: 512px; height: 512px;"></div>
+```
 
-## Deploy
+## API Reference
 
-To deploy **packages, applications, and runtimes**, follow the deployment guide:  
-https://unisphere.kaltura.com/docs/create/devops/deploy
+### Constructor
 
-For **applications and runtimes**, deployment is only part of the process.  
-After deployment, they must also be activated to become available in production.
+```typescript
+new KalturaAvatarSession(apiKey: string, config?: AvatarConfig)
+```
 
-Activation guide:  
-https://unisphere.kaltura.com/docs/create/devops/activate
+**Parameters:**
+- `apiKey` - Your Kaltura Avatar API key
+- `config` - Optional configuration:
+  - `baseUrl` - Backend API URL (required for non-default endpoints)
+  - `iceServers` - Custom ICE servers for WebRTC (optional, defaults to backend-provided TURN servers)
+  - `iceTransportPolicy` - 'all' or 'relay' (default: 'all')
+  - `retryConfig` - Retry configuration for API calls and connections
+  - `logLevel` - 'debug' | 'info' | 'warn' | 'error' (default: 'info')
+
+### Methods
+
+#### `createSession(options: CreateSessionOptions): Promise<void>`
+
+Create a new avatar session and establish WebRTC connection. Optionally auto-attach video to a container.
+
+**Options:**
+- `avatarId` (required) - ID of the avatar to use
+- `voiceId` (optional) - ID of the voice to use for TTS
+- `videoContainerId` (optional) - ID of the div element to automatically attach video to
+
+```typescript
+// With auto-attach
+await session.createSession({
+  avatarId: 'avatar-123',
+  voiceId: 'voice-456', // optional
+  videoContainerId: 'avatar-container' // optional - auto-attach video
+});
+
+// Without auto-attach
+await session.createSession({
+  avatarId: 'avatar-123',
+  voiceId: 'voice-456' // optional
+});
+```
 
----
+**Note:** When a session is created, the SDK automatically:
+1. Initializes the backend client
+2. Retrieves WHEP URL and TURN server configuration from the backend
+3. Establishes WebRTC connection
+4. Starts automatic keep-alive (every 10 seconds)
+5. Attaches video if `videoContainerId` is provided
 
-## Local Development
+#### `attachAvatar(containerId: string): void`
 
-Unisphere provides a structured local development flow that mirrors production as closely as possible, while still keeping things fast and developer-friendly.
+Attach avatar video to a container div by creating a video element inside it. If a video element already exists in the container, it will be reused.
 
-This includes:
-- Local builds  
-- Watching and rebuilding artifacts  
-- Running applications in isolation or as part of a larger workspace  
+**Parameters:**
+- `containerId` - ID of the div element to attach video to
 
-Read more about local development here:  
-https://unisphere.kaltura.com/docs/create/local-development
+```typescript
+// The SDK will create/find a video element inside this div
+session.attachAvatar('avatar-container');
+```
 
----
+**HTML:**
+```html
+<div id="avatar-container" style="width: 512px; height: 512px;">
+  <!-- Video element will be created here automatically -->
+</div>
+```
 
-## Available Commands
+**Note:** The created video element will have:
+- `autoplay` and `playsinline` attributes
+- Width and height set to 100%
+- Appropriate styling for fullscreen display within the container
 
-This workspace exposes a small set of opinionated commands to help you explore, validate, and manage the experience.
+#### `sayText(text: string): Promise<void>`
 
-### Start
-Lists all available commands for each artifact in the experience, including how to build and serve runtimes, and how to spawn applications.
+Send text for the avatar to speak. Text is automatically chunked into 3-word segments.
 
-```bash
-npm run start
+```typescript
+await session.sayText('Hello, how are you doing today?');
+// Internally split into: ['Hello, how are', 'you doing today']
 ```
 
+#### `say(mp3File: File | Blob): Promise<void>`
 
-### Info
-Displays information about the current workspace, including detected artifacts, configuration, and environment details.
+Send audio file for the avatar to speak.
 
-```bash
-npm run info
+```typescript
+const audioFile = new File([audioBlob], 'speech.mp3', { type: 'audio/mpeg' });
+await session.say(audioFile);
 ```
 
-### Check
-Validates the workspace setup and verifies that all required artifacts and configurations are in place.
+#### `interrupt(): Promise<void>`
 
-```bash
-npm run check
+Interrupt the avatar's current speech.
+
+```typescript
+await session.interrupt();
 ```
 
-### Reset
-Resets the workspace to a clean state. Useful when experimenting or when local state becomes inconsistent.
+#### `endSession(): Promise<void>`
 
-```bash
-npm run reset
+End the session and cleanup resources. This will:
+- Stop the automatic keep-alive interval
+- Disconnect the WebRTC connection
+- End the session on the backend
+- Clean up all resources
+
+```typescript
+await session.endSession();
 ```
 
+### State Management
+
+#### `getSessionId(): string | null`
+
+Get the current session ID.
+
+```typescript
+const sessionId = session.getSessionId();
+console.log('Session ID:', sessionId);
+```
+
+#### `getSessionState(): SessionState`
+
+Get current session state.
+
+```typescript
+const state = session.getSessionState();
+// States: IDLE, CREATING, READY, ENDED, ERROR
+```
+
+#### `getConnectionState(): ConnectionState`
+
+Get current WebRTC connection state.
+
+```typescript
+const connState = session.getConnectionState();
+// States: DISCONNECTED, CONNECTING, CONNECTED, FAILED, CLOSED
+```
+
+### Events
+
+#### `on(event: SessionEvent, callback: Function): void`
 
----
+Register event handlers.
+
+```typescript
+// State changes
+session.on('stateChange', (state: SessionState) => {
+  console.log('Session state:', state);
+});
+
+// Connection changes
+session.on('connectionChange', (state: ConnectionState) => {
+  console.log('Connection state:', state);
+});
+
+// Errors
+session.on('error', (error: AvatarError) => {
+  console.error('Error:', error.message, error.code);
+});
+```
+
+## Complete Example
+
+```typescript
+import {
+  KalturaAvatarSession,
+  SessionState,
+  ConnectionState,
+  AvatarError,
+} from '@unisphere/models-sdk-js';
+
+async function runAvatarDemo() {
+  // Create SDK instance with custom config
+  const session = new KalturaAvatarSession('your-api-key', {
+    baseUrl: 'https://api.avatar.example.com/v1/avatar-session',
+    logLevel: 'info',
+    retryConfig: {
+      maxAttempts: 3,
+      initialDelayMs: 500,
+      maxDelayMs: 5000,
+      backoffMultiplier: 2,
+    },
+  });
+
+  // Setup event listeners
+  session.on('stateChange', (state: SessionState) => {
+    console.log('Session state changed:', state);
+  });
+
+  session.on('connectionChange', (state: ConnectionState) => {
+    console.log('Connection state changed:', state);
+  });
+
+  session.on('error', (error: AvatarError) => {
+    console.error('SDK Error:', error.message);
+  });
+
+  try {
+    // Create session with auto-attach
+    console.log('Creating session...');
+    await session.createSession({
+      avatarId: 'avatar-123',
+      voiceId: 'voice-456',
+      videoContainerId: 'avatar-container' // Auto-attach to div
+    });
+
+    // Session is now ready, keep-alive started automatically
+    console.log('Session ID:', session.getSessionId());
+    console.log('Session State:', session.getSessionState());
+    console.log('Connection State:', session.getConnectionState());
+
+    // Say some text (automatically chunked)
+    await session.sayText('Welcome to Kaltura Avatar! How can I help you today?');
+
+    // Wait for user interaction...
+    await new Promise(resolve => setTimeout(resolve, 5000));
+
+    // Interrupt if needed
+    await session.interrupt();
+
+    // Say something else
+    await session.sayText('Thanks for trying the Kaltura Avatar SDK!');
+
+    // Wait before ending
+    await new Promise(resolve => setTimeout(resolve, 3000));
+
+    // End session (automatically stops keep-alive)
+    await session.endSession();
+    console.log('Session ended successfully');
+
+  } catch (error) {
+    if (error instanceof AvatarError) {
+      console.error('Avatar Error:', error.code, error.message);
+    } else {
+      console.error('Unexpected error:', error);
+    }
+  }
+}
+
+// HTML: <div id="avatar-container" style="width: 512px; height: 512px;"></div>
+runAvatarDemo();
+```
+
+## Architecture
+
+The SDK follows an "SDK of SDKs" pattern with clean separation of concerns:
+
+```
+KalturaAvatarSession (Public API)
+├── AvatarControlSDK (HTTP/JWT)
+│   ├── Session lifecycle (create, init, end)
+│   ├── Avatar control (sayText, say, interrupt)
+│   └── Keep-alive management
+└── AvatarRTCSDK (WebRTC)
+    ├── SignalingManager (WHEP protocol)
+    └── PeerConnectionManager (RTCPeerConnection)
+```
+
+### Session Initialization Flow
+
+1. **Create Session** - Establishes backend session and receives JWT token
+2. **Init Client** - Retrieves WHEP URL and TURN server configuration from backend
+3. **Initialize RTC SDK** - Creates RTC SDK with backend-provided configuration
+4. **Connect WebRTC** - Establishes WHEP connection using backend-provided endpoint
+5. **Start Keep-Alive** - Automatically sends keep-alive every 10 seconds
+6. **Auto-Attach (optional)** - Creates video element if `videoContainerId` provided
+
+### Benefits
+
+- **Separation of Concerns** - Each layer has a single responsibility
+- **Backend-Managed ICE Servers** - TURN configuration provided by backend, not client
+- **Automatic Keep-Alive** - Session maintenance handled automatically
+- **Testability** - Mock and test each layer independently
+- **Extensibility** - Easy to add new features or swap implementations
+- **Maintainability** - Clear boundaries and good TypeScript typing
+
+## Advanced Usage
+
+### Custom ICE Servers (Override Backend)
+
+By default, ICE servers (TURN configuration) are provided by the backend during `initClient`. However, you can override them:
+
+```typescript
+const session = new KalturaAvatarSession('your-api-key', {
+  baseUrl: 'https://api.avatar.example.com/v1/avatar-session',
+  // Override backend-provided ICE servers
+  iceServers: [
+    { urls: 'stun:stun.example.com:19302' },
+    {
+      urls: ['turn:turn.example.com:80', 'turn:turn.example.com:443'],
+      username: 'user',
+      credential: 'pass',
+    },
+  ],
+  iceTransportPolicy: 'relay', // Force TURN only
+});
+```
+
+**Note:** The SDK will fall back to your custom ICE servers only if the backend doesn't provide them.
+
+### Using Individual SDKs
+
+For advanced use cases, you can use the underlying SDKs directly:
+
+```typescript
+import {
+  AvatarControlSDK,
+  AvatarRTCSDK,
+} from '@unisphere/models-sdk-js';
+
+// HTTP control
+const controlSDK = new AvatarControlSDK({
+  baseUrl: 'http://localhost:6100/v1/avatar-session',
+  apiKey: 'your-api-key',
+  logLevel: 'debug',
+});
+
+// Create session
+const createRes = await controlSDK.createSession({
+  avatarId: 'avatar-123',
+  voiceId: 'voice-456',
+});
+
+// Initialize client to get WHEP URL and TURN servers
+const initRes = await controlSDK.initClient(createRes.sessionId);
+
+// WebRTC - initialized with backend-provided config
+const rtcSDK = new AvatarRTCSDK({
+  whepUrl: initRes.whepUrl,
+  iceServers: [{
+    urls: [
+      `turn:${initRes.turn.url}:80?transport=udp`,
+      `turn:${initRes.turn.url}:443?transport=udp`,
+      `turn:${initRes.turn.url}:80?transport=tcp`,
+    ],
+    username: initRes.turn.username,
+    credential: initRes.turn.credential,
+  }],
+});
+
+await rtcSDK.connect();
+```
+
+### Keep-Alive Management
+
+The SDK automatically sends keep-alive signals to the backend every 10 seconds once a session is created. This ensures:
+- Session remains active
+- Backend doesn't terminate idle sessions
+- Resources are properly maintained
+
+```typescript
+// Keep-alive is started automatically when session is READY
+await session.createSession({
+  avatarId: 'avatar-123',
+  videoContainerId: 'avatar-container'
+});
+// Keep-alive now running every 10 seconds
+
+// Keep-alive is stopped automatically when session ends
+await session.endSession();
+// Keep-alive stopped
+```
+
+**Configuration:**
+
+The keep-alive interval is set to 10 seconds and cannot be configured. If you need to manually manage keep-alive, use the `AvatarControlSDK` directly:
+
+```typescript
+import { AvatarControlSDK } from '@unisphere/models-sdk-js';
+
+const controlSDK = new AvatarControlSDK({
+  baseUrl: 'https://api.avatar.example.com/v1/avatar-session',
+  apiKey: 'your-api-key',
+});
+
+// Manual keep-alive
+await controlSDK.keepAlive(sessionId);
+```
+
+## Error Handling
+
+All errors are instances of `AvatarError` with a specific error code:
+
+```typescript
+try {
+  await session.sayText('Hello');
+} catch (error) {
+  if (error instanceof AvatarError) {
+    switch (error.code) {
+      case AvatarErrorCode.INVALID_STATE:
+        console.log('Wrong state - create session first');
+        break;
+      case AvatarErrorCode.API_REQUEST_FAILED:
+        console.log('API request failed - check network');
+        break;
+      case AvatarErrorCode.RTC_CONNECTION_FAILED:
+        console.log('WebRTC connection failed');
+        break;
+      default:
+        console.log('Error:', error.message);
+    }
+  }
+}
+```
 
-## Next Steps
+## Browser Support
 
-If this is your first time working with Unisphere, a great next step is to **create a runtime with a playground**.
+- Chrome/Edge 80+
+- Firefox 75+
+- Safari 14+
+- Requires WebRTC support
 
-Follow the Hello World walkthrough here:
-[https://unisphere.kaltura.com/docs/create/overview](https://unisphere.kaltura.com/docs/create/overview)
+## License
 
-The walkthrough will guide you through:
+AGPL-3.0
 
-* Creating your first runtime
-* Running it locally with a playground
-* Understanding how runtimes, packages, and applications connect together
+## Support
 
-This will give you a strong foundation for building and evolving the experience over time.
+For issues and questions, please visit our [GitHub repository](https://github.com/kaltura/models).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@unisphere/models-types",
-  "version": "1.2.0",
+  "version": "1.3.0",
   "author": "kaltura",
   "private": false,
   "license": "AGPL-3.0",