@unith-ai/core-client 1.2.0 → 1.3.0

package/README.md

@@ -0,0 +1,429 @@
+# Unith Core Client Typescript SDK
+
+An SDK library for building complex digital human experiences using javascript/typescript.
+
+## Installation
+
+Install the package in your project through package manager.
+```shell
+npm install @unith-ai/core-client
+# or
+yarn add @unith-ai/core-client
+# or
+pnpm install @unith-ai/core-client
+```
+
+## Usage
+
+This library is designed for use in plain JavaScript applications or to serve as a foundation for framework-specific implementations. Before integrating it, verify if a dedicated library exists for your particular framework. That said, it's compatible with any project built on JavaScript.
+
+### Initialize Digital Human
+
+First, initialize the Conversation instance:
+```js
+const conversation = await Conversation.startDigitalHuman(options);
+```
+
+This will establish a WebSocket connection and initialize the digital human avatar with audio/video streaming capabilities.
+
+#### Session Configuration
+
+The options passed to `startDigitalHuman` specify how the session is established:
+```js
+const conversation = await Conversation.startDigitalHuman({
+  orgId: "your-org-id",
+  headId: "your-head-id",
+  username: "anonymous",
+  password: "Password1",
+  environment: "production", // or "development"
+  element: document.getElementById("video-container"), // HTML element for video output
+  apiKey: "your-api-key",
+  mode: "default",
+  frameRate: 30,
+  streamType: "jpg", // or "vp8"
+  quality: "high",
+  crop: false,
+  showIdle: false,
+  language: "en",
+  allowWakeLock: true,
+  fadeTransitionsType: VideoTransitionType.NONE,
+});
+```
+
+#### Required Parameters
+
+- **orgId** - Your organization ID
+- **headId** - The digital human head ID to use
+- **element** - HTML element where the video will be rendered
+- **username** - Authentication username (default: "anonymous")
+- **password** - Authentication password (default: "Password1")
+
+#### Optional Parameters
+
+- **environment** - API environment ("production" or "development", default: "production")
+- **apiKey** - API key for authentication (default: "")
+- **mode** - Conversation mode (default: "default")
+- **frameRate** - Video frame rate (default: 30)
+- **streamType** - Video stream format ("jpg" or "vp8", default: "jpg")
+- **quality** - Video quality ("high", "medium", or "low", default: "high")
+- **crop** - Whether to crop the video (default: false)
+- **showIdle** - Whether to show idle video when not speaking (default: false)
+- **language** - Language code for the conversation (default: browser language)
+- **allowWakeLock** - Prevent screen from sleeping during conversation (default: true)
+- **fadeTransitionsType** - Video transition type (default: VideoTransitionType.NONE)
+
+#### Optional Callbacks
+
+Register callbacks to handle various events:
+```js
+const conversation = await Conversation.startDigitalHuman({
+  // ... required options
+  onConnect: ({ userId, headInfo, microphoneAccess }) => {
+    console.log("Connected:", userId);
+  },
+  onDisconnect: (details) => {
+    console.log("Disconnected:", details.reason);
+  },
+  onStatusChange: ({ status }) => {
+    console.log("Status changed:", status); // "connecting", "connected", "disconnecting", "disconnected"
+  },
+  onText: (event) => {
+    console.log("User message:", event.text);
+  },
+  onResponse: (event) => {
+    console.log("AI response:", event.text);
+  },
+  onJoin: (event) => {
+    console.log("Joined conversation:", event);
+  },
+  onStreaming: (event) => {
+    console.log("Streaming event:", event.type); // "audio_frame", "video_frame", "metadata", "cache", "error"
+  },
+  onMuteStatusChange: ({ isMuted }) => {
+    console.log("Mute status:", isMuted);
+  },
+  onSpeakingStart: () => {
+    console.log("Digital human started speaking");
+  },
+  onSpeakingEnd: () => {
+    console.log("Digital human stopped speaking");
+  },
+  onStoppingEnd: () => {
+    console.log("Response stopped");
+  },
+  onTimeout: () => {
+    console.log("Session timed out");
+  },
+  onTimeoutWarning: () => {
+    console.log("Session will timeout soon");
+  },
+  onKeepSession: ({ granted }) => {
+    console.log("Keep session request:", granted);
+  },
+  onError: ({ message, endConversation, type }) => {
+    console.error("Error:", message);
+    // type: "toast" or "modal"
+    // endConversation: true if session should be restarted
+  },
+});
+```
+
+#### Event Types
+
+- **onConnect** - Called when the WebSocket connection is established
+- **onDisconnect** - Called when the connection is closed
+- **onStatusChange** - Called when connection status changes
+- **onText** - Called when a user text message is received
+- **onResponse** - Called when the AI generates a response
+- **onJoin** - Called when successfully joining the conversation
+- **onStreaming** - Called for audio/video frame events
+- **onMuteStatusChange** - Called when mute status changes
+- **onSpeakingStart** - Called when the digital human starts speaking
+- **onSpeakingEnd** - Called when the digital human finishes speaking
+- **onStoppingEnd** - Called when a response is manually stopped
+- **onTimeout** - Called when the session times out due to inactivity
+- **onTimeoutWarning** - Called before the session times out
+- **onKeepSession** - Called when a keep-alive request is processed
+- **onError** - Called when an error occurs
+
+### Getting Background Video
+
+Retrieve the idle background video URL for use in welcome screens or widget mode:
+```js
+const videoUrl = await Conversation.getBackgroundVideo({
+  orgId: "your-org-id",
+  headId: "your-head-id",
+  environment: "production",
+});
+```
+
+### Instance Methods
+
+#### startSession()
+
+Start the conversation session and begin audio playback:
+```js
+await conversation.startSession();
+```
+
+This method should be called after user interaction to ensure audio context is properly initialized, especially on mobile browsers.
+
+#### sendMessage(message)
+
+Send a text message to the digital human:
+```js
+conversation.sendMessage({
+  id: 1,
+  timestamp: new Date().toISOString(),
+  speaker: "user",
+  text: "Hello, how are you?",
+  isSent: false,
+  user_id: "user-123",
+  username: "John Doe",
+  event: EventType.TEXT,
+  visible: true,
+});
+```
+
+#### keepSession(message)
+
+Send a keep-alive message to prevent session timeout:
+```js
+conversation.keepSession({
+  id: 1,
+  timestamp: new Date().toISOString(),
+  speaker: "user",
+  text: "",
+  isSent: false,
+  user_id: "user-123",
+  username: "John Doe",
+  event: EventType.KEEP_SESSION,
+  visible: true,
+});
+```
+
+#### stopCurrentResponse()
+
+Stop the current response from the digital human:
+```js
+conversation.stopCurrentResponse();
+```
+
+This clears both audio and video queues and returns the digital human to idle state.
+
+#### toggleMuteStatus()
+
+Toggle the mute status of the audio output:
+```js
+const volume = await conversation.toggleMuteStatus();
+console.log("New volume:", volume); // 0 for muted, 1 for unmuted
+```
+
+#### getUserId()
+
+Get the current user's ID:
+```js
+const userId = conversation.getUserId();
+```
+
+#### endSession()
+
+End the conversation session and clean up resources:
+```js
+await conversation.endSession();
+```
+
+This closes the WebSocket connection, releases the wake lock, and destroys audio/video outputs.
+
+#### initializeMicrophone()
+
+Initialize microphone for speech recognition (ASR):
+```js
+const asrToken = await conversation.initializeMicrophone(
+  (result) => {
+    console.log("Speech result:", result);
+  },
+  (error) => {
+    console.error("Microphone error:", error);
+  },
+  (status) => {
+    console.log("Microphone status:", status);
+  }
+);
+```
+
+### Message Structure
+
+Messages sent to and from the digital human follow this structure:
+```typescript
+interface Message {
+  id: number;
+  timestamp: string; // ISO format
+  speaker: "user" | "backend";
+  text: string;
+  isSent: boolean;
+  user_id: string;
+  username: string;
+  event: EventType.TEXT | EventType.KEEP_SESSION;
+  visible: boolean;
+  session_id?: string; // Auto-generated
+}
+```
+
+### Event Types
+
+The SDK defines the following event types:
+```typescript
+enum EventType {
+  TEXT = "text",
+  RESPONSE = "response",
+  JOIN = "join",
+  STREAMING = "streaming",
+  BINARY = "binary",
+  TIMEOUT_WARNING = "timeout_warning",
+  TIME_OUT = "timeout",
+  KEEP_SESSION = "keep_session",
+}
+```
+
+### Streaming Event Types
+
+Streaming events can have the following types:
+
+- **audio_frame** - Audio data for playback
+- **video_frame** - Video frame data
+- **metadata** - Stream control metadata (start/end)
+- **cache** - Cached video response
+- **error** - Streaming error occurred
+
+### Error Handling
+
+Always handle errors appropriately:
+```js
+try {
+  const conversation = await Conversation.startDigitalHuman({
+    orgId: "your-org-id",
+    headId: "your-head-id",
+    element: videoElement,
+    onError: ({ message, endConversation, type }) => {
+      if (type === "toast") {
+        // Show toast notification
+        showToast(message);
+        if (endConversation) {
+          // Restart the session
+          restartSession();
+        }
+      } else if (type === "modal") {
+        // Show modal dialog
+        showModal(message);
+      }
+    },
+  });
+} catch (error) {
+  console.error("Failed to start digital human:", error);
+}
+```
+
+### Common Error Types
+
+The SDK handles several error scenarios:
+
+- **resource_exhausted** - Server at capacity
+- **deadline_exceeded** - Request timeout
+- **inactivity_timeout** - Session inactive for too long
+- **connection** - WebSocket connection failed
+
+## Framework Integration Examples
+
+### React/Preact Example
+```jsx
+import { useEffect, useRef, useState } from "react";
+import { Conversation, EventType } from "@unith-ai/core-client";
+
+function DigitalHuman() {
+  const videoRef = useRef(null);
+  const conversationRef = useRef(null);
+  const [status, setStatus] = useState("disconnected");
+  const [messages, setMessages] = useState([]);
+
+  useEffect(() => {
+    const startConversation = async () => {
+      try {
+        const conversation = await Conversation.startDigitalHuman({
+          orgId: "your-org-id",
+          headId: "your-head-id",
+          element: videoRef.current,
+          onStatusChange: ({ status }) => setStatus(status),
+          onText: (event) => {
+            setMessages((prev) => [...prev, event]);
+          },
+          onResponse: (event) => {
+            setMessages((prev) => [...prev, event]);
+          },
+        });
+
+        conversationRef.current = conversation;
+        await conversation.startSession();
+      } catch (error) {
+        console.error("Failed to start:", error);
+      }
+    };
+
+    startConversation();
+
+    return () => {
+      conversationRef.current?.endSession();
+    };
+  }, []);
+
+  const sendMessage = async (text) => {
+    if (!conversationRef.current) return;
+
+    await conversationRef.current.sendMessage({
+      id: messages.length,
+      timestamp: new Date().toISOString(),
+      speaker: "user",
+      text,
+      isSent: false,
+      user_id: conversationRef.current.getUserId(),
+      username: "User",
+      event: EventType.TEXT,
+      visible: true,
+    });
+  };
+
+  return (
+    <div>
+      <div ref={videoRef} style={{ width: "100%", height: "600px" }} />
+      <div>Status: {status}</div>
+      <button onClick={() => sendMessage("Hello!")}>Send Message</button>
+    </div>
+  );
+}
+```
+
+## TypeScript Support
+
+Full TypeScript types are included:
+```typescript
+import {
+  Conversation,
+  EventType,
+  Status,
+  HeadType,
+  VideoTransitionType,
+  type ConversationOptions,
+  type Message,
+  type IncomingSocketEvent,
+} from "@unith-ai/core-client";
+```
+
+## Development
+
+Please refer to the README.md file in the root of this repository.
+
+## Contributing
+
+Please create an issue first to discuss proposed changes. Any contributions are welcome!
+
+Remember, if merged, your code will be used as part of a MIT licensed project. By submitting a Pull Request, you are giving your consent for your code to be integrated into this library.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@unith-ai/core-client",
-  "version": "1.2.0",
+  "version": "1.3.0",
   "description": "",
   "main": "./dist/lib.umd.js",
   "module": "./dist/lib.module.js",