ulai-js-sdk 0.1.0 → 0.1.2

package/README.md

@@ -0,0 +1,557 @@
+# Ulai Cloud JavaScript/TypeScript Client SDK (`ulai-js-sdk`)
+
+[![npm version](https://img.shields.io/npm/v/ulai-js-sdk.svg?color=blue)](https://www.npmjs.com/package/ulai-js-sdk)
+[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)
+[![TypeScript](https://img.shields.io/badge/TypeScript-Ready-3178C6.svg)](https://www.typescriptlang.org/)
+
+The official JavaScript and TypeScript client SDK for Ulai Cloud real-time WebRTC sessions. Enables web applications to establish ultra-low-latency, bidirectional audio streaming with AI voice agents, receive real-time streaming transcripts, and manage room lifecycle events.
+
+---
+
+## Table of Contents
+
+- [Features](#features)
+- [Installation](#installation)
+- [Quick Start](#quick-start)
+  - [Pattern 1: Connect with Pre-provisioned Session Credentials (`Room`)](#pattern-1-connect-with-pre-provisioned-session-credentials-room)
+  - [Pattern 2: Connect using Ulai Cloud API Key (`UlaiClient`)](#pattern-2-connect-using-ulai-cloud-api-key-ulaiclient)
+- [React / Next.js Example](#react--nextjs-example)
+- [Vanilla JavaScript / HTML Example](#vanilla-javascript--html-example)
+- [API Reference](#api-reference)
+  - [`Room`](#room)
+  - [Events](#events)
+  - [`UlaiClient`](#ulaiclient)
+  - [TypeScript Interfaces](#typescript-interfaces)
+- [Handling Browser Audio & Autoplay Policies](#handling-browser-audio--autoplay-policies)
+- [Network & ICE Configuration](#network--ice-configuration)
+- [License](#license)
+
+---
+
+## Features
+
+- **W3C Perfect Negotiation:** Native polite-peer pattern implementation eliminates SDP glare and race conditions during dynamic renegotiation.
+- **Resilient Audio Pipelines:** Automatic track wrapping and fallback streams compatible with Pion SFU standalone and grouped media tracks.
+- **Dynamic Candidate Buffering:** Queues incoming ICE candidates until remote session descriptions are active, avoiding `InvalidStateError`.
+- **Automatic Reconnection:** Exponential backoff with jitter for WebRTC signaling and Control Plane event sockets.
+- **Message Outbox:** Buffers signaling messages until WebSocket connections reach an open state.
+- **Live Transcripts:** Real-time event streaming for interim and final speech recognition deltas (`transcriptDelta`).
+- **Audio Management:** Direct microphone capture with hardware-level mute controls (`setMuted()`), query methods (`isMuted()`), and pre-configured noise suppression, echo cancellation, and AGC.
+
+---
+
+## Installation
+
+```bash
+# npm
+npm install ulai-js-sdk
+
+# yarn
+yarn add ulai-js-sdk
+
+# pnpm
+pnpm add ulai-js-sdk
+```
+
+---
+
+## Quick Start
+
+### Pattern 1: Connect with Pre-provisioned Session Credentials (`Room`)
+
+This is the standard pattern for web applications. Your application server provisions a session and returns join credentials (`roomId`, `ticket`, `nodeAddress`, `eventsToken`), which the client SDK uses to join:
+
+```typescript
+import { Room } from 'ulai-js-sdk';
+
+// 1. Initialize Room with your Ulai Control Plane URL
+const room = new Room('https://cp.ulai.co.in');
+
+// 2. Set up event listeners
+room.on('connected', () => {
+  console.log('Connected to Ulai WebRTC room!');
+});
+
+// Play remote incoming audio (e.g., from an AI agent or participant)
+room.on('trackSubscribed', (track: MediaStreamTrack, stream: MediaStream) => {
+  if (track.kind === 'audio') {
+    const audio = document.createElement('audio');
+    audio.srcObject = stream;
+    audio.autoplay = true;
+    audio.style.display = 'none';
+    document.body.appendChild(audio);
+  }
+});
+
+// Receive live transcripts
+room.on('transcriptDelta', (delta) => {
+  console.log(`[${delta.speaker}] ${delta.text} (final: ${delta.is_final})`);
+});
+
+room.on('sessionTerminated', () => {
+  console.log('Call ended by remote party');
+});
+
+// 3. Connect to WebRTC media and the Control Plane event feed
+await room.connect({
+  roomId: 'session-xyz',          // The session / room ID
+  ticket: 'ticket-token-abc',     // SFU WebRTC signaling auth ticket
+  nodeAddress: 'sfu.ulai.co.in',  // Address of the assigned SFU node
+  eventsToken: 'events-token-123' // Token for live events & transcripts
+});
+
+// 4. Request microphone access and stream audio to the room
+await room.publishAudio();
+
+// 5. Mute / Unmute controls
+room.setMuted(true);
+console.log('Is muted:', room.isMuted());
+
+// 6. Graceful disconnect
+room.disconnect();
+```
+
+---
+
+### Pattern 2: Connect using Ulai Cloud API Key (`UlaiClient`)
+
+If you want the client to make direct join requests to the Ulai Cloud Control Plane:
+
+```typescript
+import { UlaiClient } from 'ulai-js-sdk';
+
+const client = new UlaiClient({
+  url: 'https://cp.ulai.co.in',
+  apiKey: 'YOUR_ULAI_API_KEY', // Optional Bearer token
+});
+
+// Requests join credentials from the Control Plane and connects WebRTC + Events
+const room = await client.joinRoom({
+  roomId: 'session-xyz',
+  participantId: 'user-01',
+});
+
+// Hook into room events
+room.on('trackSubscribed', (track, stream) => {
+  const audio = document.createElement('audio');
+  audio.srcObject = stream;
+  audio.autoplay = true;
+  document.body.appendChild(audio);
+});
+
+// Publish microphone
+await room.publishAudio();
+```
+
+---
+
+## React / Next.js Example
+
+```tsx
+'use client';
+
+import React, { useState, useRef, useEffect, useCallback } from 'react';
+import { Room, TranscriptDelta } from 'ulai-js-sdk';
+
+interface TranscriptItem {
+  id: string;
+  speaker: string;
+  text: string;
+  isFinal: boolean;
+}
+
+export function VoiceRoom({
+  controlPlaneUrl = 'https://cp.ulai.co.in',
+  fetchSessionCredentials,
+}: {
+  controlPlaneUrl?: string;
+  fetchSessionCredentials: () => Promise<{
+    roomId: string;
+    ticket: string;
+    nodeAddress: string;
+    eventsToken?: string;
+  }>;
+}) {
+  const [status, setStatus] = useState<'idle' | 'connecting' | 'connected' | 'ended'>('idle');
+  const [isMuted, setIsMuted] = useState(false);
+  const [transcripts, setTranscripts] = useState<TranscriptItem[]>([]);
+  
+  const roomRef = useRef<Room | null>(null);
+  const audioRef = useRef<HTMLAudioElement | null>(null);
+
+  const cleanup = useCallback(() => {
+    if (roomRef.current) {
+      roomRef.current.disconnect();
+      roomRef.current = null;
+    }
+    if (audioRef.current) {
+      audioRef.current.srcObject = null;
+    }
+  }, []);
+
+  useEffect(() => {
+    return () => cleanup();
+  }, [cleanup]);
+
+  const startSession = async () => {
+    try {
+      setStatus('connecting');
+      setTranscripts([]);
+
+      // 1. Fetch credentials from your backend
+      const creds = await fetchSessionCredentials();
+
+      // 2. Initialize Room
+      const room = new Room(controlPlaneUrl);
+      roomRef.current = room;
+
+      // 3. Register Event Handlers
+      room.on('connected', () => setStatus('connected'));
+
+      room.on('trackSubscribed', (track, stream) => {
+        if (track.kind === 'audio' && audioRef.current) {
+          audioRef.current.srcObject = stream;
+          audioRef.current.play().catch(console.error);
+        }
+      });
+
+      room.on('transcriptDelta', (delta: TranscriptDelta) => {
+        setTranscripts((prev) => {
+          const existingIndex = prev.findIndex((t) => t.id === delta.segment_id);
+          if (existingIndex !== -1) {
+            const updated = [...prev];
+            updated[existingIndex] = {
+              ...updated[existingIndex],
+              text: delta.text,
+              isFinal: delta.is_final,
+            };
+            return updated;
+          }
+          return [
+            ...prev,
+            { id: delta.segment_id, speaker: delta.speaker, text: delta.text, isFinal: delta.is_final },
+          ];
+        });
+      });
+
+      room.on('sessionTerminated', () => {
+        setStatus('ended');
+        cleanup();
+      });
+
+      room.on('disconnected', () => {
+        setStatus('idle');
+        cleanup();
+      });
+
+      // 4. Connect WebRTC & Events
+      await room.connect({
+        roomId: creds.roomId,
+        ticket: creds.ticket,
+        nodeAddress: creds.nodeAddress,
+        eventsToken: creds.eventsToken,
+      });
+
+      // 5. Publish microphone
+      await room.publishAudio();
+    } catch (err) {
+      console.error('Failed to start session:', err);
+      setStatus('idle');
+      cleanup();
+    }
+  };
+
+  const endSession = () => {
+    cleanup();
+    setStatus('idle');
+  };
+
+  const toggleMute = () => {
+    if (!roomRef.current) return;
+    const nextMuted = !isMuted;
+    roomRef.current.setMuted(nextMuted);
+    setIsMuted(nextMuted);
+  };
+
+  return (
+    <div style={{ maxWidth: 480, padding: 24, borderRadius: 12, border: '1px solid #e2e8f0' }}>
+      <audio ref={audioRef} autoPlay playsInline style={{ display: 'none' }} />
+
+      <h3>Live Voice Session</h3>
+      <p>Status: <strong>{status.toUpperCase()}</strong></p>
+
+      <div style={{ display: 'flex', gap: 12, margin: '16px 0' }}>
+        {status === 'idle' || status === 'ended' ? (
+          <button onClick={startSession} style={{ padding: '8px 16px', background: '#2563eb', color: '#fff', borderRadius: 8 }}>
+            Start Session
+          </button>
+        ) : (
+          <>
+            <button onClick={endSession} style={{ padding: '8px 16px', background: '#dc2626', color: '#fff', borderRadius: 8 }}>
+              End Session
+            </button>
+            <button onClick={toggleMute} style={{ padding: '8px 16px', borderRadius: 8 }}>
+              {isMuted ? 'Unmute Mic' : 'Mute Mic'}
+            </button>
+          </>
+        )}
+      </div>
+
+      <div style={{ height: 200, overflowY: 'auto', background: '#f8fafc', padding: 12, borderRadius: 8 }}>
+        {transcripts.map((t) => (
+          <div key={t.id} style={{ marginBottom: 6 }}>
+            <strong>{t.speaker}: </strong>
+            <span>{t.text}</span>
+          </div>
+        ))}
+      </div>
+    </div>
+  );
+}
+```
+
+---
+
+## Vanilla JavaScript / HTML Example
+
+```html
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="UTF-8">
+  <title>Ulai Voice Session</title>
+</head>
+<body>
+  <h2>Ulai Voice Client</h2>
+  <button id="connectBtn">Connect</button>
+  <button id="muteBtn" disabled>Mute</button>
+  <div id="transcriptBox" style="margin-top: 16px; border: 1px solid #ddd; height: 220px; overflow-y: auto; padding: 10px;"></div>
+
+  <audio id="remoteAudio" autoplay playsinline></audio>
+
+  <script type="module">
+    import { Room } from 'ulai-js-sdk';
+
+    let room = null;
+    const connectBtn = document.getElementById('connectBtn');
+    const muteBtn = document.getElementById('muteBtn');
+    const transcriptBox = document.getElementById('transcriptBox');
+    const remoteAudio = document.getElementById('remoteAudio');
+
+    connectBtn.addEventListener('click', async () => {
+      if (room) {
+        room.disconnect();
+        room = null;
+        connectBtn.textContent = 'Connect';
+        muteBtn.disabled = true;
+        return;
+      }
+
+      connectBtn.textContent = 'Connecting...';
+
+      // Obtain credentials from your backend API
+      const res = await fetch('/api/get-room-credentials');
+      const creds = await res.json();
+
+      room = new Room(creds.controlPlaneUrl || 'https://cp.ulai.co.in');
+
+      room.on('connected', () => {
+        connectBtn.textContent = 'Disconnect';
+        muteBtn.disabled = false;
+      });
+
+      room.on('trackSubscribed', (track, stream) => {
+        if (track.kind === 'audio') {
+          remoteAudio.srcObject = stream;
+        }
+      });
+
+      room.on('transcriptDelta', (delta) => {
+        const item = document.createElement('div');
+        item.textContent = `[${delta.speaker}]: ${delta.text}`;
+        transcriptBox.appendChild(item);
+        transcriptBox.scrollTop = transcriptBox.scrollHeight;
+      });
+
+      room.on('sessionTerminated', () => {
+        room.disconnect();
+        room = null;
+        connectBtn.textContent = 'Connect';
+        muteBtn.disabled = true;
+      });
+
+      await room.connect({
+        roomId: creds.roomId,
+        ticket: creds.ticket,
+        nodeAddress: creds.nodeAddress,
+        eventsToken: creds.eventsToken,
+      });
+
+      await room.publishAudio();
+    });
+
+    muteBtn.addEventListener('click', () => {
+      if (!room) return;
+      const next = !room.isMuted();
+      room.setMuted(next);
+      muteBtn.textContent = next ? 'Unmute' : 'Mute';
+    });
+  </script>
+</body>
+</html>
+```
+
+---
+
+## API Reference
+
+### `Room`
+
+Primary session manager. Extends `EventEmitter<RoomEvent>`.
+
+#### Constructor
+```typescript
+new Room(controlPlaneUrl: string)
+```
+- `controlPlaneUrl`: Base URL of the Ulai Control Plane (e.g. `'https://cp.ulai.co.in'`).
+
+#### Methods
+
+- **`connect(options: JoinRoomOptions): Promise<void>`**  
+  Connects to the SFU signaling WebSocket and the Control Plane event feed.
+- **`publishAudio(): Promise<MediaStream>`**  
+  Requests microphone access (`navigator.mediaDevices.getUserMedia`) with echo cancellation, noise suppression, and auto gain control, and attaches the audio track to the WebRTC connection.
+- **`setMuted(muted: boolean): void`**  
+  Mutes or unmutes the active microphone audio track. Emits `'muteChanged'`.
+- **`isMuted(): boolean`**  
+  Returns `true` if local audio is muted or inactive.
+- **`getLocalStream(): MediaStream | null`**  
+  Returns the active local `MediaStream`.
+- **`getRemoteStreams(): MediaStream[]`**  
+  Returns all active subscribed remote `MediaStream`s.
+- **`disconnect(): void`**  
+  Stops local microphone tracks, closes the WebRTC peer connection, disconnects signaling and event sockets, and cleans up resources.
+
+#### Properties
+- **`roomId: string | null`**: Current room/session ID.
+- **`participantId: string | null`**: Current participant ID.
+
+---
+
+### Events
+
+Register listeners using `room.on(eventName, handler)`.
+
+| Event | Parameters | Description |
+|---|---|---|
+| `connected` | `()` | WebRTC connection successfully established. |
+| `disconnected` | `()` | Connection closed or lost. |
+| `reconnecting` | `()` | WebRTC connection interrupted; attempting reconnection. |
+| `reconnected` | `()` | Reconnection established and media renegotiated. |
+| `trackSubscribed` | `(track: MediaStreamTrack, stream: MediaStream)` | Remote audio track available. Attach `stream` to an `<audio>` element. |
+| `trackUnsubscribed` | `(track: MediaStreamTrack, stream: MediaStream)` | Remote track ended or removed. |
+| `transcriptDelta` | `(delta: TranscriptDelta)` | Streaming transcript update for user or remote speaker. |
+| `sessionTerminated`| `(msg: any)` | Session ended by server/remote agent. SDK auto-disconnects. |
+| `muteChanged` | `(muted: boolean)` | Local microphone mute status changed. |
+| `participantJoined`| `(msg: any)` | A participant joined the session. |
+| `participantLeft` | `(msg: any)` | A participant left the session. |
+| `roster` | `(msg: RosterMessage)` | Current participant roster update. |
+| `error` | `(error: any)` | Signaling, WebRTC, or event socket error. |
+
+---
+
+### `UlaiClient`
+
+Utility class for directly requesting room join tickets and initializing a `Room`.
+
+```typescript
+const client = new UlaiClient({
+  url: 'https://cp.ulai.co.in',
+  apiKey: 'optional_api_key'
+});
+
+const room: Room = await client.joinRoom({
+  roomId: 'session-id',
+  participantId: 'user-01'
+});
+```
+
+---
+
+### TypeScript Interfaces
+
+```typescript
+export interface JoinRoomOptions {
+  roomId: string;
+  participantId?: string;
+  eventsToken?: string;
+  ticket?: string;
+  nodeAddress?: string;
+  iceServers?: RTCIceServer[];
+  autoSubscribeAudio?: boolean;
+}
+
+export interface TranscriptDelta {
+  segment_id: string;
+  speaker: 'user' | 'agent' | string;
+  text: string;
+  is_final: boolean;
+}
+
+export interface ConnectOptions {
+  url: string;
+  apiKey?: string;
+  token?: string;
+}
+
+export interface RosterMessage {
+  participants: any[];
+}
+```
+
+---
+
+## Handling Browser Audio & Autoplay Policies
+
+Modern web browsers restrict media playback until a user interaction occurs. Follow these rules to ensure smooth audio delivery:
+
+1. **Invoke from a User Gesture:** Always call `room.connect()` and `room.publishAudio()` inside a direct user event handler (e.g., clicking a "Start Call" button).
+2. **Audio Tag Attributes:** When attaching remote streams, ensure `<audio autoplay playsinline>` attributes are present.
+3. **Catch Playback Errors:** Handle rejected audio play promises:
+   ```typescript
+   audioElement.play().catch((err) => {
+     console.warn('Audio playback deferred by browser autoplay policy:', err);
+   });
+   ```
+
+---
+
+## Network & ICE Configuration
+
+Ulai Cloud WebRTC uses:
+- **Signaling & Events:** Secure WebSockets (`wss://`) over port `443`.
+- **Media (RTP/RTCP):** UDP media routing.
+
+For environments requiring custom STUN/TURN traversal, provide custom `iceServers` in `JoinRoomOptions`:
+
+```typescript
+await room.connect({
+  roomId: 'session-xyz',
+  ticket: 'ticket-token',
+  nodeAddress: 'sfu.ulai.co.in',
+  iceServers: [
+    { urls: 'stun:stun.l.google.com:19302' },
+    {
+      urls: 'turn:turn.yourcompany.com:3478',
+      username: 'turnuser',
+      credential: 'turnpassword',
+    },
+  ],
+});
+```
+
+---
+
+## License
+
+MIT © [Ulai Team](https://ulai.co.in)

package/dist/events.d.ts

@@ -2,7 +2,20 @@ import EventEmitter from 'eventemitter3';
 export declare class EventsClient extends EventEmitter {
     private ws;
     private url;
+    private sessionId;
+    private eventsToken;
+    private lastEventId;
+    private isClosedManually;
+    private retryAttempts;
+    private readonly maxRetries;
+    private reconnectTimer;
+    private heartbeatTimer;
     constructor(cpBase: string);
+    private buildUrl;
     connect(sessionId: string, eventsToken: string): void;
+    private sendSubscribe;
+    private startHeartbeat;
+    private stopHeartbeat;
+    private cleanupSocket;
     close(): void;
 }