@rqmii_immersive/agent-react-sdk 2.0.0 → 4.0.1

package/README.md

@@ -1,422 +1,168 @@
-# Rqmii React Agent SDK
+# @rqmii_immersive/agent-react-sdk
 
-Standalone React SDK for building realtime agent experiences in web apps.
+Standalone React SDK for Rqmii Agent realtime sessions. Version 4.0.0 — complete TypeScript rewrite with reducer-based state management and event emitter system.
 
-This SDK gives you a LiveKit-style integration model:
-
-- room wrapper component (`RqmiiAgentRoom`)
-- provider (`RqmiiAgentProvider`)
-- hooks usable by any child component (`useRqmiiConnection`, `useRqmiiMessages`, `useRqmiiVoice`, `useRqmiiAvatar`)
-
-## 1. Installation
+## Installation
 
 ```bash
 npm install @rqmii_immersive/agent-react-sdk
 ```
 
-If you are testing locally from this mono-repo:
+### Peer Dependencies
 
 ```bash
-npm install file:../RqmiiAgentReactSDK
+npm install react react-dom lucide-react
+# Optional (for 3D orb):
+npm install three @react-three/fiber @react-three/drei
 ```
 
-## 2. Required Configuration
-
-You must provide both URLs to the SDK:
+## Quick Start
 
-- `restUrl`: REST API base URL (for session init/join)
-- `wsUrl`: realtime Socket.IO URL
-
-Typical env values:
-
-```env
-VITE_AGENT_API_URL=https://api.example.com
-VITE_AGENT_WS_URL=wss://realtime.example.com
-```
-
-## 3. Import Surface
-
-From the package root:
-
-```js
-import {
-  RqmiiAgentProvider,
-  RqmiiAgentRoom,
-  RqmiiCloudAvatarPlayer,
-  EpicPixelStreamingPlayer,
-  useRqmiiAgent,
-  useRqmiiConnection,
-  useRqmiiMessages,
-  useRqmiiVoice,
-  useRqmiiAvatar,
-  useRqmiiAudioElement,
-  useRqmiiPressToTalk,
-  useRqmiiVAD,
-  AGENT_TYPES,
-  CONNECTION_STATE,
-  CLOUD_STREAM_STATE,
-  SOCKET_EVENTS,
-} from "@rqmii_immersive/agent-react-sdk";
-```
-
-If your bundler requires explicit style import, also import:
-
-```js
-import "@rqmii_immersive/agent-react-sdk/style.css";
-```
+### Core Setup
 
-## 4. Quick Start (Room Wrapper)
+```tsx
+import { RqmiiAgentProvider, useRqmiiConnection, useRqmiiMessages } from '@rqmii_immersive/agent-react-sdk';
 
-```jsx
-import {
-  RqmiiAgentRoom,
-  RqmiiCloudAvatarPlayer,
-  useRqmiiConnection,
-  useRqmiiMessages,
-  useRqmiiAvatar,
-} from "@rqmii_immersive/agent-react-sdk";
+function App() {
+  return (
+    <RqmiiAgentProvider wsUrl="wss://your-server" restUrl="https://your-api">
+      <Chat />
+    </RqmiiAgentProvider>
+  );
+}
 
-function ChatUI() {
+function Chat() {
+  const { initializeSession, isConnected } = useRqmiiConnection();
   const { messages, sendTextMessage } = useRqmiiMessages();
-  const { connectionState, isSessionReady, error } = useRqmiiConnection();
-  const { isAvatarSpeaking, cloudStreamState } = useRqmiiAvatar();
 
   return (
     <div>
-      <div>Conn: {connectionState}</div>
-      <div>Ready: {String(isSessionReady)}</div>
-      <div>Avatar Speaking: {String(isAvatarSpeaking)}</div>
-      <div>Cloud Stream: {cloudStreamState}</div>
-      {error && <div style={{ color: "red" }}>{error}</div>}
-
+      {messages.map(m => <div key={m.id}>{m.text}</div>)}
       <button onClick={() => sendTextMessage("Hello")}>Send</button>
-      <pre>{JSON.stringify(messages, null, 2)}</pre>
-
-      <RqmiiCloudAvatarPlayer muted={false} />
     </div>
   );
 }
+```
+
+### Using RqmiiAgentRoom (auto-connect)
+
+```tsx
+import { RqmiiAgentRoom, useRqmiiMessages } from '@rqmii_immersive/agent-react-sdk';
 
-export default function App() {
+function App() {
   return (
     <RqmiiAgentRoom
-      wsUrl={import.meta.env.VITE_AGENT_WS_URL}
-      restUrl={import.meta.env.VITE_AGENT_API_URL}
-      session={{ mode: "initialize", apiKey: "YOUR_API_KEY", language: "en" }}
-      connect
-      onError={(e) => console.error(e)}
-      onInitialized={(data) => console.log("session initialized", data)}
+      wsUrl="wss://your-server"
+      restUrl="https://your-api"
+      session={{ apiKey: 'your-key', language: 'en' }}
     >
-      <ChatUI />
+      <Chat />
     </RqmiiAgentRoom>
   );
 }
 ```
 
-## 5. Agent-Type Integration Flows
-
-### 5.1 Cloud Avatar
-
-Use `session.mode = 'initialize'` with `apiKey` (or `publicChatId` / `agentId`).
-
-- backend returns `streaming_url`
-- render `RqmiiCloudAvatarPlayer`
-- speaking state comes from avatar events, not audio chunks
-
-```jsx
-<RqmiiAgentRoom
-  wsUrl={WS_URL}
-  restUrl={REST_URL}
-  session={{ mode: "initialize", apiKey: API_KEY, language: "en" }}
-  connect
->
-  <RqmiiCloudAvatarPlayer muted={false} />
-</RqmiiAgentRoom>
-```
-
-### 5.2 Desktop Avatar (User Join)
-
-User joins existing desktop avatar session by `uniqueId`:
-
-```jsx
-<RqmiiAgentRoom
-  wsUrl={WS_URL}
-  restUrl={REST_URL}
-  session={{ mode: "join_desktop", uniqueId: "A1B2C3" }}
-  connect
->
-  <DesktopUI />
-</RqmiiAgentRoom>
-```
-
-### 5.3 Text Agent
-
-- initialize session
-- send text using `sendTextMessage`
-- read assistant chunks/final from `messages`
-
-### 5.4 Voice / Text-and-Voice Agent
-
-- send microphone chunks via `sendAudioChunk`
-- signal utterance end via `endAudioStream`
-- audio playback handled internally via `pcm-stream-player`
-
-## 6. Avatar State Model (Important)
-
-`isAvatarSpeaking` is server-event driven only:
-
-- starts on `avatar_speaking`
-- ends on `avatar_speaking_done`
-
-This means avatar speaking state is independent from PCM playback.
-
-For cloud avatar rendering lifecycle, `useRqmiiAvatar()` exposes:
-
-- `cloudStreamState`
-- `cloudStreamError`
-- `isCloudStreamConnected`
-- `isCloudStreamLoading`
-
-`cloudStreamState` values:
-
-- `idle`
-- `connecting`
-- `loading`
-- `connected`
-- `disconnected`
-- `error`
-
-## 7. Reconnection Model
-
-The backend uses one-time session JWTs, so default Socket.IO auto-reconnect with the same token is not enough.
-
-This SDK implements the coordinator reconnect contract:
+### Full Chat Widget
 
-1. connect using initial `session_token`
-2. server emits `reconnect_credential` with `{ sessionId, reconnectToken, role }`
-3. on disconnect, SDK reconnects with reconnect handshake auth
-4. on success, server emits `reconnect_success` and rotates token
+```tsx
+import { RqmiiAgentChat } from '@rqmii_immersive/agent-react-sdk/ui';
 
-Configurable with `reconnectOptions` on provider/room:
-
-```jsx
-<RqmiiAgentRoom
-  wsUrl={WS_URL}
-  restUrl={REST_URL}
-  reconnectOptions={{
-    enabled: true,
-    maxAttempts: 8,
-    initialDelayMs: 500,
-    maxDelayMs: 5000,
-    timeoutMs: 8000,
-  }}
-  session={{ mode: "initialize", apiKey: API_KEY }}
-  connect
->
-  <UI />
-</RqmiiAgentRoom>
-```
-
-## 8. Provider-Only Integration (No Room Wrapper)
-
-Use this when you need full manual control:
-
-```jsx
-import {
-  RqmiiAgentProvider,
-  useRqmiiConnection,
-} from "@rqmii_immersive/agent-react-sdk";
-
-function Controller() {
-  const { initializeSession, joinDesktopSession, disconnect } =
-    useRqmiiConnection();
-
-  return (
-    <div>
-      <button
-        onClick={() => initializeSession({ apiKey: API_KEY, language: "en" })}
-      >
-        Init
-      </button>
-      <button onClick={() => joinDesktopSession({ uniqueId: "A1B2C3" })}>
-        Join Desktop
-      </button>
-      <button onClick={disconnect}>Disconnect</button>
-    </div>
-  );
-}
-
-export default function App() {
+function App() {
   return (
-    <RqmiiAgentProvider wsUrl={WS_URL} restUrl={REST_URL}>
-      <Controller />
-    </RqmiiAgentProvider>
+    <RqmiiAgentChat
+      wsUrl="wss://your-server"
+      restUrl="https://your-api"
+      publicChatId="your-chat-id"
+    />
   );
 }
 ```
 
-## 9. Hook Reference
-
-### `useRqmiiConnection()`
-
-Returns:
-
-- `connectionState`
-- `isConnected`
-- `isSessionReady`
-- `error`
-- `reconnectAttempt`
-- `session`
-- `initializeSession(args)`
-- `joinDesktopSession(args)`
-- `disconnect()`
-
-### `useRqmiiMessages()`
-
-Returns:
-
-- `messages`
-- `sendTextMessage(text)`
-- `clearMessages()`
-
-Message stream behavior:
-
-- server chunks update streaming assistant message (`agent_response_chunk`)
-- final response completes that message (`agent_response`)
+## Architecture
 
-### `useRqmiiVoice()`
+### Entry Points
 
-Returns:
+- `@rqmii_immersive/agent-react-sdk` — Core: provider, hooks, types, constants
+- `@rqmii_immersive/agent-react-sdk/ui` — UI components: chat widget, avatar player, orb
+- `@rqmii_immersive/agent-react-sdk/style.css` — Chat widget styles
 
-- `sendAudioChunk(audio)`
-- `endAudioStream()`
-- `isAudioPlaying`
-- `transcript`
-- `partialTranscript`
-- `interrupt()`
-- `resumeAudioPlayback()`
-- `attachAudioElement(element)`
+### State Management
 
-### `useRqmiiAvatar()`
+State is managed via `useReducer` with a single `RqmiiState` shape. All mutations go through typed actions.
 
-Returns:
+### Event System
 
-- `streamingUrl`
-- `isAvatarSpeaking`
-- `speakingSource` (`'avatar_events'`)
-- `isCloudAvatar`
-- `isDesktopAvatar`
-- `cloudStreamState`
-- `cloudStreamError`
-- `isCloudStreamConnected`
-- `isCloudStreamLoading`
-- `setCloudStreamState(nextState, error?)`
+The provider exposes an `events` EventEmitter for subscribing to lifecycle events:
 
-### `useRqmiiAudioElement(audioRef)`
+```tsx
+import { useRqmiiAgent } from '@rqmii_immersive/agent-react-sdk';
 
-Optional helper to bind an audio element and improve autoplay unlock UX.
+function Component() {
+  const { events } = useRqmiiAgent();
 
-Note: voice playback does not require HTML audio element because SDK uses WebAudio (`pcm-stream-player`).
-
-### `useRqmiiPressToTalk(options?)`
-
-Built-in press-and-release voice input mode.
-
-- streams mic chunks to backend (`audio_chunk`)
-- sends `audio_end` automatically on release
-- exposes ready-to-spread button handlers via `eventHandlers`
-
-Minimal usage:
-
-```jsx
-const { eventHandlers, isRecording } = useRqmiiPressToTalk();
-<button {...eventHandlers}>
-  {isRecording ? "Recording..." : "Hold to Talk"}
-</button>;
-```
-
-### `useRqmiiVAD(options?)`
-
-Built-in VAD (voice activity detection) mode using `@ricky0123/vad-react`.
-
-- toggle VAD listening with `toggle()`
-- auto interrupts on speech start (configurable)
-- handles pre-speech padding and streaming chunks/end events
-
-Minimal usage:
-
-```jsx
-const { toggle, isListening, isSpeaking } = useRqmiiVAD();
+  useEffect(() => {
+    const unsub = events.on('session:ready', (payload) => {
+      console.log('Session ready', payload);
+    });
+    return unsub;
+  }, []);
+}
 ```
 
-## 10. Cloud Player Wrapper API
-
-`RqmiiCloudAvatarPlayer` props:
-
-- `className`
-- `style`
-- `autoConnect` (default `true`)
-- `muted` (default `false`)
-- `onConnected()`
-- `onDisconnected()`
-- `onError(error)`
-
-Internally it forwards cloud stream lifecycle to context state.
-
-## 11. Error Handling Strategy
-
-- fatal session/socket errors populate `error` in connection state
-- `TTS_ERROR` socket errors are treated as non-fatal by default
-- reconnect failures transition to `connectionState = failed`
-
-Recommended UI behavior:
-
-- show persistent banner for `error`
-- show reconnect indicator while `connectionState === reconnecting`
-- offer retry button that calls `initializeSession` again
-
-## 12. Production Checklist
-
-- use secure `https/wss` URLs in production
-- ensure CORS allows browser origins for REST and Socket.IO
-- pass correct API key/public chat id per environment
-- monitor reconnect failures and auth errors in logs
-- handle browser autoplay policies with user gesture before first audio
+Available events: `session:ready`, `session:joined`, `session:terminated`, `session:expired`, `message:received`, `message:sent`, `audio:playing`, `audio:stopped`, `avatar:speaking`, `avatar:speaking_done`, `state:changed`, `connection:connected`, `connection:disconnected`, `connection:reconnecting`, `connection:failed`, `error`, `credits:exhausted`, `transcript`, `partial_transcript`, `interrupted`.
 
-## 13. Troubleshooting
+## API Reference
 
-### Session initializes but socket fails auth
+### RqmiiAgentProvider Props
 
-Check:
+| Prop | Type | Default | Description |
+|------|------|---------|-------------|
+| `wsUrl` | `string` | required | WebSocket server URL |
+| `restUrl` | `string` | required | REST API base URL |
+| `reconnectOptions` | `RqmiiReconnectOptions` | `{ enabled: true, maxAttempts: 8, initialDelayMs: 500, maxDelayMs: 5000, timeoutMs: 8000 }` | Reconnection config |
+| `enableAudio` | `boolean` | `true` | Enable PCM audio playback |
+| `pcmOptions` | `RqmiiPCMOptions` | `{ encoding: '16bitInt', channels: 1, sampleRate: 16000 }` | PCM player config |
+| `debug` | `boolean` | `false` | Enable debug logging |
+| `withCredentials` | `boolean` | `false` | Include cookies in CORS requests |
+| `heartbeatIntervalSeconds` | `number` | `15` | Heartbeat interval |
+| `onSessionExpired` | `(payload?) => void` | — | Callback |
+| `onSessionTerminated` | `() => void` | — | Callback |
+| `onError` | `(payload?) => void` | — | Callback |
+| `onDisconnect` | `(reason?) => void` | — | Callback |
+| `onReconnectSuccess` | `(payload?) => void` | — | Callback |
+| `onReconnectFailed` | `(payload?) => void` | — | Callback |
 
-- `wsUrl` points to `RqmiiAgentBackendJs`
-- token is passed via SDK flow (do not manually override connect handshake)
-- backend clocks are synchronized for JWT expiry
+### Hooks
 
-### Cloud avatar not visible
+| Hook | Returns |
+|------|---------|
+| `useRqmiiAgent()` | Full context value |
+| `useRqmiiConnection()` | Connection state + session actions |
+| `useRqmiiSession()` | Session state booleans |
+| `useRqmiiMessages()` | Messages array + send/clear |
+| `useRqmiiVoice()` | Audio controls + transcript |
+| `useRqmiiAvatar()` | Avatar state + cloud stream |
+| `useRqmiiAudioElement(ref)` | Attaches audio element |
+| `useRqmiiPressToTalk(options)` | PTT recording |
+| `useRqmiiVAD(options)` | Voice Activity Detection |
+| `useRqmiiEvents(event, handler, deps)` | Event subscription |
 
-Check:
+## Breaking Changes from v3
 
-- `/sessions/init` response includes `streaming_url`
-- `RqmiiCloudAvatarPlayer` is rendered
-- network allows signaling/WebRTC paths
+1. **`useRqmiiSessionState` renamed to `useRqmiiSession`** — The hook is now `useRqmiiSession()`.
 
-### Avatar speaking UI not updating
+2. **Context value spread** — The context value now includes all state fields directly (no nested `state` object). Access `ctx.messages` instead of `ctx.state.messages`.
 
-Check:
+3. **Event emitter replaces some callbacks** — While legacy callbacks (`onSessionExpired`, etc.) still work, the preferred approach is using `events.on()`.
 
-- server emits `avatar_speaking` and `avatar_speaking_done`
-- your UI reads `isAvatarSpeaking` from `useRqmiiAvatar()`
+4. **TypeScript only** — All source is now `.ts`/`.tsx`. JavaScript consumers get full type inference via bundled `.d.ts` files.
 
-### Voice playback missing for text-and-voice
+5. **`terminateSession` return type simplified** — Now returns `Promise<{ success: boolean }>` instead of the previous shape with `skipped` and `error` fields.
 
-Check:
+6. **`useRqmiiVAD` return** — Added `isErrored` field to match the VAD library's state.
 
-- agent type is not avatar (`cloud_avatar`/`desktop_avatar`)
-- browser has user interaction to unlock audio context
-- `agent_audio_chunk` events are arriving from server
+7. **Internal module structure changed** — If you were importing from internal paths (e.g., `@rqmii_immersive/agent-react-sdk/react/core/...`), those paths no longer exist. Use the public exports only.
 
-## 14. Current Scope
+## License
 
-This SDK supports React, TypeScript (ships `.d.ts`), and Next.js client-side integration.
-Use it inside client components (`"use client"`), since realtime sockets/media APIs are browser runtime features.
+UNLICENSED