@graineai/inapp-react-native 0.12.0 → 0.16.0

package/INTEGRATION.md

@@ -8,6 +8,87 @@ Package: `@graineai/inapp-react-native`
 
 ---
 
+## How it works, end to end
+
+Two things travel between your app and the agent, and keeping them straight
+explains most of this document.
+
+**Media** is the customer's voice and the agent's. **Control** is everything the
+agent knows about your app: the screen, who is on it, what just happened, and
+what it is allowed to do about it. They are different problems — media wants
+pacing and loss tolerance, control wants ordered, reliable delivery — so they are
+carried differently, and the SDK is what hides that from you.
+
+```
+  YOUR APP                                    GRAINE
+  ─────────────────────────────────────       ──────────────────────────────
+
+  ┌───────────────────────────────────┐
+  │ your screens                      │
+  │   useGraineScreen(...)  ──────┐   │
+  │   useGraineAction(...)  ──────┤   │
+  │   identify() / trackEvent()   │   │
+  └───────────────────────────────┼───┘
+                                  │  one client, one source of truth
+                                  ▼
+                     ┌─────────────────────────┐
+                     │  GraineInAppClient      │
+                     │  screen · identity      │
+                     │  events · actions       │
+                     │  buildContextFrame()    │
+                     │  maskDeep() ← PII never │
+                     │    leaves the device    │
+                     └───────────┬─────────────┘
+                                 │
+                     ┌───────────▼─────────────┐
+                     │  GraineVoiceLauncher    │
+                     │  mic · lifecycle · relay│
+                     └───────────┬─────────────┘
+                                 │  postMessage
+                     ┌───────────▼─────────────┐
+                     │  hidden WebView (1×1)   │           ┌──────────────┐
+                     │  the audio engine       │══ media ══▶│  the agent   │
+                     │  capture · playback     │           │              │
+                     │  jitter buffer · barge-in│──control ─▶│  runtime     │
+                     └─────────────────────────┘           └──────┬───────┘
+                                                                  │
+                                 ▲                                │
+                                 └──────── app_action ────────────┘
+                                    "open the offers screen"
+                                    (the agent BLOCKS on your answer)
+```
+
+**Why a WebView.** React Native has no `getUserMedia`. The alternative is a
+native audio module, which means a store release for every audio fix — and the
+audio path is exactly where the fixes are. The page is one pixel, invisible and
+unreachable; your customer only ever sees your own UI. See *Why there is no
+LiveKit step* at the end.
+
+### What travels, and when
+
+| Frame | Direction | When |
+|---|---|---|
+| `app_context` | app → agent | Every screen change and field edit, throttled. Masked first. |
+| `app_action` | agent → app | The agent wants to do something. **It blocks on your reply** (~6s). |
+| `app_action_result` | app → agent | Your handler's answer: `ok`, `refused`, or `error`. |
+| `app_event` | app → agent | Something happened that may be worth speaking about unprompted. |
+
+### The one rule that matters
+
+**One client.** `GraineProvider` builds a client unless you hand it one. If your
+app already has a module-level instance — and any app of size does, because
+`identify()` and `trackEvent()` are not all called from React — pass it in:
+
+```tsx
+<GraineProvider client={myClient} autoConnect={false}>
+```
+
+Two clients is the failure that looks like a broken product: your screens write
+the context into one, the launcher reads the other and finds nothing, and the
+agent connects, greets, and says it cannot see the screen.
+
+---
+
 ## What you need
 
 | | |
@@ -89,12 +170,18 @@ export default function App() {
       baseUrl="https://www.graine.ai"
       publishableKey="pk_live_..."
       navigationRef={navigationRef}
+      // The launcher owns the connection — see below.
+      autoConnect={false}
     >
       <NavigationContainer ref={navigationRef}>
         <RootNavigator />
       </NavigationContainer>
 
-      <GraineVoiceLauncher webView={WebView} autoStart>
+      <GraineVoiceLauncher
+        webView={WebView}
+        autoStart
+        onError={(m) => console.warn("[graine]", m)}
+      >
         {() => <GraineAgentBar />}
       </GraineVoiceLauncher>
     </GraineProvider>
@@ -102,6 +189,16 @@ export default function App() {
 }
 ```
 
+**`autoConnect={false}` is not optional here.** The provider opens a connection
+of its own by default, and `GraineVoiceLauncher` opens the one that carries the
+call. Leave both on and the agent answers twice — the customer hears two
+greetings over each other, and you are billed for two conversations. Drop the
+launcher and use text only, and `autoConnect` goes back to its default.
+
+**Pass `onError`.** A rejected key or an agent that has not been enabled for apps
+leaves the launcher rendering nothing at all, forever, with nothing in the
+console. This is the only way to find out.
+
 `navigationRef` is the line that matters. With it the SDK reads the current route
 itself, so the agent knows which screen the customer is on without you calling
 anything per screen — including the screen you add next year when nobody
@@ -135,6 +232,63 @@ agent just does not know their name.
 
 ---
 
+### Metadata before the first word
+
+Two places, and the difference is when you know the value.
+
+```tsx
+// Known at build time — carried in the init frame, before the agent speaks.
+<GraineProvider variables={{ tier: 'gold', region: 'IN' }} />
+
+// Known after sign-in — merged into that same init frame if it lands before the
+// socket opens, sent on the next context frame if it lands after.
+identify({ name: user.name, plan: user.plan });
+```
+
+`identify()` wins over `variables` on the same key: a signed-in customer is more
+specific than a default. Both are masked on the device before they leave it.
+
+### It reaches the call record, not only the prompt
+
+Traits are stored on the conversation as `app_user`, and they are what labels
+the row in Call History. **A web call has no phone number**, so without this
+every in-app conversation shows a bare session id — a page of uuids.
+
+Read from the LIVE identity rather than the init frame, so a customer who signs
+in halfway through is identified from that moment rather than from a frame that
+predates them.
+
+#### Tag the session with whatever you file people under
+
+You do not have to have a `name`. Call History looks for a label in this order:
+
+`name` → `full_name` / `fullName` → `display_name` / `displayName` → `label` →
+`username` / `user_name` → `customer_name` → `company` → `email` → **any other
+string trait**
+
+So an app that files people by policy number, customer id or internal handle is
+still legible:
+
+```tsx
+identify({ label: `Policy ${policy.number}` });   // → "Policy HDFC-4471"
+identify({ customer_name: acct.holder });          // → "R. Bhanot"
+identify({ crm_id: lead.id });                     // → falls through to the id
+```
+
+Two rules worth knowing:
+
+- **Redacted values are skipped.** `phone` and `email` are masked on the device,
+  so a row never renders `[PHONE]` as somebody's name. Send the phone anyway if
+  it is useful — the agent is told a number is on file without being told the
+  digits — just do not rely on it as the label.
+- **Identity alone is enough.** You do not need `useGraineScreen` for this. An
+  app that only calls `identify()` still tags every conversation.
+
+`identify(null)` on sign-out. An agent still greeting the previous account by
+name after a switch is worse than one that never knew it.
+
+---
+
 ## Step 3 — Report what is ON the screen
 
 The route gives the agent a screen *name*. This gives it the contents.
@@ -218,6 +372,23 @@ updating the screen.
 
 ---
 
+### Declaring them: paste, do not retype
+
+Keep the list in your repo — `docs/agent-actions.json` is the convention — and
+load it with **Embed & Widgets → App actions → Import from JSON**. A bare array
+works, so does `{ "appActions": [...] }`.
+
+Names must match your `useGraineAction` handlers exactly, and that is the whole
+argument for importing: a name one underscore out declares a tool your app will
+refuse for the life of the release, and the agent keeps trying it. `enum`
+constraints are preserved, so a value the model cannot get wrong stays that way.
+
+Import replaces the panel's contents rather than merging — the file is the
+source of truth, and a merge leaves actions declared here and implemented
+nowhere. Save is what pushes the catalogue to the agent.
+
+---
+
 ## Step 5 — Product events, without interrupting
 
 ```tsx
@@ -235,6 +406,30 @@ and the runtime decides whether it is worth interrupting for.
 
 ---
 
+### When something should be spoken about
+
+```tsx
+const { client } = useGraineAgent();
+client.reportEvent({
+  name: 'mandate_failed',
+  detail: 'Their mandate was declined twice. Offer to switch to a card.',
+});
+```
+
+`track()` is "know this when they next ask". `reportEvent()` is "this may be
+worth interrupting for". `detail` tells the agent what to OFFER — without it the
+agent reads an event name aloud, which is worse than saying nothing.
+
+Reserve it for a dead end the customer would want addressed without asking: a
+save that never reached the server, an upload that failed twice, a payment
+declined. Not a tap, not a navigation, not a save that worked.
+
+The runtime still decides whether to speak: silent while it is talking, while a
+reply is generating, for 20s after the last intervention, and after three in a
+session.
+
+---
+
 ## Step 6 — React to the conversation
 
 ```tsx
@@ -406,6 +601,12 @@ than opening a second socket. This matters under React 18 strict mode, which
 mounts effects twice: previously the first socket was orphaned with no reference
 to close it, and the runtime kept it open and billing until its own timeout.
 
+**Nothing is left holding the call.** An outstanding action's waiter is released
+in a `finally`, so a customer closing the app mid-action releases it too, and
+every wait has a deadline — an app that never answers cannot hold the turn open
+against someone who has already gone. A timeout is reported rather than
+swallowed, because silence would have the agent claim the change landed.
+
 **`close()` sends a stop frame** before dropping the socket, so the conversation
 is filed as ended rather than as a customer who vanished mid-turn. It is called
 for you on unmount and on backgrounding.
@@ -445,6 +646,29 @@ Echo. The SDK suppresses the agent's own voice from the microphone, but a very
 loud speaker defeats a level-based guard. Lower the volume, or use earphones to
 confirm that is what it is.
 
+**Nothing at all happens, and `onError` never fires either.**
+You did not pass `onError`. Without it a failed session — a rejected key, an
+agent not enabled for apps — leaves the launcher rendering nothing, silently.
+Pass it before you debug anything else.
+
+**On `rtc`: every call reports a fallback to `ws`.**
+`onTransport` gives a `reason`. `unreachable` means WebRTC could not be reached
+at all — usually no SIP realm on the org, or the SBC is not reachable from the
+customer's network. `screen_channel` means the audio connected but the control
+socket did not join, so the call was restarted on the WebSocket rather than run
+blind. Both are org configuration, not integration code.
+
+**On `rtc`: the greeting plays twice, occasionally.**
+That is the `screen_channel` fallback restarting the call. It only happens when
+the control channel fails to join within five seconds, and it is deliberate: an
+in-app agent that cannot see the screen is not a lesser version of the product.
+If it happens on every call, fix the control channel rather than living with it.
+
+**Audio is muffled or crackly on Android but fine on iOS.**
+Update the SDK. Versions before 0.15.0 asked the platform for the stream's own
+sample rate; iOS refuses and falls back to the device rate, Android grants it
+and resamples in its output stage, which is where the artefacts came from.
+
 ---
 
 ## Reference
@@ -459,12 +683,68 @@ confirm that is what it is.
 | `useGraineEvents()` | conversation lifecycle |
 | `useGraineVoice()` | only if you bring your own native audio instead of the WebView |
 
-`GraineProvider`: `baseUrl`, `publishableKey`, `navigationRef`, `includeScreens`,
-`launcherDelayMs`, `visibility`, `autoConnect`, `onProactive`.
+### `GraineProvider`
 
-`GraineVoiceLauncher`: `webView` (required), `autoStart`, `onCaption`,
-`onCallState`, and a child function receiving `{ connected, connecting, muted,
-start, end, setMuted }`.
+| Prop | Type | Notes |
+|---|---|---|
+| `publishableKey` | `string` | Required unless you pass `client`. |
+| `baseUrl` | `string` | Defaults to Graine's host. |
+| `client` | `GraineInAppClient` | Use **your** instance instead of building one. Pass this if your app has a module-level client. |
+| `navigationRef` | `ref` | The same ref you give `NavigationContainer`. Makes the agent screen-aware with no per-screen code. |
+| `includeScreens` | `string[]` | Route names where the launcher may appear. Omit for everywhere. |
+| `launcherDelayMs` | `number` | Delay before the launcher appears on an eligible screen. |
+| `visibility` | `LauncherVisibility` | Groups, continuity, per-group delays and insets. See *Controlling where the launcher appears*. |
+| `autoConnect` | `boolean` | Default `true`. Set `false` when `GraineVoiceLauncher` owns the connection — see below. |
+| `onProactive` | `(text) => void` | The agent spoke first; draw your own nudge. |
+| `endOnBackground` | `boolean` | Default `true`. You want it on. |
+
+### `GraineVoiceLauncher`
+
+| Prop | Type | Notes |
+|---|---|---|
+| `webView` | `ComponentType` | **Required.** Pass `react-native-webview`'s `WebView`. Passed in rather than imported so the package installs without it. |
+| `autoStart` | `boolean` | Start talking as soon as the page is ready. |
+| `transport` | `"ws" \| "rtc"` | Which transport carries **audio**. See below. |
+| `onTransport` | `({transport, requested}) => void` | Which transport actually carried the call, including after a fallback. |
+| `onCaption` | `({role, text, live}) => void` | Each caption line, for your own UI. |
+| `onCallState` | `({connected, connecting}) => void` | Connection state. |
+| `onEnded` | `() => void` | The call is over. Fires once. Give the goodbye ~400ms to play before you unmount. |
+| `onMicDenied` | `(reason) => void` | `NotAllowedError` is a refusal; `NotFoundError` is a device with no usable microphone. Different sentences for a customer. |
+| `onError` | `(message) => void` | Voice could not be set up at all — key rejected, agent not enabled for apps, no network. Without this the launcher renders nothing and you cannot tell why. |
+| `children` | `(api) => ReactNode` | Receives `{ connected, connecting, muted, start, end, setMuted }`. |
+
+**The launcher owns the connection.** It asks for the microphone, resolves the
+session, holds the socket and relays your screen, actions and events onto it. So
+set `autoConnect={false}` on the provider when you use it: two live connections
+on one agent is two conversations, and the customer hears both greetings.
+
+### `transport` — how audio travels
+
+**Omit it and the dashboard decides.** Agent → Embed & Widgets → *Call audio*
+sets this for every app and site running the agent, so the transport can be
+changed without shipping a release. Passing the prop overrides it.
+
+`ws` (the default) sends raw PCM on the same socket that carries screen context
+and in-app actions.
+
+`rtc` hands media to jambonz's SBC — Opus, DTLS-SRTP, an adaptive jitter buffer,
+and the platform's own echo canceller running below the microphone with the
+playout signal as its reference. Better audio on a bad network.
+
+**`rtc` carries the screen too.** The browser holds two connections — media to
+the SBC, and a control socket for `app_context`, `app_action` and `app_event` —
+and the server joins them by a session id the browser puts on the INVITE. So an
+in-app agent on `rtc` sees exactly what it sees on `ws`. Frames sent before the
+join completes are queued rather than dropped, because the first screen frame
+almost always arrives while the phone is still ringing, and it is the one the
+agent's opening line is composed from.
+
+`rtc` **falls back.** WebRTC has more ways to be unavailable than a WebSocket —
+no SIP realm on the org, an unreachable SBC, a network that eats UDP — and each
+is a reason to carry the call differently rather than refuse it. The page
+reconnects on the WebSocket and tells you through `onTransport`, so a permanent
+misconfiguration is visible instead of being merely audible. A refused
+microphone is **not** a transport problem and does not fall back.
 
 ---
 

package/README.md

@@ -114,6 +114,45 @@ the outcome through the next `useGraineScreen` update.
 
 ## Voice
 
+The short way: no native audio module, no rebuild, and audio fixes reach you
+over the air.
+
+```tsx
+import { WebView } from "react-native-webview";
+import { GraineProvider, GraineVoiceLauncher, GraineAgentBar } from "@graineai/inapp-react-native";
+
+<GraineProvider
+  baseUrl="…"
+  publishableKey="pk_live_…"
+  navigationRef={navigationRef}
+  autoConnect={false}          // the launcher owns the connection
+>
+  <RootNavigator />
+  <GraineVoiceLauncher
+    webView={WebView}
+    autoStart
+    onError={(m) => console.warn("[graine]", m)}
+  >
+    {() => <GraineAgentBar />}
+  </GraineVoiceLauncher>
+</GraineProvider>
+```
+
+`react-native-webview` is the only peer dependency and it autolinks. The
+microphone and speaker live in a hidden one-pixel WebView — your customer never
+sees a web page. Set `autoConnect={false}`: leave the provider connecting too
+and the agent answers twice, so the customer hears two greetings and you are
+billed for two conversations.
+
+Audio travels on a WebSocket by default. Set **Embed & Widgets → Call audio** to
+*High quality* for WebRTC — Opus, a jitter buffer and the device's own echo
+canceller — which carries screen context just the same and falls back on its own
+if it cannot connect. See `INTEGRATION.md`.
+
+### Bringing your own audio
+
+Only if you already have a native audio module and want it used instead.
+
 ```tsx
 <GraineProvider baseUrl="…" publishableKey="pk_live_…" voice>
 ```

package/dist/audio.js

@@ -60,7 +60,8 @@ export function decodeAgentAudio(bytes, declared, latched, fallbackPcmRate = 160
     const pcm16 = muLawWins ? asMuLaw : asPcm;
     const kind = muLawWins ? "mulaw" : "pcm";
     const sampleRate = muLawWins ? 8000 : declared?.sampleRate || fallbackPcmRate;
-    const confident = rms(pcm16) >= CONFIDENT_RMS;
+    const confident = rms(asPcm) >= CONFIDENT_RMS &&
+        zeroCrossingRate(asMuLaw) !== zeroCrossingRate(asPcm);
     return { pcm16, sampleRate, kind, latch: confident ? { kind, sampleRate } : null };
 }
 const B64 = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";

package/dist/client.d.ts

@@ -1,4 +1,4 @@
-import { type ActionHandler, type AppActionResult, type AppEvent, type ScreenContext } from "./protocol";
+import { type ActionHandler, type AppActionResult, type AppEvent, type ScreenContext } from "./protocol.js";
 export declare const SUBPROTOCOL = "graine.embed.v1";
 export interface GraineInAppOptions {
     baseUrl: string;
@@ -53,7 +53,14 @@ export declare class GraineInAppClient {
     executeAction(name: string, args?: Record<string, unknown>): Promise<AppActionResult>;
     private runAction;
     private invokeAction;
+    get isConnected(): boolean;
     getScreen(): ScreenContext | null;
+    getIdentity(): Record<string, string>;
+    getRecentEvents(): Array<{
+        name: string;
+        at: number;
+        data?: Record<string, unknown>;
+    }>;
     getAvailableActions(): string[];
     registerAction(name: string, handler: ActionHandler): () => void;
     get availableActions(): string[];
@@ -61,6 +68,12 @@ export declare class GraineInAppClient {
     reportEvent(event: AppEvent): void;
     track(name: string, data?: Record<string, unknown>): void;
     private scheduleScreen;
+    buildContextFrame(): {
+        type: "app_context";
+        context: Record<string, unknown>;
+        available_actions: string[];
+        seq: number;
+    } | null;
     private flushScreen;
     private armStall;
     identify(user: Record<string, unknown> | null): void;

package/dist/client.js

@@ -1,5 +1,5 @@
-import { ACTION_DEADLINE_MS, CONTEXT_THROTTLE_MS, } from "./protocol";
-import { maskDeep } from "./mask";
+import { ACTION_DEADLINE_MS, CONTEXT_THROTTLE_MS, } from "./protocol.js";
+import { maskDeep } from "./mask.js";
 export const SUBPROTOCOL = "graine.embed.v1";
 const PING_MS = 25000;
 export class GraineInAppClient {
@@ -259,9 +259,18 @@ export class GraineInAppClient {
             };
         }
     }
+    get isConnected() {
+        return !!this.ws && this.ws.readyState === 1;
+    }
     getScreen() {
         return this.screen;
     }
+    getIdentity() {
+        return { ...this.identity };
+    }
+    getRecentEvents() {
+        return [...this.recentEvents];
+    }
     getAvailableActions() {
         return [...this.actions.keys()];
     }
@@ -287,6 +296,7 @@ export class GraineInAppClient {
         this.scheduleScreen();
     }
     reportEvent(event) {
+        this.emit("app_event", event);
         this.send({ type: "app_event", event });
     }
     track(name, data) {
@@ -299,6 +309,7 @@ export class GraineInAppClient {
             this.scheduleScreen();
     }
     scheduleScreen() {
+        this.emit("screen", { screen: this.screen, actions: this.getAvailableActions() });
         if (this.contextTimer) {
             this.pendingScreen = true;
             return;
@@ -313,21 +324,28 @@ export class GraineInAppClient {
             }
         }, CONTEXT_THROTTLE_MS);
     }
-    flushScreen() {
-        if (!this.screen)
-            return false;
-        const { availableActions, ...rest } = this.screen;
-        return this.send({
+    buildContextFrame() {
+        const hasIdentity = Object.keys(this.identity).length > 0;
+        if (!this.screen && !hasIdentity && this.recentEvents.length === 0)
+            return null;
+        const { availableActions, ...rest } = (this.screen ?? {});
+        return {
             type: "app_context",
             context: maskDeep({
                 ...rest,
-                idle_ms: Date.now() - this.screenSince,
+                ...(this.screen ? { idle_ms: Date.now() - this.screenSince } : {}),
                 ...(Object.keys(this.identity).length ? { user: this.identity } : {}),
                 ...(this.recentEvents.length ? { recent_events: this.recentEvents } : {}),
             }),
             available_actions: (availableActions ?? this.availableActions).filter((a) => this.actions.has(a)),
             seq: ++this.screenSeq,
-        });
+        };
+    }
+    flushScreen() {
+        const frame = this.buildContextFrame();
+        if (!frame)
+            return false;
+        return this.send(frame);
     }
     armStall() {
         clearTimeout(this.stallTimer);
@@ -362,8 +380,7 @@ export class GraineInAppClient {
             flat[k] = typeof v === "string" ? v : JSON.stringify(v);
         }
         this.identity = maskDeep(flat);
-        if (this.screen)
-            this.scheduleScreen();
+        this.scheduleScreen();
     }
     noteInteraction() {
         this.screenSince = Date.now();

package/dist/index.d.ts

@@ -1,9 +1,10 @@
-export type { ActionHandler, AppActionRequest, AppActionResult, AppEvent, ScreenContext, ScreenField, } from "./protocol";
-export { GraineInAppClient, type GraineInAppOptions, type SessionConfig } from "./client";
-export { VoiceSession, liveAudioStreamAdapter, base64ToPcm16, CAPTURE_SAMPLE_RATE, type AudioAdapter, type VoiceSessionOptions, } from "./voice";
-export { EchoGuard, decodeAgentAudio, decodeMuLaw, decodePcm16, base64ToBytes, type AudioFormat, type DecodedAudio, } from "./audio";
-export { addMaskRule, maskDeep, maskString } from "./mask";
-export { GraineProvider, useGraineAgent, useGraineScreen, useGraineAction, useGraineVoice, useGraineIdentify, useGraineEvents, useGraineTrack, useGraineTap, type GraineEvent, type GraineProviderProps, type Turn, type Caption, } from "./react-native/index";
-export { GraineAgentBar, GraineLauncher, type GraineAgentBarProps, type GraineLauncherProps, type GraineBarTheme, } from "./react-native/ui";
-export { GraineVoiceLauncher, type GraineVoiceLauncherProps, type GraineVoiceApi, } from "./react-native/voice-launcher";
-export { LauncherVisibilityTracker, activeRouteName, type LauncherInset, type LauncherContinuity, type LauncherDelayPolicy, type LauncherGroup, type LauncherVisibility, type VisibilityDecision, } from "./react-native/navigation";
+export type { ActionHandler, AppActionRequest, AppActionResult, AppEvent, ScreenContext, ScreenField, } from "./protocol.js";
+export { GraineInAppClient, type GraineInAppOptions, type SessionConfig } from "./client.js";
+export { VoiceSession, liveAudioStreamAdapter, base64ToPcm16, CAPTURE_SAMPLE_RATE, type AudioAdapter, type VoiceSessionOptions, } from "./voice.js";
+export { EchoGuard, decodeAgentAudio, decodeMuLaw, decodePcm16, base64ToBytes, type AudioFormat, type DecodedAudio, } from "./audio.js";
+export { addMaskRule, maskDeep, maskString } from "./mask.js";
+export { GraineProvider, useGraineAgent, useGraineScreen, useGraineAction, useGraineVoice, useGraineIdentify, useGraineEvents, useGraineTrack, useGraineTap, type GraineEvent, type GraineProviderProps, type Turn, type Caption, } from "./react-native/index.js";
+export { RtcVoiceSessionController, RtcVoiceError, type RtcVoiceState, type RtcVoiceErrorCode, type RtcVoiceSession, type RtcVoiceOptions, } from "./react-native/voice-rtc.js";
+export { GraineAgentBar, GraineLauncher, type GraineAgentBarProps, type GraineLauncherProps, type GraineBarTheme, } from "./react-native/ui.js";
+export { GraineVoiceLauncher, type GraineVoiceLauncherProps, type GraineVoiceApi, } from "./react-native/voice-launcher.js";
+export { LauncherVisibilityTracker, activeRouteName, type LauncherInset, type LauncherContinuity, type LauncherDelayPolicy, type LauncherGroup, type LauncherVisibility, type VisibilityDecision, } from "./react-native/navigation.js";

package/dist/index.js

@@ -1,8 +1,9 @@
-export { GraineInAppClient } from "./client";
-export { VoiceSession, liveAudioStreamAdapter, base64ToPcm16, CAPTURE_SAMPLE_RATE, } from "./voice";
-export { EchoGuard, decodeAgentAudio, decodeMuLaw, decodePcm16, base64ToBytes, } from "./audio";
-export { addMaskRule, maskDeep, maskString } from "./mask";
-export { GraineProvider, useGraineAgent, useGraineScreen, useGraineAction, useGraineVoice, useGraineIdentify, useGraineEvents, useGraineTrack, useGraineTap, } from "./react-native/index";
-export { GraineAgentBar, GraineLauncher, } from "./react-native/ui";
-export { GraineVoiceLauncher, } from "./react-native/voice-launcher";
-export { LauncherVisibilityTracker, activeRouteName, } from "./react-native/navigation";
+export { GraineInAppClient } from "./client.js";
+export { VoiceSession, liveAudioStreamAdapter, base64ToPcm16, CAPTURE_SAMPLE_RATE, } from "./voice.js";
+export { EchoGuard, decodeAgentAudio, decodeMuLaw, decodePcm16, base64ToBytes, } from "./audio.js";
+export { addMaskRule, maskDeep, maskString } from "./mask.js";
+export { GraineProvider, useGraineAgent, useGraineScreen, useGraineAction, useGraineVoice, useGraineIdentify, useGraineEvents, useGraineTrack, useGraineTap, } from "./react-native/index.js";
+export { RtcVoiceSessionController, RtcVoiceError, } from "./react-native/voice-rtc.js";
+export { GraineAgentBar, GraineLauncher, } from "./react-native/ui.js";
+export { GraineVoiceLauncher, } from "./react-native/voice-launcher.js";
+export { LauncherVisibilityTracker, activeRouteName, } from "./react-native/navigation.js";

package/dist/react-native/index.d.ts

@@ -1,8 +1,8 @@
 import React from "react";
-import { GraineInAppClient, type GraineInAppOptions } from "../client";
-import { type AudioAdapter, type VoiceSessionOptions } from "../voice";
-import type { ActionHandler, ScreenContext } from "../protocol";
-import { type LauncherInset, type LauncherVisibility } from "./navigation";
+import { GraineInAppClient, type GraineInAppOptions } from "../client.js";
+import { type AudioAdapter, type VoiceSessionOptions } from "../voice.js";
+import type { ActionHandler, ScreenContext } from "../protocol.js";
+import { type LauncherInset, type LauncherVisibility } from "./navigation.js";
 interface GraineContextValue {
     client: GraineInAppClient;
     connected: boolean;
@@ -60,8 +60,9 @@ export interface Turn {
     text: string;
     live?: boolean;
 }
-export interface GraineProviderProps extends GraineInAppOptions {
+export interface GraineProviderProps extends Partial<GraineInAppOptions> {
     children: React.ReactNode;
+    client?: GraineInAppClient;
     autoConnect?: boolean;
     onProactive?: (text: string) => void;
     endOnBackground?: boolean;
@@ -72,7 +73,7 @@ export interface GraineProviderProps extends GraineInAppOptions {
     launcherDelayMs?: number;
     visibility?: LauncherVisibility;
 }
-export declare function GraineProvider({ children, autoConnect, onProactive, endOnBackground, navigationRef, includeScreens, launcherDelayMs, visibility, ...options }: GraineProviderProps): React.JSX.Element;
+export declare function GraineProvider({ children, autoConnect, onProactive, endOnBackground, navigationRef, includeScreens, launcherDelayMs, visibility, client: providedClient, ...options }: GraineProviderProps): React.JSX.Element;
 export declare function useGraineAgent(): GraineContextValue;
 export declare function useGraineEvents(handler: (e: GraineEvent) => void): void;
 export declare function useGraineTrack(): (name: string, data?: Record<string, unknown>) => void;

package/dist/react-native/index.js

@@ -1,14 +1,16 @@
 import { jsx as _jsx } from "react/jsx-runtime";
 import { createContext, useCallback, useContext, useEffect, useMemo, useRef, useState, } from "react";
 import { AppState } from "react-native";
-import { GraineInAppClient } from "../client";
-import { VoiceSession } from "../voice";
-import { LauncherVisibilityTracker, activeRouteName, } from "./navigation";
+import { GraineInAppClient } from "../client.js";
+import { VoiceSession } from "../voice.js";
+import { LauncherVisibilityTracker, activeRouteName, } from "./navigation.js";
 const Ctx = createContext(null);
-export function GraineProvider({ children, autoConnect = true, onProactive, endOnBackground = true, navigationRef, includeScreens, launcherDelayMs = 0, visibility, ...options }) {
+export function GraineProvider({ children, autoConnect = true, onProactive, endOnBackground = true, navigationRef, includeScreens, launcherDelayMs = 0, visibility, client: providedClient, ...options }) {
     const clientRef = useRef(null);
-    if (!clientRef.current)
-        clientRef.current = new GraineInAppClient(options);
+    if (!clientRef.current) {
+        clientRef.current =
+            providedClient ?? new GraineInAppClient(options);
+    }
     const client = clientRef.current;
     const [connected, setConnected] = useState(false);
     const [connecting, setConnecting] = useState(false);

package/dist/react-native/ui.js

@@ -1,7 +1,7 @@
 import { jsx as _jsx, jsxs as _jsxs } from "react/jsx-runtime";
 import { useCallback, useEffect, useMemo, useRef, useState } from "react";
 import { ActivityIndicator, Image, KeyboardAvoidingView, Platform, Pressable, ScrollView, StyleSheet, Text, TextInput, useColorScheme, View, } from "react-native";
-import { useGraineAgent } from "./index";
+import { useGraineAgent } from "./index.js";
 const DARK = {
     surface: "#17161A",
     border: "rgba(255,255,255,0.09)",

package/dist/react-native/voice-launcher.d.ts

@@ -1,5 +1,10 @@
 import React from "react";
 export interface GraineVoiceLauncherProps {
+    transport?: "ws" | "rtc";
+    onTransport?: (info: {
+        transport: "ws" | "rtc";
+        requested: "ws" | "rtc";
+    }) => void;
     webView: React.ComponentType<any>;
     autoStart?: boolean;
     onCaption?: (c: {
@@ -13,6 +18,7 @@ export interface GraineVoiceLauncherProps {
     }) => void;
     onEnded?: () => void;
     onMicDenied?: (reason: string) => void;
+    onError?: (message: string) => void;
     children?: (api: GraineVoiceApi) => React.ReactNode;
 }
 export interface GraineVoiceApi {
@@ -23,4 +29,4 @@ export interface GraineVoiceApi {
     end: () => void;
     setMuted: (m: boolean) => void;
 }
-export declare function GraineVoiceLauncher({ webView: WebView, autoStart, onCaption, onCallState, onMicDenied, onEnded, children, }: GraineVoiceLauncherProps): React.JSX.Element;
+export declare function GraineVoiceLauncher({ webView: WebView, autoStart, transport, onTransport, onCaption, onCallState, onMicDenied, onError, onEnded, children, }: GraineVoiceLauncherProps): React.JSX.Element;

package/dist/react-native/voice-launcher.js

@@ -1,10 +1,11 @@
 import { Fragment as _Fragment, jsx as _jsx, jsxs as _jsxs } from "react/jsx-runtime";
 import { useCallback, useEffect, useRef, useState } from "react";
-import { AppState, View } from "react-native";
-import { useGraineAgent } from "./index";
-export function GraineVoiceLauncher({ webView: WebView, autoStart = false, onCaption, onCallState, onMicDenied, onEnded, children, }) {
+import { AppState, PermissionsAndroid, Platform, View } from "react-native";
+import { useGraineAgent } from "./index.js";
+export function GraineVoiceLauncher({ webView: WebView, autoStart = false, transport, onTransport, onCaption, onCallState, onMicDenied, onError, onEnded, children, }) {
     const { client, appearance } = useGraineAgent();
     const ref = useRef(null);
+    const startedRef = useRef(false);
     const [connected, setConnected] = useState(false);
     const [connecting, setConnecting] = useState(false);
     const [muted, setMutedState] = useState(false);
@@ -12,22 +13,86 @@ export function GraineVoiceLauncher({ webView: WebView, autoStart = false, onCap
     const post = useCallback((msg) => {
         ref.current?.injectJavaScript(`window.postMessage(${JSON.stringify(msg)}, "*"); true;`);
     }, []);
+    const [voiceAgentId, setVoiceAgentId] = useState(client.config ? client.config.webcallAgentId || client.config.agentId : null);
+    const onErrorRef = useRef(onError);
+    onErrorRef.current = onError;
+    useEffect(() => {
+        if (voiceAgentId)
+            return;
+        let cancelled = false;
+        (async () => {
+            try {
+                const cfg = client.config ?? (await client.session());
+                if (!cancelled)
+                    setVoiceAgentId(cfg.webcallAgentId || cfg.agentId);
+            }
+            catch (err) {
+                if (!cancelled) {
+                    onErrorRef.current?.(String(err?.message || err || "Voice is unavailable."));
+                }
+            }
+        })();
+        return () => { cancelled = true; };
+    }, [client, voiceAgentId]);
+    const ensureMic = useCallback(async () => {
+        if (Platform.OS !== "android")
+            return true;
+        try {
+            const perm = PermissionsAndroid?.PERMISSIONS?.RECORD_AUDIO;
+            if (!perm)
+                return true;
+            if (await PermissionsAndroid.check(perm))
+                return true;
+            const res = await PermissionsAndroid.request(perm, {
+                title: "Microphone",
+                message: "Allow the microphone so you can talk to the assistant.",
+                buttonPositive: "Allow",
+            });
+            return res === PermissionsAndroid.RESULTS.GRANTED;
+        }
+        catch {
+            return true;
+        }
+    }, []);
     const pushContext = useCallback(() => {
-        const screen = client.getScreen();
-        if (!screen)
+        const frame = client.buildContextFrame();
+        if (!frame)
             return;
-        const { availableActions, ...rest } = screen;
         post({
             type: "graine:app-context",
-            context: rest,
-            availableActions: (availableActions ?? client.getAvailableActions()).filter((a) => client.getAvailableActions().includes(a)),
-            seq: Date.now(),
+            context: frame.context,
+            availableActions: frame.available_actions,
+            seq: frame.seq,
         });
     }, [client, post]);
     useEffect(() => {
         if (ready)
             pushContext();
     }, [ready, pushContext]);
+    const beginCall = useCallback(async () => {
+        if (client.isConnected) {
+            console.warn("[graine] The provider has its own connection open while GraineVoiceLauncher " +
+                "is starting a call. Pass autoConnect={false} to <GraineProvider> — otherwise " +
+                "the agent answers twice and the customer hears two greetings.");
+        }
+        if (!(await ensureMic())) {
+            setConnecting(false);
+            onMicDenied?.("NotAllowedError");
+            return;
+        }
+        post({ type: "graine:start-call" });
+    }, [client, ensureMic, onMicDenied, post]);
+    useEffect(() => {
+        if (!connected)
+            return;
+        return client.on("app_event", (event) => {
+            pushContext();
+            const { name, ...rest } = (event ?? {});
+            if (!name)
+                return;
+            post({ type: "graine:app-event", name, data: rest });
+        });
+    }, [client, connected, post, pushContext]);
     const onMessage = useCallback(async (e) => {
         let msg;
         try {
@@ -37,15 +102,25 @@ export function GraineVoiceLauncher({ webView: WebView, autoStart = false, onCap
             return;
         }
         switch (msg.type) {
+            case "graine:transport":
+                onTransport?.({
+                    transport: msg.transport === "rtc" ? "rtc" : "ws",
+                    requested: msg.requested === "rtc" ? "rtc" : "ws",
+                });
+                return;
             case "graine:ready":
                 setReady(true);
                 pushContext();
-                if (autoStart)
-                    post({ type: "graine:start-call" });
+                if (autoStart && !startedRef.current) {
+                    startedRef.current = true;
+                    void beginCall();
+                }
                 return;
             case "graine:call":
                 setConnected(!!msg.connected);
                 setConnecting(!!msg.connecting);
+                if (!msg.connected && !msg.connecting)
+                    startedRef.current = false;
                 onCallState?.({ connected: !!msg.connected, connecting: !!msg.connecting });
                 return;
             case "graine:muted":
@@ -75,15 +150,15 @@ export function GraineVoiceLauncher({ webView: WebView, autoStart = false, onCap
                 return;
             }
         }
-    }, [autoStart, client, onCaption, onCallState, onMicDenied, onEnded, post, pushContext]);
+    }, [autoStart, beginCall, client, onCaption, onCallState, onMicDenied, onTransport, onEnded, post, pushContext]);
     const api = {
         connected,
         connecting,
         muted,
         start: useCallback(() => {
             setConnecting(true);
-            post({ type: "graine:start-call" });
-        }, [post]),
+            void beginCall();
+        }, [beginCall]),
         end: useCallback(() => post({ type: "graine:end-call" }), [post]),
         setMuted: useCallback((m) => {
             setMutedState(m);
@@ -93,19 +168,34 @@ export function GraineVoiceLauncher({ webView: WebView, autoStart = false, onCap
     useEffect(() => () => { post({ type: "graine:end-call" }); }, [post]);
     useEffect(() => {
         let last = AppState.currentState;
+        let pending = null;
+        const CONFIRM_MS = 2000;
         const sub = AppState.addEventListener("change", (next) => {
             const wasActive = last === "active";
             last = next;
-            if (next === "background" && wasActive)
-                post({ type: "graine:end-call" });
+            if (next === "active") {
+                if (pending) {
+                    clearTimeout(pending);
+                    pending = null;
+                }
+                return;
+            }
+            if (next === "background" && wasActive && !pending) {
+                pending = setTimeout(() => {
+                    pending = null;
+                    if (AppState.currentState !== "active")
+                        post({ type: "graine:end-call" });
+                }, CONFIRM_MS);
+            }
         });
-        return () => sub.remove();
+        return () => { if (pending)
+            clearTimeout(pending); sub.remove(); };
     }, [post]);
-    const agentId = client.config?.agentId;
-    if (!agentId)
+    if (!voiceAgentId)
         return _jsx(_Fragment, { children: children?.(api) });
-    const src = `${client.opts.baseUrl.replace(/\/$/, "")}/embed/${agentId}` +
-        `?mode=voice&branding=0&embed=1` +
+    const effectiveTransport = transport ?? (appearance?.audioTransport === "rtc" ? "rtc" : "ws");
+    const src = `${client.opts.baseUrl.replace(/\/$/, "")}/embed/${voiceAgentId}` +
+        `?mode=voice&branding=0&embed=1&transport=${effectiveTransport}` +
         (appearance?.accent ? `&accent=${encodeURIComponent(String(appearance.accent).replace("#", ""))}` : "");
     return (_jsxs(_Fragment, { children: [_jsx(View, { style: { position: "absolute", width: 1, height: 1, opacity: 0, bottom: 0, left: 0 }, pointerEvents: "none", children: _jsx(WebView, { ref: ref, source: { uri: src }, originWhitelist: ["*"], javaScriptEnabled: true, domStorageEnabled: true, allowsInlineMediaPlayback: true, mediaPlaybackRequiresUserAction: false, mediaCapturePermissionGrantType: "grant", onMessage: onMessage, style: { width: 1, height: 1, backgroundColor: "transparent" } }) }), children?.(api)] }));
 }

package/dist/react-native/voice-rtc.d.ts

@@ -0,0 +1,41 @@
+export type RtcVoiceState = "idle" | "connecting" | "ringing" | "active" | "ended";
+export type RtcVoiceErrorCode = "mint_failed" | "microphone_denied" | "connect_failed" | "already_active";
+export declare class RtcVoiceError extends Error {
+    readonly code: RtcVoiceErrorCode;
+    readonly cause?: unknown;
+    constructor(code: RtcVoiceErrorCode, message: string, cause?: unknown);
+}
+export interface RtcVoiceSession {
+    server: string;
+    username: string;
+    password: string;
+    realm: string;
+    application_sid: string;
+    agent_id?: string;
+    expires_at?: string;
+}
+export interface RtcVoiceOptions {
+    publishableKey?: string;
+    baseUrl?: string;
+    getSession?: () => Promise<RtcVoiceSession>;
+    onState?: (s: RtcVoiceState) => void;
+    onEnded?: (reason: string) => void;
+    onLog?: (line: string) => void;
+    sessionId?: string;
+}
+export declare class RtcVoiceSessionController {
+    private client;
+    private call;
+    private state;
+    private startToken;
+    readonly sessionId: string;
+    private opts;
+    constructor(opts: RtcVoiceOptions);
+    getState(): RtcVoiceState;
+    private set;
+    private log;
+    private mint;
+    start(): Promise<void>;
+    setMuted(muted: boolean): void;
+    stop(): Promise<void>;
+}

package/dist/react-native/voice-rtc.js

@@ -0,0 +1,143 @@
+export class RtcVoiceError extends Error {
+    constructor(code, message, cause) {
+        super(message);
+        this.name = "RtcVoiceError";
+        this.code = code;
+        this.cause = cause;
+    }
+}
+function newSessionId() {
+    return `${Date.now().toString(16)}-${Math.random().toString(16).slice(2, 10)}`;
+}
+export class RtcVoiceSessionController {
+    constructor(opts) {
+        this.client = null;
+        this.call = null;
+        this.state = "idle";
+        this.startToken = null;
+        this.opts = opts;
+        this.sessionId = opts.sessionId || newSessionId();
+    }
+    getState() {
+        return this.state;
+    }
+    set(s) {
+        this.state = s;
+        this.opts.onState?.(s);
+    }
+    log(l) {
+        this.opts.onLog?.(l);
+    }
+    async mint() {
+        if (this.opts.getSession)
+            return this.opts.getSession();
+        if (!this.opts.publishableKey) {
+            throw new RtcVoiceError("mint_failed", "Pass publishableKey (or getSession for a custom backend).");
+        }
+        const base = (this.opts.baseUrl || "https://www.graine.ai").replace(/\/$/, "");
+        let res;
+        try {
+            res = await fetch(`${base}/api/embed/rtc-session`, {
+                method: "POST",
+                headers: { "Content-Type": "application/json" },
+                body: JSON.stringify({ publishableKey: this.opts.publishableKey }),
+            });
+        }
+        catch (e) {
+            throw new RtcVoiceError("mint_failed", "Could not reach the call service.", e);
+        }
+        if (!res.ok) {
+            let detail = "";
+            try {
+                detail = (await res.json())?.error || "";
+            }
+            catch {
+            }
+            throw new RtcVoiceError("mint_failed", detail || `Could not start a call (${res.status}).`);
+        }
+        return res.json();
+    }
+    async start() {
+        if (this.state !== "idle" && this.state !== "ended") {
+            throw new RtcVoiceError("already_active", "A call is already in progress.");
+        }
+        const token = (this.startToken = Symbol("start"));
+        this.set("connecting");
+        try {
+            const session = await this.mint();
+            if (token !== this.startToken)
+                return;
+            this.log(`session for ${session.realm}`);
+            const { createJambonzClient } = await import("@jambonz/client-sdk-react-native");
+            this.client = createJambonzClient({
+                server: session.server,
+                username: session.username,
+                password: session.password,
+                realm: session.realm,
+            });
+            this.client.on("error", (e) => this.log(`client error: ${e?.message ?? e}`));
+            await this.client.connect();
+            if (token !== this.startToken) {
+                await this.stop();
+                return;
+            }
+            this.log("registered");
+            this.call = this.client.callApplication(session.application_sid, {
+                headers: {
+                    "X-Graine-Session-Id": this.sessionId,
+                    ...(session.agent_id ? { "X-Graine-Agent-Id": session.agent_id } : {}),
+                },
+            });
+            this.call.on("stateChanged", (s) => this.log(`call: ${s}`));
+            this.call.on("accepted", () => {
+                this.set("active");
+                this.log("audio flowing");
+            });
+            this.call.on("ended", (c) => {
+                this.set("ended");
+                this.opts.onEnded?.(c?.reason || "ended");
+            });
+            this.call.on("failed", (c) => {
+                this.set("ended");
+                this.opts.onEnded?.(c?.reason || "failed");
+            });
+            this.set("ringing");
+        }
+        catch (err) {
+            if (token !== this.startToken)
+                return;
+            this.set("ended");
+            throw err instanceof RtcVoiceError
+                ? err
+                : new RtcVoiceError("connect_failed", err?.message || "Could not connect.", err);
+        }
+    }
+    setMuted(muted) {
+        try {
+            if (muted)
+                this.call?.mute?.();
+            else
+                this.call?.unmute?.();
+        }
+        catch {
+        }
+    }
+    async stop() {
+        if (this.state === "idle" || this.state === "ended")
+            return;
+        this.startToken = null;
+        try {
+            this.call?.hangup?.();
+        }
+        catch {
+        }
+        try {
+            await this.client?.disconnect?.();
+        }
+        catch {
+        }
+        this.call = null;
+        this.client = null;
+        this.set("ended");
+    }
+}

package/dist/voice.d.ts

@@ -1,4 +1,4 @@
-import type { GraineInAppClient } from "./client";
+import type { GraineInAppClient } from "./client.js";
 export declare const CAPTURE_SAMPLE_RATE = 16000;
 export declare const MIN_PLAYOUT_BUFFER_SECONDS = 0.15;
 export interface AudioAdapter {

package/dist/voice.js

@@ -1,4 +1,4 @@
-import { EchoGuard, base64ToBytes, decodeAgentAudio, decodePcm16 } from "./audio";
+import { EchoGuard, base64ToBytes, decodeAgentAudio, decodePcm16 } from "./audio.js";
 export const CAPTURE_SAMPLE_RATE = 16000;
 export const MIN_PLAYOUT_BUFFER_SECONDS = 0.15;
 export function base64ToPcm16(b64) {

package/package.json

@@ -1,6 +1,7 @@
 {
   "name": "@graineai/inapp-react-native",
-  "version": "0.12.0",
+  "version": "0.16.0",
+  "type": "module",
   "description": "Graine in-app agent for React Native — an agent that sees the screen your customer is on and can act on it.",
   "license": "MIT",
   "private": false,
@@ -35,7 +36,8 @@
   },
   "peerDependencies": {
     "react": ">=17",
-    "react-native": ">=0.68"
+    "react-native": ">=0.68",
+    "@jambonz/client-sdk-react-native": "^0.1.4"
   },
   "devDependencies": {
     "@types/react": "^18.2.0",
@@ -60,5 +62,10 @@
   "homepage": "https://www.graine.ai/docs/in-app",
   "bugs": {
     "url": "https://www.graine.ai/docs/in-app"
+  },
+  "peerDependenciesMeta": {
+    "@jambonz/client-sdk-react-native": {
+      "optional": true
+    }
   }
 }