@navai/voice-frontend 0.1.1 → 0.1.2

package/README.en.md

@@ -0,0 +1,320 @@
+# @navai/voice-frontend
+
+<p>
+  <a href="./README.es.md"><img alt="Idioma Espanol" src="https://img.shields.io/badge/Idioma-ES-0A66C2?style=for-the-badge"></a>
+  <a href="./README.en.md"><img alt="Language English" src="https://img.shields.io/badge/Language-EN-1D9A6C?style=for-the-badge"></a>
+</p>
+
+Frontend package to build Navai voice agents in web applications.
+
+It removes repeated boilerplate for:
+
+1. Realtime client secret requests.
+2. Route-aware navigation tools.
+3. Dynamic local function loading.
+4. Optional backend function bridging.
+5. React hook lifecycle for connect/disconnect.
+
+## Installation
+
+```bash
+npm install @navai/voice-frontend @openai/agents zod
+npm install react
+```
+
+## Package Architecture
+
+This package is intentionally split by concern:
+
+1. `src/backend.ts`
+HTTP client for backend routes:
+- `POST /navai/realtime/client-secret`
+- `GET /navai/functions`
+- `POST /navai/functions/execute`
+
+2. `src/runtime.ts`
+Runtime resolver for:
+- route module selection
+- function module filtering by `NAVAI_FUNCTIONS_FOLDERS`
+- optional model override
+
+3. `src/functions.ts`
+Local function loader:
+- imports modules from generated loaders
+- converts exports into normalized callable tool definitions
+
+4. `src/agent.ts`
+Agent builder:
+- creates `RealtimeAgent`
+- injects built-in tools (`navigate_to`, `execute_app_function`)
+- optionally adds direct alias tools for each allowed function
+
+5. `src/useWebVoiceAgent.ts`
+React lifecycle wrapper:
+- builds runtime config
+- requests client secret
+- discovers backend functions
+- builds agent
+- opens/closes `RealtimeSession`
+
+6. `src/routes.ts`
+Route matching helpers for natural language to path resolution.
+
+## End-to-End Runtime Flow
+
+Hook-driven runtime flow (`useWebVoiceAgent`):
+
+1. Resolve runtime config from `moduleLoaders` + `defaultRoutes` + env/options.
+2. Create backend client with `apiBaseUrl` or `NAVAI_API_URL`.
+3. On `start()`:
+- request client secret.
+- fetch backend function list.
+- build Navai agent with local + backend functions.
+- connect `RealtimeSession`.
+4. On `stop()`:
+- close session and reset state.
+
+State machine exposed by hook:
+
+- `idle`
+- `connecting`
+- `connected`
+- `error`
+
+Agent voice state exposed by the hook:
+
+- `agentVoiceState`: `idle | speaking`
+- `isAgentSpeaking`: `boolean`
+
+`agentVoiceState` is driven by realtime events `audio_start`, `audio_stopped`, and `audio_interrupted`.
+
+## Public API
+
+Main exports:
+
+- `buildNavaiAgent(...)`
+- `createNavaiBackendClient(...)`
+- `resolveNavaiFrontendRuntimeConfig(...)`
+- `loadNavaiFunctions(...)`
+- `useWebVoiceAgent(...)`
+- `resolveNavaiRoute(...)`
+- `getNavaiRoutePromptLines(...)`
+
+Useful types:
+
+- `NavaiRoute`
+- `NavaiFunctionDefinition`
+- `NavaiFunctionsRegistry`
+- `NavaiBackendFunctionDefinition`
+- `UseWebVoiceAgentOptions`
+
+## Tool Model and Behavior
+
+`buildNavaiAgent` always registers:
+
+- `navigate_to`
+- `execute_app_function`
+
+Optional direct alias tools:
+
+- for each allowed function name, a direct tool can be created.
+- reserved names are never used as direct tools (`navigate_to`, `execute_app_function`).
+- invalid tool ids are skipped (kept accessible via `execute_app_function`).
+
+Execution precedence:
+
+1. Try frontend/local function first.
+2. If missing, try backend function.
+3. If both exist with same name, frontend wins and backend is ignored with warning.
+
+## Dynamic Function Loading Internals
+
+`loadNavaiFunctions` supports module export shapes:
+
+1. Exported function.
+2. Exported class (instance methods become functions).
+3. Exported object (callable members become functions).
+
+Name normalization rules:
+
+- snake_case lowercase.
+- invalid chars removed.
+- collisions are renamed with suffixes (`_2`, `_3`, ...).
+
+Argument mapping rules:
+
+- `payload.args` or `payload.arguments` as direct args.
+- else `payload.value` as first arg.
+- else full payload as first arg.
+- context appended when arity indicates one more argument.
+
+For class methods:
+
+- constructor args: `payload.constructorArgs`.
+- method args: `payload.methodArgs`.
+
+## Runtime Resolution and Env Precedence
+
+`resolveNavaiFrontendRuntimeConfig` input priority:
+
+1. Explicit function args.
+2. Env object keys.
+3. Package defaults.
+
+Keys used:
+
+- `NAVAI_ROUTES_FILE`
+- `NAVAI_FUNCTIONS_FOLDERS`
+- `NAVAI_REALTIME_MODEL`
+
+Defaults:
+
+- routes file: `src/ai/routes.ts`
+- functions folder: `src/ai/functions-modules`
+
+Path matcher formats:
+
+- folder: `src/ai/functions-modules`
+- recursive: `src/ai/functions-modules/...`
+- wildcard: `src/features/*/voice-functions`
+- explicit file: `src/ai/functions-modules/secret.ts`
+- CSV list: `a,b,c`
+
+Fallback behavior:
+
+- if configured folders match no modules, warning is emitted.
+- resolver falls back to default functions folder.
+
+## Backend Client Behavior
+
+`createNavaiBackendClient` base URL priority:
+
+1. `apiBaseUrl` option.
+2. `env.NAVAI_API_URL`.
+3. fallback `http://localhost:3000`.
+
+Methods:
+
+- `createClientSecret(input?)`
+- `listFunctions()`
+- `executeFunction({ functionName, payload })`
+
+Error handling:
+
+- network/HTTP failures throw for create/execute.
+- function listing returns warnings and empty list on failures.
+
+## Generated Module Loader CLI
+
+This package ships:
+
+- `navai-generate-web-loaders`
+
+Default command behavior:
+
+1. Reads `.env` and process env.
+2. Resolves `NAVAI_FUNCTIONS_FOLDERS` and `NAVAI_ROUTES_FILE`.
+3. Selects modules only from configured function paths.
+4. Optionally includes configured route module if it differs from default route module.
+5. Writes `src/ai/generated-module-loaders.ts`.
+
+Manual usage:
+
+```bash
+navai-generate-web-loaders
+```
+
+Useful flags:
+
+- `--project-root <path>`
+- `--src-root <path>`
+- `--output-file <path>`
+- `--env-file <path>`
+- `--default-functions-folder <path>`
+- `--default-routes-file <path>`
+- `--type-import <module>`
+- `--export-name <identifier>`
+
+## Auto Setup on npm Install
+
+Postinstall script can auto-add missing scripts in consumer app:
+
+- `generate:module-loaders` -> `navai-generate-web-loaders`
+- `predev` -> `npm run generate:module-loaders`
+- `prebuild` -> `npm run generate:module-loaders`
+- `pretypecheck` -> `npm run generate:module-loaders`
+- `prelint` -> `npm run generate:module-loaders`
+
+Rules:
+
+- only missing scripts are added.
+- existing scripts are never overwritten.
+
+Disable auto setup:
+
+- `NAVAI_SKIP_AUTO_SETUP=1`
+- or `NAVAI_SKIP_FRONTEND_AUTO_SETUP=1`
+
+Manual setup runner:
+
+```bash
+npx navai-setup-voice-frontend
+```
+
+## Integration Examples
+
+Imperative integration:
+
+```ts
+import { RealtimeSession } from "@openai/agents/realtime";
+import { buildNavaiAgent, createNavaiBackendClient } from "@navai/voice-frontend";
+import { NAVAI_ROUTE_ITEMS } from "./ai/routes";
+import { NAVAI_WEB_MODULE_LOADERS } from "./ai/generated-module-loaders";
+
+const backend = createNavaiBackendClient({ apiBaseUrl: "http://localhost:3000" });
+const secret = await backend.createClientSecret();
+const backendList = await backend.listFunctions();
+
+const { agent, warnings } = await buildNavaiAgent({
+  navigate: (path) => router.navigate(path),
+  routes: NAVAI_ROUTE_ITEMS,
+  functionModuleLoaders: NAVAI_WEB_MODULE_LOADERS,
+  backendFunctions: backendList.functions,
+  executeBackendFunction: backend.executeFunction
+});
+
+warnings.forEach((w) => console.warn(w));
+
+const session = new RealtimeSession(agent);
+await session.connect({ apiKey: secret.value });
+```
+
+React hook integration:
+
+```ts
+import { useWebVoiceAgent } from "@navai/voice-frontend";
+import { NAVAI_WEB_MODULE_LOADERS } from "./ai/generated-module-loaders";
+import { NAVAI_ROUTE_ITEMS } from "./ai/routes";
+
+const voice = useWebVoiceAgent({
+  navigate: (path) => router.navigate(path),
+  moduleLoaders: NAVAI_WEB_MODULE_LOADERS,
+  defaultRoutes: NAVAI_ROUTE_ITEMS,
+  env: import.meta.env as Record<string, string | undefined>
+});
+```
+
+## Operational Notes
+
+- warnings are emitted with `console.warn` from runtime, backend list, and agent builder.
+- unknown function execution returns structured `ok: false` payload.
+- if route module fails to load or has invalid shape, resolver falls back to default routes.
+
+## Related Docs
+
+- Spanish version: `README.es.md`
+- English version: `README.en.md`
+- Backend package: `../voice-backend/README.md`
+- Mobile package: `../voice-mobile/README.md`
+- Playground Web: `../../apps/playground-web/README.md`
+- Playground API: `../../apps/playground-api/README.md`

package/README.es.md

@@ -0,0 +1,320 @@
+# @navai/voice-frontend
+
+<p>
+  <a href="./README.es.md"><img alt="Idioma Espanol" src="https://img.shields.io/badge/Idioma-ES-0A66C2?style=for-the-badge"></a>
+  <a href="./README.en.md"><img alt="Language English" src="https://img.shields.io/badge/Language-EN-1D9A6C?style=for-the-badge"></a>
+</p>
+
+Paquete frontend para construir agentes de voz Navai en aplicaciones web.
+
+El objetivo es evitar boilerplate repetido para:
+
+1. Solicitud de `client_secret` realtime.
+2. Tools de navegacion basadas en rutas permitidas.
+3. Carga dinamica de funciones locales.
+4. Puente opcional con funciones backend.
+5. Ciclo de vida React para conectar/desconectar sesion.
+
+## Instalacion
+
+```bash
+npm install @navai/voice-frontend @openai/agents zod
+npm install react
+```
+
+## Arquitectura del Paquete
+
+El paquete esta separado por responsabilidades:
+
+1. `src/backend.ts`
+Cliente HTTP para rutas backend:
+- `POST /navai/realtime/client-secret`
+- `GET /navai/functions`
+- `POST /navai/functions/execute`
+
+2. `src/runtime.ts`
+Resolver de runtime para:
+- seleccion de modulo de rutas
+- filtrado de modulos de funciones por `NAVAI_FUNCTIONS_FOLDERS`
+- override opcional de modelo
+
+3. `src/functions.ts`
+Loader de funciones locales:
+- importa modulos desde loaders generados
+- transforma exports en definiciones de tools normalizadas y ejecutables
+
+4. `src/agent.ts`
+Builder del agente:
+- crea `RealtimeAgent`
+- inyecta tools base (`navigate_to`, `execute_app_function`)
+- agrega aliases directos por funcion permitida cuando aplica
+
+5. `src/useWebVoiceAgent.ts`
+Wrapper de ciclo de vida React:
+- construye runtime config
+- solicita client secret
+- descubre funciones backend
+- construye el agente
+- abre/cierra `RealtimeSession`
+
+6. `src/routes.ts`
+Helpers para resolver texto natural hacia rutas permitidas.
+
+## Flujo Runtime de Punta a Punta
+
+Flujo del hook (`useWebVoiceAgent`):
+
+1. Resuelve runtime config desde `moduleLoaders` + `defaultRoutes` + env/opciones.
+2. Crea backend client con `apiBaseUrl` o `NAVAI_API_URL`.
+3. En `start()`:
+- solicita client secret.
+- solicita listado de funciones backend.
+- construye agente Navai con funciones locales + backend.
+- conecta `RealtimeSession`.
+4. En `stop()`:
+- cierra sesion y resetea estado.
+
+Maquina de estados expuesta por el hook:
+
+- `idle`
+- `connecting`
+- `connected`
+- `error`
+
+Estado de voz del agente expuesto por el hook:
+
+- `agentVoiceState`: `idle | speaking`
+- `isAgentSpeaking`: `boolean`
+
+`agentVoiceState` se actualiza con los eventos realtime `audio_start`, `audio_stopped` y `audio_interrupted`.
+
+## API Publica
+
+Exports principales:
+
+- `buildNavaiAgent(...)`
+- `createNavaiBackendClient(...)`
+- `resolveNavaiFrontendRuntimeConfig(...)`
+- `loadNavaiFunctions(...)`
+- `useWebVoiceAgent(...)`
+- `resolveNavaiRoute(...)`
+- `getNavaiRoutePromptLines(...)`
+
+Tipos utiles:
+
+- `NavaiRoute`
+- `NavaiFunctionDefinition`
+- `NavaiFunctionsRegistry`
+- `NavaiBackendFunctionDefinition`
+- `UseWebVoiceAgentOptions`
+
+## Modelo de Tools y Comportamiento
+
+`buildNavaiAgent` siempre registra:
+
+- `navigate_to`
+- `execute_app_function`
+
+Aliases directos opcionales:
+
+- para cada funcion permitida se puede crear una tool directa.
+- nombres reservados nunca se exponen como tool directa (`navigate_to`, `execute_app_function`).
+- ids invalidos de tool se omiten (la funcion sigue accesible via `execute_app_function`).
+
+Precedencia de ejecucion:
+
+1. Intenta funcion local/frontend.
+2. Si no existe, intenta funcion backend.
+3. Si ambas tienen el mismo nombre, gana frontend y backend se ignora con warning.
+
+## Internos de Carga Dinamica de Funciones
+
+`loadNavaiFunctions` soporta estos formatos de export:
+
+1. Funcion exportada.
+2. Clase exportada (metodos de instancia se vuelven funciones).
+3. Objeto exportado (miembros callables se vuelven funciones).
+
+Reglas de normalizacion de nombre:
+
+- snake_case en minusculas.
+- elimina caracteres invalidos.
+- colisiones se renombran con sufijos (`_2`, `_3`, ...).
+
+Reglas de mapeo de argumentos:
+
+- usa `payload.args` o `payload.arguments` como argumentos directos.
+- si no, usa `payload.value` como primer argumento.
+- si no, usa payload completo como primer argumento.
+- agrega contexto cuando la aridad indica un argumento adicional.
+
+Para metodos de clase:
+
+- args de constructor: `payload.constructorArgs`.
+- args del metodo: `payload.methodArgs`.
+
+## Resolucion Runtime y Precedencia de Env
+
+Prioridad de entrada en `resolveNavaiFrontendRuntimeConfig`:
+
+1. Argumentos explicitos de funcion.
+2. Claves del objeto env.
+3. Defaults del paquete.
+
+Claves usadas:
+
+- `NAVAI_ROUTES_FILE`
+- `NAVAI_FUNCTIONS_FOLDERS`
+- `NAVAI_REALTIME_MODEL`
+
+Defaults:
+
+- archivo de rutas: `src/ai/routes.ts`
+- carpeta de funciones: `src/ai/functions-modules`
+
+Formatos aceptados en matcher de rutas:
+
+- carpeta: `src/ai/functions-modules`
+- recursivo: `src/ai/functions-modules/...`
+- wildcard: `src/features/*/voice-functions`
+- archivo explicito: `src/ai/functions-modules/secret.ts`
+- CSV: `a,b,c`
+
+Comportamiento fallback:
+
+- si `NAVAI_FUNCTIONS_FOLDERS` no matchea modulos, emite warning.
+- hace fallback a carpeta de funciones por defecto.
+
+## Comportamiento del Backend Client
+
+Prioridad de base URL en `createNavaiBackendClient`:
+
+1. Opcion `apiBaseUrl`.
+2. `env.NAVAI_API_URL`.
+3. Fallback `http://localhost:3000`.
+
+Metodos:
+
+- `createClientSecret(input?)`
+- `listFunctions()`
+- `executeFunction({ functionName, payload })`
+
+Manejo de errores:
+
+- fallos de red/HTTP lanzan error en create/execute.
+- el listado de funciones retorna warnings + lista vacia en fallos.
+
+## CLI Generador de Module Loaders
+
+Este paquete incluye:
+
+- `navai-generate-web-loaders`
+
+Comportamiento por defecto:
+
+1. Lee `.env` y env del proceso.
+2. Resuelve `NAVAI_FUNCTIONS_FOLDERS` y `NAVAI_ROUTES_FILE`.
+3. Selecciona modulos solo en rutas de funciones configuradas.
+4. Incluye modulo de rutas configurado cuando difiere del modulo por defecto.
+5. Escribe `src/ai/generated-module-loaders.ts`.
+
+Uso manual:
+
+```bash
+navai-generate-web-loaders
+```
+
+Flags utiles:
+
+- `--project-root <path>`
+- `--src-root <path>`
+- `--output-file <path>`
+- `--env-file <path>`
+- `--default-functions-folder <path>`
+- `--default-routes-file <path>`
+- `--type-import <module>`
+- `--export-name <identifier>`
+
+## Auto Configuracion al Instalar desde npm
+
+El `postinstall` puede agregar scripts faltantes en el consumidor:
+
+- `generate:module-loaders` -> `navai-generate-web-loaders`
+- `predev` -> `npm run generate:module-loaders`
+- `prebuild` -> `npm run generate:module-loaders`
+- `pretypecheck` -> `npm run generate:module-loaders`
+- `prelint` -> `npm run generate:module-loaders`
+
+Reglas:
+
+- solo agrega scripts faltantes.
+- nunca sobreescribe scripts existentes.
+
+Desactivar auto setup:
+
+- `NAVAI_SKIP_AUTO_SETUP=1`
+- o `NAVAI_SKIP_FRONTEND_AUTO_SETUP=1`
+
+Ejecutor manual de setup:
+
+```bash
+npx navai-setup-voice-frontend
+```
+
+## Ejemplos de Integracion
+
+Integracion imperativa:
+
+```ts
+import { RealtimeSession } from "@openai/agents/realtime";
+import { buildNavaiAgent, createNavaiBackendClient } from "@navai/voice-frontend";
+import { NAVAI_ROUTE_ITEMS } from "./ai/routes";
+import { NAVAI_WEB_MODULE_LOADERS } from "./ai/generated-module-loaders";
+
+const backend = createNavaiBackendClient({ apiBaseUrl: "http://localhost:3000" });
+const secret = await backend.createClientSecret();
+const backendList = await backend.listFunctions();
+
+const { agent, warnings } = await buildNavaiAgent({
+  navigate: (path) => router.navigate(path),
+  routes: NAVAI_ROUTE_ITEMS,
+  functionModuleLoaders: NAVAI_WEB_MODULE_LOADERS,
+  backendFunctions: backendList.functions,
+  executeBackendFunction: backend.executeFunction
+});
+
+warnings.forEach((w) => console.warn(w));
+
+const session = new RealtimeSession(agent);
+await session.connect({ apiKey: secret.value });
+```
+
+Integracion con hook React:
+
+```ts
+import { useWebVoiceAgent } from "@navai/voice-frontend";
+import { NAVAI_WEB_MODULE_LOADERS } from "./ai/generated-module-loaders";
+import { NAVAI_ROUTE_ITEMS } from "./ai/routes";
+
+const voice = useWebVoiceAgent({
+  navigate: (path) => router.navigate(path),
+  moduleLoaders: NAVAI_WEB_MODULE_LOADERS,
+  defaultRoutes: NAVAI_ROUTE_ITEMS,
+  env: import.meta.env as Record<string, string | undefined>
+});
+```
+
+## Notas Operativas
+
+- los warnings se emiten con `console.warn` desde runtime, backend list y agent builder.
+- una funcion desconocida retorna payload estructurado `ok: false`.
+- si el modulo de rutas falla o tiene shape invalido, el resolver hace fallback a rutas por defecto.
+
+## Documentacion Relacionada
+
+- Version en espanol: `README.es.md`
+- Version en ingles: `README.en.md`
+- Paquete backend: `../voice-backend/README.md`
+- Paquete mobile: `../voice-mobile/README.md`
+- Playground Web: `../../apps/playground-web/README.md`
+- Playground API: `../../apps/playground-api/README.md`

package/README.md

@@ -1,126 +1,320 @@
-# @navai/voice-frontend
-
-Frontend helpers to integrate OpenAI Realtime voice without creating custom base plumbing in every app.
-
-## What this package provides
-
-- Route resolution helpers.
-- Optional dynamic frontend function loader.
-- `useWebVoiceAgent(...)` to bootstrap voice session from React with backend + frontend tools.
-- `buildNavaiAgent(...)` with built-in tools:
-  - `navigate_to`
-  - `execute_app_function`
-- Optional backend function bridge:
-  - frontend + backend functions can coexist under `execute_app_function`.
-- `createNavaiBackendClient(...)` to centralize calls to backend routes:
-  - `POST /navai/realtime/client-secret`
-  - `GET /navai/functions`
-  - `POST /navai/functions/execute`
-- `resolveNavaiFrontendRuntimeConfig(...)` to read routes/functions from env.
-
-## Install
-
-```bash
-npm install @navai/voice-frontend @openai/agents zod
-```
-
-Peer dependency for hooks:
-
-```bash
-npm install react
-```
-
-When installed from npm, this package auto-configures missing scripts in the consumer `package.json`:
-
-- `generate:module-loaders` -> `navai-generate-web-loaders`
-- `predev`, `prebuild`, `pretypecheck`, `prelint` -> `npm run generate:module-loaders`
-
-It only adds missing entries and never overwrites existing scripts.
-To disable auto-setup, set `NAVAI_SKIP_AUTO_SETUP=1` (or `NAVAI_SKIP_FRONTEND_AUTO_SETUP=1`) during install.
-To run setup manually later, use `npx navai-setup-voice-frontend`.
-
-## Expected app inputs
-
-1. Route data in `src/ai/routes.ts` (or any array compatible with `NavaiRoute[]`).
-2. Optional function module loaders when you want local/frontend tools.
-
-## Minimal usage (no bundler-specific APIs)
-
-```ts
-import { buildNavaiAgent, createNavaiBackendClient } from "@navai/voice-frontend";
-import { NAVAI_ROUTE_ITEMS } from "./ai/routes";
-
-const backendClient = createNavaiBackendClient({
-  apiBaseUrl: "http://localhost:3000"
-});
-const backendFunctions = await backendClient.listFunctions();
-
-const { agent } = await buildNavaiAgent({
-  navigate: (path) => routerNavigate(path),
-  routes: NAVAI_ROUTE_ITEMS,
-  backendFunctions: backendFunctions.functions,
-  executeBackendFunction: backendClient.executeFunction
-});
-```
-
-Then use `agent` with `RealtimeSession` from `@openai/agents/realtime`.
-
-## React hook usage
-
-```ts
-import { useWebVoiceAgent } from "@navai/voice-frontend";
-import { NAVAI_WEB_MODULE_LOADERS } from "./ai/generated-module-loaders";
-import { NAVAI_ROUTE_ITEMS } from "./ai/routes";
-
-const agent = useWebVoiceAgent({
-  navigate: (path) => routerNavigate(path),
-  moduleLoaders: NAVAI_WEB_MODULE_LOADERS,
-  defaultRoutes: NAVAI_ROUTE_ITEMS,
-  env: import.meta.env as Record<string, string | undefined>
-});
-```
-
-## Optional dynamic frontend functions (bundler adapter)
-
-If your bundler can provide module loaders, you can add local frontend functions too.
-
-```ts
-import { resolveNavaiFrontendRuntimeConfig } from "@navai/voice-frontend";
-
-// Vite adapter example (folder-scoped):
-const runtime = await resolveNavaiFrontendRuntimeConfig({
-  moduleLoaders: import.meta.glob(["/src/ai/functions-modules/**/*.{ts,js}"]),
-  defaultRoutes: NAVAI_ROUTE_ITEMS
-});
-```
-
-Execution rule inside `execute_app_function`:
-
-- local/frontend function is attempted first.
-- if not found locally, backend function is attempted.
-- if names conflict, frontend function wins and backend one is ignored with warning.
-
-## Runtime env keys
-
-`resolveNavaiFrontendRuntimeConfig` reads:
-
-- `NAVAI_ROUTES_FILE`
-- `NAVAI_FUNCTIONS_FOLDERS`
-- `NAVAI_REALTIME_MODEL`
-
-`createNavaiBackendClient` reads:
-
-- `NAVAI_API_URL` when you pass `env`
-- or `apiBaseUrl` directly (fallback `http://localhost:3000`)
-
-When `modelOverride` exists, pass it to:
-
-- backend request body: `POST /navai/realtime/client-secret`
-- realtime connection: `session.connect({ model })`
-
-## `NAVAI_FUNCTIONS_FOLDERS` formats
-
-- single path: `src/ai/functions-modules`
-- recursive marker: `src/ai/functions-modules/...`
-- wildcard: `src/features/*/voice-functions`
-- CSV list: `src/ai/functions,src/features/account/functions`
+# @navai/voice-frontend
+
+<p>
+  <a href="./README.es.md"><img alt="Idioma Espanol" src="https://img.shields.io/badge/Idioma-ES-0A66C2?style=for-the-badge"></a>
+  <a href="./README.en.md"><img alt="Language English" src="https://img.shields.io/badge/Language-EN-1D9A6C?style=for-the-badge"></a>
+</p>
+
+Frontend package to build Navai voice agents in web applications.
+
+It removes repeated boilerplate for:
+
+1. Realtime client secret requests.
+2. Route-aware navigation tools.
+3. Dynamic local function loading.
+4. Optional backend function bridging.
+5. React hook lifecycle for connect/disconnect.
+
+## Installation
+
+```bash
+npm install @navai/voice-frontend @openai/agents zod
+npm install react
+```
+
+## Package Architecture
+
+This package is intentionally split by concern:
+
+1. `src/backend.ts`
+HTTP client for backend routes:
+- `POST /navai/realtime/client-secret`
+- `GET /navai/functions`
+- `POST /navai/functions/execute`
+
+2. `src/runtime.ts`
+Runtime resolver for:
+- route module selection
+- function module filtering by `NAVAI_FUNCTIONS_FOLDERS`
+- optional model override
+
+3. `src/functions.ts`
+Local function loader:
+- imports modules from generated loaders
+- converts exports into normalized callable tool definitions
+
+4. `src/agent.ts`
+Agent builder:
+- creates `RealtimeAgent`
+- injects built-in tools (`navigate_to`, `execute_app_function`)
+- optionally adds direct alias tools for each allowed function
+
+5. `src/useWebVoiceAgent.ts`
+React lifecycle wrapper:
+- builds runtime config
+- requests client secret
+- discovers backend functions
+- builds agent
+- opens/closes `RealtimeSession`
+
+6. `src/routes.ts`
+Route matching helpers for natural language to path resolution.
+
+## End-to-End Runtime Flow
+
+Hook-driven runtime flow (`useWebVoiceAgent`):
+
+1. Resolve runtime config from `moduleLoaders` + `defaultRoutes` + env/options.
+2. Create backend client with `apiBaseUrl` or `NAVAI_API_URL`.
+3. On `start()`:
+- request client secret.
+- fetch backend function list.
+- build Navai agent with local + backend functions.
+- connect `RealtimeSession`.
+4. On `stop()`:
+- close session and reset state.
+
+State machine exposed by hook:
+
+- `idle`
+- `connecting`
+- `connected`
+- `error`
+
+Agent voice state exposed by the hook:
+
+- `agentVoiceState`: `idle | speaking`
+- `isAgentSpeaking`: `boolean`
+
+`agentVoiceState` is driven by realtime events `audio_start`, `audio_stopped`, and `audio_interrupted`.
+
+## Public API
+
+Main exports:
+
+- `buildNavaiAgent(...)`
+- `createNavaiBackendClient(...)`
+- `resolveNavaiFrontendRuntimeConfig(...)`
+- `loadNavaiFunctions(...)`
+- `useWebVoiceAgent(...)`
+- `resolveNavaiRoute(...)`
+- `getNavaiRoutePromptLines(...)`
+
+Useful types:
+
+- `NavaiRoute`
+- `NavaiFunctionDefinition`
+- `NavaiFunctionsRegistry`
+- `NavaiBackendFunctionDefinition`
+- `UseWebVoiceAgentOptions`
+
+## Tool Model and Behavior
+
+`buildNavaiAgent` always registers:
+
+- `navigate_to`
+- `execute_app_function`
+
+Optional direct alias tools:
+
+- for each allowed function name, a direct tool can be created.
+- reserved names are never used as direct tools (`navigate_to`, `execute_app_function`).
+- invalid tool ids are skipped (kept accessible via `execute_app_function`).
+
+Execution precedence:
+
+1. Try frontend/local function first.
+2. If missing, try backend function.
+3. If both exist with same name, frontend wins and backend is ignored with warning.
+
+## Dynamic Function Loading Internals
+
+`loadNavaiFunctions` supports module export shapes:
+
+1. Exported function.
+2. Exported class (instance methods become functions).
+3. Exported object (callable members become functions).
+
+Name normalization rules:
+
+- snake_case lowercase.
+- invalid chars removed.
+- collisions are renamed with suffixes (`_2`, `_3`, ...).
+
+Argument mapping rules:
+
+- `payload.args` or `payload.arguments` as direct args.
+- else `payload.value` as first arg.
+- else full payload as first arg.
+- context appended when arity indicates one more argument.
+
+For class methods:
+
+- constructor args: `payload.constructorArgs`.
+- method args: `payload.methodArgs`.
+
+## Runtime Resolution and Env Precedence
+
+`resolveNavaiFrontendRuntimeConfig` input priority:
+
+1. Explicit function args.
+2. Env object keys.
+3. Package defaults.
+
+Keys used:
+
+- `NAVAI_ROUTES_FILE`
+- `NAVAI_FUNCTIONS_FOLDERS`
+- `NAVAI_REALTIME_MODEL`
+
+Defaults:
+
+- routes file: `src/ai/routes.ts`
+- functions folder: `src/ai/functions-modules`
+
+Path matcher formats:
+
+- folder: `src/ai/functions-modules`
+- recursive: `src/ai/functions-modules/...`
+- wildcard: `src/features/*/voice-functions`
+- explicit file: `src/ai/functions-modules/secret.ts`
+- CSV list: `a,b,c`
+
+Fallback behavior:
+
+- if configured folders match no modules, warning is emitted.
+- resolver falls back to default functions folder.
+
+## Backend Client Behavior
+
+`createNavaiBackendClient` base URL priority:
+
+1. `apiBaseUrl` option.
+2. `env.NAVAI_API_URL`.
+3. fallback `http://localhost:3000`.
+
+Methods:
+
+- `createClientSecret(input?)`
+- `listFunctions()`
+- `executeFunction({ functionName, payload })`
+
+Error handling:
+
+- network/HTTP failures throw for create/execute.
+- function listing returns warnings and empty list on failures.
+
+## Generated Module Loader CLI
+
+This package ships:
+
+- `navai-generate-web-loaders`
+
+Default command behavior:
+
+1. Reads `.env` and process env.
+2. Resolves `NAVAI_FUNCTIONS_FOLDERS` and `NAVAI_ROUTES_FILE`.
+3. Selects modules only from configured function paths.
+4. Optionally includes configured route module if it differs from default route module.
+5. Writes `src/ai/generated-module-loaders.ts`.
+
+Manual usage:
+
+```bash
+navai-generate-web-loaders
+```
+
+Useful flags:
+
+- `--project-root <path>`
+- `--src-root <path>`
+- `--output-file <path>`
+- `--env-file <path>`
+- `--default-functions-folder <path>`
+- `--default-routes-file <path>`
+- `--type-import <module>`
+- `--export-name <identifier>`
+
+## Auto Setup on npm Install
+
+Postinstall script can auto-add missing scripts in consumer app:
+
+- `generate:module-loaders` -> `navai-generate-web-loaders`
+- `predev` -> `npm run generate:module-loaders`
+- `prebuild` -> `npm run generate:module-loaders`
+- `pretypecheck` -> `npm run generate:module-loaders`
+- `prelint` -> `npm run generate:module-loaders`
+
+Rules:
+
+- only missing scripts are added.
+- existing scripts are never overwritten.
+
+Disable auto setup:
+
+- `NAVAI_SKIP_AUTO_SETUP=1`
+- or `NAVAI_SKIP_FRONTEND_AUTO_SETUP=1`
+
+Manual setup runner:
+
+```bash
+npx navai-setup-voice-frontend
+```
+
+## Integration Examples
+
+Imperative integration:
+
+```ts
+import { RealtimeSession } from "@openai/agents/realtime";
+import { buildNavaiAgent, createNavaiBackendClient } from "@navai/voice-frontend";
+import { NAVAI_ROUTE_ITEMS } from "./ai/routes";
+import { NAVAI_WEB_MODULE_LOADERS } from "./ai/generated-module-loaders";
+
+const backend = createNavaiBackendClient({ apiBaseUrl: "http://localhost:3000" });
+const secret = await backend.createClientSecret();
+const backendList = await backend.listFunctions();
+
+const { agent, warnings } = await buildNavaiAgent({
+  navigate: (path) => router.navigate(path),
+  routes: NAVAI_ROUTE_ITEMS,
+  functionModuleLoaders: NAVAI_WEB_MODULE_LOADERS,
+  backendFunctions: backendList.functions,
+  executeBackendFunction: backend.executeFunction
+});
+
+warnings.forEach((w) => console.warn(w));
+
+const session = new RealtimeSession(agent);
+await session.connect({ apiKey: secret.value });
+```
+
+React hook integration:
+
+```ts
+import { useWebVoiceAgent } from "@navai/voice-frontend";
+import { NAVAI_WEB_MODULE_LOADERS } from "./ai/generated-module-loaders";
+import { NAVAI_ROUTE_ITEMS } from "./ai/routes";
+
+const voice = useWebVoiceAgent({
+  navigate: (path) => router.navigate(path),
+  moduleLoaders: NAVAI_WEB_MODULE_LOADERS,
+  defaultRoutes: NAVAI_ROUTE_ITEMS,
+  env: import.meta.env as Record<string, string | undefined>
+});
+```
+
+## Operational Notes
+
+- warnings are emitted with `console.warn` from runtime, backend list, and agent builder.
+- unknown function execution returns structured `ok: false` payload.
+- if route module fails to load or has invalid shape, resolver falls back to default routes.
+
+## Related Docs
+
+- Spanish version: `README.es.md`
+- English version: `README.en.md`
+- Backend package: `../voice-backend/README.md`
+- Mobile package: `../voice-mobile/README.md`
+- Playground Web: `../../apps/playground-web/README.md`
+- Playground API: `../../apps/playground-api/README.md`

package/dist/index.cjs

@@ -760,6 +760,7 @@ function emitWarnings(warnings) {
 }
 function useWebVoiceAgent(options) {
   const sessionRef = (0, import_react.useRef)(null);
+  const attachedRealtimeSessionRef = (0, import_react.useRef)(null);
   const runtimeConfigPromise = (0, import_react.useMemo)(
     () => resolveNavaiFrontendRuntimeConfig({
       moduleLoaders: options.moduleLoaders,
@@ -790,15 +791,61 @@ function useWebVoiceAgent(options) {
     [options.apiBaseUrl, options.env]
   );
   const [status, setStatus] = (0, import_react.useState)("idle");
+  const [agentVoiceState, setAgentVoiceState] = (0, import_react.useState)("idle");
   const [error, setError] = (0, import_react.useState)(null);
+  const setAgentVoiceStateIfChanged = (0, import_react.useCallback)((next) => {
+    setAgentVoiceState((current) => current === next ? current : next);
+  }, []);
+  const handleSessionAudioStart = (0, import_react.useCallback)(() => {
+    setAgentVoiceStateIfChanged("speaking");
+  }, [setAgentVoiceStateIfChanged]);
+  const handleSessionAudioStopped = (0, import_react.useCallback)(() => {
+    setAgentVoiceStateIfChanged("idle");
+  }, [setAgentVoiceStateIfChanged]);
+  const handleSessionAudioInterrupted = (0, import_react.useCallback)(() => {
+    setAgentVoiceStateIfChanged("idle");
+  }, [setAgentVoiceStateIfChanged]);
+  const handleSessionError = (0, import_react.useCallback)(() => {
+    setAgentVoiceStateIfChanged("idle");
+  }, [setAgentVoiceStateIfChanged]);
+  const detachSessionAudioListeners = (0, import_react.useCallback)(() => {
+    const attachedSession = attachedRealtimeSessionRef.current;
+    if (!attachedSession) {
+      return;
+    }
+    attachedSession.off("audio_start", handleSessionAudioStart);
+    attachedSession.off("audio_stopped", handleSessionAudioStopped);
+    attachedSession.off("audio_interrupted", handleSessionAudioInterrupted);
+    attachedSession.off("error", handleSessionError);
+    attachedRealtimeSessionRef.current = null;
+  }, [handleSessionAudioInterrupted, handleSessionAudioStart, handleSessionAudioStopped, handleSessionError]);
+  const attachSessionAudioListeners = (0, import_react.useCallback)(
+    (session) => {
+      detachSessionAudioListeners();
+      session.on("audio_start", handleSessionAudioStart);
+      session.on("audio_stopped", handleSessionAudioStopped);
+      session.on("audio_interrupted", handleSessionAudioInterrupted);
+      session.on("error", handleSessionError);
+      attachedRealtimeSessionRef.current = session;
+    },
+    [
+      detachSessionAudioListeners,
+      handleSessionAudioInterrupted,
+      handleSessionAudioStart,
+      handleSessionAudioStopped,
+      handleSessionError
+    ]
+  );
   const stop = (0, import_react.useCallback)(() => {
+    detachSessionAudioListeners();
     try {
       sessionRef.current?.close();
     } finally {
       sessionRef.current = null;
       setStatus("idle");
+      setAgentVoiceStateIfChanged("idle");
     }
-  }, []);
+  }, [detachSessionAudioListeners, setAgentVoiceStateIfChanged]);
   (0, import_react.useEffect)(() => {
     return () => {
       stop();
@@ -810,6 +857,7 @@ function useWebVoiceAgent(options) {
     }
     setError(null);
     setStatus("connecting");
+    setAgentVoiceStateIfChanged("idle");
     try {
       const runtimeConfig = await runtimeConfigPromise;
       const requestPayload = runtimeConfig.modelOverride ? { model: runtimeConfig.modelOverride } : {};
@@ -824,6 +872,7 @@ function useWebVoiceAgent(options) {
       });
       emitWarnings([...runtimeConfig.warnings, ...backendFunctionsResult.warnings, ...warnings]);
       const session = new import_realtime2.RealtimeSession(agent);
+      attachSessionAudioListeners(session);
       if (runtimeConfig.modelOverride) {
         await session.connect({ apiKey: secretPayload.value, model: runtimeConfig.modelOverride });
       } else {
@@ -835,18 +884,30 @@ function useWebVoiceAgent(options) {
       const message = formatError(startError);
       setError(message);
       setStatus("error");
+      setAgentVoiceStateIfChanged("idle");
+      detachSessionAudioListeners();
       try {
         sessionRef.current?.close();
       } catch {
       }
       sessionRef.current = null;
     }
-  }, [backendClient, options.navigate, runtimeConfigPromise, status]);
+  }, [
+    attachSessionAudioListeners,
+    backendClient,
+    detachSessionAudioListeners,
+    options.navigate,
+    runtimeConfigPromise,
+    setAgentVoiceStateIfChanged,
+    status
+  ]);
   return {
     status,
+    agentVoiceState,
     error,
     isConnecting: status === "connecting",
     isConnected: status === "connected",
+    isAgentSpeaking: agentVoiceState === "speaking",
     start,
     stop
   };

package/dist/index.d.cts

@@ -104,6 +104,7 @@ type ResolveNavaiFrontendRuntimeConfigResult = {
 declare function resolveNavaiFrontendRuntimeConfig(options: ResolveNavaiFrontendRuntimeConfigOptions): Promise<ResolveNavaiFrontendRuntimeConfigResult>;
 
 type VoiceStatus = "idle" | "connecting" | "connected" | "error";
+type AgentVoiceState = "idle" | "speaking";
 type NavaiFrontendEnv = Record<string, string | undefined>;
 type UseWebVoiceAgentOptions = {
     navigate: (path: string) => void;
@@ -119,9 +120,11 @@ type UseWebVoiceAgentOptions = {
 };
 type UseWebVoiceAgentResult = {
     status: VoiceStatus;
+    agentVoiceState: AgentVoiceState;
     error: string | null;
     isConnecting: boolean;
     isConnected: boolean;
+    isAgentSpeaking: boolean;
     start: () => Promise<void>;
     stop: () => void;
 };

package/dist/index.d.ts

@@ -104,6 +104,7 @@ type ResolveNavaiFrontendRuntimeConfigResult = {
 declare function resolveNavaiFrontendRuntimeConfig(options: ResolveNavaiFrontendRuntimeConfigOptions): Promise<ResolveNavaiFrontendRuntimeConfigResult>;
 
 type VoiceStatus = "idle" | "connecting" | "connected" | "error";
+type AgentVoiceState = "idle" | "speaking";
 type NavaiFrontendEnv = Record<string, string | undefined>;
 type UseWebVoiceAgentOptions = {
     navigate: (path: string) => void;
@@ -119,9 +120,11 @@ type UseWebVoiceAgentOptions = {
 };
 type UseWebVoiceAgentResult = {
     status: VoiceStatus;
+    agentVoiceState: AgentVoiceState;
     error: string | null;
     isConnecting: boolean;
     isConnected: boolean;
+    isAgentSpeaking: boolean;
     start: () => Promise<void>;
     stop: () => void;
 };

package/dist/index.js

@@ -728,6 +728,7 @@ function emitWarnings(warnings) {
 }
 function useWebVoiceAgent(options) {
   const sessionRef = useRef(null);
+  const attachedRealtimeSessionRef = useRef(null);
   const runtimeConfigPromise = useMemo(
     () => resolveNavaiFrontendRuntimeConfig({
       moduleLoaders: options.moduleLoaders,
@@ -758,15 +759,61 @@ function useWebVoiceAgent(options) {
     [options.apiBaseUrl, options.env]
   );
   const [status, setStatus] = useState("idle");
+  const [agentVoiceState, setAgentVoiceState] = useState("idle");
   const [error, setError] = useState(null);
+  const setAgentVoiceStateIfChanged = useCallback((next) => {
+    setAgentVoiceState((current) => current === next ? current : next);
+  }, []);
+  const handleSessionAudioStart = useCallback(() => {
+    setAgentVoiceStateIfChanged("speaking");
+  }, [setAgentVoiceStateIfChanged]);
+  const handleSessionAudioStopped = useCallback(() => {
+    setAgentVoiceStateIfChanged("idle");
+  }, [setAgentVoiceStateIfChanged]);
+  const handleSessionAudioInterrupted = useCallback(() => {
+    setAgentVoiceStateIfChanged("idle");
+  }, [setAgentVoiceStateIfChanged]);
+  const handleSessionError = useCallback(() => {
+    setAgentVoiceStateIfChanged("idle");
+  }, [setAgentVoiceStateIfChanged]);
+  const detachSessionAudioListeners = useCallback(() => {
+    const attachedSession = attachedRealtimeSessionRef.current;
+    if (!attachedSession) {
+      return;
+    }
+    attachedSession.off("audio_start", handleSessionAudioStart);
+    attachedSession.off("audio_stopped", handleSessionAudioStopped);
+    attachedSession.off("audio_interrupted", handleSessionAudioInterrupted);
+    attachedSession.off("error", handleSessionError);
+    attachedRealtimeSessionRef.current = null;
+  }, [handleSessionAudioInterrupted, handleSessionAudioStart, handleSessionAudioStopped, handleSessionError]);
+  const attachSessionAudioListeners = useCallback(
+    (session) => {
+      detachSessionAudioListeners();
+      session.on("audio_start", handleSessionAudioStart);
+      session.on("audio_stopped", handleSessionAudioStopped);
+      session.on("audio_interrupted", handleSessionAudioInterrupted);
+      session.on("error", handleSessionError);
+      attachedRealtimeSessionRef.current = session;
+    },
+    [
+      detachSessionAudioListeners,
+      handleSessionAudioInterrupted,
+      handleSessionAudioStart,
+      handleSessionAudioStopped,
+      handleSessionError
+    ]
+  );
   const stop = useCallback(() => {
+    detachSessionAudioListeners();
     try {
       sessionRef.current?.close();
     } finally {
       sessionRef.current = null;
       setStatus("idle");
+      setAgentVoiceStateIfChanged("idle");
     }
-  }, []);
+  }, [detachSessionAudioListeners, setAgentVoiceStateIfChanged]);
   useEffect(() => {
     return () => {
       stop();
@@ -778,6 +825,7 @@ function useWebVoiceAgent(options) {
     }
     setError(null);
     setStatus("connecting");
+    setAgentVoiceStateIfChanged("idle");
     try {
       const runtimeConfig = await runtimeConfigPromise;
       const requestPayload = runtimeConfig.modelOverride ? { model: runtimeConfig.modelOverride } : {};
@@ -792,6 +840,7 @@ function useWebVoiceAgent(options) {
       });
       emitWarnings([...runtimeConfig.warnings, ...backendFunctionsResult.warnings, ...warnings]);
       const session = new RealtimeSession(agent);
+      attachSessionAudioListeners(session);
       if (runtimeConfig.modelOverride) {
         await session.connect({ apiKey: secretPayload.value, model: runtimeConfig.modelOverride });
       } else {
@@ -803,18 +852,30 @@ function useWebVoiceAgent(options) {
       const message = formatError(startError);
       setError(message);
       setStatus("error");
+      setAgentVoiceStateIfChanged("idle");
+      detachSessionAudioListeners();
       try {
         sessionRef.current?.close();
       } catch {
       }
       sessionRef.current = null;
     }
-  }, [backendClient, options.navigate, runtimeConfigPromise, status]);
+  }, [
+    attachSessionAudioListeners,
+    backendClient,
+    detachSessionAudioListeners,
+    options.navigate,
+    runtimeConfigPromise,
+    setAgentVoiceStateIfChanged,
+    status
+  ]);
   return {
     status,
+    agentVoiceState,
     error,
     isConnecting: status === "connecting",
     isConnected: status === "connected",
+    isAgentSpeaking: agentVoiceState === "speaking",
     start,
     stop
   };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@navai/voice-frontend",
-  "version": "0.1.1",
+  "version": "0.1.2",
   "description": "Frontend helpers to build OpenAI Realtime voice agents",
   "type": "module",
   "main": "./dist/index.cjs",
@@ -20,7 +20,9 @@
   "files": [
     "dist",
     "bin",
-    "README.md"
+    "README.md",
+    "README.es.md",
+    "README.en.md"
   ],
   "scripts": {
     "build": "tsup src/index.ts --format cjs,esm --dts --clean",