nexus-ai-pro 0.9.0 → 1.0.0

package/API_STABILITY.md

@@ -25,6 +25,29 @@ Source files, `dist` file paths, examples, test helpers that are not exported, a
 
 The package is ESM-only and supports Node.js 22 or newer. CommonJS `require()` and deep imports into `dist` or `src` are not supported.
 
+## Realtime API stage
+
+The opt-in `nexus-ai-pro/realtime` family is public in the current development line. This includes the
+explicit `realtime/session`, `realtime/tools`, `realtime/conversation`, `realtime/openai-webrtc`,
+`realtime/openai-websocket`, `realtime/openai-server`, and `realtime/mock` subpaths. Its exported symbols,
+normalized event names, configuration fields, and conversation/export shapes follow the same patch
+compatibility rules as the rest of the package.
+
+The realtime surface is still pre-1.0. Provider-specific wire events returned by `openai-events`, values
+under `raw`, SDP/ICE behavior, and upstream model or voice availability are controlled by the provider
+and are not normalized compatibility guarantees. New normalized event variants or optional metrics may
+be added in a minor release. A necessary incompatible change during 0.x will be documented in a minor
+release with migration guidance.
+
+Platform interfaces for WebRTC and WebSocket are structural so applications can inject browser, server,
+mobile, or test adapters without a framework dependency. A documented structural member is public;
+private transport internals and unexported protocol helpers are not.
+
+`VoiceSession` and `RealtimeSession` are independent public APIs. `VoiceSession` remains a batch,
+turn-oriented transcription/completion/speech workflow. `RealtimeSession` owns a persistent transport,
+live events, interruption, and normalized realtime conversation state. Neither is a compatibility alias
+for the other.
+
 ## Deprecation process
 
 Deprecated APIs are marked with `@deprecated` in declarations and described in the changelog. During the 0.x series, removal will not normally occur earlier than the next minor release. After 1.0, removals will be reserved for major releases, except when an urgent security issue requires otherwise.

package/CHANGELOG.md

@@ -4,6 +4,30 @@ Notable changes to this project are recorded here. The format follows [Keep a Ch
 
 ## [Unreleased]
 
+### Added
+
+- An opt-in realtime package family with `realtime`, `realtime/session`, `realtime/tools`,
+  `realtime/conversation`, provider transport, server-helper, and deterministic mock subpaths. Realtime
+  code is not added to the root or batch voice import graph.
+- Framework-independent OpenAI WebRTC and WebSocket transports with injectable platform interfaces,
+  secure server-owned WebRTC negotiation, microphone and remote-audio handling, cancellation, and
+  reconnect support.
+- A persistent `RealtimeSession` and higher-level `createRealtimeAgent()` API with normalized and raw
+  events, barge-in, response cancellation, unheard-audio truncation, typed tools, confirmations,
+  idempotency keys, bounded execution, and transcript/usage state.
+- Immutable normalized conversations with JSON, OpenAI-event, text, and analytics exports, plus
+  connection, first-audio, turn, tool, interruption, reconnect, token, audio-duration, and cost metrics.
+- Realtime security controls for tool allowlists, transcript retention, PII hooks, and maximum session
+  and audio duration, with structural OpenTelemetry-compatible tracing and metrics hooks.
+- Opt-in safe-tool result caching, active-connection/audio-duration metrics, and an application-owned
+  realtime cost estimator hook without bundled pricing tables.
+- A network-free realtime scheduling example using the deterministic mock transport.
+
+### Changed
+
+- Realtime connections are documented as a separate opt-in operation family. Existing `VoiceSession`
+  behavior remains the batch transcription -> completion/tools -> speech workflow.
+
 ## [0.9.0] - 2026-07-15
 
 ### Added

package/README.md

@@ -16,6 +16,8 @@ npm install nexus-ai-pro
 ```
 
 Requires Node.js 22 or newer. `nexus-ai-pro` is ESM-only: use `import` from an ES module rather than CommonJS `require()`.
+The main Nexus pipeline is Node-oriented; the isolated realtime subpaths use portable structural
+interfaces and can be bundled for modern browsers.
 
 Install only the provider SDKs your app uses. The OpenAI SDK also powers OpenAI-compatible adapters such as OpenRouter, Groq, Mistral, DeepSeek, Azure OpenAI, LM Studio, and llama.cpp.
 
@@ -24,6 +26,8 @@ npm install openai @anthropic-ai/sdk ollama
 ```
 
 The Google Gemini provider uses Node.js `fetch`, so it does not need an extra SDK.
+Realtime transports use injected or platform WebRTC, WebSocket, and `fetch` interfaces and do not
+require the OpenAI SDK.
 
 ## Quick Start
 
@@ -77,6 +81,7 @@ You can still pass a plain `NexusAIConfig` to `new NexusAI(...)` when you want f
 - estimate tokens and cost before provider calls
 - cache exact or semantically similar prompts
 - add tools, agents, RAG context, evals, batch jobs, and queues
+- build persistent realtime voice agents with interruption, live tools, and normalized conversation state
 - keep TypeScript types around every request and response
 
 ## CLI
@@ -469,7 +474,210 @@ ai.registerVoiceProvider('private-voice', {
 });
 ```
 
-Low-level realtime transport adapters are still future work; the current voice layer covers transcription, speech generation, full request/response voice turns, and stateful voice sessions with tool calls.
+`VoiceSession` remains the batch-oriented path: each turn is transcription -> completion/tools -> optional
+speech. For a persistent connection with live microphone audio, streamed remote audio, barge-in, and
+provider events, use the separate realtime entry points below. Neither API replaces the other.
+
+## Realtime Voice Agents
+
+Realtime is opt-in and is not re-exported from the package root or from `nexus-ai-pro/voice`:
+
+```ts
+import {
+  createRealtimeAgent,
+  defineTool,
+  OpenAIRealtimeProvider,
+} from 'nexus-ai-pro/realtime';
+```
+
+The framework-independent session normalizes connection, speech, transcript, response, tool,
+interruption, error, conversation, and metrics events while retaining provider events when configured.
+The browser and server transports use structural platform interfaces, so the realtime core does not
+require React, a DOM shim, or an OpenAI SDK.
+
+### Browser WebRTC
+
+Give the browser a same-origin `sessionEndpoint`; never put a permanent provider API key in browser
+code. The endpoint receives the SDP offer and returns the SDP answer while your server owns the
+provider key and authoritative session configuration.
+
+```ts
+type BookingInput = { startTime: string; endTime: string; timezone: string };
+
+const createBooking = defineTool({
+  name: 'create_calendar_booking',
+  description: 'Create a calendar booking after the user confirms it.',
+  parameters: {
+    type: 'object',
+    properties: {
+      startTime: { type: 'string', format: 'date-time' },
+      endTime: { type: 'string', format: 'date-time' },
+      timezone: { type: 'string' },
+    },
+    required: ['startTime', 'endTime', 'timezone'],
+    additionalProperties: false,
+  },
+  requiresConfirmation: true,
+  execute: async (input: BookingInput, context) =>
+    calendar.create(input, { idempotencyKey: context.idempotencyKey, signal: context.signal }),
+});
+
+const provider = new OpenAIRealtimeProvider({
+  sessionEndpoint: '/api/realtime/session',
+});
+
+const agent = createRealtimeAgent({
+  provider,
+  model: 'gpt-realtime',
+  modalities: ['audio', 'text'],
+  instructions: 'Check availability before booking. Never write without confirmation.',
+  tools: [createBooking],
+  voice: {
+    transport: 'webrtc',
+    voice: 'alloy',
+    turnDetection: { type: 'server_vad', silenceDurationMs: 450 },
+    interruption: {
+      enabled: true,
+      cancelResponse: true,
+      truncateUnheardAudio: true,
+    },
+  },
+  toolExecution: { mode: 'automatic', timeoutMs: 8_000, maxParallelCalls: 3 },
+  reconnect: { enabled: true, maxAttempts: 3, initialDelayMs: 500 },
+  conversation: { captureTranscripts: true, captureToolCalls: true, maxRawEvents: 2_000 },
+  security: {
+    maxSessionDurationMs: 30 * 60_000,
+    maxAudioDurationMs: 20 * 60_000,
+    toolAllowlist: ['create_calendar_booking'],
+  },
+  telemetry: {
+    includeTranscripts: false,
+    estimateCost: (usage) => pricing.estimateRealtime(usage),
+  },
+});
+
+agent.on('tool.confirmation.required', ({ call }) => {
+  const approved = window.confirm(`Create booking requested by ${call.name}?`);
+  agent.confirmTool(call.callId, approved);
+});
+
+agent.on('metrics', (metrics) => {
+  console.log('speech end -> first audio', metrics.speechEndToFirstAudioMs);
+});
+
+const audioElement = document.querySelector<HTMLAudioElement>('#assistant-audio')!;
+await agent.connect({ microphone: true, audioElement });
+```
+
+WebRTC attaches the selected microphone track directly to the peer connection and assigns the remote
+media stream to `audioElement`. Browsers still require HTTPS (or localhost), user permission, and often
+a user gesture before audio playback. `sendAudio()` is intended for transports that accept binary
+chunks; normal WebRTC microphone use should stay on the media track.
+
+WebRTC first-audio timing uses the provider's `output_audio_buffer.started` event. WebSocket playback
+remains application-owned, so its player must call `markAudioPlayed()` as described below.
+
+A minimal server endpoint can use the focused server helper:
+
+```ts
+import { createOpenAIRealtimeCall } from 'nexus-ai-pro/realtime/openai-server';
+
+export async function POST(request: Request) {
+  const answerSdp = await createOpenAIRealtimeCall(await request.text(), {
+    apiKey: process.env.OPENAI_API_KEY!,
+    model: 'gpt-realtime',
+    session: {
+      instructions: 'You are a concise scheduling assistant.',
+      max_output_tokens: 1_024,
+    },
+  }, request.signal);
+
+  return new Response(answerSdp, { headers: { 'content-type': 'application/sdp' } });
+}
+```
+
+Authenticate that endpoint, authorize the tenant and requested tools, apply origin/CSRF and rate-limit
+controls, and keep its accepted session options allowlisted. The client must not choose arbitrary tools,
+instructions, limits, or credentials.
+
+### Server WebSocket
+
+Server-to-server agents can inject their WebSocket implementation and keep provider authorization in
+request headers. The package deliberately does not force a WebSocket dependency:
+
+```ts
+import {
+  OpenAIWebSocketTransport,
+  type OpenAIWebSocketFactory,
+} from 'nexus-ai-pro/realtime/openai-websocket';
+import { createRealtimeSession } from 'nexus-ai-pro/realtime/session';
+
+const webSocketFactory: OpenAIWebSocketFactory = (url, { headers, protocols }) =>
+  serverWebSocketClient.connect(url, { headers, protocols });
+
+const session = createRealtimeSession({
+  provider: 'openai',
+  model: 'gpt-realtime',
+  transport: new OpenAIWebSocketTransport({
+    apiKey: process.env.OPENAI_API_KEY!,
+    webSocketFactory,
+  }),
+  modalities: ['audio', 'text'],
+  reconnect: { enabled: true, maxAttempts: 3 },
+  connection: {
+    remoteAudioSink: (audio) => audioPlayer.enqueue(audio.data),
+  },
+});
+
+await session.connect();
+```
+
+When an application plays WebSocket audio itself, call `session.markAudioPlayed()` when the first bytes
+actually reach playback. That makes the primary latency metric—user speech ended to first assistant
+audio played—represent user experience rather than network receipt.
+
+The global browser WebSocket path accepts only an ephemeral token. Supplying a permanent API key to a
+browser adapter is rejected; use an injected server factory or an ephemeral-token provider instead.
+
+### Interruption, tools, state, and exports
+
+With interruption enabled, a `speech.started` event can stop local playback, cancel the active response,
+and, for WebSocket playback with a known position, truncate unheard assistant audio. Applications can
+also call `session.interrupt('manual')`.
+
+Realtime tools support structural `parse`, `safeParse`, or `validate` schemas, execution timeouts,
+bounded parallelism, safe-tool retries, duplicate call suppression, idempotency keys, allowlists, and
+write confirmation. A tool marked `safe: true` can also opt into cross-call caching with `cache: true`
+or `{ ttlMs, key }` when `toolExecution.cache` supplies a structural cache adapter; cache failures are
+ignored unless `cacheFailureMode: 'fail'` is configured. `requiresConfirmation` does not replace server
+authorization: re-check identity, permissions, business invariants, and idempotency in the tool
+implementation.
+
+Session state is available as immutable snapshots and four export formats:
+
+```ts
+const json = session.export('json');
+const providerEvents = session.export('openai-events');
+const transcript = session.export('text');
+const analytics = session.export('analytics');
+
+session.on('conversation.updated', (snapshot) => conversationStore.save(snapshot));
+session.on('error', (error) => telemetry.recordException(error));
+```
+
+`json` contains normalized user/assistant speech, text messages, tool calls/results, interruptions,
+errors, and latency/usage metrics. `openai-events` returns retained raw provider events; it is useful for
+diagnostics and replay but follows the upstream provider schema. Disable transcript/raw-event retention
+for sensitive workloads. `piiHook` redacts normalized conversation/event text, but deliberately does not
+rewrite raw provider exports. Telemetry transcript fields are redacted by default; `includeTranscripts`
+must be explicitly enabled. `estimateCost` accepts an application-owned pricing callback so model price
+changes are not hard-coded into the package. The built-in limits and telemetry hooks are local controls,
+not a durable session store, distributed rate limiter, or authorization system.
+
+Focused public imports are available for `realtime`, `realtime/session`, `realtime/tools`,
+`realtime/conversation`, `realtime/openai-webrtc`, `realtime/openai-websocket`,
+`realtime/openai-server`, and `realtime/mock`. The deterministic mock is suitable for tests and demos;
+see [`examples/realtime-scheduling.ts`](./examples/realtime-scheduling.ts).
 
 ## Phone Number Telephony
 
@@ -598,6 +806,7 @@ Available runtime helpers:
 - Redis and BullMQ queue adapters
 - eval runner with quality, operations, RAG, safety metrics, and optional LLM-as-judge scoring
 - voice sessions for multi-turn calls with conditional task prompts and app tools
+- realtime sessions with WebRTC/WebSocket transports, interruption, tools, metrics, and conversation exports
 - workflow templates for RAG answers, extraction, classification, comparison, support, sales, legal review, and code review
 
 Use a plain assertion for small evals, or add an LLM judge when a rubric is more useful than exact matching:
@@ -683,6 +892,8 @@ import { ContextWindowManager } from 'nexus-ai-pro/context';
 import { guardrailPolicy } from 'nexus-ai-pro/security';
 import { RedisCacheAdapter } from 'nexus-ai-pro/cache';
 import { DeepSeekProvider } from 'nexus-ai-pro/providers/deepseek';
+import { createRealtimeSession } from 'nexus-ai-pro/realtime/session';
+import { OpenAIWebRTCTransport } from 'nexus-ai-pro/realtime/openai-webrtc';
 ```
 
 Provider SDKs are optional peer dependencies. The package is ESM-only, supports Node.js 22+, and is marked with `sideEffects: false`. Only the entry points listed in the package export map are public; deep imports into `dist` or `src` are unsupported.
@@ -699,6 +910,8 @@ npm run example:basic
 npm run example:security
 npm run example:optimizer
 npm run example:agent
+npm run build && npx tsx examples/realtime-scheduling.ts
+npx tsc -p tsconfig.examples.browser.json
 ```
 
 Repository example files cover:
@@ -710,6 +923,7 @@ Repository example files cover:
 - classifier calibration
 - domain workflows
 - custom providers
+- deterministic realtime scheduling with tools, confirmation, metrics, and conversation exports
 - import examples under `examples/exports`
 
 ## Test Commands

package/ROADMAP.md

@@ -3,6 +3,39 @@
 This roadmap is a design proposal, not a compatibility promise. Stable and experimental
 surfaces are defined in [API_STABILITY.md](./API_STABILITY.md).
 
+## Delivered in the current development line: realtime voice foundation
+
+The first native realtime milestone is implemented as an opt-in package family rather than as another
+`complete()` mode or an eager dependency of `NexusAI`:
+
+- persistent provider-neutral `RealtimeSession` and higher-level `createRealtimeAgent()` APIs;
+- OpenAI WebRTC for browser microphone/remote-media connections and WebSocket for server agents;
+- structural, injectable browser/server interfaces plus a deterministic mock transport;
+- normalized speech, transcript, text, response, tool, interruption, error, conversation, metrics, and
+  retained raw-provider events;
+- barge-in, response cancellation, optional unheard-audio truncation, reconnect backoff, and cancellation;
+- typed/validated tools with automatic or manual execution, confirmation, timeouts, bounded parallelism,
+  safe retries, opt-in read caching, duplicate suppression, allowlists, and idempotency keys;
+- normalized JSON conversation state and OpenAI-event, text, and analytics exports;
+- server-owned SDP/client-secret helpers, retention controls, PII hooks, session/audio limits, and
+  OpenTelemetry-compatible hooks.
+
+Current limitations are intentional and should remain visible:
+
+- OpenAI is the only native realtime provider; provider-neutral contracts have not yet been validated
+  against a second wire protocol.
+- The package supplies framework-independent transports, not React hooks, UI/audio-player components,
+  mobile SDK wrappers, SIP, or a telephone gateway.
+- WebRTC requires application-owned HTTPS session negotiation and browser permission/autoplay handling.
+  WebSocket consumers are responsible for audio capture/playback and for injecting a server socket
+  implementation when authenticated headers are required.
+- Conversation snapshots and exports are in memory. Durable storage, distributed deduplication,
+  cross-process session recovery, and provider-state replay remain application responsibilities.
+- Compatibility/load validation across Safari, iOS Safari, Firefox, Android Chrome, long conversations,
+  and large concurrent session counts is still required before making broad production-scale claims.
+- Realtime video tracks, SIP transports, durable cost accounting beyond the configurable local estimate,
+  and a published latency benchmark methodology remain follow-on work.
+
 ## Recommended next release: first-class image generation
 
 Image generation should be a separate operation family rather than another variation of
@@ -164,7 +197,8 @@ code should only perform the provider call and normalize its result.
 
 1. True mixed text/asset outputs and tool results that pass asset references without base64 JSON.
 2. OCR, captioning, visual question answering, image embeddings, and visual moderation.
-3. Realtime voice/video sessions with interruption handling, turn detection, and live tool calls.
+3. Realtime follow-ons: a second provider, SIP/video transports, durable recovery, browser compatibility
+   automation, and published latency/load benchmarks. The native voice foundation is delivered above.
 4. Video generation through the same asynchronous operation, job, and asset contracts.
 5. Media search and RAG using image embeddings.
 6. MCP client/server adapters with asset and tool interoperability.
@@ -185,4 +219,3 @@ code should only perform the provider call and normalize its result.
 6. Add true multimodal outputs and media evaluation.
 7. Promote the media API only after packed-package tests, provider conformance, and image evals
    cover the supported capability matrix.
-

package/SECURITY.md

@@ -13,3 +13,34 @@ Include the affected version, impact, a minimal reproduction, and any suggested
 ## Scope and safe operation
 
 Nexus guardrails reduce common AI application risks, but they are not a complete authorization or security boundary. Applications remain responsible for access control, provider-side safety controls, tool authorization, network egress policy, secret storage, and review of high-impact model actions.
+
+## Realtime voice security
+
+- Never ship a permanent provider API key to a browser or mobile client. For WebRTC, configure a
+  same-origin `sessionEndpoint` that performs provider negotiation or returns a short-lived client
+  secret. For WebSocket API-key authentication, run the transport on a trusted server and inject a
+  WebSocket factory that can set authorization headers.
+- Authenticate and authorize the session endpoint before contacting the provider. Restrict origins,
+  apply CSRF and rate-limit controls, cap request size and duration, and derive the model, tools,
+  instructions, retention, and safety settings from trusted server policy rather than client input.
+- Treat microphone permission, remote audio, provider data-channel events, transcripts, and raw events
+  as sensitive data. Request microphone access only after a user action, stop owned tracks on cleanup,
+  minimize retention, and avoid putting transcripts or audio in logs and telemetry by default.
+- Use `security.maxSessionDurationMs`, `security.maxAudioDurationMs`, `security.toolAllowlist`,
+  `security.retainTranscripts`, `security.piiHook`, and bounded raw-event retention as defense-in-depth.
+  These process-local controls do not replace tenant quotas, durable retention enforcement, or a DLP
+  system.
+- A realtime tool confirmation is a user-experience checkpoint, not authorization. Every read and write
+  tool must enforce tenant access and validate arguments server-side. Writes should require explicit
+  confirmation, use the supplied idempotency key, and remain safe under retries and duplicate provider
+  events.
+- Redact API keys, ephemeral tokens, SDP diagnostics, tool arguments/results, and PII from errors and
+  telemetry. Realtime telemetry transcript fields are redacted by default; set
+  `telemetry.includeTranscripts: true` only when the destination and retention policy are explicitly
+  approved. `piiHook` applies to normalized events and conversation state, not retained raw events.
+- Reconnects can repeat provider events or tool requests and do not guarantee restoration of remote
+  conversation state. Keep write tools idempotent, deduplicate by call ID/idempotency key, and surface
+  recovery failures to the user.
+- Browser WebRTC still requires a backend security boundary. A direct media path reduces latency; it
+  does not remove the need for server-owned credentials, session policy, tool authorization, abuse
+  monitoring, and provider-side safety controls.

package/dist/realtime/conversation.d.ts

@@ -0,0 +1,19 @@
+import { type RealtimeIdFactory } from './id.js';
+import type { ConversationMetrics, RealtimeAnalyticsExport, RealtimeConversation, RealtimeConversationExportFormat, RealtimeConversationOptions, RealtimeEvent, RealtimeProviderName, RealtimeServerEvent } from './types.js';
+export interface CreateRealtimeConversationOptions {
+    id?: string;
+    provider: RealtimeProviderName;
+    model: string;
+    startedAt?: string;
+    metadata?: Record<string, unknown>;
+    idFactory?: RealtimeIdFactory;
+}
+export interface RealtimeConversationReducerOptions extends RealtimeConversationOptions {
+    idFactory?: RealtimeIdFactory;
+}
+export declare function createConversationMetrics(): ConversationMetrics;
+export declare function createRealtimeConversation(options: CreateRealtimeConversationOptions): RealtimeConversation;
+export declare function reduceRealtimeConversation(conversation: RealtimeConversation, event: RealtimeEvent, options?: RealtimeConversationReducerOptions): RealtimeConversation;
+export declare function withConversationMetrics(conversation: RealtimeConversation, metrics: Partial<ConversationMetrics>): RealtimeConversation;
+export declare function snapshotRealtimeConversation(conversation: RealtimeConversation): RealtimeConversation;
+export declare function exportRealtimeConversation(conversation: RealtimeConversation, format: RealtimeConversationExportFormat, rawEvents?: RealtimeServerEvent[]): RealtimeConversation | RealtimeServerEvent[] | string | RealtimeAnalyticsExport;