nexus-ai-pro 0.8.1 → 1.0.0

package/API_STABILITY.md

@@ -0,0 +1,55 @@
+# API Stability Policy
+
+This policy describes compatibility guarantees for `nexus-ai-pro` beginning with version 0.9.0.
+
+## Release stages
+
+The package is pre-1.0. Minor releases may contain breaking changes when they are necessary, but those changes will be called out in the changelog with migration guidance. Patch releases are intended to remain backward compatible.
+
+The following surfaces are public and covered by this policy:
+
+- exports documented in the root package or an explicit `package.json` subpath;
+- exported TypeScript types and runtime symbols;
+- documented configuration keys, response shapes, and CLI commands.
+
+Source files, `dist` file paths, examples, test helpers that are not exported, and undocumented implementation details are not public API. Import only from `nexus-ai-pro` or one of its explicit subpaths.
+
+## Compatibility rules
+
+- A public symbol will not be removed or incompatibly changed in a patch release.
+- Deprecations will be documented before removal whenever practical.
+- New optional fields, providers, exports, error subclasses, and enum-like union members may be added in a minor release.
+- TypeScript improvements that expose previously invalid usage may arrive in a patch release.
+- Security fixes may tighten validation or blocking behavior in a patch release when preserving the old behavior would leave users exposed.
+- Provider behavior can change when an upstream API changes; Nexus will preserve its normalized request and response contracts where possible.
+
+The package is ESM-only and supports Node.js 22 or newer. CommonJS `require()` and deep imports into `dist` or `src` are not supported.
+
+## Realtime API stage
+
+The opt-in `nexus-ai-pro/realtime` family is public in the current development line. This includes the
+explicit `realtime/session`, `realtime/tools`, `realtime/conversation`, `realtime/openai-webrtc`,
+`realtime/openai-websocket`, `realtime/openai-server`, and `realtime/mock` subpaths. Its exported symbols,
+normalized event names, configuration fields, and conversation/export shapes follow the same patch
+compatibility rules as the rest of the package.
+
+The realtime surface is still pre-1.0. Provider-specific wire events returned by `openai-events`, values
+under `raw`, SDP/ICE behavior, and upstream model or voice availability are controlled by the provider
+and are not normalized compatibility guarantees. New normalized event variants or optional metrics may
+be added in a minor release. A necessary incompatible change during 0.x will be documented in a minor
+release with migration guidance.
+
+Platform interfaces for WebRTC and WebSocket are structural so applications can inject browser, server,
+mobile, or test adapters without a framework dependency. A documented structural member is public;
+private transport internals and unexported protocol helpers are not.
+
+`VoiceSession` and `RealtimeSession` are independent public APIs. `VoiceSession` remains a batch,
+turn-oriented transcription/completion/speech workflow. `RealtimeSession` owns a persistent transport,
+live events, interruption, and normalized realtime conversation state. Neither is a compatibility alias
+for the other.
+
+## Deprecation process
+
+Deprecated APIs are marked with `@deprecated` in declarations and described in the changelog. During the 0.x series, removal will not normally occur earlier than the next minor release. After 1.0, removals will be reserved for major releases, except when an urgent security issue requires otherwise.
+
+Report accidental compatibility regressions through the project issue tracker. Security issues should follow [SECURITY.md](./SECURITY.md).

package/CHANGELOG.md

@@ -0,0 +1,59 @@
+# Changelog
+
+Notable changes to this project are recorded here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and releases use semantic versioning while accounting for the pre-1.0 stability policy.
+
+## [Unreleased]
+
+### Added
+
+- An opt-in realtime package family with `realtime`, `realtime/session`, `realtime/tools`,
+  `realtime/conversation`, provider transport, server-helper, and deterministic mock subpaths. Realtime
+  code is not added to the root or batch voice import graph.
+- Framework-independent OpenAI WebRTC and WebSocket transports with injectable platform interfaces,
+  secure server-owned WebRTC negotiation, microphone and remote-audio handling, cancellation, and
+  reconnect support.
+- A persistent `RealtimeSession` and higher-level `createRealtimeAgent()` API with normalized and raw
+  events, barge-in, response cancellation, unheard-audio truncation, typed tools, confirmations,
+  idempotency keys, bounded execution, and transcript/usage state.
+- Immutable normalized conversations with JSON, OpenAI-event, text, and analytics exports, plus
+  connection, first-audio, turn, tool, interruption, reconnect, token, audio-duration, and cost metrics.
+- Realtime security controls for tool allowlists, transcript retention, PII hooks, and maximum session
+  and audio duration, with structural OpenTelemetry-compatible tracing and metrics hooks.
+- Opt-in safe-tool result caching, active-connection/audio-duration metrics, and an application-owned
+  realtime cost estimator hook without bundled pricing tables.
+- A network-free realtime scheduling example using the deterministic mock transport.
+
+### Changed
+
+- Realtime connections are documented as a separate opt-in operation family. Existing `VoiceSession`
+  behavior remains the batch transcription -> completion/tools -> speech workflow.
+
+## [0.9.0] - 2026-07-15
+
+### Added
+
+- Full JSON Schema validation through Ajv and standard format validation through `ajv-formats`.
+- Node.js 22 and 24 CI gates for linting, builds, tests, consumer type checks, and packed-package installation.
+- OIDC trusted publishing with npm provenance for tagged releases.
+- API stability, security, and contribution policies.
+- Biome formatting/lint gates, test/example type checks, and built-in Node.js coverage thresholds.
+
+### Changed
+
+- The supported runtime is now Node.js 22 or newer and the package is explicitly ESM-only.
+- Provider, cache, and job entry points now use an explicit export map instead of wildcard exports.
+- Published tarballs contain the runtime and essential package documentation, excluding repository examples and marketing assets.
+- Provider peer dependency ranges are bounded to tested SDK lines.
+- Build and publish scripts always clean and rebuild `dist` before packing.
+- Security-enabled output streams validate the complete bounded response before yielding chunks.
+
+### Fixed
+
+- Blocking output policies now stop unsafe completions instead of returning blocked content as successful output.
+- Streaming redaction protects values that cross provider chunk boundaries.
+- Provider timeouts abort in-flight completion and streaming requests.
+- URL fetching rejects unsafe private-network targets and limits response reads.
+- CLI and audit findings no longer disclose detected secret values.
+
+[Unreleased]: https://github.com/mkhitar-abrahamyan/nexus-ai/compare/v0.9.0...HEAD
+[0.9.0]: https://github.com/mkhitar-abrahamyan/nexus-ai/releases/tag/v0.9.0

package/README.md

@@ -7,6 +7,7 @@ Use the whole pipeline for production AI features, or turn pieces off when you o
 - NPM: https://www.npmjs.com/package/nexus-ai-pro
 - GitHub: https://github.com/mkhitar-abrahamyan/nexus-ai
 - Full technical notes and manual test cases: [NEXUS.md](./NEXUS.md)
+- Planned image generation and infrastructure work: [ROADMAP.md](./ROADMAP.md)
 
 ## Install
 
@@ -14,6 +15,10 @@ Use the whole pipeline for production AI features, or turn pieces off when you o
 npm install nexus-ai-pro
 ```
 
+Requires Node.js 22 or newer. `nexus-ai-pro` is ESM-only: use `import` from an ES module rather than CommonJS `require()`.
+The main Nexus pipeline is Node-oriented; the isolated realtime subpaths use portable structural
+interfaces and can be bundled for modern browsers.
+
 Install only the provider SDKs your app uses. The OpenAI SDK also powers OpenAI-compatible adapters such as OpenRouter, Groq, Mistral, DeepSeek, Azure OpenAI, LM Studio, and llama.cpp.
 
 ```bash
@@ -21,6 +26,8 @@ npm install openai @anthropic-ai/sdk ollama
 ```
 
 The Google Gemini provider uses Node.js `fetch`, so it does not need an extra SDK.
+Realtime transports use injected or platform WebRTC, WebSocket, and `fetch` interfaces and do not
+require the OpenAI SDK.
 
 ## Quick Start
 
@@ -74,6 +81,7 @@ You can still pass a plain `NexusAIConfig` to `new NexusAI(...)` when you want f
 - estimate tokens and cost before provider calls
 - cache exact or semantically similar prompts
 - add tools, agents, RAG context, evals, batch jobs, and queues
+- build persistent realtime voice agents with interruption, live tools, and normalized conversation state
 - keep TypeScript types around every request and response
 
 ## CLI
@@ -246,6 +254,12 @@ for await (const chunk of stream) {
 
 All providers normalize streaming chunks to `text`, `tool_call`, `done`, or `error`.
 
+When security is enabled, output streams are correctness-first: Nexus buffers and validates the
+complete output before yielding any chunk, up to 1 MiB or 100,000 chunks. This prevents secrets,
+PII, and blocked phrases split across provider chunks from leaking partially, at the cost of
+token-by-token latency. `security: 'off'` preserves immediate streaming when that trade-off is
+explicitly acceptable for the application.
+
 ### Tools and Agents
 
 ```ts
@@ -460,7 +474,210 @@ ai.registerVoiceProvider('private-voice', {
 });
 ```
 
-Low-level realtime transport adapters are still future work; the current voice layer covers transcription, speech generation, full request/response voice turns, and stateful voice sessions with tool calls.
+`VoiceSession` remains the batch-oriented path: each turn is transcription -> completion/tools -> optional
+speech. For a persistent connection with live microphone audio, streamed remote audio, barge-in, and
+provider events, use the separate realtime entry points below. Neither API replaces the other.
+
+## Realtime Voice Agents
+
+Realtime is opt-in and is not re-exported from the package root or from `nexus-ai-pro/voice`:
+
+```ts
+import {
+  createRealtimeAgent,
+  defineTool,
+  OpenAIRealtimeProvider,
+} from 'nexus-ai-pro/realtime';
+```
+
+The framework-independent session normalizes connection, speech, transcript, response, tool,
+interruption, error, conversation, and metrics events while retaining provider events when configured.
+The browser and server transports use structural platform interfaces, so the realtime core does not
+require React, a DOM shim, or an OpenAI SDK.
+
+### Browser WebRTC
+
+Give the browser a same-origin `sessionEndpoint`; never put a permanent provider API key in browser
+code. The endpoint receives the SDP offer and returns the SDP answer while your server owns the
+provider key and authoritative session configuration.
+
+```ts
+type BookingInput = { startTime: string; endTime: string; timezone: string };
+
+const createBooking = defineTool({
+  name: 'create_calendar_booking',
+  description: 'Create a calendar booking after the user confirms it.',
+  parameters: {
+    type: 'object',
+    properties: {
+      startTime: { type: 'string', format: 'date-time' },
+      endTime: { type: 'string', format: 'date-time' },
+      timezone: { type: 'string' },
+    },
+    required: ['startTime', 'endTime', 'timezone'],
+    additionalProperties: false,
+  },
+  requiresConfirmation: true,
+  execute: async (input: BookingInput, context) =>
+    calendar.create(input, { idempotencyKey: context.idempotencyKey, signal: context.signal }),
+});
+
+const provider = new OpenAIRealtimeProvider({
+  sessionEndpoint: '/api/realtime/session',
+});
+
+const agent = createRealtimeAgent({
+  provider,
+  model: 'gpt-realtime',
+  modalities: ['audio', 'text'],
+  instructions: 'Check availability before booking. Never write without confirmation.',
+  tools: [createBooking],
+  voice: {
+    transport: 'webrtc',
+    voice: 'alloy',
+    turnDetection: { type: 'server_vad', silenceDurationMs: 450 },
+    interruption: {
+      enabled: true,
+      cancelResponse: true,
+      truncateUnheardAudio: true,
+    },
+  },
+  toolExecution: { mode: 'automatic', timeoutMs: 8_000, maxParallelCalls: 3 },
+  reconnect: { enabled: true, maxAttempts: 3, initialDelayMs: 500 },
+  conversation: { captureTranscripts: true, captureToolCalls: true, maxRawEvents: 2_000 },
+  security: {
+    maxSessionDurationMs: 30 * 60_000,
+    maxAudioDurationMs: 20 * 60_000,
+    toolAllowlist: ['create_calendar_booking'],
+  },
+  telemetry: {
+    includeTranscripts: false,
+    estimateCost: (usage) => pricing.estimateRealtime(usage),
+  },
+});
+
+agent.on('tool.confirmation.required', ({ call }) => {
+  const approved = window.confirm(`Create booking requested by ${call.name}?`);
+  agent.confirmTool(call.callId, approved);
+});
+
+agent.on('metrics', (metrics) => {
+  console.log('speech end -> first audio', metrics.speechEndToFirstAudioMs);
+});
+
+const audioElement = document.querySelector<HTMLAudioElement>('#assistant-audio')!;
+await agent.connect({ microphone: true, audioElement });
+```
+
+WebRTC attaches the selected microphone track directly to the peer connection and assigns the remote
+media stream to `audioElement`. Browsers still require HTTPS (or localhost), user permission, and often
+a user gesture before audio playback. `sendAudio()` is intended for transports that accept binary
+chunks; normal WebRTC microphone use should stay on the media track.
+
+WebRTC first-audio timing uses the provider's `output_audio_buffer.started` event. WebSocket playback
+remains application-owned, so its player must call `markAudioPlayed()` as described below.
+
+A minimal server endpoint can use the focused server helper:
+
+```ts
+import { createOpenAIRealtimeCall } from 'nexus-ai-pro/realtime/openai-server';
+
+export async function POST(request: Request) {
+  const answerSdp = await createOpenAIRealtimeCall(await request.text(), {
+    apiKey: process.env.OPENAI_API_KEY!,
+    model: 'gpt-realtime',
+    session: {
+      instructions: 'You are a concise scheduling assistant.',
+      max_output_tokens: 1_024,
+    },
+  }, request.signal);
+
+  return new Response(answerSdp, { headers: { 'content-type': 'application/sdp' } });
+}
+```
+
+Authenticate that endpoint, authorize the tenant and requested tools, apply origin/CSRF and rate-limit
+controls, and keep its accepted session options allowlisted. The client must not choose arbitrary tools,
+instructions, limits, or credentials.
+
+### Server WebSocket
+
+Server-to-server agents can inject their WebSocket implementation and keep provider authorization in
+request headers. The package deliberately does not force a WebSocket dependency:
+
+```ts
+import {
+  OpenAIWebSocketTransport,
+  type OpenAIWebSocketFactory,
+} from 'nexus-ai-pro/realtime/openai-websocket';
+import { createRealtimeSession } from 'nexus-ai-pro/realtime/session';
+
+const webSocketFactory: OpenAIWebSocketFactory = (url, { headers, protocols }) =>
+  serverWebSocketClient.connect(url, { headers, protocols });
+
+const session = createRealtimeSession({
+  provider: 'openai',
+  model: 'gpt-realtime',
+  transport: new OpenAIWebSocketTransport({
+    apiKey: process.env.OPENAI_API_KEY!,
+    webSocketFactory,
+  }),
+  modalities: ['audio', 'text'],
+  reconnect: { enabled: true, maxAttempts: 3 },
+  connection: {
+    remoteAudioSink: (audio) => audioPlayer.enqueue(audio.data),
+  },
+});
+
+await session.connect();
+```
+
+When an application plays WebSocket audio itself, call `session.markAudioPlayed()` when the first bytes
+actually reach playback. That makes the primary latency metric—user speech ended to first assistant
+audio played—represent user experience rather than network receipt.
+
+The global browser WebSocket path accepts only an ephemeral token. Supplying a permanent API key to a
+browser adapter is rejected; use an injected server factory or an ephemeral-token provider instead.
+
+### Interruption, tools, state, and exports
+
+With interruption enabled, a `speech.started` event can stop local playback, cancel the active response,
+and, for WebSocket playback with a known position, truncate unheard assistant audio. Applications can
+also call `session.interrupt('manual')`.
+
+Realtime tools support structural `parse`, `safeParse`, or `validate` schemas, execution timeouts,
+bounded parallelism, safe-tool retries, duplicate call suppression, idempotency keys, allowlists, and
+write confirmation. A tool marked `safe: true` can also opt into cross-call caching with `cache: true`
+or `{ ttlMs, key }` when `toolExecution.cache` supplies a structural cache adapter; cache failures are
+ignored unless `cacheFailureMode: 'fail'` is configured. `requiresConfirmation` does not replace server
+authorization: re-check identity, permissions, business invariants, and idempotency in the tool
+implementation.
+
+Session state is available as immutable snapshots and four export formats:
+
+```ts
+const json = session.export('json');
+const providerEvents = session.export('openai-events');
+const transcript = session.export('text');
+const analytics = session.export('analytics');
+
+session.on('conversation.updated', (snapshot) => conversationStore.save(snapshot));
+session.on('error', (error) => telemetry.recordException(error));
+```
+
+`json` contains normalized user/assistant speech, text messages, tool calls/results, interruptions,
+errors, and latency/usage metrics. `openai-events` returns retained raw provider events; it is useful for
+diagnostics and replay but follows the upstream provider schema. Disable transcript/raw-event retention
+for sensitive workloads. `piiHook` redacts normalized conversation/event text, but deliberately does not
+rewrite raw provider exports. Telemetry transcript fields are redacted by default; `includeTranscripts`
+must be explicitly enabled. `estimateCost` accepts an application-owned pricing callback so model price
+changes are not hard-coded into the package. The built-in limits and telemetry hooks are local controls,
+not a durable session store, distributed rate limiter, or authorization system.
+
+Focused public imports are available for `realtime`, `realtime/session`, `realtime/tools`,
+`realtime/conversation`, `realtime/openai-webrtc`, `realtime/openai-websocket`,
+`realtime/openai-server`, and `realtime/mock`. The deterministic mock is suitable for tests and demos;
+see [`examples/realtime-scheduling.ts`](./examples/realtime-scheduling.ts).
 
 ## Phone Number Telephony
 
@@ -559,6 +776,12 @@ Useful helpers:
 - URL and tool allowlist checks
 - reusable policies through `guardrailPolicy(...)`
 - `hardenPrompt(...)` for clearly delimiting untrusted input
+- `createFetchUrlTool(...)` with DNS pinning, redirect validation, private-network blocking, and
+  bounded response reads
+
+CLI scans and audit records redact detected values by default. `nexus scan --reveal-values` is
+restricted to an interactive terminal; raw audit data requires both `includeSensitiveData: true`
+and an explicit custom sink.
 
 ## Caching, Jobs, and Evals
 
@@ -583,6 +806,7 @@ Available runtime helpers:
 - Redis and BullMQ queue adapters
 - eval runner with quality, operations, RAG, safety metrics, and optional LLM-as-judge scoring
 - voice sessions for multi-turn calls with conditional task prompts and app tools
+- realtime sessions with WebRTC/WebSocket transports, interruption, tools, metrics, and conversation exports
 - workflow templates for RAG answers, extraction, classification, comparison, support, sales, legal review, and code review
 
 Use a plain assertion for small evals, or add an LLM judge when a rubric is more useful than exact matching:
@@ -668,9 +892,11 @@ import { ContextWindowManager } from 'nexus-ai-pro/context';
 import { guardrailPolicy } from 'nexus-ai-pro/security';
 import { RedisCacheAdapter } from 'nexus-ai-pro/cache';
 import { DeepSeekProvider } from 'nexus-ai-pro/providers/deepseek';
+import { createRealtimeSession } from 'nexus-ai-pro/realtime/session';
+import { OpenAIWebRTCTransport } from 'nexus-ai-pro/realtime/openai-webrtc';
 ```
 
-Provider SDKs are optional peer dependencies. The package is ESM-first and marked with `sideEffects: false`.
+Provider SDKs are optional peer dependencies. The package is ESM-only, supports Node.js 22+, and is marked with `sideEffects: false`. Only the entry points listed in the package export map are public; deep imports into `dist` or `src` are unsupported.
 
 ## Examples
 
@@ -684,9 +910,11 @@ npm run example:basic
 npm run example:security
 npm run example:optimizer
 npm run example:agent
+npm run build && npx tsx examples/realtime-scheduling.ts
+npx tsc -p tsconfig.examples.browser.json
 ```
 
-Included example files cover:
+Repository example files cover:
 
 - OpenTelemetry with Node HTTP and Express
 - BullMQ workers
@@ -695,25 +923,34 @@ Included example files cover:
 - classifier calibration
 - domain workflows
 - custom providers
+- deterministic realtime scheduling with tools, confirmation, metrics, and conversation exports
 - import examples under `examples/exports`
 
 ## Test Commands
 
 ```bash
 npm install
-npm run build
-npm test
-npm run test:types
-npm run test:clean-install
-npm run test:conformance:mock
+npm run check
+npm run check:release
 ```
 
+`check` runs formatting and lint gates, source/test/example type checks, the build, unit and mock
+conformance tests, coverage thresholds, package import checks, and an external type-consumer test.
+`check:release` additionally verifies the dry-run tarball and a clean packed-package install.
+
 Real provider conformance is opt-in:
 
 ```bash
 npm run test:conformance:real
 ```
 
+## Roadmap
+
+The next recommended feature family is first-class image generation and editing through a
+provider-neutral `ImageManager`, portable asset types, durable operation handles, visual safety,
+asset storage, and media-specific evaluations. The design and prioritized infrastructure backlog
+are in [ROADMAP.md](./ROADMAP.md).
+
 ## Production Notes
 
 - Use direct or rules-based routing when model choice is already known.