@ai-ecoverse/pi-audio 1.10.0 → 1.12.0

package/README.md

@@ -23,6 +23,8 @@ pi-audio does not apply that patch. Hosts such as slicc-agent record monkeypatch
 
 `@ai-ecoverse/pi-audio/durable` appends a `voice.truncate` entry for a `VoiceCut`. The entry has no model messages. Its `edits` replace the target assistant message with the text heard so far and keep tool calls. The stored entry stays intact, and pi-durable derives the shorter message for the next model request. `@earendil-works/pi-durable` is an optional peer (`^1.1.0`); the root package does not import it. `VoiceCut.entryId` is the decimal string of pi-durable's numeric entry id.
 
+`@ai-ecoverse/pi-audio/xai` registers xAI speech adapters when `registerXAI()` is called. `grok-voice-transcribe-2.0` posts multipart audio to `/stt` (fields before `file`) and streams raw binary frames on `wss://…/stt` after `transcript.created`, with Smart Turn (`turnDetection: 'semantic'`). The catalog id is shared, so `audioCost` uses the streaming rate of $0.20 per hour. `grok-voice-tts-1.0` posts JSON to `/tts` for a string and streams `text.delta` / `audio.delta` on `wss://…/tts` for an async iterable (`textStreaming: true`). Voices are eve, ara, rex, sal, and leo. Codecs are mp3, pcm, wav, mulaw, and alaw.
+
 `@ai-ecoverse/pi-audio/openai` registers OpenAI and OpenRouter speech adapters when `registerOpenAI()` is called. Batch transcription posts multipart audio to `/audio/transcriptions`. OpenRouter (`provider` `openrouter`, or a base URL on `openrouter.ai`) posts JSON `input_audio` instead. Speech posts JSON to `/audio/speech` and reads the raw audio response. OpenAI can return pcm, mp3, wav, opus, or flac. OpenRouter returns pcm or mp3. Realtime transcription uses a WebSocket at `/realtime?intent=transcription` for `gpt-realtime-whisper` and `gpt-live-transcribe`. The catalog key is type, provider, and id, so the batch model keeps the id `gpt-4o-transcribe`.
 
 `convertAudio` and `convertAudioSource` convert between `AudioFormat`s: PCM16, float32, G.711 μ-law and A-law, mono and stereo, sample rate, and PCM16 WAV. Opus, MP3 and FLAC pass through only when the format does not change. `encodeAudioBase64` and `decodeAudioBase64` are the wire helpers for JSON protocols. `AudioChunk.data` stays binary. The same converter runs in an AudioWorklet from `@ai-ecoverse/pi-audio/worklet` (`AUDIO_CONVERT_PROCESSOR`, `registerAudioConvertWorklet`). Adapters declare `inputFormats` and `outputFormats` and do not resample on their own.
@@ -35,6 +37,10 @@ pi-audio does not apply that patch. Hosts such as slicc-agent record monkeypatch
 
 `RealtimeLedger` meters one session. Input seconds come from `appendAudio`. Output seconds come from each `output_audio` event: PCM, mulaw, and alaw from the byte length, and a compressed encoding from character alignment. `sendText` and each delivered `inject` add one text message. Tool results do not. `close()` prices those meters with `priceAudio`. Sum that usage once. `response_done.usage` is the adapter's per-response figure, not a second copy of the session total.
 
+`@ai-ecoverse/pi-audio/gemini-live` registers `gemini-3.8-live`, `gemini-live-2.5-flash-native-audio`, and `gemini-3.1-flash-live-preview`. Setup enables sliding-window compression and session resumption. `gemini-3.8-live` leaves function behavior at the model default and accepts `vocabulary` as `customVocabulary`. The 2.5 model declares `NON_BLOCKING`. The 3.1 preview declares `BLOCKING`. `respond` maps `Delivery` onto `scheduling` (`INTERRUPT`, `WHEN_IDLE`, `SILENT`) and returns `status`, `retryable`, and `message`. A `goAway` reconnects with the latest resumable handle. There is no truncate call. The adapter takes a recorded `wire` and never calls Google.
+
+`mintRealtimeToken(model, constraints, options)` returns `{ value, expiresAt, model }` and calls `options.fetch` (or `globalThis.fetch`). OpenAI `POST /v1/realtime/client_secrets` sends a realtime session with instructions, voice, and tools. xAI `POST /v1/realtime/client_secrets` sends only `expires_after.seconds` of 300. Gemini `POST /v1beta/auth_tokens` locks `models/<id>`, instructions, voice, and tools in `liveConnectConstraints` and returns `name`, which the live adapter sends as `access_token`. ElevenLabs `GET /v1/convai/conversation/get-signed-url` takes `agent_id` (or `GET /conversation/token` when the only transport is WebRTC) and expires in 15 minutes. Fish `POST /v1/agent/sessions` sends `agent_id` plus `overrides.system_prompt` and `overrides.voice_id`. `value` is the LiveKit participant token. `expiresAt` is milliseconds. A numeric `expires_at` below `1e12` is unix seconds.
+
 `talkerTools` is `ask_agent` and `read`. pi-audio defines `askAgent` (a pi `Tool` whose parameters are `request` and optional `context`) and the `Thinker` interface. The host implements `Thinker` and supplies `read`. `bindThinker` runs direct tools with `directTimeoutMs` and truncates their text to `directMaxChars`. On a blocking `ask_agent` it answers with `ack` immediately and injects the result later. On an async call it `respond`s with the result and `options.result`. Progress is injected with `options.progress`. `tool_call_cancelled` aborts the thinker. The binding also yields `thinker_progress` / `thinker_result` captions and `thinking` / `tool` / `idle` phases.
 
 ## Development

package/dist/gemini-live/index.js

@@ -0,0 +1,71 @@
+import { registerVoiceModel } from "../models.js";
+import { registerRealtimeApiProvider } from "../registry.js";
+import { connectGeminiSession, pcmFormat } from "./session.js";
+const voices = [
+  { id: "Aoede", name: "Aoede" },
+  { id: "Puck", name: "Puck" },
+  { id: "Charon", name: "Charon" },
+  { id: "Kore", name: "Kore" },
+  { id: "Fenrir", name: "Fenrir" },
+  { id: "Zephyr", name: "Zephyr" }
+];
+const audioCost = { perInputMinute: 5e-3, perOutputMinute: 0.018 };
+const endpoint = "wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent";
+function liveModel(id, name, toolCalls, video) {
+  return {
+    id,
+    name,
+    api: "gemini-live",
+    provider: "google",
+    baseUrl: endpoint,
+    type: "realtime",
+    transports: ["websocket"],
+    toolCalls,
+    inputFormat: pcmFormat(16e3),
+    outputFormat: pcmFormat(24e3),
+    video: video ? { maxFps: 1 } : void 0,
+    maxSessionMs: 9e5,
+    resumable: true,
+    ephemeralAuth: true,
+    voices,
+    provisioning: "none",
+    delivery: { interrupt: "native", when_idle: "native", silent: "native" },
+    truncation: "none",
+    liveToolUpdate: false,
+    cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
+    audioCost
+  };
+}
+const gemini38Live = liveModel("gemini-3.8-live", "Gemini 3.8 Live", "async", true);
+const gemini25NativeAudio = liveModel(
+  "gemini-live-2.5-flash-native-audio",
+  "Gemini 2.5 Flash Live",
+  "async",
+  true
+);
+const gemini31FlashLive = liveModel(
+  "gemini-3.1-flash-live-preview",
+  "Gemini 3.1 Flash Live",
+  "blocking",
+  false
+);
+function registerGeminiLive(options = {}) {
+  const unregisterModels = [gemini38Live, gemini25NativeAudio, gemini31FlashLive].map(
+    registerVoiceModel
+  );
+  const unregisterApi = registerRealtimeApiProvider({
+    api: "gemini-live",
+    connect: (model, context, connectOptions) => connectGeminiSession(model, context, connectOptions, options)
+  });
+  return () => {
+    for (const unregister of unregisterModels) unregister();
+    unregisterApi();
+  };
+}
+export {
+  gemini25NativeAudio,
+  gemini31FlashLive,
+  gemini38Live,
+  registerGeminiLive
+};
+//# sourceMappingURL=index.js.map

package/dist/gemini-live/index.js.map

@@ -0,0 +1,7 @@
+{
+  "version": 3,
+  "sources": ["../../src/gemini-live/index.ts"],
+  "sourcesContent": ["import type { RealtimeModel } from '../models.ts';\nimport { registerVoiceModel } from '../models.ts';\nimport { registerRealtimeApiProvider } from '../registry.ts';\nimport { connectGeminiSession, type GeminiOptions, pcmFormat } from './session.ts';\n\nconst voices = [\n  { id: 'Aoede', name: 'Aoede' },\n  { id: 'Puck', name: 'Puck' },\n  { id: 'Charon', name: 'Charon' },\n  { id: 'Kore', name: 'Kore' },\n  { id: 'Fenrir', name: 'Fenrir' },\n  { id: 'Zephyr', name: 'Zephyr' },\n];\n\nconst audioCost = { perInputMinute: 0.005, perOutputMinute: 0.018 };\n\nconst endpoint =\n  'wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent';\n\nfunction liveModel(\n  id: string,\n  name: string,\n  toolCalls: RealtimeModel<'gemini-live'>['toolCalls'],\n  video: boolean\n): RealtimeModel<'gemini-live'> {\n  return {\n    id,\n    name,\n    api: 'gemini-live',\n    provider: 'google',\n    baseUrl: endpoint,\n    type: 'realtime',\n    transports: ['websocket'],\n    toolCalls,\n    inputFormat: pcmFormat(16000),\n    outputFormat: pcmFormat(24000),\n    video: video ? { maxFps: 1 } : undefined,\n    maxSessionMs: 900_000,\n    resumable: true,\n    ephemeralAuth: true,\n    voices,\n    provisioning: 'none',\n    delivery: { interrupt: 'native', when_idle: 'native', silent: 'native' },\n    truncation: 'none',\n    liveToolUpdate: false,\n    cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },\n    audioCost,\n  };\n}\n\nexport const gemini38Live = liveModel('gemini-3.8-live', 'Gemini 3.8 Live', 'async', true);\n\nexport const gemini25NativeAudio = liveModel(\n  'gemini-live-2.5-flash-native-audio',\n  'Gemini 2.5 Flash Live',\n  'async',\n  true\n);\n\nexport const gemini31FlashLive = liveModel(\n  'gemini-3.1-flash-live-preview',\n  'Gemini 3.1 Flash Live',\n  'blocking',\n  false\n);\n\nexport function registerGeminiLive(options: GeminiOptions = {}): () => void {\n  const unregisterModels = [gemini38Live, gemini25NativeAudio, gemini31FlashLive].map(\n    registerVoiceModel\n  );\n  const unregisterApi = registerRealtimeApiProvider({\n    api: 'gemini-live',\n    connect: (model, context, connectOptions) =>\n      connectGeminiSession(model, context, connectOptions, options),\n  });\n  return () => {\n    for (const unregister of unregisterModels) unregister();\n    unregisterApi();\n  };\n}\n\nexport type { GeminiOptions, VoiceWireFactory } from './session.ts';\n"],
+  "mappings": "AACA,SAAS,0BAA0B;AACnC,SAAS,mCAAmC;AAC5C,SAAS,sBAA0C,iBAAiB;AAEpE,MAAM,SAAS;AAAA,EACb,EAAE,IAAI,SAAS,MAAM,QAAQ;AAAA,EAC7B,EAAE,IAAI,QAAQ,MAAM,OAAO;AAAA,EAC3B,EAAE,IAAI,UAAU,MAAM,SAAS;AAAA,EAC/B,EAAE,IAAI,QAAQ,MAAM,OAAO;AAAA,EAC3B,EAAE,IAAI,UAAU,MAAM,SAAS;AAAA,EAC/B,EAAE,IAAI,UAAU,MAAM,SAAS;AACjC;AAEA,MAAM,YAAY,EAAE,gBAAgB,MAAO,iBAAiB,MAAM;AAElE,MAAM,WACJ;AAEF,SAAS,UACP,IACA,MACA,WACA,OAC8B;AAC9B,SAAO;AAAA,IACL;AAAA,IACA;AAAA,IACA,KAAK;AAAA,IACL,UAAU;AAAA,IACV,SAAS;AAAA,IACT,MAAM;AAAA,IACN,YAAY,CAAC,WAAW;AAAA,IACxB;AAAA,IACA,aAAa,UAAU,IAAK;AAAA,IAC5B,cAAc,UAAU,IAAK;AAAA,IAC7B,OAAO,QAAQ,EAAE,QAAQ,EAAE,IAAI;AAAA,IAC/B,cAAc;AAAA,IACd,WAAW;AAAA,IACX,eAAe;AAAA,IACf;AAAA,IACA,cAAc;AAAA,IACd,UAAU,EAAE,WAAW,UAAU,WAAW,UAAU,QAAQ,SAAS;AAAA,IACvE,YAAY;AAAA,IACZ,gBAAgB;AAAA,IAChB,MAAM,EAAE,OAAO,GAAG,QAAQ,GAAG,WAAW,GAAG,YAAY,EAAE;AAAA,IACzD;AAAA,EACF;AACF;AAEO,MAAM,eAAe,UAAU,mBAAmB,mBAAmB,SAAS,IAAI;AAElF,MAAM,sBAAsB;AAAA,EACjC;AAAA,EACA;AAAA,EACA;AAAA,EACA;AACF;AAEO,MAAM,oBAAoB;AAAA,EAC/B;AAAA,EACA;AAAA,EACA;AAAA,EACA;AACF;AAEO,SAAS,mBAAmB,UAAyB,CAAC,GAAe;AAC1E,QAAM,mBAAmB,CAAC,cAAc,qBAAqB,iBAAiB,EAAE;AAAA,IAC9E;AAAA,EACF;AACA,QAAM,gBAAgB,4BAA4B;AAAA,IAChD,KAAK;AAAA,IACL,SAAS,CAAC,OAAO,SAAS,mBACxB,qBAAqB,OAAO,SAAS,gBAAgB,OAAO;AAAA,EAChE,CAAC;AACD,SAAO,MAAM;AACX,eAAW,cAAc,iBAAkB,YAAW;AACtD,kBAAc;AAAA,EAChB;AACF;",
+  "names": []
+}