@jambonz/schema 0.4.6 → 0.4.8

package/AGENTS.md

@@ -35,7 +35,7 @@ The verb schemas and JSON structure are identical in both modes. The difference
 - **Webhook**: Simple IVR, call routing, voicemail, basic gather-and-respond patterns.
 - **WebSocket**: LLM-powered voice agents, real-time audio streaming, complex conversational flows, anything requiring bidirectional communication, or asynchronous logic, or streaming tts.
 
-**IMPORTANT**: Any application that uses a speech-to-speech verb (`openai_s2s`, `google_s2s`, `deepgram_s2s`, `ultravox_s2s`, `elevenlabs_s2s`, `s2s`, or `agent`) MUST use WebSocket transport, not webhooks. These verbs require persistent bidirectional communication for real-time audio and events.
+**IMPORTANT**: Any application that uses a speech-to-speech verb (`openai_s2s`, `gptlive_s2s`, `google_s2s`, `deepgram_s2s`, `ultravox_s2s`, `qwen_s2s`, `elevenlabs_s2s`, `s2s`, or `agent`) MUST use WebSocket transport, not webhooks. These verbs require persistent bidirectional communication for real-time audio and events.
 
 ## Schema
 
@@ -62,7 +62,8 @@ Two tools are available:
 - **gather** — Collect speech (STT) and/or DTMF input. The workhorse for interactive menus and voice input.
 
 ### AI & Real-time
-- **openai_s2s** / **google_s2s** / **deepgram_s2s** / **ultravox_s2s** — Connect the caller to a vendor-specific LLM for real-time voice conversation. These are the **preferred** verbs when the vendor is known. Each handles the full STT→LLM→TTS flow with the vendor pre-set.
+- **openai_s2s** / **google_s2s** / **deepgram_s2s** / **ultravox_s2s** / **qwen_s2s** — Connect the caller to a vendor-specific LLM for real-time voice conversation. These are the **preferred** verbs when the vendor is known. Each handles the full STT→LLM→TTS flow with the vendor pre-set.
+- **gptlive_s2s** — Connect the caller to OpenAI's GPT Live API (limited-access alpha). A different wire protocol from `openai_s2s` (Realtime): `llmOptions` carries only a `session_update`, there is no `response_create`, and tool calls arrive via a Responses-targeted *delegation*. See [GPT Live S2S specifics](#gpt-live-s2s-specifics) below.
 - **elevenlabs_s2s** — Connect the caller to an ElevenLabs Conversational AI agent. **Unlike other s2s vendors**, ElevenLabs requires a pre-configured `agent_id` (created in the ElevenLabs dashboard) rather than a model and messages. See [ElevenLabs S2S specifics](#elevenlabs-s2s-specifics) below.
 - **s2s** — Generic LLM voice conversation verb. Use only when the vendor is determined at runtime (e.g. from an env var). Requires `vendor` to be specified.
 - **agent** — Higher-level voice AI agent with integrated turn detection. Mix-and-match STT, LLM, and TTS vendors.
@@ -98,7 +99,7 @@ Two tools are available:
 **IMPORTANT — Code generation rules:**
 
 1. **Always use `stream`, never `listen`** — they are synonyms; `stream` is the preferred name.
-2. **Always use the vendor-specific shortcut when the LLM vendor is known** — use `openai_s2s`, `google_s2s`, `elevenlabs_s2s`, `deepgram_s2s`, or `ultravox_s2s`. Do NOT use `llm` or `s2s` with a `vendor` property when a shortcut exists.
+2. **Always use the vendor-specific shortcut when the LLM vendor is known** — use `openai_s2s`, `gptlive_s2s`, `google_s2s`, `elevenlabs_s2s`, `deepgram_s2s`, `qwen_s2s`, or `ultravox_s2s`. Do NOT use `llm` or `s2s` with a `vendor` property when a shortcut exists.
 3. **Use `s2s` (not `llm`) when the vendor is dynamic** — e.g. the vendor comes from an env var or runtime config. Both `s2s` and `llm` are synonyms, but prefer `s2s`.
 4. **Never use `llm` in generated code** — it is a legacy name. Use either a vendor shortcut or `s2s`.
 
@@ -161,6 +162,51 @@ ElevenLabs works differently from other s2s vendors. Instead of passing a model
 ]
 ```
 
+### GPT Live S2S Specifics
+
+OpenAI's GPT Live API (limited-access alpha) is a different wire protocol from the OpenAI Realtime API behind `openai_s2s`, so it has its own verb and its own `llmOptions` shape.
+
+**Key differences from `openai_s2s`:**
+- `llmOptions` takes only `session_update` — there is **no** `response_create`. The model starts and drives the conversation itself.
+- Do NOT put `model` inside `session_update`; the model travels in the connection URL (set the verb's `model` property instead, default `gpt-live-1-boulder-alpha`).
+- Work the model wants done arrives as a **delegation**, selected by `session_update.delegation.type`:
+  - `client` — the model asks your app for free-form text context. Answer with an `llm:update` carrying `delegation.context.append` (max 500 tokens, exactly one `input_text` part).
+  - `responses` — the model runs a Responses API turn whose `response.*` events stream inline. Function calls are answered with `delegation.function_call_output.create`.
+- A `responses` delegation carries a **required nested object** whose `model` is **mandatory**, and tools go INSIDE it:
+  ```json
+  "delegation": {
+    "type": "responses",
+    "responses": {
+      "model": "gpt-5.5",
+      "tools": [{ "type": "function", "name": "get_weather", "description": "...", "parameters": {} }]
+    }
+  }
+  ```
+  Tools placed at `delegation.tools` are rejected by the server. `delegation.responses.model` is the Responses-side model, separate from the GPT Live voice model in the verb's `model` property.
+- Tools — `mcpServers`, `handoff` and `hangup` — only work with `delegation.type: "responses"` **and** `delegation.responses.model` set; with a `client` delegation there is no function-calling protocol to declare them on, and the verb is rejected.
+- Mid-call `llm:update` accepts only `session.update`, `session.context.append`, `delegation.context.append`, `delegation.function_call_output.create` and `session.close`. Realtime events (`response.create`, `response.cancel`, `input_audio_buffer.*`, `conversation.item.create`) are rejected.
+- A later `session.update` is sparse (omitted fields keep their values), but replacing `delegation` requires the complete delegation object.
+
+```json
+[
+  {
+    "verb": "gptlive_s2s",
+    "model": "gpt-live-1-boulder-alpha",
+    "auth": { "apiKey": "sk-..." },
+    "llmOptions": {
+      "session_update": {
+        "instructions": "You are a helpful voice assistant. Be concise.",
+        "audio": { "output": { "voice": "marin" } },
+        "delegation": { "type": "client" }
+      }
+    },
+    "actionHook": "/s2s-complete",
+    "eventHook": "/event",
+    "events": ["all"]
+  }
+]
+```
+
 ### Dial with Fallback
 ```json
 [

package/components/synthesizer.schema.json

@@ -8,7 +8,7 @@
     "vendor": {
       "type": "string",
       "description": "The TTS vendor to use. Must match a vendor configured in the jambonz platform.",
-      "examples": ["google", "aws", "microsoft", "elevenlabs", "cartesia", "gradium", "nineninesix", "deepgram", "ibm", "nvidia", "wellsaid", "whisper", "custom"]
+      "examples": ["google", "aws", "microsoft", "elevenlabs", "cartesia", "gradium", "nineninesix", "deepgram", "deepgramflux", "ibm", "nvidia", "wellsaid", "whisper", "custom"]
     },
     "label": {
       "type": "string",

package/docs/components/synthesizer.md

@@ -5,6 +5,7 @@ Refer to vendor documentation for supported models, voices, and vendor-specific
 ### Deepgram
 
 - [TTS Models](https://developers.deepgram.com/docs/tts-models)
+- [Flux TTS](https://developers.deepgram.com/docs/flux-tts/overview) (vendor `deepgramflux`)
 
 ### Google
 

package/jambonz-app.schema.json

@@ -28,6 +28,7 @@
         { "$ref": "verbs/deepgram_s2s" },
         { "$ref": "verbs/ultravox_s2s" },
         { "$ref": "verbs/qwen_s2s" },
+        { "$ref": "verbs/gptlive_s2s" },
         { "$ref": "verbs/dialogflow" },
         { "$ref": "verbs/agent" },
         { "$ref": "verbs/transfer" },

package/lib/normalize.js

@@ -15,6 +15,7 @@ const verbTransforms = new Map([
   ['voiceagent_s2s', {verb: 'llm', properties: {vendor: 'voiceagent'}}],
   ['ultravox_s2s', {verb: 'llm', properties: {vendor: 'ultravox'}}],
   ['qwen_s2s', {verb: 'llm', properties: {vendor: 'qwen'}}],
+  ['gptlive_s2s', {verb: 'llm', properties: {vendor: 'gptlive'}}],
 ]);
 
 function applyVerbTransform(name, data) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@jambonz/schema",
-  "version": "0.4.6",
+  "version": "0.4.8",
   "description": "JSON Schema definitions and validation for jambonz verb applications",
   "main": "index.js",
   "scripts": {

package/verbs/gptlive_s2s.schema.json

@@ -0,0 +1,240 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://jambonz.org/schema/verbs/gptlive_s2s",
+  "minVersion": "11.1.0",
+  "title": "GPT Live S2S",
+  "description": "Shortcut for 'llm' with vendor automatically set to 'gptlive'. Connects the caller to OpenAI's GPT Live API (limited-access alpha, e.g. model gpt-live-1-boulder-alpha) for real-time speech-to-speech voice conversation. This is a different wire protocol from the OpenAI Realtime API used by 'openai_s2s': llmOptions takes only a session_update (there is no response_create \u2014 the model starts and drives the conversation itself), and work the model wants done arrives as a delegation. Set session_update.delegation.type to 'client' to be asked for free-form text context (answer with a delegation.context.append via llm:update), or to 'responses' to run a Responses API turn that can make function calls (answer with delegation.function_call_output.create). A 'responses' delegation carries a required nested object, delegation.responses, whose 'model' is mandatory and whose 'tools' take the flat Responses function shape. Tool support \u2014 mcpServers, handoff and hangup \u2014 requires delegation.type 'responses' with delegation.responses.model set. auth takes the OpenAI apiKey; the model is carried in the connection URL, so do not set session_update.model.",
+  "type": "object",
+  "allOf": [
+    {
+      "$ref": "../components/llm-base"
+    },
+    {
+      "$comment": "handoff, hangup and mcpServers need a function-calling channel, which only a Responses delegation provides. The feature-server throws when the verb executes otherwise, dropping the call \u2014 catch it at validation instead.",
+      "if": {
+        "anyOf": [
+          {
+            "required": [
+              "handoff"
+            ]
+          },
+          {
+            "required": [
+              "hangup"
+            ]
+          },
+          {
+            "required": [
+              "mcpServers"
+            ]
+          }
+        ]
+      },
+      "then": {
+        "properties": {
+          "llmOptions": {
+            "properties": {
+              "session_update": {
+                "properties": {
+                  "delegation": {
+                    "properties": {
+                      "type": {
+                        "const": "responses"
+                      }
+                    },
+                    "required": [
+                      "type",
+                      "responses"
+                    ]
+                  }
+                },
+                "required": [
+                  "delegation"
+                ]
+              }
+            }
+          }
+        }
+      }
+    }
+  ],
+  "properties": {
+    "verb": {
+      "const": "gptlive_s2s",
+      "description": "The verb name."
+    },
+    "vendor": {
+      "type": "string",
+      "const": "gptlive",
+      "description": "The LLM vendor (always 'gptlive' for this shortcut)."
+    },
+    "auth": {
+      "type": "object",
+      "description": "OpenAI credentials.",
+      "properties": {
+        "apiKey": {
+          "type": "string",
+          "description": "The OpenAI API key. Required \u2014 the session cannot be authenticated without it.",
+          "minLength": 1
+        }
+      },
+      "required": [
+        "apiKey"
+      ],
+      "additionalProperties": true
+    },
+    "llmOptions": {
+      "type": "object",
+      "properties": {
+        "session_update": {
+          "type": "object",
+          "description": "Startup configuration, sent as the first session.update client event. Required: the server withholds session.started \u2014 and therefore caller audio \u2014 until it arrives.",
+          "properties": {
+            "model": {
+              "not": {},
+              "description": "Not allowed: the model travels in the connection URL. Set the verb's top-level 'model' property instead."
+            },
+            "delegation": {
+              "type": "object",
+              "description": "How the model delegates work it cannot do itself. Replacing this mid-call requires the complete object.",
+              "properties": {
+                "type": {
+                  "type": "string",
+                  "enum": [
+                    "client",
+                    "responses"
+                  ],
+                  "description": "'client' asks the application for free-form text context; 'responses' runs a Responses API turn that can make function calls. Tools \u2014 mcpServers, handoff, hangup \u2014 require 'responses'."
+                },
+                "responses": {
+                  "type": "object",
+                  "description": "Configuration for a Responses-targeted delegation. Required when type is 'responses'.",
+                  "properties": {
+                    "model": {
+                      "type": "string",
+                      "description": "The Responses-side model the delegated turns run on (e.g. gpt-5.5). Required by the server, and distinct from the GPT Live voice model in the connection URL.",
+                      "examples": [
+                        "gpt-5.5"
+                      ]
+                    },
+                    "tools": {
+                      "type": "array",
+                      "items": {
+                        "type": "object"
+                      },
+                      "description": "Function tools in the flat Responses shape: {type:'function', name, description, parameters}. jambonz prepends the injected handoff/hangup/MCP tools here."
+                    },
+                    "instructions": {
+                      "type": "string"
+                    }
+                  },
+                  "required": [
+                    "model"
+                  ],
+                  "additionalProperties": true
+                },
+                "tools": {
+                  "not": {},
+                  "description": "Not allowed here: a Responses delegation's tools belong at delegation.responses.tools. Tools placed at this level are rejected by the server at session start."
+                }
+              },
+              "additionalProperties": true,
+              "if": {
+                "properties": {
+                  "type": {
+                    "const": "responses"
+                  }
+                },
+                "required": [
+                  "type"
+                ]
+              },
+              "then": {
+                "required": [
+                  "responses"
+                ]
+              }
+            }
+          },
+          "additionalProperties": true
+        }
+      },
+      "required": [
+        "session_update"
+      ],
+      "additionalProperties": true
+    }
+  },
+  "required": [
+    "auth",
+    "llmOptions"
+  ],
+  "examples": [
+    {
+      "verb": "gptlive_s2s",
+      "model": "gpt-live-1-boulder-alpha",
+      "auth": {
+        "apiKey": "sk-..."
+      },
+      "llmOptions": {
+        "session_update": {
+          "instructions": "You are a helpful voice assistant. Be concise.",
+          "audio": {
+            "output": {
+              "voice": "marin"
+            }
+          },
+          "delegation": {
+            "type": "client"
+          }
+        }
+      },
+      "eventHook": "/s2s-event",
+      "actionHook": "/s2s-complete"
+    },
+    {
+      "verb": "gptlive_s2s",
+      "model": "gpt-live-1-boulder-alpha",
+      "auth": {
+        "apiKey": "sk-..."
+      },
+      "llmOptions": {
+        "session_update": {
+          "instructions": "You are a helpful voice assistant. Be concise.",
+          "audio": {
+            "output": {
+              "voice": "marin"
+            }
+          },
+          "delegation": {
+            "type": "responses",
+            "responses": {
+              "model": "gpt-5.5",
+              "tools": [
+                {
+                  "type": "function",
+                  "name": "get_weather",
+                  "description": "Get the current weather for a city",
+                  "parameters": {
+                    "type": "object",
+                    "properties": {
+                      "city": {
+                        "type": "string"
+                      }
+                    },
+                    "required": [
+                      "city"
+                    ]
+                  }
+                }
+              ]
+            }
+          }
+        }
+      },
+      "toolHook": "/s2s-tool-call",
+      "eventHook": "/s2s-event",
+      "actionHook": "/s2s-complete"
+    }
+  ]
+}