@jambonz/schema 0.4.31 → 0.4.32

package/docs/verbs/agent.md

@@ -291,6 +291,44 @@ session.updateAgent({
 });
 ```
 
+## Your own harness (`llm.vendor: "harness"`)
+
+With `vendor: "harness"` the customer's own code decides what the agent says, while jambonz keeps STT, turn-taking, barge-in, TTS and the turn record. feature-server opens one WebSocket per call to the URL in the account's **Harness** LLM credential (fields `url` and `api_key`, looked up by vendor and `label` like any LLM credential), with subprotocol `harness.jambonz.org` and `Authorization: Bearer <api_key>`.
+
+```json
+{
+  "verb": "agent",
+  "stt": {"vendor": "deepgramflux", "language": "en-US"},
+  "tts": {"vendor": "cartesia", "voice": "..."},
+  "llm": {
+    "vendor": "harness",
+    "label": "my-harness",
+    "llmOptions": {"tools": [{"name": "lookup_account", "description": "...", "parameters": {"type": "object"}}]},
+    "harness": {
+      "metadata": {"agent": "outage-line"},
+      "firstChunkTimeoutMs": 4000,
+      "holdingPhrase": "One moment.",
+      "onHarnessFailure": {"say": "Sorry, I'm having trouble. Transferring you now.", "transfer": "+15085550100"}
+    }
+  },
+  "toolHook": "/tool",
+  "eventHook": "/event",
+  "actionHook": "/done"
+}
+```
+
+The harness owns prompts, context, fillers and reprompts, so with it `llmOptions` may carry only `tools`, and `model`, `auth`, `connectOptions`, `mcpServers`, `toolFiller`, `earlyGeneration: true`, `speechCleanup: true` and `noResponseMaxPrompts` are rejected by the schema. The tools are sent to the harness in `session:start` as OpenAI function definitions; when the harness sends `tool:call`, jambonz runs it through the `toolHook` exactly as for a model (`agent:tool-call`, answered with `llm:tool-output`) and returns `tool:result`. `handoff`, when present, gives the defaults for transfers the harness asks for.
+
+What jambonz sends: `session:start`, `speech:start`, `transcript`, `turn:start` (`input` is `speech`, `greeting` or `no_input`), `turn:cancel`, `event` (`user_interruption`, `reply_spoken`, `reply_dropped`, `silence`, `transfer`, `recording`, `application_update`, `error`), `command:result`, `tool:result`, `session:end`. What the harness sends: `reply:chunk` / `reply:done` (OpenAI `ChatCompletionChunk`s; `choices[0].delta.content` is spoken), `tool:call`, and `command` (`hangup`, `transfer`, `record`, `barge_in`, `listen`). The full protocol is the harness protocol reference.
+
+Notes on this implementation:
+- `application_update` carries the application's `agent:update` type as `update_type` (the message's own `type` is `event`), with the rest of the update in `data`.
+- `user_interruption.spoken` is exact with TTS vendors that report alignment, and estimated from playout time otherwise.
+- `noResponseTimeout` sends a `silence` event (`seconds`, `streak`) each time it elapses; jambonz does not reprompt.
+- `play` and `listen` with `hints` are not supported yet and are refused in `command:result`; `listen` with `language` works where the STT vendor can change language mid-call (Deepgram Flux).
+- Failures (socket cannot open, no reply twice in `firstChunkTimeoutMs`, a dropped socket that cannot be reopened within 2 s) run `onHarnessFailure`; the actionHook reports `completionReason: "harness_failure"` with `harness_error: {reason, message}` and the eventHook gets `harness_failure`.
+- Each `turn_end` carries `engine: "harness"`, `harness.turn_record` (the harness's `reply:done.turn_record`, up to 16 KB) and `latency.harness_first_chunk_ms` in place of `llm_ms`.
+
 ## LLM configuration
 
 The `llm` property is the only required field. It configures the text-to-text LLM:

package/docs/verbs/dialogflow.md

@@ -2,6 +2,17 @@
 
 The dialogflow verb connects the caller to a Google Dialogflow agent (ES, CX, or CES). See the verb schema for all properties.
 
+## Credentials without a key in the verb
+
+`credentials` may name a Google speech credential on the account instead of carrying a service account key:
+
+```json
+{"verb": "dialogflow", "model": "ces", "project": "my-project", "region": "us", "agent": "<CES app id>",
+ "lang": "en-US", "credentials": {"vendor": "google", "label": "my-ces"}}
+```
+
+An object with only `vendor: "google"` and optionally `label` is a reference. feature-server looks up the Google speech credential with that label (no label, or null, means the unlabelled one), account first and then service provider, and uses its service key. The credential does not have to be enabled or tested for STT or TTS. If none is found the verb ends at once: the actionHook gets `dialogflowResult: "error"` with `error.code: "credential_not_found"`, and feature-server logs the account sid, vendor and label. An inline service account works as before.
+
 ## Sending input into a CES conversation mid-call
 
 With `model: "ces"`, an application can send input into the running conversation without restarting the verb. Updates can be sent via:

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@jambonz/schema",
-  "version": "0.4.31",
+  "version": "0.4.32",
   "description": "JSON Schema definitions and validation for jambonz verb applications",
   "main": "index.js",
   "scripts": {

package/verbs/agent.schema.json

@@ -130,19 +130,19 @@
     },
     "llm": {
       "type": "object",
-      "description": "LLM configuration for the agent. Optional: when omitted, or when `vendor`/`model` are omitted, they fall back to the application's configured default LLM (like `stt`/`tts`).",
+      "description": "LLM configuration for the agent. Optional: when omitted, or when `vendor`/`model` are omitted, they fall back to the application's configured default LLM (like `stt`/`tts`). With `vendor: \"harness\"` a customer-run harness decides what to say instead of a model, over one WebSocket per call opened by jambonz; jambonz keeps STT, turn-taking, barge-in and TTS. See `harness`.",
       "properties": {
         "vendor": {
           "type": "string",
-          "description": "LLM vendor id. Must match a `@jambonz/llm` registered adapter. Optional: falls back to the application's default LLM vendor."
+          "description": "LLM vendor id. Must match a `@jambonz/llm` registered adapter, or be \"harness\" for a customer-run harness (the account's Harness credential, looked up by vendor and label, gives its WebSocket URL and key). Optional: falls back to the application's default LLM vendor."
         },
         "model": {
           "type": "string",
           "description": "Vendor-specific model id (e.g. 'gpt-4o', 'claude-sonnet-4-5-20250929'). Optional: falls back to the application's default LLM model."
         },
         "label": {
-          "type": "string",
-          "description": "Optional label to disambiguate when the account has multiple credentials for the same vendor."
+          "type": ["string", "null"],
+          "description": "Optional label to disambiguate when the account has multiple credentials for the same vendor. null or omitted selects the unlabelled credential."
         },
         "auth": {
           "type": "object",
@@ -204,13 +204,50 @@
             },
             "tools": {
               "type": "array",
-              "description": "Tool / function definitions available to the model. The MCP-flat shape `{name, description, parameters}` is canonical; the OpenAI-wrapped form `{type:'function', function:{...}}` is also accepted.",
+              "description": "Tool / function definitions available to the model. The MCP-flat shape `{name, description, parameters}` is canonical; the OpenAI-wrapped form `{type:'function', function:{...}}` is also accepted. With vendor \"harness\" these are sent to the harness in session:start as OpenAI function definitions, and the harness runs them with tool:call through the toolHook.",
               "items": {
                 "type": "object"
               }
             }
           },
           "additionalProperties": false
+        },
+        "harness": {
+          "type": "object",
+          "description": "Harness settings, only with vendor \"harness\". The harness owns prompts, context, fillers and reprompts, so with it `llmOptions` may carry only `tools`, `model`/`auth`/`connectOptions` are not allowed, and `mcpServers`, `toolFiller`, `earlyGeneration`, `speechCleanup: true` and `noResponseMaxPrompts` are rejected. `noResponseTimeout` still sets when the harness is sent a `silence` event; jambonz does not reprompt on its own. `handoff`, when present, supplies the defaults (mode, disposition, timeout, callerId) for transfers the harness asks for.",
+          "properties": {
+            "metadata": {
+              "type": "object",
+              "additionalProperties": true,
+              "description": "Free-form JSON passed to the harness in session:start (for example agent id, version and per-call variables)."
+            },
+            "firstChunkTimeoutMs": {
+              "type": "integer",
+              "minimum": 500,
+              "default": 4000,
+              "description": "Milliseconds from the end of the caller's turn to the harness's first reply:chunk (or reply:done). On the first timeout holdingPhrase is spoken once; on a second timeout onHarnessFailure runs. While a tool:call from the harness is outstanding, the second timeout waits for it. Default: 4000."
+            },
+            "holdingPhrase": {
+              "type": "string",
+              "description": "Spoken once when the harness has not started its reply within firstChunkTimeoutMs, for example 'One moment.'. When omitted nothing is spoken and the second timeout still applies."
+            },
+            "onHarnessFailure": {
+              "type": "object",
+              "description": "What jambonz does when the harness cannot be reached, stops answering, or drops and cannot be reconnected within 2 seconds: speak `say`, then transfer to `transfer` if given, otherwise hang up. The actionHook reports completionReason 'harness_failure' with a harness_error object.",
+              "properties": {
+                "say": {
+                  "type": "string",
+                  "description": "Spoken to the caller before the transfer or hangup."
+                },
+                "transfer": {
+                  "type": "string",
+                  "description": "Where to transfer the caller: an E.164 number, a sip: URI, or a registered user name. Uses the verb's `handoff` settings when present, otherwise a blind transfer by dial."
+                }
+              },
+              "additionalProperties": false
+            }
+          },
+          "additionalProperties": false
         }
       },
       "additionalProperties": false
@@ -221,7 +258,7 @@
     },
     "eventHook": {
       "$ref": "../components/actionHook",
-      "description": "A webhook invoked for agent events. Receives event types: 'user_transcript' (user speech recognized), 'llm_response' (assistant reply), 'user_interruption' (barge-in detected), and 'turn_end' (end-of-turn summary with transcript, response, and latency metrics)."
+      "description": "A webhook invoked for agent events. Receives event types: 'user_transcript' (user speech recognized), 'llm_response' (assistant reply), 'user_interruption' (barge-in detected), and 'turn_end' (end-of-turn summary with transcript, response, and latency metrics). With vendor \"harness\" it also receives 'harness_failure' ({reason, message}) when onHarnessFailure runs, and turn_end carries `engine: \"harness\"`, `harness.turn_record` and `latency.harness_first_chunk_ms`."
     },
     "toolHook": {
       "$ref": "../components/actionHook",
@@ -343,6 +380,41 @@
       "description": "Configuration for playing filler audio while tool calls are in progress. Prevents silence during long-running tool executions."
     }
   },
+  "allOf": [
+    {
+      "if": {
+        "properties": {
+          "llm": {
+            "properties": { "vendor": { "const": "harness" } },
+            "required": ["vendor"]
+          }
+        },
+        "required": ["llm"]
+      },
+      "then": {
+        "properties": {
+          "llm": {
+            "properties": {
+              "model": false,
+              "auth": false,
+              "connectOptions": false,
+              "llmOptions": { "propertyNames": { "enum": ["tools"] } }
+            }
+          },
+          "mcpServers": { "maxItems": 0 },
+          "toolFiller": { "const": false },
+          "earlyGeneration": { "const": false },
+          "speechCleanup": { "const": false },
+          "noResponseMaxPrompts": { "const": 0 }
+        }
+      },
+      "else": {
+        "properties": {
+          "llm": { "not": { "required": ["harness"] } }
+        }
+      }
+    }
+  ],
   "$defs": {
     "llmMessage": {
       "type": "object",
@@ -454,6 +526,48 @@
         "sticky": false
       },
       "actionHook": "/agent-complete"
+    },
+    {
+      "verb": "agent",
+      "stt": {
+        "vendor": "deepgramflux",
+        "language": "en-US"
+      },
+      "tts": {
+        "vendor": "cartesia",
+        "voice": "sonic-english"
+      },
+      "llm": {
+        "vendor": "harness",
+        "label": "my-harness",
+        "llmOptions": {
+          "tools": [
+            {
+              "name": "lookup_account",
+              "description": "Look up the caller's account by phone number.",
+              "parameters": {
+                "type": "object",
+                "properties": { "phone": { "type": "string" } },
+                "required": ["phone"]
+              }
+            }
+          ]
+        },
+        "harness": {
+          "metadata": { "agent": "outage-line", "tenant": "acme" },
+          "firstChunkTimeoutMs": 4000,
+          "holdingPhrase": "One moment.",
+          "onHarnessFailure": {
+            "say": "Sorry, I'm having trouble. Transferring you now.",
+            "transfer": "+15085550100"
+          }
+        }
+      },
+      "turnDetection": "stt",
+      "bargeIn": { "enable": true },
+      "toolHook": "/tool",
+      "eventHook": "/event",
+      "actionHook": "/done"
     }
   ]
 }

package/verbs/dialogflow.schema.json

@@ -24,7 +24,7 @@
           "type": "string"
         }
       ],
-      "description": "Google service account credentials as a JSON object or stringified JSON."
+      "description": "Google service account credentials as a JSON object or stringified JSON. Alternatively a reference to a Google speech credential on the account, so no key is carried in the verb: {\"vendor\": \"google\", \"label\": \"my-ces\"} (label optional; omit it or use null for the unlabelled Google credential). An object with only `vendor` (\"google\") and optionally `label` is treated as a reference; feature-server looks the credential up by vendor and label, account first then service provider, and uses its service key. If none is found the verb ends at once and its actionHook reports dialogflowResult 'error' with error.code 'credential_not_found'."
     },
     "project": {
       "type": "string",