@kuralle-syrinx/browser-client 4.2.0 → 4.4.0

package/README.md

@@ -0,0 +1,142 @@
+# @kuralle-syrinx/browser-client
+
+The browser SDK for the Syrinx WebSocket voice protocol. Microphone capture,
+opus encoding, jitter-buffered playback, reconnect — and the typed message
+stream a voice agent sends back.
+
+**This is what you build your own voice UI on.** [Syrinx Studio](https://github.com/kuralle/syrinx/tree/main/apps/studio)
+is built entirely on it; so is anything you want to embed voice into.
+
+```bash
+npm install @kuralle-syrinx/browser-client
+```
+
+## Talk to an agent
+
+```ts
+import { SyrinxBrowserClient } from "@kuralle-syrinx/browser-client";
+
+const client = new SyrinxBrowserClient({
+  url: "ws://localhost:4173/ws",
+  audioContext: new AudioContext(),
+  jitterBuffer: { targetBufferMs: 100 },
+});
+
+client.on((event) => {
+  if (event.type === "message" && event.message.type === "stt_output") {
+    console.log("you said:", event.message.transcript);
+  }
+  if (event.type === "message" && event.message.type === "agent_chunk") {
+    process.stdout.write(event.message.text);
+  }
+});
+
+await client.connect();
+```
+
+Send a turn as text instead of speaking — useful for iterating on prompts and
+tools without paying STT/TTS latency or cost:
+
+```ts
+client.sendText("What's the deadline for the CS masters?");
+```
+
+## What the server sends you
+
+`SyrinxStudioMessage` is a discriminated union of roughly twenty variants. The
+ones you will reach for first:
+
+| Message | Carries |
+| --- | --- |
+| `ready` | negotiated sample rates, encoding, resume window |
+| `speech_started` / `speech_ended` | turn boundaries |
+| `stt_chunk` / `stt_output` | interim and final transcript (+ confidence) |
+| `agent_chunk` / `agent_end` | the reply, streamed |
+| `agent_tool_call` / `agent_tool_result` | tool use |
+| `tool_call_started` \| `delayed` \| `complete` \| `failed` | the "thinking" cue lifecycle |
+| `agent_interrupted` | barge-in, **with a reason** |
+| `turn_complete`, `tts_chunk`, `tts_end` | turn and audio lifecycle |
+| `metrics` | per-turn latency decomposition |
+| `error` | `component`, `category`, `message` |
+
+> **Handle unknown types gracefully.** On Cloudflare the agents SDK also emits
+> `cf_agent_identity` and `cf_agent_mcp_servers`, which are not in this union,
+> and future versions will add more. Ignore what you do not recognise rather
+> than throwing.
+
+## Structured session state — `/record`
+
+Folding that stream by hand is tedious and easy to get wrong. The `/record`
+subpath does it for you, and is **dependency-free and DOM-free** so it also runs
+under Node (in a test, a CLI, or CI):
+
+```ts
+import { buildSessionRecord } from "@kuralle-syrinx/browser-client/record";
+
+const record = buildSessionRecord(messages.map((m, i) => ({ message: m, atMs: i })));
+
+record.turns[0]?.userTranscript;  // what the caller said
+record.turns[0]?.agentText;       // what the agent replied
+record.turns[0]?.interrupted;     // { atMs, reason } when barge-in cut it off
+record.turns[0]?.toolCalls;       // merged across all four cue phases
+record.turns[0]?.timings;         // optional — see the caveat below
+```
+
+It is a **pure reducer**: same messages in, same record out, no side effects.
+That is what lets you render a UI from a recorded fixture in a test instead of
+needing a live provider.
+
+It is also **bounded** — 50 turns and 500 events per turn by default, evicting
+oldest and reporting `droppedTurns` / `droppedEvents` rather than losing data
+silently.
+
+> **`timings` is optional and you must handle its absence.** It comes from the
+> `metrics` message, which the Node host emits and the Cloudflare Workers path
+> currently does not. Render a turn without timings rather than assuming they
+> arrive — showing zeroes would read as a zero-latency turn.
+
+## Derived views — `/agent-state` and `/turn-timeline`
+
+Both are pure, Node-safe, and derived from the same record.
+
+```ts
+import { deriveAgentState, isStalled } from "@kuralle-syrinx/browser-client/agent-state";
+import { buildTurnTimeline } from "@kuralle-syrinx/browser-client/turn-timeline";
+```
+
+`deriveAgentState` gives you `idle → listening → endpointing → thinking →
+speaking → interrupted` — what a client renders as a listening indicator. Note
+this is *conversational* state; `SessionState` in `@kuralle-syrinx/core` is
+session lifecycle and is a different thing. `isStalled` flags a state held far
+past plausibility, because a thirty-second "thinking" is a hung tool call, not
+patience.
+
+`buildTurnTimeline` turns `metrics` into a latency waterfall with the slowest
+segment marked. It also flags replies faster than `FAST_TURN_FLOOR_MS` (700ms) —
+**a sub-second reply is usually the endpointer firing while the caller is still
+speaking**, not a fast agent.
+
+## Session aggregates — `/session-metrics`
+
+```ts
+import { buildSessionMetrics } from "@kuralle-syrinx/browser-client/session-metrics";
+```
+
+Median / p95 / max per stage across a session, plus the ids of turns that
+replied implausibly fast. Percentiles are **nearest-rank**, so every number
+reported is a measurement that actually occurred rather than an interpolation no
+turn ever produced.
+
+## Exports
+
+| Path | Contents | Runs under Node? |
+| --- | --- | --- |
+| `.` | client, transport, audio, message types | no — needs `AudioContext` |
+| `./record` | `SessionRecord` assembler | yes |
+| `./agent-state` | conversational state machine | yes |
+| `./turn-timeline` | per-turn latency waterfall | yes |
+| `./session-metrics` | session aggregates | yes |
+
+## Licence
+
+MIT.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@kuralle-syrinx/browser-client",
-  "version": "4.2.0",
+  "version": "4.4.0",
   "private": false,
   "description": "Browser client for the Syrinx WebSocket audio protocol — mic capture, opus, jitter buffer, playout progress reporting, reconnect",
   "keywords": [
@@ -25,14 +25,22 @@
   "type": "module",
   "main": "./src/index.ts",
   "types": "./src/index.ts",
+  "exports": {
+    ".": "./src/index.ts",
+    "./record": "./src/session-record.ts",
+    "./agent-state": "./src/agent-state.ts",
+    "./turn-timeline": "./src/turn-timeline.ts",
+    "./session-metrics": "./src/session-metrics.ts",
+    "./turn-recorder": "./src/turn-recorder.ts"
+  },
   "dependencies": {
     "@evan/opus": "1.0.3",
-    "@kuralle-syrinx/core": "4.2.0"
+    "@kuralle-syrinx/core": "4.4.0"
   },
   "devDependencies": {
     "typescript": "^5.7.0",
-    "vite": "^6.0.0",
-    "vitest": "^2.1.0"
+    "vite": "^6.4.3",
+    "vitest": "^3.2.6"
   },
   "scripts": {
     "dev": "vite --port 5173",

package/src/agent-state.ts

@@ -0,0 +1,107 @@
+// SPDX-License-Identifier: MIT
+//
+// Conversational agent state, derived on the client from messages the server
+// already sends. Distinct from `SessionState` in core, which is session
+// *lifecycle* (uninitialized/ready/closed) — this is what the caller experiences:
+// is it listening to me, thinking, or talking?
+//
+// A pause reads as "broken" without this. It is what a client renders as a
+// listening indicator and what makes a turn timeline legible.
+//
+// Derived rather than sent: every signal already exists on the wire, so this
+// needs no server change and works identically on Node and Workers.
+
+import type { SyrinxStudioMessage } from "./index.js";
+
+export type AgentState =
+  | "idle"        // no session activity
+  | "listening"   // user is speaking
+  | "endpointing" // user stopped; deciding whether the turn is over
+  | "thinking"    // reasoner (and any tools) running
+  | "speaking"    // assistant audio playing
+  | "interrupted"; // barge-in cut the assistant off
+
+export interface AgentStateSnapshot {
+  readonly state: AgentState;
+  /** ms the state has been held, from the caller's clock. Lets a UI flag a stuck state. */
+  readonly sinceMs: number;
+  readonly turnId?: string;
+}
+
+type AnyMessage = { readonly type: string; readonly [k: string]: unknown };
+
+/**
+ * Fold one message into the state. Pure; the caller supplies the clock so this
+ * stays deterministic under test.
+ *
+ * Ordering note: `speech_started` wins over everything, because a caller talking
+ * is the highest-priority fact on the wire — that is also what makes barge-in
+ * legible (speaking → listening without waiting for the interrupt message).
+ */
+export function nextAgentState(
+  prev: AgentStateSnapshot,
+  message: SyrinxStudioMessage | AnyMessage,
+  atMs: number,
+): AgentStateSnapshot {
+  const type = typeof (message as AnyMessage).type === "string" ? (message as AnyMessage).type : "";
+  const turnId = typeof (message as AnyMessage).turnId === "string" ? ((message as AnyMessage).turnId as string) : prev.turnId;
+  const to = (state: AgentState): AgentStateSnapshot =>
+    state === prev.state ? { ...prev, turnId } : { state, sinceMs: atMs, turnId };
+
+  switch (type) {
+    case "speech_started":
+      return to("listening");
+    case "speech_ended":
+      // Speech stopped, but the turn is not necessarily over — this is exactly
+      // the window where a premature endpoint cuts a caller off.
+      return to("endpointing");
+    case "stt_output":
+    case "turn_complete":
+      // A final transcript means the turn closed; the reasoner is now running.
+      return to("thinking");
+    case "agent_tool_call":
+    case "tool_call_started":
+    case "tool_call_delayed":
+      return to("thinking");
+    case "agent_chunk":
+      // Text is arriving but nothing is audible yet.
+      return prev.state === "speaking" ? to("speaking") : to("thinking");
+    case "tts_chunk":
+      return to("speaking");
+    case "agent_interrupted":
+      return to("interrupted");
+    case "tts_end":
+    case "agent_end":
+      // Only settle to idle from a talking state; an interrupt stands until the
+      // next turn starts, so the UI can show what happened.
+      return prev.state === "speaking" ? to("idle") : prev;
+    default:
+      return prev;
+  }
+}
+
+export const INITIAL_AGENT_STATE: AgentStateSnapshot = { state: "idle", sinceMs: 0 };
+
+/** Fold a whole stream. Equivalent to repeated `nextAgentState`. */
+export function deriveAgentState(
+  messages: readonly { readonly message: SyrinxStudioMessage | AnyMessage; readonly atMs: number }[],
+): AgentStateSnapshot {
+  return messages.reduce((s, { message, atMs }) => nextAgentState(s, message, atMs), INITIAL_AGENT_STATE);
+}
+
+/**
+ * How long a state may plausibly be held before it indicates a stall. A UI should
+ * flag rather than hide these — a 30s "thinking" is a hung tool call, not patience.
+ */
+export const STATE_STALL_THRESHOLD_MS: Readonly<Record<AgentState, number>> = {
+  idle: Number.POSITIVE_INFINITY,
+  listening: 60_000,
+  endpointing: 5_000,
+  thinking: 30_000,
+  speaking: 120_000,
+  interrupted: 10_000,
+};
+
+export function isStalled(snapshot: AgentStateSnapshot, nowMs: number): boolean {
+  return nowMs - snapshot.sinceMs > STATE_STALL_THRESHOLD_MS[snapshot.state];
+}

package/src/index.ts

@@ -28,6 +28,7 @@ import {
   resamplePcm16Streaming,
   type StreamingPcm16Resampler,
 } from "@kuralle-syrinx/core/audio";
+import type { TurnEndOwner, TurnEndReason } from "@kuralle-syrinx/core";
 import {
   decodeBrowserAssistantAudio,
   encodeBrowserAudioEnvelopeFrame,
@@ -68,6 +69,8 @@ export type SyrinxStudioMessage =
         readonly encoding: "pcm_s16le" | "opus";
         readonly supportedInputCodecs?: readonly string[];
         readonly channels: 1;
+        /** Node states this; the Workers/DO host does not. Optional for that reason. */
+        readonly targetFrameDurationMs?: number;
         readonly binaryEnvelope?: "syrinx.audio.v1";
         readonly rawBinaryInput?: boolean;
       };
@@ -114,6 +117,10 @@ export type SyrinxStudioMessage =
       readonly llmTTFTMs?: number;
       readonly ttsTTFBMs?: number;
       readonly e2eMs?: number;
+      /** Which owner decided the turn ended. Omitted when the backend did not say. */
+      readonly endpointingOwner?: TurnEndOwner;
+      /** Why the turn ended. Omitted when the backend did not say. */
+      readonly endpointingReason?: TurnEndReason;
     }
   | { readonly type: "error"; readonly component?: string; readonly category?: string; readonly message: string };
 
@@ -187,6 +194,11 @@ const RECONNECT_MIN_STABLE_MS = 5_000;
 const RECONNECT_MAX_QUICK_FAILURES = 3;
 const KEEPALIVE_INTERVAL_MS = 10_000;
 
+// Endpointing decision values carried on the `metrics` message. Forward-compatible:
+// an unknown value (a newer backend) is ignored rather than dropping the message.
+const ENDPOINTING_OWNERS = new Set<string>(["provider_stt", "smart_turn", "timer", "text"]);
+const ENDPOINTING_REASONS = new Set<string>(["end_of_speech", "force_finalized", "typed"]);
+
 export class SyrinxBrowserClient {
   private readonly transport: ClientTransport;
   private readonly handlers = new Set<SyrinxBrowserClientHandler>();
@@ -535,7 +547,15 @@ export class SyrinxBrowserClient {
   private handleJsonMessage(data: unknown): void {
     if (typeof data !== "string") return;
     try {
-      const message = parseStudioMessage(JSON.parse(data) as unknown);
+      const parsed = JSON.parse(data) as unknown;
+      // Cloudflare Agents SDK control frames (cf_agent_state, cf_agent_mcp_servers, …) are
+      // broadcast by the Agent base class on the same socket a cf-agents `withVoice` worker
+      // uses for voice. They are not part of the Syrinx voice protocol — ignore them rather
+      // than throwing (which surfaced a spurious `error` event → a false "Error" badge).
+      if (isRecord(parsed) && typeof parsed["type"] === "string" && parsed["type"].startsWith("cf_agent")) {
+        return;
+      }
+      const message = parseStudioMessage(parsed);
       if (message.type === "ready") {
         if (message.sessionId !== undefined) this.currentSessionId = message.sessionId;
         if (message.audio?.outputSampleRateHz && message.audio.outputSampleRateHz !== this.outputSampleRateHz) {
@@ -801,6 +821,14 @@ function parseStudioMessage(value: unknown): SyrinxStudioMessage {
     };
   }
   if (type === "metrics") {
+    const endpointingOwner = optionalEndpointingEnum<TurnEndOwner>(
+      value.endpointingOwner,
+      ENDPOINTING_OWNERS,
+    );
+    const endpointingReason = optionalEndpointingEnum<TurnEndReason>(
+      value.endpointingReason,
+      ENDPOINTING_REASONS,
+    );
     return {
       type,
       turnId: optionalString(value.turnId, "metrics.turnId"),
@@ -814,6 +842,8 @@ function parseStudioMessage(value: unknown): SyrinxStudioMessage {
       llmTTFTMs: optionalNumber(value.llmTTFTMs, "metrics.llmTTFTMs"),
       ttsTTFBMs: optionalNumber(value.ttsTTFBMs, "metrics.ttsTTFBMs"),
       e2eMs: optionalNumber(value.e2eMs, "metrics.e2eMs"),
+      ...(endpointingOwner !== undefined ? { endpointingOwner } : {}),
+      ...(endpointingReason !== undefined ? { endpointingReason } : {}),
     };
   }
   if (type === "error") {
@@ -925,6 +955,12 @@ function optionalBoolean(value: unknown, name: string): boolean | undefined {
   return value;
 }
 
+function optionalEndpointingEnum<T extends string>(value: unknown, allowed: ReadonlySet<string>): T | undefined {
+  // Forward-compatible: an unknown value is treated as absent, not an error, so a
+  // newer backend adding an enum member does not break an older client's message.
+  return typeof value === "string" && allowed.has(value) ? (value as T) : undefined;
+}
+
 function requiredLiteral<T extends string | number>(value: unknown, expected: T, name: string): T {
   if (value !== expected) throw new Error(`${name} must be ${String(expected)}`);
   return expected;

package/src/session-metrics.ts

@@ -0,0 +1,90 @@
+// SPDX-License-Identifier: MIT
+//
+// Session-level latency aggregates. A one-off spike and a consistent regression
+// look identical turn by turn; you need the distribution to tell them apart.
+//
+// Deliberately reports a FLOOR as well as a ceiling. A sub-second reply is the
+// number every voice framework optimises for, and in a turn-taking system it is
+// usually evidence the endpointer fired while the caller was still speaking.
+// Celebrating it is how "fast" agents ship that interrupt people.
+
+import type { TurnRecord, TurnTimings } from "./session-record.js";
+import { FAST_TURN_FLOOR_MS } from "./turn-timeline.js";
+
+export interface StageStats {
+  readonly stage: string;
+  readonly label: string;
+  readonly count: number;
+  readonly medianMs: number;
+  readonly p95Ms: number;
+  readonly maxMs: number;
+}
+
+export interface SessionMetrics {
+  readonly turnCount: number;
+  /** Turns that carried timings. The Workers path sends none, so this can be 0. */
+  readonly measuredTurnCount: number;
+  readonly stages: readonly StageStats[];
+  /** Turns whose end-to-end fell below the floor — probable premature endpointing. */
+  readonly suspiciouslyFastTurnIds: readonly string[];
+  readonly floorMs: number;
+  /** True when nothing could be measured, so the UI says why instead of showing zeroes. */
+  readonly unavailable: boolean;
+}
+
+const STAGES: readonly { readonly key: keyof TurnTimings; readonly stage: string; readonly label: string }[] = [
+  { key: "sttMs", stage: "stt", label: "Hearing you" },
+  { key: "llmTTFTMs", stage: "llm", label: "Thinking (to first word)" },
+  { key: "ttsTTFBMs", stage: "tts", label: "Voice (to first audio)" },
+  { key: "e2eMs", stage: "e2e", label: "End to end" },
+];
+
+/**
+ * Nearest-rank percentile. Chosen over interpolation deliberately: every value
+ * reported is a real measurement that occurred, so a p95 can be traced back to
+ * an actual turn rather than being a number no turn ever had.
+ */
+export function percentile(sortedAsc: readonly number[], p: number): number {
+  if (sortedAsc.length === 0) return 0;
+  const rank = Math.ceil((p / 100) * sortedAsc.length);
+  const idx = Math.min(sortedAsc.length - 1, Math.max(0, rank - 1));
+  return sortedAsc[idx] ?? 0;
+}
+
+export function buildSessionMetrics(
+  turns: readonly TurnRecord[],
+  floorMs: number = FAST_TURN_FLOOR_MS,
+): SessionMetrics {
+  const measured = turns.filter((t) => t.timings !== undefined);
+
+  const stages = STAGES.map(({ key, stage, label }) => {
+    const values = measured
+      .map((t) => t.timings?.[key])
+      .filter((v): v is number => typeof v === "number")
+      .sort((a, b) => a - b);
+    return {
+      stage,
+      label,
+      count: values.length,
+      medianMs: percentile(values, 50),
+      p95Ms: percentile(values, 95),
+      maxMs: values.length > 0 ? (values[values.length - 1] ?? 0) : 0,
+    };
+  }).filter((s) => s.count > 0);
+
+  const suspiciouslyFastTurnIds = measured
+    .filter((t) => {
+      const e2e = t.timings?.e2eMs;
+      return typeof e2e === "number" && e2e > 0 && e2e < floorMs;
+    })
+    .map((t) => t.turnId);
+
+  return {
+    turnCount: turns.length,
+    measuredTurnCount: measured.length,
+    stages,
+    suspiciouslyFastTurnIds,
+    floorMs,
+    unavailable: measured.length === 0,
+  };
+}