@voqalize/avatar 0.1.0 → 0.2.1

package/README.md

@@ -5,8 +5,8 @@ any face and go by any name — the demos call theirs Kiran. The server owns
 meaning (what state the agent is in, what it's saying, where it's looking); the
 client owns motion (how a face actually moves when it means those things).
 
-No build step, no runtime dependencies, ~200KB of ES modules — about two thirds
-of that is the three face rigs, and a host that ships one face pays for one.
+No build step, no runtime dependencies, ~270KB of ES modules — a third of that
+is the three face rigs, and a host that ships one face pays for one.
 
 The two load-bearing interfaces are specified in
 [docs/contract-protocol.md](docs/contract-protocol.md) (server ↔ widget) and
@@ -17,17 +17,30 @@ npm install @voqalize/avatar      # the browser half
 pip install voqalize-avatar       # the pipecat half
 ```
 
-```js
-import { createAvatar } from '@voqalize/avatar';
+```jsx
+import { Avatar } from '@voqalize/avatar';
+
+<Avatar client={pipecatClient} className="call-tile" />
+```
 
-const avatar = createAvatar({ mount: '#avatar' });
+```python
+from voqalize_avatar import AvatarProcessor
 
-avatar.setState('LISTENING', { emotion: 'warm' });
-avatar.interject('MM_HMM');
-avatar.setGaze('SCREEN_WORK');
-avatar.speak({ audio: audioEl, cues });   // cues: [{t: 0, v: 'D'}, ...]
+pipeline = Pipeline([..., tts, AvatarProcessor(), transport.output()])
 ```
 
+That is the integration, both halves of it. The processor infers the agent's
+state from the frames already flowing past it and streams viseme letters synced
+to the audio; the component renders a face that listens, thinks, claims the
+floor, speaks and yields it. Nothing to configure, no binaries to install.
+
+Under that one component is a ~30-channel parameter mixer with a full driving
+API (`setState`, `interject`, `gesture`, `speak`, `perform`, …). It is not an
+npm entrypoint — one component is the whole public surface, deliberately, and
+[docs/removed.md](docs/removed.md) says what that cost and how to undo it. The
+rest of this README documents that machinery, because it is what you read when
+you are authoring a face or debugging a mouth.
+
 ## What's in the box
 
 The repo is one system in three pieces, because that is how it gets consumed —
@@ -37,37 +50,37 @@ why this is a library rather than a product.
 
 | piece | where | what it is |
 |---|---|---|
-| the widget | `src/` → `@voqalize/avatar` | the face. Dependency-free ES modules, no build step, mounts anywhere |
-| the client | `client/` → `@voqalize/avatar/{pipecat,react}` | the dispatcher: turn clock, cue splice, React mount |
+| the widget | `src/` | the face. Dependency-free ES modules, no build step, mounts anywhere. Internal since 0.2 |
+| the client | `client/` → `@voqalize/avatar` | the dispatcher (turn clock, cue splice) and the `<Avatar>` component over it |
 | the backend | `py/` → `voqalize-avatar` (PyPI) | a pipecat `FrameProcessor` that infers state from stock frames and streams visemes |
 | the aligner | `native/avatarsync/` | the Rhubarb Lip Sync fork the backend drives — A–H letters from text *and* from audio |
 
-Browser side, three entry points:
-
-```js
-import { createAvatar, AVATAR_NAMES } from '@voqalize/avatar';         // no deps
-import { AvatarClient } from '@voqalize/avatar/pipecat';               // + pipecat client
-import { Avatar, useAvatar } from '@voqalize/avatar/react';            // + React >= 18
-```
-
-The root export is the widget itself and pulls in nothing. `/pipecat` adds
-`AvatarClient`, which anchors the turn clock and splices cue tracks; it needs
-`@pipecat-ai/client-js` only if you call `attach()`. `/react` adds a mount
-lifecycle and nothing else:
+Browser side, one entry point:
 
 ```jsx
-<Avatar client={pipecatClient} className="call-tile" />
+import { Avatar } from '@voqalize/avatar';    // peers: react >= 18, @pipecat-ai/client-js
+
+<Avatar client={pipecatClient} avatar="peep" className="call-tile" />
 ```
 
+`client` is the live `PipecatClient` (or `null` before connect); `avatar` picks
+a face and is read once, at mount. Everything else is forwarded to the mount
+`<div>`, so it sizes and styles like the tile it lives in. There is nothing to
+configure because the server already says all of it.
+
 Server side, the whole integration is one processor between your TTS and your
 output transport — see `py/` and `docs/contract-protocol.md`:
 
 ```python
-from voqalize_avatar import AvatarProcessor, AvatarStateMachine
+from voqalize_avatar import AvatarProcessor
 
-pipeline = Pipeline([..., tts, AvatarProcessor(state_machine=AvatarStateMachine()), transport.output()])
+pipeline = Pipeline([..., tts, AvatarProcessor(), transport.output()])
 ```
 
+It takes no arguments. `StartFrame` tells it the sample rate, the aligner rides
+inside the wheel, and an application that needs to say something the pipeline
+cannot infer subclasses `AvatarStateMachine` — see `py/README.md`.
+
 That much is inferred from stock pipecat frames, with no application code.
 States that depend on what your application is *doing* — a tool call that
 should read as *reviewing the screen* rather than *thinking* — are signalled
@@ -281,21 +294,11 @@ Rough recipe if you're rolling your own:
 Latency note: alignment needs the whole audio, so for streaming TTS run it per
 sentence chunk and `pushCues()` each chunk as it completes.
 
-### Tier 3 — amplitude fallback (client-side, zero server work)
-
-Already implemented in `src/audio-fallback.js`. Attach any `MediaStream`,
-`HTMLMediaElement` or WebAudio node and it derives openness from RMS and a
-rough shape family from spectral tilt (sibilant → `B`, low-heavy → `F`/`E`,
-otherwise open vowels by level).
-
-```js
-avatar.setAudioFallback(audioElement);
-```
-
-It is not lip-sync — it's a mouth that moves plausibly with the voice. Use it
-when the viseme stream is unavailable or has fallen behind. The mixer engages it
-only when nothing better is playing, so you can leave it attached permanently as
-a safety net.
+There used to be a Tier 3: a client-side amplitude/spectral guesser
+(`setAudioFallback`) for a server that sends no cues at all. It was removed in
+0.2 — there is no such server any more, and a second, lower-fidelity mouth
+standing behind the real one mostly made a broken mouth harder to diagnose. See
+[docs/removed.md](docs/removed.md) § Amplitude lipsync.
 
 ---
 
@@ -354,12 +357,11 @@ speak. The head comes *up*, not down — a lowered head reads as yielding.
 
 Backchannels only create rapport when they are *contingent* — a nod coupled to
 the speaker's pauses reads as understanding; the same nod on a random timer
-reads as distracting (the research is unambiguous on this). Give the widget the
-user's voice and the listening engine does the rest:
+reads as distracting (the research is unambiguous on this). Tell the widget when
+the user holds the floor and the listening engine does the rest:
 
 ```js
-avatar.setUserAudio(micStreamOrElement);   // internal RMS VAD; null to detach
-avatar.setUserSpeaking(true / false);      // or run your own VAD; this wins
+avatar.setUserSpeaking(true / false);      // the server endpointer's own signal
 avatar.on('backchannel', (id) => log(id)); // every autonomous ack, announced
 ```
 
@@ -367,7 +369,9 @@ While the user holds the floor the avatar leans in a touch; at pause onsets it
 acknowledges within ~250–600ms (probability, refractory gap and nod choice all
 tuned from listening-corpus numbers — long user utterances earn the bigger
 nods). With no signal attached, the old plausible random cadence remains as the
-fallback. The server can always `interject()` explicitly; autonomous acks
+fallback. (The widget could once derive this itself with its own VAD over the
+mic stream; that raced the server's endpointer and went in 0.2 —
+[docs/removed.md](docs/removed.md) § Client-side VAD.) The server can always `interject()` explicitly; autonomous acks
 suppress themselves around it.
 
 ### Action timelines
@@ -571,6 +575,36 @@ attachAudio('OKAY', '/audio/agent-okay.mp3');
 Clips ramp in over 70ms and out over 150ms, are interruptible, and a repeat of
 the clip already playing collapses rather than stacking.
 
+### Hand gestures
+
+`gesture(id)` — `HI`, `BYE`, `THUMBS_UP`, `ONE_MOMENT`. A hand rises into the
+bottom of the frame, and the matching interjection above plays with it, because
+a hand that arrives while the head sits perfectly still is not attached to
+anybody.
+
+There is still no arm. The hand enters the way a webcam sees one — only fingers
+and palm ever clear the edge, the wrist never does — which is what makes it a
+different proposition from the articulated forearm chain this project removed.
+It is not part of the rig either: no parameter channel, no per-face geometry.
+Every avatar gets it from its own `viewBox` and theme, and a face that never
+plays a gesture renders exactly what it rendered before.
+
+Two guarantees worth stating, because they are the reasons it could ship at
+all: **nothing but a single digit ever passes the mouth** (lipsync is the
+headline feature — a gesture is free to fire mid-speech), and the hand never
+leaves the frame sideways. `checkHandFraming(meta)` asserts both against the
+real timelines for every registered avatar, in `sweep()`.
+
+```js
+avatar.gesture('HI');
+avatar.setHandSide(-1);          // which side it enters from; +1 is the default
+avatar.gesturing;                // the id in flight, or null
+createAvatar({ mount, hand: false });   // no hand; gesture() plays the face half alone
+```
+
+`gesture` is a separate verb from `interject` on the wire too — `interject('WAVE')`
+is still the face alone, so a server that upgrades gets no hand until it asks.
+
 ### Misc
 
 ```js
@@ -578,11 +612,10 @@ avatar.blink(true);              // true = double blink
 avatar.setMouthGain(1.2);        // articulation: scales the viseme shapes as authored
 avatar.setGestureGain(0.8);      // scales every clip delta
 avatar.setMotionGain(0.8);       // scales the whole idle/body liveness layer
-avatar.setAudioFallback(el);     // null to detach
 avatar.setOverrides({ ... });    // direct param injection, for tuning UIs
 avatar.setOverrides(null);
 avatar.params;                   // live smoothed parameter vector (read-only)
-avatar.state / .emotion / .gaze / .speaking / .clip / .performing / .audioLevel
+avatar.state / .emotion / .gaze / .speaking / .clip / .gesturing / .performing
 avatar.mouthGain / .gestureGain / .motionGain / .svg / .meta
 avatar.destroy();
 ```
@@ -592,7 +625,7 @@ are properties, not methods. (`meta` is the mounted avatar's descriptor — the
 call demo sizes its tile from `meta.viewBox`.)
 
 Events: `state` (new state name), `speakEnd`, `clipEnd` (clip id),
-`backchannel` (autonomous ack id), `performEnd`.
+`backchannel` (autonomous ack id), `gestureEnd` (hand gesture id), `performEnd`.
 
 ---
 
@@ -613,16 +646,18 @@ Events: `state` (new state name), `speakEnd`, `clipEnd` (clip id),
 | `src/idle.js` | per-state liveness profiles, the `ListeningEngine`, autonomous backchannel |
 | `src/clips.js` | keyframe player for gesture timelines |
 | `src/interjections.js` | the 26 clips |
+| `src/hand.js` | the frame-edge hand: four gestures, placed from `META.viewBox` |
 | `src/perform.js` | the action-timeline player behind `perform()` |
-| `src/audio-fallback.js` | WebAudio amplitude/spectral lipsync |
-| `src/avatar.js` | public API, the per-frame mixer, and the `AVATARS` registry |
+| `src/avatar.js` | the driving API, the per-frame mixer, and the `AVATARS` registry |
 | `client/src/AvatarClient.ts` | the dispatcher: turn-clock anchoring and the cue splice, framework-free |
 | `client/src/useAvatar.ts` `client/src/Avatar.tsx` | the React binding — a mount lifecycle over `AvatarClient` |
+| `client/src/index.ts` | the one export: `<Avatar>` |
 | `client/src/types.ts` | the wire vocabulary in TypeScript. Kept in step with `contract-protocol.md` and `messages.py` |
 | `py/src/voqalize_avatar/` | the pipecat backend: state machine, processor, viseme engine, `avatarsync` pool |
 | `native/avatarsync/` | the Rhubarb Lip Sync fork — text leg and audio leg — plus its patch and build script |
 | `docs/contract-protocol.md` `docs/contract-avatar.md` | the two binding interface contracts |
 | `docs/design-library-split.md` | why this is a library, and what each published artifact owns |
+| `docs/removed.md` | what 0.2 deleted from the public surface, why, and how to get any of it back |
 | `docs/research-biomechanics.md` | the citations behind the motion constants |
 | `tools/` | headless render / sweep / pixel-diff CLI (dev-only dependencies) |
 | `serve.py` | the dev server — `Cache-Control: no-store`. Use this one |

package/client/dist/Avatar.d.ts

@@ -1,24 +1,27 @@
 /**
- * Avatar — a call-tile-ready wrapper around the widget.
- *
- * Thin by design: `useAvatar` does the work; this is the div it mounts into
- * plus prop plumbing.
+ * Avatar — a call-tile-ready wrapper around the widget, and the package's
+ * whole public surface.
  *
  *     <Avatar client={session.client} className="avatar-tile" />
  *
+ * Two props of its own; everything else is forwarded to the mount `<div>`, so
+ * it sizes and styles like the tile it lives in. There is nothing to configure
+ * because there is nothing the server does not already say: the
+ * `AvatarProcessor` in the pipeline drives the state, the gaze and the mouth.
+ *
  * The widget has no hot-swap-avatar API — `createFace` runs once per mount —
- * so `avatar`/`theme`/`mouthGain`/`gestureGain` are read once, at mount. To
- * switch avatars at runtime, remount with a `key` prop:
+ * so `avatar` is read once, at mount. To switch faces at runtime, remount with
+ * a `key` prop:
  *
  *     <Avatar key={name} avatar={name} client={session.client} />
  */
-import type { CSSProperties } from "react";
-import type { UseAvatarOptions } from "./useAvatar.js";
-export interface AvatarProps extends UseAvatarOptions {
-    className?: string;
-    style?: CSSProperties;
-    /** Forwarded to the mount `<div>`. */
-    "aria-label"?: string;
+import type { HTMLAttributes } from "react";
+import type { PipecatClient } from "@pipecat-ai/client-js";
+export interface AvatarProps extends HTMLAttributes<HTMLDivElement> {
+    /** The live `PipecatClient`, or `null` before connect. */
+    client?: PipecatClient | null;
+    /** Which face. Omit for the default. Read at mount only — see above. */
+    avatar?: string;
 }
-export declare function Avatar({ className, style, "aria-label": ariaLabel, ...options }: AvatarProps): import("react").JSX.Element;
+export declare function Avatar({ client, avatar, ...rest }: AvatarProps): import("react").JSX.Element;
 //# sourceMappingURL=Avatar.d.ts.map

package/client/dist/Avatar.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"Avatar.d.ts","sourceRoot":"","sources":["../src/Avatar.tsx"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;GAaG;AAEH,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,OAAO,CAAC;AAC3C,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,gBAAgB,CAAC;AAGvD,MAAM,WAAW,WAAY,SAAQ,gBAAgB;IACnD,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,KAAK,CAAC,EAAE,aAAa,CAAC;IACtB,sCAAsC;IACtC,YAAY,CAAC,EAAE,MAAM,CAAC;CACvB;AAED,wBAAgB,MAAM,CAAC,EAAE,SAAS,EAAE,KAAK,EAAE,YAAY,EAAE,SAAS,EAAE,GAAG,OAAO,EAAE,EAAE,WAAW,+BAW5F"}
+{"version":3,"file":"Avatar.d.ts","sourceRoot":"","sources":["../src/Avatar.tsx"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;GAgBG;AAEH,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,OAAO,CAAC;AAC5C,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,uBAAuB,CAAC;AAG3D,MAAM,WAAW,WAAY,SAAQ,cAAc,CAAC,cAAc,CAAC;IACjE,0DAA0D;IAC1D,MAAM,CAAC,EAAE,aAAa,GAAG,IAAI,CAAC;IAC9B,wEAAwE;IACxE,MAAM,CAAC,EAAE,MAAM,CAAC;CACjB;AAED,wBAAgB,MAAM,CAAC,EAAE,MAAM,EAAE,MAAM,EAAE,GAAG,IAAI,EAAE,EAAE,WAAW,+BAG9D"}

package/client/dist/Avatar.js

@@ -1,7 +1,7 @@
 import { jsx as _jsx } from "react/jsx-runtime";
 import { useAvatar } from "./useAvatar.js";
-export function Avatar({ className, style, "aria-label": ariaLabel, ...options }) {
-    const { containerRef } = useAvatar(options);
-    return (_jsx("div", { ref: containerRef, className: className, style: style, "aria-label": ariaLabel ?? "avatar", role: "img" }));
+export function Avatar({ client, avatar, ...rest }) {
+    const { containerRef } = useAvatar({ client, avatar });
+    return _jsx("div", { role: "img", "aria-label": "avatar", ...rest, ref: containerRef });
 }
 //# sourceMappingURL=Avatar.js.map

package/client/dist/Avatar.js.map

@@ -1 +1 @@
-{"version":3,"file":"Avatar.js","sourceRoot":"","sources":["../src/Avatar.tsx"],"names":[],"mappings":";AAiBA,OAAO,EAAE,SAAS,EAAE,MAAM,gBAAgB,CAAC;AAS3C,MAAM,UAAU,MAAM,CAAC,EAAE,SAAS,EAAE,KAAK,EAAE,YAAY,EAAE,SAAS,EAAE,GAAG,OAAO,EAAe;IAC3F,MAAM,EAAE,YAAY,EAAE,GAAG,SAAS,CAAC,OAAO,CAAC,CAAC;IAC5C,OAAO,CACL,cACE,GAAG,EAAE,YAAY,EACjB,SAAS,EAAE,SAAS,EACpB,KAAK,EAAE,KAAK,gBACA,SAAS,IAAI,QAAQ,EACjC,IAAI,EAAC,KAAK,GACV,CACH,CAAC;AACJ,CAAC"}
+{"version":3,"file":"Avatar.js","sourceRoot":"","sources":["../src/Avatar.tsx"],"names":[],"mappings":";AAoBA,OAAO,EAAE,SAAS,EAAE,MAAM,gBAAgB,CAAC;AAS3C,MAAM,UAAU,MAAM,CAAC,EAAE,MAAM,EAAE,MAAM,EAAE,GAAG,IAAI,EAAe;IAC7D,MAAM,EAAE,YAAY,EAAE,GAAG,SAAS,CAAC,EAAE,MAAM,EAAE,MAAM,EAAE,CAAC,CAAC;IACvD,OAAO,cAAK,IAAI,EAAC,KAAK,gBAAY,QAAQ,KAAK,IAAI,EAAE,GAAG,EAAE,YAAY,GAAI,CAAC;AAC7E,CAAC"}

package/client/dist/AvatarClient.d.ts

@@ -32,10 +32,10 @@
  *      it would eat into the intentional video-first safety margin rather than
  *      improve it. Left as a documented option, not built.
  *
- * `attach()` still subscribes to both pipecat events, but only to report a
- * **diagnostic** drift (`onSpeakingDrift`) between our anchor and pipecat's —
- * useful for noticing in logs if the two ever separate by more than jitter,
- * never used to move `t0` itself.
+ * `attach()` therefore subscribes to exactly one pipecat event,
+ * `serverMessage`. It used to also subscribe to both speaking events to report
+ * a diagnostic drift between our anchor and pipecat's; that hook is gone with
+ * the rest of the observability surface (`docs/removed.md` § Client callbacks).
  *
  * ## Cue splice
  *
@@ -70,48 +70,28 @@
  */
 import type { PipecatClient } from "@pipecat-ai/client-js";
 import type { AvatarApi } from "../../src/avatar.js";
-import { type AvatarCommand, type AvatarCue, type AvatarHintCmd, type AvatarUnknownCmd } from "./types.js";
+import { type AvatarCommand, type AvatarCue } from "./types.js";
+/**
+ * Internal. Not exported from the package — see `index.ts` for the public
+ * surface, which is `<Avatar>` and nothing else.
+ *
+ * There is deliberately no `accept` predicate here any more. Avatar commands
+ * travel in one envelope, `{type:"avatar"}`, in both directions and from every
+ * source: a `AvatarProcessor` in the pipeline and a brain driving the face
+ * out of band emit the same shape. A per-deployment predicate meant the
+ * library could not state what an avatar message *is*, which is the one thing
+ * a wire format has to be able to say. See docs/removed.md § The accept
+ * predicate.
+ */
 export interface AvatarClientOptions {
-    /** `{cmd:"hint"}` is a no-op hook today — the widget's listening engine
-     * already handles acks; a host may still want to know a hint arrived. */
-    onHint?: (kind: string, msg: AvatarHintCmd) => void;
-    /** An unrecognized `cmd` (forward compat) — the protocol says ignore
-     * silently, so this is purely an observability hook, not required. */
-    onUnknownCmd?: (msg: AvatarUnknownCmd) => void;
     /** A dispatch threw (e.g. an unknown state or interjection id, which the
      * widget throws on). Defaults to `console.warn`. */
     onError?: (err: unknown, msg: AvatarCommand) => void;
-    /** Diagnostic only (see the class doc's "Turn clock anchoring" section) —
-     * never moves the anchor, just reports how far pipecat's own
-     * botStartedSpeaking/botStoppedSpeaking landed from it. */
-    onSpeakingDrift?: (info: {
-        event: "start" | "stop";
-        ctx: string | null;
-        driftMs: number;
-    }) => void;
-    /**
-     * Which server-messages `attach()` should look inside. Defaults to the
-     * protocol's own envelope, `type === "avatar"`.
-     *
-     * The escape hatch exists because an application may tunnel avatar commands
-     * inside a message type of its own — one deployment routes them through a
-     * generic `ui_command` envelope so an LLM tool call can drive the face — and
-     * teaching this library that envelope would be teaching it one consumer's
-     * private vocabulary. Widen it here instead:
-     *
-     *     accept: (m) => m.type === "avatar" ||
-     *                    (m.type === "ui_command" && m.action === "avatar")
-     *
-     * The predicate only decides *whether to look*; the payload still has to
-     * carry a string `cmd` to dispatch at all.
-     */
-    accept?: (message: Record<string, unknown>) => boolean;
     /** Override for tests. Defaults to `performance.now`. */
     now?: () => number;
 }
 /**
- * The three `RTVIEvent` members `attach()` subscribes to, spelled as their
- * values.
+ * The one `RTVIEvent` member `attach()` subscribes to, spelled as its value.
  *
  * Written out rather than imported because that enum was this module's *only*
  * runtime reference to `@pipecat-ai/client-js`, and one runtime reference makes
@@ -127,43 +107,29 @@ export interface AvatarClientOptions {
  */
 export declare const RTVI_EVENTS: {
     readonly serverMessage: "serverMessage";
-    readonly botStartedSpeaking: "botStartedSpeaking";
-    readonly botStoppedSpeaking: "botStoppedSpeaking";
 };
 export declare class AvatarClient {
     private readonly avatar;
     private readonly opts;
     private readonly now;
-    private readonly accept;
     private turn;
     constructor(avatar: AvatarApi, opts?: AvatarClientOptions);
     /** The active turn's ctx, or `null` between turns. For tests and telemetry. */
     get turnCtx(): string | null;
     /** The active turn's canonical (already-spliced) cue track. For tests and telemetry. */
     get turnCues(): AvatarCue[];
-    /** Dispatch one avatar command. Accepts anything with a string `cmd` — an
-     * already-unwrapped `{type:"avatar", cmd, ...}` server message, or a bare
-     * `{cmd, ...}` payload from whatever else the host is carrying them in.
-     * Unknown `cmd`s are ignored, per the wire protocol's forward-compat rule. */
+    /** Dispatch one server message. Anything that isn't in the avatar envelope
+     * is not ours and is ignored; so is an envelope carrying a `cmd` this build
+     * has never heard of, per the wire protocol's forward-compat rule. */
     dispatch(raw: unknown): void;
     private handleState;
-    private handlePerform;
-    /** Ride the named turn's clock if it's the one we're currently anchored to;
-     * otherwise (no active turn, or `perform` names a ctx we never saw a
-     * `speech start` for) fall back to a fresh clock anchored at this call — the
-     * same "elapsed ms since this call" default `avatar.perform()` itself uses
-     * when given no clock and no audio. */
-    private resolveClock;
     private ensureTurn;
     private handleCues;
     private handleSpeech;
-    private reportDrift;
     /**
      * Subscribe to a live `PipecatClient`'s server messages and dispatch the
-     * avatar commands among them. Which messages count is the `accept` option;
-     * by default, the protocol's own `{type:"avatar"}` envelope.
-     *
-     * Also wires the diagnostic drift cross-check described in the class doc.
+     * avatar commands among them — the ones in the protocol's own
+     * `{type:"avatar"}` envelope, which `isAvatarMessage` is the definition of.
      * Never throws on a malformed or irrelevant message.
      *
      * @returns an unsubscribe function; call it on unmount or disconnect.

package/client/dist/AvatarClient.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"AvatarClient.d.ts","sourceRoot":"","sources":["../src/AvatarClient.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAqEG;AAEH,OAAO,KAAK,EAAE,aAAa,EAAa,MAAM,uBAAuB,CAAC;AACtE,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,qBAAqB,CAAC;AACrD,OAAO,EAGL,KAAK,aAAa,EAClB,KAAK,SAAS,EAEd,KAAK,aAAa,EAIlB,KAAK,gBAAgB,EACtB,MAAM,YAAY,CAAC;AAYpB,MAAM,WAAW,mBAAmB;IAClC;6EACyE;IACzE,MAAM,CAAC,EAAE,CAAC,IAAI,EAAE,MAAM,EAAE,GAAG,EAAE,aAAa,KAAK,IAAI,CAAC;IACpD;0EACsE;IACtE,YAAY,CAAC,EAAE,CAAC,GAAG,EAAE,gBAAgB,KAAK,IAAI,CAAC;IAC/C;wDACoD;IACpD,OAAO,CAAC,EAAE,CAAC,GAAG,EAAE,OAAO,EAAE,GAAG,EAAE,aAAa,KAAK,IAAI,CAAC;IACrD;;+DAE2D;IAC3D,eAAe,CAAC,EAAE,CAAC,IAAI,EAAE;QAAE,KAAK,EAAE,OAAO,GAAG,MAAM,CAAC;QAAC,GAAG,EAAE,MAAM,GAAG,IAAI,CAAC;QAAC,OAAO,EAAE,MAAM,CAAA;KAAE,KAAK,IAAI,CAAC;IACnG;;;;;;;;;;;;;;;OAeG;IACH,MAAM,CAAC,EAAE,CAAC,OAAO,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,KAAK,OAAO,CAAC;IACvD,yDAAyD;IACzD,GAAG,CAAC,EAAE,MAAM,MAAM,CAAC;CACpB;AAED;;;;;;;;;;;;;;;GAeG;AACH,eAAO,MAAM,WAAW;;;;CAImB,CAAC;AAU5C,qBAAa,YAAY;IACvB,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAY;IACnC,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAsB;IAC3C,OAAO,CAAC,QAAQ,CAAC,GAAG,CAAe;IACnC,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAgD;IACvE,OAAO,CAAC,IAAI,CAAqB;gBAErB,MAAM,EAAE,SAAS,EAAE,IAAI,GAAE,mBAAwB;IAO7D,+EAA+E;IAC/E,IAAI,OAAO,IAAI,MAAM,GAAG,IAAI,CAE3B;IAED,wFAAwF;IACxF,IAAI,QAAQ,IAAI,SAAS,EAAE,CAE1B;IAED;;;kFAG8E;IAC9E,QAAQ,CAAC,GAAG,EAAE,OAAO,GAAG,IAAI;IAsC5B,OAAO,CAAC,WAAW;IAUnB,OAAO,CAAC,aAAa;IAIrB;;;;2CAIuC;IACvC,OAAO,CAAC,YAAY;IAQpB,OAAO,CAAC,UAAU;IAWlB,OAAO,CAAC,UAAU;IAkBlB,OAAO,CAAC,YAAY;IAmBpB,OAAO,CAAC,WAAW;IAOnB;;;;;;;;;OASG;IACH,MAAM,CAAC,MAAM,EAAE,aAAa,GAAG,MAAM,IAAI;CAuB1C"}
+{"version":3,"file":"AvatarClient.d.ts","sourceRoot":"","sources":["../src/AvatarClient.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAqEG;AAEH,OAAO,KAAK,EAAE,aAAa,EAAa,MAAM,uBAAuB,CAAC;AACtE,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,qBAAqB,CAAC;AACrD,OAAO,EAEL,KAAK,aAAa,EAClB,KAAK,SAAS,EAIf,MAAM,YAAY,CAAC;AAYpB;;;;;;;;;;;GAWG;AACH,MAAM,WAAW,mBAAmB;IAClC;wDACoD;IACpD,OAAO,CAAC,EAAE,CAAC,GAAG,EAAE,OAAO,EAAE,GAAG,EAAE,aAAa,KAAK,IAAI,CAAC;IACrD,yDAAyD;IACzD,GAAG,CAAC,EAAE,MAAM,MAAM,CAAC;CACpB;AAED;;;;;;;;;;;;;;GAcG;AACH,eAAO,MAAM,WAAW;;CAEmB,CAAC;AAU5C,qBAAa,YAAY;IACvB,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAY;IACnC,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAsB;IAC3C,OAAO,CAAC,QAAQ,CAAC,GAAG,CAAe;IACnC,OAAO,CAAC,IAAI,CAAqB;gBAErB,MAAM,EAAE,SAAS,EAAE,IAAI,GAAE,mBAAwB;IAM7D,+EAA+E;IAC/E,IAAI,OAAO,IAAI,MAAM,GAAG,IAAI,CAE3B;IAED,wFAAwF;IACxF,IAAI,QAAQ,IAAI,SAAS,EAAE,CAE1B;IAED;;0EAEsE;IACtE,QAAQ,CAAC,GAAG,EAAE,OAAO,GAAG,IAAI;IAkC5B,OAAO,CAAC,WAAW;IAUnB,OAAO,CAAC,UAAU;IAWlB,OAAO,CAAC,UAAU;IAkBlB,OAAO,CAAC,YAAY;IAkBpB;;;;;;;OAOG;IACH,MAAM,CAAC,MAAM,EAAE,aAAa,GAAG,MAAM,IAAI;CAM1C"}

package/client/dist/AvatarClient.js

@@ -32,10 +32,10 @@
  *      it would eat into the intentional video-first safety margin rather than
  *      improve it. Left as a documented option, not built.
  *
- * `attach()` still subscribes to both pipecat events, but only to report a
- * **diagnostic** drift (`onSpeakingDrift`) between our anchor and pipecat's —
- * useful for noticing in logs if the two ever separate by more than jitter,
- * never used to move `t0` itself.
+ * `attach()` therefore subscribes to exactly one pipecat event,
+ * `serverMessage`. It used to also subscribe to both speaking events to report
+ * a diagnostic drift between our anchor and pipecat's; that hook is gone with
+ * the rest of the observability surface (`docs/removed.md` § Client callbacks).
  *
  * ## Cue splice
  *
@@ -68,10 +68,9 @@
  * chunk of a turn starts speak()" means the first *widget* call, not
  * necessarily the first *message*.
  */
-import { AVATAR_MESSAGE_TYPE, isAvatarMessage, } from "./types.js";
+import { isAvatarMessage, } from "./types.js";
 /**
- * The three `RTVIEvent` members `attach()` subscribes to, spelled as their
- * values.
+ * The one `RTVIEvent` member `attach()` subscribes to, spelled as its value.
  *
  * Written out rather than imported because that enum was this module's *only*
  * runtime reference to `@pipecat-ai/client-js`, and one runtime reference makes
@@ -87,8 +86,6 @@ import { AVATAR_MESSAGE_TYPE, isAvatarMessage, } from "./types.js";
  */
 export const RTVI_EVENTS = {
     serverMessage: "serverMessage",
-    botStartedSpeaking: "botStartedSpeaking",
-    botStoppedSpeaking: "botStoppedSpeaking",
 };
 /** Defensive unwrap for the `RTVIEvent.ServerMessage` `{ data }` quirk: some
  * transports deliver the payload directly and some wrap it once more. */
@@ -101,13 +98,11 @@ export class AvatarClient {
     avatar;
     opts;
     now;
-    accept;
     turn = null;
     constructor(avatar, opts = {}) {
         this.avatar = avatar;
         this.opts = opts;
         this.now = opts.now ?? (() => performance.now());
-        this.accept = opts.accept ?? ((m) => m.type === AVATAR_MESSAGE_TYPE);
     }
     /** The active turn's ctx, or `null` between turns. For tests and telemetry. */
     get turnCtx() {
@@ -117,10 +112,9 @@ export class AvatarClient {
     get turnCues() {
         return this.turn ? [...this.turn.cues] : [];
     }
-    /** Dispatch one avatar command. Accepts anything with a string `cmd` — an
-     * already-unwrapped `{type:"avatar", cmd, ...}` server message, or a bare
-     * `{cmd, ...}` payload from whatever else the host is carrying them in.
-     * Unknown `cmd`s are ignored, per the wire protocol's forward-compat rule. */
+    /** Dispatch one server message. Anything that isn't in the avatar envelope
+     * is not ours and is ignored; so is an envelope carrying a `cmd` this build
+     * has never heard of, per the wire protocol's forward-compat rule. */
     dispatch(raw) {
         if (!isAvatarMessage(raw))
             return;
@@ -133,8 +127,8 @@ export class AvatarClient {
                 case "interject":
                     this.avatar.interject(msg.id);
                     break;
-                case "perform":
-                    this.handlePerform(msg);
+                case "gesture":
+                    this.avatar.gesture(msg.id);
                     break;
                 case "cues":
                     this.handleCues(msg);
@@ -145,14 +139,10 @@ export class AvatarClient {
                 case "user":
                     this.avatar.setUserSpeaking(msg.speaking);
                     break;
-                case "hint": {
-                    const hint = msg;
-                    this.opts.onHint?.(hint.kind, hint);
-                    break;
-                }
-                default:
-                    this.opts.onUnknownCmd?.(msg);
-                    break;
+                // No default: an unknown `cmd` is a newer server talking to an older
+                // widget, and the protocol's forward-compat rule says ignore it. There
+                // is no callback for it — a hook nobody could act on is observability,
+                // not an interface (`docs/removed.md` § Client callbacks).
             }
         }
         catch (err) {
@@ -171,28 +161,13 @@ export class AvatarClient {
         // particular message takes effect.
         this.avatar.setState(msg.name, { emotion: msg.emotion, gaze: msg.gaze });
     }
-    handlePerform(msg) {
-        this.avatar.perform(msg.actions, { clock: this.resolveClock(msg.ctx) });
-    }
-    /** Ride the named turn's clock if it's the one we're currently anchored to;
-     * otherwise (no active turn, or `perform` names a ctx we never saw a
-     * `speech start` for) fall back to a fresh clock anchored at this call — the
-     * same "elapsed ms since this call" default `avatar.perform()` itself uses
-     * when given no clock and no audio. */
-    resolveClock(ctx) {
-        if (ctx && this.turn && this.turn.ctx === ctx && this.turn.clock) {
-            return this.turn.clock;
-        }
-        const start = this.now();
-        return () => this.now() - start;
-    }
     ensureTurn(ctx) {
         if (!this.turn || this.turn.ctx !== ctx) {
             // A different ctx supersedes whatever turn we had — a stale trailing
             // message for the old ctx will find `this.turn.ctx !== ctx` in
             // handleSpeech's stop-guard and be ignored, rather than cutting off the
             // new turn.
-            this.turn = { ctx, cues: [], started: false, clock: null, t0: null };
+            this.turn = { ctx, cues: [], started: false, clock: null };
         }
         return this.turn;
     }
@@ -218,7 +193,6 @@ export class AvatarClient {
             const turn = this.ensureTurn(msg.ctx);
             const t0 = this.now();
             const clock = () => this.now() - t0;
-            turn.t0 = t0;
             turn.clock = clock;
             turn.started = true;
             this.avatar.speak({ cues: turn.cues, clock });
@@ -231,44 +205,19 @@ export class AvatarClient {
             this.turn = null;
         }
     }
-    reportDrift(event) {
-        if (!this.opts.onSpeakingDrift)
-            return;
-        const t0 = this.turn?.t0;
-        if (t0 == null)
-            return;
-        this.opts.onSpeakingDrift({ event, ctx: this.turn?.ctx ?? null, driftMs: this.now() - t0 });
-    }
     /**
      * Subscribe to a live `PipecatClient`'s server messages and dispatch the
-     * avatar commands among them. Which messages count is the `accept` option;
-     * by default, the protocol's own `{type:"avatar"}` envelope.
-     *
-     * Also wires the diagnostic drift cross-check described in the class doc.
+     * avatar commands among them — the ones in the protocol's own
+     * `{type:"avatar"}` envelope, which `isAvatarMessage` is the definition of.
      * Never throws on a malformed or irrelevant message.
      *
      * @returns an unsubscribe function; call it on unmount or disconnect.
      */
     attach(client) {
-        const onServerMessage = (raw) => {
-            const message = unwrapServerMessage(raw);
-            if (!this.accept(message))
-                return;
-            this.dispatch(message);
-        };
-        const onBotStartedSpeaking = () => this.reportDrift("start");
-        const onBotStoppedSpeaking = () => this.reportDrift("stop");
+        const onServerMessage = (raw) => this.dispatch(unwrapServerMessage(raw));
         const serverMessage = RTVI_EVENTS.serverMessage;
-        const started = RTVI_EVENTS.botStartedSpeaking;
-        const stopped = RTVI_EVENTS.botStoppedSpeaking;
         client.on(serverMessage, onServerMessage);
-        client.on(started, onBotStartedSpeaking);
-        client.on(stopped, onBotStoppedSpeaking);
-        return () => {
-            client.off(serverMessage, onServerMessage);
-            client.off(started, onBotStartedSpeaking);
-            client.off(stopped, onBotStoppedSpeaking);
-        };
+        return () => client.off(serverMessage, onServerMessage);
     }
 }
 //# sourceMappingURL=AvatarClient.js.map