pipecat-anva 0.1.0__tar.gz

pipecat_anva-0.1.0/.gitignore

@@ -0,0 +1,6 @@
+.npmrc
+dist/
+node_modules/
+__pycache__/
+*.egg-info/
+.venv/

pipecat_anva-0.1.0/PKG-INFO

@@ -0,0 +1,103 @@
+Metadata-Version: 2.5
+Name: pipecat-anva
+Version: 0.1.0
+Summary: Anva video avatar service for Pipecat: a lip-synced face for your voice agent, on any transport.
+Project-URL: Homepage, https://anva.ai
+Project-URL: Documentation, https://anva.ai/docs#pipecat
+Project-URL: Source, https://github.com/Anva-avatars/anva-sdk
+Author-email: Anva <anva.ai.2026@gmail.com>
+License: MIT
+Keywords: anva,avatar,daily,pipecat,voice,webrtc
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Multimedia :: Video
+Requires-Python: >=3.10
+Requires-Dist: aiohttp>=3.9
+Requires-Dist: aiortc>=1.9
+Requires-Dist: av>=12
+Requires-Dist: numpy>=1.24
+Requires-Dist: pipecat-ai>=1.0
+Requires-Dist: websockets>=14
+Description-Content-Type: text/markdown
+
+# Anva avatars for Pipecat
+
+Give your Pipecat voice agent a face. The pipeline keeps its own STT, LLM and
+TTS; `AnvaVideoService` sits after the TTS, sends its audio to Anva and pushes
+back the avatar's video and voice, lips in time, as ordinary output frames for
+whatever transport you use: Daily, LiveKit, WebRTC, anything with video out.
+
+```sh
+pip install pipecat-anva
+```
+
+```python
+from pipecat_anva import AnvaVideoService
+
+anva = AnvaVideoService(api_key=os.environ["ANVA_API_KEY"], avatar_id="av_...")
+
+pipeline = Pipeline([
+    transport.input(),
+    stt,
+    context_aggregator.user(),
+    llm,
+    tts,
+    anva,                     # after the TTS, before the transport
+    transport.output(),
+    context_aggregator.assistant(),
+])
+```
+
+Give the transport video out at the avatar's size, for example with Daily:
+
+```python
+DailyParams(video_out_enabled=True, video_out_width=480, video_out_height=480, ...)
+```
+
+Anva renders 480×480 at 25 fps by default. The service does not scale the
+picture; set `video_out_width`/`video_out_height` to what the avatar sends
+or let your transport scale it.
+
+## What happens
+
+1. On pipeline start the service creates an Anva `avatar_only` session with
+   your API key, opens its events socket and connects a WebRTC viewer to it,
+   the way Anva's embed player does. Nothing is billed until the viewer is
+   connected.
+2. `TTSAudioRawFrame`s are converted to what Anva's lip-sync takes (24 kHz
+   mono PCM) and sent as one utterance per `TTSStartedFrame` /
+   `TTSStoppedFrame`. The pipeline's own copy of the audio stops at the
+   service.
+3. The avatar's video comes back as `OutputImageRawFrame`s (RGB) and its
+   voice as `TTSAudioRawFrame`s at the pipeline's TTS sample rate, in sync.
+4. An `InterruptionFrame` stops the avatar mid-sentence and drops what was
+   queued.
+5. `EndFrame` / `CancelFrame` end the Anva session, which is what stops the
+   billing.
+
+Sending is paced by Anva's playback reports: the service keeps at most four
+seconds of unplayed audio in flight, so a long answer is never front-loaded.
+
+## Options
+
+| Argument | Meaning |
+|---|---|
+| `api_key` | Anva API key (anva.ai → API keys). The events socket needs the Developer plan or above. |
+| `avatar_id` / `preset_id` | Exactly one. A preset's voice and prompt do not apply: the pipeline speaks. |
+| `api_url` | Anva base URL (`https://anva.ai`). |
+| `max_duration_seconds` | End the Anva session this long after it goes live. |
+| `metadata` | Echoed on Anva's session and webhooks. |
+
+A refused session (wrong avatar, plan limits, no capacity) is reported with
+`push_error`; the pipeline keeps running without a face.
+
+## Tests
+
+The tests run against a stand-in for Anva, no account needed:
+
+```sh
+pip install -e . pytest pytest-asyncio
+python -m pytest tests
+```

pipecat_anva-0.1.0/README.md

@@ -0,0 +1,79 @@
+# Anva avatars for Pipecat
+
+Give your Pipecat voice agent a face. The pipeline keeps its own STT, LLM and
+TTS; `AnvaVideoService` sits after the TTS, sends its audio to Anva and pushes
+back the avatar's video and voice, lips in time, as ordinary output frames for
+whatever transport you use: Daily, LiveKit, WebRTC, anything with video out.
+
+```sh
+pip install pipecat-anva
+```
+
+```python
+from pipecat_anva import AnvaVideoService
+
+anva = AnvaVideoService(api_key=os.environ["ANVA_API_KEY"], avatar_id="av_...")
+
+pipeline = Pipeline([
+    transport.input(),
+    stt,
+    context_aggregator.user(),
+    llm,
+    tts,
+    anva,                     # after the TTS, before the transport
+    transport.output(),
+    context_aggregator.assistant(),
+])
+```
+
+Give the transport video out at the avatar's size, for example with Daily:
+
+```python
+DailyParams(video_out_enabled=True, video_out_width=480, video_out_height=480, ...)
+```
+
+Anva renders 480×480 at 25 fps by default. The service does not scale the
+picture; set `video_out_width`/`video_out_height` to what the avatar sends
+or let your transport scale it.
+
+## What happens
+
+1. On pipeline start the service creates an Anva `avatar_only` session with
+   your API key, opens its events socket and connects a WebRTC viewer to it,
+   the way Anva's embed player does. Nothing is billed until the viewer is
+   connected.
+2. `TTSAudioRawFrame`s are converted to what Anva's lip-sync takes (24 kHz
+   mono PCM) and sent as one utterance per `TTSStartedFrame` /
+   `TTSStoppedFrame`. The pipeline's own copy of the audio stops at the
+   service.
+3. The avatar's video comes back as `OutputImageRawFrame`s (RGB) and its
+   voice as `TTSAudioRawFrame`s at the pipeline's TTS sample rate, in sync.
+4. An `InterruptionFrame` stops the avatar mid-sentence and drops what was
+   queued.
+5. `EndFrame` / `CancelFrame` end the Anva session, which is what stops the
+   billing.
+
+Sending is paced by Anva's playback reports: the service keeps at most four
+seconds of unplayed audio in flight, so a long answer is never front-loaded.
+
+## Options
+
+| Argument | Meaning |
+|---|---|
+| `api_key` | Anva API key (anva.ai → API keys). The events socket needs the Developer plan or above. |
+| `avatar_id` / `preset_id` | Exactly one. A preset's voice and prompt do not apply: the pipeline speaks. |
+| `api_url` | Anva base URL (`https://anva.ai`). |
+| `max_duration_seconds` | End the Anva session this long after it goes live. |
+| `metadata` | Echoed on Anva's session and webhooks. |
+
+A refused session (wrong avatar, plan limits, no capacity) is reported with
+`push_error`; the pipeline keeps running without a face.
+
+## Tests
+
+The tests run against a stand-in for Anva, no account needed:
+
+```sh
+pip install -e . pytest pytest-asyncio
+python -m pytest tests
+```

pipecat_anva-0.1.0/pipecat_anva/__init__.py

@@ -0,0 +1,15 @@
+"""Anva avatars for Pipecat.
+
+    from pipecat_anva import AnvaVideoService
+
+    avatar = AnvaVideoService(api_key=..., avatar_id="av_...")
+    pipeline = Pipeline([transport.input(), stt, llm, tts, avatar, transport.output()])
+
+The service takes the TTS audio, sends it to Anva, and puts back the avatar's
+video and its audio, in sync, for whatever transport the pipeline uses.
+"""
+from .session import AnvaError, AnvaSession
+from .version import __version__
+from .video import AnvaVideoService
+
+__all__ = ["AnvaVideoService", "AnvaSession", "AnvaError", "__version__"]

pipecat_anva-0.1.0/pipecat_anva/session.py

@@ -0,0 +1,409 @@
+"""An Anva ``avatar_only`` session driven from Python.
+
+Three connections make one avatar: the REST call that creates the session, the
+events socket that carries speech in and progress out, and a WebRTC viewer that
+receives the avatar's video and audio -- the same media a browser would get
+from Anva's embed player, decoded here instead. Speech goes in as raw PCM in
+utterances; Anva's core plays each one through the avatar and reports what it
+accepted and played, which is what paces the sending.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import base64
+import json
+import logging
+import uuid
+from dataclasses import dataclass, field
+from typing import Any, Awaitable, Callable
+
+import aiohttp
+import numpy as np
+import websockets
+from aiortc import RTCPeerConnection, RTCSessionDescription
+from av.audio.frame import AudioFrame
+from av.audio.resampler import AudioResampler
+
+log = logging.getLogger("pipecat_anva")
+
+DEFAULT_API_URL = "https://anva.ai"
+#: What Anva's speech ingress takes.
+CORE_RATE = 24000
+#: The most one append may carry.
+CHUNK_SAMPLES = 12000
+#: Keep a second under the core's five-second window of unplayed audio.
+OUTSTANDING_LIMIT = 4 * CORE_RATE
+#: Anva's longest utterance.
+UTTERANCE_LIMIT = 120 * CORE_RATE
+
+FrameCallback = Callable[[Any], Awaitable[None]]
+
+
+class AnvaError(Exception):
+    """Anva could not start or run the session."""
+
+
+async def _cancel_task(task: asyncio.Task) -> None:
+    task.cancel()
+    try:
+        await task
+    except (asyncio.CancelledError, Exception):  # noqa: BLE001
+        pass
+
+
+@dataclass
+class Utterance:
+    """One stretch of speech from the agent, converted to core samples."""
+
+    turn_id: str
+    pending: bytearray = field(default_factory=bytearray)
+    pushed: int = 0
+    closed: bool = False
+    cancelled: bool = False
+    wake: asyncio.Event = field(default_factory=asyncio.Event)
+
+    def push(self, pcm: bytes) -> None:
+        if self.cancelled or not pcm:
+            return
+        room = UTTERANCE_LIMIT - self.pushed
+        if room <= 0:
+            return
+        pcm = pcm[: room * 2]
+        self.pending.extend(pcm)
+        self.pushed += len(pcm) // 2
+        self.wake.set()
+
+    def close(self) -> None:
+        self.closed = True
+        self.wake.set()
+
+    def cancel(self) -> None:
+        self.cancelled = True
+        self.pending.clear()
+        self.wake.set()
+
+    def take(self) -> bytes:
+        """A full chunk, or whatever is left once the utterance is closed."""
+        if len(self.pending) >= CHUNK_SAMPLES * 2 or (self.closed and self.pending):
+            n = min(len(self.pending), CHUNK_SAMPLES * 2)
+            chunk = bytes(self.pending[:n])
+            del self.pending[:n]
+            return chunk
+        return b""
+
+
+class AnvaSession:
+    """Create, watch and speak through one Anva avatar session.
+
+    ``on_video`` and ``on_audio`` receive PyAV frames as they arrive from the
+    avatar. ``task_factory`` is how background tasks are started, so a host
+    such as Pipecat can own them.
+    """
+
+    def __init__(
+        self,
+        *,
+        api_key: str,
+        avatar_id: str | None = None,
+        preset_id: str | None = None,
+        api_url: str = DEFAULT_API_URL,
+        max_duration_seconds: int | None = None,
+        metadata: dict[str, Any] | None = None,
+        on_video: FrameCallback | None = None,
+        on_audio: FrameCallback | None = None,
+        task_factory: Callable[[Awaitable[Any]], asyncio.Task] | None = None,
+        task_cancel: Callable[[asyncio.Task], Awaitable[None]] | None = None,
+    ) -> None:
+        if bool(avatar_id) == bool(preset_id):
+            raise AnvaError("give exactly one of avatar_id and preset_id")
+        if not api_key:
+            raise AnvaError("api_key is required")
+        self._api_key = api_key
+        self._avatar_id, self._preset_id = avatar_id, preset_id
+        self._api_url = api_url.rstrip("/")
+        self._max_duration_seconds = max_duration_seconds
+        self._metadata = metadata
+        self._on_video, self._on_audio = on_video, on_audio
+        self._task_factory = task_factory or asyncio.create_task
+        self._task_cancel = task_cancel or _cancel_task
+        self._http: aiohttp.ClientSession | None = None
+        self._ws: Any = None
+        self._pc: RTCPeerConnection | None = None
+        self._tasks: list[asyncio.Task] = []
+        self.session_id: str | None = None
+        self.session_token: str | None = None
+        self.live = asyncio.Event()
+        self.ended = asyncio.Event()
+        # Every sentence the pipeline speaks waits its turn; an interruption drains it.
+        self._queue: asyncio.Queue[Utterance] = asyncio.Queue()
+        self._current: Utterance | None = None
+        self._states: asyncio.Queue[dict] = asyncio.Queue(maxsize=64)
+        self._turns = 0
+        self._resampler: AudioResampler | None = None
+        self._resampler_key: tuple[int, int] | None = None
+
+    # -- lifecycle ---------------------------------------------------------
+    async def start(self) -> None:
+        self._http = aiohttp.ClientSession()
+        await self._create()
+        await self._connect_events()
+        await self._attach_viewer()
+        try:
+            await asyncio.wait_for(self.live.wait(), timeout=60)
+        except asyncio.TimeoutError as exc:
+            raise AnvaError("Anva did not report the session live within 60 s") from exc
+        self._tasks.append(self._task_factory(self._feed()))
+
+    async def close(self) -> None:
+        for task in list(self._tasks):
+            await self._task_cancel(task)
+        self._tasks.clear()
+        if self._pc is not None:
+            await self._pc.close()
+            self._pc = None
+        if self._ws is not None:
+            try:
+                await self._ws.close()
+            except Exception:  # noqa: BLE001
+                pass
+            self._ws = None
+        if self._http is not None:
+            if self.session_id:
+                try:
+                    async with self._http.delete(
+                        f"{self._api_url}/api/v2/sessions/{self.session_id}", headers=self._auth(),
+                        timeout=aiohttp.ClientTimeout(total=15),
+                    ):
+                        pass
+                except Exception:  # noqa: BLE001
+                    log.debug("could not end the Anva session", exc_info=True)
+            await self._http.close()
+            self._http = None
+        self.ended.set()
+
+    def _auth(self) -> dict[str, str]:
+        return {"Authorization": f"Bearer {self._api_key}"}
+
+    async def _create(self) -> None:
+        assert self._http is not None
+        body: dict[str, Any] = {"service_mode": "avatar_only"}
+        if self._avatar_id:
+            body["avatar_id"] = self._avatar_id
+        else:
+            body["preset_id"] = self._preset_id
+        if self._max_duration_seconds:
+            body["max_duration_seconds"] = self._max_duration_seconds
+        if self._metadata:
+            body["metadata"] = self._metadata
+        async with self._http.post(f"{self._api_url}/api/v2/sessions", json=body, headers=self._auth(),
+                                   timeout=aiohttp.ClientTimeout(total=30)) as response:
+            text = await response.text()
+            if response.status != 201:
+                raise AnvaError(f"Anva refused the session ({response.status}): {text}")
+            data = json.loads(text)
+        self.session_id = data["session_id"]
+        self.session_token = data["session_token"]
+        self._events_url = data["events_ws_url"]
+        log.info("Anva session %s created", self.session_id)
+
+    async def _connect_events(self) -> None:
+        self._ws = await websockets.connect(self._events_url, additional_headers=self._auth(), max_size=1 << 20)
+        self._tasks.append(self._task_factory(self._read_events()))
+
+    async def _read_events(self) -> None:
+        assert self._ws is not None
+        try:
+            async for raw in self._ws:
+                try:
+                    event = json.loads(raw)
+                except ValueError:
+                    continue
+                typ, payload = event.get("type"), event.get("payload") or {}
+                if typ == "session.live":
+                    self.live.set()
+                elif typ == "session.ended":
+                    log.info("Anva session %s ended: %s", self.session_id, payload.get("reason"))
+                    self.ended.set()
+                elif typ == "speech.state":
+                    if self._states.full():
+                        self._states.get_nowait()
+                    self._states.put_nowait(payload)
+                elif typ == "error":
+                    log.warning("Anva session %s error: %s", self.session_id, payload)
+        except websockets.ConnectionClosed:
+            log.info("Anva events socket closed for %s", self.session_id)
+            self.ended.set()
+
+    async def _attach_viewer(self) -> None:
+        """Receive the avatar over WebRTC, as the embed player does."""
+        assert self._http is not None
+        pc = RTCPeerConnection()
+        self._pc = pc
+        pc.addTransceiver("video", direction="recvonly")
+        pc.addTransceiver("audio", direction="recvonly")
+
+        @pc.on("track")
+        def on_track(track):
+            self._tasks.append(self._task_factory(self._consume(track)))
+
+        offer = await pc.createOffer()
+        await pc.setLocalDescription(offer)
+        body = {"session_id": self.session_id, "session_url_token": self.session_token,
+                "sdp": pc.localDescription.sdp, "type": pc.localDescription.type,
+                "client_instance_id": "pipecat-anva-" + uuid.uuid4().hex[:12], "client_generation": 1}
+        for attempt in range(8):
+            async with self._http.post(f"{self._api_url}/offer", json=body,
+                                       timeout=aiohttp.ClientTimeout(total=60)) as response:
+                text = await response.text()
+                if response.status == 200:
+                    answer = json.loads(text)
+                    break
+                # Capacity and warming answers say when to try again.
+                retry_after = 5
+                try:
+                    retry_after = int(json.loads(text).get("retry_after", retry_after))
+                except (ValueError, AttributeError):
+                    pass
+                if response.status == 503 and attempt < 7:
+                    log.info("Anva viewer not admitted yet (%s); retrying in %ss", text[:120], retry_after)
+                    await asyncio.sleep(retry_after)
+                    continue
+                raise AnvaError(f"Anva refused the viewer ({response.status}): {text}")
+        await pc.setRemoteDescription(RTCSessionDescription(sdp=answer["sdp"], type=answer["type"]))
+
+    async def _consume(self, track) -> None:
+        callback = self._on_video if track.kind == "video" else self._on_audio
+        try:
+            while True:
+                frame = await track.recv()
+                if callback is not None:
+                    await callback(frame)
+        except Exception:  # noqa: BLE001  the track ended with the connection
+            log.debug("Anva %s track ended", track.kind, exc_info=True)
+
+    # -- speech in ---------------------------------------------------------
+    def begin_utterance(self) -> Utterance:
+        """The agent starts speaking; audio follows in push_audio."""
+        self._turns += 1
+        u = Utterance(turn_id=f"pipecat-{self._turns}")
+        self._queue.put_nowait(u)
+        return u
+
+    def push_audio(self, u: Utterance, pcm: bytes, sample_rate: int, channels: int) -> None:
+        """Add TTS audio to an utterance, converting it to what Anva takes."""
+        if u.cancelled or not pcm:
+            return
+        if channels == 1 and sample_rate == CORE_RATE:
+            u.push(pcm)
+            return
+        if self._resampler_key != (sample_rate, channels):
+            self._resampler = AudioResampler("s16", "mono", CORE_RATE)
+            self._resampler_key = (sample_rate, channels)
+        frame = AudioFrame.from_ndarray(np.frombuffer(pcm, dtype=np.int16)[None, :],
+                                        layout="mono" if channels == 1 else "stereo")
+        frame.sample_rate = sample_rate
+        assert self._resampler is not None
+        for out in self._resampler.resample(frame):
+            u.push(out.to_ndarray().astype(np.int16).tobytes())
+
+    def end_utterance(self, u: Utterance) -> None:
+        if self._resampler is not None:
+            for out in self._resampler.resample(None):
+                u.push(out.to_ndarray().astype(np.int16).tobytes())
+            self._resampler, self._resampler_key = None, None
+        u.close()
+
+    async def interrupt(self) -> None:
+        """Stop what is playing and drop what is queued."""
+        while not self._queue.empty():
+            self._queue.get_nowait().cancel()
+        if self._current is not None:
+            self._current.cancel()
+        # What the resampler still holds belonged to the cut-off sentence.
+        self._resampler, self._resampler_key = None, None
+
+    async def _send(self, typ: str, payload: dict[str, Any]) -> None:
+        if self._ws is None:
+            raise AnvaError("events socket is not connected")
+        await self._ws.send(json.dumps({"type": typ, "payload": payload}))
+
+    async def _feed(self) -> None:
+        """Serve utterances to the core one at a time, within its window."""
+        while True:
+            u = await self._queue.get()
+            self._current = u
+            try:
+                await self._serve(u)
+            except asyncio.CancelledError:
+                raise
+            except Exception:  # noqa: BLE001
+                log.warning("Anva utterance %s failed", u.turn_id, exc_info=True)
+            finally:
+                self._current = None
+
+    async def _serve(self, u: Utterance) -> None:
+        if u.cancelled:
+            return
+        await self._send("speech.start", {"turn_id": u.turn_id, "codec": "pcm_s16le",
+                                          "sample_rate": CORE_RATE, "channels": 1})
+        sent = played = seq = 0
+        done = False
+
+        def note(state: dict) -> None:
+            nonlocal played, done
+            if state.get("turn_id") != u.turn_id:
+                return
+            played = max(played, int(state.get("played_samples") or 0))
+            if state.get("state") in ("finished", "cancelled", "error"):
+                done = True
+
+        while not done:
+            if u.cancelled:
+                await self._send("speech.cancel", {"turn_id": u.turn_id})
+                return
+            chunk = u.take()
+            if chunk:
+                while sent - played > OUTSTANDING_LIMIT and not u.cancelled:
+                    try:
+                        note(await asyncio.wait_for(self._states.get(), timeout=0.25))
+                    except asyncio.TimeoutError:
+                        pass
+                if u.cancelled:
+                    continue
+                await self._send("speech.append", {"turn_id": u.turn_id, "seq": seq, "start_sample": sent,
+                                                   "data": base64.b64encode(chunk).decode()})
+                seq += 1
+                sent += len(chunk) // 2
+                continue
+            if u.closed:
+                if sent == 0:
+                    await self._send("speech.cancel", {"turn_id": u.turn_id})
+                    return
+                await self._send("speech.done", {"turn_id": u.turn_id, "total_samples": sent})
+                remaining = (sent - played) / CORE_RATE + 10
+                deadline = asyncio.get_running_loop().time() + remaining
+                while not done and not u.cancelled:
+                    wait = deadline - asyncio.get_running_loop().time()
+                    if wait <= 0:
+                        log.info("Anva did not confirm playback of %s; moving on", u.turn_id)
+                        return
+                    try:
+                        note(await asyncio.wait_for(self._states.get(), timeout=min(wait, 0.5)))
+                    except asyncio.TimeoutError:
+                        pass
+                if u.cancelled and not done:
+                    await self._send("speech.cancel", {"turn_id": u.turn_id})
+                return
+            # Nothing ready: wait for audio, a report, or the end.
+            u.wake.clear()
+            state_wait = asyncio.ensure_future(self._states.get())
+            wake_wait = asyncio.ensure_future(u.wake.wait())
+            finished, _ = await asyncio.wait({state_wait, wake_wait}, return_when=asyncio.FIRST_COMPLETED)
+            for task in (state_wait, wake_wait):
+                if task in finished:
+                    result = task.result()
+                    if task is state_wait:
+                        note(result)
+                else:
+                    task.cancel()

pipecat_anva-0.1.0/pipecat_anva/version.py

@@ -0,0 +1 @@
+__version__ = "0.1.0"

pipecat_anva-0.1.0/pipecat_anva/video.py

@@ -0,0 +1,165 @@
+"""Anva as a Pipecat video service.
+
+Sits after the TTS in the pipeline. The TTS audio goes to Anva instead of the
+transport; what comes back is the avatar's video and its audio, in sync, which
+the service pushes downstream as ordinary output frames for whatever transport
+the pipeline uses -- Daily, LiveKit, a phone line's video, a browser.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any
+
+import numpy as np
+from av.audio.resampler import AudioResampler
+from pipecat.frames.frames import (
+    CancelFrame,
+    EndFrame,
+    Frame,
+    InterruptionFrame,
+    OutputImageRawFrame,
+    TTSAudioRawFrame,
+    TTSStartedFrame,
+    TTSStoppedFrame,
+)
+from pipecat.processors.frame_processor import FrameDirection, FrameProcessorSetup
+from pipecat.services.ai_service import AIService
+from pipecat.services.settings import ServiceSettings
+
+from .session import AnvaSession, Utterance
+
+
+@dataclass
+class AnvaVideoSettings(ServiceSettings):
+    """Settings for the Anva video service."""
+
+
+class AnvaVideoService(AIService):
+    """Anva video service: a lip-synced avatar for the pipeline's TTS.
+
+    Consumes ``TTSAudioRawFrame``s and produces ``OutputImageRawFrame``s and
+    ``TTSAudioRawFrame``s carrying the avatar's synchronized video and audio.
+    An interruption stops the avatar mid-sentence.
+    """
+
+    Settings = AnvaVideoSettings
+    _settings: Settings
+
+    def __init__(
+        self,
+        *,
+        api_key: str,
+        avatar_id: str | None = None,
+        preset_id: str | None = None,
+        api_url: str = "https://anva.ai",
+        max_duration_seconds: int | None = None,
+        metadata: dict[str, Any] | None = None,
+        settings: Settings | None = None,
+        **kwargs,
+    ):
+        """Create the service.
+
+        Args:
+            api_key: Anva API key.
+            avatar_id: An avatar from your Anva account. Exactly one of this and preset_id.
+            preset_id: A saved preset. Its voice and prompt do not apply: the pipeline speaks.
+            api_url: Anva base URL.
+            max_duration_seconds: End the Anva session this long after it goes live.
+            metadata: Echoed on the Anva session and its webhooks.
+            settings: Service settings.
+            **kwargs: Passed to AIService.
+        """
+        default_settings = ServiceSettings(model=None)
+        if settings is not None:
+            default_settings.apply_update(settings)
+        super().__init__(settings=default_settings, **kwargs)
+        self._session = AnvaSession(
+            api_key=api_key, avatar_id=avatar_id, preset_id=preset_id, api_url=api_url,
+            max_duration_seconds=max_duration_seconds, metadata=metadata,
+            on_video=self._on_video, on_audio=self._on_audio,
+            task_factory=self.create_task, task_cancel=self.cancel_task,
+        )
+        self._utterance: Utterance | None = None
+        self._out_rate = 24000
+        self._out_resampler: AudioResampler | None = None
+        self._out_key: tuple[int, int] | None = None
+        self._started = False
+
+    async def setup(self, setup: FrameProcessorSetup):
+        await super().setup(setup)
+        await self._start_connection()
+
+    async def stop(self, frame: EndFrame):
+        await super().stop(frame)
+        await self._stop_connection()
+
+    async def cancel(self, frame: CancelFrame):
+        await super().cancel(frame)
+        await self._stop_connection()
+
+    async def cleanup(self):
+        await super().cleanup()
+        await self._stop_connection()
+
+    async def _start_connection(self):
+        if self._started:
+            return
+        try:
+            await self._session.start()
+            self._started = True
+        except Exception as e:  # noqa: BLE001
+            await self.push_error(error_msg=f"Unable to start the Anva avatar: {e}", exception=e)
+
+    async def _stop_connection(self):
+        if not self._started:
+            return
+        self._started = False
+        await self._session.close()
+
+    async def process_frame(self, frame: Frame, direction: FrameDirection):
+        await super().process_frame(frame, direction)
+        if not self._started:
+            # No avatar: the pipeline speaks on its own.
+            await self.push_frame(frame, direction)
+            return
+        if isinstance(frame, TTSStartedFrame):
+            self._utterance = self._session.begin_utterance()
+        elif isinstance(frame, TTSAudioRawFrame):
+            # The avatar's own audio comes back in sync with its face; the
+            # pipeline's copy stops here.
+            self._out_rate = frame.sample_rate
+            if self._utterance is None:
+                self._utterance = self._session.begin_utterance()
+            self._session.push_audio(self._utterance, frame.audio, frame.sample_rate, frame.num_channels)
+            return
+        elif isinstance(frame, TTSStoppedFrame):
+            if self._utterance is not None:
+                self._session.end_utterance(self._utterance)
+                self._utterance = None
+            return
+        elif isinstance(frame, InterruptionFrame):
+            await self._session.interrupt()
+            self._utterance = None
+        await self.push_frame(frame, direction)
+
+    async def _on_video(self, frame) -> None:
+        rgb = frame.to_ndarray(format="rgb24")
+        await self.push_frame(OutputImageRawFrame(image=rgb.tobytes(), size=(frame.width, frame.height), format="RGB"))
+
+    async def _on_audio(self, frame) -> None:
+        key = (frame.sample_rate, self._out_rate)
+        if self._out_key != key:
+            self._out_resampler = AudioResampler("s16", "mono", self._out_rate)
+            self._out_key = key
+        assert self._out_resampler is not None
+        for out in self._out_resampler.resample(frame):
+            samples = out.to_ndarray().astype(np.int16)
+            if samples.any():
+                await self.push_frame(TTSAudioRawFrame(audio=samples.tobytes(), sample_rate=self._out_rate, num_channels=1))
+
+    def __repr__(self) -> str:
+        return f"AnvaVideoService(session={self._session.session_id})"
+
+
+__all__ = ["AnvaVideoService", "AnvaVideoSettings"]

pipecat_anva-0.1.0/pyproject.toml

@@ -0,0 +1,39 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "pipecat-anva"
+dynamic = ["version"]
+description = "Anva video avatar service for Pipecat: a lip-synced face for your voice agent, on any transport."
+readme = "README.md"
+license = { text = "MIT" }
+requires-python = ">=3.10"
+authors = [{ name = "Anva", email = "anva.ai.2026@gmail.com" }]
+keywords = ["pipecat", "daily", "avatar", "anva", "webrtc", "voice"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Topic :: Multimedia :: Video",
+]
+dependencies = [
+    "pipecat-ai>=1.0",
+    "aiohttp>=3.9",
+    "aiortc>=1.9",
+    "av>=12",
+    "numpy>=1.24",
+    "websockets>=14",
+]
+
+[project.urls]
+Homepage = "https://anva.ai"
+Documentation = "https://anva.ai/docs#pipecat"
+Source = "https://github.com/Anva-avatars/anva-sdk"
+
+[tool.hatch.version]
+path = "pipecat_anva/version.py"
+
+[tool.hatch.build.targets.wheel]
+packages = ["pipecat_anva"]

pipecat_anva-0.1.0/tests/fake_anva.py

@@ -0,0 +1,161 @@
+"""Anva's API and node as the Pipecat service sees them, without a network.
+
+Creates sessions, runs the events socket with instant playback reports, and
+answers the viewer's WebRTC offer with a moving picture and a tone, the way the
+real node does.
+"""
+from __future__ import annotations
+
+import asyncio
+import base64
+import fractions
+import json
+import time
+
+import numpy as np
+from aiohttp import WSMsgType, web
+from aiortc import RTCPeerConnection, RTCSessionDescription
+from aiortc.mediastreams import MediaStreamTrack, VideoStreamTrack
+from av import AudioFrame, VideoFrame
+
+WIDTH, HEIGHT = 160, 120
+TONE_RATE = 48000
+
+
+class Picture(VideoStreamTrack):
+    """A frame whose colour moves, so decoded frames are not all alike."""
+
+    async def recv(self):
+        pts, time_base = await self.next_timestamp()
+        arr = np.zeros((HEIGHT, WIDTH, 3), np.uint8)
+        arr[..., 0] = (pts // 3000) % 256
+        arr[..., 2] = 120
+        frame = VideoFrame.from_ndarray(arr, format="rgb24")
+        frame.pts, frame.time_base = pts, time_base
+        return frame
+
+
+class Tone(MediaStreamTrack):
+    """A 440 Hz tone at 48 kHz stereo, 20 ms a frame, as the node's Opus track decodes."""
+
+    kind = "audio"
+
+    def __init__(self) -> None:
+        super().__init__()
+        self._pos = 0
+        self._start: float | None = None
+
+    async def recv(self):
+        samples = 960
+        if self._start is None:
+            self._start = time.monotonic()
+        wait = self._start + self._pos / TONE_RATE - time.monotonic()
+        if wait > 0:
+            await asyncio.sleep(wait)
+        t = (np.arange(samples) + self._pos) / TONE_RATE
+        mono = (np.sin(2 * np.pi * 440 * t) * 8000).astype(np.int16)
+        frame = AudioFrame.from_ndarray(np.repeat(mono, 2)[None, :], format="s16", layout="stereo")
+        frame.sample_rate = TONE_RATE
+        frame.pts, frame.time_base = self._pos, fractions.Fraction(1, TONE_RATE)
+        self._pos += samples
+        return frame
+
+
+class FakeAnva:
+    """The parts of Anva that pipecat-anva talks to."""
+
+    def __init__(self) -> None:
+        self.app = web.Application()
+        self.app.router.add_post("/api/v2/sessions", self.create)
+        self.app.router.add_delete("/api/v2/sessions/{id}", self.delete)
+        self.app.router.add_get("/api/v2/sessions/{id}/events", self.events)
+        self.app.router.add_post("/offer", self.offer)
+        self.created: list[dict] = []
+        self.deleted: list[str] = []
+        self.offers: list[dict] = []
+        self.commands: list[dict] = []
+        self.ws_auth: str | None = None
+        self.pcs: list[RTCPeerConnection] = []
+        self.port = 0
+        #: When set, session creation answers with this status instead.
+        self.refuse_with: int | None = None
+        self._runner: web.AppRunner | None = None
+
+    async def start(self) -> str:
+        self._runner = web.AppRunner(self.app)
+        await self._runner.setup()
+        site = web.TCPSite(self._runner, "127.0.0.1", 0)
+        await site.start()
+        self.port = site._server.sockets[0].getsockname()[1]  # noqa: SLF001
+        return f"http://127.0.0.1:{self.port}"
+
+    async def stop(self) -> None:
+        for pc in self.pcs:
+            await pc.close()
+        if self._runner is not None:
+            await self._runner.cleanup()
+
+    def commands_of(self, typ: str) -> list[dict]:
+        return [c["payload"] for c in self.commands if c["type"] == typ]
+
+    async def create(self, request: web.Request) -> web.Response:
+        if self.refuse_with:
+            return web.json_response({"error": {"code": "no_capacity", "message": "busy"}}, status=self.refuse_with)
+        body = await request.json()
+        self.created.append({"auth": request.headers.get("Authorization"), "body": body})
+        sid = f"apisess-{len(self.created)}"
+        return web.json_response({
+            "session_id": sid, "session_token": "anva_gr_test", "status": "created",
+            "events_ws_url": f"ws://127.0.0.1:{self.port}/api/v2/sessions/{sid}/events",
+        }, status=201)
+
+    async def delete(self, request: web.Request) -> web.Response:
+        self.deleted.append(request.match_info["id"])
+        return web.json_response({"session_id": request.match_info["id"], "status": "ended"})
+
+    async def events(self, request: web.Request) -> web.WebSocketResponse:
+        self.ws_auth = request.headers.get("Authorization")
+        sid = request.match_info["id"]
+        ws = web.WebSocketResponse()
+        await ws.prepare(request)
+        await ws.send_json({"type": "session.info", "payload": {"session_id": sid, "status": "created"}})
+        await ws.send_json({"type": "session.live", "payload": {"session_id": sid}})
+        accepted = 0
+
+        async def report(turn: str, state: str, played: int, **extra) -> None:
+            await ws.send_json({"type": "speech.state", "payload": {
+                "turn_id": turn, "state": state, "accepted_samples": accepted, "played_samples": played, **extra}})
+
+        async for msg in ws:
+            if msg.type != WSMsgType.TEXT:
+                continue
+            env = json.loads(msg.data)
+            typ, payload = env["type"], env.get("payload") or {}
+            record = dict(payload)
+            if "data" in record:
+                record["samples"] = len(base64.b64decode(record.pop("data"))) // 2
+            self.commands.append({"type": typ, "payload": record})
+            turn = payload.get("turn_id", "")
+            # Playback is instant here: what is accepted is played.
+            if typ == "speech.start":
+                accepted = 0
+                await report(turn, "started", 0)
+            elif typ == "speech.append":
+                accepted += record["samples"]
+                await report(turn, "accepted", accepted, next_seq=payload["seq"] + 1)
+            elif typ == "speech.done":
+                await report(turn, "finished", payload["total_samples"])
+            elif typ == "speech.cancel":
+                await report(turn, "cancelled", accepted)
+        return ws
+
+    async def offer(self, request: web.Request) -> web.Response:
+        body = await request.json()
+        self.offers.append(body)
+        pc = RTCPeerConnection()
+        self.pcs.append(pc)
+        pc.addTrack(Picture())
+        pc.addTrack(Tone())
+        await pc.setRemoteDescription(RTCSessionDescription(sdp=body["sdp"], type=body["type"]))
+        await pc.setLocalDescription(await pc.createAnswer())
+        return web.json_response({"type": pc.localDescription.type, "sdp": pc.localDescription.sdp})

pipecat_anva-0.1.0/tests/test_service.py

@@ -0,0 +1,75 @@
+"""The service inside a Pipecat pipeline, against the fake Anva."""
+from __future__ import annotations
+
+import numpy as np
+import pytest
+from pipecat.frames.frames import (
+    InterruptionFrame,
+    OutputImageRawFrame,
+    TTSAudioRawFrame,
+    TTSStartedFrame,
+    TTSStoppedFrame,
+)
+from pipecat.tests.utils import SleepFrame, run_test
+
+from pipecat_anva import AnvaVideoService
+from pipecat_anva.session import CORE_RATE
+
+from .fake_anva import FakeAnva
+from .test_session import tone
+
+
+@pytest.mark.asyncio
+async def test_tts_audio_becomes_the_avatars_picture_and_voice():
+    anva = FakeAnva()
+    base = await anva.start()
+    service = AnvaVideoService(api_key="k", avatar_id="av_1", api_url=base)
+    sent = [TTSAudioRawFrame(audio=tone(0.4, 16000), sample_rate=16000, num_channels=1) for _ in range(3)]
+    try:
+        down, _ = await run_test(
+            service,
+            frames_to_send=[TTSStartedFrame(), *sent, TTSStoppedFrame(), SleepFrame(2.0)],
+        )
+    finally:
+        await anva.stop()
+
+    assert anva.created[0]["body"]["service_mode"] == "avatar_only"
+    assert anva.commands_of("speech.start")[0]["turn_id"] == "pipecat-1"
+    total = sum(a["samples"] for a in anva.commands_of("speech.append"))
+    assert abs(total - int(1.2 * CORE_RATE)) <= 64
+    assert anva.commands_of("speech.done") == [{"turn_id": "pipecat-1", "total_samples": total}]
+
+    assert any(isinstance(f, TTSStartedFrame) for f in down), "the start of speech is passed on"
+    assert not any(isinstance(f, TTSStoppedFrame) for f in down), "the pipeline's own audio and its end stop here"
+    images = [f for f in down if isinstance(f, OutputImageRawFrame)]
+    assert len(images) >= 5 and images[0].size == (160, 120) and images[0].format == "RGB"
+    assert len(images[0].image) == 160 * 120 * 3
+    audio = [f for f in down if isinstance(f, TTSAudioRawFrame)]
+    assert audio, "the avatar's voice comes back as audio"
+    assert all(f.sample_rate == 16000 and f.num_channels == 1 for f in audio), "at the pipeline's TTS rate"
+    assert not any(f.audio in {s.audio for s in sent} for f in audio), "and it is the avatar's, not the copy"
+    samples = np.frombuffer(b"".join(f.audio for f in audio), dtype=np.int16)
+    assert samples.size >= 16000 and np.abs(samples).max() > 3000, "a second or more of a real tone"
+    assert anva.deleted == ["apisess-1"], "the end of the pipeline ends the Anva session"
+
+
+@pytest.mark.asyncio
+async def test_an_interruption_stops_the_avatar():
+    anva = FakeAnva()
+    base = await anva.start()
+    service = AnvaVideoService(api_key="k", avatar_id="av_1", api_url=base)
+    try:
+        await run_test(
+            service,
+            frames_to_send=[
+                TTSStartedFrame(),
+                TTSAudioRawFrame(audio=tone(0.5, CORE_RATE), sample_rate=CORE_RATE, num_channels=1),
+                SleepFrame(0.5),
+                InterruptionFrame(),
+                SleepFrame(0.5),
+            ],
+        )
+    finally:
+        await anva.stop()
+    assert [c["turn_id"] for c in anva.commands_of("speech.cancel")] == ["pipecat-1"]
+    assert not anva.commands_of("speech.done")

pipecat_anva-0.1.0/tests/test_session.py

@@ -0,0 +1,135 @@
+"""The session on its own: what it sends Anva and what it makes of the answers."""
+from __future__ import annotations
+
+import asyncio
+
+import numpy as np
+import pytest
+
+from pipecat_anva import AnvaError, AnvaSession
+from pipecat_anva.session import CHUNK_SAMPLES, CORE_RATE, Utterance
+
+from .fake_anva import FakeAnva
+
+
+def tone(seconds: float, rate: int, channels: int = 1) -> bytes:
+    t = np.arange(int(seconds * rate)) / rate
+    mono = (np.sin(2 * np.pi * 300 * t) * 6000).astype(np.int16)
+    return (np.repeat(mono, channels) if channels > 1 else mono).tobytes()
+
+
+def test_one_avatar_reference_and_a_key_are_required():
+    with pytest.raises(AnvaError, match="exactly one"):
+        AnvaSession(api_key="k")
+    with pytest.raises(AnvaError, match="exactly one"):
+        AnvaSession(api_key="k", avatar_id="a", preset_id="p")
+    with pytest.raises(AnvaError, match="api_key"):
+        AnvaSession(api_key="", avatar_id="a")
+    AnvaSession(api_key="k", preset_id="p", api_url="https://anva.ai/")
+
+
+def test_utterances_are_taken_in_chunks_and_the_tail_only_when_closed():
+    u = Utterance(turn_id="t")
+    u.push(b"\0\0" * (CHUNK_SAMPLES + 10))
+    assert len(u.take()) == CHUNK_SAMPLES * 2
+    assert u.take() == b"", "a partial chunk waits for more audio"
+    u.close()
+    assert len(u.take()) == 20
+    assert u.take() == b""
+    u.cancel()
+    u.push(b"\0\0" * 5)
+    assert u.take() == b"" and u.cancelled
+
+
+def test_pipeline_audio_is_converted_to_what_anva_takes():
+    s = AnvaSession(api_key="k", avatar_id="a")
+    u = s.begin_utterance()
+    s.push_audio(u, tone(0.5, 48000, 2), 48000, 2)
+    s.push_audio(u, tone(0.25, 48000, 2), 48000, 2)
+    s.end_utterance(u)
+    assert abs(u.pushed - int(0.75 * CORE_RATE)) <= 64
+    # Native audio is taken as it is, sample for sample.
+    v = s.begin_utterance()
+    s.push_audio(v, tone(0.1, CORE_RATE), CORE_RATE, 1)
+    assert v.pushed == int(0.1 * CORE_RATE)
+
+
+@pytest.mark.asyncio
+async def test_speech_reaches_anva_as_turns_with_receipts():
+    anva = FakeAnva()
+    base = await anva.start()
+    frames: list = []
+
+    async def on_video(frame):
+        frames.append(frame)
+
+    session = AnvaSession(api_key="key-1", avatar_id="av_1", api_url=base, metadata={"k": "v"},
+                          on_video=on_video)
+    try:
+        await session.start()
+        assert anva.created[0]["auth"] == "Bearer key-1"
+        assert anva.created[0]["body"] == {"service_mode": "avatar_only", "avatar_id": "av_1", "metadata": {"k": "v"}}
+        assert anva.ws_auth == "Bearer key-1"
+        assert anva.offers[0]["session_id"] == "apisess-1" and anva.offers[0]["session_url_token"] == "anva_gr_test"
+        assert anva.offers[0]["type"] == "offer" and anva.offers[0]["client_generation"] == 1
+
+        u = session.begin_utterance()
+        session.push_audio(u, tone(1.2, 16000), 16000, 1)
+        session.end_utterance(u)
+        for _ in range(100):
+            if anva.commands_of("speech.done"):
+                break
+            await asyncio.sleep(0.05)
+        starts = anva.commands_of("speech.start")
+        assert starts == [{"turn_id": "pipecat-1", "codec": "pcm_s16le", "sample_rate": CORE_RATE, "channels": 1}]
+        appends = anva.commands_of("speech.append")
+        assert [a["seq"] for a in appends] == list(range(len(appends)))
+        assert all(a["samples"] <= CHUNK_SAMPLES for a in appends)
+        total = sum(a["samples"] for a in appends)
+        assert abs(total - int(1.2 * CORE_RATE)) <= 64
+        assert anva.commands_of("speech.done") == [{"turn_id": "pipecat-1", "total_samples": total}]
+
+        # An interruption cancels what is playing and drops what waits.
+        second = session.begin_utterance()
+        session.push_audio(second, tone(3, CORE_RATE), CORE_RATE, 1)
+        third = session.begin_utterance()
+        session.push_audio(third, tone(1, CORE_RATE), CORE_RATE, 1)
+        for _ in range(100):
+            if any(a["turn_id"] == "pipecat-2" for a in anva.commands_of("speech.append")):
+                break
+            await asyncio.sleep(0.05)
+        await session.interrupt()
+        for _ in range(100):
+            if anva.commands_of("speech.cancel"):
+                break
+            await asyncio.sleep(0.05)
+        assert [c["turn_id"] for c in anva.commands_of("speech.cancel")] == ["pipecat-2"]
+        await asyncio.sleep(0.3)
+        assert not any(c["payload"].get("turn_id") == "pipecat-3" for c in anva.commands), \
+            "a queued utterance dropped by the interruption must never start"
+
+        # The viewer receives the avatar's picture.
+        for _ in range(100):
+            if len(frames) >= 3:
+                break
+            await asyncio.sleep(0.05)
+        assert len(frames) >= 3 and frames[0].width == 160 and frames[0].height == 120
+    finally:
+        await session.close()
+        await anva.stop()
+    assert anva.deleted == ["apisess-1"]
+    assert session.ended.is_set()
+
+
+@pytest.mark.asyncio
+async def test_a_refusal_is_an_error_not_a_hang():
+    anva = FakeAnva()
+    anva.refuse_with = 503
+    base = await anva.start()
+    session = AnvaSession(api_key="k", avatar_id="a", api_url=base)
+    try:
+        with pytest.raises(AnvaError, match="503"):
+            await session.start()
+    finally:
+        await session.close()
+        await anva.stop()