pipecat-siphon 0.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- pipecat_siphon-0.2.0/.gitignore +17 -0
- pipecat_siphon-0.2.0/CHANGELOG.md +87 -0
- pipecat_siphon-0.2.0/LICENSE +21 -0
- pipecat_siphon-0.2.0/PKG-INFO +426 -0
- pipecat_siphon-0.2.0/README.md +395 -0
- pipecat_siphon-0.2.0/examples/agent_bot.py +270 -0
- pipecat_siphon-0.2.0/examples/echo_bot.py +139 -0
- pipecat_siphon-0.2.0/integration/README.md +260 -0
- pipecat_siphon-0.2.0/integration/bot/Dockerfile +21 -0
- pipecat_siphon-0.2.0/integration/bot/harness_bot.py +388 -0
- pipecat_siphon-0.2.0/integration/docker-compose.yaml +203 -0
- pipecat_siphon-0.2.0/integration/proxy/control_tap.py +138 -0
- pipecat_siphon-0.2.0/integration/proxy/harness_script.py +86 -0
- pipecat_siphon-0.2.0/integration/proxy/siphon.yaml +107 -0
- pipecat_siphon-0.2.0/integration/run.sh +145 -0
- pipecat_siphon-0.2.0/integration/scenarios/roundtrip_uac.xml +114 -0
- pipecat_siphon-0.2.0/integration/scenarios/turntaking_uac.xml +118 -0
- pipecat_siphon-0.2.0/integration/scenarios/wideband_uac.xml +115 -0
- pipecat_siphon-0.2.0/integration/tools/__init__.py +1 -0
- pipecat_siphon-0.2.0/integration/tools/analyse.py +601 -0
- pipecat_siphon-0.2.0/integration/tools/capture.py +137 -0
- pipecat_siphon-0.2.0/integration/tools/make_fixtures.py +174 -0
- pipecat_siphon-0.2.0/integration/tools/rtp_pcap.py +115 -0
- pipecat_siphon-0.2.0/integration/tools/signals.py +309 -0
- pipecat_siphon-0.2.0/integration/tools/wait_for.py +167 -0
- pipecat_siphon-0.2.0/integration/uac/Dockerfile +32 -0
- pipecat_siphon-0.2.0/integration/uac/prepare.sh +30 -0
- pipecat_siphon-0.2.0/integration/uac/run_scenario.sh +99 -0
- pipecat_siphon-0.2.0/pyproject.toml +109 -0
- pipecat_siphon-0.2.0/src/pipecat_siphon/__init__.py +69 -0
- pipecat_siphon-0.2.0/src/pipecat_siphon/protocol.py +656 -0
- pipecat_siphon-0.2.0/src/pipecat_siphon/py.typed +0 -0
- pipecat_siphon-0.2.0/src/pipecat_siphon/serializer.py +588 -0
- pipecat_siphon-0.2.0/tests/test_end_to_end.py +250 -0
- pipecat_siphon-0.2.0/tests/test_protocol.py +221 -0
- pipecat_siphon-0.2.0/tests/test_serializer.py +638 -0
- pipecat_siphon-0.2.0/tests/wire_fixtures.py +112 -0
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
__pycache__/
|
|
2
|
+
*.py[cod]
|
|
3
|
+
.venv/
|
|
4
|
+
.mypy_cache/
|
|
5
|
+
.ruff_cache/
|
|
6
|
+
.pytest_cache/
|
|
7
|
+
dist/
|
|
8
|
+
build/
|
|
9
|
+
*.egg-info/
|
|
10
|
+
|
|
11
|
+
# Local-only conventions and tooling scratch — never committed.
|
|
12
|
+
CLAUDE.md
|
|
13
|
+
.claude/
|
|
14
|
+
|
|
15
|
+
# Scratch git worktrees — a stray `git add -A` would commit a second copy of the tree.
|
|
16
|
+
/.worktrees/
|
|
17
|
+
/worktrees/
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project are documented here. The format follows
|
|
4
|
+
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project uses
|
|
5
|
+
[semantic versioning](https://semver.org/spec/v2.0.0.html).
|
|
6
|
+
|
|
7
|
+
## [Unreleased]
|
|
8
|
+
|
|
9
|
+
## [0.2.0] - 2026-09-04
|
|
10
|
+
|
|
11
|
+
The first release published to PyPI. `0.1.0` was the initial code drop and never left the
|
|
12
|
+
repository, so `pip install pipecat-siphon` starts here.
|
|
13
|
+
|
|
14
|
+
### Changed
|
|
15
|
+
|
|
16
|
+
- Moved to `pipecat-ai` 1.8, and the floor with it (`>=1.8.0,<2`). 1.8.0 moved the pipeline's
|
|
17
|
+
sample rates out of `StartFrame` and into the `FrameProcessorSetup` a transport hands every
|
|
18
|
+
processor, deprecating the old reads; `SiphonFrameSerializer.setup()` now takes that object,
|
|
19
|
+
which is what the transport was already passing. The non-fatal `error` mapping also stops
|
|
20
|
+
passing `ErrorFrame(fatal=False)`, another 1.8 deprecation and the default anyway -- a fatal
|
|
21
|
+
engine error still becomes a `CancelWorkerFrame`, which is what pipecat now recommends instead
|
|
22
|
+
of a fatal frame. The wire, the frame mapping and every parameter are unchanged.
|
|
23
|
+
|
|
24
|
+
- Documentation aligned to the siphon-rtp 0.3.x/0.4.x line. The bridge control envelope is
|
|
25
|
+
untouched across all of it -- `crates/siphon-rtp-media/src/bridge/protocol.rs` has not changed
|
|
26
|
+
since before 0.2.0 -- so the serializer, the protocol module and the engine-emitted wire fixtures
|
|
27
|
+
are unaffected and there is no code change here. What did change is the surface a *controller*
|
|
28
|
+
drives, and the README now says so: the selectable wire rate (`ws_sample_rate` /
|
|
29
|
+
`ws_tee_sample_rate`) is released rather than unreleased, with the advice to set it to the
|
|
30
|
+
pipeline's own rate so the one conversion in the path is the engine's; the turn-taking flags gain
|
|
31
|
+
`ws_vad_engine` (`energy` / `neural`) and `ws_vad_min_speech_ms`; a new section covers which
|
|
32
|
+
callers a takeover can terminate now that `answer_local` handles SDES-SRTP, DTLS-SRTP and full
|
|
33
|
+
ICE, with the stable refusal tokens for the shapes it cannot; and another covers the 0.4.0
|
|
34
|
+
takeover-bridge lifecycle, where `attach_ws_bridge` / `detach_ws_bridge` can put a bot on a call
|
|
35
|
+
that is already up, move it to a different bot, or hand the two parties back to each other, and
|
|
36
|
+
`ws_bridge_ended` finally tells a controller when the bot's socket died.
|
|
37
|
+
|
|
38
|
+
### Added
|
|
39
|
+
|
|
40
|
+
- `examples/agent_bot.py`, a conversational demo bot: the caller's speech goes to a streaming
|
|
41
|
+
recognizer, the transcript to Claude, and the reply to a streaming voice, with the turn taking
|
|
42
|
+
left to the engine (`ws_vad` / `ws_barge_in`) rather than to a VAD analyzer in the pipeline. It
|
|
43
|
+
runs the pipeline at the negotiated wire rate, so neither side of the socket resamples.
|
|
44
|
+
|
|
45
|
+
- The harness runs on a fresh clone. siphon-sip and siphon-rtp default to their published
|
|
46
|
+
release images rather than to builds of checkouts beside this repository, so `./run.sh` needs
|
|
47
|
+
Docker and nothing else; only the bot and the UAC are built, from this repository. Naming a
|
|
48
|
+
checkout (`SIPHON_RTP_PATH` / `SIPHON_SIP_PATH`) still builds that component from source, which
|
|
49
|
+
is what you want when the engine change is yours, and `SIPHON_RTP_IMAGE` / `SIPHON_SIP_IMAGE`
|
|
50
|
+
pin a different release without one.
|
|
51
|
+
|
|
52
|
+
- A four-way integration harness under `integration/`: SIPp to siphon-sip to siphon-rtp to a
|
|
53
|
+
pipecat bot using this serializer, in compose, with no mocks in the path. Three scenarios, all
|
|
54
|
+
asserting on audio rather than on connectivity -- a Goertzel check on the RTP that returns to
|
|
55
|
+
the caller for the round trip, VAD edge timing plus barge-in flush latency for turn taking, and
|
|
56
|
+
a wideband run that negotiates a 16 kHz wire over an 8 kHz G.711 leg with `ws_sample_rate` and
|
|
57
|
+
checks the audio comes back at the pitch it went out at, which is the case this serializer's
|
|
58
|
+
rate reconciliation exists for. Not wired into CI; see `integration/README.md` for why.
|
|
59
|
+
|
|
60
|
+
## [0.1.0] - 2026-08-20
|
|
61
|
+
|
|
62
|
+
First release.
|
|
63
|
+
|
|
64
|
+
### Added
|
|
65
|
+
|
|
66
|
+
- `SiphonFrameSerializer`, a `pipecat.serializers.base_serializer.FrameSerializer` for the
|
|
67
|
+
siphon-rtp media WebSocket protocol: binary L16/PCMU/PCMA audio frames plus the
|
|
68
|
+
`{"type", "data"}` JSON control envelope, in both directions.
|
|
69
|
+
- `pipecat_siphon.protocol`, a dependency-free transcription of the engine-side message
|
|
70
|
+
definitions, byte-exact against the engine's own `serde` output (field order, camelCase keys,
|
|
71
|
+
omit-when-absent optionals).
|
|
72
|
+
- Negotiated-wire-rate handling: the `start` envelope's `sampleRate` is authoritative and is
|
|
73
|
+
reconciled against the pipeline's rates with pipecat's own streaming resampler, in both
|
|
74
|
+
directions, including a `media_renegotiate` mid-stream.
|
|
75
|
+
- Stereo tee support: channel 0 caller, channel 1 callee, selectable per `stereo_input`
|
|
76
|
+
(`caller`, `callee`, `mix`, `interleaved`).
|
|
77
|
+
- Engine-side VAD turn boundaries (`speech_started` / `speech_stopped`) mapped to either the VAD
|
|
78
|
+
frames pipecat's default turn-start strategy consumes or explicit user-turn frames, plus an
|
|
79
|
+
optional interruption.
|
|
80
|
+
- DTMF, playout marks, graceful `stop`, and fatal/non-fatal `error` handling.
|
|
81
|
+
- A runnable parrot-bot example under `examples/`.
|
|
82
|
+
|
|
83
|
+
### Notes
|
|
84
|
+
|
|
85
|
+
- Verified against `pipecat-ai` 1.7.0 on Python 3.11 and 3.13.
|
|
86
|
+
- `FrameSerializer` in pipecat 1.7 has no `type` property and no `FrameSerializerType`; the
|
|
87
|
+
transport decides binary versus text from the return type of `serialize`.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 SIPhon Contributors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,426 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: pipecat-siphon
|
|
3
|
+
Version: 0.2.0
|
|
4
|
+
Summary: Pipecat frame serializer for the siphon-rtp media WebSocket protocol
|
|
5
|
+
Project-URL: Homepage, https://github.com/siphon-project/pipecat-siphon
|
|
6
|
+
Project-URL: Repository, https://github.com/siphon-project/pipecat-siphon
|
|
7
|
+
Project-URL: Changelog, https://github.com/siphon-project/pipecat-siphon/blob/main/CHANGELOG.md
|
|
8
|
+
Project-URL: Issues, https://github.com/siphon-project/pipecat-siphon/issues
|
|
9
|
+
Author: SIPhon contributors
|
|
10
|
+
License-Expression: MIT
|
|
11
|
+
License-File: LICENSE
|
|
12
|
+
Keywords: pipecat,rtp,siphon-rtp,telephony,voice-ai,websocket
|
|
13
|
+
Classifier: Development Status :: 4 - Beta
|
|
14
|
+
Classifier: Intended Audience :: Developers
|
|
15
|
+
Classifier: Programming Language :: Python :: 3
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
19
|
+
Classifier: Topic :: Communications :: Telephony
|
|
20
|
+
Classifier: Topic :: Multimedia :: Sound/Audio
|
|
21
|
+
Classifier: Typing :: Typed
|
|
22
|
+
Requires-Python: >=3.11
|
|
23
|
+
Requires-Dist: pipecat-ai<2,>=1.8.0
|
|
24
|
+
Provides-Extra: dev
|
|
25
|
+
Requires-Dist: mypy>=1.14; extra == 'dev'
|
|
26
|
+
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
|
|
27
|
+
Requires-Dist: pytest>=8; extra == 'dev'
|
|
28
|
+
Requires-Dist: ruff>=0.8; extra == 'dev'
|
|
29
|
+
Requires-Dist: websockets>=13.1; extra == 'dev'
|
|
30
|
+
Description-Content-Type: text/markdown
|
|
31
|
+
|
|
32
|
+
# pipecat-siphon
|
|
33
|
+
|
|
34
|
+
A [pipecat](https://github.com/pipecat-ai/pipecat) frame serializer for the **siphon-rtp media
|
|
35
|
+
WebSocket protocol**.
|
|
36
|
+
|
|
37
|
+
siphon-rtp is a media engine that can bridge a call leg's audio to an external WebSocket server:
|
|
38
|
+
it decodes RTP to linear PCM, streams it up, and encodes PCM coming back down into RTP toward the
|
|
39
|
+
caller. Your bot never touches RTP, jitter buffers, or codecs. This package is the small adapter
|
|
40
|
+
that lets a pipecat pipeline read and write that wire.
|
|
41
|
+
|
|
42
|
+
siphon-rtp keeps its own native wire on purpose. Every telephony vendor has a different one, and
|
|
43
|
+
serializers exist precisely so an engine does not have to pretend to be somebody else. There is no
|
|
44
|
+
Twilio/Telnyx emulation mode here, and there will not be one. The adapter lives on this side, and
|
|
45
|
+
it is small: siphon-rtp's binary-L16-plus-JSON-envelope wire is *simpler* than the base64-in-JSON
|
|
46
|
+
formats pipecat already ships support for.
|
|
47
|
+
|
|
48
|
+
Pipecat's contributing guide directs new service and transport integrations to
|
|
49
|
+
community-maintained external packages rather than pull requests into the core repository, so this
|
|
50
|
+
is the sanctioned shape for an integration like this one.
|
|
51
|
+
|
|
52
|
+
## Install
|
|
53
|
+
|
|
54
|
+
```bash
|
|
55
|
+
pip install pipecat-siphon
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
It depends on `pipecat-ai>=1.8.0,<2` and nothing else. Python 3.11 or newer, matching pipecat.
|
|
59
|
+
|
|
60
|
+
## The shape of the integration
|
|
61
|
+
|
|
62
|
+
**The engine dials out.** siphon-rtp is the WebSocket *client*; your bot is the *server*. So the
|
|
63
|
+
example under `examples/` stands up a server and waits, rather than connecting anywhere.
|
|
64
|
+
|
|
65
|
+
```python
|
|
66
|
+
from pipecat.pipeline.pipeline import Pipeline
|
|
67
|
+
from pipecat.transports.websocket.server import (
|
|
68
|
+
SingleClientWebsocketServerParams,
|
|
69
|
+
SingleClientWebsocketServerTransport,
|
|
70
|
+
)
|
|
71
|
+
from pipecat_siphon import SiphonFrameSerializer
|
|
72
|
+
|
|
73
|
+
transport = SingleClientWebsocketServerTransport(
|
|
74
|
+
host="127.0.0.1",
|
|
75
|
+
port=9001,
|
|
76
|
+
params=SingleClientWebsocketServerParams(
|
|
77
|
+
audio_in_enabled=True,
|
|
78
|
+
audio_out_enabled=True,
|
|
79
|
+
serializer=SiphonFrameSerializer(),
|
|
80
|
+
),
|
|
81
|
+
)
|
|
82
|
+
|
|
83
|
+
pipeline = Pipeline([transport.input(), your_processors, transport.output()])
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
Then point the engine at it, in the `profile` of a native-JSON `offer`:
|
|
87
|
+
|
|
88
|
+
```json
|
|
89
|
+
{
|
|
90
|
+
"id": 1,
|
|
91
|
+
"command": "offer",
|
|
92
|
+
"call_id": "call-7@198.51.100.2",
|
|
93
|
+
"from_tag": "caller",
|
|
94
|
+
"sdp": "v=0\r\n...",
|
|
95
|
+
"profile": { "ws_uri": "ws://127.0.0.1:9001/stream", "ws_vad": true, "ws_barge_in": true }
|
|
96
|
+
}
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
`ws_uri` is *takeover*: the WebSocket server is the far side of the call. `ws_tee` (or the
|
|
100
|
+
`attach_ws_tee` verb) is a *tee*: the call keeps relaying between two real parties and a copy of
|
|
101
|
+
the decoded audio is streamed out, send-only. Both use the same wire, and this serializer handles
|
|
102
|
+
both; on a tee it refuses to write audio back, because the engine would not inject it anyway.
|
|
103
|
+
|
|
104
|
+
## Which callers can reach a bot
|
|
105
|
+
|
|
106
|
+
A takeover makes the engine the caller's *only* peer, so the engine has to terminate whatever the
|
|
107
|
+
caller negotiated and say so in the answer it writes back. `answer_local` writes that answer itself;
|
|
108
|
+
`offer`/`answer` rewrites it from the B leg's SDP, and a takeover call has no B leg. So the verb
|
|
109
|
+
decides what the caller may be (siphon-rtp 0.3.0):
|
|
110
|
+
|
|
111
|
+
| Caller's `m=audio` | `offer` + `answer` | `answer_local` |
|
|
112
|
+
|---|---|---|
|
|
113
|
+
| `RTP/AVP` (plaintext) | supported | supported |
|
|
114
|
+
| `RTP/AVP` + ICE | refused (`ws-takeover-ice-offerer`) | supported with `--ice-full`, else refused (`ws-takeover-ice-unsupported`) |
|
|
115
|
+
| `RTP/SAVP` + `a=crypto` (SDES-SRTP) | refused (`ws-takeover-secure-offerer`) | supported |
|
|
116
|
+
| `UDP/TLS/RTP/SAVPF` (DTLS-SRTP) | refused (`ws-takeover-secure-offerer`) | supported |
|
|
117
|
+
|
|
118
|
+
On a supported secure takeover the engine mints its **own** keying rather than echoing the caller's,
|
|
119
|
+
decrypts the caller's ingress before it becomes the audio your bot hears, and encrypts the downlink
|
|
120
|
+
on the way back out. It is fail-closed: nothing leaves in the clear toward a peer that negotiated
|
|
121
|
+
encryption. Every refusal comes back at offer time with a stable token at the front of the reason,
|
|
122
|
+
so a controller can branch on it without parsing prose. Before 0.3.0 those same offers returned `ok`
|
|
123
|
+
and produced a call that answered and bridged nowhere, which is worth knowing if you are moving a
|
|
124
|
+
bot off an older engine.
|
|
125
|
+
|
|
126
|
+
None of this reaches the WebSocket. The bot sees the same `start` envelope and the same L16 frames
|
|
127
|
+
whether the caller is a plaintext SIP phone or a WebRTC endpoint.
|
|
128
|
+
|
|
129
|
+
## Attaching a bot to a call that is already up
|
|
130
|
+
|
|
131
|
+
`ws_uri` puts a bot on the call at negotiation time and keeps it there until the call ends. From
|
|
132
|
+
siphon-rtp **0.4.0** a controller can also do it mid-call, with two verbs:
|
|
133
|
+
|
|
134
|
+
* **`attach_ws_bridge`** (`call_id`, `from_tag`, `ws_uri`) on a call that has no bridge takes a live
|
|
135
|
+
two-party relay over: leg A's audio goes to your bot and the A↔B path is unwired, so the other
|
|
136
|
+
party hears nothing until it is detached. On a call that already has one it **re-points** it at a
|
|
137
|
+
different server without renegotiating — codec, wire rate, VAD, echo cancellation, source gate,
|
|
138
|
+
SRTP keying and ICE selection all carry across, so there is no re-INVITE and nothing reverts to a
|
|
139
|
+
default.
|
|
140
|
+
* **`detach_ws_bridge`** puts the two parties back together, reinstalling the exact forward rules the
|
|
141
|
+
takeover displaced.
|
|
142
|
+
|
|
143
|
+
For a pipecat deployment that is the difference between "the bot answers the call" and "the bot joins
|
|
144
|
+
a call in progress" — an agent taking over from a human, a supervisor handing a caller to a bot, or a
|
|
145
|
+
bot swapped out mid-conversation. From this side of the socket nothing changes: your server gets the
|
|
146
|
+
same `start` envelope and the same audio, and this serializer does not care which verb put it there.
|
|
147
|
+
|
|
148
|
+
The same release added **`ws_bridge_started` / `ws_bridge_ended`** events. The end reason
|
|
149
|
+
(`detached`, `server_closed`, `server_stopped`, `call_ended`, `transport_error`) is how a controller
|
|
150
|
+
finds out the bot's socket died — worth wiring up, because a takeover bridge is the caller's *only*
|
|
151
|
+
far side, so a bot that goes away is a live call with nobody on it.
|
|
152
|
+
|
|
153
|
+
## The wire
|
|
154
|
+
|
|
155
|
+
Two frame kinds share one socket:
|
|
156
|
+
|
|
157
|
+
* **binary** frames carry raw audio in the negotiated format, one frame per ptime, no base64;
|
|
158
|
+
* **text** frames carry a `{"type": "<snake_case tag>", "data": {...}}` control envelope whose
|
|
159
|
+
`data` object uses camelCase field names.
|
|
160
|
+
|
|
161
|
+
The first text frame is always `start`:
|
|
162
|
+
|
|
163
|
+
```json
|
|
164
|
+
{
|
|
165
|
+
"type": "start",
|
|
166
|
+
"data": {
|
|
167
|
+
"streamId": "ws-call-7@198.51.100.2",
|
|
168
|
+
"callId": "call-7@198.51.100.2",
|
|
169
|
+
"direction": "duplex",
|
|
170
|
+
"media": {
|
|
171
|
+
"encoding": "L16", "sampleRate": 8000, "channels": 1,
|
|
172
|
+
"bitDepth": 16, "endianness": "little", "ptime": 20
|
|
173
|
+
}
|
|
174
|
+
}
|
|
175
|
+
}
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
## Sample rates: the wire rate is negotiated, and authoritative
|
|
179
|
+
|
|
180
|
+
`media.sampleRate` in the `start` envelope is **the wire rate**, in both directions, for the life of
|
|
181
|
+
the stream, and it is authoritative. Never assume 8000. Frame against the number `start` gives you,
|
|
182
|
+
and expect it to differ from what your pipeline runs at.
|
|
183
|
+
|
|
184
|
+
From siphon-rtp **0.3.0** that rate is selectable independently of the call's codec, so an 8 kHz
|
|
185
|
+
G.711 caller can stream 16 kHz L16 to a bot that wants wideband input:
|
|
186
|
+
|
|
187
|
+
| Knob | Where it goes |
|
|
188
|
+
|---|---|
|
|
189
|
+
| `ws_sample_rate` | the takeover profile, alongside `ws_uri`. Applied in both directions: the engine resamples the leg's uplink into it, and resamples the bot's downlink back into the codec's rate before re-encoding. |
|
|
190
|
+
| `ws_tee_sample_rate` | the tee profile, alongside `ws_tee`. Every tapped leg is resampled into it, so a stereo tee across two different codecs still produces one coherent stream. |
|
|
191
|
+
| `sample_rate` | the `attach_ws_tee` verb, for a tee attached after the call is up. |
|
|
192
|
+
|
|
193
|
+
Valid rates are multiples of 1000 from 8000 to 48000. Anything else fails the offer, answer or
|
|
194
|
+
attach with a typed error before the call is dialled — the engine never silently clamps to a rate
|
|
195
|
+
you did not ask for. Leave the knob out and the wire follows the leg's codec rate (8000 for G.711,
|
|
196
|
+
16000 for G.722/AMR-WB) with no conversion built at all, exactly as before 0.3.0.
|
|
197
|
+
|
|
198
|
+
**Set it to whatever your pipeline runs at.** The conversion then happens in the engine instead of
|
|
199
|
+
here, which is the better place for it: siphon-rtp resamples frame by frame at the RTP boundary with
|
|
200
|
+
nothing held back, while pipecat's SOXR stream resampler at VHQ swallows the first few chunks and
|
|
201
|
+
then delivers in bursts. Same audio, steadier timing, and one less stage between the caller and the
|
|
202
|
+
model.
|
|
203
|
+
|
|
204
|
+
On 0.2.1 and earlier the knobs do not exist. An older engine *ignores* the field rather than
|
|
205
|
+
rejecting it, so you get the codec rate back with nothing to point at — check the engine version,
|
|
206
|
+
or just read the rate out of `start`, which is right on every build.
|
|
207
|
+
|
|
208
|
+
This serializer handles the mismatch for you. `setup(StartFrame)` records the pipeline's
|
|
209
|
+
`audio_in_sample_rate`; the wire rate arrives later, in `start` (the engine cannot dial you before
|
|
210
|
+
you are listening, so the ordering is always pipeline-first). Every frame is then resampled
|
|
211
|
+
between the two with pipecat's own `create_stream_resampler`, in both directions, and a
|
|
212
|
+
`media_renegotiate` mid-stream moves the wire rate without restarting anything. When the rates
|
|
213
|
+
already match, nothing is resampled and the bytes pass through untouched.
|
|
214
|
+
|
|
215
|
+
If you want to fix the wire rate before `start` arrives, set `wire_sample_rate` in the params;
|
|
216
|
+
`start` still wins the moment it lands.
|
|
217
|
+
|
|
218
|
+
## Stereo and track separation
|
|
219
|
+
|
|
220
|
+
A tee with `channels: 2` interleaves the two legs: **channel 0 is the caller, channel 1 is the
|
|
221
|
+
callee**. The `stereo_input` param decides what the pipeline sees:
|
|
222
|
+
|
|
223
|
+
| `stereo_input` | Result |
|
|
224
|
+
|---|---|
|
|
225
|
+
| `"caller"` (default) | Mono, channel 0 only. Keeps the far party out of your ASR. |
|
|
226
|
+
| `"callee"` | Mono, channel 1 only. |
|
|
227
|
+
| `"mix"` | Mono, the two legs summed at half amplitude each. |
|
|
228
|
+
| `"interleaved"` | A 2-channel `InputAudioRawFrame`, each channel resampled on its own. |
|
|
229
|
+
|
|
230
|
+
Outbound, if the wire is stereo the bot's mono audio is written to both channels: a bot has one
|
|
231
|
+
voice, and silencing a channel would be a stranger default than duplicating it.
|
|
232
|
+
|
|
233
|
+
## Message matrix
|
|
234
|
+
|
|
235
|
+
### Engine → pipecat (`deserialize`)
|
|
236
|
+
|
|
237
|
+
| Wire message | Pipecat frame | Notes |
|
|
238
|
+
|---|---|---|
|
|
239
|
+
| binary audio | `InputAudioRawFrame` | Resampled to the pipeline rate; L16/PCMU/PCMA; endianness honoured; stereo split per `stereo_input`. |
|
|
240
|
+
| `start` | *(none)* | Captures `streamId`, `callId`, `direction`, `tracks` and the media format. Readable afterwards on `.stream_id` / `.media_format` / `.wire_sample_rate` / `.direction` / `.tracks`. |
|
|
241
|
+
| `media_renegotiate` | *(none)* | Replaces the wire format mid-stream. |
|
|
242
|
+
| `speech_started` | `VADUserStartedSpeakingFrame` | Default (`speech_frames="vad"`). Pipecat's default `VADUserTurnStartStrategy` turns it into a user turn start *and* an interruption, so the engine's VAD (`ws_vad`) replaces a local one with no extra wiring. With `speech_frames="user"` it becomes `UserStartedSpeakingFrame` instead, for `ExternalUserTurnStartStrategy`. |
|
|
243
|
+
| `speech_stopped` | `VADUserStoppedSpeakingFrame` | The turn endpoint, past the VAD hangover. `UserStoppedSpeakingFrame` in `"user"` mode. |
|
|
244
|
+
| `dtmf` | `InputDTMFFrame` | `0`–`9`, `*`, `#`. See the unmapped list below for `A`–`D`. |
|
|
245
|
+
| `mark` | `BotStoppedSpeakingFrame` | A playout boundary was rendered, or skipped on a `clear` (the engine answers a `clear` with a mark named `cleared`). Disable with `emit_bot_speaking_frames=False`. |
|
|
246
|
+
| `stop` | `EndWorkerFrame` | The pipeline worker converts it into a pipeline-wide `EndFrame`, so the transports shut down in order. The engine's `reason` is carried through. |
|
|
247
|
+
| `error` (`fatal: true`) | `CancelWorkerFrame` | Logged at `error`, then the pipeline is cancelled. The engine closes the socket after a fatal error, so this has to be the frame that ends things. |
|
|
248
|
+
| `error` (`fatal: false`) | `ErrorFrame` | Logged at `error` and surfaced; the stream continues. |
|
|
249
|
+
| `event` | *(none)* | Logged at `debug`. Opaque passthrough, so there is no frame it maps to. |
|
|
250
|
+
| `play_start`, `play_stop`, `clear` | *(none)* | Server-to-engine verbs. If one arrives *from* the engine the peer is not siphon-rtp, so it is logged and ignored rather than acted on. |
|
|
251
|
+
| malformed / truncated | *(none)* | Logged at `warning` and dropped. The socket is untrusted input; `deserialize` never raises. |
|
|
252
|
+
|
|
253
|
+
### Pipecat → engine (`serialize`)
|
|
254
|
+
|
|
255
|
+
| Pipecat frame | Wire message | Notes |
|
|
256
|
+
|---|---|---|
|
|
257
|
+
| `OutputAudioRawFrame` (any `AudioRawFrame`) | binary audio | Resampled to the wire rate, folded to mono or duplicated to stereo as the wire requires, byte-swapped for a big-endian wire, encoded to PCMU/PCMA if that is what was negotiated. Dropped on a send-only (tee) stream. |
|
|
258
|
+
| `InterruptionFrame` | `clear` | Barge-in: flush anything the engine still has queued for playout. `reason` from `clear_reason`, default `barge_in`. |
|
|
259
|
+
| `OutputTransportMessageFrame`, `OutputTransportMessageUrgentFrame` | `event` | A `{"name": ..., "payload": ...}` message keeps its name; anything else is wrapped under `event_name`. RTVI messages are filtered out by the base class unless you set `ignore_rtvi_messages=False`. |
|
|
260
|
+
| `EndFrame`, `CancelFrame` | `stop` | Carries the frame's own `reason` when it has one, else `stop_reason`. |
|
|
261
|
+
| everything else | *(none)* | Not sent. |
|
|
262
|
+
|
|
263
|
+
### Deliberately unmapped
|
|
264
|
+
|
|
265
|
+
* **`A`–`D` DTMF digits.** RFC 4733 defines events 12–15 as the A–D digits and the engine will
|
|
266
|
+
report them, but pipecat's `KeypadEntry` only models `0`–`9`, `*` and `#`. Nothing downstream
|
|
267
|
+
could consume an invented frame, so those digits are logged at `debug` and dropped.
|
|
268
|
+
* **`play_start` / `play_stop`.** Downlink audio needs no announcement on this wire: binary frames
|
|
269
|
+
are enough, and the engine's v1 takeover bridge rejects an inline (base64) `play_start` with an
|
|
270
|
+
`error` anyway. There is no pipecat frame that means "start a named playback segment", so
|
|
271
|
+
nothing is emitted and nothing is consumed.
|
|
272
|
+
* **`event` inbound.** An opaque application payload has no pipecat frame it corresponds to; it is
|
|
273
|
+
logged rather than guessed at.
|
|
274
|
+
* **`FrameSerializer.type` / `FrameSerializerType`.** Older pipecat releases had a `type` property
|
|
275
|
+
on serializers. It does not exist in pipecat 1.8 — the transport decides binary versus text from
|
|
276
|
+
whether `serialize` returned `bytes` or `str` — so this package does not define one.
|
|
277
|
+
|
|
278
|
+
## Turn taking and barge-in
|
|
279
|
+
|
|
280
|
+
Engine profile flags move the turn work into the media engine, which is usually where you want it
|
|
281
|
+
for telephony:
|
|
282
|
+
|
|
283
|
+
| Flag | What it does |
|
|
284
|
+
|---|---|
|
|
285
|
+
| `ws_vad` | Emit `speech_started` / `speech_stopped` on the caller's speech edges. |
|
|
286
|
+
| `ws_barge_in` | Flush queued playout the instant the caller talks over the bot, with no server round-trip. Implies `ws_vad`. |
|
|
287
|
+
| `ws_vad_engine` | `"energy"` (default) or `"neural"` — which detector runs. New in 0.3.0. |
|
|
288
|
+
| `ws_vad_min_speech_ms` | A **leading** run: the uplink must read as speech continuously for this long before the start edge (and barge-in) fires. New in 0.3.0. |
|
|
289
|
+
| `ws_vad_threshold`, `ws_vad_hangover_ms` | Tune the energy detector specifically: the mean-square energy that counts as speech, and how long speech is held after it drops before `speech_stopped`. |
|
|
290
|
+
| `echo_cancellation` | Cancel the bot's own audio coming back through the caller's handset. |
|
|
291
|
+
|
|
292
|
+
With `ws_vad` on and the default `speech_frames="vad"`, the engine's VAD drives pipecat's turn
|
|
293
|
+
machinery directly and you do not need a local VAD in the pipeline at all — no
|
|
294
|
+
`SileroVADAnalyzer` on the transport, no second copy of the audio being classified.
|
|
295
|
+
|
|
296
|
+
Which detector: the energy gate answers "is something loud here", so as a *turn* detector it fires
|
|
297
|
+
on mains hum, breathing and fan noise. `"neural"` runs a Silero v5 forward pass inside the engine
|
|
298
|
+
and answers "is what is here speech" instead, for about 27 µs per 20 ms frame on an 8 kHz leg, with
|
|
299
|
+
a 32 ms detection floor from the network's own window. For a conversational bot, `"neural"` plus
|
|
300
|
+
`ws_vad_min_speech_ms` somewhere in 60–120 ms is the pairing that stops a cough, a door or one burst
|
|
301
|
+
of echo cutting the bot off mid-sentence. A detector the engine cannot build for the leg fails the
|
|
302
|
+
offer rather than quietly falling back to the one you were avoiding.
|
|
303
|
+
|
|
304
|
+
`echo_cancellation` is not optional once barge-in is on. Echo of speech is speech to any detector,
|
|
305
|
+
the neural one included, so on a handsfree or loudspeaker endpoint the bot's own voice returning up
|
|
306
|
+
the caller's uplink reads as the caller interrupting.
|
|
307
|
+
|
|
308
|
+
**If you are pinned to siphon-rtp 0.3.0, upgrade.** That release counted the energy detector's
|
|
309
|
+
trailing hangover in milliseconds rather than in ptime frames, so `speech_stopped` — and the
|
|
310
|
+
`VADUserStoppedSpeakingFrame` this serializer emits from it — landed twenty times too late at a
|
|
311
|
+
20 ms ptime, i.e. never inside a normal turn, while `speech_started` and barge-in kept working and
|
|
312
|
+
made turn taking look healthy. Fixed in 0.3.1.
|
|
313
|
+
|
|
314
|
+
Put together, the profile for a conversational bot on 0.3.0 — wideband wire on a narrowband call,
|
|
315
|
+
engine-side turn taking, no local VAD:
|
|
316
|
+
|
|
317
|
+
```json
|
|
318
|
+
"profile": {
|
|
319
|
+
"ws_uri": "ws://198.51.100.10:9001/stream",
|
|
320
|
+
"ws_sample_rate": 16000,
|
|
321
|
+
"ws_vad": true,
|
|
322
|
+
"ws_barge_in": true,
|
|
323
|
+
"ws_vad_engine": "neural",
|
|
324
|
+
"ws_vad_min_speech_ms": 100,
|
|
325
|
+
"echo_cancellation": true
|
|
326
|
+
}
|
|
327
|
+
```
|
|
328
|
+
|
|
329
|
+
If you would rather drive the turn explicitly, set `speech_frames="user"` and wire
|
|
330
|
+
`ExternalUserTurnStartStrategy` / `ExternalUserTurnStopStrategy`. That strategy is constructed with
|
|
331
|
+
interruptions disabled, so `auto_interrupt=True` (the default) also emits an
|
|
332
|
+
`InterruptionWorkerFrame` after the `UserStartedSpeakingFrame`. Pipecat's serializer API hands the
|
|
333
|
+
transport one frame per WebSocket message, so the second frame is queued and drains on the next
|
|
334
|
+
message — at most one ptime later, since the engine's takeover ticker never stops sending.
|
|
335
|
+
|
|
336
|
+
Barge-in in the other direction is automatic: any `InterruptionFrame` in the pipeline becomes a
|
|
337
|
+
`clear` on the wire.
|
|
338
|
+
|
|
339
|
+
## Params
|
|
340
|
+
|
|
341
|
+
All of these live on `SiphonFrameSerializer.InputParams`, which extends pipecat's own
|
|
342
|
+
`FrameSerializer.InputParams` (so `ignore_rtvi_messages` and `resampler_clear_after_secs` are there
|
|
343
|
+
too).
|
|
344
|
+
|
|
345
|
+
| Param | Default | Meaning |
|
|
346
|
+
|---|---|---|
|
|
347
|
+
| `sample_rate` | `None` | Override the pipeline input rate instead of taking `StartFrame.audio_in_sample_rate`. |
|
|
348
|
+
| `wire_sample_rate` | `None` | Wire rate assumed before `start` arrives. `start` always wins afterwards. |
|
|
349
|
+
| `wire_ptime` | `None` | Packetization time assumed before `start`. |
|
|
350
|
+
| `speech_frames` | `"vad"` | `"vad"`, `"user"` or `"none"`. See the matrix above. |
|
|
351
|
+
| `auto_interrupt` | `True` | With `speech_frames="user"`, also emit an `InterruptionWorkerFrame` on `speech_started`. |
|
|
352
|
+
| `emit_bot_speaking_frames` | `True` | Emit `BotStoppedSpeakingFrame` on `mark`. |
|
|
353
|
+
| `stereo_input` | `"caller"` | `"caller"`, `"callee"`, `"mix"` or `"interleaved"`. |
|
|
354
|
+
| `clear_reason` | `"barge_in"` | `reason` on the `clear` an interruption produces. |
|
|
355
|
+
| `stop_reason` | `"pipeline_ended"` | `reason` on the `stop` an `EndFrame`/`CancelFrame` produces when the frame has none. |
|
|
356
|
+
| `event_name` | `"pipecat"` | `name` for an `event` built from an unnamed transport message. |
|
|
357
|
+
|
|
358
|
+
## Example
|
|
359
|
+
|
|
360
|
+
`examples/echo_bot.py` is the minimum that proves the path: a WebSocket server, this serializer,
|
|
361
|
+
and a processor that echoes the caller's audio back into the call. Run it and point an engine at
|
|
362
|
+
it:
|
|
363
|
+
|
|
364
|
+
```bash
|
|
365
|
+
python examples/echo_bot.py --host 127.0.0.1 --port 9001
|
|
366
|
+
```
|
|
367
|
+
|
|
368
|
+
Then offer a call with `"profile": {"ws_uri": "ws://127.0.0.1:9001/stream"}` and you hear yourself
|
|
369
|
+
back, one jitter-buffer frame plus one playout frame later.
|
|
370
|
+
|
|
371
|
+
`examples/agent_bot.py` is that same path with the echo processor replaced by an STT → LLM → TTS
|
|
372
|
+
chain, so a caller talks to a model instead of to themselves. It needs three API keys in the
|
|
373
|
+
environment and `pip install "pipecat-ai[anthropic,deepgram,cartesia]"`; the vendors are one line
|
|
374
|
+
each to swap, and the file explains which knobs are there for latency rather than for taste. The
|
|
375
|
+
turn taking is the engine's, not the pipeline's — there is no VAD analyzer in it at all.
|
|
376
|
+
|
|
377
|
+
## Integration harness
|
|
378
|
+
|
|
379
|
+
`integration/` holds a four-way harness that runs the whole chain with nothing mocked: SIPp
|
|
380
|
+
places a call through siphon-sip, siphon-rtp bridges the leg to a pipecat bot using this
|
|
381
|
+
serializer, and the assertions are on the audio that comes back rather than on the socket being
|
|
382
|
+
open. Three scenarios: a known signal survives the round trip, checked spectrally against a tshark
|
|
383
|
+
capture; the engine's VAD edges land where the fixture puts them and barge-in flushes the bot's
|
|
384
|
+
queued speech; and the same call again with `ws_sample_rate: 16000`, where the wire runs at 16 kHz
|
|
385
|
+
over an 8 kHz G.711 leg and the audio has to come back at the pitch it went out at.
|
|
386
|
+
|
|
387
|
+
```bash
|
|
388
|
+
cd integration && ./run.sh
|
|
389
|
+
```
|
|
390
|
+
|
|
391
|
+
It needs Docker and nothing else: siphon-sip and siphon-rtp are pulled as their published
|
|
392
|
+
release images, so this runs on a fresh clone. Point `SIPHON_RTP_PATH` / `SIPHON_SIP_PATH` at a
|
|
393
|
+
checkout to build either from source instead. Not part of CI yet; see
|
|
394
|
+
[`integration/README.md`](integration/README.md).
|
|
395
|
+
|
|
396
|
+
## Compatibility
|
|
397
|
+
|
|
398
|
+
| | Verified against |
|
|
399
|
+
|---|---|
|
|
400
|
+
| pipecat | `pipecat-ai` 1.8.1 |
|
|
401
|
+
| siphon-rtp | 0.4.x, end to end through the harness under `integration/`. The bridge protocol lives in `crates/siphon-rtp-media/src/bridge/protocol.rs`, and that file has not been touched since before 0.2.0 — its diff across every release since is empty — so the same bytes work against 0.2.x, 0.3.x and 0.4.x alike. What those releases add is surface a *controller* uses: the wire-rate and detector knobs (0.3.0), callers a takeover can terminate (0.3.0), and the attach/detach bridge lifecycle (0.4.0). None of it changes the wire this package speaks. |
|
|
402
|
+
| siphon-sip | 1.7.0+ for the harness, which is the release that carries `ws_sample_rate` and the other 0.3.0 profile flags through to the engine. Not a dependency of this package — the serializer never sees the signalling side. |
|
|
403
|
+
| Python | 3.13 (declared support 3.11+, matching pipecat's floor) |
|
|
404
|
+
|
|
405
|
+
The control-frame fixtures under `tests/wire_fixtures.py` are byte-exact strings produced by
|
|
406
|
+
compiling the engine's own Rust definition against `serde_json` — not by this package's encoder.
|
|
407
|
+
A shared bug on both sides of an encode/decode pair sails straight through a round-trip test; it
|
|
408
|
+
does not survive a comparison against bytes another implementation produced.
|
|
409
|
+
|
|
410
|
+
Note on version skew: an unrecognised control `type` is dropped with a warning rather than
|
|
411
|
+
raising, so a bot pinned to this release keeps working against a newer engine that adds a message.
|
|
412
|
+
|
|
413
|
+
## Development
|
|
414
|
+
|
|
415
|
+
```bash
|
|
416
|
+
uv venv --python 3.13 .venv
|
|
417
|
+
uv pip install -e ".[dev]"
|
|
418
|
+
.venv/bin/python -m pytest
|
|
419
|
+
.venv/bin/ruff check .
|
|
420
|
+
.venv/bin/ruff format --check .
|
|
421
|
+
.venv/bin/mypy
|
|
422
|
+
```
|
|
423
|
+
|
|
424
|
+
## License
|
|
425
|
+
|
|
426
|
+
MIT. See [LICENSE](LICENSE).
|