livekit-plugins-duplexjev 0.1.1__tar.gz → 0.3.0__tar.gz

{livekit_plugins_duplexjev-0.1.1 → livekit_plugins_duplexjev-0.3.0}/CHANGELOG.md

@@ -1,5 +1,18 @@
 # Changelog
 
+## 0.3.0 (2026-10-13)
+
+- `readout="act"`: ask "What should the assistant do now?" and report P(reply now) + P(short acknowledgement) as the
+  end-of-turn probability (also gives the backchannel probability from the same answer). On LiveKit eot-bench
+  (English, 32B-Para v1.2, with the agent's previous message) false cut-offs at 300 ms: 18.4% vs 19.7%.
+- `last_turn_class` (finished / not_finished / backchannel / hesitating, or reply / keep_listening / acknowledge).
+
+## 0.2.0 (2026-10-13)
+
+- Turn-taking context: `detector.attach(session)` (or `detector.set_assistant_said(text)`) sends the agent's previous
+  message with every request. Needs a gateway from `duplexjev>=0.6`. On LiveKit eot-bench (English,
+  DuplexJev-32B-Para v1.2) false cut-offs at a 300 ms budget drop from 23.5% to 19.7%.
+
 ## 0.1.1 (2026-10-12)
 
 - The turn probability is now P(finished) + P(backchannel): a short "ok" / "yeah" with nobody else talking also

{livekit_plugins_duplexjev-0.1.1 → livekit_plugins_duplexjev-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: livekit-plugins-duplexjev
-Version: 0.1.1
+Version: 0.3.0
 Summary: Audio end-of-turn detection for LiveKit Agents with DuplexJev: turn state, emotion and non-verbal sounds from one request, no transcript needed.
 Project-URL: Homepage, https://github.com/adventists-ai/duplexjev/tree/main/integrations/livekit
 Project-URL: Models, https://huggingface.co/adventists-ai/DuplexJev-4B-Para
@@ -66,6 +66,18 @@ session = AgentSession(
 )
 ```
 
+**Give it the agent's last message** (new in 0.2.0). Turn-taking is clearly better when the model knows what the
+agent had just said. Keep a reference to the detector and attach it to the session once:
+
+```python
+detector = duplexjev.TurnDetector(api_url="http://localhost:8420")
+session = AgentSession(..., turn_handling={"turn_detection": detector})
+detector.attach(session)        # follows conversation_item_added; or detector.set_assistant_said(text)
+```
+
+It needs a gateway from `duplexjev>=0.6`. On LiveKit eot-bench (English, DuplexJev-32B-Para v1.2) this cuts false
+cut-offs at a 300 ms budget from 23.5% to 19.7%.
+
 `AgentSession(turn_detection=duplexjev.TurnDetector(), ...)` works too (LiveKit marks that argument as deprecated).
 A complete agent is in [`examples/agent.py`](examples/agent.py).
 
@@ -80,10 +92,10 @@ commits the turn, as with any turn detector, because the LLM needs the text.
 
 ## Run the model yourself (recommended for real-time)
 
-The default URL is our free trial API (rate limited to 30 requests per minute; the round trip from outside mainland China adds network latency). For a live agent, serve the model next to it (DuplexJev-4B-Para needs about 10 GB of GPU memory):
+The default URL is our free trial API (rate limited to 30 requests per minute; the round trip from outside mainland China adds network latency). For a live agent, serve the model next to it (DuplexJev-4B-Para needs about 10 GB of GPU memory; for the best turn-taking use DuplexJev-32B-Para v1.2 on one 80 GB GPU):
 
 ```bash
-pip install "vllm[audio]>=0.29" duplexjev-vllm "duplexjev[server]>=0.4.1"
+pip install "vllm[audio]>=0.29" duplexjev-vllm "duplexjev[server]>=0.6"
 vllm serve adventists-ai/DuplexJev-4B-Para --max-model-len 4096 --port 8000 &
 duplexjev gateway --vllm http://localhost:8000/v1 --port 8420
 ```
@@ -105,6 +117,10 @@ Then pass `api_url="http://localhost:8420"` (or set `DUPLEXJEV_API_URL`).
 | `trim_silence` | `True` | drop the silence before the user started talking (keeps 0.3 s) |
 | `prediction_timeout` | `4.0` hosted, `1.0` own gateway | how long LiveKit waits for a prediction |
 | `fallback_probability` | `1.0` | P(finished) reported if the gateway fails, so the conversation never blocks |
+| `readout` | `"turn"` | `"act"` asks "What should the assistant do now?" (P(reply) + P(short acknowledgement)); slightly better on eot-bench with context |
+
+After each prediction `detector.last_turn_class` is `finished`, `not_finished`, `backchannel` or `hesitating`
+(`reply`, `keep_listening`, `acknowledge` with `readout="act"`), for your own routing.
 
 ## License
 

{livekit_plugins_duplexjev-0.1.1 → livekit_plugins_duplexjev-0.3.0}/README.md

@@ -40,6 +40,18 @@ session = AgentSession(
 )
 ```
 
+**Give it the agent's last message** (new in 0.2.0). Turn-taking is clearly better when the model knows what the
+agent had just said. Keep a reference to the detector and attach it to the session once:
+
+```python
+detector = duplexjev.TurnDetector(api_url="http://localhost:8420")
+session = AgentSession(..., turn_handling={"turn_detection": detector})
+detector.attach(session)        # follows conversation_item_added; or detector.set_assistant_said(text)
+```
+
+It needs a gateway from `duplexjev>=0.6`. On LiveKit eot-bench (English, DuplexJev-32B-Para v1.2) this cuts false
+cut-offs at a 300 ms budget from 23.5% to 19.7%.
+
 `AgentSession(turn_detection=duplexjev.TurnDetector(), ...)` works too (LiveKit marks that argument as deprecated).
 A complete agent is in [`examples/agent.py`](examples/agent.py).
 
@@ -54,10 +66,10 @@ commits the turn, as with any turn detector, because the LLM needs the text.
 
 ## Run the model yourself (recommended for real-time)
 
-The default URL is our free trial API (rate limited to 30 requests per minute; the round trip from outside mainland China adds network latency). For a live agent, serve the model next to it (DuplexJev-4B-Para needs about 10 GB of GPU memory):
+The default URL is our free trial API (rate limited to 30 requests per minute; the round trip from outside mainland China adds network latency). For a live agent, serve the model next to it (DuplexJev-4B-Para needs about 10 GB of GPU memory; for the best turn-taking use DuplexJev-32B-Para v1.2 on one 80 GB GPU):
 
 ```bash
-pip install "vllm[audio]>=0.29" duplexjev-vllm "duplexjev[server]>=0.4.1"
+pip install "vllm[audio]>=0.29" duplexjev-vllm "duplexjev[server]>=0.6"
 vllm serve adventists-ai/DuplexJev-4B-Para --max-model-len 4096 --port 8000 &
 duplexjev gateway --vllm http://localhost:8000/v1 --port 8420
 ```
@@ -79,6 +91,10 @@ Then pass `api_url="http://localhost:8420"` (or set `DUPLEXJEV_API_URL`).
 | `trim_silence` | `True` | drop the silence before the user started talking (keeps 0.3 s) |
 | `prediction_timeout` | `4.0` hosted, `1.0` own gateway | how long LiveKit waits for a prediction |
 | `fallback_probability` | `1.0` | P(finished) reported if the gateway fails, so the conversation never blocks |
+| `readout` | `"turn"` | `"act"` asks "What should the assistant do now?" (P(reply) + P(short acknowledgement)); slightly better on eot-bench with context |
+
+After each prediction `detector.last_turn_class` is `finished`, `not_finished`, `backchannel` or `hesitating`
+(`reply`, `keep_listening`, `acknowledge` with `readout="act"`), for your own routing.
 
 ## License
 

{livekit_plugins_duplexjev-0.1.1 → livekit_plugins_duplexjev-0.3.0}/livekit/plugins/duplexjev/turn_detector.py

@@ -8,6 +8,10 @@ exposed on ``last_answers`` / ``on_answers``.
 This module implements LiveKit's *streaming* (audio) turn-detector protocol: ``AgentSession`` pushes every input
 audio frame into the stream, calls ``predict()`` once the VAD has seen ~200 ms of silence, and waits up to
 ``prediction_timeout`` for the resulting ``TurnDetectionEvent``.
+
+Turn-taking is markedly better when the model also knows what the agent had just said. Call
+``detector.attach(session)`` once after creating the ``AgentSession`` (it follows ``conversation_item_added``), or
+set it yourself with ``detector.set_assistant_said(text)``.
 """
 
 from __future__ import annotations
@@ -52,6 +56,9 @@ REPLY_QUESTION = {
            "options": ["马上回复", "继续听", "简短回应一下"], "lang": "zh"},
 }
 
+TURN_CLASSES = ["finished", "not_finished", "backchannel", "hesitating"]
+ACT_CLASSES = ["reply", "keep_listening", "acknowledge"]
+
 PRESETS = {
     "emotion": {"en": ("What is the speaker's emotional state?", ["neutral", "happy", "angry", "sad"]),
                 "zh": ("说话人当时的情绪状态是？", ["中性", "高兴", "生气", "伤心"])},
@@ -113,6 +120,9 @@ class TurnDetector:
         extra_questions: names from ``PRESETS`` (``"emotion"``, ``"sound"``, ``"barge_in"``) or question dicts
             ``{"id", "text", "options"}``, answered in the same request; see ``last_answers``.
         on_answers: optional callback ``f(answers: dict)`` called after every successful prediction.
+        readout: ``"turn"`` (turn-state question, default) or ``"act"``: ask "What should the assistant do now?" and
+            report P(reply now) + P(short acknowledgement); slightly better on eot-bench with the agent's previous
+            message (18.4% vs 19.7% false cut-offs at 300 ms, English, 32B-Para v1.2).
         window_s: seconds of the current user turn sent with each request (the most recent ones).
         trim_silence: drop the quiet audio before the user started talking, keeping 0.3 s of pre-roll.
         prediction_timeout: how long LiveKit waits for a prediction before committing without one. Defaults to
@@ -138,6 +148,7 @@ class TurnDetector:
         prediction_timeout: float | None = None,
         fallback_probability: float = 1.0,
         http_session: aiohttp.ClientSession | None = None,
+        readout: str = "turn",
     ) -> None:
         if lang not in TURN_QUESTION:
             raise ValueError("lang must be 'en' or 'zh'")
@@ -160,8 +171,12 @@ class TurnDetector:
         self._fallback_probability = fallback_probability
         self._http_session = http_session
 
-        self._questions: list[dict[str, Any]] = [TURN_QUESTION[lang]]
-        if backchannel_threshold is not None:
+        if readout not in ("turn", "act"):
+            raise ValueError("readout must be 'turn' or 'act'")
+        self._readout = readout
+        main = dict(REPLY_QUESTION[lang], id="turn") if readout == "act" else TURN_QUESTION[lang]
+        self._questions: list[dict[str, Any]] = [main]
+        if backchannel_threshold is not None and readout == "turn":
             self._questions.append(REPLY_QUESTION[lang])
         self._questions += [self._as_question(q) for q in (extra_questions or [])]
         ids = [q["id"] for q in self._questions]
@@ -173,6 +188,30 @@ class TurnDetector:
         """Answers of the latest successful request: ``{id: {"answer", "confidence", "probs"}}``."""
         self.last_server_ms: float | None = None
         """Server-side model time of the latest request, in milliseconds."""
+        self.last_turn_class: str | None = None
+        """Most likely class of the latest request: finished / not_finished / backchannel / hesitating
+        (``readout="act"``: reply / keep_listening / acknowledge)."""
+        self.assistant_said: str = ""
+        """The agent's previous message, sent with every request (last 300 characters)."""
+
+    def set_assistant_said(self, text: str | None) -> None:
+        """What the agent said just before the user's current turn."""
+        self.assistant_said = " ".join((text or "").split())[-300:]
+
+    def attach(self, session: Any) -> None:
+        """Keep ``assistant_said`` up to date from an ``AgentSession``'s ``conversation_item_added`` events."""
+
+        def _on_item(ev: Any) -> None:
+            item = getattr(ev, "item", None)
+            if getattr(item, "role", None) != "assistant":
+                return
+            text = getattr(item, "text_content", None)
+            if callable(text):
+                text = text()
+            if text:
+                self.set_assistant_said(text)
+
+        session.on("conversation_item_added", _on_item)
 
     def _as_question(self, q: str | dict[str, Any]) -> dict[str, Any]:
         if isinstance(q, str):
@@ -366,6 +405,8 @@ class TurnDetectorStream:
         det = self._detector
         payload = {"audio_b64": base64.b64encode(wav).decode(), "format": "wav", "lang": det._lang,
                    "questions": det._questions}
+        if det.assistant_said:
+            payload["assistant_said"] = det.assistant_said
         try:
             async with self._http().post(
                 det._decide_url, json=payload, headers=det._headers,
@@ -377,9 +418,12 @@ class TurnDetectorStream:
                 data = await resp.json()
             answers: dict[str, Any] = data["answers"]
             probs = answers["turn"]["probs"]
-            o = TURN_QUESTION[det._lang]["options"]
-            # finished + backchannel: a short "ok" / "yeah" with nobody else talking also ends the turn
+            o = (REPLY_QUESTION if det._readout == "act" else TURN_QUESTION)[det._lang]["options"]
+            # turn: finished + backchannel (a short "ok" with nobody else talking also ends the turn);
+            # act: reply now + short acknowledgement
             p_finished = float(probs.get(o[0], 0.0)) + float(probs.get(o[2], 0.0))
+            names = ACT_CLASSES if det._readout == "act" else TURN_CLASSES
+            det.last_turn_class = names[max(range(len(o)), key=lambda i: float(probs.get(o[i], 0.0)))]
         except asyncio.CancelledError:
             raise
         except Exception as e:  # network or server error: never block the conversation
@@ -398,8 +442,10 @@ class TurnDetectorStream:
         if data.get("model"):
             det._model_name = str(data["model"])
         backchannel = None
-        if "reply" in answers and det._backchannel_threshold is not None:
-            backchannel = float(answers["reply"]["probs"].get(REPLY_QUESTION[det._lang]["options"][2], 0.0))
+        if det._backchannel_threshold is not None:
+            src = answers["turn"] if det._readout == "act" else answers.get("reply")
+            if src is not None:
+                backchannel = float(src["probs"].get(REPLY_QUESTION[det._lang]["options"][2], 0.0))
         now = time.time()
         self.last_answers = det.last_answers = answers
         self.last_server_ms = det.last_server_ms = server_ms

livekit_plugins_duplexjev-0.3.0/livekit/plugins/duplexjev/version.py

@@ -0,0 +1 @@
+__version__ = "0.3.0"

{livekit_plugins_duplexjev-0.1.1 → livekit_plugins_duplexjev-0.3.0}/tests/test_turn_detector.py

@@ -172,3 +172,59 @@ async def test_agent_session_accepts_it(gateway):
     det = duplexjev.TurnDetector(api_url=gateway.url)
     session = AgentSession(turn_handling={"turn_detection": det})
     assert session.turn_detection is det
+
+
+async def test_assistant_said_is_sent_and_attach_follows_session(gateway):
+    det = duplexjev.TurnDetector(api_url=gateway.url)
+    stream = det.stream()
+    push(stream, 1.0)
+    await asyncio.wait_for(stream.predict(), 2)
+    assert "assistant_said" not in gateway.requests[-1]
+
+    handlers = {}
+
+    class FakeSession:
+        def on(self, name, fn):
+            handlers[name] = fn
+
+    class Item:
+        def __init__(self, role, text):
+            self.role, self.text_content = role, text
+
+    class Ev:
+        def __init__(self, item):
+            self.item = item
+
+    det.attach(FakeSession())
+    handlers["conversation_item_added"](Ev(Item("user", "hello")))
+    assert det.assistant_said == ""
+    handlers["conversation_item_added"](Ev(Item("assistant", "Sure,  what time\nworks for you?")))
+    assert det.assistant_said == "Sure, what time works for you?"
+    await asyncio.wait_for(stream.predict(), 2)
+    assert gateway.requests[-1]["assistant_said"] == "Sure, what time works for you?"
+    det.set_assistant_said("y" * 500)
+    assert len(det.assistant_said) == 300
+    await stream.aclose()
+
+
+async def test_act_readout_and_turn_class(gateway):
+    det = duplexjev.TurnDetector(api_url=gateway.url, readout="act", backchannel_threshold=0.5)
+    stream = det.stream()
+    push(stream, 1.0)
+    gateway.p_finished = 0.8
+    ev = await asyncio.wait_for(stream.predict(), 2)
+    qs = gateway.requests[-1]["questions"]
+    assert [q["id"] for q in qs] == ["turn"] and qs[0]["text"] == "What should the assistant do now?"
+    assert ev.end_of_turn_probability == pytest.approx(0.8) and det.last_turn_class == "reply"
+    assert ev.backchannel_probability == pytest.approx(0.0)
+    gateway.p_finished = 0.2
+    ev = await asyncio.wait_for(stream.predict(), 2)
+    assert det.last_turn_class == "keep_listening"
+    await stream.aclose()
+    t = duplexjev.TurnDetector(api_url=gateway.url)
+    s = t.stream(); push(s, 1.0); gateway.p_finished = 0.9
+    await asyncio.wait_for(s.predict(), 2)
+    assert t.last_turn_class == "finished"
+    await s.aclose()
+    with pytest.raises(ValueError):
+        duplexjev.TurnDetector(readout="x")

livekit_plugins_duplexjev-0.1.1/livekit/plugins/duplexjev/version.py

@@ -1 +0,0 @@
-__version__ = "0.1.1"