pipecat-memorysync 1.1.0__tar.gz → 1.1.1__tar.gz

{pipecat_memorysync-1.1.0 → pipecat_memorysync-1.1.1}/CHANGELOG.md

@@ -2,6 +2,19 @@
 
 All notable changes to `pipecat-memorysync` are documented here.
 
+## 1.1.1 — 2026-08-29
+
+- **Immediate, loss-proof capture.** 1.1.0's turn-complete merging
+  deferred the current utterance until the turn completed — on a browser
+  disconnect that deferral raced the brief cancel salvage window and
+  could lose the newest (usually most important) turn over slow
+  networks. Capture is immediate again: every new user/assistant message
+  is in flight the moment its frame passes, BEFORE the LLM replies.
+  Junk filtering and fact extraction now happen server-side (the
+  platform's low-value gate + conversational ingestion), so client-side
+  merging is unnecessary. Suite: 17 checks on Pipecat v1.8.1, including
+  the exact disconnect sequence that previously lost data.
+
 ## 1.1.0 — 2026-08-29
 
 - **Turn-complete capture.** Voice aggregators split one utterance across

{pipecat_memorysync-1.1.0 → pipecat_memorysync-1.1.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: pipecat-memorysync
-Version: 1.1.0
+Version: 1.1.1
 Summary: MemorySync for Pipecat: budgeted memory recall that never stalls a voice reply, delta-only conversation persistence with idempotency seeds, and a pipeline that cannot be broken by a memory outage.
 Project-URL: Homepage, https://docs.memorysync.io/guides/pipecat
 Project-URL: Documentation, https://docs.memorysync.io/guides/pipecat
@@ -81,10 +81,12 @@ enriched or not, on time.
 - **Budgeted recall.** Enrichment runs under a hard timeout (default
   **1.2 s**). A slow or dead memory backend means an unenriched frame, never a
   stalled voice reply.
-- **Turn-complete capture.** Voice aggregators split one utterance across
-  several context messages at speech pauses; consecutive fragments merge
-  into ONE stored turn when the assistant's reply completes it (the
-  in-progress tail flushes at end of call) — no per-fragment junk rows.
+- **Immediate, loss-proof capture.** Every new message is sent the moment
+  its frame passes — the current utterance is in flight BEFORE the LLM
+  replies, so a disconnect can never lose it. Junk filtering and fact
+  extraction happen server-side (the platform's low-value gate +
+  conversational ingestion): the memory store receives curated facts,
+  not transcript clutter.
 - **Delta-only capture.** Only messages *not seen before* are stored, tracked
   by deterministic idempotency seeds. Growing a 50-message context does not
   re-store 50 messages per turn.

{pipecat_memorysync-1.1.0 → pipecat_memorysync-1.1.1}/README.md

@@ -57,10 +57,12 @@ enriched or not, on time.
 - **Budgeted recall.** Enrichment runs under a hard timeout (default
   **1.2 s**). A slow or dead memory backend means an unenriched frame, never a
   stalled voice reply.
-- **Turn-complete capture.** Voice aggregators split one utterance across
-  several context messages at speech pauses; consecutive fragments merge
-  into ONE stored turn when the assistant's reply completes it (the
-  in-progress tail flushes at end of call) — no per-fragment junk rows.
+- **Immediate, loss-proof capture.** Every new message is sent the moment
+  its frame passes — the current utterance is in flight BEFORE the LLM
+  replies, so a disconnect can never lose it. Junk filtering and fact
+  extraction happen server-side (the platform's low-value gate +
+  conversational ingestion): the memory store receives curated facts,
+  not transcript clutter.
 - **Delta-only capture.** Only messages *not seen before* are stored, tracked
   by deterministic idempotency seeds. Growing a 50-message context does not
   re-store 50 messages per turn.

{pipecat_memorysync-1.1.0 → pipecat_memorysync-1.1.1}/src/pipecat_memorysync/_version.py

@@ -1,3 +1,3 @@
 """Version for pipecat-memorysync. Single source of truth."""
 
-__version__ = "1.1.0"
+__version__ = "1.1.1"

{pipecat_memorysync-1.1.0 → pipecat_memorysync-1.1.1}/src/pipecat_memorysync/service.py

@@ -7,13 +7,13 @@ predecessors don't:
 1. **Recall is budgeted.** Enrichment waits at most ``recall_timeout``
    seconds (default 1.2). On timeout or failure the context frame passes
    through unenriched — a voice reply is never stalled by a slow network.
-2. **Capture is turn-complete and delta-only.** Voice aggregators split
-   one utterance across several context messages at speech pauses; this
-   service merges consecutive new user fragments and stores them as ONE
-   verbatim turn once the assistant's reply marks the turn complete (the
-   in-progress tail is flushed at end of call). Only NEW messages are
-   considered each frame — never the whole conversation re-sent — and
-   every stored turn carries a cross-adapter fnv1a64 idempotency seed.
+2. **Capture is immediate and delta-only.** Every NEW user/assistant
+   message is sent to the platform the moment it appears in a context
+   frame — the current utterance is already in flight BEFORE the LLM
+   replies, so a disconnect can never lose it. Junk filtering and fact
+   extraction are the platform's job (the low-value gate + conversational
+   ingestion), not the client's. Only new messages are sent each frame,
+   with cross-adapter fnv1a64 idempotency seeds.
 3. **Nothing raises, nothing is dropped.** Every failure path logs and
    pushes the ORIGINAL frame through. The pipeline cannot stall and the
    LLM always gets its context.
@@ -141,14 +141,9 @@ class MemorySyncMemoryService(FrameProcessor):
         self.params = params
 
         self._stored_seeds: Set[str] = set()
-        # Message-level consumption ledger: which context messages have
-        # been merged into a stored turn. Distinct from _stored_seeds
-        # (which dedups at merged-turn granularity) so a failed store can
-        # release its messages for re-capture on the next frame.
+        # Message-level ledger: which context messages have been sent.
+        # A failed store releases its message so the next frame retries.
         self._seen_messages: Set[str] = set()
-        # Snapshot of the in-progress user utterance (fragments after the
-        # last assistant reply) — flushed as one turn at end of call.
-        self._tail_pending: List[tuple[str, str]] = []
         self._store_tasks: Set[asyncio.Task] = set()
         self._last_query: Optional[str] = None
 
@@ -249,25 +244,21 @@ class MemorySyncMemoryService(FrameProcessor):
         body = "\n".join(lines)
         return f"{self.params.system_prompt}\n{body}\n\n{CONTEXT_GUARD}"
 
-    # ── capture: turn-complete, delta-only, background ─────────────────
+    # ── capture: immediate, delta-only, background ─────────────────────
 
     def _capture_delta(self, context: Any) -> None:
-        """Queue storage for completed turns built from NEW messages only.
-
-        Voice aggregators append each speech fragment as its own user
-        message ("Do you know that my favorite", "dinner", "and", …).
-        Storing them individually floods memory with junk rows, so:
-
-        - consecutive NEW user messages merge into ONE turn, stored the
-          moment an assistant message follows them (turn completed);
-        - user messages after the last assistant reply are an in-progress
-          utterance — they stay pending (snapshotted for the end-of-call
-          flush) instead of being stored piecemeal;
-        - NEW assistant messages store immediately, flushing any pending
-          user group first so ordering survives.
+        """Queue storage for NEW messages the moment they appear.
+
+        The platform owns junk filtering (the low-value gate) and fact
+        extraction (conversational ingestion), so the adapter's single
+        duty is delivering every turn reliably and EARLY. The current
+        user utterance is in the frame BEFORE the LLM replies — sending
+        it immediately means a mid-call disconnect can never lose it.
+        (1.1.0 deferred the current utterance until the turn completed;
+        that deferral raced the disconnect salvage window and could drop
+        the newest — usually most important — turn. Never again.)
         """
         header = self.params.system_prompt
-        entries: List[tuple[str, str]] = []  # (speaker_role, trimmed_text)
         for message in context.get_messages():
             role = message.get("role")
             if role not in ("user", "assistant"):
@@ -277,57 +268,12 @@ class MemorySyncMemoryService(FrameProcessor):
                 continue  # our own injection (user-role mode) never re-enters
             speaker_role = "human" if role == "user" else "ai"
             trimmed = text if len(text) <= MAX_TURN_CHARS else text[:MAX_TURN_CHARS] + "…"
-            entries.append((speaker_role, trimmed))
-
-        last_ai = -1
-        for index, (speaker_role, _text) in enumerate(entries):
-            if speaker_role == "ai":
-                last_ai = index
-
-        group_texts: List[str] = []
-        group_keys: List[str] = []
-        for index, (speaker_role, text) in enumerate(entries):
-            key = f"{speaker_role}:{text}"
-            if speaker_role == "human":
-                if index > last_ai:
-                    continue  # in-progress utterance — tail snapshot below
-                if key in self._seen_messages:
-                    continue
-                group_texts.append(text)
-                group_keys.append(key)
-            else:
-                if group_texts:
-                    self._store_merged(group_texts, group_keys)
-                    group_texts, group_keys = [], []
-                if key not in self._seen_messages:
-                    self._seen_messages.add(key)
-                    self._bound_seen()
-                    self._queue_store("ai", text, [key])
-
-        self._tail_pending = [
-            (text, f"human:{text}")
-            for speaker_role, text in entries[last_ai + 1:]
-            if speaker_role == "human" and f"human:{text}" not in self._seen_messages
-        ]
-
-    def _store_merged(self, texts: List[str], keys: List[str]) -> None:
-        """One utterance from its fragments: mark consumed, merge, store."""
-        for key in keys:
+            key = f"{speaker_role}:{trimmed}"
+            if key in self._seen_messages:
+                continue
             self._seen_messages.add(key)
-        self._bound_seen()
-        merged = " ".join(texts)
-        if len(merged) > MAX_TURN_CHARS:
-            merged = merged[:MAX_TURN_CHARS] + "…"
-        self._queue_store("human", merged, keys)
-
-    def _flush_tail(self) -> None:
-        """Store the in-progress utterance (end-of-call, no reply coming)."""
-        if not self._tail_pending:
-            return
-        texts = [text for text, _key in self._tail_pending]
-        keys = [key for _text, key in self._tail_pending]
-        self._tail_pending = []
-        self._store_merged(texts, keys)
+            self._bound_seen()
+            self._queue_store(speaker_role, trimmed, [key])
 
     def _bound_seen(self) -> None:
         if len(self._seen_messages) > 4096:
@@ -360,7 +306,7 @@ class MemorySyncMemoryService(FrameProcessor):
             )
         except Exception as exc:  # noqa: BLE001
             # The write never landed: release both ledgers so the next
-            # frame re-captures these messages and retries the store.
+            # frame re-captures this message and retries the store.
             self._stored_seeds.discard(seed)
             for key in msg_keys:
                 self._seen_messages.discard(key)
@@ -392,12 +338,7 @@ class MemorySyncMemoryService(FrameProcessor):
     # ── lifecycle ──────────────────────────────────────────────────────
 
     async def _flush(self, *, timeout: float) -> None:
-        """Store the in-progress utterance, then let queued stores land,
-        bounded. Never raises."""
-        try:
-            self._flush_tail()
-        except Exception:  # noqa: BLE001
-            pass
+        """Let queued stores land, bounded. Never raises."""
         pending = [t for t in self._store_tasks if not t.done()]
         if not pending:
             return

{pipecat_memorysync-1.1.0 → pipecat_memorysync-1.1.1}/tests/test_service.py

@@ -252,61 +252,47 @@ async def test_missing_user_id_is_loud_at_construction(mock):
         MemorySyncMemoryService(api_key="ms_x", user_id="", transport=mock.transport())
 
 
-# ── capture: turn-complete merging (voice fragmenting) ────────────────
+# ── capture: immediate and loss-proof ─────────────────────────────────
 
 
-async def test_speech_fragments_merge_into_one_turn(mock, make_service):
-    """VAD pauses split one utterance across context messages — they must
-    store as ONE merged row when the assistant reply completes the turn,
-    never as per-fragment junk rows ("and", "dinner", …)."""
+async def test_current_utterance_stores_before_any_reply(mock, make_service):
+    """The user's words must be in flight the moment the frame passes —
+    BEFORE the LLM replies — so a disconnect can never lose them."""
     service = make_service()
-    fragments = [
-        "Do you know that my favorite",
-        "dinner",
-        "and",
-        "dinner is my favorite dish is biryani.",
-    ]
-    # The aggregator's view grows frame by frame while the user speaks…
-    growing = [
-        context_of([{"role": "user", "content": f} for f in fragments[: i + 1]])
-        for i in range(len(fragments))
-    ]
-    # …and the turn completes when the assistant replies.
-    completed = context_of(
-        [{"role": "user", "content": f} for f in fragments]
-        + [{"role": "assistant", "content": "Got it, biryani it is!"}]
-    )
-    await drive(service, *growing, completed)
-    await drain(mock, 2)
-
-    texts = sorted(r["text"] for r in mock.rows)
-    assert texts == [
-        "ai: Got it, biryani it is!",
-        "human: Do you know that my favorite dinner and dinner is my favorite dish is biryani.",
+    ctx = context_of([
+        {"role": "user", "content": "My age is twenty two and I completed my bachelor's."},
+    ])
+    await drive(service, ctx)
+    await drain(mock, 1)
+    assert [r["text"] for r in mock.rows] == [
+        "human: My age is twenty two and I completed my bachelor's."
     ]
-    assert mock.add_turn_calls() == 2, "four fragments + one reply = exactly two rows"
 
 
-async def test_in_progress_utterance_flushes_once_at_end_of_call(mock, make_service):
-    """A call that ends mid-utterance (no reply coming) must not lose the
-    tail — and must store it merged, not per fragment."""
+async def test_demo_disconnect_sequence_loses_nothing(mock, make_service):
+    """The exact sequence that lost data in 1.1.0: greeting frame, then a
+    frame carrying the reply + the important utterance, then immediate
+    CancelFrame (browser disconnect). Every message must already be
+    stored — nothing may depend on a post-cancel flush window."""
     service = make_service()
-    ctx1 = context_of([{"role": "user", "content": "my name is Abdullah"}])
-    ctx2 = context_of([
-        {"role": "user", "content": "my name is Abdullah"},
-        {"role": "user", "content": "and my favorite color is black"},
+    f1 = context_of([{"role": "user", "content": "Hello there, anyone home?"}])
+    f2 = context_of([
+        {"role": "user", "content": "Hello there, anyone home?"},
+        {"role": "assistant", "content": "Hi! How can I help you today?"},
+        {"role": "user", "content": "My age is twenty two and I completed my bachelor's."},
     ])
-    await drive(service, ctx1, ctx2)  # run_test's EndFrame triggers the flush
-    await drain(mock, 1)
-
-    assert [r["text"] for r in mock.rows] == [
-        "human: my name is Abdullah and my favorite color is black"
+    await drive(service, f1, f2)  # run_test tears the pipeline down right after
+    await drain(mock, 3)
+    texts = sorted(r["text"] for r in mock.rows)
+    assert texts == [
+        "ai: Hi! How can I help you today?",
+        "human: Hello there, anyone home?",
+        "human: My age is twenty two and I completed my bachelor's.",
     ]
-    assert mock.add_turn_calls() == 1
 
 
-async def test_failed_store_releases_fragments_for_retry(mock, make_service):
-    """A store that never landed must not consume its messages — the next
+async def test_failed_store_releases_message_for_retry(mock, make_service):
+    """A store that never landed must not consume its message — the next
     frame re-captures and retries, converging on one row."""
     service = make_service()
     mock.fail_next = 503