yrby 0.2.1 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d6580b564ad0ffb8573b7e5f81acdba9ad4b59648d4c2b25c5914d144cef0623
-  data.tar.gz: 1ee082cbd115166217d4ac48a62e32d2d0f161564d02644dfe3a23731822ca87
+  metadata.gz: 2dcb5650668e6cc3f1e0ef1349426ba875d5580a945e90a62216e61a24bf9cef
+  data.tar.gz: eead1e7159caeacf517febe05149d0c55491dcceaf6fbbbe2f3c40fdf7620978
 SHA512:
-  metadata.gz: da995eb5f169b62d3a83bfe138be28894b66fd9048a091efb624019ed6e87b520429c8ea29df17eef5ad55929e4b7680190fb84fad06b1f952af72c0bec300d7
-  data.tar.gz: a6bd21c7f3d353037fd877b0246bca1221bdd894bd7dc4f1a811fdd74fcbae26dc7a0d087d9bba13c79cbfb8134e4c588a2493e527a57ed31d6e1faed3504e8a
+  metadata.gz: 68948615ae023becf04deea8c767cdd075ccd7097bbcecefb52868a7c2257fe4bd39d20bbac83b1f6a47d78a2889d76e590a83b7df0d9f6e1ccea1a8fdfa115d
+  data.tar.gz: 70716f2948661046b71843e66dcec62e5cf3e2b5c1509e9ea76c49db6039909bc02f510a90e3b9f44054b20ea3c2297d10878834b3c751abd1a28c72d0d2cabc

data/CHANGELOG.md

@@ -6,6 +6,72 @@ to follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+## [0.3.0] - 2026-07-01
+
+### Fixed
+
+- **Sync no longer serves un-integrable pending structs.** When a doc holds a
+  *pending* struct (a gappy update whose causally-prior update is missing — e.g.
+  legacy data recorded before the `update_ready?` gate existed), its integrated
+  state vector is empty but `encode_state_as_update` merges the pending bytes back
+  in. Answering a peer's `SyncStep1` with that state handed the peer content it
+  couldn't integrate, so it parked the same pending forever and the empty-SV /
+  non-empty-content mismatch drove endless resync traffic (observed as a browser
+  re-sending frames several times a second). `handle_sync_message` now answers
+  `SyncStep1` with **integrated-only** state, so a server never serves a struct it
+  can't integrate itself. Neutralizes existing poisoned server state on deploy —
+  no migration needed. The server's own pending is untouched and still heals if
+  the missing dependency later arrives (only then does the content become
+  visible in sync). Live delta relay (`Update` frames) is unchanged.
+
+### Added
+
+- `Doc#pending?` — true if the doc holds un-integrable pending structs or a
+  pending delete set (content waiting on a missing causally-prior update).
+- `Doc#compacted_state_update` — like `encode_state_as_update` (full state) but
+  **gap-free**: excludes pending structs/delete set. Use it when persisting or
+  serving state other peers will apply. Non-destructive — the doc keeps its
+  pending (so it can still heal), and `encode_state_as_update` stays lossless for
+  raw-update recovery.
+
+## [0.2.3] - 2026-07-01
+
+### Fixed
+
+- `Doc#update_advances?` is now exact for **delete-bearing** updates, so an
+  already-applied pure-delete retry no longer reports as advancing. Previously any
+  update carrying a delete set returned `true` (record it) because deletes don't
+  move the state vector, so the cheap state-vector probe couldn't prove a
+  duplicate. A lost-ack retry of a deletion the server had already integrated was
+  therefore re-recorded and re-broadcast every time. For delete-bearing updates we
+  now compare the full encoded document state (which includes the delete set)
+  before vs. after a trial apply on an isolated probe: a genuinely new deletion
+  changes it (`true`); an already-applied retry re-encodes identically (`false`).
+  Insert/format-only updates keep the cheaper state-vector path, so only
+  delete-bearing frames — a minority — pay for the exact comparison. The exactly-
+  once guarantee is unchanged in the safe direction: a real deletion is never
+  dropped.
+
+  This lets `yrby-actioncable` (and any caller gating `on_change` on
+  `update_advances?`) settle a duplicate pure-delete frame as `:applied` — acked,
+  but not stored or relayed — so apps no longer need an app-level
+  encode-and-compare guard around their durable writes.
+
+## [0.2.2] - 2026-06-30
+
+### Fixed
+
+- `Doc#read_xml` now recovers text from **nested** Lexical/Lexxy blocks. Lexical
+  embeds child blocks (list items, table cells, nested lists) as `Y.XmlText`
+  embeds that `get_string` silently drops, so lists and tables previously came
+  back empty. `read_xml` now walks the embeds: text runs build a line, inline
+  children (links) join it, and nested block children flush and recurse — so a
+  document with headings, formatted text, links, bullet/numbered/check/nested
+  lists, blockquotes, code blocks and tables extracts every piece of text.
+  Lexical decorator elements (horizontal rule, image) are skipped instead of
+  emitting their `<UNDEFINED …>` serialization. ProseMirror handling is
+  unchanged.
+
 ## [0.2.1] - 2026-06-29
 
 ### Changed
@@ -15,15 +81,9 @@ to follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [0.2.0] - 2026-06-28
 
-First release under the **`yrby`** name (the project was previously developed
-as `yrb-lite`). The public Ruby interface is the top-level module **`Y`** —
-mirroring the `y-rb` gem's `Y::Doc` interface.
-
-### Changed
-- **Renamed `yrb-lite` → `yrby`.** Module `YrbLite` → top-level `Y`
-  (`Y::Doc`, `Y::Error`, `Y::VERSION`). Require path `require "yrb_lite"` →
-  `require "y"`. (The native extension crate shipped as `y_ruby` in 0.2.0; see
-  0.2.1 for its rename to `yrby`.)
+First release. The public Ruby interface is the top-level module **`Y`**
+(`Y::Doc`, `Y::Error`, `Y::VERSION`), loaded with `require "y"` — mirroring the
+`y-rb` gem's `Y::Doc` interface.
 
 ### Added
 - Native `Doc#read_text` and `Doc#read_map` readers — reconstruct plain text and

data/README.md

@@ -41,23 +41,25 @@ npm install yrby-client
   one-include ActionCable concern.
 - Authoritative record-before-distribute semantics: each document change can be
   recorded durably before it goes out to anyone.
-- 
+- Optional server-side reads: `Doc#read_text` and `Doc#read_map` reconstruct a
+  document's contents in Ruby - no Node process - for search, exports, validation,
+  or server-side rendering.
 
-## Why "lite"
+## Scope
 
-The "lite" is the size of the surface. `yrby` binds just the part of `y-crdt` you
-need to *sync and persist* collaborative documents - a `Doc`, awareness, and the
-y-websocket protocol primitives. The Ruby side treats a document as opaque CRDT
-state: it applies updates, answers sync handshakes, and records deltas, but never
-reaches in to read or edit the contents. The browser editor owns the document's
-shape.
+`yrby` binds just the part of `y-crdt` you need to *sync and persist* collaborative
+documents - a `Doc`, awareness, and the y-websocket protocol primitives. By default
+the Ruby side treats a document as opaque CRDT state: it applies updates, answers
+sync handshakes, and records deltas without reaching into the contents - the browser
+editor owns the document's shape. When you do need to look inside, `Doc#read_text`
+and `Doc#read_map` reconstruct it server-side, in Ruby.
 
-## What isn't "lite"
+## Durability and delivery
 
-The surface area may be "lite", but a core focus is on durability, resiliency, delivery
+The surface is intentionally small, but the focus is durability, resiliency, delivery
 guarantees, correctness, and thread safety.
 
-Towards that goal, `yrby` adds capabilities that may even stand out in the Yjs ecosystem:
+Towards that goal, `yrby` adds capabilities that stand out even in the Yjs ecosystem:
 
 - Built-in update acknowledgement: the `ActionCableProvider` in `yrby-client` will continue to
   send updates until an ack is received from the server. [`yrby-actioncable`](https://rubygems.org/gems/yrby-actioncable)
@@ -140,18 +142,39 @@ doc = Y::Doc.new(12345) # specific client ID (used for CRDT identity)
 
 # Encoding
 doc.encode_state_vector           # => current state vector
-doc.encode_state_as_update        # => full update
+doc.encode_state_as_update        # => full update (lossless: keeps pending)
 doc.encode_state_as_update(sv)    # => update diff against state vector
+doc.compacted_state_update        # => full update, gap-free (excludes pending)
 
 # Applying updates
 doc.apply_update(update_bytes)    # apply raw V1 update
+doc.pending?                      # => true if holding un-integrable pending structs
 
 # Sync protocol
 doc.sync_step1                    # => SyncStep1 message (this doc's state vector)
 doc.handle_sync_message(data)     # => [msg_type, sync_type, response]; answers a
-                                  #    peer's SyncStep1 with a SyncStep2
+                                  #    peer's SyncStep1 with an integrated-only
+                                  #    SyncStep2 (never serves pending structs)
 ```
 
+### Pending structs and gap-free state
+
+If a doc applies an update whose causally-prior update is missing (a "gappy"
+update), yrs parks it as a **pending** struct: the integrated state vector stays
+empty, but the pending block is held as a recovery buffer and heals if the
+missing dependency later arrives. `Doc#pending?` reports this.
+
+Pending structs are *not* document state, so they must not cross the sync
+boundary — a peer that receives one can't integrate it and gets stuck. Two
+guarantees keep serving safe:
+
+- `handle_sync_message` answers `SyncStep1` with **integrated-only** state, so a
+  server never serves a struct it can't integrate itself (this is automatic).
+- `Doc#compacted_state_update` gives you the same gap-free full-state update for
+  when you persist or hand off state yourself. It's non-destructive (the doc
+  keeps its pending), while `encode_state_as_update` stays lossless so you can
+  still preserve the raw pending bytes for recovery.
+
 ### Protocol codec (module functions)
 
 Classifying and unwrapping wire frames is stateless, so it's exposed as

data/ext/yrby/src/lib.rs

@@ -8,7 +8,10 @@ use yrs::{Doc, GetString, ReadTxn, Transact};
 
 mod protocol;
 mod read;
-use protocol::{classify_message, merged_doc_update, update_advances_doc, update_is_ready};
+use protocol::{
+    classify_message, has_pending, integrated_update, merged_doc_update, update_advances_doc,
+    update_is_ready,
+};
 
 /// Wrapper around yrs Doc.
 ///
@@ -188,6 +191,28 @@ impl RbDoc {
         })
     }
 
+    /// True if the doc holds un-integrable pending structs or a pending delete
+    /// set — content that couldn't integrate because a causally-prior update is
+    /// missing. Such content is a recovery buffer, not document state; it heals if
+    /// the missing dependency later arrives. A pure read.
+    fn pending(&self) -> bool {
+        let doc = &self.0;
+        nogvl(move || has_pending(doc))
+    }
+
+    /// Like `encode_state_as_update` (full state), but **gap-free**: it excludes
+    /// any pending (un-integrable) structs and pending delete set. Use this when
+    /// persisting or serving state that other peers will apply — serving pending
+    /// content poisons their sync. Non-destructive: this doc keeps its pending, so
+    /// a genuine gap still heals if its dependency arrives. (`encode_state_as_update`
+    /// stays lossless for raw-update recovery.)
+    fn compacted_state_update(&self) -> Result<RString, Error> {
+        let doc = &self.0;
+        let update = nogvl(move || integrated_update(doc, &yrs::StateVector::default()))
+            .map_err(yrb_error)?;
+        Ok(binary_string(&update))
+    }
+
     /// Encode state as update (optionally diffed against a state vector)
     fn encode_state_as_update(&self, args: &[Value]) -> Result<RString, Error> {
         let sv_bytes: Option<Vec<u8>> = if args.is_empty() {
@@ -263,9 +288,13 @@ impl RbDoc {
                 match msg {
                     Message::Sync(sync_msg) => match sync_msg {
                         SyncMessage::SyncStep1(sv) => {
-                            // Respond with SyncStep2
-                            let txn = doc.transact();
-                            let update = txn.encode_state_as_update_v1(&sv);
+                            // Respond with SyncStep2 carrying only *integrated*
+                            // state. Never hand a peer un-integrable pending
+                            // structs: the peer would park the same pending
+                            // forever and the state-vector/content mismatch drives
+                            // endless resync traffic. (integrated_update is a no-op
+                            // fast path when nothing is pending.)
+                            let update = integrated_update(doc, &sv)?;
                             let response = Message::Sync(SyncMessage::SyncStep2(update));
                             Ok((0, 0, response.encode_v1()))
                         }
@@ -366,6 +395,11 @@ fn init(ruby: &Ruby) -> Result<(), Error> {
     doc_class.define_method("read_text", method!(RbDoc::read_text, 1))?;
     doc_class.define_method("read_xml", method!(RbDoc::read_xml, 1))?;
     doc_class.define_method("read_map", method!(RbDoc::read_map, 1))?;
+    doc_class.define_method("pending?", method!(RbDoc::pending, 0))?;
+    doc_class.define_method(
+        "compacted_state_update",
+        method!(RbDoc::compacted_state_update, 0),
+    )?;
     doc_class.define_method("update_ready?", method!(RbDoc::update_ready, 1))?;
     doc_class.define_method("update_advances?", method!(RbDoc::update_advances, 1))?;
     doc_class.define_method("sync_step1", method!(RbDoc::sync_step1, 0))?;

data/ext/yrby/src/protocol.rs

@@ -8,7 +8,7 @@ use yrs::encoding::read::{Cursor, Read};
 use yrs::sync::protocol::MessageReader;
 use yrs::sync::{Message, SyncMessage};
 use yrs::updates::decoder::{Decode, DecoderV1};
-use yrs::{Doc, ReadTxn, Transact};
+use yrs::{Doc, ReadTxn, StateVector, Transact, Update, WriteTxn};
 
 /// Classify a frame: a non-zero code only for exactly one well-formed message
 /// that consumes the whole buffer (the codes are the match arms below).
@@ -77,26 +77,38 @@ pub(crate) fn update_is_ready(doc: &Doc, update_bytes: &[u8]) -> Result<bool, St
 }
 
 /// True if applying `update_bytes` would actually change `doc`, i.e. it carries
-/// content the doc doesn't already have. This lets the server make durable side
-/// effects exactly-once: a lost-ack retry re-sends an update the server already
-/// applied; that retry is causally ready (so `update_is_ready` is true) but must
-/// not re-run `on_change`.
+/// content (an insert, a format, or a deletion) the doc doesn't already have.
+/// This lets the server make durable side effects exactly-once: a lost-ack retry
+/// re-sends an update the server already applied; that retry is causally ready
+/// (so `update_is_ready` is true) but must not re-run `on_change`.
 ///
 /// We can't read the update's own state vector to decide this: yrs reports an
 /// empty state_vector() for a causally-pending diff (e.g. a resync delta whose
 /// structs depend on updates the doc has but the standalone update doesn't),
-/// which would look identical to a no-op. So measure the real effect: seed an
-/// independent probe with the doc's current state, apply the update there, and
-/// see whether the state vector grew. Deletes don't move the state vector, so we
-/// can't cheaply prove a delete-bearing update is a duplicate; we conservatively
-/// report it as advancing (record it). That can still double-record a pure-delete
-/// retry, but it never drops a real deletion, which is the safe direction.
-/// Assumes the update is already causally ready.
+/// which would look identical to a no-op. So measure the real effect on an
+/// independent probe seeded with the doc's current state (never mutating the real
+/// doc), then compare the probe before and after applying the update:
+///
+/// - **Insert/format-only updates** grow the probe's state vector, so comparing
+///   the state vector is enough — and cheaper than a full re-encode.
+/// - **Delete-bearing updates** don't move the state vector (a deletion tombstones
+///   an existing struct rather than adding one), so we compare the full encoded
+///   state, which carries the delete set. An already-applied pure-delete retry
+///   re-encodes byte-identically → false; a genuinely new deletion changes the
+///   delete set → true. This is exact but pays for two full encodes, so only
+///   delete-bearing frames — a minority — take that path.
+///
+/// Earlier this branch was conservative: any delete-bearing update returned true
+/// (record it), which double-recorded and re-broadcast pure-delete retries the
+/// server had already integrated. The exact comparison removes that duplication
+/// while still never dropping a real deletion. Assumes the update is already
+/// causally ready.
 pub(crate) fn update_advances_doc(doc: &Doc, update_bytes: &[u8]) -> Result<bool, String> {
     let update = yrs::Update::decode_v1(update_bytes).map_err(|e| e.to_string())?;
-    if !update.delete_set().is_empty() {
-        return Ok(true); // can't cheaply prove a delete is a duplicate; record it
-    }
+    let has_deletes = !update.delete_set().is_empty();
+
+    // Seed an independent probe with the doc's current state so we can measure the
+    // update's effect without mutating the real doc.
     let probe = Doc::new();
     let current = doc
         .transact()
@@ -105,30 +117,78 @@ pub(crate) fn update_advances_doc(doc: &Doc, update_bytes: &[u8]) -> Result<bool
         .transact_mut()
         .apply_update(yrs::Update::decode_v1(&current).map_err(|e| e.to_string())?)
         .map_err(|e| e.to_string())?;
-    let before = probe.transact().state_vector();
-    probe
-        .transact_mut()
-        .apply_update(update)
-        .map_err(|e| e.to_string())?;
-    let after = probe.transact().state_vector();
-    Ok(after != before)
+
+    if has_deletes {
+        // Deletes don't move the state vector; compare the full encoded state
+        // (which includes the delete set), before vs. after, on the same probe.
+        let before = probe
+            .transact()
+            .encode_state_as_update_v1(&yrs::StateVector::default());
+        probe
+            .transact_mut()
+            .apply_update(update)
+            .map_err(|e| e.to_string())?;
+        let after = probe
+            .transact()
+            .encode_state_as_update_v1(&yrs::StateVector::default());
+        Ok(before != after)
+    } else {
+        let before = probe.transact().state_vector();
+        probe
+            .transact_mut()
+            .apply_update(update)
+            .map_err(|e| e.to_string())?;
+        let after = probe.transact().state_vector();
+        Ok(before != after)
+    }
 }
 
-/// True if the doc holds pending structs or a pending delete set: blocks that
-/// couldn't integrate because a dependency is missing. Test-only: asserts the
-/// causal-chain parking behavior in the unit tests below.
-#[cfg(test)]
-pub(crate) fn doc_has_pending(doc: &Doc) -> bool {
+/// True if the doc holds un-integrable pending structs or a pending delete set:
+/// blocks that couldn't integrate because a causally-prior update is missing. A
+/// pure read; does not mutate.
+pub(crate) fn has_pending(doc: &Doc) -> bool {
     let txn = doc.transact();
     txn.store().pending_update().is_some() || txn.store().pending_ds().is_some()
 }
 
+/// Encode the doc's **integrated** state as a v1 update diffed against `sv`,
+/// excluding any pending (un-integrable) structs and pending delete set.
+///
+/// Pending blocks are a recovery buffer, not document state. Serving them across
+/// the sync boundary hands a peer content it can't integrate, so the peer parks
+/// the same pending forever and the state-vector/content mismatch drives endless
+/// resync traffic. `encode_state_as_update_v1` merges pending back in (see yrs
+/// `merge_pending_v1`), so to get a gap-free encode we rebuild the state into a
+/// throwaway doc and `prune_pending` there before re-encoding.
+///
+/// Non-destructive: the prune happens only on the throwaway copy; `doc` keeps its
+/// pending, so a genuine gap still heals if its missing dependency later arrives.
+pub(crate) fn integrated_update(doc: &Doc, sv: &StateVector) -> Result<Vec<u8>, String> {
+    // Fast path: with nothing pending the direct encode is already gap-free, so
+    // the clean common case keeps the zero-copy behavior.
+    if !has_pending(doc) {
+        return Ok(doc.transact().encode_state_as_update_v1(sv));
+    }
+    let full = doc
+        .transact()
+        .encode_state_as_update_v1(&StateVector::default());
+    let clean = Doc::new();
+    {
+        let mut txn = clean.transact_mut();
+        txn.apply_update(Update::decode_v1(&full).map_err(|e| e.to_string())?)
+            .map_err(|e| e.to_string())?;
+        txn.prune_pending();
+    }
+    let out = clean.transact().encode_state_as_update_v1(sv);
+    Ok(out)
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
     use yrs::sync::Awareness;
     use yrs::updates::encoder::Encode;
-    use yrs::Text;
+    use yrs::{GetString, Text};
 
     fn text_update(content: &str) -> Vec<u8> {
         let doc = Doc::new();
@@ -246,6 +306,87 @@ mod tests {
         );
     }
 
+    #[test]
+    fn update_advances_is_exact_for_pure_delete_retries() {
+        // Build "hello", snapshot the pre-delete content, then delete a char and
+        // capture just that deletion as a diff (only a delete set, no new structs).
+        let doc = Doc::new();
+        let text = doc.get_or_insert_text("content");
+        text.insert(&mut doc.transact_mut(), 0, "hello");
+        let content_state = doc
+            .transact()
+            .encode_state_as_update_v1(&yrs::StateVector::default());
+        let sv_before = doc.transact().state_vector();
+        text.remove_range(&mut doc.transact_mut(), 0, 1); // delete "h"
+        let delete = doc.transact().encode_state_as_update_v1(&sv_before);
+
+        assert!(
+            !yrs::Update::decode_v1(&delete)
+                .unwrap()
+                .delete_set()
+                .is_empty(),
+            "the diff carries a delete set"
+        );
+
+        // A server holding the pre-delete content, but not the deletion yet.
+        let server = Doc::new();
+        server
+            .transact_mut()
+            .apply_update(yrs::Update::decode_v1(&content_state).unwrap())
+            .unwrap();
+
+        // The deletion is new: it advances (must be recorded).
+        assert!(
+            update_advances_doc(&server, &delete).unwrap(),
+            "a not-yet-applied deletion advances the doc"
+        );
+
+        // Apply it; now the byte-identical pure-delete retry must NOT advance.
+        // (This is the behavior change: it used to conservatively return true.)
+        server
+            .transact_mut()
+            .apply_update(yrs::Update::decode_v1(&delete).unwrap())
+            .unwrap();
+        assert!(
+            !update_advances_doc(&server, &delete).unwrap(),
+            "an already-applied pure-delete retry does not advance"
+        );
+    }
+
+    #[test]
+    fn update_advances_for_a_delete_bundled_with_new_content() {
+        // A delete-bearing update that ALSO carries a new struct still advances,
+        // even after the pure-delete part would be a no-op on its own.
+        let doc = Doc::new();
+        let text = doc.get_or_insert_text("content");
+        text.insert(&mut doc.transact_mut(), 0, "hello");
+        let content_state = doc
+            .transact()
+            .encode_state_as_update_v1(&yrs::StateVector::default());
+        let sv_before = doc.transact().state_vector();
+        text.remove_range(&mut doc.transact_mut(), 0, 1); // delete "h"
+        text.insert(&mut doc.transact_mut(), 4, "!"); // and add "!"
+        let mixed = doc.transact().encode_state_as_update_v1(&sv_before);
+
+        let server = Doc::new();
+        server
+            .transact_mut()
+            .apply_update(yrs::Update::decode_v1(&content_state).unwrap())
+            .unwrap();
+        assert!(
+            update_advances_doc(&server, &mixed).unwrap(),
+            "an insert+delete update advances"
+        );
+        server
+            .transact_mut()
+            .apply_update(yrs::Update::decode_v1(&mixed).unwrap())
+            .unwrap();
+        assert!(
+            !update_advances_doc(&server, &mixed).unwrap(),
+            "its byte-identical retry does not advance"
+        );
+    }
+
     #[test]
     fn merged_doc_update_extracts_and_skips_no_ops() {
         // A document update yields a delta that reconstructs the content.
@@ -308,24 +449,213 @@ mod tests {
             !update_is_ready(&doc, u3).unwrap(),
             "u3 depends on the missing u2"
         );
-        assert!(
-            !doc_has_pending(&doc),
-            "nothing pending until u3 is applied"
-        );
+        assert!(!has_pending(&doc), "nothing pending until u3 is applied");
 
         // Applying u3 anyway parks it as a pending struct.
         doc.transact_mut()
             .apply_update(yrs::Update::decode_v1(u3).unwrap())
             .unwrap();
-        assert!(
-            doc_has_pending(&doc),
-            "u3 is pending: its parent u2 is missing"
-        );
+        assert!(has_pending(&doc), "u3 is pending: its parent u2 is missing");
 
         // Once u2 arrives (via resync), u3 integrates and pending clears.
         doc.transact_mut()
             .apply_update(yrs::Update::decode_v1(u2).unwrap())
             .unwrap();
-        assert!(!doc_has_pending(&doc), "u2 arrived; u3 integrated");
+        assert!(!has_pending(&doc), "u2 arrived; u3 integrated");
+    }
+
+    // Build a causal gap: `first` inserts "a", `dependent` inserts "b" after it,
+    // so `dependent` alone parks as pending on a doc that lacks `first`.
+    fn gap_pair() -> (Vec<u8>, Vec<u8>) {
+        let src = Doc::new();
+        let txt = src.get_or_insert_text("notepad");
+        txt.insert(&mut src.transact_mut(), 0, "a");
+        let first = src
+            .transact()
+            .encode_state_as_update_v1(&yrs::StateVector::default());
+        let sv = src.transact().state_vector();
+        txt.insert(&mut src.transact_mut(), 1, "b");
+        let dependent = src.transact().encode_state_as_update_v1(&sv);
+        (first, dependent)
+    }
+
+    #[test]
+    fn integrated_update_strips_pending_and_is_non_destructive() {
+        let (_first, dependent) = gap_pair();
+        let doc = Doc::new();
+        doc.transact_mut()
+            .apply_update(yrs::Update::decode_v1(&dependent).unwrap())
+            .unwrap();
+        assert!(has_pending(&doc), "the gappy update parked as pending");
+
+        // encode_state_as_update carries the pending; integrated_update does not.
+        let full = doc
+            .transact()
+            .encode_state_as_update_v1(&yrs::StateVector::default());
+        let gap_free = integrated_update(&doc, &yrs::StateVector::default()).unwrap();
+        assert_ne!(full, gap_free, "integrated_update drops the pending bytes");
+
+        // Applying the gap-free encode to a fresh peer must NOT poison it.
+        let peer = Doc::new();
+        peer.transact_mut()
+            .apply_update(yrs::Update::decode_v1(&gap_free).unwrap())
+            .unwrap();
+        assert!(
+            !has_pending(&peer),
+            "peer got no pending from the gap-free state"
+        );
+
+        // Non-destructive: the source doc keeps its pending (so it can still heal).
+        assert!(
+            has_pending(&doc),
+            "integrated_update did not mutate the source"
+        );
+    }
+
+    #[test]
+    fn integrated_update_fast_path_matches_direct_encode_when_clean() {
+        // No pending -> byte-identical to encode_state_as_update (zero-copy path).
+        let (first, _dependent) = gap_pair();
+        let doc = Doc::new();
+        doc.transact_mut()
+            .apply_update(yrs::Update::decode_v1(&first).unwrap())
+            .unwrap();
+        assert!(!has_pending(&doc));
+        let direct = doc
+            .transact()
+            .encode_state_as_update_v1(&yrs::StateVector::default());
+        let via = integrated_update(&doc, &yrs::StateVector::default()).unwrap();
+        assert_eq!(direct, via);
+    }
+
+    #[test]
+    fn a_healed_gap_serves_its_content() {
+        // After the missing dependency arrives, the (formerly pending) content is
+        // integrated and integrated_update includes it.
+        let (first, dependent) = gap_pair();
+        let doc = Doc::new();
+        doc.transact_mut()
+            .apply_update(yrs::Update::decode_v1(&dependent).unwrap())
+            .unwrap();
+        doc.transact_mut()
+            .apply_update(yrs::Update::decode_v1(&first).unwrap())
+            .unwrap();
+        assert!(!has_pending(&doc), "gap healed once first arrived");
+        let gap_free = integrated_update(&doc, &yrs::StateVector::default()).unwrap();
+        let peer = Doc::new();
+        peer.transact_mut()
+            .apply_update(yrs::Update::decode_v1(&gap_free).unwrap())
+            .unwrap();
+        assert_eq!(
+            peer.get_or_insert_text("notepad")
+                .get_string(&peer.transact()),
+            "ab"
+        );
+    }
+
+    // A gappy insert from its own independent client: inserts two chars and
+    // returns only the second delta, which depends on the (missing) first.
+    fn independent_gappy_insert() -> Vec<u8> {
+        let src = Doc::new();
+        let txt = src.get_or_insert_text("notepad");
+        txt.insert(&mut src.transact_mut(), 0, "x");
+        let sv = src.transact().state_vector();
+        txt.insert(&mut src.transact_mut(), 1, "y");
+        let txn = src.transact();
+        txn.encode_state_as_update_v1(&sv)
+    }
+
+    #[test]
+    fn integrated_update_keeps_content_and_drops_pending_when_mixed() {
+        // The realistic case: a doc with real integrated content AND a pending
+        // struct. Pruning must keep the content and drop only the pending.
+        let (first, _dep) = gap_pair();
+        let doc = Doc::new();
+        doc.transact_mut()
+            .apply_update(yrs::Update::decode_v1(&first).unwrap())
+            .unwrap(); // integrated "a"
+        doc.transact_mut()
+            .apply_update(yrs::Update::decode_v1(&independent_gappy_insert()).unwrap())
+            .unwrap(); // + a pending struct from another client
+        assert!(has_pending(&doc));
+
+        let gap_free = integrated_update(&doc, &yrs::StateVector::default()).unwrap();
+        let peer = Doc::new();
+        peer.transact_mut()
+            .apply_update(yrs::Update::decode_v1(&gap_free).unwrap())
+            .unwrap();
+        assert_eq!(
+            peer.get_or_insert_text("notepad")
+                .get_string(&peer.transact()),
+            "a",
+            "kept the integrated content"
+        );
+        assert!(!has_pending(&peer), "dropped the pending");
+    }
+
+    #[test]
+    fn integrated_update_diffs_against_a_peer_sv_and_excludes_pending() {
+        // The production signature: `handle_sync_message` calls
+        // integrated_update(doc, peer_sv). A peer already holding the integrated
+        // content should get a diff carrying no new content and no pending.
+        let (first, _dep) = gap_pair();
+        let server = Doc::new();
+        server
+            .transact_mut()
+            .apply_update(yrs::Update::decode_v1(&first).unwrap())
+            .unwrap();
+        server
+            .transact_mut()
+            .apply_update(yrs::Update::decode_v1(&independent_gappy_insert()).unwrap())
+            .unwrap();
+
+        let peer = Doc::new();
+        peer.transact_mut()
+            .apply_update(yrs::Update::decode_v1(&first).unwrap())
+            .unwrap();
+        let peer_sv = peer.transact().state_vector();
+
+        let diff = integrated_update(&server, &peer_sv).unwrap();
+        peer.transact_mut()
+            .apply_update(yrs::Update::decode_v1(&diff).unwrap())
+            .unwrap();
+        assert_eq!(
+            peer.get_or_insert_text("notepad")
+                .get_string(&peer.transact()),
+            "a"
+        );
+        assert!(!has_pending(&peer), "the diff carried no pending");
+    }
+
+    #[test]
+    fn integrated_update_strips_a_pending_delete_set() {
+        // A deletion whose target struct is absent parks as a pending *delete
+        // set* -- the delete-side counterpart to a pending struct.
+        let src = Doc::new();
+        let txt = src.get_or_insert_text("notepad");
+        txt.insert(&mut src.transact_mut(), 0, "z");
+        let sv = src.transact().state_vector();
+        txt.remove_range(&mut src.transact_mut(), 0, 1);
+        let deletion = src.transact().encode_state_as_update_v1(&sv); // delete-only
+
+        let doc = Doc::new();
+        doc.transact_mut()
+            .apply_update(yrs::Update::decode_v1(&deletion).unwrap())
+            .unwrap();
+        assert!(
+            has_pending(&doc),
+            "the orphan deletion parked as a pending delete set"
+        );
+
+        let gap_free = integrated_update(&doc, &yrs::StateVector::default()).unwrap();
+        let peer = Doc::new();
+        peer.transact_mut()
+            .apply_update(yrs::Update::decode_v1(&gap_free).unwrap())
+            .unwrap();
+        assert!(!has_pending(&peer), "the pending delete set was not served");
+        assert!(
+            has_pending(&doc),
+            "non-destructive: source keeps its pending"
+        );
     }
 }

data/ext/yrby/src/read.rs

@@ -4,29 +4,99 @@
 
 use std::collections::HashMap;
 use std::sync::Arc;
-use yrs::{Any, Array, GetString, Map, MapRef, Out, ReadTxn, XmlFragment, XmlFragmentRef, XmlOut};
+use yrs::types::text::YChange;
+use yrs::{
+    Any, Array, GetString, Map, MapRef, Out, ReadTxn, Text, Xml, XmlFragment, XmlFragmentRef,
+    XmlOut, XmlTextRef,
+};
 
 /// Read an XML-shaped root as text, one top-level block per line.
 ///
-/// ProseMirror stores blocks as `Y.XmlElement` children (`<paragraph>…`);
-/// Lexical stores each block as a sibling `Y.XmlText` (its node metadata is an
-/// embed, which yrs omits from the string). We serialize each top-level child and
-/// join with "\n", so adjacent blocks don't merge into one run of words. Without
-/// the separator, Lexical — whose blocks carry no element tags — would glue
-/// paragraphs together (e.g. "first paragraphsecond paragraph"), breaking word
-/// boundaries for search/preview. Element tags are kept (the caller strips them);
-/// deeper nesting is flattened, but its inner tags still separate words after
-/// stripping.
+/// Two editors store their documents differently, and both are handled:
+///
+/// - **ProseMirror** (Tiptap) stores blocks as `Y.XmlElement` children
+///   (`<paragraph>…`). `get_string` already recurses these (tags included; the
+///   caller strips them), so we keep that path.
+/// - **Lexical** (Lexxy) stores every node as a `Y.XmlText`, and nests child
+///   blocks (list items, table cells, nested lists) as *embedded* `Y.XmlText`s —
+///   which `get_string` silently omits, dropping all that content. So for a
+///   Lexical block we walk its content (`Text::diff`) instead: text runs build a
+///   line, inline children (links) join it, and nested block children flush the
+///   line and recurse. Each leaf block becomes one line, so words never glue
+///   across blocks and lists/tables come through intact.
 pub fn xml_blocks_text<T: ReadTxn>(txn: &T, fragment: &XmlFragmentRef) -> String {
-    fragment
-        .children(txn)
-        .map(|node| match node {
-            XmlOut::Element(e) => e.get_string(txn),
-            XmlOut::Text(t) => t.get_string(txn),
-            XmlOut::Fragment(f) => f.get_string(txn),
-        })
-        .collect::<Vec<_>>()
-        .join("\n")
+    let mut out: Vec<String> = Vec::new();
+    for node in fragment.children(txn) {
+        match node {
+            XmlOut::Text(t) => walk_lexical_block(txn, &t, &mut out),
+            XmlOut::Element(e) => {
+                // ProseMirror blocks have a tag but no `__type`; get_string recurses
+                // them (tags kept, caller strips). A Lexical decorator (horizontal
+                // rule, image) is an XmlElement *with* a `__type` and no extractable
+                // text -- skip it rather than emit its `<UNDEFINED …>` serialization.
+                if e.get_attribute(txn, "__type").is_none() {
+                    out.push(e.get_string(txn));
+                }
+            }
+            XmlOut::Fragment(f) => out.push(f.get_string(txn)),
+        }
+    }
+    out.join("\n")
+}
+
+/// Lexical node `__type`s whose text belongs on the surrounding line rather than
+/// a new block (e.g. a link inside a paragraph). Everything else with embedded
+/// child `Y.XmlText`s is treated as a block and recursed.
+fn is_inline_lexical_type(ty: &str) -> bool {
+    matches!(
+        ty,
+        "text" | "link" | "autolink" | "linebreak" | "tab" | "hashtag" | "mark" | "overflow"
+    )
+}
+
+/// A Lexical node's `__type` (stored as an XML attribute on its `Y.XmlText`).
+fn lexical_type<T: ReadTxn>(txn: &T, t: &XmlTextRef) -> String {
+    match t.get_attribute(txn, "__type") {
+        Some(Out::Any(Any::String(s))) => s.to_string(),
+        _ => String::new(),
+    }
+}
+
+/// Gather the text of an inline Lexical element (its text runs and any nested
+/// inline elements) without introducing block breaks.
+fn inline_lexical_text<T: ReadTxn>(txn: &T, t: &XmlTextRef, buf: &mut String) {
+    for d in t.diff(txn, YChange::identity) {
+        match d.insert {
+            Out::Any(Any::String(s)) => buf.push_str(&s),
+            Out::YXmlText(child) => inline_lexical_text(txn, &child, buf),
+            _ => {} // per-text-node metadata map, decorator embeds: no text
+        }
+    }
+}
+
+/// Walk a Lexical block (`Y.XmlText`), pushing one line per leaf block. Text runs
+/// accumulate; inline children join the line; block children flush it and recurse.
+fn walk_lexical_block<T: ReadTxn>(txn: &T, t: &XmlTextRef, out: &mut Vec<String>) {
+    let mut line = String::new();
+    for d in t.diff(txn, YChange::identity) {
+        match d.insert {
+            Out::Any(Any::String(s)) => line.push_str(&s),
+            Out::YXmlText(child) => {
+                if is_inline_lexical_type(&lexical_type(txn, &child)) {
+                    inline_lexical_text(txn, &child, &mut line);
+                } else {
+                    if !line.is_empty() {
+                        out.push(std::mem::take(&mut line));
+                    }
+                    walk_lexical_block(txn, &child, out);
+                }
+            }
+            _ => {} // per-text-node metadata map; embeds we don't read for text
+        }
+    }
+    if !line.is_empty() {
+        out.push(line);
+    }
 }
 
 /// Read a `Y.Map` root as a JSON object string (keys sorted for stable output).
@@ -185,4 +255,49 @@ mod tests {
         let txn = doc.transact();
         assert_eq!(map_json(&txn, &map), "{}");
     }
+
+    #[test]
+    fn lexical_complex_doc_extracts_all_nested_text() {
+        // A real Lexxy/Lexical doc with every block type: headings, formatted
+        // text, an inline link, bullet + NESTED bullet + numbered + check lists,
+        // a quote, a code block, a horizontal rule, and a table. Every piece of
+        // text -- including list items, the nested sub-list, and table cells --
+        // must come through (get_string alone dropped all the nested ones).
+        use yrs::updates::decoder::Decode;
+        use yrs::{Transact, Update};
+        let bytes = include_bytes!("fixtures/lexical_rich.bin");
+        let doc = Doc::new();
+        {
+            let mut txn = doc.transact_mut();
+            txn.apply_update(Update::decode_v1(bytes).unwrap()).unwrap();
+        }
+        let txn = doc.transact();
+        let frag = txn.get_xml_fragment("root").unwrap();
+        let text = xml_blocks_text(&txn, &frag);
+        for expected in [
+            "Heading One",
+            "Heading Two",
+            "Plain, bold, italic, strike, underline, and code.",
+            "Visit the website for more.", // link text stays inline
+            "First bullet",
+            "Second bullet",
+            "Nested A", // nested sub-list
+            "Nested B",
+            "Step one",
+            "Step two",
+            "Done item",
+            "Todo item",
+            "A blockquote about CRDTs.",
+            "const x = 1;", // code block (keeps its internal newline)
+            "console.log(x);",
+            "Name", // table header cells
+            "Role",
+            "Ada", // table body cells
+            "Engineer",
+        ] {
+            assert!(text.contains(expected), "missing {expected:?} in:\n{text}");
+        }
+        // The inline link must NOT have been split onto its own line.
+        assert!(text.contains("Visit the website for more."));
+    }
 }

data/lib/y/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Y
-  VERSION = "0.2.1"
+  VERSION = "0.3.0"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: yrby
 version: !ruby/object:Gem::Version
-  version: 0.2.1
+  version: 0.3.0
 platform: ruby
 authors:
 - JP Camara