yrby 0.2.1 → 0.2.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d6580b564ad0ffb8573b7e5f81acdba9ad4b59648d4c2b25c5914d144cef0623
-  data.tar.gz: 1ee082cbd115166217d4ac48a62e32d2d0f161564d02644dfe3a23731822ca87
+  metadata.gz: a5928987b16a5d3a3933b95875dff276e4e18364b46ed0e33baa8e48b5b2e847
+  data.tar.gz: 2676689ddee9403765353c4612055fb120ce9e1c6dcdbeb956c482eed313374d
 SHA512:
-  metadata.gz: da995eb5f169b62d3a83bfe138be28894b66fd9048a091efb624019ed6e87b520429c8ea29df17eef5ad55929e4b7680190fb84fad06b1f952af72c0bec300d7
-  data.tar.gz: a6bd21c7f3d353037fd877b0246bca1221bdd894bd7dc4f1a811fdd74fcbae26dc7a0d087d9bba13c79cbfb8134e4c588a2493e527a57ed31d6e1faed3504e8a
+  metadata.gz: 7e13256e4881804c3330bf69ce86203bd2aea4bda9313db042f7d368221130f6624285cbddea1a9b61b967c3831635ace45b1291496100872ad8a93c97be2e10
+  data.tar.gz: fbf1c8c9ec0f0b23190578d562f75fb14a14ec3d562b54b86437c5abda5bd48a97f2da216927f66fde46b5d52fa12d38b4f39b8073b8c6222d656fd09fd2c24f

data/CHANGELOG.md

@@ -6,6 +6,44 @@ to follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+## [0.2.3] - 2026-07-01
+
+### Fixed
+
+- `Doc#update_advances?` is now exact for **delete-bearing** updates, so an
+  already-applied pure-delete retry no longer reports as advancing. Previously any
+  update carrying a delete set returned `true` (record it) because deletes don't
+  move the state vector, so the cheap state-vector probe couldn't prove a
+  duplicate. A lost-ack retry of a deletion the server had already integrated was
+  therefore re-recorded and re-broadcast every time. For delete-bearing updates we
+  now compare the full encoded document state (which includes the delete set)
+  before vs. after a trial apply on an isolated probe: a genuinely new deletion
+  changes it (`true`); an already-applied retry re-encodes identically (`false`).
+  Insert/format-only updates keep the cheaper state-vector path, so only
+  delete-bearing frames — a minority — pay for the exact comparison. The exactly-
+  once guarantee is unchanged in the safe direction: a real deletion is never
+  dropped.
+
+  This lets `yrby-actioncable` (and any caller gating `on_change` on
+  `update_advances?`) settle a duplicate pure-delete frame as `:applied` — acked,
+  but not stored or relayed — so apps no longer need an app-level
+  encode-and-compare guard around their durable writes.
+
+## [0.2.2] - 2026-06-30
+
+### Fixed
+
+- `Doc#read_xml` now recovers text from **nested** Lexical/Lexxy blocks. Lexical
+  embeds child blocks (list items, table cells, nested lists) as `Y.XmlText`
+  embeds that `get_string` silently drops, so lists and tables previously came
+  back empty. `read_xml` now walks the embeds: text runs build a line, inline
+  children (links) join it, and nested block children flush and recurse — so a
+  document with headings, formatted text, links, bullet/numbered/check/nested
+  lists, blockquotes, code blocks and tables extracts every piece of text.
+  Lexical decorator elements (horizontal rule, image) are skipped instead of
+  emitting their `<UNDEFINED …>` serialization. ProseMirror handling is
+  unchanged.
+
 ## [0.2.1] - 2026-06-29
 
 ### Changed
@@ -15,15 +53,9 @@ to follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [0.2.0] - 2026-06-28
 
-First release under the **`yrby`** name (the project was previously developed
-as `yrb-lite`). The public Ruby interface is the top-level module **`Y`** —
-mirroring the `y-rb` gem's `Y::Doc` interface.
-
-### Changed
-- **Renamed `yrb-lite` → `yrby`.** Module `YrbLite` → top-level `Y`
-  (`Y::Doc`, `Y::Error`, `Y::VERSION`). Require path `require "yrb_lite"` →
-  `require "y"`. (The native extension crate shipped as `y_ruby` in 0.2.0; see
-  0.2.1 for its rename to `yrby`.)
+First release. The public Ruby interface is the top-level module **`Y`**
+(`Y::Doc`, `Y::Error`, `Y::VERSION`), loaded with `require "y"` — mirroring the
+`y-rb` gem's `Y::Doc` interface.
 
 ### Added
 - Native `Doc#read_text` and `Doc#read_map` readers — reconstruct plain text and

data/README.md

@@ -41,23 +41,25 @@ npm install yrby-client
   one-include ActionCable concern.
 - Authoritative record-before-distribute semantics: each document change can be
   recorded durably before it goes out to anyone.
-- 
+- Optional server-side reads: `Doc#read_text` and `Doc#read_map` reconstruct a
+  document's contents in Ruby - no Node process - for search, exports, validation,
+  or server-side rendering.
 
-## Why "lite"
+## Scope
 
-The "lite" is the size of the surface. `yrby` binds just the part of `y-crdt` you
-need to *sync and persist* collaborative documents - a `Doc`, awareness, and the
-y-websocket protocol primitives. The Ruby side treats a document as opaque CRDT
-state: it applies updates, answers sync handshakes, and records deltas, but never
-reaches in to read or edit the contents. The browser editor owns the document's
-shape.
+`yrby` binds just the part of `y-crdt` you need to *sync and persist* collaborative
+documents - a `Doc`, awareness, and the y-websocket protocol primitives. By default
+the Ruby side treats a document as opaque CRDT state: it applies updates, answers
+sync handshakes, and records deltas without reaching into the contents - the browser
+editor owns the document's shape. When you do need to look inside, `Doc#read_text`
+and `Doc#read_map` reconstruct it server-side, in Ruby.
 
-## What isn't "lite"
+## Durability and delivery
 
-The surface area may be "lite", but a core focus is on durability, resiliency, delivery
+The surface is intentionally small, but the focus is durability, resiliency, delivery
 guarantees, correctness, and thread safety.
 
-Towards that goal, `yrby` adds capabilities that may even stand out in the Yjs ecosystem:
+Towards that goal, `yrby` adds capabilities that stand out even in the Yjs ecosystem:
 
 - Built-in update acknowledgement: the `ActionCableProvider` in `yrby-client` will continue to
   send updates until an ack is received from the server. [`yrby-actioncable`](https://rubygems.org/gems/yrby-actioncable)

data/ext/yrby/src/protocol.rs

@@ -77,26 +77,38 @@ pub(crate) fn update_is_ready(doc: &Doc, update_bytes: &[u8]) -> Result<bool, St
 }
 
 /// True if applying `update_bytes` would actually change `doc`, i.e. it carries
-/// content the doc doesn't already have. This lets the server make durable side
-/// effects exactly-once: a lost-ack retry re-sends an update the server already
-/// applied; that retry is causally ready (so `update_is_ready` is true) but must
-/// not re-run `on_change`.
+/// content (an insert, a format, or a deletion) the doc doesn't already have.
+/// This lets the server make durable side effects exactly-once: a lost-ack retry
+/// re-sends an update the server already applied; that retry is causally ready
+/// (so `update_is_ready` is true) but must not re-run `on_change`.
 ///
 /// We can't read the update's own state vector to decide this: yrs reports an
 /// empty state_vector() for a causally-pending diff (e.g. a resync delta whose
 /// structs depend on updates the doc has but the standalone update doesn't),
-/// which would look identical to a no-op. So measure the real effect: seed an
-/// independent probe with the doc's current state, apply the update there, and
-/// see whether the state vector grew. Deletes don't move the state vector, so we
-/// can't cheaply prove a delete-bearing update is a duplicate; we conservatively
-/// report it as advancing (record it). That can still double-record a pure-delete
-/// retry, but it never drops a real deletion, which is the safe direction.
-/// Assumes the update is already causally ready.
+/// which would look identical to a no-op. So measure the real effect on an
+/// independent probe seeded with the doc's current state (never mutating the real
+/// doc), then compare the probe before and after applying the update:
+///
+/// - **Insert/format-only updates** grow the probe's state vector, so comparing
+///   the state vector is enough — and cheaper than a full re-encode.
+/// - **Delete-bearing updates** don't move the state vector (a deletion tombstones
+///   an existing struct rather than adding one), so we compare the full encoded
+///   state, which carries the delete set. An already-applied pure-delete retry
+///   re-encodes byte-identically → false; a genuinely new deletion changes the
+///   delete set → true. This is exact but pays for two full encodes, so only
+///   delete-bearing frames — a minority — take that path.
+///
+/// Earlier this branch was conservative: any delete-bearing update returned true
+/// (record it), which double-recorded and re-broadcast pure-delete retries the
+/// server had already integrated. The exact comparison removes that duplication
+/// while still never dropping a real deletion. Assumes the update is already
+/// causally ready.
 pub(crate) fn update_advances_doc(doc: &Doc, update_bytes: &[u8]) -> Result<bool, String> {
     let update = yrs::Update::decode_v1(update_bytes).map_err(|e| e.to_string())?;
-    if !update.delete_set().is_empty() {
-        return Ok(true); // can't cheaply prove a delete is a duplicate; record it
-    }
+    let has_deletes = !update.delete_set().is_empty();
+
+    // Seed an independent probe with the doc's current state so we can measure the
+    // update's effect without mutating the real doc.
     let probe = Doc::new();
     let current = doc
         .transact()
@@ -105,13 +117,30 @@ pub(crate) fn update_advances_doc(doc: &Doc, update_bytes: &[u8]) -> Result<bool
         .transact_mut()
         .apply_update(yrs::Update::decode_v1(&current).map_err(|e| e.to_string())?)
         .map_err(|e| e.to_string())?;
-    let before = probe.transact().state_vector();
-    probe
-        .transact_mut()
-        .apply_update(update)
-        .map_err(|e| e.to_string())?;
-    let after = probe.transact().state_vector();
-    Ok(after != before)
+
+    if has_deletes {
+        // Deletes don't move the state vector; compare the full encoded state
+        // (which includes the delete set), before vs. after, on the same probe.
+        let before = probe
+            .transact()
+            .encode_state_as_update_v1(&yrs::StateVector::default());
+        probe
+            .transact_mut()
+            .apply_update(update)
+            .map_err(|e| e.to_string())?;
+        let after = probe
+            .transact()
+            .encode_state_as_update_v1(&yrs::StateVector::default());
+        Ok(before != after)
+    } else {
+        let before = probe.transact().state_vector();
+        probe
+            .transact_mut()
+            .apply_update(update)
+            .map_err(|e| e.to_string())?;
+        let after = probe.transact().state_vector();
+        Ok(before != after)
+    }
 }
 
 /// True if the doc holds pending structs or a pending delete set: blocks that
@@ -246,6 +275,87 @@ mod tests {
         );
     }
 
+    #[test]
+    fn update_advances_is_exact_for_pure_delete_retries() {
+        // Build "hello", snapshot the pre-delete content, then delete a char and
+        // capture just that deletion as a diff (only a delete set, no new structs).
+        let doc = Doc::new();
+        let text = doc.get_or_insert_text("content");
+        text.insert(&mut doc.transact_mut(), 0, "hello");
+        let content_state = doc
+            .transact()
+            .encode_state_as_update_v1(&yrs::StateVector::default());
+        let sv_before = doc.transact().state_vector();
+        text.remove_range(&mut doc.transact_mut(), 0, 1); // delete "h"
+        let delete = doc.transact().encode_state_as_update_v1(&sv_before);
+
+        assert!(
+            !yrs::Update::decode_v1(&delete)
+                .unwrap()
+                .delete_set()
+                .is_empty(),
+            "the diff carries a delete set"
+        );
+
+        // A server holding the pre-delete content, but not the deletion yet.
+        let server = Doc::new();
+        server
+            .transact_mut()
+            .apply_update(yrs::Update::decode_v1(&content_state).unwrap())
+            .unwrap();
+
+        // The deletion is new: it advances (must be recorded).
+        assert!(
+            update_advances_doc(&server, &delete).unwrap(),
+            "a not-yet-applied deletion advances the doc"
+        );
+
+        // Apply it; now the byte-identical pure-delete retry must NOT advance.
+        // (This is the behavior change: it used to conservatively return true.)
+        server
+            .transact_mut()
+            .apply_update(yrs::Update::decode_v1(&delete).unwrap())
+            .unwrap();
+        assert!(
+            !update_advances_doc(&server, &delete).unwrap(),
+            "an already-applied pure-delete retry does not advance"
+        );
+    }
+
+    #[test]
+    fn update_advances_for_a_delete_bundled_with_new_content() {
+        // A delete-bearing update that ALSO carries a new struct still advances,
+        // even after the pure-delete part would be a no-op on its own.
+        let doc = Doc::new();
+        let text = doc.get_or_insert_text("content");
+        text.insert(&mut doc.transact_mut(), 0, "hello");
+        let content_state = doc
+            .transact()
+            .encode_state_as_update_v1(&yrs::StateVector::default());
+        let sv_before = doc.transact().state_vector();
+        text.remove_range(&mut doc.transact_mut(), 0, 1); // delete "h"
+        text.insert(&mut doc.transact_mut(), 4, "!"); // and add "!"
+        let mixed = doc.transact().encode_state_as_update_v1(&sv_before);
+
+        let server = Doc::new();
+        server
+            .transact_mut()
+            .apply_update(yrs::Update::decode_v1(&content_state).unwrap())
+            .unwrap();
+        assert!(
+            update_advances_doc(&server, &mixed).unwrap(),
+            "an insert+delete update advances"
+        );
+        server
+            .transact_mut()
+            .apply_update(yrs::Update::decode_v1(&mixed).unwrap())
+            .unwrap();
+        assert!(
+            !update_advances_doc(&server, &mixed).unwrap(),
+            "its byte-identical retry does not advance"
+        );
+    }
+
     #[test]
     fn merged_doc_update_extracts_and_skips_no_ops() {
         // A document update yields a delta that reconstructs the content.

data/ext/yrby/src/read.rs

@@ -4,29 +4,99 @@
 
 use std::collections::HashMap;
 use std::sync::Arc;
-use yrs::{Any, Array, GetString, Map, MapRef, Out, ReadTxn, XmlFragment, XmlFragmentRef, XmlOut};
+use yrs::types::text::YChange;
+use yrs::{
+    Any, Array, GetString, Map, MapRef, Out, ReadTxn, Text, Xml, XmlFragment, XmlFragmentRef,
+    XmlOut, XmlTextRef,
+};
 
 /// Read an XML-shaped root as text, one top-level block per line.
 ///
-/// ProseMirror stores blocks as `Y.XmlElement` children (`<paragraph>…`);
-/// Lexical stores each block as a sibling `Y.XmlText` (its node metadata is an
-/// embed, which yrs omits from the string). We serialize each top-level child and
-/// join with "\n", so adjacent blocks don't merge into one run of words. Without
-/// the separator, Lexical — whose blocks carry no element tags — would glue
-/// paragraphs together (e.g. "first paragraphsecond paragraph"), breaking word
-/// boundaries for search/preview. Element tags are kept (the caller strips them);
-/// deeper nesting is flattened, but its inner tags still separate words after
-/// stripping.
+/// Two editors store their documents differently, and both are handled:
+///
+/// - **ProseMirror** (Tiptap) stores blocks as `Y.XmlElement` children
+///   (`<paragraph>…`). `get_string` already recurses these (tags included; the
+///   caller strips them), so we keep that path.
+/// - **Lexical** (Lexxy) stores every node as a `Y.XmlText`, and nests child
+///   blocks (list items, table cells, nested lists) as *embedded* `Y.XmlText`s —
+///   which `get_string` silently omits, dropping all that content. So for a
+///   Lexical block we walk its content (`Text::diff`) instead: text runs build a
+///   line, inline children (links) join it, and nested block children flush the
+///   line and recurse. Each leaf block becomes one line, so words never glue
+///   across blocks and lists/tables come through intact.
 pub fn xml_blocks_text<T: ReadTxn>(txn: &T, fragment: &XmlFragmentRef) -> String {
-    fragment
-        .children(txn)
-        .map(|node| match node {
-            XmlOut::Element(e) => e.get_string(txn),
-            XmlOut::Text(t) => t.get_string(txn),
-            XmlOut::Fragment(f) => f.get_string(txn),
-        })
-        .collect::<Vec<_>>()
-        .join("\n")
+    let mut out: Vec<String> = Vec::new();
+    for node in fragment.children(txn) {
+        match node {
+            XmlOut::Text(t) => walk_lexical_block(txn, &t, &mut out),
+            XmlOut::Element(e) => {
+                // ProseMirror blocks have a tag but no `__type`; get_string recurses
+                // them (tags kept, caller strips). A Lexical decorator (horizontal
+                // rule, image) is an XmlElement *with* a `__type` and no extractable
+                // text -- skip it rather than emit its `<UNDEFINED …>` serialization.
+                if e.get_attribute(txn, "__type").is_none() {
+                    out.push(e.get_string(txn));
+                }
+            }
+            XmlOut::Fragment(f) => out.push(f.get_string(txn)),
+        }
+    }
+    out.join("\n")
+}
+
+/// Lexical node `__type`s whose text belongs on the surrounding line rather than
+/// a new block (e.g. a link inside a paragraph). Everything else with embedded
+/// child `Y.XmlText`s is treated as a block and recursed.
+fn is_inline_lexical_type(ty: &str) -> bool {
+    matches!(
+        ty,
+        "text" | "link" | "autolink" | "linebreak" | "tab" | "hashtag" | "mark" | "overflow"
+    )
+}
+
+/// A Lexical node's `__type` (stored as an XML attribute on its `Y.XmlText`).
+fn lexical_type<T: ReadTxn>(txn: &T, t: &XmlTextRef) -> String {
+    match t.get_attribute(txn, "__type") {
+        Some(Out::Any(Any::String(s))) => s.to_string(),
+        _ => String::new(),
+    }
+}
+
+/// Gather the text of an inline Lexical element (its text runs and any nested
+/// inline elements) without introducing block breaks.
+fn inline_lexical_text<T: ReadTxn>(txn: &T, t: &XmlTextRef, buf: &mut String) {
+    for d in t.diff(txn, YChange::identity) {
+        match d.insert {
+            Out::Any(Any::String(s)) => buf.push_str(&s),
+            Out::YXmlText(child) => inline_lexical_text(txn, &child, buf),
+            _ => {} // per-text-node metadata map, decorator embeds: no text
+        }
+    }
+}
+
+/// Walk a Lexical block (`Y.XmlText`), pushing one line per leaf block. Text runs
+/// accumulate; inline children join the line; block children flush it and recurse.
+fn walk_lexical_block<T: ReadTxn>(txn: &T, t: &XmlTextRef, out: &mut Vec<String>) {
+    let mut line = String::new();
+    for d in t.diff(txn, YChange::identity) {
+        match d.insert {
+            Out::Any(Any::String(s)) => line.push_str(&s),
+            Out::YXmlText(child) => {
+                if is_inline_lexical_type(&lexical_type(txn, &child)) {
+                    inline_lexical_text(txn, &child, &mut line);
+                } else {
+                    if !line.is_empty() {
+                        out.push(std::mem::take(&mut line));
+                    }
+                    walk_lexical_block(txn, &child, out);
+                }
+            }
+            _ => {} // per-text-node metadata map; embeds we don't read for text
+        }
+    }
+    if !line.is_empty() {
+        out.push(line);
+    }
 }
 
 /// Read a `Y.Map` root as a JSON object string (keys sorted for stable output).
@@ -185,4 +255,49 @@ mod tests {
         let txn = doc.transact();
         assert_eq!(map_json(&txn, &map), "{}");
     }
+
+    #[test]
+    fn lexical_complex_doc_extracts_all_nested_text() {
+        // A real Lexxy/Lexical doc with every block type: headings, formatted
+        // text, an inline link, bullet + NESTED bullet + numbered + check lists,
+        // a quote, a code block, a horizontal rule, and a table. Every piece of
+        // text -- including list items, the nested sub-list, and table cells --
+        // must come through (get_string alone dropped all the nested ones).
+        use yrs::updates::decoder::Decode;
+        use yrs::{Transact, Update};
+        let bytes = include_bytes!("fixtures/lexical_rich.bin");
+        let doc = Doc::new();
+        {
+            let mut txn = doc.transact_mut();
+            txn.apply_update(Update::decode_v1(bytes).unwrap()).unwrap();
+        }
+        let txn = doc.transact();
+        let frag = txn.get_xml_fragment("root").unwrap();
+        let text = xml_blocks_text(&txn, &frag);
+        for expected in [
+            "Heading One",
+            "Heading Two",
+            "Plain, bold, italic, strike, underline, and code.",
+            "Visit the website for more.", // link text stays inline
+            "First bullet",
+            "Second bullet",
+            "Nested A", // nested sub-list
+            "Nested B",
+            "Step one",
+            "Step two",
+            "Done item",
+            "Todo item",
+            "A blockquote about CRDTs.",
+            "const x = 1;", // code block (keeps its internal newline)
+            "console.log(x);",
+            "Name", // table header cells
+            "Role",
+            "Ada", // table body cells
+            "Engineer",
+        ] {
+            assert!(text.contains(expected), "missing {expected:?} in:\n{text}");
+        }
+        // The inline link must NOT have been split onto its own line.
+        assert!(text.contains("Visit the website for more."));
+    }
 }

data/lib/y/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Y
-  VERSION = "0.2.1"
+  VERSION = "0.2.3"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: yrby
 version: !ruby/object:Gem::Version
-  version: 0.2.1
+  version: 0.2.3
 platform: ruby
 authors:
 - JP Camara