yrby-rails 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG-rails.md +123 -0
- data/LICENSE +21 -0
- data/README.md +753 -0
- data/app/models/y/document.rb +182 -0
- data/app/models/y/document_update.rb +10 -0
- data/lib/generators/yrby/install/USAGE +11 -0
- data/lib/generators/yrby/install/install_generator.rb +44 -0
- data/lib/generators/yrby/install/templates/document_channel.rb +36 -0
- data/lib/generators/yrby/tables/tables_generator.rb +41 -0
- data/lib/generators/yrby/tables/templates/create_y_tables.rb +24 -0
- data/lib/y/action_cable/sync.rb +333 -0
- data/lib/y/action_cable.rb +23 -0
- data/lib/yrby/engine.rb +11 -0
- data/lib/yrby/rails/version.rb +7 -0
- data/lib/yrby-rails.rb +6 -0
- metadata +175 -0
data/README.md
ADDED
|
@@ -0,0 +1,753 @@
|
|
|
1
|
+
# yrby
|
|
2
|
+
|
|
3
|
+
[](https://github.com/jpcamara/yrby/actions/workflows/ci.yml)
|
|
4
|
+
|
|
5
|
+
Collaborative editing for Rails, backed by [y-crdt](https://github.com/y-crdt/y-crdt)
|
|
6
|
+
(the Rust library behind Y.js). Your Rails server speaks the y-websocket sync
|
|
7
|
+
protocol directly, so there's no separate Node process hosting the Y.js
|
|
8
|
+
documents. Pronounced "yer-bee".
|
|
9
|
+
|
|
10
|
+

|
|
11
|
+
|
|
12
|
+
```ruby
|
|
13
|
+
class DocumentChannel < ApplicationCable::Channel
|
|
14
|
+
include Y::ActionCable
|
|
15
|
+
|
|
16
|
+
on_load { |key| Y::Document.load_state(key) }
|
|
17
|
+
on_change { |key, update| Y::Document.append(key, update) }
|
|
18
|
+
|
|
19
|
+
def subscribed = sync_subscribed(params[:id])
|
|
20
|
+
def receive(data) = sync_receive(data, params[:id])
|
|
21
|
+
end
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
On the browser, use the `ActionCableProvider` from the
|
|
25
|
+
[`yrby-client`](https://www.npmjs.com/package/yrby-client) npm package.
|
|
26
|
+
Integrates with any editor that includes Y.js support, such as Tiptap, ProseMirror
|
|
27
|
+
and [Lexxy](https://www.npmjs.com/package/lexxy-realtime).
|
|
28
|
+
|
|
29
|
+
## Usage
|
|
30
|
+
|
|
31
|
+
Install the gem and npm package:
|
|
32
|
+
|
|
33
|
+
```
|
|
34
|
+
gem install yrby-actioncable # depends on yrby
|
|
35
|
+
npm install yrby-client
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
## What you get
|
|
39
|
+
|
|
40
|
+
- A thread-safe Ruby `Doc` you can share across Ruby threads/fibers, and native CRDT work
|
|
41
|
+
runs with the GVL released.
|
|
42
|
+
- The y-websocket protocol (document sync plus awareness/presence) as a
|
|
43
|
+
one-include ActionCable concern.
|
|
44
|
+
- Authoritative record-before-distribute semantics: each document change can be
|
|
45
|
+
recorded durably before it goes out to anyone.
|
|
46
|
+
- Optional server-side reads: `Doc#read_text` and `Doc#read_map` reconstruct a
|
|
47
|
+
document's contents in Ruby - no Node process - for search, exports, validation,
|
|
48
|
+
or server-side rendering.
|
|
49
|
+
|
|
50
|
+
## Scope
|
|
51
|
+
|
|
52
|
+
`yrby` binds just the part of `y-crdt` you need to *sync and persist* collaborative
|
|
53
|
+
documents - a `Doc`, awareness, and the y-websocket protocol primitives. By default
|
|
54
|
+
the Ruby side treats a document as opaque CRDT state: it applies updates, answers
|
|
55
|
+
sync handshakes, and records deltas without reaching into the contents - the browser
|
|
56
|
+
editor owns the document's shape. When you do need to look inside, `Doc#read_text`
|
|
57
|
+
and `Doc#read_map` reconstruct it server-side, in Ruby.
|
|
58
|
+
|
|
59
|
+
## Durability and delivery
|
|
60
|
+
|
|
61
|
+
The surface is intentionally small, but the focus is durability, resiliency, delivery
|
|
62
|
+
guarantees, correctness, and thread safety.
|
|
63
|
+
|
|
64
|
+
Towards that goal, `yrby` adds opinionated defaults on top of normal Yjs syncing:
|
|
65
|
+
|
|
66
|
+
- Built-in update acknowledgement: the `ActionCableProvider` in `yrby-client` will continue to
|
|
67
|
+
send updates until an ack is received from the server. [`yrby-actioncable`](https://rubygems.org/gems/yrby-actioncable)
|
|
68
|
+
only sends an ack when applying an update is successful. The goal is at-least-once delivery,
|
|
69
|
+
and because CRDTs are idempotent a duplicate update is effectively a no-op.
|
|
70
|
+
- Gap detection in document updates: before applying an update and sending an ack to the client,
|
|
71
|
+
`yrby` checks whether the update results in any causal gap. Ie, an update comes through
|
|
72
|
+
which depends on a previous update that is not yet present in the document. This can result in
|
|
73
|
+
a document stuck with "pending" updates, which will _never_ apply if the missing update is not sent.
|
|
74
|
+
To avoid this, `yrby` does not apply the update, and starts a new y-protocol sync with the client.
|
|
75
|
+
That will cause the client to synchronize its document with the server, sending through any updates
|
|
76
|
+
that may have been missed
|
|
77
|
+
|
|
78
|
+
## What about [yrb](https://github.com/y-crdt/yrb)?
|
|
79
|
+
|
|
80
|
+
`yrb` has a much larger interface that gives you most of the Yjs type system -
|
|
81
|
+
shared text, arrays, maps, XML - to build and query documents in Ruby. It was a great
|
|
82
|
+
inspiration for my use of Yjs in Ruby/Rails, and I originally considered building
|
|
83
|
+
on top of it. There are a few reasons I went with `yrby` instead:
|
|
84
|
+
|
|
85
|
+
- `yrb` is largely unmaintained. It was built as an experiment for GitLab, and the original
|
|
86
|
+
author mostly moved onto other projects.
|
|
87
|
+
- [It isn't thread-safe](https://github.com/y-crdt/yrb/issues/72). It segfaults in a threaded
|
|
88
|
+
environment (such as ActionCable...)
|
|
89
|
+
- It's a much larger set of features to maintain, which most people don't need. The vast
|
|
90
|
+
majority of people manipulate Y.js documents in the browser, not from a server-side language.
|
|
91
|
+
|
|
92
|
+
## Testing
|
|
93
|
+
|
|
94
|
+
Ruby and Rust unit tests cover the core. CI also runs the npm client tests and a
|
|
95
|
+
Rails demo smoke slice against the real ActionCable stack. The demo includes
|
|
96
|
+
heavier local suites for hostile input, crash recovery, multi-browser editing,
|
|
97
|
+
AnyCable, and load testing. The benchmark note below is from a single laptop.
|
|
98
|
+
Issues and PRs are welcome.
|
|
99
|
+
|
|
100
|
+
## Install
|
|
101
|
+
|
|
102
|
+
```ruby
|
|
103
|
+
# Core CRDT + protocol primitives:
|
|
104
|
+
gem "yrby"
|
|
105
|
+
|
|
106
|
+
# For the Rails side (the sync channel, document models, the generator).
|
|
107
|
+
# Formerly yrby-actioncable; that name stops at 0.3.1.
|
|
108
|
+
gem "yrby-rails"
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
Requires Ruby 3.4 or newer. The release workflow builds precompiled gems for
|
|
112
|
+
Ruby 3.4 and 4.0 across the supported Ruby platforms, with native smoke tests
|
|
113
|
+
on Linux x86_64 and macOS arm64. Installing from a matching platform gem needs
|
|
114
|
+
no Rust; a source build needs [Rust](https://rustup.rs).
|
|
115
|
+
|
|
116
|
+
To work on the gem itself:
|
|
117
|
+
|
|
118
|
+
```bash
|
|
119
|
+
git clone https://github.com/jpcamara/yrby
|
|
120
|
+
cd yrby
|
|
121
|
+
bundle install
|
|
122
|
+
bundle exec rake compile test
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
The rest of the dev setup, plus the demo, is in [CONTRIBUTING.md](CONTRIBUTING.md).
|
|
126
|
+
|
|
127
|
+
## Docs
|
|
128
|
+
|
|
129
|
+
- The ActionCable concern and a quickstart are [below](#actioncable-integration).
|
|
130
|
+
- [`examples/actioncable-demo`](examples/actioncable-demo): a runnable Rails +
|
|
131
|
+
Tiptap app with collaborative cursors, the AnyCable setup, a Postgres store,
|
|
132
|
+
and the test/load suites.
|
|
133
|
+
- [CHANGELOG.md](CHANGELOG.md) and [CONTRIBUTING.md](CONTRIBUTING.md).
|
|
134
|
+
|
|
135
|
+
## Editors
|
|
136
|
+
|
|
137
|
+
yrby syncs opaque Yjs updates, so it works with any editor that has a Yjs
|
|
138
|
+
binding. The demo app runs four, and CI drives each one in real Chrome:
|
|
139
|
+
concurrent typing with every keystroke accounted for, remote cursors,
|
|
140
|
+
local-only undo, and byte parity between the server-side renderers and the
|
|
141
|
+
editor's own serializer. Each page is a working integration to copy from:
|
|
142
|
+
|
|
143
|
+
| Editor | Yjs binding | Demo code |
|
|
144
|
+
|---|---|---|
|
|
145
|
+
| [Tiptap](https://tiptap.dev) (v2) | `@tiptap/extension-collaboration` | [`app.js`](examples/actioncable-demo/frontend/src/app.js) |
|
|
146
|
+
| [Lexxy](https://github.com/basecamp/lexxy) (Lexical) | [`lexxy-realtime`](https://www.npmjs.com/package/lexxy-realtime) | [`lexxy.js`](examples/actioncable-demo/frontend/src/lexxy.js) |
|
|
147
|
+
| [Rhino Editor](https://github.com/KonnorRogers/rhino-editor) (Tiptap 3) | `@tiptap/extension-collaboration` + `-caret` | [`rhino.js`](examples/actioncable-demo/frontend/src/rhino.js) |
|
|
148
|
+
| [CodeMirror 6](https://codemirror.net) | `y-codemirror.next` | [`codemirror.js`](examples/actioncable-demo/frontend/src/codemirror.js) |
|
|
149
|
+
|
|
150
|
+
The demo also syncs plain Yjs shapes with no editor at all — a whiteboard
|
|
151
|
+
on a `Y.Map`, a kanban board on a `Y.Array`, a co-filled form — over the
|
|
152
|
+
same channel. The demo README's "Using this in your own app" section has
|
|
153
|
+
the integration recipe, and its `NoteMaterializer` shows how to render a
|
|
154
|
+
document to ActionText server-side with `Y::Tiptap` or `Y::Lexxy`.
|
|
155
|
+
|
|
156
|
+
## Usage
|
|
157
|
+
|
|
158
|
+
### Doc (Low-Level Document Sync)
|
|
159
|
+
|
|
160
|
+
```ruby
|
|
161
|
+
require "y"
|
|
162
|
+
|
|
163
|
+
# Create docs
|
|
164
|
+
doc = Y::Doc.new # random client ID
|
|
165
|
+
doc = Y::Doc.new(12345) # specific client ID (used for CRDT identity)
|
|
166
|
+
|
|
167
|
+
# Encoding
|
|
168
|
+
doc.encode_state_vector # => current state vector
|
|
169
|
+
doc.encode_state_as_update # => full update (lossless: keeps pending)
|
|
170
|
+
doc.encode_state_as_update(sv) # => update diff against state vector
|
|
171
|
+
doc.compacted_state_update # => full update, gap-free (excludes pending)
|
|
172
|
+
|
|
173
|
+
# Applying updates
|
|
174
|
+
doc.apply_update(update_bytes) # apply raw V1 update
|
|
175
|
+
doc.pending? # => true if holding un-integrable pending structs
|
|
176
|
+
|
|
177
|
+
# Sync protocol
|
|
178
|
+
doc.sync_step1 # => SyncStep1 message (this doc's state vector)
|
|
179
|
+
doc.handle_sync_message(data) # => [msg_type, sync_type, response]; answers a
|
|
180
|
+
# peer's SyncStep1 with an integrated-only
|
|
181
|
+
# SyncStep2 (never serves pending structs)
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
### Reading document contents
|
|
185
|
+
|
|
186
|
+
Reconstruct a document server-side — search, exports, emails, SSR — with no
|
|
187
|
+
Node process:
|
|
188
|
+
|
|
189
|
+
```ruby
|
|
190
|
+
doc.read_text("prosemirror") # => plain text of a Y.Text root, or nil
|
|
191
|
+
doc.read_xml("root") # => text of an XML root, one block per line
|
|
192
|
+
doc.read_map("state") # => a Y.Map root as a JSON string; JSON.parse it
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
### Pending structs and gap-free state
|
|
196
|
+
|
|
197
|
+
If a doc applies an update whose causally-prior update is missing (a "gappy"
|
|
198
|
+
update), yrs parks it as a **pending** struct: the integrated state vector stays
|
|
199
|
+
empty, but the pending block is held as a recovery buffer and heals if the
|
|
200
|
+
missing dependency later arrives. `Doc#pending?` reports this.
|
|
201
|
+
|
|
202
|
+
Pending structs are *not* document state, so they must not cross the sync
|
|
203
|
+
boundary — a peer that receives one can't integrate it and gets stuck. Two
|
|
204
|
+
guarantees keep serving safe:
|
|
205
|
+
|
|
206
|
+
- `handle_sync_message` answers `SyncStep1` with **integrated-only** state, so a
|
|
207
|
+
server never serves a struct it can't integrate itself (this is automatic).
|
|
208
|
+
- `Doc#compacted_state_update` gives you the same gap-free full-state update for
|
|
209
|
+
when you persist or hand off state yourself. It's non-destructive (the doc
|
|
210
|
+
keeps its pending), while `encode_state_as_update` stays lossless so you can
|
|
211
|
+
still preserve the raw pending bytes for recovery.
|
|
212
|
+
|
|
213
|
+
### Rendering to HTML
|
|
214
|
+
|
|
215
|
+
Schema-pinned renderers turn a collaborative document into HTML on the
|
|
216
|
+
server, with no Node process or headless editor. Each is an editor-specific
|
|
217
|
+
class (byte-for-byte with that editor's own serializer) built on a core base
|
|
218
|
+
any other editor extends with rules: `Y::Tiptap` on `Y::ProseMirror` for
|
|
219
|
+
ProseMirror documents, and `Y::Lexxy` (the
|
|
220
|
+
[Lexxy](https://github.com/basecamp/lexxy) editor) on `Y::Lexical`. Each
|
|
221
|
+
returns `nil` for a root that belongs to the other schema.
|
|
222
|
+
|
|
223
|
+
#### `Y::Tiptap` (and `Y::ProseMirror`, its base)
|
|
224
|
+
|
|
225
|
+
```ruby
|
|
226
|
+
tiptap = Y::Tiptap.new(doc)
|
|
227
|
+
tiptap.to_html # the "default" fragment (Tiptap's default root)
|
|
228
|
+
tiptap.to_html("content") # or another XML root
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
The output matches Tiptap's own `getHTML()`, checked byte-for-byte in the tests
|
|
232
|
+
against a document captured from a real editor. It follows
|
|
233
|
+
[`tiptap-php`](https://github.com/ueberdosis/tiptap-php) and reads both name
|
|
234
|
+
styles editors use — Tiptap's `bulletList`/`bold` and prosemirror-schema-basic's
|
|
235
|
+
`bullet_list`/`strong`.
|
|
236
|
+
|
|
237
|
+
It covers paragraphs, headings, blockquotes, bullet/ordered/task lists, code
|
|
238
|
+
blocks, links, images, mentions, details, hard breaks, horizontal rules,
|
|
239
|
+
tables, text styles (color, font family), and every text mark. A table renders
|
|
240
|
+
as semantic `<table><tbody>`, without the column-width styling Tiptap's editor
|
|
241
|
+
view adds.
|
|
242
|
+
|
|
243
|
+
The support is layered like the Lexical side: `Y::ProseMirror` covers core
|
|
244
|
+
ProseMirror natively — prosemirror-schema-basic plus the prosemirror-tables
|
|
245
|
+
family — and Tiptap's extension nodes (task lists, mentions, the details
|
|
246
|
+
family) are `Y::Tiptap`'s rule set (`Y::Tiptap::NODES`), built on the
|
|
247
|
+
extension API below. Marks stay in the base: mark rendering (nesting order,
|
|
248
|
+
`textStyle` CSS, `code` exclusivity) runs through native text-run machinery
|
|
249
|
+
that node rules don't reach, so `Y::ProseMirror` renders Tiptap's mark set
|
|
250
|
+
as-is and `rules.mark` overrides individual marks.
|
|
251
|
+
|
|
252
|
+
#### `Y::Lexxy` (and `Y::Lexical`, its base)
|
|
253
|
+
|
|
254
|
+
```ruby
|
|
255
|
+
lexxy = Y::Lexxy.new(doc)
|
|
256
|
+
lexxy.to_html # the "root" fragment (Lexical's default root name)
|
|
257
|
+
lexxy.to_html("notepad") # or another XML root
|
|
258
|
+
```
|
|
259
|
+
|
|
260
|
+
The HTML is identical to what a `lexxy-editor` submits to Rails (its `value`).
|
|
261
|
+
The tests check this byte-for-byte against a document captured from a real
|
|
262
|
+
editor. Stock Lexical has no canonical serializer — every editor configures
|
|
263
|
+
its own — so the editor-specific class carries the editor's name, and
|
|
264
|
+
`Y::Lexical` is the core-Lexical base: paragraphs, headings, quotes, code,
|
|
265
|
+
lists, tables, links, and the full text-format model, for any other Lexical
|
|
266
|
+
editor to extend with rules.
|
|
267
|
+
|
|
268
|
+
It handles the whole Lexxy 0.9.x node set: paragraphs, headings, every text
|
|
269
|
+
format and their combinations, links, the four list types and nesting,
|
|
270
|
+
blockquotes, code blocks, tabs and soft breaks, horizontal rules, tables with
|
|
271
|
+
header cells, image galleries, and ActionText attachments (uploads and
|
|
272
|
+
mentions both emit `<action-text-attachment>` elements that ActionText can
|
|
273
|
+
re-render).
|
|
274
|
+
|
|
275
|
+
Internally that support is layered: `Y::Lexical` covers core Lexical
|
|
276
|
+
structure natively, and everything Lexxy adds — its node types (attachments,
|
|
277
|
+
galleries) and its decorations of core nodes (the table wrapper, header-cell
|
|
278
|
+
styling, nested-list classes) — is `Y::Lexxy`'s rule set
|
|
279
|
+
(`Y::Lexxy::NODES`), built on the extension API below. The gem's own Lexxy
|
|
280
|
+
support is the API's first consumer: an app rule for one of those types
|
|
281
|
+
simply replaces it.
|
|
282
|
+
|
|
283
|
+
In both renderers an unknown node keeps its content — text and nested blocks
|
|
284
|
+
fall back to readable markup rather than disappearing.
|
|
285
|
+
|
|
286
|
+
#### Custom nodes and marks
|
|
287
|
+
|
|
288
|
+
The built-in schemas are pinned to what Tiptap and Lexxy ship, but apps add
|
|
289
|
+
their own node types. Both renderers take rules for them. A rule is checked
|
|
290
|
+
before the built-in schema, so it can add a node type or replace how a
|
|
291
|
+
built-in renders.
|
|
292
|
+
|
|
293
|
+
Rules register in a block — one `rules.node` call per type. A declarative
|
|
294
|
+
rule is markup as data, rendered natively:
|
|
295
|
+
|
|
296
|
+
```ruby
|
|
297
|
+
tiptap = Y::Tiptap.new(doc) do |rules|
|
|
298
|
+
rules.node "callout", tag: "aside",
|
|
299
|
+
attrs: { "class" => ["callout callout--", :kind] },
|
|
300
|
+
contains: :blocks
|
|
301
|
+
end
|
|
302
|
+
```
|
|
303
|
+
|
|
304
|
+
`tag` names the element. `attrs` values are templates: a string is a literal,
|
|
305
|
+
a symbol reads that attribute off the node, an array concatenates both kinds;
|
|
306
|
+
an attribute that resolves empty is left out. `text` (same template form)
|
|
307
|
+
emits literal text content. `contains` declares what lives inside the node — `:inline` (formatted text,
|
|
308
|
+
the default), `:blocks` (child block nodes — a container), or `:none` (a
|
|
309
|
+
leaf). `void: true` skips the closing tag.
|
|
310
|
+
|
|
311
|
+
You don't have to guess any of those names or shapes. Editors store types
|
|
312
|
+
and attributes under names you'd never predict (Rhino's strike mark is
|
|
313
|
+
`rhino-strike`; Lexical prefixes its own props `__`), so ask a real
|
|
314
|
+
document instead — make one in your editor using your custom node, then:
|
|
315
|
+
|
|
316
|
+
```ruby
|
|
317
|
+
Y::Tiptap.new(doc).node_types
|
|
318
|
+
# => { "callout" => { "count" => 2, "attrs" => ["kind"],
|
|
319
|
+
# "children" => ["paragraph"], "text" => false,
|
|
320
|
+
# "handled" => nil },
|
|
321
|
+
# "paragraph" => { ..., "handled" => "builtin" } }
|
|
322
|
+
```
|
|
323
|
+
|
|
324
|
+
`handled` nil marks the types that still need a rule; `attrs` are the stored
|
|
325
|
+
names your templates and blocks will read; `children` plus `text` is how you
|
|
326
|
+
pick `contains:` (child block types → `:blocks`; text → `:inline`).
|
|
327
|
+
|
|
328
|
+
When markup-as-data isn't enough, give the node a block:
|
|
329
|
+
|
|
330
|
+
```ruby
|
|
331
|
+
lexical = Y::Lexical.new(doc) do |rules|
|
|
332
|
+
rules.node "video_embed" do |node|
|
|
333
|
+
src = ERB::Util.html_escape(node.attrs["__src"])
|
|
334
|
+
%(<video controls src="#{src}"></video>)
|
|
335
|
+
end
|
|
336
|
+
end
|
|
337
|
+
```
|
|
338
|
+
|
|
339
|
+
The block gets the node's type, its stored attributes, `node.content` — the
|
|
340
|
+
children, already rendered to HTML — and `node.child_types`, the node's
|
|
341
|
+
element/block children by type, in document order. `child_types` answers the
|
|
342
|
+
structural questions attributes can't: how many images a gallery holds, or
|
|
343
|
+
whether a list item carries a nested list. Whatever the block returns is
|
|
344
|
+
spliced into the output as-is: it's trusted HTML, so escape any values you
|
|
345
|
+
interpolate. To set the content mode for a callback, give the node both —
|
|
346
|
+
`rules.node "embed", contains: :blocks do |node| ... end`.
|
|
347
|
+
|
|
348
|
+
Callbacks never run while the document is locked. The render finishes first
|
|
349
|
+
(inside one read transaction, GVL released), then the blocks run and their
|
|
350
|
+
output is spliced in — so a callback can safely read or even write the same
|
|
351
|
+
doc. With no callback rules, `to_html` skips the splicing entirely.
|
|
352
|
+
|
|
353
|
+
Blocks are the escape hatch for everything the declarative form can't say,
|
|
354
|
+
and they're proven sufficient: `Y::Lexxy` and `Y::Tiptap` are themselves
|
|
355
|
+
built on this API (`lib/y/lexxy.rb`, `lib/y/tiptap.rb`) — simple nodes as
|
|
356
|
+
declarative hashes, everything with logic as plain methods mapped by node
|
|
357
|
+
type (a `Method` responds to `call` like any lambda) — and the fixture tests
|
|
358
|
+
hold their output byte-identical to a live editor's.
|
|
359
|
+
|
|
360
|
+
The ProseMirror side also takes custom marks:
|
|
361
|
+
|
|
362
|
+
```ruby
|
|
363
|
+
tiptap = Y::Tiptap.new(doc) do |rules|
|
|
364
|
+
rules.mark "comment", tag: "span", attrs: { "data-comment-id" => :id }
|
|
365
|
+
end
|
|
366
|
+
```
|
|
367
|
+
|
|
368
|
+
Symbol refs resolve against the mark's own attributes. A custom mark wraps
|
|
369
|
+
outside every built-in mark; several on one run nest alphabetically. A rule
|
|
370
|
+
for a built-in mark name (`"bold"`) replaces its built-in tag.
|
|
371
|
+
|
|
372
|
+
##### Worked examples
|
|
373
|
+
|
|
374
|
+
A video-embed node from an app's Tiptap extension — a type the pinned schema
|
|
375
|
+
has never heard of:
|
|
376
|
+
|
|
377
|
+
```ruby
|
|
378
|
+
tiptap = Y::Tiptap.new(doc) do |rules|
|
|
379
|
+
rules.node "videoEmbed" do |node|
|
|
380
|
+
src = ERB::Util.html_escape(node.attrs["src"])
|
|
381
|
+
title = ERB::Util.html_escape(node.attrs["title"] || "Video")
|
|
382
|
+
%(<figure class="video"><iframe src="#{src}" title="#{title}" allowfullscreen></iframe></figure>)
|
|
383
|
+
end
|
|
384
|
+
end
|
|
385
|
+
```
|
|
386
|
+
|
|
387
|
+
Resolving mentions against the database. Blocks run after the document read
|
|
388
|
+
has finished, so hitting ActiveRecord (or the doc itself) inside one is safe:
|
|
389
|
+
|
|
390
|
+
```ruby
|
|
391
|
+
tiptap = Y::Tiptap.new(doc) do |rules|
|
|
392
|
+
rules.node "mention" do |node|
|
|
393
|
+
user = User.find_by(id: node.attrs["id"])
|
|
394
|
+
next "<span>@unknown</span>" unless user
|
|
395
|
+
|
|
396
|
+
%(<a class="mention" href="/users/#{user.id}">@#{ERB::Util.html_escape(user.handle)}</a>)
|
|
397
|
+
end
|
|
398
|
+
end
|
|
399
|
+
```
|
|
400
|
+
|
|
401
|
+
Overriding a shipped rule — rendering Lexxy uploads as real image markup
|
|
402
|
+
instead of the `<action-text-attachment>` elements ActionText re-renders:
|
|
403
|
+
|
|
404
|
+
```ruby
|
|
405
|
+
lexxy = Y::Lexxy.new(doc) do |rules|
|
|
406
|
+
rules.node "action_text_attachment" do |node|
|
|
407
|
+
src = ERB::Util.html_escape(node.attrs["src"])
|
|
408
|
+
alt = ERB::Util.html_escape(node.attrs["altText"].to_s)
|
|
409
|
+
caption = node.attrs["caption"].to_s
|
|
410
|
+
html = %(<img src="#{src}" alt="#{alt}" loading="lazy">)
|
|
411
|
+
html += "<figcaption>#{ERB::Util.html_escape(caption)}</figcaption>" unless caption.empty?
|
|
412
|
+
"<figure>#{html}</figure>"
|
|
413
|
+
end
|
|
414
|
+
end
|
|
415
|
+
```
|
|
416
|
+
|
|
417
|
+
Markup that depends on structure — `node.child_types` lists the node's
|
|
418
|
+
element/block children in document order, so a layout container can size
|
|
419
|
+
itself by its column count while the columns themselves stay declarative:
|
|
420
|
+
|
|
421
|
+
```ruby
|
|
422
|
+
tiptap = Y::Tiptap.new(doc) do |rules|
|
|
423
|
+
rules.node "columns", contains: :blocks do |node|
|
|
424
|
+
%(<div class="columns columns--#{node.child_types.length}">#{node.content}</div>)
|
|
425
|
+
end
|
|
426
|
+
rules.node "column", tag: "div", attrs: { "class" => "column" }, contains: :blocks
|
|
427
|
+
end
|
|
428
|
+
```
|
|
429
|
+
|
|
430
|
+
Content-aware overrides — dropping the empty paragraphs an editor keeps
|
|
431
|
+
around the cursor, since `node.content` arrives already rendered:
|
|
432
|
+
|
|
433
|
+
```ruby
|
|
434
|
+
lexical = Y::Lexical.new(doc) do |rules|
|
|
435
|
+
rules.node "paragraph" do |node|
|
|
436
|
+
node.content.empty? ? "" : "<p>#{node.content}</p>"
|
|
437
|
+
end
|
|
438
|
+
end
|
|
439
|
+
```
|
|
440
|
+
|
|
441
|
+
For a larger reference, the gem's own editor schemas ship this way — see
|
|
442
|
+
`Y::Lexxy::NODES` in `lib/y/lexxy.rb` (declarative hashes for the simple
|
|
443
|
+
nodes, a plain method per node that needs logic — galleries, list items,
|
|
444
|
+
header cells, both attachment types — mapped with `method(:name)`) and
|
|
445
|
+
`Y::Tiptap::NODES` in `lib/y/tiptap.rb` (task lists, mentions, the details
|
|
446
|
+
family).
|
|
447
|
+
|
|
448
|
+
### Protocol codec (module functions)
|
|
449
|
+
|
|
450
|
+
Classifying and unwrapping wire frames is stateless, so it's exposed as
|
|
451
|
+
`Y` module functions rather than a class. The server never holds presence
|
|
452
|
+
or document state to route a frame — presence lives in the browser clients, and
|
|
453
|
+
the server only relays awareness frames opaquely.
|
|
454
|
+
|
|
455
|
+
```ruby
|
|
456
|
+
Y.message_kind(frame) # => 0 drop / 1 step1 / 2 update / 3 awareness / 4 query
|
|
457
|
+
Y.update_from_message(frame) # => the document delta carried by a frame, or nil
|
|
458
|
+
Y.wrap_update(update_bytes) # => wrap a raw doc update as a sync Update frame
|
|
459
|
+
```
|
|
460
|
+
|
|
461
|
+
### ActionCable Integration
|
|
462
|
+
|
|
463
|
+
In a Rails app, one generator creates the channel and the migration:
|
|
464
|
+
|
|
465
|
+
```bash
|
|
466
|
+
bin/rails generate yrby:install
|
|
467
|
+
bin/rails db:migrate
|
|
468
|
+
```
|
|
469
|
+
|
|
470
|
+
The models ship in the gem, the way Action Text owns
|
|
471
|
+
`ActionText::RichText`:
|
|
472
|
+
|
|
473
|
+
- **`Y::Document`** — one row per document, addressed two ways: by `key`
|
|
474
|
+
(what a channel addresses — one opaque, unique string, sometimes
|
|
475
|
+
app-supplied, never parsed) and, optionally, by polymorphic `record` +
|
|
476
|
+
`name` (which model attribute it backs; `name` is the attribute name,
|
|
477
|
+
`"body"` — one document per attribute per record, the
|
|
478
|
+
ActionText::RichText scheme). Key-only documents leave the binding nil.
|
|
479
|
+
Either side can arrive first: `Y::Document.for(record, name)` finds or
|
|
480
|
+
creates the binding, derives a readable key (`post/1/body`), and adopts
|
|
481
|
+
a key-only row already holding that key, so a channel writing first and
|
|
482
|
+
a binding created later converge on one document. The row also holds
|
|
483
|
+
the merged `state` snapshot — CRDT state only; derived data (rendered
|
|
484
|
+
HTML, search text) is the application's job, typically in the channel's
|
|
485
|
+
on_change. `.load_state(key)` / `.append(key, update)` are the store
|
|
486
|
+
calls the generated channel uses.
|
|
487
|
+
- **`Y::DocumentUpdate`** — the uncompacted tail: one delta per row,
|
|
488
|
+
compacted into `state` and deleted once the tail reaches `compact_every`
|
|
489
|
+
(default 64). Loading reads the snapshot plus the current tail; an
|
|
490
|
+
empty tail returns `state` directly. Compaction serializes on a
|
|
491
|
+
per-document row lock and skips causally-gapped rows — they're
|
|
492
|
+
quarantined until they heal rather than compacted into state or
|
|
493
|
+
deleted. Destroying a document deletes its updates with it.
|
|
494
|
+
|
|
495
|
+
The migration creates `y_documents` and `y_document_updates`. To rename
|
|
496
|
+
them, edit the generated migration and point `Y::Document.table_name` /
|
|
497
|
+
`Y::DocumentUpdate.table_name` at the new names in an initializer.
|
|
498
|
+
|
|
499
|
+
Storage is swappable: the channel only needs `on_load` and `on_change`
|
|
500
|
+
answered, and they can point at anything.
|
|
501
|
+
|
|
502
|
+
`include Y::ActionCable` (from the `yrby-rails` gem) is the channel
|
|
503
|
+
integration: the y-websocket protocol (document sync +
|
|
504
|
+
awareness/presence) over ActionCable. (`include Y::ActionCable::Sync`
|
|
505
|
+
keeps working and has the same effect.)
|
|
506
|
+
|
|
507
|
+
```ruby
|
|
508
|
+
# app/channels/document_channel.rb
|
|
509
|
+
class DocumentChannel < ApplicationCable::Channel
|
|
510
|
+
include Y::ActionCable
|
|
511
|
+
|
|
512
|
+
on_load { |key| Y::Document.load_state(key) } # rebuild from storage
|
|
513
|
+
on_change { |key, update| Y::Document.append(key, update) } # record, then broadcast
|
|
514
|
+
|
|
515
|
+
def subscribed
|
|
516
|
+
return reject unless authorized?(params[:id])
|
|
517
|
+
|
|
518
|
+
sync_subscribed params[:id]
|
|
519
|
+
end
|
|
520
|
+
|
|
521
|
+
def receive(data)
|
|
522
|
+
sync_receive(data, params[:id])
|
|
523
|
+
end
|
|
524
|
+
|
|
525
|
+
private
|
|
526
|
+
|
|
527
|
+
# Everyone is denied until you wire this to your app's auth.
|
|
528
|
+
def authorized?(_document_key) = false
|
|
529
|
+
end
|
|
530
|
+
```
|
|
531
|
+
|
|
532
|
+
The concern is store-backed. A handshake is answered from `on_load`; document
|
|
533
|
+
changes are checked against that durable state, recorded through `on_change`,
|
|
534
|
+
then broadcast. Nothing authoritative is kept in ActionCable process memory, so
|
|
535
|
+
AnyCable RPC workers, Puma workers, and separate dynos can all handle messages
|
|
536
|
+
for the same document as long as they share the same store and cable adapter.
|
|
537
|
+
|
|
538
|
+
`on_load` and `on_change` are required. If either is missing, the channel fails
|
|
539
|
+
before it can acknowledge or broadcast edits. Presence is ephemeral:
|
|
540
|
+
awareness frames are relayed, and `yrby-client` sends a best-effort
|
|
541
|
+
presence-removal frame on disconnect/pagehide, with the client-side awareness
|
|
542
|
+
timeout as the fallback for abrupt disconnects.
|
|
543
|
+
|
|
544
|
+
Incoming frames are validated as a single well-formed protocol message before
|
|
545
|
+
anything processes or relays them. Malformed, truncated, multi-message,
|
|
546
|
+
oversized, or unknown frames are dropped. A bad frame can't crash the process: a
|
|
547
|
+
Rust panic is caught at the FFI boundary and re-raised as a Ruby exception. And
|
|
548
|
+
no single client can relay garbage that breaks the others in a room.
|
|
549
|
+
|
|
550
|
+
#### Delivery guarantees
|
|
551
|
+
|
|
552
|
+
The contract is the same at every scale — one process, or hundreds across many
|
|
553
|
+
servers:
|
|
554
|
+
|
|
555
|
+
- **The document always converges.** CRDT updates are commutative and
|
|
556
|
+
idempotent, so out-of-order, duplicate, or concurrent delivery all converge to
|
|
557
|
+
the same correct document. This needs no coordination and holds everywhere.
|
|
558
|
+
- **The durable log never goes gappy.** An update is recorded only once its
|
|
559
|
+
causal dependencies are already in the store (checked against `on_load`); a
|
|
560
|
+
causally-incomplete update triggers a resync instead, so the log always
|
|
561
|
+
rebuilds cleanly.
|
|
562
|
+
- **`on_change` is at-least-once, and the durable guarantee is that replaying the
|
|
563
|
+
log reconstructs the document.** Every update triggers `on_change` before it's acked or
|
|
564
|
+
broadcast (record-before-distribute). If exactly-once updates matter for you, **you
|
|
565
|
+
must make `on_change` idempotent**. But remember that the CRDT can handle duplicates.
|
|
566
|
+
- **A raising `on_change` rejects the update implicitly.** If the block raises,
|
|
567
|
+
the update is neither acked nor broadcast (record-before-distribute stops both).
|
|
568
|
+
There is no negative-ack: the client simply never receives the ack, keeps the
|
|
569
|
+
update pending, and retransmits on its timer/reconnect. This is built for
|
|
570
|
+
*transient* failures (the store is briefly down → a retry lands). A block that
|
|
571
|
+
raises *deterministically* — a validation that always fails for this edit —
|
|
572
|
+
will be retried forever, since nothing tells the client to stop. Enforce hard
|
|
573
|
+
rejections before the edit reaches `on_change` (channel authorization in
|
|
574
|
+
`subscribed`), not by raising inside it.
|
|
575
|
+
- **An over-cap frame is dropped the same silent way.** A frame larger than
|
|
576
|
+
`max_frame_bytes` (default 8 MiB) is dropped before decoding — no ack, no
|
|
577
|
+
broadcast — to bound the work a client can force. For a genuine document
|
|
578
|
+
update that means the same implicit rejection as above: unacked, retransmitted
|
|
579
|
+
forever. Normal typing never approaches the cap, but a large paste, an embedded
|
|
580
|
+
image, or a big initial `SyncStep2` can. The drop is logged (`warn` for
|
|
581
|
+
over-cap, `debug` for undecodable) with the document key and update id so it's
|
|
582
|
+
findable; override `sync_log_context` on the channel to add a user/connection
|
|
583
|
+
id. Size the cap for your largest expected payload, and reject
|
|
584
|
+
genuinely-too-big content upstream rather than relying on the cap to reject it
|
|
585
|
+
gracefully.
|
|
586
|
+
|
|
587
|
+
#### Multi-process deployments
|
|
588
|
+
|
|
589
|
+
Most Rails apps run several processes, and any of them might serve a given document.
|
|
590
|
+
Two pieces keep them in step.
|
|
591
|
+
|
|
592
|
+
Broadcasts cross processes through the Action Cable adapter, so it needs to something
|
|
593
|
+
like `redis` or `solid_cable`, not `async`. With that in place, a change
|
|
594
|
+
on one process reaches clients on all of them.
|
|
595
|
+
|
|
596
|
+
Every process rebuilds document state from the durable store through `on_load`.
|
|
597
|
+
Because changes are recorded before broadcast, record-before-distribute holds
|
|
598
|
+
across processes: whichever process receives a change records it to the shared
|
|
599
|
+
store before anyone, anywhere, sees it.
|
|
600
|
+
|
|
601
|
+
`bun multiprocess.mjs` in the demo runs clients across two processes and checks
|
|
602
|
+
convergence, fresh reads on both, presence across processes, and one shared log.
|
|
603
|
+
|
|
604
|
+
##### AnyCable
|
|
605
|
+
|
|
606
|
+
`yrby` fully supports AnyCable.
|
|
607
|
+
|
|
608
|
+
The demo checks this against a real anycable-go + RPC server
|
|
609
|
+
(`frontend/anycable_probe.mjs`, `anycable_concurrent.mjs`): liveness, the
|
|
610
|
+
yrby client provider, cross-process reads, and concurrent convergence.
|
|
611
|
+
|
|
612
|
+
##### Demo
|
|
613
|
+
|
|
614
|
+
[`examples/actioncable-demo`](examples/actioncable-demo) is a full Rails + Tiptap
|
|
615
|
+
app using the yrby provider, with end-to-end tests.
|
|
616
|
+
|
|
617
|
+
#### Record Before Distribute
|
|
618
|
+
|
|
619
|
+
Every document change is handed to the `on_change` handler before broadcasting.
|
|
620
|
+
It is up to you to durably record it:
|
|
621
|
+
|
|
622
|
+
```ruby
|
|
623
|
+
class DocumentChannel < ApplicationCable::Channel
|
|
624
|
+
include Y::ActionCable
|
|
625
|
+
|
|
626
|
+
# ...
|
|
627
|
+
|
|
628
|
+
on_change do |key, update|
|
|
629
|
+
# Synchronous, durable write. `update` is the exact CRDT delta.
|
|
630
|
+
AuditLog.append!(key, update) # raise to REJECT the change
|
|
631
|
+
end
|
|
632
|
+
|
|
633
|
+
# ...
|
|
634
|
+
end
|
|
635
|
+
```
|
|
636
|
+
|
|
637
|
+
If the recorder raises (say the store is down), the change is rejected: not
|
|
638
|
+
applied, not sent to anyone. The cost is a synchronous durable write on the path
|
|
639
|
+
of every change. There's no in-gem per-document lock; concurrent writes to one
|
|
640
|
+
document can both record (at-least-once), and since CRDT apply is idempotent a
|
|
641
|
+
duplicate record replays to the same document.
|
|
642
|
+
|
|
643
|
+
The demo wires `on_change` to a durable Postgres-backed log by default, and checks
|
|
644
|
+
end to end that the log alone rebuilds the document.
|
|
645
|
+
|
|
646
|
+
#### Reliable delivery (acks)
|
|
647
|
+
|
|
648
|
+
yrby document delivery is ack-tracked. Browser document updates carry an
|
|
649
|
+
`"id"`, and the server replies `{ "ack": <id> }` once `on_change` has succesfully fired.
|
|
650
|
+
A causally-gapped update is not acked; the server sends a resync request, and
|
|
651
|
+
the client keeps the update queued until it lands.
|
|
652
|
+
|
|
653
|
+
```
|
|
654
|
+
client -> server { "update": "<base64 update>", "id": 42 }
|
|
655
|
+
server -> client { "ack": 42 } # update accepted; safe to forget
|
|
656
|
+
```
|
|
657
|
+
|
|
658
|
+
`yrby-client`'s `ActionCableProvider` handles this automatically. It keeps
|
|
659
|
+
the unacknowledged local document tail in a queue and sends the merged tail as a
|
|
660
|
+
single causally-complete delta. The id is the highest sequence in the batch, so
|
|
661
|
+
one `{ ack: id }` cumulatively confirms everything up to it. Because CRDT apply
|
|
662
|
+
is idempotent, a resend that already landed is a harmless no-op that just
|
|
663
|
+
re-acks. Awareness stays ephemeral and is not acked.
|
|
664
|
+
|
|
665
|
+
Presence (cursors, selections) is owned by the browser clients — the server
|
|
666
|
+
never sets or holds presence state, it only relays awareness frames opaquely.
|
|
667
|
+
See `yrby-client` for the client-side awareness API.
|
|
668
|
+
|
|
669
|
+
## Thread Safety
|
|
670
|
+
|
|
671
|
+
A `Doc` is safe to share across Ruby threads — used concurrently from Puma
|
|
672
|
+
workers, ActionCable connection threads, or background jobs without external
|
|
673
|
+
locking.
|
|
674
|
+
|
|
675
|
+
`test/thread_safety_test.rb` runs shared docs, the full sync handshake, and
|
|
676
|
+
fan-in sync across 8 threads at once, and checks the interleaving doesn't change
|
|
677
|
+
convergence.
|
|
678
|
+
|
|
679
|
+
### Parallelism (GVL release)
|
|
680
|
+
|
|
681
|
+
Every method that does real CRDT work (applying updates, encoding state,
|
|
682
|
+
handling sync messages) releases Ruby's Global VM Lock
|
|
683
|
+
(`rb_thread_call_without_gvl`) while the native code runs. That buys two things.
|
|
684
|
+
|
|
685
|
+
CRDT work runs in parallel across Ruby threads on MRI, not just
|
|
686
|
+
JRuby/TruffleRuby. `bench/parallelism_bench.rb` measures over 2x wall-clock
|
|
687
|
+
speedup applying a ~900 KB update concurrently; native code that held the GVL
|
|
688
|
+
couldn't beat serial time.
|
|
689
|
+
|
|
690
|
+
A slow operation also can't stall the VM. A thread applying a large update holds
|
|
691
|
+
the doc's write lock without holding the GVL, so other Ruby threads keep running
|
|
692
|
+
instead of queuing behind it.
|
|
693
|
+
|
|
694
|
+
Each method has the same shape: copy Ruby byte strings first, drop the GVL, do
|
|
695
|
+
the yrs work while taking and releasing native locks entirely inside the
|
|
696
|
+
closure, take the GVL back, then build Ruby objects. No Ruby API is touched
|
|
697
|
+
without the GVL, and no native lock is held while reacquiring it, so the lock
|
|
698
|
+
order can't deadlock. Panics in native code are caught and re-raised as Ruby
|
|
699
|
+
exceptions.
|
|
700
|
+
|
|
701
|
+
## Message Type Constants
|
|
702
|
+
|
|
703
|
+
```ruby
|
|
704
|
+
Y::MSG_SYNC # 0 - Document sync messages
|
|
705
|
+
Y::MSG_AWARENESS # 1 - User presence data
|
|
706
|
+
|
|
707
|
+
Y::MSG_SYNC_STEP1 # 0 - State vector request
|
|
708
|
+
Y::MSG_SYNC_STEP2 # 1 - Update response
|
|
709
|
+
Y::MSG_SYNC_UPDATE # 2 - Incremental update
|
|
710
|
+
```
|
|
711
|
+
|
|
712
|
+
## Sync Flow
|
|
713
|
+
|
|
714
|
+
```
|
|
715
|
+
Client A Server
|
|
716
|
+
| |
|
|
717
|
+
|-------- connect() ------------->|
|
|
718
|
+
| (SyncStep1 + Awareness) |
|
|
719
|
+
| |
|
|
720
|
+
|<--- handle_sync_message resp ---|
|
|
721
|
+
| (SyncStep2) |
|
|
722
|
+
| |
|
|
723
|
+
| (Document synchronized!) |
|
|
724
|
+
| |
|
|
725
|
+
|<------- updates ----------------|
|
|
726
|
+
|-------- updates --------------->|
|
|
727
|
+
```
|
|
728
|
+
|
|
729
|
+
## Development
|
|
730
|
+
|
|
731
|
+
```bash
|
|
732
|
+
# Setup
|
|
733
|
+
bundle install
|
|
734
|
+
|
|
735
|
+
# Build extension
|
|
736
|
+
rake compile
|
|
737
|
+
|
|
738
|
+
# Run tests
|
|
739
|
+
rake test
|
|
740
|
+
|
|
741
|
+
# Clean build artifacts
|
|
742
|
+
rake clean
|
|
743
|
+
```
|
|
744
|
+
|
|
745
|
+
## License
|
|
746
|
+
|
|
747
|
+
MIT License
|
|
748
|
+
|
|
749
|
+
## Acknowledgments
|
|
750
|
+
|
|
751
|
+
- [y-crdt/yrs](https://github.com/y-crdt/y-crdt) - The Rust implementation of Y.js
|
|
752
|
+
- [Magnus](https://github.com/matsadler/magnus) - Ruby bindings for Rust
|
|
753
|
+
- [rb-sys](https://github.com/oxidize-rb/rb-sys) - Rust extensions for Ruby
|