instant_record 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +24 -0
- data/LICENSE.txt +21 -0
- data/README.md +378 -0
- data/app/controllers/instant_record/bootstraps_controller.rb +34 -0
- data/app/controllers/instant_record/events_controller.rb +59 -0
- data/app/controllers/instant_record/mutations_controller.rb +21 -0
- data/app/controllers/instant_record/records_controller.rb +46 -0
- data/app/models/instant_record/applied_mutation.rb +9 -0
- data/app/models/instant_record/change.rb +30 -0
- data/app/models/instant_record/mutation_applier.rb +133 -0
- data/config/routes.rb +7 -0
- data/db/migrate/20260721000002_create_instant_record_outbox.rb +12 -0
- data/db/migrate/20260721000003_create_instant_record_sync_metadata.rb +9 -0
- data/db/migrate/20260721000101_create_instant_record_changes.rb +13 -0
- data/db/migrate/20260721000102_create_instant_record_applied_mutations.rb +10 -0
- data/lib/generators/instant_record/install/install_generator.rb +53 -0
- data/lib/generators/instant_record/install/templates/database.js +32 -0
- data/lib/generators/instant_record/install/templates/index.html +69 -0
- data/lib/generators/instant_record/install/templates/rails.sw.js +750 -0
- data/lib/instant_record/client/notifier.rb +23 -0
- data/lib/instant_record/client/transport.rb +90 -0
- data/lib/instant_record/client.rb +465 -0
- data/lib/instant_record/configuration.rb +19 -0
- data/lib/instant_record/engine.rb +52 -0
- data/lib/instant_record/local_schema.rb +75 -0
- data/lib/instant_record/outbox_mutation.rb +24 -0
- data/lib/instant_record/pglite_compat.rb +40 -0
- data/lib/instant_record/runtime_scoped.rb +21 -0
- data/lib/instant_record/sync_metadata.rb +16 -0
- data/lib/instant_record/sync_window.rb +82 -0
- data/lib/instant_record/syncable.rb +106 -0
- data/lib/instant_record/version.rb +3 -0
- data/lib/instant_record.rb +356 -0
- data/lib/tasks/instant_record.rake +53 -0
- metadata +108 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: e36ca64624187dc76c35e76ec89e98d6bcd58aa154b8cbed2674bfab317196c9
|
|
4
|
+
data.tar.gz: 1b8abb4f4b2184f500d761572870ff23cec96815d7cc79ec7197058fc97fdbff
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: a3909b291e85633d5e5f6f044ab42e33c9570df8ecc7c22aafbb3821dcd10c51be34f4b9cc504f5482d5c10085db7ec737c905c73a9ff70026435af232570bf2
|
|
7
|
+
data.tar.gz: 337d2df1ce4ca243d71249f0afe607f78d4ec86ced28988577caf1c94108fd3af850e25b07b19b1fb1d7f0d858e7b75d180deb3daa3a0875879f42c9b93d22df
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 0.1.0
|
|
4
|
+
|
|
5
|
+
First release. Real Active Record models running in the browser on ruby.wasm
|
|
6
|
+
and PGlite, writing optimistically through a durable outbox that syncs to a
|
|
7
|
+
Rails + Postgres server.
|
|
8
|
+
|
|
9
|
+
- Browser-runtime migrations: `InstantRecord.prepare_database!` brings a returning visitor's local database up to the app's current schema. `DatabaseTasks.prepare_all` alone cannot — the browser VM's working directory is `/`, so ActiveRecord's relative `db/migrate` resolves to nothing and the migrator silently reports zero pending. `InstantRecord::LocalSchema` reconciles `schema_migrations` by the rule `assume_migrated_upto_version` would have used, then migrates only what is genuinely newer. Ships `instant_record/pglite_compat`, without which no in-browser migration could run at all (Rails' PostgreSQL statement pool asks the PGlite connection for a `status` it does not define, and DDL clears the pool).
|
|
10
|
+
- Microsecond timestamps on every wire path, in both directions, via `InstantRecord.wire_attributes`. The change log and outbox previously serialized through ActiveSupport's JSON encoder (milliseconds) while bootstrap and records kept all six places; the mismatch lost `(created_at, id)` keyset neighbours between history pages and biased last-write-wins against whichever side was rounded.
|
|
11
|
+
- Writes that lose last-write-wins are no longer badged as delivered: the server returns the row that won, the client reconciles to it, and `on_discarded_change` lets a model observe what was thrown away. A client running ahead of the server's schema now gets a rejection rather than jamming the outbox behind a mutation that would retry forever.
|
|
12
|
+
- `InstantRecord.server_url` for app paths that must reach the authoritative server from either runtime.
|
|
13
|
+
- Windowed sync: `sync_window limit:, partition_by:` on Syncable models bounds what syncs — fresh clients hydrate from `GET /bootstrap` (windowed snapshot + cursor) instead of replaying the whole change log, a cold boot evicts windowed models back to their window (pending rows never evicted), and older rows page in through `GET /records` keyset cursors via `InstantRecord.fetch_history` (local-first, idempotent, no outbox noise). The service worker gains an `instant_record.fetch_history` page message sharing the tick's single-flight guard.
|
|
14
|
+
- Swiss Slack demo: thousands of seeded history messages (unlogged `insert_all` backfill, preserved across resets), windowed conversation rendering with scroll-up pagination, and in-place DOM morphing on sync — no more full-page reload on the conversation view.
|
|
15
|
+
- Demo pages that let a visitor check the claims rather than take them on trust: `/console` (a REPL against whichever runtime served the page — the eval action is defined inside `browser_only`, so it does not exist on the server), `/source` (each page's own source, read off disk by the runtime that rendered it), `/receipts` (bundle size and query latency measured on the visitor's device, local and server side by side), and `/migrate` (shipping a v2 migration on demand against a local database that still holds rows and a full outbox).
|
|
16
|
+
- Initial spike: gem skeleton, engine, Syncable concern, sync protocol, demo app.
|
|
17
|
+
- Auto-mount the sync engine; models opt in by including `Syncable` (no registration).
|
|
18
|
+
- `server_only` / `browser_only` blocks for runtime-scoped model rules.
|
|
19
|
+
- `instant_record:install` generator and `instant_record:build` task; opt-in `build_on_precompile`.
|
|
20
|
+
- Ruby-owned sync loop: `InstantRecord.configure` / `start` / `sync_now`; transport moved from service-worker JS into Ruby via JS-fetch interop.
|
|
21
|
+
- SSE endpoint rewritten as a Rack streaming body (no `ActionController::Live`): no thread per stream, no database connection pinned while idle, query cache bypassed in the poll loop. Falcon documented and measured as the recommended server for SSE-heavy deployments (500 concurrent streams, p50 378ms).
|
|
22
|
+
- Browser bundle shrunk 87.2MB -> 61.7MB raw (17.1MB gzipped): `public/` unpacked, mail-family frameworks excluded, wasm-opt strip pass wired into `wasmify:pack`.
|
|
23
|
+
- `InstantRecord::RuntimeScoped`: `server_only`/`browser_only` blocks now work on any class (controllers, jobs) via `extend`; Syncable models get them automatically as before. Demo and README show server-only auth/scoping patterns.
|
|
24
|
+
- Review-driven simplification pass: sync loop consolidated into `InstantRecord::Client` (hash-native, no intra-VM JSON round-trips); fixed tick starvation — change polls now request `window=0` so a tick never holds the server's SSE tail window; cursor persisted once per poll; `MutationApplier` extracted from the mutations controller; `Change.poll` owns the connection-scoped uncached read; shared CORS/mount-path constants; SSE window configurable (`config.instant_record.sse_window_seconds`) with ±10% reconnect jitter; service worker template is single-source (demo copies via `rake pwa:sync`); gem tests run standalone and share helpers.
|
data/LICENSE.txt
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Kieran Klaassen
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in
|
|
13
|
+
all copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
|
21
|
+
THE SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,378 @@
|
|
|
1
|
+
# InstantRecord
|
|
2
|
+
|
|
3
|
+
Rails models that run in the browser, sync to your server, and keep working offline.
|
|
4
|
+
|
|
5
|
+
## The Problem
|
|
6
|
+
|
|
7
|
+
Apps like Linear feel instant because they never make you wait on the network: every read and write hits a **local** database, the UI renders from local data, and syncing with the server happens in the background. No spinners, no loading states, and offline works for free.
|
|
8
|
+
|
|
9
|
+
Building that today means a JavaScript sync engine, a client-side store, an API layer, and client-side state management — a whole second data architecture that leaves Rails behind.
|
|
10
|
+
|
|
11
|
+
InstantRecord is a bet that Rails developers can have local-first without leaving Ruby:
|
|
12
|
+
|
|
13
|
+
- **Instant** — `Issue.create!` commits to a Postgres running inside the browser and the UI re-renders immediately. The network is never in the hot path.
|
|
14
|
+
- **Durable** — every write records a mutation in the same local transaction, so nothing is lost to a crash, a reload, or a dead connection.
|
|
15
|
+
- **Convergent** — a background loop drains mutations to your Rails server (the source of truth) and streams everyone else's changes back over SSE.
|
|
16
|
+
- **Still just Rails** — the same `ApplicationRecord` model file runs on both sides. Validations, scopes, callbacks — no API client, no duplicate schema, no JavaScript data layer.
|
|
17
|
+
|
|
18
|
+
:construction: **This is a proof of concept.** The core loop is built and demonstrated: instant optimistic writes, offline-durable outbox, two clients converging over SSE, and server rejections rolling back cleanly. See [What's proven](#whats-proven).
|
|
19
|
+
|
|
20
|
+
## The Two Runtimes
|
|
21
|
+
|
|
22
|
+
The same model file loads into two Ruby runtimes, each backed by its own Postgres:
|
|
23
|
+
|
|
24
|
+
| | In the browser | On the server |
|
|
25
|
+
|---|---|---|
|
|
26
|
+
| Ruby | CRuby compiled to WebAssembly, inside a **service worker** | CRuby + Puma |
|
|
27
|
+
| Database | [PGlite](https://pglite.dev) — Postgres-in-wasm, persisted to IndexedDB | PostgreSQL |
|
|
28
|
+
| Writes | **Optimistic** — instant local commit, queued for sync | **Authoritative** — validated, versioned, logged |
|
|
29
|
+
| Your controllers & views | Run here when the page is served by the service worker | Run here at `localhost:3000` |
|
|
30
|
+
|
|
31
|
+
```mermaid
|
|
32
|
+
flowchart LR
|
|
33
|
+
subgraph tab [Your browser tab]
|
|
34
|
+
UI[Your views] --> VM["Your Rails app on ruby.wasm"]
|
|
35
|
+
VM --> PG[(PGlite)]
|
|
36
|
+
end
|
|
37
|
+
VM -->|mutations up| S[Your Rails app on a server]
|
|
38
|
+
S --> P[(Postgres)]
|
|
39
|
+
S -->|changes down over SSE| VM
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
## Usage
|
|
43
|
+
|
|
44
|
+
### 1. Install
|
|
45
|
+
|
|
46
|
+
```ruby
|
|
47
|
+
# Gemfile
|
|
48
|
+
gem "instant_record"
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
[wasmify-rails](https://github.com/palkan/wasmify-rails) comes along as a dependency — it provides the browser build tasks and the PGlite database adapter.
|
|
52
|
+
|
|
53
|
+
### 2. Make a model syncable
|
|
54
|
+
|
|
55
|
+
This is the whole opt-in:
|
|
56
|
+
|
|
57
|
+
```ruby
|
|
58
|
+
class Issue < ApplicationRecord
|
|
59
|
+
include InstantRecord::Syncable
|
|
60
|
+
|
|
61
|
+
validates :title, presence: true
|
|
62
|
+
end
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
Syncable models need three columns — a string/uuid primary key, `server_version:integer`, and `sync_state:string`. The gem's own tables (outbox, sync metadata, change log, idempotency ledger) install automatically as engine migrations — just run `bin/rails db:migrate`.
|
|
66
|
+
|
|
67
|
+
Rules that should only run on the server (authorization, quotas, anything the client shouldn't decide) stay in the same file, scoped with a block:
|
|
68
|
+
|
|
69
|
+
```ruby
|
|
70
|
+
class Issue < ApplicationRecord
|
|
71
|
+
include InstantRecord::Syncable
|
|
72
|
+
|
|
73
|
+
validates :title, presence: true
|
|
74
|
+
|
|
75
|
+
server_only do
|
|
76
|
+
validate :quota_not_exceeded # server rejects; the client rolls back
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
browser_only do
|
|
80
|
+
after_commit :flash_confetti # local-runtime behavior
|
|
81
|
+
end
|
|
82
|
+
end
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
There's no server setup: the sync engine auto-mounts at `/instant_record` on the server — `POST /mutations` (batched, idempotent writes) and `GET /events` (SSE change stream) — and every model that includes `Syncable` is syncable. Two optional knobs:
|
|
86
|
+
|
|
87
|
+
```ruby
|
|
88
|
+
# config/application.rb — move or disable the mount
|
|
89
|
+
config.instant_record.mount_path = "/sync" # or false to mount manually
|
|
90
|
+
|
|
91
|
+
# config/initializers/instant_record.rb — restrict syncing to an explicit allowlist
|
|
92
|
+
InstantRecord.sync(Issue, Todo)
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
### 3. Build the browser bundle
|
|
96
|
+
|
|
97
|
+
Your app compiles to a wasm module that boots in a service worker. One generator sets everything up — the wasm environment, the PGlite database config, and a PWA shell wired with the sync driver:
|
|
98
|
+
|
|
99
|
+
```sh
|
|
100
|
+
bin/rails generate instant_record:install
|
|
101
|
+
bin/rails instant_record:build # compiles the Ruby core (once) + packs app.wasm
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
**Deploying:** the bundle is a build artifact, like precompiled assets. Either build it in your deploy pipeline with `bin/rails instant_record:build`, or opt in to building during asset precompilation:
|
|
105
|
+
|
|
106
|
+
```ruby
|
|
107
|
+
# config/application.rb
|
|
108
|
+
config.instant_record.build_on_precompile = true
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
(Off by default — the build needs the wasm toolchain and network access for ruby.wasm artifacts.)
|
|
112
|
+
|
|
113
|
+
### 4. Write your app like it's normal Rails
|
|
114
|
+
|
|
115
|
+
No special APIs in your controllers or views. When the page is served by the service worker, this code executes **in the browser**, reading and writing the local database:
|
|
116
|
+
|
|
117
|
+
```ruby
|
|
118
|
+
class IssuesController < ApplicationController
|
|
119
|
+
def index
|
|
120
|
+
@issues = Issue.order(created_at: :desc) # instant: local Postgres read
|
|
121
|
+
end
|
|
122
|
+
|
|
123
|
+
def create
|
|
124
|
+
Issue.create!(issue_params) # instant: local commit + queued for sync
|
|
125
|
+
redirect_to root_path
|
|
126
|
+
end
|
|
127
|
+
end
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
```erb
|
|
131
|
+
<% @issues.each do |issue| %>
|
|
132
|
+
<li>
|
|
133
|
+
<%= issue.title %>
|
|
134
|
+
<span class="badge"><%= issue.sync_state %></span> <%# "pending" until the server acks %>
|
|
135
|
+
</li>
|
|
136
|
+
<% end %>
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
Useful helpers:
|
|
140
|
+
|
|
141
|
+
- `issue.sync_state` — `"pending"` (not yet acked) or `"synced"`
|
|
142
|
+
- `InstantRecord.pending_count` — outbox size, for an "unsynced changes" indicator
|
|
143
|
+
- `InstantRecord.sync_now` — force a sync pass (the background loop runs anyway)
|
|
144
|
+
- `server_only { ... }` / `browser_only { ... }` — runtime-scoped model rules
|
|
145
|
+
- `InstantRecord.browser?` — the underlying runtime check, for anywhere else
|
|
146
|
+
- `InstantRecord::Client.applying_remote?` — whether the write running right now arrived from the server, rather than being made here
|
|
147
|
+
- `InstantRecord.server_url(path)` — the URL that reaches the authoritative server from either runtime, honouring a customised `mount_path`
|
|
148
|
+
|
|
149
|
+
Convergence is last-write-wins on `updated_at`, and the value that loses is gone from the row. A model can watch for that:
|
|
150
|
+
|
|
151
|
+
```ruby
|
|
152
|
+
class Issue < ApplicationRecord
|
|
153
|
+
include InstantRecord::Syncable
|
|
154
|
+
|
|
155
|
+
# A value from the server that last-write-wins threw away. The block runs with
|
|
156
|
+
# the losing attribute hash at the only moment it exists here — nothing is
|
|
157
|
+
# assigned, no callback fires, and the gem keeps no copy.
|
|
158
|
+
on_discarded_change { |attributes| Rails.logger.info("dropped #{attributes["title"].inspect}") }
|
|
159
|
+
end
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
#### Server-only controller behavior: auth and scoping
|
|
163
|
+
|
|
164
|
+
Controllers load in both runtimes too — the browser runs them against local data, the server against Postgres. Anything that depends on sessions, cookies, or secrets exists only on the server, so scope it there:
|
|
165
|
+
|
|
166
|
+
```ruby
|
|
167
|
+
class IssuesController < ApplicationController
|
|
168
|
+
extend InstantRecord::RuntimeScoped
|
|
169
|
+
|
|
170
|
+
server_only do
|
|
171
|
+
before_action :authenticate_user! # sessions and secrets: server only
|
|
172
|
+
end
|
|
173
|
+
|
|
174
|
+
browser_only do
|
|
175
|
+
skip_forgery_protection # no CSRF secrets in the local runtime
|
|
176
|
+
end
|
|
177
|
+
|
|
178
|
+
def index
|
|
179
|
+
@issues = issues_scope.order(created_at: :desc)
|
|
180
|
+
end
|
|
181
|
+
|
|
182
|
+
private
|
|
183
|
+
|
|
184
|
+
def issues_scope
|
|
185
|
+
# Server: enforce per-user scoping. Browser: the local database already
|
|
186
|
+
# holds only what the server synced to this client, so Issue.all IS the
|
|
187
|
+
# scoped set.
|
|
188
|
+
InstantRecord.browser? ? Issue.all : current_user.issues
|
|
189
|
+
end
|
|
190
|
+
end
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
**Where authorization actually lives:** browser-side checks are UX, never enforcement — the user owns the client and everything in it. Real enforcement has exactly two places, both on the server:
|
|
194
|
+
|
|
195
|
+
1. **The mutations endpoint** — server-side validations and `server_only` model rules reject writes the client wasn't allowed to make; the client rolls back.
|
|
196
|
+
2. **What you sync down** — a client can only render what the change stream delivered to it. (Per-user scoping of the stream itself — `syncable_to` — is on the roadmap; today the PoC syncs every registered model to every client.)
|
|
197
|
+
|
|
198
|
+
A `server_only` controller filter protects the server-rendered surface; it cannot and need not protect the browser's local rendering.
|
|
199
|
+
|
|
200
|
+
### 5. Sync just happens
|
|
201
|
+
|
|
202
|
+
The sync loop is Ruby, configured in Ruby:
|
|
203
|
+
|
|
204
|
+
```ruby
|
|
205
|
+
# config/initializers/instant_record.rb (optional — these are the defaults)
|
|
206
|
+
InstantRecord.configure do |config|
|
|
207
|
+
config.endpoint = "/instant_record" # absolute URL for cross-origin setups
|
|
208
|
+
config.sync_interval = 3 # base backoff for retrying a stuck outbox
|
|
209
|
+
end
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
The browser shell calls `InstantRecord.start` once at boot; from there Ruby owns the drain, the change stream, reconnects, and reconciliation. (`InstantRecord.sync_now` forces a pass manually.) The only JavaScript left is a timer — wasm Ruby can't sleep without blocking the VM, so the service worker provides the clock and nothing else.
|
|
213
|
+
|
|
214
|
+
Behind the scenes:
|
|
215
|
+
|
|
216
|
+
- **First load** — a new client hydrates from a bootstrap snapshot (current state plus the sync cursor, windowed per model — see below) and remembers its position, so later visits paint instantly from IndexedDB and fetch only what's new.
|
|
217
|
+
- **Every write** — committed locally first, then delivered to the server in the background. Offline just means "delivered later"; queued writes survive reloads.
|
|
218
|
+
- **Other clients** — every accepted change streams to all connected clients over SSE, and their pages re-render from local data. Concurrent edits resolve last-write-wins.
|
|
219
|
+
- **Rejections** — if the server refuses a write (failed validation, server-only rule), the client rolls the local record back to server state and stops retrying.
|
|
220
|
+
|
|
221
|
+
### 6. Window large tables
|
|
222
|
+
|
|
223
|
+
Tables with unbounded history (chat messages, events, logs) don't have to sync whole. Declare a **sync window** and only the newest rows reach clients:
|
|
224
|
+
|
|
225
|
+
```ruby
|
|
226
|
+
class Message < ApplicationRecord
|
|
227
|
+
include InstantRecord::Syncable
|
|
228
|
+
|
|
229
|
+
sync_window limit: 50, partition_by: :channel_id # newest 50 per channel
|
|
230
|
+
end
|
|
231
|
+
```
|
|
232
|
+
|
|
233
|
+
Windows order by `(created_at, id)`, so a windowed model needs a `created_at` column — and an index on `(partition, created_at, id)` keeps the queries flat at depth.
|
|
234
|
+
|
|
235
|
+
Declaring a window changes three things:
|
|
236
|
+
|
|
237
|
+
- **Fresh clients bootstrap.** A client that has never synced hydrates from `GET /instant_record/bootstrap` — current state, windowed per model, plus the change-log cursor — instead of replaying the whole change log. Windowless models still serialize in full.
|
|
238
|
+
- **Boot-time eviction.** On a cold boot the browser trims each windowed model back to its window per partition. Rows with `sync_state: "pending"` are never evicted.
|
|
239
|
+
- **History pages on demand.** Older rows stream in through keyset cursors (never OFFSET), applied idempotently with no outbox noise:
|
|
240
|
+
|
|
241
|
+
```ruby
|
|
242
|
+
InstantRecord.fetch_history(Message,
|
|
243
|
+
partition: channel_id,
|
|
244
|
+
before: { created_at: oldest.created_at.iso8601(6), id: oldest.id },
|
|
245
|
+
limit: 50)
|
|
246
|
+
# => { ok: true, applied: 50, has_more: true } — or :busy while a sync pass runs
|
|
247
|
+
```
|
|
248
|
+
|
|
249
|
+
In the browser this serves repeat scrolls and offline scrollback from the local database and only hits `GET /instant_record/records` for pages it doesn't hold. Page JavaScript reaches it through a service-worker message (never from inside a request — the wasm VM can't re-enter itself):
|
|
250
|
+
|
|
251
|
+
```js
|
|
252
|
+
const channel = new MessageChannel();
|
|
253
|
+
channel.port1.onmessage = (event) => { /* {ok, has_more, error} */ };
|
|
254
|
+
navigator.serviceWorker.controller.postMessage(
|
|
255
|
+
{ type: "instant_record.fetch_history",
|
|
256
|
+
request: { type: "Message", partition: channelId,
|
|
257
|
+
before: { created_at: oldestAt, id: oldestId }, limit: 50 } },
|
|
258
|
+
[channel.port2],
|
|
259
|
+
);
|
|
260
|
+
```
|
|
261
|
+
|
|
262
|
+
**Render the top of the scrollback server-side.** A windowed view needs a slot above its oldest row — a scroll-up sentinel, or a "beginning of history" marker once there's nothing left. Render that slot in the initial HTML even when you don't yet know which it is, and let JavaScript fill it in place. Injecting it after a probe shifts every row below it by its height on each load.
|
|
263
|
+
|
|
264
|
+
The Slack demo's infinite scroll is the reference wiring — IntersectionObserver sentinel, scroll anchoring, in-place DOM updates — in the Stimulus controller `demo/app/javascript/controllers/conversation_controller.js`.
|
|
265
|
+
|
|
266
|
+
## Running the demo
|
|
267
|
+
|
|
268
|
+
Requirements: Ruby 3.3.3, Node, PostgreSQL, [wasmtime](https://wasmtime.dev) and [wasi-vfs](https://github.com/kateinoigakukun/wasi-vfs).
|
|
269
|
+
|
|
270
|
+
```sh
|
|
271
|
+
# Gem tests
|
|
272
|
+
bundle install && bundle exec rake test
|
|
273
|
+
|
|
274
|
+
# One-time setup
|
|
275
|
+
cd demo
|
|
276
|
+
bundle install
|
|
277
|
+
bin/rails db:prepare
|
|
278
|
+
bin/rails test
|
|
279
|
+
cd pwa && yarn install && cd ..
|
|
280
|
+
|
|
281
|
+
# One-time browser build (~5 min)
|
|
282
|
+
bin/rails instant_record:build # writes pwa/public/app.wasm (~62 MB)
|
|
283
|
+
|
|
284
|
+
# Run it — sync server on :3000, PWA on :5173
|
|
285
|
+
bin/dev
|
|
286
|
+
```
|
|
287
|
+
|
|
288
|
+
`bin/dev` runs both processes under foreman (see `demo/Procfile.dev`): the Rails sync server, and the Vite server that hosts the service worker, `app.wasm`, and PGlite's wasm.
|
|
289
|
+
|
|
290
|
+
Open http://localhost:5173/boot.html, wait for **Service Worker Ready**, then open http://localhost:5173/ — that page is rendered by Rails in your browser.
|
|
291
|
+
|
|
292
|
+
Things to try:
|
|
293
|
+
|
|
294
|
+
- **Instant writes** — create an issue; it appears immediately with a `pending` badge that flips to `synced`.
|
|
295
|
+
- **Two clients** — open http://127.0.0.1:5173/ too (different origin = separate service worker + separate local database). Changes propagate between windows through the server. A fresh client catches up automatically.
|
|
296
|
+
- **Infinite scroll** — `/slack` seeds thousands of messages in `#general`, but a fresh client only syncs the newest 50 per conversation. Scroll to the top and older pages stream in; messages arriving mid-read update the page without a reload or a scroll jump.
|
|
297
|
+
- **Offline** — stop the Rails server, create issues (they stay `pending`), reload the tab (still there), restart the server and watch them drain.
|
|
298
|
+
- **Rejection** — create an issue titled `reject me`. It appears optimistically, the server refuses it, and it disappears on reconcile.
|
|
299
|
+
|
|
300
|
+
## Scaling the SSE stream
|
|
301
|
+
|
|
302
|
+
Every client holds one persistent SSE connection — the service worker keeps it open in JavaScript and hands each event to Ruby, because Ruby awaiting a chunk itself would suspend the sync guard and starve outbox drains for the length of the window. Concurrent streams therefore scale with open tabs; plan your server accordingly. The engine's stream endpoint is a plain Rack streaming body (no `ActionController::Live`, no thread per stream) that releases its database connection between polls, so it runs well on both servers; the ceiling is the server's concurrency model. Measured on the demo (M-series MacBook, dev mode):
|
|
303
|
+
|
|
304
|
+
| Server | Concurrent SSE clients | Result |
|
|
305
|
+
|---|---|---|
|
|
306
|
+
| Puma (3 threads default) | 2 | Works — p50 delivery 103ms |
|
|
307
|
+
| Puma (3 threads default) | 3 | Streams consumed every thread; the sync write itself starved — 0 delivered |
|
|
308
|
+
| Puma (3 threads default) | 50 | Server unresponsive for minutes (rails/rails#55762's failure mode) |
|
|
309
|
+
| Falcon (fiber per request) | 200 | 200/200 delivered, p50 442ms, 0 errors |
|
|
310
|
+
| Falcon (fiber per request) | 500 | 500/500 delivered, p50 378ms, ~13 MB RSS, 0 errors |
|
|
311
|
+
|
|
312
|
+
**Recommendation:** Puma is fine for development and a handful of clients. For SSE-heavy deployments, run [Falcon](https://github.com/socketry/falcon) — streams become fibers, `sleep` and `pg` yield to the scheduler, and thousands of idle connections are cheap:
|
|
313
|
+
|
|
314
|
+
```ruby
|
|
315
|
+
# Gemfile
|
|
316
|
+
gem "falcon", require: false
|
|
317
|
+
```
|
|
318
|
+
|
|
319
|
+
```ruby
|
|
320
|
+
# config/application.rb — fiber-scoped execution state under Falcon only
|
|
321
|
+
config.active_support.isolation_level = :fiber if defined?(Falcon)
|
|
322
|
+
```
|
|
323
|
+
|
|
324
|
+
```sh
|
|
325
|
+
bundle exec falcon serve --bind http://localhost:3000
|
|
326
|
+
```
|
|
327
|
+
|
|
328
|
+
Browser clients receive changes over that held stream — measured at **130ms** from a server-side write to the row appearing, bounded by the server's 0.5s change-log poll rather than by any client interval. Reopening with `?after=<cursor>` replays whatever a dropped connection missed, so nothing polls for changes. The tail window is configurable via `config.instant_record.sse_window_seconds`.
|
|
329
|
+
|
|
330
|
+
There is no heartbeat in the browser: a sync pass runs on a local write, at boot, and when the network returns, then retries with backoff only while the outbox still has something in it. An idle tab issues no requests at all beyond holding its stream. Reproduce the numbers with `demo/script/sse_load_spike.rb`.
|
|
331
|
+
|
|
332
|
+
## What's proven
|
|
333
|
+
|
|
334
|
+
Measured on the demo (Chrome, M-series MacBook):
|
|
335
|
+
|
|
336
|
+
- Rails 8.1.3 boots under wasm32-wasi with the full gem bundle, including this gem.
|
|
337
|
+
- Warm boot — VM init plus PGlite schema prepare — in **~1.8 seconds**; the `app.wasm` module is **61.9 MB** raw, **17.1 MB** gzipped over the wire.
|
|
338
|
+
- Instant optimistic create with local persistence across reloads.
|
|
339
|
+
- Two independent clients converging through POST + SSE, including fresh-client catch-up.
|
|
340
|
+
- Server rejection reconciled by rolling the local record back.
|
|
341
|
+
- ~5,000 seeded messages in one channel: a fresh client bootstraps only the newest window per conversation, older pages stream in on scroll, and a cold boot evicts the local database back to the window.
|
|
342
|
+
- 226 tests (116 gem + 110 demo) covering the atomic outbox, idempotent apply, last-write-wins both ways, cursor resume, rejection reconcile, bootstrap hydration, keyset history pages, eviction, migration over a populated local store, and schema skew in both directions.
|
|
343
|
+
|
|
344
|
+
## Roadmap ideas
|
|
345
|
+
|
|
346
|
+
Sketched but **not built**:
|
|
347
|
+
|
|
348
|
+
```ruby
|
|
349
|
+
# Model-level conflict resolution beyond last-write-wins
|
|
350
|
+
class Issue < ApplicationRecord
|
|
351
|
+
include InstantRecord::Syncable
|
|
352
|
+
resolves_conflict_on :state, prefer: :closed
|
|
353
|
+
end
|
|
354
|
+
```
|
|
355
|
+
|
|
356
|
+
Also out of scope for the PoC, on purpose: CRDTs, Postgres logical replication, multi-tab leader election, attachments, authentication/authorization on the sync endpoints, and migration generators. An Inertia-compatible layer (server-seeded props hydrating the local database, plus a JS read surface via PGlite live queries) is captured as a follow-up direction in `docs/plans/`.
|
|
357
|
+
|
|
358
|
+
## History
|
|
359
|
+
|
|
360
|
+
View the [changelog](CHANGELOG.md).
|
|
361
|
+
|
|
362
|
+
## Contributing
|
|
363
|
+
|
|
364
|
+
Everyone is encouraged to help improve this project. Here are a few ways you can help:
|
|
365
|
+
|
|
366
|
+
- [Report bugs](https://github.com/kieranklaassen/instant_record/issues)
|
|
367
|
+
- Fix bugs and [submit pull requests](https://github.com/kieranklaassen/instant_record/pulls)
|
|
368
|
+
- Write, clarify, or fix documentation
|
|
369
|
+
- Suggest or add new features
|
|
370
|
+
|
|
371
|
+
To get started with development:
|
|
372
|
+
|
|
373
|
+
```sh
|
|
374
|
+
git clone https://github.com/kieranklaassen/instant_record.git
|
|
375
|
+
cd instant_record
|
|
376
|
+
bundle install
|
|
377
|
+
bundle exec rake test
|
|
378
|
+
```
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
module InstantRecord
|
|
2
|
+
# First-sync hydration: current state (windowed per model when a sync_window
|
|
3
|
+
# is declared) plus the change-log cursor, so a fresh client never replays
|
|
4
|
+
# the whole change log. The cursor is read BEFORE the rows, inside one
|
|
5
|
+
# transaction: events committed between cursor and response re-apply
|
|
6
|
+
# idempotently client-side, whereas the reverse order would skip them.
|
|
7
|
+
class BootstrapsController < ActionController::API
|
|
8
|
+
before_action { headers.merge!(InstantRecord::CORS_HEADERS) }
|
|
9
|
+
|
|
10
|
+
def show
|
|
11
|
+
payload = ActiveRecord::Base.transaction do
|
|
12
|
+
cursor = Change.maximum(:id) || 0
|
|
13
|
+
{ cursor: cursor, records: bootstrap_models.flat_map { |model| serialize_rows(model) } }
|
|
14
|
+
end
|
|
15
|
+
render json: payload
|
|
16
|
+
end
|
|
17
|
+
|
|
18
|
+
private
|
|
19
|
+
|
|
20
|
+
# Eager-load first so development doesn't miss lazily-loaded models;
|
|
21
|
+
# syncable_models then resolves like synced_model (allowlist or all
|
|
22
|
+
# Syncable includers).
|
|
23
|
+
def bootstrap_models
|
|
24
|
+
Rails.application.eager_load! unless Rails.application.config.eager_load
|
|
25
|
+
InstantRecord.syncable_models
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
def serialize_rows(model)
|
|
29
|
+
window = model.instant_record_sync_window
|
|
30
|
+
scope = window ? window.in_window : model.all
|
|
31
|
+
scope.map { |record| InstantRecord.record_payload(record) }
|
|
32
|
+
end
|
|
33
|
+
end
|
|
34
|
+
end
|
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
module InstantRecord
|
|
2
|
+
class EventsController < ActionController::Base
|
|
3
|
+
# SSE change stream as a lazily-enumerated response body — not
|
|
4
|
+
# ActionController::Live (which burns a thread per stream and bypasses the
|
|
5
|
+
# fiber scheduler). An #each body is the portable Rack streaming
|
|
6
|
+
# interface: Puma and Falcon both write each chunk as it is yielded.
|
|
7
|
+
# Under Falcon each stream is a fiber, so `sleep` and pg queries suspend
|
|
8
|
+
# the fiber; under Puma the body holds the request thread.
|
|
9
|
+
#
|
|
10
|
+
# The change-log row id is the event id, which doubles as the client's
|
|
11
|
+
# cursor: catch-up from ?after= (or the Last-Event-ID reconnect header),
|
|
12
|
+
# then tail for a bounded window and close. ?window= lets a client shorten
|
|
13
|
+
# the tail — the gem's own sync loop polls with window=0 (catch-up only)
|
|
14
|
+
# so a tick never holds a long stream.
|
|
15
|
+
EVENT_STREAM_HEADERS = {
|
|
16
|
+
"content-type" => "text/event-stream",
|
|
17
|
+
"cache-control" => "no-store",
|
|
18
|
+
"x-accel-buffering" => "no"
|
|
19
|
+
}.freeze
|
|
20
|
+
|
|
21
|
+
class EventStream
|
|
22
|
+
def initialize(cursor:, deadline:)
|
|
23
|
+
@cursor = cursor
|
|
24
|
+
@deadline = deadline
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
def each
|
|
28
|
+
loop do
|
|
29
|
+
Change.poll(@cursor).each do |change|
|
|
30
|
+
@cursor = change.id
|
|
31
|
+
yield "id: #{change.id}\nevent: change\ndata: #{change.as_event.to_json}\n\n"
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
break if Time.current >= @deadline
|
|
35
|
+
sleep 0.5
|
|
36
|
+
end
|
|
37
|
+
end
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
def index
|
|
41
|
+
cursor = (request.headers["Last-Event-ID"] || params[:after] || 0).to_i
|
|
42
|
+
|
|
43
|
+
headers.merge!(EVENT_STREAM_HEADERS).merge!(InstantRecord::CORS_HEADERS)
|
|
44
|
+
self.response_body = EventStream.new(cursor: cursor, deadline: Time.current + window_seconds)
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
private
|
|
48
|
+
|
|
49
|
+
def window_seconds
|
|
50
|
+
configured = Rails.application.config.instant_record.sse_window_seconds.to_f
|
|
51
|
+
requested = params[:window]&.to_f
|
|
52
|
+
seconds = requested ? requested.clamp(0.0, configured) : configured
|
|
53
|
+
|
|
54
|
+
# +/-10% jitter so a herd of streams opened together doesn't reconnect
|
|
55
|
+
# in lockstep at every window boundary.
|
|
56
|
+
seconds.positive? ? seconds * (0.9 + rand * 0.2) : seconds
|
|
57
|
+
end
|
|
58
|
+
end
|
|
59
|
+
end
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
module InstantRecord
|
|
2
|
+
class MutationsController < ActionController::API
|
|
3
|
+
before_action { headers.merge!(InstantRecord::CORS_HEADERS) }
|
|
4
|
+
|
|
5
|
+
def preflight
|
|
6
|
+
head :no_content
|
|
7
|
+
end
|
|
8
|
+
|
|
9
|
+
# Accepts a batch of client mutations; MutationApplier owns the sync
|
|
10
|
+
# semantics (idempotency, LWW, versioning, change logging).
|
|
11
|
+
def create
|
|
12
|
+
results = Array(params.require(:mutations)).map do |raw|
|
|
13
|
+
MutationApplier.apply(
|
|
14
|
+
raw.permit(:id, :record_type, :record_id, :operation, :base_version, changes: {})
|
|
15
|
+
)
|
|
16
|
+
end
|
|
17
|
+
|
|
18
|
+
render json: { results: results }
|
|
19
|
+
end
|
|
20
|
+
end
|
|
21
|
+
end
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
module InstantRecord
|
|
2
|
+
# Keyset-cursor history pages for windowed models: rows strictly below the
|
|
3
|
+
# (created_at, id) tuple, newest first. Backs InstantRecord.fetch_history;
|
|
4
|
+
# never uses OFFSET, so page cost stays flat at any depth.
|
|
5
|
+
class RecordsController < ActionController::API
|
|
6
|
+
# Independent of any model's window size: the ceiling a client can
|
|
7
|
+
# request in one page.
|
|
8
|
+
HARD_LIMIT = 200
|
|
9
|
+
|
|
10
|
+
before_action { headers.merge!(InstantRecord::CORS_HEADERS) }
|
|
11
|
+
|
|
12
|
+
def index
|
|
13
|
+
model = InstantRecord.synced_model(params[:type])
|
|
14
|
+
return head :not_found unless model
|
|
15
|
+
|
|
16
|
+
window = model.instant_record_sync_window
|
|
17
|
+
return head :unprocessable_entity unless window
|
|
18
|
+
return head :bad_request if window.partition_by && params[:partition].blank?
|
|
19
|
+
|
|
20
|
+
before_at = parse_time(params[:before_created_at])
|
|
21
|
+
return head :bad_request if before_at.nil? || params[:before_id].blank?
|
|
22
|
+
|
|
23
|
+
limit = (params[:limit].presence || window.limit).to_i.clamp(1, HARD_LIMIT)
|
|
24
|
+
rows = window
|
|
25
|
+
.keyset_below(at: before_at, id: params[:before_id], partition: params[:partition])
|
|
26
|
+
.limit(limit + 1)
|
|
27
|
+
.to_a
|
|
28
|
+
|
|
29
|
+
render json: {
|
|
30
|
+
records: rows.first(limit).map { |record| InstantRecord.record_payload(record) },
|
|
31
|
+
has_more: rows.size > limit
|
|
32
|
+
}
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
private
|
|
36
|
+
|
|
37
|
+
# Time.zone.parse returns nil for unparseable input but raises
|
|
38
|
+
# ArgumentError for out-of-range components ("25:61"); treat both as a bad
|
|
39
|
+
# cursor so a crafted param is a 400, not a 500.
|
|
40
|
+
def parse_time(value)
|
|
41
|
+
Time.zone.parse(value.to_s)
|
|
42
|
+
rescue ArgumentError
|
|
43
|
+
nil
|
|
44
|
+
end
|
|
45
|
+
end
|
|
46
|
+
end
|
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
module InstantRecord
|
|
2
|
+
# Idempotency ledger: one row per mutation id ever processed, storing the
|
|
3
|
+
# result so replays return the original outcome without re-applying.
|
|
4
|
+
class AppliedMutation < ActiveRecord::Base
|
|
5
|
+
self.table_name = "instant_record_applied_mutations"
|
|
6
|
+
|
|
7
|
+
serialize :result_payload, coder: JSON
|
|
8
|
+
end
|
|
9
|
+
end
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
module InstantRecord
|
|
2
|
+
# Server-side ordered change log. The auto-increment id is the SSE cursor.
|
|
3
|
+
class Change < ActiveRecord::Base
|
|
4
|
+
self.table_name = "instant_record_changes"
|
|
5
|
+
|
|
6
|
+
serialize :attributes_payload, coder: JSON
|
|
7
|
+
|
|
8
|
+
scope :after, ->(cursor) { where("id > ?", cursor.to_i).order(:id) }
|
|
9
|
+
|
|
10
|
+
# One poll of the change log for a streaming loop. Block-scoped connection
|
|
11
|
+
# checkout — an idle stream must not pin a database connection while it
|
|
12
|
+
# sleeps — and uncached, because the request executor's query cache would
|
|
13
|
+
# otherwise return the first poll's result forever.
|
|
14
|
+
def self.poll(cursor)
|
|
15
|
+
connection_pool.with_connection do
|
|
16
|
+
uncached { after(cursor).to_a }
|
|
17
|
+
end
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
def as_event
|
|
21
|
+
{
|
|
22
|
+
type: record_type,
|
|
23
|
+
id: record_id,
|
|
24
|
+
operation: operation,
|
|
25
|
+
version: version,
|
|
26
|
+
attributes: attributes_payload
|
|
27
|
+
}
|
|
28
|
+
end
|
|
29
|
+
end
|
|
30
|
+
end
|