upkeep-rails 0.1.9 → 0.1.12
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +105 -195
- data/docs/drafts/turbo-streams-is-cache-invalidation-you-write-by-hand.md +240 -0
- data/docs/handoffs/_archive/2026-06-10-main.md +47 -0
- data/docs/how-it-works.md +8 -0
- data/lib/generators/upkeep/install/install_generator.rb +59 -0
- data/lib/generators/upkeep/install/templates/subscription.js +6 -5
- data/lib/generators/upkeep/install/templates/upkeep.rb +8 -7
- data/lib/upkeep/delivery/turbo_streams.rb +40 -15
- data/lib/upkeep/dependencies.rb +55 -5
- data/lib/upkeep/invalidation/planner.rb +48 -10
- data/lib/upkeep/rails/cable/channel.rb +27 -5
- data/lib/upkeep/rails/cable/subscriber_identity.rb +21 -1
- data/lib/upkeep/rails/client_subscription.rb +12 -12
- data/lib/upkeep/rails/cluster_guard.rb +57 -0
- data/lib/upkeep/rails/configuration.rb +9 -16
- data/lib/upkeep/rails/controller_runtime.rb +17 -0
- data/lib/upkeep/rails/railtie.rb +1 -10
- data/lib/upkeep/rails/testing.rb +1 -1
- data/lib/upkeep/rails.rb +58 -17
- data/lib/upkeep/runtime.rb +39 -2
- data/lib/upkeep/shared_streams.rb +17 -3
- data/lib/upkeep/subscriptions/active_record_store.rb +53 -62
- data/lib/upkeep/subscriptions/active_record_subscription_persistence.rb +6 -3
- data/lib/upkeep/subscriptions/active_registry.rb +0 -7
- data/lib/upkeep/subscriptions/base_store.rb +106 -0
- data/lib/upkeep/subscriptions/layered_reverse_index.rb +9 -13
- data/lib/upkeep/subscriptions/lookup_instrumentation.rb +32 -0
- data/lib/upkeep/subscriptions/persistent_reverse_index.rb +4 -1
- data/lib/upkeep/subscriptions/store.rb +38 -64
- data/lib/upkeep/version.rb +1 -1
- data/upkeep-rails.gemspec +0 -1
- metadata +7 -24
- data/lib/upkeep/rails/delivery_job.rb +0 -29
- data/lib/upkeep/subscriptions/async_durable_writer.rb +0 -131
|
@@ -0,0 +1,240 @@
|
|
|
1
|
+
# Live Rails pages without writing broadcasts
|
|
2
|
+
|
|
3
|
+
---
|
|
4
|
+
|
|
5
|
+
A user posts a comment. Easy: `@comment.save`, redirect, done. Then someone asks for it to show up live, and the Turbo docs have you covered: add `broadcasts_to :post` and you're on the air. Then the comment count in the sidebar goes stale, so you add a broadcast for that. Then the moderation queue needs one. Then the author's profile page, which shows their latest activity. Six months later the `Comment` model broadcasts to four streams, and there's a comment in the code that says `# don't remove, the dashboard breaks`.
|
|
6
|
+
|
|
7
|
+
If any of this sounds familiar, you're in good company. Nobody plans this; every one of those broadcasts was the reasonable next step on the day it was written. In this post we'll look at why this pattern decays no matter how careful you are, and we'll introduce [Upkeep](https://github.com/fc-anjos/upkeep-rails), an open source gem we built that takes a different approach: instead of you telling Rails which pages a write affects, it watches your pages render and works that out on its own.
|
|
8
|
+
|
|
9
|
+
## The list nobody checks
|
|
10
|
+
|
|
11
|
+
Here's the standard Turbo Streams shape, straight from any tutorial. The write action names the parts of the UI it affects:
|
|
12
|
+
|
|
13
|
+
```ruby
|
|
14
|
+
class CardsController < ApplicationController
|
|
15
|
+
def create
|
|
16
|
+
@board = Board.find(params[:board_id])
|
|
17
|
+
@card = @board.cards.create!(card_params)
|
|
18
|
+
|
|
19
|
+
respond_to do |format|
|
|
20
|
+
format.turbo_stream
|
|
21
|
+
end
|
|
22
|
+
end
|
|
23
|
+
end
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
```erb
|
|
27
|
+
<%= turbo_stream.append "cards", partial: "card", locals: { card: @card } %>
|
|
28
|
+
<%= turbo_stream.update "board_open_count", @board.open_count %>
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
This works, and it ships. Now let's look closer at what those stream lines really are. Each one is a rule, and the rule says: when a card changes, this part of this page needs updating. You wrote that rule by hand, and here's the trouble: the app will happily let it go stale. Suppose a teammate adds an open-card count to the dashboard. It renders correctly, it ships, and it's out of date within the hour, because nobody knew the dashboard now needed a broadcast too. No test fails, since stale pages don't raise. Sooner or later a bug report arrives that says "sometimes the dashboard is behind," and those reports keep coming, because nothing in the app checks that the broadcast list still matches the UI.
|
|
32
|
+
|
|
33
|
+
The decay runs the other way too. Someone redesigns the board page and removes the open count, and the `turbo_stream.update "board_open_count"` line stays behind, broadcasting into an element that no longer exists. Rails will happily verify your routes, your migrations, and even your N+1 queries if you install a gem for it. This list, though, has no verifier at all.
|
|
34
|
+
|
|
35
|
+
If the pattern sounds familiar beyond Turbo, it should: this is cache invalidation. A hand-maintained set of "when X changes, Y is stale" rules is exactly the thing we all agreed was one of the two hard problems, and fragment caching moved away from it years ago with key-based expiration, precisely because hand-written invalidation doesn't survive contact with a changing codebase. Live updates are the same problem in a party hat.
|
|
36
|
+
|
|
37
|
+
There's also a second, trickier version of this problem: a broadcast doesn't just need the right target, it needs the right audience. If a partial renders differently for an admin than for a visitor, sending one viewer's HTML to everyone on the stream is a leak, and hand-written broadcasting leaves that entirely up to you. Keep this one in mind, because it comes back later. For now, let's stay with staleness, and with the question we kept coming back to while building live features: what if these rules could be derived from the pages themselves, instead of written by hand?
|
|
38
|
+
|
|
39
|
+
## Deriving the rules from renders
|
|
40
|
+
|
|
41
|
+
That's what Upkeep does. When a page renders during an ordinary GET request, Upkeep watches: which records were read, which columns, what query produced that list of cards. It stores that as a subscription for the page. Later, when a write commits, it checks the write against the stored subscriptions and sends Turbo Streams to exactly the browsers whose pages went stale. The create action is left with one job:
|
|
42
|
+
|
|
43
|
+
```ruby
|
|
44
|
+
class CardsController < ApplicationController
|
|
45
|
+
def create
|
|
46
|
+
Board.find(params[:board_id]).cards.create!(card_params)
|
|
47
|
+
head :no_content
|
|
48
|
+
end
|
|
49
|
+
end
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
The stream template is gone, and so is the `broadcasts_to` line in the model. Upkeep already knows the targets, because it saw them render: it plans an `append` into the card list it observed and an `update` for the count it observed, and it delivers them to the subscribed browsers when the create commits.
|
|
53
|
+
|
|
54
|
+
And when your teammate adds that open-card count to the dashboard next month? The dashboard's next render records the new dependency, and the page is live from its first request. Nobody had to remember anything, and that's the whole idea: the dependency list is rebuilt from real renders, so it can't drift out of date.
|
|
55
|
+
|
|
56
|
+
Getting started is the usual two steps:
|
|
57
|
+
|
|
58
|
+
```ruby
|
|
59
|
+
gem "upkeep-rails"
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
```sh
|
|
63
|
+
bin/rails generate upkeep:install
|
|
64
|
+
bin/rails db:migrate
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
The installer creates the subscription tables, writes an initializer, and imports the browser client. Requirements are Ruby 3.2+, Rails 7.1+, and Turbo 2.0+.
|
|
68
|
+
|
|
69
|
+
### Declaring identities: the one new concept
|
|
70
|
+
|
|
71
|
+
If your pages depend on the current user, there's one more step, and it's fair to present it as the one genuinely new concept Upkeep asks you to learn. With hand-written broadcasts, Rails never asks which session or `Current` values shaped a page, because you're the one deciding who gets each broadcast. Upkeep makes that decision for you, and it won't make it on a guess. So the mapping between what the page read at render time and what the ActionCable connection can prove at subscribe time has to come from you.
|
|
72
|
+
|
|
73
|
+
Let's set it up for a Devise app. Devise authenticates through Warden, so we declare a `:viewer` identity backed by the Warden user:
|
|
74
|
+
|
|
75
|
+
```ruby
|
|
76
|
+
# config/initializers/upkeep.rb
|
|
77
|
+
Upkeep::Rails.configure do |config|
|
|
78
|
+
config.identify :viewer, warden: :user do
|
|
79
|
+
subscribe { |connection| connection.current_user }
|
|
80
|
+
end
|
|
81
|
+
end
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
Reading it as a sentence: "when a render reads the Warden `:user`, deliver its updates only to subscribers whose connection presents the same `current_user`." The `subscribe` block is evaluated against your ActionCable connection, so the connection needs to expose that value, which is the standard Devise-and-ActionCable setup you may already have:
|
|
85
|
+
|
|
86
|
+
```ruby
|
|
87
|
+
# app/channels/application_cable/connection.rb
|
|
88
|
+
module ApplicationCable
|
|
89
|
+
class Connection < ActionCable::Connection::Base
|
|
90
|
+
identified_by :current_user
|
|
91
|
+
|
|
92
|
+
def connect
|
|
93
|
+
self.current_user = env["warden"].user
|
|
94
|
+
end
|
|
95
|
+
end
|
|
96
|
+
end
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
One more wrinkle worth handling on day one: logged-out pages. When nobody is signed in, the Warden read comes back `nil`, and Upkeep needs to know whether `nil` means "a viewer we couldn't identify" (don't share!) or "anonymous public" (share freely). You tell it with `absent_if`:
|
|
100
|
+
|
|
101
|
+
```ruby
|
|
102
|
+
config.identify :viewer, warden: :user do
|
|
103
|
+
absent_if { |value| value.nil? }
|
|
104
|
+
subscribe { |connection| connection.current_user }
|
|
105
|
+
end
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
With that in place, your public pages get the cheap shared broadcasting from the previous section, and your signed-in pages get updates scoped to the right viewer. If a page reads an identity you haven't declared, Upkeep refuses to make that page live and names the missing declaration, so a forgotten mapping shows up as a development-time message rather than as someone else's HTML.
|
|
109
|
+
|
|
110
|
+
For a typical Devise app, this is a few lines written once. If your app derives viewer state in several creative ways, budget a real hour for it, and let the refusal messages point you to each source that needs declaring. The README covers the other three shapes: `Current` attributes, session keys, and cookies.
|
|
111
|
+
|
|
112
|
+
### Where does all this live?
|
|
113
|
+
|
|
114
|
+
A reasonable question at this point: Upkeep is remembering what every open page depends on, so where does that memory go, and what keeps it from growing forever?
|
|
115
|
+
|
|
116
|
+
The installer's migration creates three tables. `upkeep_subscriptions` holds one row per subscribed page, with the page's dependency graph serialized as JSON. `upkeep_subscription_index_entries` is the reverse index that write matching runs against: table names, columns, and predicate digests, laid out so a committed write can find its candidate subscriptions with an indexed lookup instead of scanning graphs. And `upkeep_subscription_shape_index_entries` deduplicates that index across identical pages, so ten thousand people looking at the same public feed share index rows instead of multiplying them. The subscription row is written when the page renders, and the index rows are written when the browser's ActionCable connection comes up, which means half-loaded pages that never connect don't clutter the index.
|
|
117
|
+
|
|
118
|
+
Cleanup follows the same pattern Solid Cache and Solid Cable use for their tables: it rides on the gem's own traffic, with no scheduler for you to remember. Three rules keep the tables bounded. When a browser disconnects cleanly, its subscription row is deleted on the spot. While a browser stays connected, its channel touches the subscription every twenty minutes, so a dashboard left open for a week is never mistaken for garbage. And everything that hasn't been touched within the TTL, twenty-four hours by default via `config.subscription_ttl`, gets collected opportunistically: every hundredth registration prunes one bounded batch of expired rows, wrapped so a cleanup hiccup can never fail a request. Crashed tabs and dropped connections leave rows behind, and the next day's traffic quietly sweeps them out.
|
|
119
|
+
|
|
120
|
+
Deploys get a tidy answer from machinery you've already seen. Subscription shapes carry the gem version in their digest, and shared stream names carry a digest of the render recipe, so a subscription stored under last week's templates simply stops matching anything the new code broadcasts. It can't replay an old template or deliver against a renamed target; the worst it can do is nothing, until the TTL collects it. Refuse rather than guess, applied to time.
|
|
121
|
+
|
|
122
|
+
## Wait, doesn't automatic invalidation always guess wrong?
|
|
123
|
+
|
|
124
|
+
Fair question! Automatic invalidation has a deserved reputation: systems that guess which pages depend on which data tend to guess wide, refresh too much, and eventually get turned off. So let's spend a minute on why Upkeep can afford to be precise.
|
|
125
|
+
|
|
126
|
+
The trick is that Active Record queries are, most of the time, structured data. Rails builds them as Arel, and Arel can be read like a description: which table, which columns, which predicates, which ordering. Take a page that renders this:
|
|
127
|
+
|
|
128
|
+
```ruby
|
|
129
|
+
@cards = Card.where(status: "open").order(:position)
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
Upkeep reads the structure behind that relation and stores what the query means. An insert affects this page only if the new row's status is open. An update matters only if it touched `status` or `position`, or belongs to a row that's already rendered. Any other write to the cards table can churn all day without producing a single broadcast, because the stored structure proves the page doesn't care.
|
|
133
|
+
|
|
134
|
+
Let's peek under the hood, because this part is less magical than it sounds. At render time, Upkeep walks the Arel tree of every relation the page reads. The walker is a plain old `case` statement over node types (here it is from the gem's source, trimmed down):
|
|
135
|
+
|
|
136
|
+
```ruby
|
|
137
|
+
# lib/upkeep/active_record_query.rb
|
|
138
|
+
def walk(value, source: false)
|
|
139
|
+
case value
|
|
140
|
+
when Arel::Attributes::Attribute
|
|
141
|
+
attribute(value)
|
|
142
|
+
when Arel::Nodes::Equality
|
|
143
|
+
walk(value.left, source: source)
|
|
144
|
+
walk(value.right, source: source) if value.right.is_a?(Arel::Attributes::Attribute)
|
|
145
|
+
when Arel::Nodes::HomogeneousIn
|
|
146
|
+
walk(value.attribute, source: source)
|
|
147
|
+
when Arel::Table
|
|
148
|
+
table(value.name)
|
|
149
|
+
# ... more structural node types ...
|
|
150
|
+
when Arel::Nodes::StringJoin
|
|
151
|
+
opaque_table!("raw SQL join")
|
|
152
|
+
when Arel::Nodes::BoundSqlLiteral, Arel::Nodes::SqlLiteral
|
|
153
|
+
source ? opaque_table!("raw SQL source") : opaque_column!("raw SQL predicate or order expression")
|
|
154
|
+
end
|
|
155
|
+
end
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
Both halves of the story are visible in one screen. Structural nodes like `Equality` and `Table` contribute columns and tables to the page's dependency record, while a `SqlLiteral` or a string join lands in an `opaque_table!` branch, and that's the refusal we'll get to in a moment.
|
|
159
|
+
|
|
160
|
+
The other half of the mechanism runs at write time. Every stored collection dependency knows how to answer one question about a committed change, and the method that answers it fits on a napkin:
|
|
161
|
+
|
|
162
|
+
```ruby
|
|
163
|
+
# lib/upkeep/dependencies.rb
|
|
164
|
+
def matches_change?(change)
|
|
165
|
+
return false unless table_columns.key?(change.fetch(:table))
|
|
166
|
+
|
|
167
|
+
predicate_match = predicate_match(change)
|
|
168
|
+
return predicate_match unless predicate_match == UNKNOWN
|
|
169
|
+
|
|
170
|
+
return true if create_change?(change)
|
|
171
|
+
return true if delete_change?(change)
|
|
172
|
+
|
|
173
|
+
table_columns.fetch(change.fetch(:table)).intersect?(change.fetch(:changed_attributes, []))
|
|
174
|
+
end
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
Reading it top to bottom: a write to a different table never matches; if the stored predicate can decide (the new row's `status` is `"open"`, or it isn't), its answer wins; when the predicate can't decide, creates and deletes match because they can change membership, and updates match only if the changed columns overlap the ones this page depends on. That last line is the "churn all day" guarantee from a moment ago, as one `intersect?` call. When a write does match, Upkeep picks the narrowest update it can justify: an `append` for a new member, a `replace` for a changed one, and when the narrow proof isn't available, a broader but still proven re-render of the enclosing container.
|
|
178
|
+
|
|
179
|
+
Now for the important part: what happens when the query isn't structured data?
|
|
180
|
+
|
|
181
|
+
```ruby
|
|
182
|
+
Story.where("score >= 0")
|
|
183
|
+
User.joins("INNER JOIN posts ON ...")
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
Rails hands these to the database as opaque strings, so there's no structure left for Upkeep to read. Here Upkeep makes the call that shapes the whole tool: it declines to track that page. It raises in development so you find out right away, warns in production, and the page renders as ordinary, perfectly functional, non-live HTML. Often the fix is a one-line rewrite into something structural, like `Story.where(Story.arel_table[:score].gteq(0))`. Sometimes there is no structural rewrite, with full-text search against a raw `tsvector` being the classic case, and for those you opt the request out and keep the rest of the page reactive.
|
|
187
|
+
|
|
188
|
+
We'd rather give you a page that's honestly static than one that's confidently wrong. A hand-written broadcast that's out of date fails silently in production, and we've all seen how vague those bug reports get. A derived rule that can't be proven fails loudly, at development time, with a message that tells you which query to fix.
|
|
189
|
+
|
|
190
|
+
## The bonus we didn't expect: fan-out gets cheap
|
|
191
|
+
|
|
192
|
+
We built the proof machinery for correctness, and then it handed us a performance feature.
|
|
193
|
+
|
|
194
|
+
Think about the pages where live updates matter most: feeds, boards, leaderboards, dashboards. Lots of people staring at the same data. With hand-written broadcasting, the safe general pattern is to do the rendering work per stream, because nothing in the app can tell whether two subscribers are seeing identical HTML. The fan-out cost grows with your audience.
|
|
195
|
+
|
|
196
|
+
Upkeep can tell. Identity reads, meaning `Current.user`, Warden, session, and cookies, are part of what it records at render time, and so is their absence. A page that read no viewer-specific data is provably public, and "provably" is doing real work in that sentence. Here's the method that decides, straight from the gem:
|
|
197
|
+
|
|
198
|
+
```ruby
|
|
199
|
+
# lib/upkeep/shared_streams.rb
|
|
200
|
+
def identity_signature_for(graph, frame_id)
|
|
201
|
+
identity_dependencies = graph.contained_node_ids(frame_id)
|
|
202
|
+
.flat_map { |owner_id| graph.dependencies_for(owner_id) }
|
|
203
|
+
.select { |dependency| Dependencies.partitioning_identity?(dependency) }
|
|
204
|
+
.uniq(&:cache_key)
|
|
205
|
+
return "public" if identity_dependencies.empty?
|
|
206
|
+
|
|
207
|
+
Digest::SHA256.hexdigest(identity_dependencies.map(&:identity_key).sort_by(&:inspect).inspect)[0, 16]
|
|
208
|
+
end
|
|
209
|
+
```
|
|
210
|
+
|
|
211
|
+
It gathers every identity read recorded under a frame during rendering. If the list is empty, the frame is public, and public frames with the same render recipe hash to the same shared stream name. A provably public page has a pleasant property: every subscriber is seeing the same bytes. So when the data changes, Upkeep renders the update once and broadcasts it once, whether twelve browsers are watching or ten thousand.
|
|
212
|
+
|
|
213
|
+
## What Upkeep refuses to do
|
|
214
|
+
|
|
215
|
+
A tool that promises to refuse rather than guess owes you the list of what it refuses. Here it is.
|
|
216
|
+
|
|
217
|
+
| It can't track | Because | So |
|
|
218
|
+
| ------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
|
|
219
|
+
| Opaque relations: raw SQL predicates, raw joins, raw `from`, opaque order expressions | Rails no longer exposes enough structure to prove table, column, and predicate coverage | The boundary is refused and the page renders as ordinary HTML |
|
|
220
|
+
| Reads from Redis, HTTP APIs, files, globals, memoized service state | Active Record commit facts can't select these reads | They aren't live dependencies; a replay triggered by something else will pick up their current value |
|
|
221
|
+
| Writes that bypass Active Record: direct connection SQL, other datastores | Without a write fact there's nothing to match against stored subscriptions | No refresh is scheduled from that write |
|
|
222
|
+
| Viewer-specific pages whose identity you haven't declared | Upkeep won't infer who should receive HTML from naming conventions | Live registration is refused rather than risking the wrong browser |
|
|
223
|
+
| Templates that fail Herb's strict ERB parse | Narrow update targets are derived from template structure | You get broad page-level updates and a diagnostic instead of surgical ones |
|
|
224
|
+
|
|
225
|
+
Every row is the same decision applied to a different surface: where correctness can't be proven, the page behaves like the plain Rails HTML it always was. One distinction worth knowing before you read your logs: a _deoptimization_ means Upkeep found a broader target it can still prove, so the page stays live with a coarser update, while a _refusal_ means the page isn't live at all. The diagnostics tell you which one you got, and why.
|
|
226
|
+
|
|
227
|
+
## Enjoy deleting that comment
|
|
228
|
+
|
|
229
|
+
This post started with a `# don't remove, the dashboard breaks` comment guarding four hand-written broadcasts. With subscriptions derived from renders, that comment has nothing left to protect. Install the gem, take the broadcasts out, save a record, and watch the right pages update on their own.
|
|
230
|
+
|
|
231
|
+
```sh
|
|
232
|
+
bin/rails generate upkeep:install
|
|
233
|
+
bin/rails db:migrate
|
|
234
|
+
```
|
|
235
|
+
|
|
236
|
+
And the audience problem from earlier? Upkeep records every identity read a page makes, including `Current.user`, Warden, session, and cookies, and only delivers an update to subscribers who can prove the same identity over ActionCable. If a page depends on an identity you haven't declared, it refuses to make that page live rather than send viewer-specific HTML to the wrong browser.
|
|
237
|
+
|
|
238
|
+
---
|
|
239
|
+
|
|
240
|
+
_Upkeep is MIT-licensed and lives at [github.com/TODO-repo-link](https://github.com/TODO-repo-link). Issues, questions, and opaque-query war stories are all welcome._
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
---
|
|
2
|
+
created: 2026-06-10T14:25:18+00:00
|
|
3
|
+
branch: main
|
|
4
|
+
trigger: compact
|
|
5
|
+
restored: true
|
|
6
|
+
restored_at: 2026-06-10T14:27:24+00:00
|
|
7
|
+
topic: main
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
# Handoff: main
|
|
11
|
+
|
|
12
|
+
## Goal
|
|
13
|
+
|
|
14
|
+
<!-- to be enriched by LLM -->
|
|
15
|
+
|
|
16
|
+
## Current State
|
|
17
|
+
|
|
18
|
+
<!-- to be enriched by LLM -->
|
|
19
|
+
|
|
20
|
+
## Key Decisions
|
|
21
|
+
|
|
22
|
+
<!-- to be enriched by LLM -->
|
|
23
|
+
|
|
24
|
+
## Modified Files
|
|
25
|
+
|
|
26
|
+
- .claude/
|
|
27
|
+
- docs/demo/
|
|
28
|
+
- docs/handoffs/
|
|
29
|
+
- docs/known-gaps.md
|
|
30
|
+
- docs/reddit-announcement-framing.md
|
|
31
|
+
- docs/reddit-post-research.md
|
|
32
|
+
|
|
33
|
+
## Failed Approaches
|
|
34
|
+
|
|
35
|
+
<!-- to be enriched by LLM -->
|
|
36
|
+
|
|
37
|
+
## Files to Read
|
|
38
|
+
|
|
39
|
+
- (no plan or spec files found)
|
|
40
|
+
|
|
41
|
+
## Next Steps
|
|
42
|
+
|
|
43
|
+
<!-- to be enriched by LLM -->
|
|
44
|
+
|
|
45
|
+
## Open Questions
|
|
46
|
+
|
|
47
|
+
<!-- to be enriched by LLM -->
|
data/docs/how-it-works.md
CHANGED
|
@@ -100,6 +100,14 @@ Page-level fallbacks use Turbo Stream `refresh method="morph"
|
|
|
100
100
|
scroll="preserve"` instead of replacing `<html>` or writing a new document from
|
|
101
101
|
JavaScript.
|
|
102
102
|
|
|
103
|
+
When a change was committed while handling a browser request, the refresh tag
|
|
104
|
+
also carries that request's Turbo id as `request-id` (from
|
|
105
|
+
`Turbo.current_request_id` or the `X-Turbo-Request-Id` header). Turbo's client
|
|
106
|
+
ignores refreshes for its own recent requests, so a write performed during a
|
|
107
|
+
page view — view tracking, for example — cannot refresh the viewer who caused
|
|
108
|
+
it into a self-refresh loop. Writes from jobs or the console carry no request
|
|
109
|
+
id and refresh everyone.
|
|
110
|
+
|
|
103
111
|
## Deoptimization
|
|
104
112
|
|
|
105
113
|
A deoptimization means Upkeep can still prove correctness, but not the cheapest
|
|
@@ -10,6 +10,16 @@ module Upkeep
|
|
|
10
10
|
|
|
11
11
|
source_root File.expand_path("templates", __dir__)
|
|
12
12
|
|
|
13
|
+
SOLID_CABLE_DEVELOPMENT_BLOCK = <<~YAML
|
|
14
|
+
development:
|
|
15
|
+
adapter: solid_cable
|
|
16
|
+
connects_to:
|
|
17
|
+
database:
|
|
18
|
+
writing: cable
|
|
19
|
+
polling_interval: 0.1.seconds
|
|
20
|
+
message_retention: 1.day
|
|
21
|
+
YAML
|
|
22
|
+
|
|
13
23
|
def self.next_migration_number(dirname)
|
|
14
24
|
ActiveRecord::Generators::Base.next_migration_number(dirname)
|
|
15
25
|
end
|
|
@@ -37,6 +47,27 @@ module Upkeep
|
|
|
37
47
|
route %(mount ActionCable.server => "/cable")
|
|
38
48
|
end
|
|
39
49
|
|
|
50
|
+
def configure_development_cable_adapter
|
|
51
|
+
unless solid_cable_in_gemfile?
|
|
52
|
+
show_solid_cable_guidance
|
|
53
|
+
return
|
|
54
|
+
end
|
|
55
|
+
return unless cable_config_path.exist?
|
|
56
|
+
|
|
57
|
+
content = cable_config_path.read
|
|
58
|
+
block = development_cable_block(content)
|
|
59
|
+
return if block&.include?("solid_cable")
|
|
60
|
+
|
|
61
|
+
updated = if block
|
|
62
|
+
content.sub(block, "#{SOLID_CABLE_DEVELOPMENT_BLOCK}\n")
|
|
63
|
+
else
|
|
64
|
+
"#{SOLID_CABLE_DEVELOPMENT_BLOCK}\n#{content}"
|
|
65
|
+
end
|
|
66
|
+
File.write(cable_config_path, updated)
|
|
67
|
+
say "Updated config/cable.yml development to the solid_cable adapter."
|
|
68
|
+
say "Ensure config/database.yml defines a cable database for development and loads db/cable_schema.rb."
|
|
69
|
+
end
|
|
70
|
+
|
|
40
71
|
def show_identity_setup_guidance
|
|
41
72
|
usages = detected_identity_usages
|
|
42
73
|
return if usages.empty?
|
|
@@ -52,6 +83,26 @@ module Upkeep
|
|
|
52
83
|
|
|
53
84
|
private
|
|
54
85
|
|
|
86
|
+
def solid_cable_in_gemfile?
|
|
87
|
+
gemfile_path.exist? && gemfile_path.read.match?(/^\s*gem\s+["']solid_cable["']/)
|
|
88
|
+
end
|
|
89
|
+
|
|
90
|
+
def development_cable_block(content)
|
|
91
|
+
content[/^development:\n(?:(?:[ \t].*)?\n)*/]
|
|
92
|
+
end
|
|
93
|
+
|
|
94
|
+
def show_solid_cable_guidance
|
|
95
|
+
say "\nAction Cable adapter", :yellow
|
|
96
|
+
say "Upkeep delivers live updates through standard Action Cable broadcasts."
|
|
97
|
+
say "The async development cable adapter is in-process only; with a clustered server"
|
|
98
|
+
say "(puma workers > 0), broadcasts from one process never reach browsers connected to another."
|
|
99
|
+
say "Add solid_cable and point config/cable.yml development at it:"
|
|
100
|
+
say " bundle add solid_cable"
|
|
101
|
+
say " bin/rails solid_cable:install"
|
|
102
|
+
say " bin/rails db:prepare"
|
|
103
|
+
say ""
|
|
104
|
+
end
|
|
105
|
+
|
|
55
106
|
def migration_exists?(name)
|
|
56
107
|
Dir.glob(destination_path("db/migrate/*.rb")).any? do |path|
|
|
57
108
|
File.basename(path).include?(name)
|
|
@@ -120,6 +171,14 @@ module Upkeep
|
|
|
120
171
|
Pathname(destination_path("config/importmap.rb"))
|
|
121
172
|
end
|
|
122
173
|
|
|
174
|
+
def gemfile_path
|
|
175
|
+
Pathname(destination_path("Gemfile"))
|
|
176
|
+
end
|
|
177
|
+
|
|
178
|
+
def cable_config_path
|
|
179
|
+
Pathname(destination_path("config/cable.yml"))
|
|
180
|
+
end
|
|
181
|
+
|
|
123
182
|
def destination_path(path)
|
|
124
183
|
File.join(destination_root, path)
|
|
125
184
|
end
|
|
@@ -12,10 +12,11 @@ function cableConsumer() {
|
|
|
12
12
|
}
|
|
13
13
|
|
|
14
14
|
function parsePayload(element) {
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
15
|
+
return {
|
|
16
|
+
channel: element.getAttribute("channel"),
|
|
17
|
+
subscription_id: element.getAttribute("subscription-id"),
|
|
18
|
+
activation_token: element.getAttribute("activation-token"),
|
|
19
|
+
stream_name: element.getAttribute("stream-name")
|
|
19
20
|
}
|
|
20
21
|
}
|
|
21
22
|
|
|
@@ -66,7 +67,7 @@ class UpkeepSubscriptionSourceElement extends HTMLElement {
|
|
|
66
67
|
|
|
67
68
|
rejected: () => {
|
|
68
69
|
console.error(
|
|
69
|
-
"[upkeep] subscription rejected by the server;
|
|
70
|
+
"[upkeep] subscription rejected by the server; the rejection reason is in the server log",
|
|
70
71
|
{ subscription_id: payload.subscription_id, channel: payload.channel || "Upkeep::Rails::Cable::Channel" }
|
|
71
72
|
)
|
|
72
73
|
}
|
|
@@ -5,15 +5,16 @@ Upkeep::Rails.configure do |config|
|
|
|
5
5
|
|
|
6
6
|
config.enabled = app_config.fetch(:enabled, true)
|
|
7
7
|
config.subscription_store = app_config.fetch(:subscription_store, Rails.env.test? ? :memory : :active_record)
|
|
8
|
-
config.
|
|
9
|
-
config.delivery_queue = app_config.fetch(:delivery_queue, :upkeep_realtime)
|
|
8
|
+
config.deliver_inline = app_config.fetch(:deliver_inline, false)
|
|
10
9
|
|
|
11
10
|
# Delivery setup:
|
|
12
|
-
# Upkeep
|
|
13
|
-
#
|
|
14
|
-
#
|
|
15
|
-
# adapter
|
|
16
|
-
#
|
|
11
|
+
# Upkeep delivers committed changes on an in-process background dispatcher in
|
|
12
|
+
# every environment; no job backend is required. Broadcasts are standard
|
|
13
|
+
# Action Cable broadcasts, so multi-process deployments need a cross-process
|
|
14
|
+
# cable adapter (we recommend solid_cable) so a write handled by one process
|
|
15
|
+
# reaches browsers connected to another. Set config.deliver_inline = true in
|
|
16
|
+
# tests or console sessions that need delivery to run synchronously in the
|
|
17
|
+
# caller.
|
|
17
18
|
#
|
|
18
19
|
# Test setup:
|
|
19
20
|
# The generated test default is the in-process memory store. It follows the
|
|
@@ -19,10 +19,15 @@ module Upkeep
|
|
|
19
19
|
:subscriber_ids,
|
|
20
20
|
:matched_dependency_keys,
|
|
21
21
|
:deoptimization_reason,
|
|
22
|
-
:render_duration_ms
|
|
22
|
+
:render_duration_ms,
|
|
23
|
+
:request_id
|
|
23
24
|
) do
|
|
24
25
|
def to_html
|
|
25
|
-
|
|
26
|
+
if action == "refresh"
|
|
27
|
+
attributes = %(action="refresh" method="morph" scroll="preserve")
|
|
28
|
+
attributes = %(#{attributes} request-id="#{CGI.escapeHTML(request_id)}") if request_id
|
|
29
|
+
return %(<turbo-stream #{attributes}></turbo-stream>)
|
|
30
|
+
end
|
|
26
31
|
|
|
27
32
|
attributes = %(action="#{CGI.escapeHTML(action)}" targets="#{CGI.escapeHTML(target_selector)}")
|
|
28
33
|
attributes = %(#{attributes} method="morph") if morph_action?
|
|
@@ -54,7 +59,8 @@ module Upkeep
|
|
|
54
59
|
subscriber_ids: subscriber_ids,
|
|
55
60
|
matched_dependency_keys: matched_dependency_keys,
|
|
56
61
|
deoptimization_reason: deoptimization_reason,
|
|
57
|
-
render_duration_ms: render_duration_ms
|
|
62
|
+
render_duration_ms: render_duration_ms,
|
|
63
|
+
request_id: request_id
|
|
58
64
|
}
|
|
59
65
|
end
|
|
60
66
|
end
|
|
@@ -69,7 +75,15 @@ module Upkeep
|
|
|
69
75
|
end
|
|
70
76
|
|
|
71
77
|
def body
|
|
72
|
-
|
|
78
|
+
seen_refresh_tags = {}
|
|
79
|
+
streams.filter_map do |stream|
|
|
80
|
+
tag = stream.to_html
|
|
81
|
+
next tag unless stream.action == "refresh"
|
|
82
|
+
next if seen_refresh_tags.key?(tag)
|
|
83
|
+
|
|
84
|
+
seen_refresh_tags[tag] = true
|
|
85
|
+
tag
|
|
86
|
+
end.join("\n")
|
|
73
87
|
end
|
|
74
88
|
|
|
75
89
|
def report
|
|
@@ -138,7 +152,7 @@ module Upkeep
|
|
|
138
152
|
}
|
|
139
153
|
|
|
140
154
|
ActiveSupport::Notifications.instrument("build_turbo_streams.upkeep", payload) do
|
|
141
|
-
streams = plans.flat_map { |plan| stream_targets(plan.targets) }.compact
|
|
155
|
+
streams = plans.flat_map { |plan| stream_targets(plan.targets, request_id: plan.request_id) }.compact
|
|
142
156
|
batch = Batch.new(merge_streams(streams))
|
|
143
157
|
payload.merge!(payload_for(batch, rendered_streams: streams))
|
|
144
158
|
batch
|
|
@@ -161,15 +175,16 @@ module Upkeep
|
|
|
161
175
|
}
|
|
162
176
|
end
|
|
163
177
|
|
|
164
|
-
def stream_targets(planned_targets)
|
|
178
|
+
def stream_targets(planned_targets, request_id:)
|
|
165
179
|
return [] if planned_targets.empty?
|
|
166
|
-
return [stream_for(planned_targets.first)] if planned_targets.one?
|
|
180
|
+
return [stream_for(planned_targets.first, request_id: request_id)] if planned_targets.one?
|
|
167
181
|
|
|
168
182
|
planned_targets.group_by { |planned_target| render_group_key(planned_target) }.map do |_key, targets|
|
|
169
183
|
stream_for(
|
|
170
184
|
targets.first,
|
|
171
185
|
subscriber_ids: targets.flat_map(&:subscriber_ids),
|
|
172
|
-
matched_dependency_keys: targets.flat_map(&:matched_dependency_keys)
|
|
186
|
+
matched_dependency_keys: targets.flat_map(&:matched_dependency_keys),
|
|
187
|
+
request_id: request_id
|
|
173
188
|
)
|
|
174
189
|
end
|
|
175
190
|
end
|
|
@@ -177,8 +192,8 @@ module Upkeep
|
|
|
177
192
|
# The write that produced these changes has already committed; an isolated render/targeting
|
|
178
193
|
# failure for one target must never propagate back into the writer's request. Rescue per
|
|
179
194
|
# target, surface the failure via instrumentation, and keep delivering the other targets.
|
|
180
|
-
def stream_for(planned_target, subscriber_ids: planned_target.subscriber_ids, matched_dependency_keys: planned_target.matched_dependency_keys)
|
|
181
|
-
build_stream(planned_target, subscriber_ids: subscriber_ids, matched_dependency_keys: matched_dependency_keys)
|
|
195
|
+
def stream_for(planned_target, request_id:, subscriber_ids: planned_target.subscriber_ids, matched_dependency_keys: planned_target.matched_dependency_keys)
|
|
196
|
+
build_stream(planned_target, subscriber_ids: subscriber_ids, matched_dependency_keys: matched_dependency_keys, request_id: request_id)
|
|
182
197
|
rescue StandardError => error
|
|
183
198
|
ActiveSupport::Notifications.instrument(
|
|
184
199
|
DELIVERY_ERROR,
|
|
@@ -192,7 +207,7 @@ module Upkeep
|
|
|
192
207
|
nil
|
|
193
208
|
end
|
|
194
209
|
|
|
195
|
-
def build_stream(planned_target, subscriber_ids:, matched_dependency_keys:)
|
|
210
|
+
def build_stream(planned_target, subscriber_ids:, matched_dependency_keys:, request_id:)
|
|
196
211
|
html, render_duration_ms = render_target(planned_target)
|
|
197
212
|
|
|
198
213
|
Stream.new(
|
|
@@ -206,7 +221,10 @@ module Upkeep
|
|
|
206
221
|
subscriber_ids.uniq.sort_by(&:to_s),
|
|
207
222
|
matched_dependency_keys.uniq,
|
|
208
223
|
planned_target.deoptimization_reason,
|
|
209
|
-
render_duration_ms
|
|
224
|
+
render_duration_ms,
|
|
225
|
+
# Only Turbo's refresh stream action consults request-id; keeping it nil on
|
|
226
|
+
# rendered streams lets identical payloads from different requests dedup.
|
|
227
|
+
planned_target.action == "refresh" ? request_id : nil
|
|
210
228
|
)
|
|
211
229
|
end
|
|
212
230
|
|
|
@@ -226,11 +244,15 @@ module Upkeep
|
|
|
226
244
|
planned_target.target.id,
|
|
227
245
|
planned_target.identity_signature,
|
|
228
246
|
planned_target.sharing_signature,
|
|
247
|
+
planned_target.deployment_signature,
|
|
229
248
|
SharedStreams.signature_for(planned_target.recipe),
|
|
230
249
|
planned_target.deoptimization_reason
|
|
231
250
|
]
|
|
232
251
|
end
|
|
233
252
|
|
|
253
|
+
# request_id splits the key only for refresh streams (nil everywhere else):
|
|
254
|
+
# refreshes from distinct requests must stay separate tags so each originating
|
|
255
|
+
# client skips only its own and still refreshes for the other request's change.
|
|
234
256
|
def merge_streams(streams)
|
|
235
257
|
streams.each_with_object({}) do |stream, indexed_streams|
|
|
236
258
|
key = [
|
|
@@ -240,7 +262,8 @@ module Upkeep
|
|
|
240
262
|
stream.identity_signature,
|
|
241
263
|
stream.shared_stream_name,
|
|
242
264
|
stream.html_digest,
|
|
243
|
-
stream.deoptimization_reason
|
|
265
|
+
stream.deoptimization_reason,
|
|
266
|
+
stream.request_id
|
|
244
267
|
]
|
|
245
268
|
indexed_streams[key] = merge_stream(indexed_streams[key], stream)
|
|
246
269
|
end.values
|
|
@@ -260,7 +283,8 @@ module Upkeep
|
|
|
260
283
|
(existing.subscriber_ids + stream.subscriber_ids).uniq.sort_by(&:to_s),
|
|
261
284
|
(existing.matched_dependency_keys + stream.matched_dependency_keys).uniq,
|
|
262
285
|
existing.deoptimization_reason,
|
|
263
|
-
(existing.render_duration_ms + stream.render_duration_ms).round(3)
|
|
286
|
+
(existing.render_duration_ms + stream.render_duration_ms).round(3),
|
|
287
|
+
existing.request_id
|
|
264
288
|
)
|
|
265
289
|
end
|
|
266
290
|
|
|
@@ -275,7 +299,8 @@ module Upkeep
|
|
|
275
299
|
SharedStreams.stream_name(
|
|
276
300
|
target: planned_target.shared_stream_target,
|
|
277
301
|
identity_signature: planned_target.identity_signature,
|
|
278
|
-
sharing_signature: planned_target.sharing_signature
|
|
302
|
+
sharing_signature: planned_target.sharing_signature,
|
|
303
|
+
deployment_signature: planned_target.deployment_signature
|
|
279
304
|
)
|
|
280
305
|
end
|
|
281
306
|
|