upkeep-rails 0.2.5-aarch64-linux-gnu
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/LICENSE.txt +21 -0
- data/README.md +244 -0
- data/docs/drafts/turbo-streams-is-cache-invalidation-you-write-by-hand.md +240 -0
- data/docs/how-it-works.md +329 -0
- data/docs/plans/sqlglot-active-record-migration.md +211 -0
- data/docs/plans/turbo-frame-subscription-composition.md +90 -0
- data/docs/spikes/SQLGLOT_VS_AREL_FINDINGS.md +136 -0
- data/docs/spikes/query-source-analysis-comparison/FINDINGS.md +169 -0
- data/docs/spikes/sqlglot-first-active-record/README.md +26 -0
- data/docs/spikes/sqlglot-query-analysis/FINDINGS.md +110 -0
- data/docs/spikes/sqlglot-query-analysis/PULSE_1802_FINDINGS.md +95 -0
- data/docs/spikes/sqlglot-semantic-bindings/README.md +29 -0
- data/lib/generators/upkeep/install/install_generator.rb +192 -0
- data/lib/generators/upkeep/install/templates/create_upkeep_subscriptions.rb.erb +49 -0
- data/lib/generators/upkeep/install/templates/subscription.js +288 -0
- data/lib/generators/upkeep/install/templates/upkeep.rb +65 -0
- data/lib/upkeep/active_record_query.rb +248 -0
- data/lib/upkeep/capture/request.rb +150 -0
- data/lib/upkeep/dag/subscription_shape.rb +244 -0
- data/lib/upkeep/dag.rb +454 -0
- data/lib/upkeep/delivery/action_cable_adapter.rb +48 -0
- data/lib/upkeep/delivery/async_dispatcher.rb +102 -0
- data/lib/upkeep/delivery/broadcast_transport.rb +89 -0
- data/lib/upkeep/delivery/transport.rb +194 -0
- data/lib/upkeep/delivery/turbo_streams.rb +339 -0
- data/lib/upkeep/delivery.rb +7 -0
- data/lib/upkeep/dependencies.rb +600 -0
- data/lib/upkeep/herb/developer_report.rb +135 -0
- data/lib/upkeep/herb/manifest_cache.rb +83 -0
- data/lib/upkeep/herb/manifest_diff.rb +183 -0
- data/lib/upkeep/herb/source_instrumenter.rb +149 -0
- data/lib/upkeep/herb/template_manifest.rb +548 -0
- data/lib/upkeep/invalidation/collection_append.rb +84 -0
- data/lib/upkeep/invalidation/collection_member_replace.rb +78 -0
- data/lib/upkeep/invalidation/collection_prepend.rb +84 -0
- data/lib/upkeep/invalidation/collection_remove.rb +57 -0
- data/lib/upkeep/invalidation/planner.rb +411 -0
- data/lib/upkeep/invalidation.rb +7 -0
- data/lib/upkeep/rails/action_view_capture.rb +1007 -0
- data/lib/upkeep/rails/activation_token.rb +55 -0
- data/lib/upkeep/rails/cable/channel.rb +165 -0
- data/lib/upkeep/rails/cable/subscriber_identity.rb +361 -0
- data/lib/upkeep/rails/cable.rb +4 -0
- data/lib/upkeep/rails/client_subscription.rb +65 -0
- data/lib/upkeep/rails/cluster_guard.rb +57 -0
- data/lib/upkeep/rails/configuration.rb +252 -0
- data/lib/upkeep/rails/controller_runtime.rb +187 -0
- data/lib/upkeep/rails/install.rb +28 -0
- data/lib/upkeep/rails/job_runtime.rb +43 -0
- data/lib/upkeep/rails/railtie.rb +44 -0
- data/lib/upkeep/rails/replay.rb +244 -0
- data/lib/upkeep/rails/testing.rb +259 -0
- data/lib/upkeep/rails.rb +466 -0
- data/lib/upkeep/replay.rb +462 -0
- data/lib/upkeep/runtime.rb +1276 -0
- data/lib/upkeep/shared_streams.rb +86 -0
- data/lib/upkeep/sql_dependency_analysis.rb +553 -0
- data/lib/upkeep/sqlglot/libsqlglot_rust.so +0 -0
- data/lib/upkeep/sqlglot/native.rb +121 -0
- data/lib/upkeep/sqlglot/native_library.rb +23 -0
- data/lib/upkeep/sqlglot.rb +367 -0
- data/lib/upkeep/subscriptions/active_record_store.rb +398 -0
- data/lib/upkeep/subscriptions/active_record_subscription_persistence.rb +411 -0
- data/lib/upkeep/subscriptions/active_registry.rb +80 -0
- data/lib/upkeep/subscriptions/base_store.rb +110 -0
- data/lib/upkeep/subscriptions/json_snapshot.rb +98 -0
- data/lib/upkeep/subscriptions/layered_reverse_index.rb +125 -0
- data/lib/upkeep/subscriptions/lookup_instrumentation.rb +32 -0
- data/lib/upkeep/subscriptions/persistent_reverse_index.rb +228 -0
- data/lib/upkeep/subscriptions/registrar.rb +36 -0
- data/lib/upkeep/subscriptions/reverse_index.rb +313 -0
- data/lib/upkeep/subscriptions/shape.rb +117 -0
- data/lib/upkeep/subscriptions/store.rb +349 -0
- data/lib/upkeep/subscriptions.rb +7 -0
- data/lib/upkeep/targeting.rb +146 -0
- data/lib/upkeep/version.rb +5 -0
- data/lib/upkeep-rails.rb +3 -0
- data/lib/upkeep.rb +15 -0
- data/upkeep-rails.gemspec +66 -0
- metadata +327 -0
|
@@ -0,0 +1,329 @@
|
|
|
1
|
+
# How Upkeep Works
|
|
2
|
+
|
|
3
|
+
This document explains the runtime model behind Upkeep Rails. It intentionally
|
|
4
|
+
does not cover installation or common configuration; use the
|
|
5
|
+
[README](../README.md) for that public API surface.
|
|
6
|
+
|
|
7
|
+
## Rendered Pages
|
|
8
|
+
|
|
9
|
+
A rendered page is a successful HTML GET that Upkeep can keep fresh. The
|
|
10
|
+
request runs normally through Rails. Upkeep observes the controller, Action View
|
|
11
|
+
rendering, Active Record reads, request inputs, and identity inputs used by the
|
|
12
|
+
response.
|
|
13
|
+
|
|
14
|
+
Upkeep only captures successful HTML responses. Non-HTML responses, redirects,
|
|
15
|
+
failed responses, and explicit non-page interactions continue to behave like
|
|
16
|
+
ordinary Rails responses.
|
|
17
|
+
|
|
18
|
+
## Frames
|
|
19
|
+
|
|
20
|
+
A frame is a rendered page, template, partial, collection render site, or
|
|
21
|
+
fragment with a stable delivery target.
|
|
22
|
+
|
|
23
|
+
Frames let Upkeep refresh a specific part of the page instead of replaying the
|
|
24
|
+
whole response when a narrower update is proven safe. A page frame is the broad
|
|
25
|
+
fallback. A render-site or fragment frame is narrower.
|
|
26
|
+
|
|
27
|
+
Upkeep instruments Action View templates and adds internal `data-upkeep-*`
|
|
28
|
+
markers for page roots, fragment roots, and safe collection render-site
|
|
29
|
+
containers. Normal templates do not need to call helper APIs directly.
|
|
30
|
+
|
|
31
|
+
The `upkeep_frame` helper is an advanced escape hatch for generated or
|
|
32
|
+
helper-built boundaries that cannot be derived from template source. Ordinary
|
|
33
|
+
ERB and partial collections should not need it.
|
|
34
|
+
|
|
35
|
+
### Controller Work and Render Regions
|
|
36
|
+
|
|
37
|
+
Dependencies inherit the frame that is active when the read occurs. A query
|
|
38
|
+
inside a partial or `upkeep_frame` belongs to that render region. A query in a
|
|
39
|
+
controller callback or action runs before rendering, so its narrowest sound
|
|
40
|
+
boundary is the page:
|
|
41
|
+
|
|
42
|
+
```ruby
|
|
43
|
+
before_action do
|
|
44
|
+
@show_reminder = !current_user.time_logs.where(log_date: Date.current).exists?
|
|
45
|
+
end
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
Rails does not record which later template expression consumes an instance
|
|
49
|
+
variable. Ruby may also pass or access that value dynamically, so Upkeep cannot
|
|
50
|
+
reliably infer that `@show_reminder` affects only one element. A matching
|
|
51
|
+
`TimeLog` change can therefore select a page replay.
|
|
52
|
+
|
|
53
|
+
When controller work exists only to render one section, execute it inside that
|
|
54
|
+
section instead:
|
|
55
|
+
|
|
56
|
+
```erb
|
|
57
|
+
<%= upkeep_frame "layout/time-log-reminder" do %>
|
|
58
|
+
<div data-upkeep-render-site="layout/time-log-reminder">
|
|
59
|
+
<% if time_log_reminder_data %>
|
|
60
|
+
<%= render "shared/time_log_reminder" %>
|
|
61
|
+
<% end %>
|
|
62
|
+
</div>
|
|
63
|
+
<% end %>
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
The helper or partial can perform the query while this frame is active. Upkeep
|
|
67
|
+
then associates the dependency with the reminder region instead of the page.
|
|
68
|
+
The region must always render a stable target, including when its conditional
|
|
69
|
+
content is empty, so it can transition in either direction.
|
|
70
|
+
|
|
71
|
+
This is a granularity rule, not a refusal: controller-level reads remain live
|
|
72
|
+
through a conservative page replay. Use an explicit region only when that
|
|
73
|
+
broader replay is undesirable and the application knows the true presentation
|
|
74
|
+
boundary.
|
|
75
|
+
|
|
76
|
+
## Surfaces
|
|
77
|
+
|
|
78
|
+
A surface is the set of facts about future writes that would make a frame
|
|
79
|
+
stale.
|
|
80
|
+
|
|
81
|
+
For Active Record, Upkeep derives surfaces from observed record attributes,
|
|
82
|
+
rendered collections, and relation shape where Rails exposes structural Arel
|
|
83
|
+
queries. A rendered collection of open cards ordered by position produces a
|
|
84
|
+
surface tied to the cards table, the columns that decide membership and order,
|
|
85
|
+
and the records rendered in that collection.
|
|
86
|
+
|
|
87
|
+
When a write commits, Upkeep compares the write facts with registered surfaces.
|
|
88
|
+
Only frames whose surfaces can be affected are selected for delivery.
|
|
89
|
+
|
|
90
|
+
Controller requests and Active Job executions both establish a change-capture
|
|
91
|
+
boundary. After the boundary completes, Upkeep plans and delivers updates for
|
|
92
|
+
the committed Active Record changes. Sidekiq jobs using the Active Job adapter
|
|
93
|
+
are covered by this lifecycle; direct Sidekiq workers are not.
|
|
94
|
+
|
|
95
|
+
## Identity Boundaries
|
|
96
|
+
|
|
97
|
+
An identity boundary is state that decides who may receive a live update.
|
|
98
|
+
|
|
99
|
+
Upkeep records observed CurrentAttributes, Warden, session, cookie, and request
|
|
100
|
+
reads for replay and sharing. It does not infer subscriber identity by naming
|
|
101
|
+
convention. The app declares which render-time value maps to which
|
|
102
|
+
subscribe-time ActionCable value.
|
|
103
|
+
|
|
104
|
+
The safety rule is simple: if rendered output depends on a non-public identity,
|
|
105
|
+
only subscribers proving the same identity may receive that output. If Upkeep
|
|
106
|
+
cannot identify the boundary, it refuses live registration rather than sending
|
|
107
|
+
viewer-specific HTML to the wrong browser.
|
|
108
|
+
|
|
109
|
+
Absent identities are public. For example, if a logged-out page reads a nil
|
|
110
|
+
viewer, that nil value can be treated as anonymous-public instead of
|
|
111
|
+
subscriber-specific.
|
|
112
|
+
|
|
113
|
+
## Subscriptions
|
|
114
|
+
|
|
115
|
+
A subscription is the browser's live connection back to the captured page.
|
|
116
|
+
|
|
117
|
+
Upkeep injects a body-scoped `<upkeep-subscription-source>` marker into
|
|
118
|
+
successful HTML responses. The generated browser bootstrap upgrades that marker
|
|
119
|
+
into a Turbo stream source, subscribes over ActionCable, and lets Turbo process
|
|
120
|
+
received stream payloads.
|
|
121
|
+
|
|
122
|
+
The server stores a replayable subscription graph for the rendered page. The
|
|
123
|
+
graph contains frames, dependencies, target metadata, replay recipes, request
|
|
124
|
+
inputs, and identity information needed to plan later updates.
|
|
125
|
+
|
|
126
|
+
Turbo Frames are graph scopes inside that page subscription. A frame response
|
|
127
|
+
replaces the matching scope while the shell and sibling scopes remain intact.
|
|
128
|
+
The browser connects the composed subscription before disconnecting the
|
|
129
|
+
previous one, so a frame visit never leaves the visible page without its full
|
|
130
|
+
dependency graph.
|
|
131
|
+
|
|
132
|
+
## Proven Delivery
|
|
133
|
+
|
|
134
|
+
Proven delivery means Upkeep only emits the narrowest Turbo operation it can
|
|
135
|
+
justify.
|
|
136
|
+
|
|
137
|
+
Depending on the proof available, delivery may use:
|
|
138
|
+
|
|
139
|
+
- `append`
|
|
140
|
+
- `prepend`
|
|
141
|
+
- `remove`
|
|
142
|
+
- `replace`
|
|
143
|
+
- `update`
|
|
144
|
+
- Turbo page `refresh`
|
|
145
|
+
|
|
146
|
+
Render-site replays use Turbo Stream `update method="morph"` against the real
|
|
147
|
+
HTML element Upkeep marked as the render site. The stream template is the
|
|
148
|
+
render site's children, so `update` preserves the legal container element and
|
|
149
|
+
swaps its contents.
|
|
150
|
+
|
|
151
|
+
Page-level fallbacks use Turbo Stream `refresh method="morph"
|
|
152
|
+
scroll="preserve"` instead of replacing `<html>` or writing a new document from
|
|
153
|
+
JavaScript.
|
|
154
|
+
|
|
155
|
+
When a change was committed while handling a GET or HEAD request, the refresh
|
|
156
|
+
tag also carries that request's Turbo id as `request-id` (from
|
|
157
|
+
`Turbo.current_request_id` or the `X-Turbo-Request-Id` header). Turbo 8's client
|
|
158
|
+
ignores refreshes for its own recent requests, so view tracking cannot refresh
|
|
159
|
+
the viewer who caused it into a self-refresh loop. Mutations omit the request id
|
|
160
|
+
so the originating tab still refreshes when its response does not render every
|
|
161
|
+
affected region. Writes from jobs or the console also refresh everyone.
|
|
162
|
+
|
|
163
|
+
## Deoptimization
|
|
164
|
+
|
|
165
|
+
A deoptimization means Upkeep can still prove correctness, but not the cheapest
|
|
166
|
+
operation.
|
|
167
|
+
|
|
168
|
+
For example, a collection member update might not have enough proof for a
|
|
169
|
+
single member `replace`, but the enclosing render site might still be safe to
|
|
170
|
+
rerender. In that case, the page remains live and Upkeep falls back to the
|
|
171
|
+
broader proven target.
|
|
172
|
+
|
|
173
|
+
Planning and delivery telemetry record deoptimization reasons so benchmarks and
|
|
174
|
+
tests can separate safety fallbacks from true refusals.
|
|
175
|
+
|
|
176
|
+
## Refused Boundaries
|
|
177
|
+
|
|
178
|
+
A refused boundary means Upkeep cannot prove correctness.
|
|
179
|
+
|
|
180
|
+
If Upkeep cannot answer which future write facts can make a rendered result
|
|
181
|
+
stale, which target can be replayed or patched, or which identity inputs decide
|
|
182
|
+
sharing, it refuses the live boundary.
|
|
183
|
+
|
|
184
|
+
This is intentional. A boundary that cannot be proven should behave like
|
|
185
|
+
ordinary Rails HTML instead of registering a broad or unsafe live dependency.
|
|
186
|
+
|
|
187
|
+
Refusal is different from deoptimization:
|
|
188
|
+
|
|
189
|
+
- refusal: Upkeep cannot prove correctness, so the boundary is not live
|
|
190
|
+
- deoptimization: Upkeep can prove correctness through a broader target, so the
|
|
191
|
+
boundary remains live
|
|
192
|
+
|
|
193
|
+
## What Upkeep Observes
|
|
194
|
+
|
|
195
|
+
Render structure:
|
|
196
|
+
|
|
197
|
+
- Rails-resolved page templates
|
|
198
|
+
- partial and object partial renders
|
|
199
|
+
- Action View-instrumented collection render sites and child fragments
|
|
200
|
+
- polymorphic `render @records` collection shorthand when runtime rendering
|
|
201
|
+
confirms a collection
|
|
202
|
+
- `tag.*` and `content_tag` containers lowered by Herb into ordinary template
|
|
203
|
+
structure
|
|
204
|
+
- single-root fragment targets and legal render-site container targets
|
|
205
|
+
|
|
206
|
+
Template parsing:
|
|
207
|
+
|
|
208
|
+
- Upkeep plans narrow source-derived targets only from templates that pass
|
|
209
|
+
Herb's strict parser.
|
|
210
|
+
- If strict parsing fails but Herb can recover with `strict: false`, Upkeep
|
|
211
|
+
reports the strict parser diagnostics as warnings and may still add broad
|
|
212
|
+
page or fragment root markers.
|
|
213
|
+
- Recovered render sites are diagnostic only. Fix strict warnings before
|
|
214
|
+
expecting narrow collection updates from that template.
|
|
215
|
+
|
|
216
|
+
Data dependencies:
|
|
217
|
+
|
|
218
|
+
- Active Record attribute reads
|
|
219
|
+
- Active Record relation collection renders
|
|
220
|
+
- Active Record callback writes
|
|
221
|
+
- supported bulk `update_all` and `delete_all` writes
|
|
222
|
+
- relation table and column coverage derived from Arel where Rails exposes a
|
|
223
|
+
structural query shape
|
|
224
|
+
|
|
225
|
+
Identity and ambient inputs:
|
|
226
|
+
|
|
227
|
+
- `ActiveSupport::CurrentAttributes` reads
|
|
228
|
+
- Warden and Devise user reads through Warden
|
|
229
|
+
- session and cookie reads
|
|
230
|
+
- request values such as host, path, params, user agent, and remote IP
|
|
231
|
+
- declared Upkeep identities that map observed render-time values to
|
|
232
|
+
ActionCable subscribe-time values
|
|
233
|
+
|
|
234
|
+
## What Upkeep Cannot Capture
|
|
235
|
+
|
|
236
|
+
Upkeep captures reactive facts, not arbitrary Ruby execution. A boundary is
|
|
237
|
+
capturable only when Upkeep can prove the future write facts that affect it,
|
|
238
|
+
the target that can be replayed or patched, and the identity inputs that decide
|
|
239
|
+
whether it can be shared.
|
|
240
|
+
|
|
241
|
+
These surfaces are not capturable today:
|
|
242
|
+
|
|
243
|
+
| Surface | Why it is not capturable | Runtime behavior |
|
|
244
|
+
| --- | --- | --- |
|
|
245
|
+
| Opaque Active Record relations: raw SQL predicates, raw joins, raw `from` sources, unknown table aliases, opaque order expressions, or opaque pluck columns. | Rails no longer exposes enough structure to prove table, column, predicate, and lifecycle coverage. | Upkeep refuses the live boundary instead of widening to an unsafe dependency. |
|
|
246
|
+
| Controller queries that are never rendered as a collection boundary. | There is no DOM collection surface where membership can be appended, removed, prepended, or replaced. | The page can still render normally. Scalar relation output may be tracked as a page-level dependency, but it does not unlock collection stream planning. |
|
|
247
|
+
| Reads from external stores or process state: Redis, HTTP APIs, files, global variables, class variables, singleton caches, background thread state, or service memoization. | Active Record commit facts cannot select these reads, and Upkeep has no source adapter for their lifecycle. | They are not live dependencies. If another observed dependency causes a replay, normal Rails code may read the new value during that replay. |
|
|
248
|
+
| Writes outside observed Active Record paths: direct connection SQL, writes in another datastore, or side effects that do not emit Upkeep change facts. | Upkeep cannot match a future change to an existing surface without a write fact. | No refresh is scheduled from that write. |
|
|
249
|
+
| Replay inputs that cannot be rebuilt: arbitrary objects, procs, IO handles, open clients, or values that only exist in one Ruby process. | A captured target must be replayable later, often in a different request context. | Non-replayable values block the narrow replay path until represented as stable data. |
|
|
250
|
+
| Patch targets Upkeep cannot identify in rendered HTML. | Delivery needs a stable page, render-site, fragment, or member target. | Upkeep uses the narrowest proven target. If no safe target exists, the boundary is refused. |
|
|
251
|
+
|
|
252
|
+
## Query Shapes
|
|
253
|
+
|
|
254
|
+
Collection dependencies are accepted only with proven column coverage. Opaque
|
|
255
|
+
predicates or table-only sources are refused instead of widening into broad
|
|
256
|
+
invalidation.
|
|
257
|
+
|
|
258
|
+
Controller materialization is supported when the rendered value keeps a
|
|
259
|
+
structural relation proof:
|
|
260
|
+
|
|
261
|
+
```ruby
|
|
262
|
+
def index
|
|
263
|
+
@cards = Card.where(status: "open").order(:position).to_a
|
|
264
|
+
end
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
```erb
|
|
268
|
+
<%= render partial: "cards/card", collection: @cards, as: :card %>
|
|
269
|
+
```
|
|
270
|
+
|
|
271
|
+
Upkeep attaches the collection dependency to the rendered collection boundary,
|
|
272
|
+
not to every controller query. A materialized relation that is never rendered as
|
|
273
|
+
a collection is not a lifecycle dependency by itself.
|
|
274
|
+
|
|
275
|
+
Scalar relation output is tracked as a page-level query dependency:
|
|
276
|
+
|
|
277
|
+
```ruby
|
|
278
|
+
@tag_names = Tag.where(active: true).pluck(:name)
|
|
279
|
+
```
|
|
280
|
+
|
|
281
|
+
Simple plucked columns are live and can select a page replay when they change.
|
|
282
|
+
They are not collection dependencies, so they do not participate in
|
|
283
|
+
append/remove/prepend planning.
|
|
284
|
+
|
|
285
|
+
## Testing Model
|
|
286
|
+
|
|
287
|
+
Use `Upkeep::Rails::Testing` for app-level assertions around subscription
|
|
288
|
+
registration and delivery.
|
|
289
|
+
|
|
290
|
+
Structure tests around behavior, not store internals:
|
|
291
|
+
|
|
292
|
+
- Most request and system tests can run against the memory store. Memory has
|
|
293
|
+
the same public lifecycle as ActiveRecord: registration is fetchable
|
|
294
|
+
immediately, lookup visibility starts on activation, touch updates liveness,
|
|
295
|
+
unregister and prune remove lookup entries, and delivery uses the same
|
|
296
|
+
planner surface.
|
|
297
|
+
- Keep a smaller ActiveRecord-backed integration slice for production-only
|
|
298
|
+
concerns: generated migration shape, schema validation, durable rows,
|
|
299
|
+
reload and rehydration, async persistence, and cross-process lookup.
|
|
300
|
+
- Do not assert implementation details that are unique to one store unless the
|
|
301
|
+
test is explicitly about that implementation. For app behavior, assert the
|
|
302
|
+
marker, activation, streams, broadcasts, and rendered bytes.
|
|
303
|
+
|
|
304
|
+
Useful helpers:
|
|
305
|
+
|
|
306
|
+
- `assert_upkeep_subscription_registered`
|
|
307
|
+
- `upkeep_subscription`
|
|
308
|
+
- `upkeep_stream_names`
|
|
309
|
+
- `activate_upkeep_subscription!`
|
|
310
|
+
- `capture_upkeep_broadcasts`
|
|
311
|
+
- `drain_upkeep_delivery!`
|
|
312
|
+
- `capture_upkeep_change_facts`
|
|
313
|
+
- `upkeep_match_report`
|
|
314
|
+
|
|
315
|
+
Use `capture_upkeep_broadcasts` when an app test needs to assert rendered
|
|
316
|
+
Turbo Stream payloads without depending on the host app's Action Cable test
|
|
317
|
+
adapter. The helper captures Upkeep delivery after planning and rendering, but
|
|
318
|
+
before the transport broadcasts.
|
|
319
|
+
|
|
320
|
+
Use `capture_upkeep_change_facts` and `upkeep_match_report` when debugging an
|
|
321
|
+
invalidation miss. Capture the committed facts produced by the request, then
|
|
322
|
+
dry-run them against the current subscription store. The report returns the
|
|
323
|
+
candidate count, matched count, miss reason, and render targets without
|
|
324
|
+
broadcasting.
|
|
325
|
+
|
|
326
|
+
For structural subscription debugging, call `subscription.explain` or
|
|
327
|
+
`Upkeep::Rails.subscriptions.explain(subscription.id)`. Explanations summarize
|
|
328
|
+
the dependency tables and attributes, identity, frame count, lookup keys, and
|
|
329
|
+
metadata without requiring store-specific instance-variable inspection.
|
|
@@ -0,0 +1,211 @@
|
|
|
1
|
+
# SQLGlot Active Record query-analysis migration
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
Make SQLGlot the only production decoder for Active Record relation
|
|
6
|
+
dependencies in the next Upkeep release. Use the former Arel oracle to freeze a
|
|
7
|
+
parity corpus, then delete the oracle once the SQLGlot, raw-SQL, and adapter
|
|
8
|
+
corpora prove the same dependencies.
|
|
9
|
+
|
|
10
|
+
“Drop Arel” means Upkeep production code does not call `Relation#arel`, inspect
|
|
11
|
+
Arel nodes, or select a structured/unstructured policy. Active Record may still
|
|
12
|
+
use Arel internally to generate `Relation#to_sql`.
|
|
13
|
+
|
|
14
|
+
## API boundary
|
|
15
|
+
|
|
16
|
+
Upkeep owns a small Ruby wrapper over the released `sql-glot-rust` C ABI. Its
|
|
17
|
+
names, arguments, and result shapes follow the native API:
|
|
18
|
+
|
|
19
|
+
1. Wrap the existing parse and generation API:
|
|
20
|
+
- `Upkeep::SQLGlot.parse(sql, dialect:)`
|
|
21
|
+
- `Upkeep::SQLGlot.generate(statement, dialect:)`
|
|
22
|
+
- `Upkeep::SQLGlot.transpile(sql, from:, to:)`
|
|
23
|
+
2. Project the released semantic API:
|
|
24
|
+
- `MappingSchema`
|
|
25
|
+
- `qualify_columns(statement, schema)`
|
|
26
|
+
- `build_scope(statement)`
|
|
27
|
+
- `lineage(column, statement, schema, config)`
|
|
28
|
+
3. Preserve Rust/Python scope and lineage fields. Do not flatten child scope
|
|
29
|
+
collections or introduce an Upkeep-shaped SQLGlot response.
|
|
30
|
+
|
|
31
|
+
There is deliberately no combined `Upkeep::SQLGlot.analyze` API.
|
|
32
|
+
|
|
33
|
+
Upkeep-specific lowering remains separate from the sibling API:
|
|
34
|
+
|
|
35
|
+
```text
|
|
36
|
+
Upkeep::SQLGlot.parse + SQL schema
|
|
37
|
+
│
|
|
38
|
+
▼
|
|
39
|
+
Upkeep::SQLGlot::MappingSchema
|
|
40
|
+
Upkeep::SQLGlot.qualify_columns
|
|
41
|
+
Upkeep::SQLGlot.build_scope
|
|
42
|
+
│
|
|
43
|
+
▼
|
|
44
|
+
Upkeep::SQLDependencyAnalysis
|
|
45
|
+
│
|
|
46
|
+
▼
|
|
47
|
+
Upkeep::ActiveRecordQuery::Result
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
`SQLDependencyAnalysis` owns physical dependency sources, referenced columns,
|
|
51
|
+
simple predicate DNF, equality edges, query shape, and conservative warnings.
|
|
52
|
+
`ActiveRecordQuery` owns model table, primary key, adapter dialect, schema
|
|
53
|
+
extraction, and the internal result contract.
|
|
54
|
+
|
|
55
|
+
This release is intentionally breaking. There are no existing users to migrate,
|
|
56
|
+
so no compatibility decoder, cache migration, top-level `Sqlglot` namespace,
|
|
57
|
+
or mixed installation is supported. Installing the release requires a full
|
|
58
|
+
bundle/gem reinstall so Bundler selects the matching Upkeep platform gem.
|
|
59
|
+
|
|
60
|
+
## Native SQLGlot packaging
|
|
61
|
+
|
|
62
|
+
`upkeep-rails` depends on `ffi` and binds `sql-glot-rust` v0.10.26 directly.
|
|
63
|
+
The Ruby boundary lives under `Upkeep::SQLGlot`; there is no external
|
|
64
|
+
`sqlglot` Ruby gem dependency and no Upkeep-specific Rust crate or C ABI.
|
|
65
|
+
|
|
66
|
+
There is one logical gem. Each
|
|
67
|
+
release consists of five platform-specific `upkeep-rails` artifacts:
|
|
68
|
+
|
|
69
|
+
- `x86_64-linux-gnu`;
|
|
70
|
+
- `aarch64-linux-gnu`;
|
|
71
|
+
- `x86_64-darwin`;
|
|
72
|
+
- `arm64-darwin`; and
|
|
73
|
+
- `x64-mingw-ucrt`.
|
|
74
|
+
|
|
75
|
+
The released v0.10.26 tag is built in release CI and its unmodified shared
|
|
76
|
+
library is included in each artifact. Rust sources are build inputs and are not
|
|
77
|
+
shipped in the gem, so installation never requires Rust. There is deliberately
|
|
78
|
+
no generic source artifact.
|
|
79
|
+
|
|
80
|
+
The v0.10.26 scope builder preserves CTE references as scope sources. Upkeep
|
|
81
|
+
does not treat a scope source as a physical table; the AST lowerer resolves the
|
|
82
|
+
logical CTE to its physical child and scope validation checks every
|
|
83
|
+
schema-backed source.
|
|
84
|
+
|
|
85
|
+
Active Record SQL types remain available in the Ruby `MappingSchema`. The
|
|
86
|
+
native dependency schema represents every column type as `UNKNOWN` because
|
|
87
|
+
qualification and lineage inspect table and column identity, not column type.
|
|
88
|
+
Database-specific types therefore cannot prevent dependency extraction, and
|
|
89
|
+
Upkeep does not maintain a parallel SQL type catalog.
|
|
90
|
+
|
|
91
|
+
`qualify_columns` expands wildcards by design. Upkeep restores wildcard
|
|
92
|
+
projection nodes before dependency lowering so `SELECT table.*` does not add
|
|
93
|
+
every model attribute to collection invalidation. Qualification in predicates,
|
|
94
|
+
joins, grouping, ordering, and explicit projections is retained.
|
|
95
|
+
|
|
96
|
+
## Implementation stages
|
|
97
|
+
|
|
98
|
+
### 1. Establish SQLGlot as an internal runtime boundary
|
|
99
|
+
|
|
100
|
+
- Bind the released Rust library through `ffi`.
|
|
101
|
+
- Add adapter-to-dialect mapping.
|
|
102
|
+
- Convert Active Record schema metadata into SQLGlot-compatible table/column
|
|
103
|
+
metadata, retaining SQL types where available.
|
|
104
|
+
- Fail closed with an actionable SQL analysis error; never silently subscribe
|
|
105
|
+
only to the primary table after an unknown joined source.
|
|
106
|
+
|
|
107
|
+
### 2. Productize dependency lowering
|
|
108
|
+
|
|
109
|
+
- Move the SQL-first spike into `Upkeep::SQLDependencyAnalysis`.
|
|
110
|
+
- Accept SQL AST, dialect, and schema only—no relation or model objects.
|
|
111
|
+
- Keep SQL constructs inside this decoder and lower them into a generic graph:
|
|
112
|
+
physical tables, columns, equality edges, predicate groups, and query shape.
|
|
113
|
+
- Treat CTEs and derived sources as logical sources backed by physical tables.
|
|
114
|
+
- Record ambiguity as conservative coverage or an unsupported analysis, never
|
|
115
|
+
as a missing dependency.
|
|
116
|
+
|
|
117
|
+
### 3. Switch the Active Record runtime path
|
|
118
|
+
|
|
119
|
+
- Make `ActiveRecordQuery.analyze` parse `relation.to_sql` with SQLGlot.
|
|
120
|
+
- Preserve `ActiveRecordQuery::Result` so invalidation/replay consumers do not
|
|
121
|
+
need a simultaneous rewrite.
|
|
122
|
+
- Give write observation a separate `analyze_for_write` entry point. It may
|
|
123
|
+
conservatively describe the model table when semantic predicate analysis
|
|
124
|
+
fails; this is not a collection-query decoder or runtime fallback.
|
|
125
|
+
- Update opaque-query guidance so it no longer asks users to rewrite SQL as
|
|
126
|
+
Arel.
|
|
127
|
+
|
|
128
|
+
### 4. Move Arel to an oracle
|
|
129
|
+
|
|
130
|
+
- Move the current collector to test support and rename it
|
|
131
|
+
`ArelQueryAnalysisOracle`.
|
|
132
|
+
- Production `lib/` must contain no Arel node dispatch and no
|
|
133
|
+
`Relation#arel` call.
|
|
134
|
+
- Compare the oracle with SQLGlot for relations the oracle can prove.
|
|
135
|
+
- SQLGlot may report additional safe dependencies. It must not omit any table,
|
|
136
|
+
column, predicate, or query-shape restriction proven by the oracle.
|
|
137
|
+
- Raw SQL cases that the oracle rejects are asserted directly against SQLGlot.
|
|
138
|
+
|
|
139
|
+
### 5. Verification and release hardening
|
|
140
|
+
|
|
141
|
+
- Cover existing query-analysis tests and Pulse #1802 edge cases.
|
|
142
|
+
- Add raw predicate, raw join, alias/self-join, correlated subquery, CTE,
|
|
143
|
+
function/operator, `IN`, null, negation, OR-DNF, group/having/distinct,
|
|
144
|
+
limit/offset, and parse-error cases.
|
|
145
|
+
- Run the full Upkeep suite.
|
|
146
|
+
- Benchmark parsing separately from invalidation fan-out. This milestone does
|
|
147
|
+
not add an Upkeep-owned analysis or schema cache.
|
|
148
|
+
- Package the released native library for every supported platform.
|
|
149
|
+
|
|
150
|
+
## Oracle exit criteria
|
|
151
|
+
|
|
152
|
+
Arel can be deleted after:
|
|
153
|
+
|
|
154
|
+
- the supported adapter corpus has no known dependency false negatives;
|
|
155
|
+
- SQLGlot covers every relation the oracle proves;
|
|
156
|
+
- raw SQL coverage has dedicated assertions;
|
|
157
|
+
- unsupported SQL fails with query, dialect, and parser diagnostics;
|
|
158
|
+
- production code has no oracle switch or fallback;
|
|
159
|
+
- analysis stays outside invalidation fan-out and meets the agreed performance
|
|
160
|
+
budget.
|
|
161
|
+
|
|
162
|
+
## Current milestone
|
|
163
|
+
|
|
164
|
+
This milestone is complete in the 0.2.0 release branch: `Upkeep::SQLGlot` is
|
|
165
|
+
the only production SQL decoder, the former oracle outputs are frozen as
|
|
166
|
+
ordinary expected values, and the Arel collector has been deleted. Remaining
|
|
167
|
+
Arel usage in tests constructs Active Record input queries only; no Upkeep
|
|
168
|
+
analyzer inspects Arel.
|
|
169
|
+
|
|
170
|
+
## Execution status
|
|
171
|
+
|
|
172
|
+
Completed:
|
|
173
|
+
|
|
174
|
+
- [x] Added `Upkeep::SQLGlot` as the production runtime parser.
|
|
175
|
+
- [x] Added generic SQL AST dependency lowering in
|
|
176
|
+
`Upkeep::SQLDependencyAnalysis`.
|
|
177
|
+
- [x] Switched `ActiveRecordQuery.analyze` from `Relation#arel` to
|
|
178
|
+
`Upkeep::SQLGlot.parse(relation.to_sql, dialect:)`.
|
|
179
|
+
- [x] Kept schema access on Active Record's existing schema cache; Upkeep owns
|
|
180
|
+
no analysis or schema cache.
|
|
181
|
+
- [x] Added a separate conservative write-analysis path without introducing a
|
|
182
|
+
collection-query fallback.
|
|
183
|
+
- [x] Used `ArelQueryAnalysisOracle` to freeze the parity corpus, then deleted
|
|
184
|
+
the collector and its test-support require.
|
|
185
|
+
- [x] Added direct raw-SQL coverage, including correlated subqueries, CTEs,
|
|
186
|
+
set operations, PostgreSQL operators, and MySQL/SQLite functions.
|
|
187
|
+
- [x] Bound parse, generate, transpile, `MappingSchema`, qualification, scope,
|
|
188
|
+
and lineage directly to `sql-glot-rust` v0.10.26.
|
|
189
|
+
- [x] Decoupled dependency schemas from database type parsing while retaining
|
|
190
|
+
the original SQL types at the Ruby boundary.
|
|
191
|
+
- [x] Preserved Rust scope and lineage fields and locked the corrected CTE
|
|
192
|
+
scope-source behavior in a binding test.
|
|
193
|
+
- [x] Added five platform-specific `upkeep-rails` artifacts for macOS, Linux,
|
|
194
|
+
and Windows, with no source gem or install-time Cargo build.
|
|
195
|
+
- [x] Added an adapter corpus for PostgreSQL, MySQL, and SQLite.
|
|
196
|
+
- [x] Added a warm semantic-analysis performance gate with a 2 ms CI budget;
|
|
197
|
+
the local 10,000-iteration mean is approximately 146 µs.
|
|
198
|
+
- [x] Verified the full suite and the focused performance gate.
|
|
199
|
+
- [x] Removed compatibility and mixed-installation handling; this release
|
|
200
|
+
requires a full reinstall.
|
|
201
|
+
|
|
202
|
+
Upstream and ecosystem follow-up:
|
|
203
|
+
|
|
204
|
+
- [x] Open the semantic Ruby bindings in `sql-glot-ruby`.
|
|
205
|
+
- [x] Merge the CTE source-overwrite fix in `sql-glot-rust`; it shipped in
|
|
206
|
+
v0.10.24 through
|
|
207
|
+
[`protegrity/sql-glot-rust#26`](https://github.com/protegrity/sql-glot-rust/pull/26).
|
|
208
|
+
- [x] Merge and release the Rust semantic C ABI in v0.10.25.
|
|
209
|
+
- [x] Delete the Upkeep-specific Rust bridge and external Ruby SQLGlot
|
|
210
|
+
dependency.
|
|
211
|
+
- [ ] Continue the Ruby wrapper PRs as optional ecosystem contributions.
|
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
# Turbo Frame subscription composition
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
Keep one authoritative Upkeep subscription for the visible page across Turbo
|
|
6
|
+
Frame navigation.
|
|
7
|
+
|
|
8
|
+
A frame response is not a page response. Turbo extracts only the matching
|
|
9
|
+
`<turbo-frame>`, and frame endpoints may omit the application shell and sibling
|
|
10
|
+
frames. Replacing the page subscription with the response capture would
|
|
11
|
+
therefore lose dependencies that are still visible. Keeping both captures would
|
|
12
|
+
retain stale dependencies and duplicate invalidation fan-out.
|
|
13
|
+
|
|
14
|
+
## Model
|
|
15
|
+
|
|
16
|
+
`turbo_frame_tag` establishes a generic render scope in the dependency graph:
|
|
17
|
+
|
|
18
|
+
```text
|
|
19
|
+
page
|
|
20
|
+
├── shell
|
|
21
|
+
├── turbo_frame:filters
|
|
22
|
+
│ └── current frame dependencies
|
|
23
|
+
└── turbo_frame:details
|
|
24
|
+
└── current frame dependencies
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
For a Turbo Frame GET, the browser sends the current subscription identity and
|
|
28
|
+
Turbo's target frame id. Upkeep captures the response, finds the matching render
|
|
29
|
+
scope in both graphs, and creates a new pending subscription from:
|
|
30
|
+
|
|
31
|
+
```text
|
|
32
|
+
old page graph - old target subtree + response target subtree
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
The old subscription remains active while Action Cable connects the candidate.
|
|
36
|
+
After `connected`, the client installs the candidate as current and
|
|
37
|
+
unsubscribes the old subscription. Rejected or superseded candidates never
|
|
38
|
+
replace the active page graph.
|
|
39
|
+
|
|
40
|
+
## Invariants
|
|
41
|
+
|
|
42
|
+
- There is one current page subscription per browser document.
|
|
43
|
+
- A page with no reactive full render keeps a dormant coordinator; its first
|
|
44
|
+
reactive frame establishes the subscription.
|
|
45
|
+
- A frame transition changes only the matching render scope.
|
|
46
|
+
- Shell and sibling dependencies survive frame navigation.
|
|
47
|
+
- Replaced dependencies leave the reverse index when the old cable subscription
|
|
48
|
+
disconnects.
|
|
49
|
+
- The server accepts a base subscription only with a valid activation token and
|
|
50
|
+
the same derived subscriber identity.
|
|
51
|
+
- A response cannot become current unless its Action Cable subscription
|
|
52
|
+
connects.
|
|
53
|
+
- Concurrent frame responses are ordered by the browser transition generation;
|
|
54
|
+
a late response cannot restore an older graph.
|
|
55
|
+
- Expired or structurally incompatible base graphs request a full Turbo render
|
|
56
|
+
instead of discarding already-active scopes.
|
|
57
|
+
|
|
58
|
+
## Protocol
|
|
59
|
+
|
|
60
|
+
Before a frame fetch, the client adds these headers when a current subscription
|
|
61
|
+
exists:
|
|
62
|
+
|
|
63
|
+
- `X-Upkeep-Subscription-Id`
|
|
64
|
+
- `X-Upkeep-Subscription-Token`
|
|
65
|
+
|
|
66
|
+
After composing and registering the replacement, the response adds:
|
|
67
|
+
|
|
68
|
+
- `X-Upkeep-Subscription-Id`
|
|
69
|
+
- `X-Upkeep-Subscription-Token`
|
|
70
|
+
- `X-Upkeep-Subscription-Channel`
|
|
71
|
+
- `X-Upkeep-Subscription-Stream`
|
|
72
|
+
|
|
73
|
+
`turbo:frame-render` is the commit point. The custom subscription source reads
|
|
74
|
+
the response headers and starts the atomic cable transition. Full-page Turbo
|
|
75
|
+
Drive visits continue to replace the body-scoped source element normally.
|
|
76
|
+
|
|
77
|
+
## Implementation
|
|
78
|
+
|
|
79
|
+
- [x] Capture `turbo_frame_tag` blocks as `turbo_frame` graph nodes with
|
|
80
|
+
controller replay recipes.
|
|
81
|
+
- [x] Add generic DAG subtree replacement and recorder composition.
|
|
82
|
+
- [x] Validate and compose frame registrations against the active page
|
|
83
|
+
subscription.
|
|
84
|
+
- [x] Return candidate subscription metadata in response headers.
|
|
85
|
+
- [x] Keep the body source stable during frame visits and atomically switch its
|
|
86
|
+
Action Cable subscription.
|
|
87
|
+
- [x] Cover graph composition, replay/targeting, request protocol, race/rejection
|
|
88
|
+
behavior, and a real Turbo Frame navigation followed by live invalidation.
|
|
89
|
+
- [x] Regenerate Pulse's installed client, run the full Upkeep proof suite, and
|
|
90
|
+
repeat the Pulse browser scenario without a page reload.
|