upkeep-rails 0.1.6

data/docs/guides/getting-started.md

@@ -0,0 +1,462 @@
+# Getting Started
+
+This guide shows the Rails app path: add the package, run the installer,
+configure ActionCable identity, render normal Rails views, and let Active
+Record commits deliver targeted Turbo Stream payloads.
+
+## Add The Package
+
+Install the published gem:
+
+```ruby
+# Gemfile
+gem "upkeep-rails"
+```
+
+The Railtie installs hooks when Rails loads Active Record, Action Controller,
+and Action View. The runtime is enabled by default. Production uses the
+ActiveRecord subscription store by default and fails fast when the Upkeep
+subscription tables are missing. The runtime can be disabled per environment:
+
+```ruby
+# config/environments/test.rb
+config.upkeep.enabled = false
+```
+
+Use `config.upkeep.*` from Rails application or environment config. Use
+`Upkeep::Rails.configure` from `config/initializers/upkeep.rb`.
+
+Opaque reactive boundaries are refused instead of being widened into broad
+invalidation. Development/test raises by default so unsupported query shapes are
+found early; production warns and skips subscription registration by default:
+
+```ruby
+# config/environments/development.rb
+config.upkeep.refused_boundary_behavior = :raise # or :warn
+```
+
+## Run The Installer
+
+```sh
+bin/rails generate upkeep:install
+bin/rails db:migrate
+```
+
+The installer creates:
+
+- `config/initializers/upkeep.rb`
+- `db/migrate/*_create_upkeep_subscriptions.rb`
+- `app/javascript/upkeep/subscription.js`
+- an import from `app/javascript/application.js`
+- importmap pins for Turbo and ActionCable when `config/importmap.rb` exists
+- an ActionCable mount in `config/routes.rb` when one is not present
+
+The browser bootstrap is vendored into the app. After upgrading
+`upkeep-rails`, rerun the installer or compare
+`app/javascript/upkeep/subscription.js` with the gem template so the client
+payload still matches the server channel.
+
+## Subscription Storage
+
+The generated initializer keeps production on durable storage and uses memory
+for ordinary app tests:
+
+```ruby
+# config/initializers/upkeep.rb
+Upkeep::Rails.configure do |config|
+  app_config = Rails.application.config.upkeep
+
+  config.enabled = app_config.fetch(:enabled, true)
+  config.subscription_store = app_config.fetch(:subscription_store, Rails.env.test? ? :memory : :active_record)
+  config.delivery_adapter = app_config.fetch(:delivery_adapter, Rails.env.production? ? :active_job : :async)
+  config.delivery_queue = app_config.fetch(:delivery_queue, :upkeep_realtime)
+end
+```
+
+The ActiveRecord store persists subscriptions and reverse-index entries so any
+Puma worker can plan invalidations for subscriptions captured by another
+worker. The memory store has the same public lifecycle but keeps all state in
+the current process. It is the right default for request/system tests that only
+need to prove marker registration, activation, planning, delivery, and rendered
+bytes.
+
+Use an ActiveRecord-backed test environment or CI job when you want to exercise
+production-only storage behavior:
+
+```ruby
+# config/environments/test.rb
+config.upkeep.subscription_store = :active_record
+```
+
+The benchmark app uses this migration shape:
+
+```ruby
+create_table :upkeep_subscriptions, id: :string do |t|
+  t.string :subscriber_id, null: false
+  t.json :recorder_snapshot, null: false
+  t.json :metadata
+  t.string :subscription_shape_key
+  t.timestamps
+end
+add_index :upkeep_subscriptions, :subscriber_id
+add_index :upkeep_subscriptions, :subscription_shape_key,
+  name: "idx_upkeep_subscriptions_on_shape_key"
+
+create_table :upkeep_subscription_index_entries do |t|
+  t.string :subscription_id, null: false
+  t.string :lookup_key_digest, null: false
+  t.string :dependency_source, null: false
+  t.string :lookup_table, null: false
+  t.json :lookup_record_id_snapshot
+  t.string :lookup_attribute, null: false
+  t.string :dependency_table, null: false
+  t.string :dependency_predicate_digest
+  t.json :dependency_metadata_snapshot
+  t.json :owner_ids_snapshot, null: false
+  t.timestamps
+end
+add_index :upkeep_subscription_index_entries, :subscription_id
+add_index :upkeep_subscription_index_entries, :lookup_key_digest
+add_foreign_key :upkeep_subscription_index_entries,
+  :upkeep_subscriptions,
+  column: :subscription_id,
+  on_delete: :cascade
+
+create_table :upkeep_subscription_shape_index_entries do |t|
+  t.string :subscription_shape_key, null: false
+  t.string :lookup_key_digest, null: false
+  t.string :dependency_source, null: false
+  t.string :lookup_table, null: false
+  t.json :lookup_record_id_snapshot
+  t.string :lookup_attribute, null: false
+  t.string :dependency_table, null: false
+  t.string :dependency_predicate_digest
+  t.json :dependency_metadata_snapshot
+  t.json :owner_ids_snapshot, null: false
+  t.timestamps
+end
+add_index :upkeep_subscription_shape_index_entries,
+  :subscription_shape_key,
+  name: "idx_upkeep_sub_shape_entries_on_shape_key"
+add_index :upkeep_subscription_shape_index_entries,
+  :lookup_key_digest,
+  name: "idx_upkeep_sub_shape_entries_on_lookup_digest"
+```
+
+## Delivery Jobs And ActionCable
+
+Upkeep uses Active Job for production delivery. A mutation request records
+committed Active Record facts, enqueues `Upkeep::Rails::DeliveryJob`, and lets
+that job plan, render, and broadcast the Turbo Stream payloads:
+
+```ruby
+# Sidekiq
+config.active_job.queue_adapter = :sidekiq
+
+# or Solid Queue
+config.active_job.queue_adapter = :solid_queue
+```
+
+The job queue does not replace ActionCable. It only decides where Upkeep's
+delivery job runs. The job still calls `ActionCable.server.broadcast`, so
+multi-process apps need a shared ActionCable adapter:
+
+```yml
+# Redis-backed cable
+production:
+  adapter: redis
+  url: <%= ENV.fetch("REDIS_URL") %>
+  channel_prefix: my_app_production
+```
+
+```yml
+# No Redis: database-backed cable
+production:
+  adapter: solid_cable
+  connects_to:
+    database:
+      writing: cable
+```
+
+Solid Queue plus Solid Cable is the no-Redis setup. Sidekiq normally brings
+Redis for jobs; it can still broadcast through either Redis-backed ActionCable
+or Solid Cable.
+
+## Configure Subscriber Identity
+
+Mount ActionCable:
+
+```ruby
+# config/routes.rb
+mount ActionCable.server => "/cable"
+```
+
+Declare each identity boundary that should partition live updates:
+
+```ruby
+# config/initializers/upkeep.rb
+Upkeep::Rails.configure do |config|
+  config.identify :viewer, current: ["Current", :user] do
+    subscribe { |connection| connection.current_user }
+  end
+end
+```
+
+Read the declaration in three parts:
+
+| Part | Meaning |
+| --- | --- |
+| `:viewer` | The identity component name. Choose the role the value plays: `:viewer`, `:account`, `:tenant`, `:locale`, etc. |
+| `current: ["Current", :user]` | The render-side source. This declaration applies when the page reads `Current.user`. |
+| `subscribe { |connection| connection.current_user }` | The ActionCable-side proof. It must return the same logical user when the browser subscribes. |
+
+Choose the source keyword from the API the render path actually reads:
+
+| Render code reads | Use |
+| --- | --- |
+| `Current.user` | `current: ["Current", :user]` |
+| Devise `current_user`, `user_signed_in?`, or `warden.user(:user)` | `warden: :user` |
+| `session[:user_id]` | `session: :user_id` |
+| `cookies[:account_id]` | `cookie: :account_id` |
+
+If a Devise app copies the signed-in user into `Current.user`, declare the
+source that the page actually reads. Declare both only when both sources are
+genuine render inputs.
+
+Expose the matching value on the ActionCable connection. Active Record records,
+scalars, and GlobalID values are supported identity components:
+
+```ruby
+module ApplicationCable
+  class Connection < ActionCable::Connection::Base
+    identified_by :current_user
+
+    def connect
+      self.current_user = User.find_by(id: request.session[:user_id])
+    end
+  end
+end
+```
+
+For Devise/Warden, declare the Warden scope and make the cable connection expose
+the same user:
+
+```ruby
+# config/initializers/upkeep.rb
+Upkeep::Rails.configure do |config|
+  config.identify :viewer, warden: :user do
+    subscribe { |connection| connection.current_user }
+  end
+end
+```
+
+```ruby
+# app/channels/application_cable/connection.rb
+module ApplicationCable
+  class Connection < ActionCable::Connection::Base
+    identified_by :current_user
+
+    def connect
+      self.current_user = env["warden"]&.user(:user)
+    end
+  end
+end
+```
+
+Use `warden: :admin` or another scope when the rendered Devise helper is scoped
+to that role. Reject nil users in `connect` only if every cable subscription in
+the app requires login; otherwise nil remains an absent identity for public
+pages.
+
+By default, `nil` means the declared boundary is absent. Logged-out pages can
+therefore stay anonymous-public. If your app uses another sentinel, declare it
+with `absent_if`.
+
+The `subscribe` block receives an Upkeep connection context. Use
+`connection.current_user`, `connection.session`, and `connection.cookies`; do
+not reach through to ActionCable's raw request object.
+
+Clients subscribe to `Upkeep::Rails::Cable::Channel` with the
+`subscription_id` from the injected `<upkeep-subscription-source>` element. The
+server channel validates the subscription id and streams from the canonical
+subscriber stream plus any shared streams attached to that graph.
+
+The generated browser bootstrap upgrades that body element into a Turbo stream
+source with `Turbo.session.connectStreamSource`, subscribes through
+`@rails/actioncable`, and lets Turbo process received stream payloads. The
+source is `data-turbo-temporary` so Turbo does not cache stale subscription
+handles.
+
+For more than one Puma worker, configure ActionCable with a shared adapter
+such as Redis or Solid Cable. The subscription store decides who should receive
+work; ActionCable decides which worker owns each WebSocket connection.
+
+## Render Normal Rails Views
+
+Controllers load normal Active Record models and relations:
+
+```ruby
+class BoardsController < ApplicationController
+  def show
+    @board = Board.find(params[:id])
+    @cards = @board.cards.order(:position)
+  end
+end
+```
+
+Templates render normal ERB and partial collections:
+
+```erb
+<main>
+  <h1><%= @board.name %></h1>
+
+  <ul id="cards">
+    <%= render partial: "cards/card", collection: @cards, as: :card %>
+  </ul>
+</main>
+```
+
+Successful HTML GET responses are captured automatically. Upkeep records page,
+render-site, and fragment frames from Rails renderer hooks, then injects a
+subscription source into the response.
+
+Polymorphic collection shorthand is supported when runtime rendering confirms
+that the rendered object is a collection:
+
+```erb
+<ul id="cards">
+  <%= render @cards %>
+</ul>
+```
+
+Rails tag-helper containers are supported too. Herb lowers `tag.*` and
+`content_tag` blocks into template structure, and Upkeep writes its internal
+markers through the helper call:
+
+```erb
+<%= tag.ul id: "cards" do %>
+  <%= render partial: "cards/card", collection: @cards, as: :card %>
+<% end %>
+```
+
+Upkeep trusts narrow source-derived targets only when Herb's strict parser
+accepts the template. If strict parsing fails but non-strict parsing recovers,
+the app still renders normally, Upkeep reports the strict diagnostics as
+warnings, and broad page or fragment markers may still be added. Recovered
+render sites are reported as candidates only; fix the strict warnings before
+expecting narrow collection updates from that template.
+
+Controller materialization is supported when the rendered value keeps a
+structural relation proof:
+
+```ruby
+def index
+  @cards = Card.where(status: "open").order(:position).to_a
+end
+```
+
+```erb
+<%= render partial: "cards/card", collection: @cards, as: :card %>
+```
+
+Upkeep attaches the collection dependency to the rendered collection boundary,
+not to every controller query. A materialized relation that is never rendered
+as a collection is not a lifecycle dependency by itself.
+
+Scalar relation output is tracked as a page-level query dependency:
+
+```ruby
+@tag_names = Tag.where(active: true).pluck(:name)
+```
+
+Simple plucked columns are live and can select a page replay when they change.
+They are not collection dependencies, so they do not participate in
+append/remove/prepend planning.
+
+Session, cookie, and request reads are observed inputs:
+
+```ruby
+@viewer = session[:viewer]
+@tag_filter = cookies[:tag_filters]
+@agent = request.user_agent
+```
+
+Replay stores only observed values needed to rerun the page. Unread session
+keys, cookies, and request headers are not copied into the replay payload.
+
+## Mutate Through Active Record
+
+Write paths keep doing domain work:
+
+```ruby
+class CardsController < ApplicationController
+  def update
+    card = Card.find(params[:id])
+    card.update!(card_params)
+    head :ok
+  end
+end
+```
+
+After commit, Upkeep records the changed table, id, and attributes. The planner
+uses the reverse index to select affected subscribers and render targets.
+Collection lookup entries are keyed by proven table and column pairs, so a write
+to `cards.title` does not select a collection whose only proven dependency is
+`cards.status`. Identity, request, session, and cookie reads are still recorded
+on the graph for replay and sharing, but lifecycle writes do not index or select
+them.
+
+Bulk writes are observed through Active Record relations:
+
+```ruby
+Card.where(board_id: board.id, status: "open").update_all(status: "done")
+```
+
+For structurally visible relations, Upkeep derives the involved tables and
+columns through Arel. Opaque collection relations are refused instead of being
+registered as broad reactive dependencies; rewrite them with structural
+Active Record or Arel predicates before relying on Upkeep updates.
+
+Non-GET controller actions do not register subscriptions. They still capture
+Active Record lifecycle changes and deliver to existing subscribers.
+
+## Verify The Integration
+
+The benchmark app checks the integration path with `Upkeep::Rails::Testing`:
+
+```ruby
+include Upkeep::Rails::Testing
+
+get board_path(board)
+assert_response :success
+assert_upkeep_subscription_registered
+activate_upkeep_subscription!
+```
+
+For a streamed mutation, capture ActionCable broadcasts for the subscription's
+stream names, perform the mutation, drain delivery, and assert the payload:
+
+```ruby
+include ActionCable::TestHelper
+include Upkeep::Rails::Testing
+
+broadcasts = capture_upkeep_broadcasts do
+  patch board_card_path(board, card), params: { card: { title: "Updated" } }
+  drain_upkeep_delivery!
+end
+
+assert_includes broadcasts.join, "Updated"
+```
+
+## Current Boundaries
+
+- HTML GET responses are the subscription capture path.
+- Non-GET requests are mutation/delivery paths, not subscription capture paths.
+- Non-HTML templates are outside the Herb-backed template planning surface.
+- The app does not declare query dependencies, and subscriber identity is only
+  created from explicit `config.identify` declarations. If a render depends
+  on hidden process state outside the observed Rails surfaces, Upkeep cannot
+  prove subscriber ownership for that value.
+- Opaque relation predicates, raw SQL joins/sources, and opaque pluck
+  expressions raise in development/test by default. In warning mode they refuse
+  the boundary and skip subscription registration instead of broadening.

data/docs/handoff-2026-05-15.md

@@ -0,0 +1,230 @@
+# Upkeep Rails Handoff - 2026-05-15
+
+This handoff captures the current state of the `lobsters-upkeep-benchmark`
+branch after the subscription storage and cold-churn work.
+
+## Current Position
+
+Upkeep is much healthier than when cold setup was taking tens of seconds, but
+it is still behind Turbo on the cold connection churn gate.
+
+Latest same-run comparison:
+
+- Report:
+  `benchmark/results/matrix-compare-20260514233725.md`
+- Workload:
+  `matrix/cold_connect_churn_chat`
+- Shape:
+  200 users, 1 Puma worker, 5 Puma threads, Upkeep on port 3000, Turbo on port
+  3001.
+
+| p95 metric | Upkeep | Turbo | Read |
+| --- | ---: | ---: | --- |
+| Setup total | 1229 ms | 408.55 ms | Upkeep is about 3.0x slower |
+| Login HTTP | 456.9 ms | 140 ms | Upkeep is about 3.3x slower |
+| Page request | 389.9 ms | 134.55 ms | Upkeep is about 2.9x slower |
+| WebSocket connect | 361.5 ms | 138.06 ms | Upkeep is about 2.6x slower |
+| Subscribe ack | 6 ms | 4 ms | Close |
+| Subscription registration | 0.13 ms | 0.15 ms | Upkeep is slightly faster |
+
+Interpretation:
+
+- Subscription registration is not the current bottleneck.
+- The remaining gap is cold admission: login, page render with recording,
+  WebSocket connect, and subscription ack under churn.
+- Upkeep had one subscription timeout in the latest comparison run. Treat the
+  result as useful signal, but do not call the gate clean until the timeout is
+  explained or eliminated.
+
+## What Cold Admission Means
+
+Cold admission is the time for a brand-new client to become a live subscriber
+while the app is already running. In the benchmark, `setup_total` starts before
+login and stops after the ActionCable subscription ack.
+
+The path is:
+
+1. Log in.
+2. Render `/rooms/:id`.
+3. Extract the subscription marker from HTML.
+4. Open `/cable`.
+5. Subscribe to the channel.
+6. Receive the ack.
+
+Turbo is expected to have a smaller cold path because it emits a signed stream
+name. Upkeep has to render while recording dependencies, emit a subscription
+id, register the graph, and then accept the socket subscription. That is an
+inherent constant tax, but the current 3x gap is not inherent.
+
+## Landed Work
+
+Recent commits to know:
+
+- `d201dc9` - `Optimize ActiveRecord subscription persistence`
+- `4acb7cb` - `Use typed persistent reverse index rows`
+- `bfb8c93` - `Cancel stale subscriptions on unsubscribe`
+
+The ActiveRecord subscription store is now split into smaller concerns:
+
+- `ActiveRecordStore` coordinates the store.
+- `ActiveRegistry` handles live in-process subscriptions.
+- `AsyncDurableWriter` batches durable writes.
+- `ActiveRecordSubscriptionPersistence` owns database writes and deletes.
+- `PersistentReverseIndex` and `LayeredReverseIndex` handle persistent and
+  live lookup.
+- `JsonSnapshot` owns inspectable replay/frame payload encoding.
+
+Current storage rules:
+
+- Production storage uses the ActiveRecord store.
+- The memory store is for development/test configuration.
+- Registration is live-first: the in-process registry updates synchronously,
+  then durable writes happen in a coalesced writer.
+- Channel unsubscribe unregisters the active subscription, cancels queued
+  durable writes, deletes persisted rows for inflight work, and removes
+  reverse-index entries incrementally.
+
+Measured lifecycle result:
+
+- Upkeep-only run `20260514232743` lowered cold churn setup p95 to
+  1097.75 ms.
+- Final `upkeep_subscriptions` row count: 0.
+- Final `upkeep_subscription_index_entries` row count: 0.
+- Real persistence batch p95: 137.6 ms.
+
+## Current Design Boundaries
+
+Keep these contracts separate:
+
+- Subscription storage: active and durable graph lookup.
+- Dispatching: render/delivery work scheduling.
+- ActionCable broadcast bus: how messages reach connected clients.
+- Benchmarking/telemetry: measurement only, not runtime policy.
+
+Multi-worker support is not done just because subscriptions persist. A
+multi-worker deployment needs:
+
+- A shared ActionCable adapter.
+- Durable graph lookup for workers that did not render the page.
+- Dispatch work that can leave the Puma process or use the app queue adapter.
+- Metrics that separate queue delay, invalidation proof, render cost, and
+  fanout cost.
+
+Do not add a second production persistence path. The direction is one
+production store contract and explicit queue/bus configuration around it.
+
+## Deoptimization Policy
+
+Keep the deoptimization surface small and actionable.
+
+Live deoptimizations stay only when Upkeep has already proven correctness and
+is merely choosing a broader operation than the cheapest possible Turbo Stream.
+Missing proof for a reactive boundary should refuse registration in a way that
+raises in development/test and warns or refuses in production-style settings.
+
+Every refusal should eventually include:
+
+- Stable reason name.
+- The concrete proof that was missing.
+- A refactor suggestion.
+- Enough context to connect it to a template or relation boundary.
+
+## What Not To Chase Next
+
+Fibers are not the next optimization for the measured bottleneck. The slow path
+is blocking ActiveRecord/SQLite persistence plus render/recording work under
+contention. Fibers would not make those writes non-blocking.
+
+A previous experiment with lowering the durable writer thread priority made
+setup p95 worse, around 2914 ms. Do not reintroduce it without a benchmark that
+proves a different result.
+
+Do not optimize subscription registration first. It is already at 0.13 ms p95
+in the latest comparison, roughly equal to Turbo.
+
+## Recommended Next Steps
+
+1. Diagnose the cold-admission gap with phase-level server and client evidence.
+
+   The server p95 phases in the latest report are small compared with the
+   client-observed gap:
+
+   | Server phase p95 | Upkeep | Turbo |
+   | --- | ---: | ---: |
+   | `sessions#create` | 8.28 ms | 5.98 ms |
+   | `rooms#show` | 12.69 ms | 3.83 ms |
+   | Subscription registration | 0.13 ms | 0.15 ms |
+   | Subscription confirmation | 0.06 ms | 0.04 ms |
+
+   That mismatch suggests queueing, contention, connection scheduling, or
+   missing instrumentation around the page/socket boundary. Add measurement
+   before changing behavior.
+
+2. Explain the single Upkeep subscription timeout.
+
+   Check `benchmark/results/matrix-chat_upkeep_cold_connect_churn-20260514233725.log`
+   and `benchmark/results/upkeep-app-server-20260514233725.log`. Determine
+   whether the timeout is ActionCable scheduling, subscription lookup, test
+   harness timing, or server pressure.
+
+3. Break down `/rooms/:id` render recording cost.
+
+   Compare Upkeep and Turbo page render work at the template/component level.
+   Separate normal Rails render time from dependency recording, marker
+   generation, subscription graph construction, and HTML size.
+
+4. Add an ActionCable connect phase for Upkeep in the comparison report.
+
+   The latest report shows Turbo cable connect p95 but `--` for Upkeep. That
+   makes the cold admission gap harder to localize.
+
+5. Only after the gap is localized, implement the smallest optimization and
+   rerun the same gate.
+
+   The loop should stay:
+
+   - Implement one optimization.
+   - Remove superseded code paths in the same pass.
+   - Update `docs/cost-model-roadmap.md` with the measured result.
+   - Run tests.
+   - Commit.
+
+## Commands
+
+Activate the expected Ruby/toolchain before Ruby or Bundler commands:
+
+```sh
+eval "$(mise activate zsh)"
+```
+
+Run the full test suite:
+
+```sh
+eval "$(mise activate zsh)" && ruby -S bundle exec ruby -Itest -e 'Dir["test/**/*_test.rb"].sort.each { |file| require_relative file }'
+```
+
+Run the cold churn comparison:
+
+```sh
+mkdir -p /tmp/upkeep-asdf-helper
+ln -sf /usr/bin/true /tmp/upkeep-asdf-helper/asdf
+eval "$(mise activate zsh)"
+PATH="/tmp/upkeep-asdf-helper:$PATH" BENCH_FAMILY=matrix BENCH_WORKLOAD=cold_connect_churn_chat BENCH_TIER=gate ruby benchmark/bin/run
+```
+
+Run the Upkeep-only gate:
+
+```sh
+mkdir -p /tmp/upkeep-asdf-helper
+ln -sf /usr/bin/true /tmp/upkeep-asdf-helper/asdf
+eval "$(mise activate zsh)"
+PATH="/tmp/upkeep-asdf-helper:$PATH" BENCH_UPKEEP_ONLY=1 BENCH_FAMILY=matrix BENCH_WORKLOAD=cold_connect_churn_chat BENCH_TIER=gate ruby benchmark/bin/run
+```
+
+## Current Bottom Line
+
+Upkeep now has a credible subscription storage architecture and the hot
+registration path is no longer the issue. The project should reframe the next
+performance work around cold admission and reliability under churn. Turbo is
+still ahead there; Upkeep has to get the cold tax low enough that precise warm
+invalidation and shared delivery work can pay it back.