upkeep-rails 0.1.0

data/docs/guides/getting-started.md

@@ -0,0 +1,282 @@
+# Getting Started
+
+This guide shows the Rails app path: add the package, run the installer,
+configure ActionCable identity, render normal Rails views, and let Active
+Record commits deliver targeted Turbo Stream payloads.
+
+## Add The Package
+
+Install the published gem:
+
+```ruby
+# Gemfile
+gem "upkeep-rails"
+```
+
+The Railtie installs hooks when Rails loads Active Record, Action Controller,
+and Action View. The runtime is enabled by default. Production uses the
+ActiveRecord subscription store by default and fails fast when the Upkeep
+subscription tables are missing. The runtime can be disabled per environment:
+
+```ruby
+# config/environments/test.rb
+config.upkeep.enabled = false
+```
+
+Opaque reactive boundaries are refused instead of being widened into broad
+invalidation. Development/test raises by default so unsupported query shapes are
+found early; production warns and skips subscription registration by default:
+
+```ruby
+config.upkeep.refused_boundary_behavior = :raise # or :warn
+```
+
+## Run The Installer
+
+```sh
+bin/rails generate upkeep:install
+bin/rails db:migrate
+```
+
+The installer creates:
+
+- `config/initializers/upkeep.rb`
+- `db/migrate/*_create_upkeep_subscriptions.rb`
+- `app/javascript/upkeep/subscription.js`
+- an import from `app/javascript/application.js`
+- importmap pins for Turbo and ActionCable when `config/importmap.rb` exists
+- an ActionCable mount in `config/routes.rb` when one is not present
+
+## Subscription Storage
+
+Production subscription storage is explicit:
+
+```ruby
+# config/initializers/upkeep.rb
+Rails.application.configure do
+  config.upkeep.enabled = true
+  config.upkeep.subscription_store = :active_record
+end
+```
+
+The ActiveRecord store persists subscriptions and reverse-index entries so any
+Puma worker can plan invalidations for subscriptions captured by another
+worker. For development or isolated tests that do not run the installer
+migration, opt into the in-process store explicitly:
+
+```ruby
+config.upkeep.subscription_store = :memory
+```
+
+The benchmark app uses this migration shape:
+
+```ruby
+create_table :upkeep_subscriptions, id: :string do |t|
+  t.string :subscriber_id, null: false
+  t.json :recorder_snapshot, null: false
+  t.json :metadata
+  t.timestamps
+end
+add_index :upkeep_subscriptions, :subscriber_id
+
+create_table :upkeep_subscription_index_entries do |t|
+  t.string :subscription_id, null: false
+  t.string :lookup_key_digest, null: false
+  t.string :dependency_source, null: false
+  t.string :lookup_table, null: false
+  t.json :lookup_record_id_snapshot
+  t.string :lookup_attribute, null: false
+  t.string :dependency_table, null: false
+  t.string :dependency_predicate_digest
+  t.json :dependency_metadata_snapshot
+  t.json :owner_ids_snapshot, null: false
+  t.timestamps
+end
+add_index :upkeep_subscription_index_entries, :subscription_id
+add_index :upkeep_subscription_index_entries, :lookup_key_digest
+add_foreign_key :upkeep_subscription_index_entries,
+  :upkeep_subscriptions,
+  column: :subscription_id,
+  on_delete: :cascade
+```
+
+## Configure ActionCable Identity
+
+Mount ActionCable:
+
+```ruby
+# config/routes.rb
+mount ActionCable.server => "/cable"
+```
+
+Expose a canonical server identity on the ActionCable connection. Active Record
+records, scalars, and GlobalID values are supported identity components:
+
+```ruby
+module ApplicationCable
+  class Connection < ActionCable::Connection::Base
+    identified_by :current_user
+
+    def connect
+      self.current_user = User.find_by(id: request.session[:user_id]) ||
+        reject_unauthorized_connection
+    end
+  end
+end
+```
+
+Clients subscribe to `Upkeep::Rails::Cable::Channel` with the
+`subscription_id` from the injected `data-upkeep-subscription` marker. The
+server channel validates the subscription id and streams from the canonical
+subscriber stream plus any shared streams attached to that graph.
+
+The generated browser bootstrap reads those markers, subscribes through
+`@rails/actioncable`, and applies received Turbo Stream payloads.
+
+For more than one Puma worker, configure ActionCable with a shared adapter
+such as Redis or Solid Cable. The subscription store decides who should receive
+work; ActionCable decides which worker owns each WebSocket connection.
+
+## Render Normal Rails Views
+
+Controllers load normal Active Record models and relations:
+
+```ruby
+class BoardsController < ApplicationController
+  def show
+    @board = Board.find(params[:id])
+    @cards = @board.cards.order(:position)
+  end
+end
+```
+
+Templates render normal ERB and partial collections:
+
+```erb
+<main>
+  <h1><%= @board.name %></h1>
+
+  <ul id="cards">
+    <%= render partial: "cards/card", collection: @cards, as: :card %>
+  </ul>
+</main>
+```
+
+Successful HTML GET responses are captured automatically. Upkeep records page,
+render-site, and fragment frames from Rails renderer hooks, then injects a
+subscription marker into the response.
+
+Controller materialization is supported when the rendered value keeps a
+structural relation proof:
+
+```ruby
+def index
+  @cards = Card.where(status: "open").order(:position).to_a
+end
+```
+
+```erb
+<%= render partial: "cards/card", collection: @cards, as: :card %>
+```
+
+Upkeep attaches the collection dependency to the rendered collection boundary,
+not to every controller query. A materialized relation that is never rendered
+as a collection is not a lifecycle dependency by itself.
+
+Scalar relation output is tracked as a page-level query dependency:
+
+```ruby
+@tag_names = Tag.where(active: true).pluck(:name)
+```
+
+Simple plucked columns are live and can select a page replay when they change.
+They are not collection dependencies, so they do not participate in
+append/remove/prepend planning.
+
+Session, cookie, and request reads are observed inputs:
+
+```ruby
+@viewer = session[:viewer]
+@tag_filter = cookies[:tag_filters]
+@agent = request.user_agent
+```
+
+Replay stores only observed values needed to rerun the page. Unread session
+keys, cookies, and request headers are not copied into the replay payload.
+
+## Mutate Through Active Record
+
+Write paths keep doing domain work:
+
+```ruby
+class CardsController < ApplicationController
+  def update
+    card = Card.find(params[:id])
+    card.update!(card_params)
+    head :ok
+  end
+end
+```
+
+After commit, Upkeep records the changed table, id, and attributes. The planner
+uses the reverse index to select affected subscribers and render targets.
+Collection lookup entries are keyed by proven table and column pairs, so a write
+to `cards.title` does not select a collection whose only proven dependency is
+`cards.status`. Identity, request, session, and cookie reads are still recorded
+on the graph for replay and sharing, but lifecycle writes do not index or select
+them.
+
+Bulk writes are observed through Active Record relations:
+
+```ruby
+Card.where(board_id: board.id, status: "open").update_all(status: "done")
+```
+
+For structurally visible relations, Upkeep derives the involved tables and
+columns through Arel. Opaque collection relations are refused instead of being
+registered as broad reactive dependencies; rewrite them with structural
+Active Record or Arel predicates before relying on Upkeep updates.
+
+Non-GET controller actions do not register subscriptions. They still capture
+Active Record lifecycle changes and deliver to existing subscribers.
+
+## Verify The Integration
+
+The benchmark app checks the integration path with `Upkeep::Rails::Testing`:
+
+```ruby
+include Upkeep::Rails::Testing
+
+get board_path(board)
+assert_response :success
+assert_upkeep_subscription_registered
+assert_instance_of Upkeep::Subscriptions::ActiveRecordStore,
+  Upkeep::Rails.subscriptions
+```
+
+For a streamed mutation, capture ActionCable broadcasts for the subscription's
+stream names, perform the mutation, drain delivery, and assert the payload:
+
+```ruby
+include ActionCable::TestHelper
+include Upkeep::Rails::Testing
+
+broadcasts = capture_upkeep_broadcasts do
+  patch board_card_path(board, card), params: { card: { title: "Updated" } }
+  Upkeep::Rails.drain_delivery!
+end
+
+assert_includes broadcasts.join, "Updated"
+```
+
+## Current Boundaries
+
+- HTML GET responses are the subscription capture path.
+- Non-GET requests are mutation/delivery paths, not subscription capture paths.
+- Non-HTML templates are outside the Herb-backed template planning surface.
+- The app does not declare query or identity dependencies. If a render depends
+  on hidden process state outside the observed Rails surfaces, Upkeep cannot
+  prove subscriber ownership for that value.
+- Opaque relation predicates, raw SQL joins/sources, and opaque pluck
+  expressions raise in development/test by default. In warning mode they refuse
+  the boundary and skip subscription registration instead of broadening.

data/docs/handoff-2026-05-15.md

@@ -0,0 +1,230 @@
+# Upkeep Rails Handoff - 2026-05-15
+
+This handoff captures the current state of the `lobsters-upkeep-benchmark`
+branch after the subscription storage and cold-churn work.
+
+## Current Position
+
+Upkeep is much healthier than when cold setup was taking tens of seconds, but
+it is still behind Turbo on the cold connection churn gate.
+
+Latest same-run comparison:
+
+- Report:
+  `benchmark/results/matrix-compare-20260514233725.md`
+- Workload:
+  `matrix/cold_connect_churn_chat`
+- Shape:
+  200 users, 1 Puma worker, 5 Puma threads, Upkeep on port 3000, Turbo on port
+  3001.
+
+| p95 metric | Upkeep | Turbo | Read |
+| --- | ---: | ---: | --- |
+| Setup total | 1229 ms | 408.55 ms | Upkeep is about 3.0x slower |
+| Login HTTP | 456.9 ms | 140 ms | Upkeep is about 3.3x slower |
+| Page request | 389.9 ms | 134.55 ms | Upkeep is about 2.9x slower |
+| WebSocket connect | 361.5 ms | 138.06 ms | Upkeep is about 2.6x slower |
+| Subscribe ack | 6 ms | 4 ms | Close |
+| Subscription registration | 0.13 ms | 0.15 ms | Upkeep is slightly faster |
+
+Interpretation:
+
+- Subscription registration is not the current bottleneck.
+- The remaining gap is cold admission: login, page render with recording,
+  WebSocket connect, and subscription ack under churn.
+- Upkeep had one subscription timeout in the latest comparison run. Treat the
+  result as useful signal, but do not call the gate clean until the timeout is
+  explained or eliminated.
+
+## What Cold Admission Means
+
+Cold admission is the time for a brand-new client to become a live subscriber
+while the app is already running. In the benchmark, `setup_total` starts before
+login and stops after the ActionCable subscription ack.
+
+The path is:
+
+1. Log in.
+2. Render `/rooms/:id`.
+3. Extract the subscription marker from HTML.
+4. Open `/cable`.
+5. Subscribe to the channel.
+6. Receive the ack.
+
+Turbo is expected to have a smaller cold path because it emits a signed stream
+name. Upkeep has to render while recording dependencies, emit a subscription
+id, register the graph, and then accept the socket subscription. That is an
+inherent constant tax, but the current 3x gap is not inherent.
+
+## Landed Work
+
+Recent commits to know:
+
+- `d201dc9` - `Optimize ActiveRecord subscription persistence`
+- `4acb7cb` - `Use typed persistent reverse index rows`
+- `bfb8c93` - `Cancel stale subscriptions on unsubscribe`
+
+The ActiveRecord subscription store is now split into smaller concerns:
+
+- `ActiveRecordStore` coordinates the store.
+- `ActiveRegistry` handles live in-process subscriptions.
+- `AsyncDurableWriter` batches durable writes.
+- `ActiveRecordSubscriptionPersistence` owns database writes and deletes.
+- `PersistentReverseIndex` and `LayeredReverseIndex` handle persistent and
+  live lookup.
+- `JsonSnapshot` owns inspectable replay/frame payload encoding.
+
+Current storage rules:
+
+- Production storage uses the ActiveRecord store.
+- The memory store is for development/test configuration.
+- Registration is live-first: the in-process registry updates synchronously,
+  then durable writes happen in a coalesced writer.
+- Channel unsubscribe unregisters the active subscription, cancels queued
+  durable writes, deletes persisted rows for inflight work, and removes
+  reverse-index entries incrementally.
+
+Measured lifecycle result:
+
+- Upkeep-only run `20260514232743` lowered cold churn setup p95 to
+  1097.75 ms.
+- Final `upkeep_subscriptions` row count: 0.
+- Final `upkeep_subscription_index_entries` row count: 0.
+- Real persistence batch p95: 137.6 ms.
+
+## Current Design Boundaries
+
+Keep these contracts separate:
+
+- Subscription storage: active and durable graph lookup.
+- Dispatching: render/delivery work scheduling.
+- ActionCable broadcast bus: how messages reach connected clients.
+- Benchmarking/telemetry: measurement only, not runtime policy.
+
+Multi-worker support is not done just because subscriptions persist. A
+multi-worker deployment needs:
+
+- A shared ActionCable adapter.
+- Durable graph lookup for workers that did not render the page.
+- Dispatch work that can leave the Puma process or use the app queue adapter.
+- Metrics that separate queue delay, invalidation proof, render cost, and
+  fanout cost.
+
+Do not add a second production persistence path. The direction is one
+production store contract and explicit queue/bus configuration around it.
+
+## Deoptimization Policy
+
+Keep the deoptimization surface small and actionable.
+
+Live deoptimizations stay only when Upkeep has already proven correctness and
+is merely choosing a broader operation than the cheapest possible Turbo Stream.
+Missing proof for a reactive boundary should refuse registration in a way that
+raises in development/test and warns or refuses in production-style settings.
+
+Every refusal should eventually include:
+
+- Stable reason name.
+- The concrete proof that was missing.
+- A refactor suggestion.
+- Enough context to connect it to a template or relation boundary.
+
+## What Not To Chase Next
+
+Fibers are not the next optimization for the measured bottleneck. The slow path
+is blocking ActiveRecord/SQLite persistence plus render/recording work under
+contention. Fibers would not make those writes non-blocking.
+
+A previous experiment with lowering the durable writer thread priority made
+setup p95 worse, around 2914 ms. Do not reintroduce it without a benchmark that
+proves a different result.
+
+Do not optimize subscription registration first. It is already at 0.13 ms p95
+in the latest comparison, roughly equal to Turbo.
+
+## Recommended Next Steps
+
+1. Diagnose the cold-admission gap with phase-level server and client evidence.
+
+   The server p95 phases in the latest report are small compared with the
+   client-observed gap:
+
+   | Server phase p95 | Upkeep | Turbo |
+   | --- | ---: | ---: |
+   | `sessions#create` | 8.28 ms | 5.98 ms |
+   | `rooms#show` | 12.69 ms | 3.83 ms |
+   | Subscription registration | 0.13 ms | 0.15 ms |
+   | Subscription confirmation | 0.06 ms | 0.04 ms |
+
+   That mismatch suggests queueing, contention, connection scheduling, or
+   missing instrumentation around the page/socket boundary. Add measurement
+   before changing behavior.
+
+2. Explain the single Upkeep subscription timeout.
+
+   Check `benchmark/results/matrix-chat_upkeep_cold_connect_churn-20260514233725.log`
+   and `benchmark/results/upkeep-app-server-20260514233725.log`. Determine
+   whether the timeout is ActionCable scheduling, subscription lookup, test
+   harness timing, or server pressure.
+
+3. Break down `/rooms/:id` render recording cost.
+
+   Compare Upkeep and Turbo page render work at the template/component level.
+   Separate normal Rails render time from dependency recording, marker
+   generation, subscription graph construction, and HTML size.
+
+4. Add an ActionCable connect phase for Upkeep in the comparison report.
+
+   The latest report shows Turbo cable connect p95 but `--` for Upkeep. That
+   makes the cold admission gap harder to localize.
+
+5. Only after the gap is localized, implement the smallest optimization and
+   rerun the same gate.
+
+   The loop should stay:
+
+   - Implement one optimization.
+   - Remove superseded code paths in the same pass.
+   - Update `docs/cost-model-roadmap.md` with the measured result.
+   - Run tests.
+   - Commit.
+
+## Commands
+
+Activate the expected Ruby/toolchain before Ruby or Bundler commands:
+
+```sh
+eval "$(mise activate zsh)"
+```
+
+Run the full test suite:
+
+```sh
+eval "$(mise activate zsh)" && ruby -S bundle exec ruby -Itest -e 'Dir["test/**/*_test.rb"].sort.each { |file| require_relative file }'
+```
+
+Run the cold churn comparison:
+
+```sh
+mkdir -p /tmp/upkeep-asdf-helper
+ln -sf /usr/bin/true /tmp/upkeep-asdf-helper/asdf
+eval "$(mise activate zsh)"
+PATH="/tmp/upkeep-asdf-helper:$PATH" BENCH_FAMILY=matrix BENCH_WORKLOAD=cold_connect_churn_chat BENCH_TIER=gate ruby benchmark/bin/run
+```
+
+Run the Upkeep-only gate:
+
+```sh
+mkdir -p /tmp/upkeep-asdf-helper
+ln -sf /usr/bin/true /tmp/upkeep-asdf-helper/asdf
+eval "$(mise activate zsh)"
+PATH="/tmp/upkeep-asdf-helper:$PATH" BENCH_UPKEEP_ONLY=1 BENCH_FAMILY=matrix BENCH_WORKLOAD=cold_connect_churn_chat BENCH_TIER=gate ruby benchmark/bin/run
+```
+
+## Current Bottom Line
+
+Upkeep now has a credible subscription storage architecture and the hot
+registration path is no longer the issue. The project should reframe the next
+performance work around cold admission and reliability under churn. Turbo is
+still ahead there; Upkeep has to get the cold tax low enough that precise warm
+invalidation and shared delivery work can pay it back.