upkeep-rails 0.1.0

data/docs/production_roadmap.md

@@ -0,0 +1,372 @@
+# Upkeep Rails Production Roadmap
+
+This repo contains the Rails-native reactive rendering gem and its in-repo
+dogfood apps.
+The production path is organized around structural proof, not host-maintained
+configuration. The framework must derive dependency and identity inputs from
+the render and request surfaces it observes.
+
+## Runtime Boundary
+
+Upkeep keeps three production concerns separate:
+
+- Subscription store: the queryable graph and reverse index used to decide
+  which subscribers and targets are affected by a lifecycle event. The current
+  production store is `config.upkeep.subscription_store = :active_record`.
+  `:memory` is an explicit development/test store only.
+- Dispatcher: the async execution boundary that moves delivery work off the
+  request thread. The current dispatcher is process-local batching. A future
+  ActiveJob dispatcher can use Solid Queue, Redis-backed queues, or whichever
+  queue adapter the host app has configured.
+- Broadcast bus: the final delivery path to browser connections. Upkeep uses
+  ActionCable; multi-worker deployments need a shared ActionCable adapter such
+  as Redis or Solid Cable.
+
+Missing production subscription tables are a boot/configuration error, not a
+reason to fall back to memory.
+
+## Design Boundary
+
+Rails runtime renderers are the authority for render shape. Upkeep should hook
+the normalized renderer choke points Rails already owns:
+
+- `ActionView::TemplateRenderer` for template/page frames.
+- `ActionView::PartialRenderer` and `ActionView::ObjectRenderer` for partial
+  and object-derived partial frames.
+- `ActionView::CollectionRenderer` for collection render-site frames and child
+  partial frames.
+- `ActionView::Renderer` only as an entry boundary when a lower hook cannot
+  prove the needed frame.
+
+Herb is an address and validation layer. It can produce stable source-location
+ids, verify single-root fragment eligibility, and improve static DOM tagging.
+It must not become a catalog of every Rails render spelling. If support for a
+render shape requires syntax-specific cases, the hook is at the wrong layer or
+the runtime proof is incomplete.
+
+## Production Gates
+
+### Gate 1: Gem-Shaped Runtime
+
+- Gem specification, load paths, `Upkeep::Rails` namespace, and `Railtie`.
+- Idempotent install hooks for ActionView, Active Record, ActionDispatch,
+  Warden, and `ActiveSupport::CurrentAttributes`.
+- Runtime code separated from proof-only templates and in-memory domain models.
+- Test runner that exercises both the proof harness and Rails integration tests.
+
+Exit criteria:
+
+- A Rails app can require the gem and boot with hooks installed.
+- Hooks are controlled by environment/config.
+- `bin/test` runs the gem tests and the proof runner.
+
+### Gate 2: ActionView Graph Capture
+
+- Template, partial, object partial, and collection render paths create graph
+  frames from Rails-resolved templates and locals.
+- Collection renders create render-site frames without parsing render syntax.
+- Active Record attribute and collection dependencies attach to the current
+  frame.
+- Current user, Warden user, CurrentAttributes, session, cookie, and request
+  dependencies attach to the frame that reads them.
+
+Exit criteria:
+
+- Tests prove several render spellings produce equivalent graph shape through
+  Rails renderer hooks.
+- The graph report explains selected targets through frame nodes, dependency
+  nodes, owner edges, and containment edges.
+
+### Gate 3: Canonical Replay Inputs
+
+- Page replay stores controller/action, params, request environment, assigns,
+  lookup context, variants, formats, and observed ambient identity inputs.
+- Render-site replay stores the Rails-resolved collection boundary and enough
+  canonical inputs to rerender membership for one subscriber.
+- Fragment replay stores Rails-resolved template identity, locals, variants,
+  formats, and identity inputs.
+- Replay never depends on host-declared dependency lists.
+
+Exit criteria:
+
+- Page, render-site, and fragment recipes render the selected target bytes in a
+  real Rails app.
+- Replay fails closed when canonical inputs are not structurally available.
+
+### Gate 4: Subscription Storage And Reverse Index
+
+- Store per-subscriber graph, identity signature, replay recipes, and selected
+  target metadata in the configured subscription store.
+- Build reverse indexes from lifecycle-selectable dependency keys to
+  subscriber graph owners.
+- Walk from changed dependency to affected frames across stored graphs.
+- Deduplicate by subscriber, target id, and identity signature.
+- Fail fast when production is configured for `:active_record` but the Upkeep
+  subscription tables are missing.
+
+Exit criteria:
+
+- An Active Record commit selects affected subscriber frames without scanning
+  every stored graph.
+- Identity-partitioned targets produce separate payloads unless identity
+  signatures prove equivalence.
+
+### Gate 5: Delivery Pipeline
+
+- Generate Turbo Stream replacement payloads for selected targets.
+- Partition payloads by identity signature.
+- Handle disconnect cleanup, retries, and backpressure.
+- Keep delivery separate from correctness proof; delivery consumes selected
+  target payloads and does not decide who is eligible.
+
+Exit criteria:
+
+- Benchmark app tests prove no cross-subscriber payload sharing when identity
+  observations differ.
+- Delivery reports expose selected dependency, frame, identity, and payload
+  digest evidence.
+
+### Gate 6: Production Hardening
+
+- Fail-closed handling for unknown dependencies, missing replay inputs, and
+  unaddressable DOM roots.
+- Benchmark workload coverage for the maintained Rails apps.
+- Memory, fan-out, and replay latency measurements.
+- Compatibility matrix across supported Rails versions.
+
+Exit criteria:
+
+- The gem can be used by the maintained benchmark app with no proof-only code
+  paths.
+- The green bar includes unit tests, Rails integration tests, proof runner, and
+  benchmark app tests.
+
+## First Session
+
+The first session targets Gates 1 and 2:
+
+1. Add gem/Railtie structure with idempotent install hooks.
+2. Add an ActionView capture proof that renders through Rails, not the in-memory
+   proof harness.
+3. Prove multiple render spellings produce graph frames through renderer hooks.
+4. Keep subscription storage and Turbo delivery out of the session unless the
+   renderer proof lands cleanly.
+
+The session is complete when the repo has commits for the roadmap, gem
+structure, and Rails-renderer graph capture with passing tests.
+
+## Second Session
+
+The second session targets the first Gate 3 proof:
+
+1. Attach replay recipes to Rails ActionView page, fragment, and render-site
+   frames.
+2. Replay Rails-resolved templates with captured locals and fresh Active Record
+   records or relations.
+3. Prove fragment replay reflects a changed record attribute.
+4. Prove render-site replay reflects a changed collection membership.
+5. Keep controller/request replay and delivery out of the session unless the
+   ActionView replay proof lands cleanly.
+
+The session is complete when Rails renderer tests prove selected ActionView
+targets can be rerendered from frame recipes without a full-page extraction.
+
+Current coverage:
+
+- Page recipes rerender Rails templates with fresh Active Record relations.
+- Render-site recipes rerender Rails collection boundaries with fresh
+  membership.
+- Fragment recipes rerender Rails-resolved partial templates with fresh Active
+  Record records.
+
+## Third Session
+
+The third session completes the first controller/request replay proof:
+
+1. Derive page replay from the controller attached to the Rails view context.
+2. Capture controller class, action name, request method, path, query string,
+   and path-parameter keys in the page frame metadata.
+3. Replay the page by re-entering `controller.class.action(action).call(env)`.
+4. Prove controller page replay reruns the action with request parameters and
+   fresh Active Record state.
+5. Keep durable subscription storage and delivery out of the session.
+
+Current coverage:
+
+- Controller-backed page recipes rerun the controller action.
+- Request query parameters are preserved for replay.
+- Controller replay uses fresh Active Record state through the action's own
+  query path.
+- Path parameter values are carried in the replay environment; route-level
+  dispatch through a full Rails application remains a later Gate 3 step.
+
+## Fourth Session
+
+The fourth session targets Gate 4:
+
+1. Store subscriber render graphs with their replay recipes and identity
+   signatures.
+2. Build a reverse index from observed dependency keys to graph owners.
+3. Plan invalidations from committed Active Record changes by walking from
+   matched dependency owners to containing frames.
+4. Deduplicate planned work by subscriber, target, and identity signature.
+5. Prove that identity-partitioned targets keep separate payloads and that
+   ambient identity observations stay attached to the replay target.
+
+Current coverage:
+
+- Active Record collection membership changes select render-site targets across
+  stored subscriptions through the reverse index.
+- Active Record attribute changes select fragment targets without scanning each
+  stored graph.
+- Duplicate subscriptions for the same subscriber, target, and identity
+  signature collapse to one planned target.
+- `Current.user` identity observations partition fragment payloads for users
+  with different visibility.
+- CurrentAttributes, session, cookie, request, and Warden observations are
+  preserved on page targets that read those surfaces.
+- ActiveRecord is the explicit production subscription store. Memory storage is
+  still available for development/test, but it is no longer selected as a
+  hidden fallback.
+
+## Fifth Session
+
+The fifth session targets the first Gate 5 delivery pass:
+
+1. Convert planned targets into Turbo Stream replacement payloads.
+2. Address page, fragment, and render-site targets through their observed
+   Upkeep DOM attributes.
+3. Partition payloads by subscriber, target, identity signature, and rendered
+   payload digest.
+4. Share rendered work across subscribers only when the target, identity
+   signature, sharing inputs, and replay recipe prove equivalence; merge
+   separately rendered payloads when the payload bytes also match.
+5. Report target, subscriber, identity, dependency, and payload digest evidence
+   for each stream and envelope.
+
+Current coverage:
+
+- Public fragment payloads and equivalent shared render-site payloads are
+  grouped before replay when the action, target, identity signature, sharing
+  signature, replay recipe, and deoptimization reason match.
+- Render-site payloads target the collection wrapper through a Turbo Stream
+  `targets` selector.
+- Identity-partitioned fragment payloads produce separate subscriber envelopes
+  when visible bytes differ.
+- Delivery reports expose the selected target, identity signature, matched
+  dependency keys, subscriber ids, and payload digest.
+
+Remaining Gate 5 work:
+
+- Keep the Rails runtime path on the broadcast transport and the lower-level
+  connection transport covered as separate delivery boundaries.
+
+## Sixth Session
+
+The sixth session targets the Gate 5 transport boundary:
+
+1. Track active subscriber connections by subscriber id.
+2. Deliver subscriber envelopes through an adapter protocol.
+3. Clean queued retry work when a subscriber disconnects.
+4. Retry failed sends up to a bounded attempt count.
+5. Apply per-connection backpressure with a bounded retry queue.
+6. Report delivery outcomes without selecting or filtering eligible targets.
+
+Current coverage:
+
+- Connected subscribers receive only their own envelopes.
+- Disconnected subscribers are reported without invoking an adapter.
+- Failed sends queue retry work and retry delivery preserves the original
+  envelope.
+- Repeated failures drop after the configured retry limit.
+- A full retry queue reports backpressure without growing retained work.
+- The Rails runtime delivery path broadcasts through ActionCable subscriber
+  streams without depending on process-local WebSocket connection ownership.
+
+Open Gate 5 work:
+
+- Keep delivery outcome reporting aligned across the connection transport and
+  the broadcast transport.
+- Add an ActiveJob dispatcher mode after the storage boundary is stable, so apps
+  can route async delivery through Solid Queue or another configured job
+  adapter without changing the subscription store.
+
+## Seventh Session
+
+The seventh session targets the Rails cable broadcast adapter:
+
+1. Add an ActionCable delivery adapter for the transport boundary.
+2. Broadcast each subscriber envelope body to a canonical subscriber stream.
+3. Hide raw subscriber ids from stream names.
+4. Emit delivery notifications with stream, subscriber, digest, and byte-size
+   evidence.
+5. Prove ActionCable broadcast failures flow through transport retry handling.
+
+Current coverage:
+
+- The adapter broadcasts envelope bodies through `ActionCable.server.broadcast`
+  compatible servers.
+- Subscriber stream names are derived from a digest of the subscriber id.
+- `deliver.upkeep` notifications expose broadcast evidence without carrying
+  rendered payload bytes.
+- Transport retries failed ActionCable broadcasts through the existing retry
+  queue.
+- The Rails cable subscription side derives server identity from the
+  ActionCable connection and streams from each canonical identity stream.
+- The maintained Upkeep benchmark app covers streamed delivery through the
+  derived subscriber stream.
+- The install generator creates subscription storage, an initializer, an
+  ActionCable route mount, importmap wiring, and the browser subscription
+  bootstrap.
+- The browser subscription bootstrap reads injected subscription markers,
+  subscribes through ActionCable, and applies Turbo Stream payloads.
+
+## Benchmark Layout
+
+The maintained benchmark apps live under `benchmark/upkeep-app` and
+`benchmark/turbo-app` in this repo. The committed benchmark tree contains
+source and harness files; generated results, logs, temporary files, runtime
+databases, Bundler lockfiles, Playwright install artifacts, and credentials are
+excluded.
+
+Current coverage:
+
+- The Herb surface probe scans the in-repo benchmark apps.
+- `benchmark/bin/run` defaults to the `matrix` family, which targets the
+  in-repo `upkeep-app` and `turbo-app`.
+- The gem test suite verifies the benchmark tree points at the in-repo apps
+  and excludes generated runtime artifacts.
+- `bin/test` runs the gem tests, both benchmark app test suites, and the proof
+  runner as one gate.
+- The benchmark app suites cover authenticated board rendering, room rendering,
+  shared feed rendering, authorization boundaries, and helper-hidden render
+  idioms.
+- The Rails cable subscription boundary derives subscriber identity from
+  ActionCable connection identifiers and rejects unidentified connections.
+- Subscription graphs persist graph snapshots and reverse-index rows before
+  registration returns, then register into the active in-process reverse index
+  on the request path.
+- Persisted Active Record subscription rows rehydrate recipes that render after
+  a fresh store load.
+- Controller requests automatically capture render graphs, register successful
+  HTML responses, inject the client subscription marker, and drain committed
+  Active Record changes into streamed delivery.
+- The Upkeep benchmark app covers automatic subscription registration and
+  streamed delivery through the canonical subscriber stream.
+- Active Record collection dependencies derive table and column coverage from
+  the relation's Arel shape. Proven relations index concrete table/column
+  lookup keys; unrelated column writes do not select those collections; opaque
+  predicates and table sources are refused instead of registering broad
+  invalidation dependencies.
+- Identity, request, session, and cookie dependencies stay on the stored graph
+  for replay and sharing, but do not create lifecycle invalidation lookup rows.
+- Collection append delivery is gated by Active Record relation shape and
+  remains a delivery optimization over the canonical render-site replacement
+  recipe.
+
+Open work:
+
+- Add compatibility matrix runs, memory/fan-out measurements, and replay
+  latency gates for the maintained benchmark workloads.
+- Add a multi-worker benchmark gate with the ActiveRecord subscription store
+  and a shared ActionCable adapter.

data/docs/shared-warm-scale-roadmap.md

@@ -0,0 +1,214 @@
+# Shared Warm Scale Roadmap
+
+This roadmap targets the shape where Upkeep should outperform Turbo: many
+subscribers share an identity-free page and one mutation fans out to all of
+them.
+
+Last updated: 2026-05-21.
+
+Current code state:
+
+- Base runtime/docs commit: `474fb9c` (`Instrument capture and document Upkeep Rails`).
+- Current pass adds gated request/action profiling, operation-scoped capture
+  metrics, mutation action/delivery timings, client/server phase correlation,
+  benchmark phase labels, lightweight subscription-shape tracing, static
+  template metadata caching, shared-stream signature memoization, and queued
+  dependency flushing.
+- Latest verification: `bundle exec rake test` passed with `186` runs and
+  `1172` assertions.
+- Latest shared benchmark data is the 2026-05-21 ramped 200-subscriber report
+  below.
+
+## Environment
+
+From a fresh zsh shell in any worktree:
+
+```sh
+cd /path/to/upkeep-rails
+eval "$(mise activate zsh)"
+ruby -v
+bundle exec rake test
+```
+
+Do not use system Ruby. The repo expects Ruby from mise, currently `3.4.7`.
+
+## Baseline
+
+Latest ramped identity-free 200-subscriber report:
+
+- Report: `benchmark/results/identity-free-feed-compare-20260521113822.md`
+- Upkeep setup p95: `58ms`
+- Turbo setup p95: `38.05ms`
+- Upkeep page render p95: `47ms`
+- Turbo page render p95: `24.05ms`
+- Upkeep WebSocket connect p95: `9ms`
+- Turbo WebSocket connect p95: `11ms`
+- Upkeep subscribe call p95: `6ms`
+- Turbo subscribe call p95: `2ms`
+- Upkeep subscribe ack p95: `5ms`
+- Turbo subscribe ack p95: `1.05ms`
+- Upkeep write POST p95: `58.90ms`
+- Turbo write POST p95: `58.69ms`
+- Upkeep update-settled p95: `191ms`
+- Turbo update-settled p95: `342.10ms`
+- Upkeep render groups: `1`
+- Upkeep represented subscribers: `200`
+- Turbo refresh GETs: `200`
+- Upkeep render dedup savings: `199`
+- Upkeep plans: `1`
+- Upkeep stream batches: `1`
+- Subscription shape cache: `199` hits, `1` miss, `0` bypasses
+- Shape timing: hit key p95 `0.031ms`; hit total p95 `0.049ms`; miss total
+  `4.324ms`
+- Setup page server timing: Upkeep p95 `34.28ms`; Turbo p95 `10.12ms`
+- Write request server timing: Upkeep p95 `15.74ms`; Turbo p95 `34.74ms`
+- `GET /feed` request-capture timing: total p95 `33.081ms`; action p95
+  `14.896ms`; view p95 `15.456ms`; template p95 `11.104ms`; collection render
+  p95 `4.362ms`; SQL p95 `0.191ms`; register p95 `3.814ms`
+- `GET /feed` recorder timing: dependency p95 `3.916ms`; frame p95 `1.364ms`;
+  shape trace p95 `3.588ms`
+- Subscribe channel server timing: total p95 `1.884ms`; activation p95
+  `1.776ms`
+- Upkeep live deopts: `0`
+
+The shared primitive is working: one anonymous shape, one plan, one render, and
+one stream batch fan out to 200 subscribers. This pass removed the full-recorder
+shape key from the cache path, removed the O(n) cohort rebuild from each
+subscriber activation, moved subscription-shape identity into DAG/runtime, and
+made shape hits use a recorder-side rolling trace digest. This pass then
+trimmed the trace and dependency path further by removing unused trace snapshots,
+excluding duplicate manifest data from shape terms, memoizing render-site stream
+signatures, caching static template metadata, and flushing dependencies in
+batches per frame/request. Shape hits and server-side subscription activation
+are no longer shared-path bottlenecks. The next work is reducing Action View
+template/page capture overhead while preserving the one-render fanout path.
+
+## Priorities
+
+1. Close page/capture fixed cost
+
+   Upkeep update delivery is much faster, and shape hits are no longer a
+   meaningful cost. In the latest shared report, setup page server p95 is
+   `34.28ms` versus Turbo `10.12ms`. `GET /feed` request-capture total p95 is
+   `33.081ms`, action p95 is `14.896ms`, view p95 is `15.456ms`, and
+   registration p95 is `3.814ms`.
+
+   Candidate work:
+
+   - Reduce Action View capture overhead around template and collection render
+     instrumentation. SQL is only `0.273ms` p95 in the shared report.
+   - Continue reducing Action View template capture overhead; template p95 is
+     now the largest named subphase at `11.104ms`.
+   - Keep dependency flushing batched and use `recorder_dependency_flush_count`
+     to separate read count from unique graph inserts.
+   - Investigate whether page/layout frame capture can be represented with one
+     page boundary without losing layout identity safety.
+   - Preserve the rolling trace digest as order-sensitive: false cache misses
+     are acceptable, unsafe sharing is not.
+
+   Target outcome:
+
+   - 200-subscriber `GET /feed` request-capture total p95 falls from
+     `33.081ms` toward `15-20ms`.
+   - Page render p95 closes most of the remaining shared setup gap.
+
+2. Explain client setup gap
+
+   Server-side subscribe p95 is `1.884ms` while client-observed subscribe ack
+   p95 is `5ms`, and setup p95 is `58ms`. Server/client phase correlation
+   now shows that the largest shared setup gap is setup page request time:
+   Upkeep `34.28ms` versus Turbo `10.12ms`.
+
+   Candidate work:
+
+   - Keep phase labels on page setup, Turbo refresh, write POST, WebSocket
+     connect, cable open, subscription registration, and confirmation.
+   - Separate browser/k6 scheduling delay from Rails server time when comparing
+     setup p95.
+   - Filter health checks out of request-capture operation reporting so
+     `GET /feed` stays the visible target.
+
+   Target outcome:
+
+   - The report continues to explain the difference between `33.081ms`
+     server capture p95 and `58ms` client setup p95 without mixing Turbo
+     refresh GETs into setup-page timings.
+
+3. Explain and reduce write-path variance
+
+   Latest write POST p95 is `58.90ms` for Upkeep and `58.69ms` for Turbo.
+   Server-side write request p95 is `15.74ms` for Upkeep and `34.74ms` for
+   Turbo. Keep this instrumented so future fanout work does not hide write-side
+   regressions.
+
+   Candidate work:
+
+   - Split the current mutation action timing into Active Record write,
+     invalidation planning, stream build, ActionCable enqueue, and response
+     return.
+   - Separate writer VU timing from subscriber delivery timing in the report.
+   - Verify the write regression is not caused by shape-cache bookkeeping.
+
+   Target outcome:
+
+   - The report identifies whether write latency is database, planning, delivery
+     enqueue, or benchmark noise.
+
+4. Fanout packing
+
+   Upkeep still has one unavoidable frame per subscriber, but payload and
+   envelope work should happen once per render group.
+
+   Candidate work:
+
+   - Serialize payload once.
+   - Build frame/envelope once per stream group when possible.
+   - Keep per-connection work to stream lookup and socket write.
+
+   Target outcome:
+
+   - 500/1000 subscriber runs preserve one render group and avoid a new CPU
+     bottleneck in payload packing.
+
+5. Benchmark/report hygiene
+
+   The ramped benchmark is now the right warm shared shape. The report should
+   continue to make fixed-cost and shared-cost claims separately.
+
+   Candidate work:
+
+   - Keep setup ramp/window fields, `steady_state_setup_leaks`, plans, stream
+     batches, represented subscribers, and shape-cache counters in markdown and
+     JSON.
+   - Keep shape-cache timing, request-capture timing, and server-side subscribe
+     timing in markdown and JSON.
+   - Keep client setup phase correlation and mutation write phase timing.
+   - Keep request-capture timings grouped by operation.
+   - Keep cold burst capacity and single-subscriber cold setup as separate
+     reports.
+
+## Success Criteria
+
+- 200-subscriber update-settled p95 remains materially faster than Turbo.
+- 200-subscriber setup p95 explains and narrows the current `58ms` versus
+  Turbo `38.05ms` gap while preserving one-render fanout.
+- 500-subscriber report shows one render group, high delivery/render ratio, and
+  no setup leaks.
+- Planning/build batch count for the one-write shared feed path stays at one.
+- Shape cache has one miss and N-1 hits for identical anonymous pages.
+- Anonymous deopts remain zero for identity-free pages.
+- Live delivery deopts remain zero for the common feed create case.
+
+## Validation
+
+Use the mise environment in every shell:
+
+```sh
+eval "$(mise activate zsh)"
+bundle exec rake test
+BENCH_FAMILY=render_dedup BENCH_WORKLOAD=identity_free_feed_compare BENCH_TIER=report BENCH_VUS=200 ruby benchmark/bin/run
+BENCH_FAMILY=render_dedup BENCH_WORKLOAD=identity_free_feed_compare BENCH_TIER=report BENCH_VUS=500 ruby benchmark/bin/run
+```
+
+Use the warm shared benchmark for shared-update economics. Do not use cold
+burst failure or accept-queue behavior to judge render dedup performance.