upkeep-rails 0.1.0

data/docs/single-subscriber-cold-roadmap.md

@@ -0,0 +1,192 @@
+# Single Subscriber And Cold Fixed-Cost Roadmap
+
+This roadmap targets the cases where Upkeep has little or no render sharing
+amortization: one active subscriber, cold page setup, cable activation, and
+short-lived anonymous pages.
+
+Last updated: 2026-05-21.
+
+Current code state:
+
+- Base runtime/docs commit: `474fb9c` (`Instrument capture and document Upkeep Rails`).
+- Current pass adds gated request/action profiling, operation-scoped capture
+  metrics, mutation action/delivery timings, client/server phase correlation,
+  benchmark phase labels, lightweight subscription-shape tracing, static
+  template metadata caching, shared-stream signature memoization, and queued
+  dependency flushing.
+- Latest verification: `bundle exec rake test` passed with `186` runs and
+  `1172` assertions.
+- Latest single-subscriber benchmark data is the 2026-05-21 identity-free
+  report below.
+
+## Environment
+
+From a fresh zsh shell in any worktree:
+
+```sh
+cd /path/to/upkeep-rails
+eval "$(mise activate zsh)"
+ruby -v
+bundle exec rake test
+```
+
+Do not use system Ruby. The repo expects Ruby from mise, currently `3.4.7`.
+
+## Baseline
+
+Latest identity-free 1-subscriber report:
+
+- Report: `benchmark/results/identity-free-feed-compare-20260521114026.md`
+- Upkeep setup p95: `140ms`
+- Turbo setup p95: `104ms`
+- Upkeep page render p95: `91ms`
+- Turbo page render p95: `73ms`
+- Upkeep WebSocket connect p95: `38ms`
+- Turbo WebSocket connect p95: `17ms`
+- Upkeep subscribe call p95: `10ms`
+- Turbo subscribe call p95: `14ms`
+- Upkeep subscribe ack p95: `10ms`
+- Turbo subscribe ack p95: `13ms`
+- Upkeep write POST p95: `38.85ms`
+- Turbo write POST p95: `66.37ms`
+- Upkeep update-settled p95: `90ms`
+- Turbo update-settled p95: `96ms`
+- Subscription shape cache: `1` miss, `0` hits, `0` bypasses
+- Shape miss timing: total `4.221ms`, key `1.743ms`, index template `1.003ms`
+- Setup page server timing: Upkeep p95 `83.88ms`; Turbo p95 `66.02ms`
+- Write request server timing: Upkeep p95 `23.78ms`; Turbo p95 `32.72ms`
+- `GET /feed` request-capture timing: total p95 `82.921ms`; action p95
+  `70.594ms`; view p95 `55.578ms`; template p95 `42.352ms`; collection render
+  p95 `13.226ms`; SQL p95 `1.577ms`; register p95 `12.129ms`
+- `GET /feed` recorder timing: dependency p95 `5.783ms`; frame p95 `2.418ms`;
+  shape trace p95 `4.264ms`
+- Subscribe channel server timing: total `2.085ms`, activation `1.95ms`
+
+With one subscriber there is still no render dedup savings. The latest cold
+report is a one-sample fixed-cost probe and is noisier than the shared report.
+It still shows the right target: the remaining gap is not SQL, shape cache hit
+cost, or server-side channel subscribe. This pass reduced recorder dependency
+and shape tracing substantially, so the dominant cold fixed cost is now the
+first page/action path, especially Action View template and collection capture,
+plus first registration.
+
+## Priorities
+
+1. Decompose cold page/capture setup
+
+   Upkeep page render p95 is now `91ms` versus Turbo `73ms`, while setup p95
+   is `140ms` versus Turbo `104ms`. `GET /feed` request-capture action p95 is
+   `70.594ms`, view p95 is `55.578ms`, and template p95 is `42.352ms`.
+
+   Candidate work:
+
+   - Reduce Action View capture overhead around template and collection render
+     instrumentation. SQL is only `1.735ms` in this run.
+   - Keep ActionCable open/confirmation separate from server-side channel
+     subscribe; latest server-side subscribe total is only `2.085ms`.
+   - Continue reducing Action View template capture overhead; template p95 is
+     `42.352ms` in the latest one-subscriber probe.
+   - Investigate whether page/layout frame capture can be represented with one
+     page boundary without losing layout identity safety.
+   - Keep queued dependency flushing; the recorder shape trace has already
+     dropped from `10.04ms` to `4.264ms`.
+
+   Target outcome:
+
+   - 1-subscriber page render p95 closes the current `18ms` gap to Turbo.
+   - Setup telemetry explains the remaining `36ms` setup gap.
+
+2. Finish shape miss/key cleanup
+
+   Shape keys no longer serialize the full recorder snapshot, and the
+   subscription-shape trace now records normal capture shape terms in
+   DAG/runtime. Shared-hit key p95 is `0.031ms`; the single-subscriber miss path
+   is still `1.743ms` and total miss is `4.221ms`, but it is a first-shape
+   cost.
+
+   Remaining work:
+
+   - Preserve the rolling trace digest for hot hits.
+   - Identify why first-shape key generation still costs `1.743ms`.
+   - Move template digest and manifest fingerprint normalization out of the
+     miss path where possible.
+   - Keep cache misses exact without replay-payload hashing.
+
+   Target outcome:
+
+   - Shape-key generation is sub-millisecond for both miss and hit paths.
+   - Cache misses stay exact, and false misses remain safe.
+
+3. Reduce subscribe ack fixed cost
+
+   Server-side subscribe is no longer the main cost, but client-observed suback
+   can still be noisy. Keep the server/client split visible.
+
+   Candidate work:
+
+   - Keep cable subscribe phase telemetry: fetch, authorization, activation,
+     active-index register, stream attach, confirmation.
+   - Keep benchmark phase labels on WebSocket open, cable open, subscription
+     registration, and confirmation.
+   - Keep durable writer and persistent index work outside the ack critical path.
+   - Preserve strict activation semantics; no compatibility shims for old rows.
+
+   Target outcome:
+
+   - Server-side subscribe p95 stays within a few milliseconds of Turbo in both
+     1- and 200-subscriber identity-free reports.
+
+4. Single-subscriber write fast path
+
+   The latest one-subscriber run recovered on write and update-settled, but the
+   one-sample p95 is noisy. Keep this lane explicit: if planning resolves to
+   exactly one active subscriber, skip shared-group work that only pays off for
+   cohorts.
+
+   Candidate checks:
+
+   - No render dedup accounting on the hot path for a single target unless the
+     runtime already produced the shared group.
+   - No batch fanout structure for one connection.
+   - No persistent index dependency for active, in-process delivery.
+
+   Target outcome:
+
+   - 1-subscriber update-settled returns to at or below Turbo.
+   - Planning/build telemetry shows one lookup, one plan, one render, one
+     transmit.
+
+5. Cold benchmark separation
+
+   Keep cold burst capacity separate from warm update economics.
+
+   Benchmark shapes:
+
+   - Cold burst capacity: simultaneous new users, accepts, page render, cable
+     open, subscribe ack.
+   - Cold ramp setup: staged subscribers without accept queue loss.
+   - Warm single update: subscribers are ready, one mutation settles.
+
+## Success Criteria
+
+- 1-subscriber first-write miss behavior is explained and bounded.
+- `steady_state_setup_leaks` is zero in warm single-subscriber runs.
+- 1-subscriber setup p95 closes most of the current `140ms` versus Turbo
+  `104ms` gap.
+- 1-subscriber page render p95 closes most of the current `91ms` versus Turbo
+  `73ms` gap.
+- Shape miss telemetry shows where first-render cost remains.
+- 1-subscriber update-settled p95 remains at or below Turbo.
+
+## Validation
+
+Use the mise environment in every shell:
+
+```sh
+eval "$(mise activate zsh)"
+bundle exec rake test
+BENCH_FAMILY=render_dedup BENCH_WORKLOAD=identity_free_feed_compare BENCH_TIER=report BENCH_VUS=1 ruby benchmark/bin/run
+```
+
+When changing cold setup behavior, also run the relevant cold benchmark instead
+of using the warm shared report as a proxy.

data/docs/testing.md

@@ -0,0 +1,113 @@
+# Testing
+
+The Rails package green bar includes the gem tests, the maintained benchmark
+apps, and the proof runner. The benchmark apps live inside this repo under
+`benchmark/`.
+
+## Commands
+
+Run the gem test suite:
+
+```sh
+mise exec -- ruby -S rake test
+```
+
+Run the full gate:
+
+```sh
+mise exec -- ruby bin/test
+```
+
+`bin/test` runs:
+
+- all tests under `test/`;
+- `benchmark/upkeep-app`'s Rails test suite;
+- `benchmark/turbo-app`'s Rails test suite;
+- `bin/run`, which writes proof reports to `results/`.
+
+Run the proof runner directly:
+
+```sh
+mise exec -- ruby bin/run
+```
+
+## What The Gate Covers
+
+Gem tests cover:
+
+- Action View frame capture and replay;
+- controller request replay;
+- Active Record/Arel query dependency analysis;
+- subscription storage and reverse-index lookups;
+- refused opaque collection boundaries;
+- column-scoped collection lookup keys;
+- invalidation planning;
+- Turbo Stream delivery partitioning;
+- render grouping before replay;
+- ActionCable subscriber identity and channel behavior;
+- transport retry and backpressure behavior.
+
+The Upkeep benchmark app covers:
+
+- authenticated board rendering;
+- room rendering;
+- shared feed rendering;
+- authorization boundaries;
+- helper-hidden render idioms;
+- automatic subscription registration;
+- streamed delivery through canonical subscriber streams.
+
+The Turbo benchmark app remains the comparison app for workload parity.
+
+## App-Level Assertions
+
+Maintained Rails apps should assert that successful HTML GETs register a
+subscription:
+
+```ruby
+include Upkeep::Rails::Testing
+
+get board_path(board)
+assert_response :success
+assert_upkeep_subscription_registered
+```
+
+They should also assert that unauthorized bytes do not render:
+
+```ruby
+get board_path(private_board)
+assert_response :forbidden
+refute_includes response.body, "Private card"
+```
+
+For streamed delivery, include `ActionCable::TestHelper`, capture the
+registered stream names through `Upkeep::Rails::Testing`, perform a mutation,
+drain delivery, and assert the broadcast payload:
+
+```ruby
+include ActionCable::TestHelper
+include Upkeep::Rails::Testing
+
+broadcasts = capture_upkeep_broadcasts do
+  patch board_card_path(board, card), params: { card: { title: "Updated" } }
+  assert_response :ok
+  Upkeep::Rails.drain_delivery!
+end
+
+assert_includes broadcasts.join, "Updated"
+refute_includes broadcasts.join, "Other subscriber secret"
+```
+
+## Result Reports
+
+The proof runner writes:
+
+- `results/herb_surface.json`
+- `results/active_record_surface.json`
+- `results/end_to_end_proof.json`
+- `results/identity_safety_proof.json`
+- `results/auth_surfaces_proof.json`
+
+These reports are debugging artifacts for maintained app work. They are useful
+when a change alters frame coverage, dependency metadata, identity
+partitioning, or target selection.

data/lib/generators/upkeep/install/install_generator.rb

@@ -0,0 +1,90 @@
+# frozen_string_literal: true
+
+require "rails/generators"
+require "rails/generators/active_record"
+require "pathname"
+
+module Upkeep
+  class InstallGenerator < ::Rails::Generators::Base
+    include ::Rails::Generators::Migration
+
+    source_root File.expand_path("templates", __dir__)
+
+    def self.next_migration_number(dirname)
+      ActiveRecord::Generators::Base.next_migration_number(dirname)
+    end
+
+    def create_subscription_migration
+      return if migration_exists?("create_upkeep_subscriptions")
+
+      @migration_version = ActiveRecord::Migration.current_version
+      migration_template "create_upkeep_subscriptions.rb.erb", "db/migrate/create_upkeep_subscriptions.rb"
+    end
+
+    def create_initializer
+      template "upkeep.rb", "config/initializers/upkeep.rb"
+    end
+
+    def create_browser_bootstrap
+      template "subscription.js", "app/javascript/upkeep/subscription.js"
+      append_application_import
+      pin_action_cable
+    end
+
+    def mount_action_cable
+      return if routes_path.exist? && routes_path.read.include?("ActionCable.server")
+
+      route %(mount ActionCable.server => "/cable")
+    end
+
+    private
+
+    def migration_exists?(name)
+      Dir.glob(destination_path("db/migrate/*.rb")).any? do |path|
+        File.basename(path).include?(name)
+      end
+    end
+
+    def append_application_import
+      return unless application_js_path.exist?
+
+      append_import("@hotwired/turbo-rails")
+      append_import("./upkeep/subscription")
+    end
+
+    def pin_action_cable
+      return unless importmap_path.exist?
+
+      pin_importmap("@hotwired/turbo-rails", "turbo.min.js")
+      pin_importmap("@rails/actioncable", "actioncable.esm.js")
+    end
+
+    def append_import(specifier)
+      return if application_js_path.read.include?(specifier)
+
+      append_to_file application_js_path.to_s, %(import "#{specifier}"\n)
+    end
+
+    def pin_importmap(specifier, asset)
+      return if importmap_path.read.include?(%("#{specifier}"))
+
+      append_to_file importmap_path.to_s, %(pin "#{specifier}", to: "#{asset}"\n)
+    end
+
+    def routes_path
+      Pathname(destination_path("config/routes.rb"))
+    end
+
+    def application_js_path
+      Pathname(destination_path("app/javascript/application.js"))
+    end
+
+    def importmap_path
+      Pathname(destination_path("config/importmap.rb"))
+    end
+
+    def destination_path(path)
+      File.join(destination_root, path)
+    end
+  end
+end

data/lib/generators/upkeep/install/templates/create_upkeep_subscriptions.rb.erb

@@ -0,0 +1,31 @@
+class CreateUpkeepSubscriptions < ActiveRecord::Migration[<%= @migration_version %>]
+  def change
+    create_table :upkeep_subscriptions, id: :string do |t|
+      t.string :subscriber_id, null: false
+      t.json :recorder_snapshot, null: false
+      t.json :metadata
+      t.timestamps
+    end
+    add_index :upkeep_subscriptions, :subscriber_id
+
+    create_table :upkeep_subscription_index_entries do |t|
+      t.string :subscription_id, null: false
+      t.string :lookup_key_digest, null: false
+      t.string :dependency_source, null: false
+      t.string :lookup_table, null: false
+      t.json :lookup_record_id_snapshot
+      t.string :lookup_attribute, null: false
+      t.string :dependency_table, null: false
+      t.string :dependency_predicate_digest
+      t.json :dependency_metadata_snapshot
+      t.json :owner_ids_snapshot, null: false
+      t.timestamps
+    end
+    add_index :upkeep_subscription_index_entries, :subscription_id
+    add_index :upkeep_subscription_index_entries, :lookup_key_digest
+    add_foreign_key :upkeep_subscription_index_entries,
+      :upkeep_subscriptions,
+      column: :subscription_id,
+      on_delete: :cascade
+  end
+end

data/lib/generators/upkeep/install/templates/subscription.js

@@ -0,0 +1,107 @@
+import { createConsumer } from "@rails/actioncable"
+import { Turbo } from "@hotwired/turbo-rails"
+
+const MARKER_SELECTOR = "script[data-upkeep-subscription]"
+
+let consumer
+const subscriptions = new Map()
+
+function cableConsumer() {
+  consumer ||= createConsumer()
+  return consumer
+}
+
+function markerPayloads() {
+  return Array.from(document.querySelectorAll(MARKER_SELECTOR)).map((marker) =>
+    JSON.parse(marker.textContent || "{}")
+  )
+}
+
+function currentSubscriptionIds() {
+  return new Set(markerPayloads().map((payload) => payload.subscription_id).filter(Boolean))
+}
+
+function applyTurboStreams(html) {
+  if (applyDocumentPageStream(html)) return
+
+  if (Turbo?.renderStreamMessage) {
+    Turbo.renderStreamMessage(String(html))
+    return
+  }
+
+  const template = document.createElement("template")
+  template.innerHTML = String(html)
+
+  template.content.querySelectorAll("turbo-stream").forEach((stream) => {
+    document.body.appendChild(stream)
+  })
+}
+
+function applyDocumentPageStream(html) {
+  const template = document.createElement("template")
+  template.innerHTML = String(html)
+
+  const stream = Array.from(template.content.querySelectorAll("turbo-stream")).find((candidate) =>
+    targetsDocumentElement(candidate)
+  )
+  if (!stream) return false
+
+  const nextDocument = stream.querySelector("template")?.innerHTML
+  if (!nextDocument) return false
+
+  document.open()
+  document.write(nextDocument)
+  document.close()
+  return true
+}
+
+function targetsDocumentElement(stream) {
+  const selector = stream.getAttribute("targets") || stream.getAttribute("target")
+  if (!selector) return false
+
+  try {
+    return Array.from(document.querySelectorAll(selector)).includes(document.documentElement)
+  } catch {
+    return false
+  }
+}
+
+function subscribe(payload) {
+  if (!payload.subscription_id || subscriptions.has(payload.subscription_id)) return
+
+  const subscription = cableConsumer().subscriptions.create(
+    {
+      channel: payload.channel || "Upkeep::Rails::Cable::Channel",
+      subscription_id: payload.subscription_id
+    },
+    {
+      received(data) {
+        applyTurboStreams(data)
+      }
+    }
+  )
+
+  subscriptions.set(payload.subscription_id, subscription)
+}
+
+function unsubscribeMissing() {
+  const liveIds = currentSubscriptionIds()
+
+  subscriptions.forEach((subscription, subscriptionId) => {
+    if (liveIds.has(subscriptionId)) return
+
+    subscription.unsubscribe()
+    subscriptions.delete(subscriptionId)
+  })
+}
+
+export function connectUpkeepSubscriptions() {
+  markerPayloads().forEach(subscribe)
+  unsubscribeMissing()
+}
+
+document.addEventListener("DOMContentLoaded", connectUpkeepSubscriptions)
+document.addEventListener("turbo:load", connectUpkeepSubscriptions)
+document.addEventListener("turbo:render", connectUpkeepSubscriptions)
+
+connectUpkeepSubscriptions()

data/lib/generators/upkeep/install/templates/upkeep.rb

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+Rails.application.configure do
+  config.upkeep.enabled = true
+  config.upkeep.subscription_store = :active_record
+end