upkeep-rails 0.1.0

data/docs/architecture/ambient-inputs-roadmap.md

@@ -0,0 +1,306 @@
+# Ambient Inputs Roadmap
+
+Rails apps commonly derive render state through controller and request
+surfaces before the view renders. Upkeep should support that idiom without
+requiring app-declared identity dimensions and without overfitting to any
+single benchmark app.
+
+The design rule is: observe the Rails-owned surface, scope the value to the
+target that needs it, replay only the observed inputs, and never turn ambient
+inputs into lifecycle invalidation lookup rows.
+
+## Rails Idioms To Support
+
+- `before_action` authentication that derives `@user` from session, Warden,
+  Devise, cookies, or another Rails request surface.
+- Controller-built Active Record relations whose shape depends on the current
+  user, params, request values, or cookie-backed preferences.
+- Views and helpers that read controller instance state such as `@user`,
+  `@page`, `@domain`, and filter state prepared by the controller.
+- Cookie-backed preferences and filters such as theme, compact mode, and tag
+  filters.
+- Request-shaped pages that depend on params, path, subdomain, request method,
+  user agent, or remote IP.
+- Mixed public/private pages where public render sites can still share while
+  identity-bound fragments or render sites remain partitioned.
+
+## Non-Goals
+
+- No whole-session snapshot as the steady-state replay model.
+- No Lobsters-specific keys such as `session[:u]` or `cookies[:tag_filters]`
+  in the runtime.
+- No host-maintained allow-list of identity dimensions.
+- No session, cookie, request, Warden, CurrentAttributes, or ActionCable
+  dependency rows in lifecycle invalidation indexes.
+- No silent broad fallback for opaque Active Record relation shape.
+
+## Current Checkpoint
+
+The Lobsters benchmark app is present as a broad Rails workload. Controller
+page replay now rebuilds Rack session state and plain cookie headers from
+observed reads instead of snapshotting the whole Rack session or original
+`HTTP_COOKIE`. Request reads are recorded in the same ambient replay-input
+ledger, but target-level request scoping remains a later phase.
+
+The reverse-index boundary is already correct: lifecycle lookup keys are still
+limited to Active Record attribute and collection dependencies.
+
+## Implementation Loop
+
+For each step:
+
+1. Add or update focused tests first.
+2. Implement the narrow runtime behavior.
+3. Remove the old broader path in the same slice.
+4. Run the focused tests and `git diff --check`.
+5. Update this roadmap with status and evidence.
+6. Commit before moving to the next step.
+
+## Phase 1: Observed Ambient Replay Inputs
+
+Goal: replay only ambient values that were actually observed.
+
+Work:
+
+- Add a recorder-owned ambient replay input ledger for session, cookie, and
+  request reads.
+- Have session, cookie, and request observers record both the identity
+  dependency and the replay input.
+- Replace whole `rack.session` snapshots with a replay session built from
+  observed session keys plus a stable session id when Rails exposes one.
+- Keep raw replay values out of dependency identity keys; dependency identity
+  continues to use private fingerprints.
+- Add tests proving unrelated session keys do not enter replay recipes or
+  shared stream signatures.
+
+Exit criteria:
+
+- A controller page that reads `session[:account_token]` can replay.
+- An unread session key is absent from the recipe.
+- Changing an unread session key does not change a public render-site sharing
+  signature.
+- No whole-session snapshot path remains.
+
+Status: landed for observed session replay inputs. Evidence:
+
+- Controller page replay survives `session[:account_token]`-style reads.
+- Unread Rack session keys are absent from the serialized recipe.
+- Controller page replay survives plain `cookies[:theme]`-style reads.
+- Unread cookie keys are absent from the replay `HTTP_COOKIE` header.
+- Changing an unread session key does not change the public render-site shared
+  stream names for the page.
+- The old whole-session `to_hash` / `to_h` snapshot path has been removed from
+  controller page replay.
+
+Remaining follow-up: none for this phase.
+
+## Phase 2: Scoped Identity And Replay Inputs
+
+Goal: identity and replay inputs should affect only the targets that depend on
+them.
+
+Work:
+
+- Make page targets inherit request-level ambient inputs from controller and
+  before-action reads.
+- Keep render-site and fragment targets public only when their render subtree
+  has no ambient identity dependency and their replay recipe does not depend on
+  unproven controller ambient state.
+- Partition render sites that directly read session, cookie, request, Warden,
+  CurrentAttributes, or ActionCable identity.
+- Deopt to a known enclosing page target, or refuse the narrower boundary, when
+  a render-site recipe depends on hidden controller state that cannot be scoped.
+
+Exit criteria:
+
+- A public render site inside a session-authenticated page can still use a
+  shared stream when its bytes do not depend on the user.
+- A render site that reads `cookies[:theme]` is identity-bound by that cookie.
+- A helper that depends on hidden controller state either gets a safe enclosing
+  target or a refused-boundary reason.
+
+Status: page/request identity inheritance and render-subtree identity landed.
+Evidence:
+
+- Page frame identity profiles include request-level ambient reads from
+  controller and before-action work.
+- Render-site frames inside the same page remain public when their render
+  subtree has no direct ambient identity dependency.
+- Public render-site shared stream names remain available inside a
+  session-backed controller page.
+- Render-site identity profiles include ambient reads from descendant member
+  partials.
+- Render sites with identity-bound member subtrees do not register public
+  shared stream names.
+- Hidden controller-state cases now stay explicit in reports instead of
+  silently widening. Herb fallback analysis reports named page fallback reasons
+  such as `preloaded_plain_data`, `helper_hidden_collection`,
+  `multi_root_partial`, and `page_dependency_without_narrower_frame`, and the
+  developer report attaches refactor guidance.
+
+Remaining follow-up: none for this phase.
+
+## Phase 3: Controller Relation Provenance
+
+Goal: support controller-built relations without treating every query
+materialization as a render dependency.
+
+Work:
+
+- Track relation provenance when a controller materializes an Active Record
+  relation under observation.
+- Attach collection dependencies when a relation or its materialized result is
+  rendered, not simply because any controller query ran.
+- For page replay, allow page-level dependencies from controller queries whose
+  relation shape is structural.
+- Raise/refuse opaque controller relation shape in strict mode instead of
+  silently ignoring it or widening to a broad table dependency.
+- Remove broad `Relation#exec_queries` dependency capture once rendered
+  relation provenance covers the needed cases.
+
+Exit criteria:
+
+- A controller-built `@stories = Story.where(status: "active")` collection
+  invalidates through the rendered collection boundary.
+- An auth lookup such as `User.find_by(session_token: session[:u])` does not
+  become an unrelated rendered collection dependency.
+- Opaque controller SQL gets a guided refused-boundary reason.
+
+Status:
+
+- Opaque relation materialization under observation no longer disappears
+  silently. Strict mode raises before the query executes; warning mode records
+  a refused boundary with `opaque_active_record_relation` and skips the
+  collection dependency instead of widening to a broad table dependency.
+- Controller materialized relations now retain relation provenance on the
+  returned collection. When that materialized result is rendered as a Rails
+  collection, the render-site receives the Active Record collection dependency,
+  the replay snapshot stays relation-shaped, and `Relation#exec_queries` no
+  longer registers a broad collection dependency on its own.
+- Structural `Relation#pluck` calls now register `active_record_query`
+  dependencies instead of `active_record_collection` dependencies. They can
+  select a page replay when scalar query output changes, but they are not
+  eligible for collection append/remove/prepend planning. Opaque pluck
+  expressions raise in strict mode and refuse the boundary in warning mode.
+- Hidden controller relation materialization is not live by itself. If the
+  materialized result is rendered as a collection, relation provenance attaches
+  the collection dependency to that render-site. If scalar query output is
+  rendered, use `pluck`/query dependencies. If record attributes affect output,
+  those attribute reads are the lifecycle dependency. Otherwise the hidden
+  query is intentionally not a lifecycle invalidation surface.
+
+Remaining follow-up: none for this phase.
+
+## Phase 4: Replay Privacy And Size Gates
+
+Goal: make replay inputs inspectable without retaining unrelated secrets.
+
+Work:
+
+- Keep raw replay values only where replay requires them.
+- Redact or fingerprint values in debug/report surfaces.
+- Add recipe-size assertions for controller page capture.
+- Add a no-secret regression test using session keys that look like OAuth,
+  TOTP, CSRF, and redirect state but are never read by the rendered request.
+
+Exit criteria:
+
+- Controller recipes stay bounded when the Rack session contains unrelated
+  keys.
+- Reports show source/key/class evidence without exposing raw private values.
+- Security-sensitive unread session keys are absent from serialized recipes.
+
+Status:
+
+- Graph reports no longer embed full replay recipes. They report recipe kind,
+  target, replay keys, replay byte size, and replay digest so diagnostics can
+  reason about cost without exposing raw replay values.
+- Controller page recipe tests now assert that unread security-shaped session
+  keys such as OAuth state, TOTP, CSRF, and redirect state are absent from
+  serialized replay payloads.
+- Controller page replay payload size is asserted against observed inputs so
+  unrelated large session values cannot silently bloat recipes.
+- Unread cookie values and unrelated request headers now have explicit
+  no-secret/size tests. Observed request headers needed for replay, such as
+  `request.user_agent`, are copied through a small allowlist; unread
+  authorization or CSRF-style headers stay out of replay payloads.
+
+Remaining follow-up: none for this phase.
+
+## Phase 5: Lobsters Acceptance Workload
+
+Goal: use Lobsters as a generic Rails stress test, not as a runtime contract.
+
+Work:
+
+- Capture and replay representative anonymous and logged-in Lobsters pages.
+- Cover session-backed user identity, cookie-backed tag filters, controller
+  ivars, and request-shaped pages.
+- Prove public regions remain shareable where the render subtree is public.
+- Prove identity-bound regions partition when user or cookie state changes.
+- Add benchmark counters for ambient replay input count, recipe bytes, refused
+  boundaries, live deopts, render groups, and render counts.
+
+Exit criteria:
+
+- Lobsters acceptance tests pass without runtime special cases.
+- Recipe sizes and render grouping evidence are reported.
+- Refused boundaries are actionable and not hidden as benchmark failures.
+
+Status:
+
+- Lobsters surface tests pass for anonymous index, logged-in story discussion,
+  comment creation, and story voting without runtime special cases.
+- Current Upkeep subscription acceptance tests replaced the stale
+  context-token/relay-region probes. They now assert current subscription
+  markers, refused-boundary absence, graph/report counters, replay byte bounds,
+  cookie-backed tag-filter replay, and logged-in stream partitioning.
+- Hot-path Lobsters queries used by those pages were refactored from opaque
+  SQL strings to structural Active Record/Arel predicates or orders.
+- Non-GET controller actions no longer run subscription observation. Mutation
+  requests still capture lifecycle change events and deliver, but opaque
+  mutation-only queries do not participate in render dependency capture.
+- Benchmark metrics now expose an `upkeep_reactivity` summary outside the
+  integration suite. It reports stored subscription graph counts, frame and
+  dependency counts, replay recipe byte totals, ambient replay input counts,
+  refused-boundary reasons, live deoptimization reasons, render groups, and
+  render counts. The benchmark markdown reports surface the same counters.
+
+Evidence:
+
+- `bin/rails test test/integration` in `benchmark/lobsters-upkeep`: 6 runs, 74
+  assertions, 0 failures, 0 errors.
+- `ruby -Itest test/benchmark/layout_test.rb` covers the benchmark metrics
+  summary shape and redaction.
+
+Remaining follow-up: none for this phase.
+
+## Phase 6: Documentation And Cleanup
+
+Goal: make the supported Rails idioms clear to app authors.
+
+Work:
+
+- Update getting-started and identity docs with observed ambient input examples.
+- Document refused-boundary examples for hidden controller state and opaque
+  controller relation shape.
+- Remove transitional wording and code paths once phases land.
+
+Exit criteria:
+
+- Docs explain what stays live, what deopts, and what refuses.
+- No roadmap item depends on maintaining both whole-session and observed-input
+  replay paths.
+
+Status:
+
+- Getting started now documents observed session/cookie/request inputs,
+  materialized relation provenance, scalar `pluck` query dependencies, and the
+  GET-only subscription capture boundary.
+- Identity and sharing docs now explain replay privacy, request-header
+  allowlisting, opaque relation/pluck refused boundaries, and hidden controller
+  state refactor options.
+- The roadmap no longer depends on a whole-session replay path or maintaining
+  both broad relation materialization and rendered relation provenance paths.
+
+Remaining follow-up: none.

data/docs/architecture/herb-roadmap.md

@@ -0,0 +1,324 @@
+# Herb Roadmap
+
+Upkeep should use Herb as the compile-time structural layer for Rails views, not
+as the first source of runtime truth. Herb answers where updates can land. Rails
+runtime observation answers which data and identity surfaces a subscriber
+actually read.
+
+This roadmap is deliberately phased. Each phase lands only after its gate
+passes, and each new path replaces the previous implementation instead of
+leaving duplicate code paths behind.
+
+## Current Status
+
+- Phase 1 landed: `TemplateManifest` is the single Herb coverage map, and the
+  Herb surface probe reports from it.
+- Runtime alignment landed: graph frames and selected targets are checked
+  against manifest entries, with concrete deopt reasons.
+- Fallback taxonomy landed: page fallbacks are explained as manifest mismatch,
+  missing render site, helper-hidden collection, multi-root partial, preloaded
+  plain data, or page dependency without a narrower frame.
+- Phase 2 landed in the proof renderer: `SourceInstrumenter` owns render-site
+  wrapping and fragment-root marker insertion, and post-render partial root
+  tagging has been retired.
+- Phase 3 landed in the proof renderer: page, fragment, and render-site frames
+  carry manifest path/fingerprint provenance, and alignment reports stale or
+  missing provenance as explicit deopt reasons.
+- Phase 4 landed as a classifier: `ManifestDiff` uses `Herb.diff` to separate
+  no-op edits, content-only topology refreshes, topology rebuilds, and unsafe
+  full rebuilds.
+- Phase 5 landed in the proof renderer: replay recipes carry the manifest
+  path/fingerprint for the frame or render site they replay.
+- Phase 6 landed as a report: `DeveloperReport` turns manifest coverage and
+  proof fallback reasons into actionable template guidance.
+- Production rollout is next: move Herb from proof renderer to real ActionView,
+  use diff classification for manifest cache updates, use manifest addresses in
+  replay, add performance gates, then make Herb the default and only
+  instrumentation path if the gates pass.
+- Production Phase A landed: real ActionView ERB templates are instrumented
+  through `SourceInstrumenter`, Rails frame payloads carry manifest provenance,
+  and collection render-site IDs come from Herb source offsets.
+- Production Phase B landed: `ManifestCache` stores manifests by template path
+  and source digest, classifying changed source with `ManifestDiff`.
+- Production Phase C landed: replay recipes render matching manifest-backed
+  targets directly, with DOM extraction kept only as a mismatch fallback.
+- Production Phase D landed as an internal gate: `PerformanceGate` records
+  capture time, allocations, replay time, replay SQL count, graph size, payload
+  bytes, and direct manifest replay coverage.
+- Default Herb cutover landed: Herb is a runtime dependency, runtime Herb files
+  ship with the gem, and the proof-only local `herb_loader` path was removed.
+
+## Baseline
+
+Before making Herb critical, keep measuring the current runtime:
+
+- selected target distribution: fragment, render site, page
+- page fallback reasons
+- subscription capture time and allocations
+- replay time and SQL count
+- Turbo Stream payload bytes
+- graph size: nodes, edges, dependencies
+- DOM patch correctness against a fresh full render
+
+These numbers are the comparison point for every phase.
+
+## Phase 1: Template Manifest
+
+Hypothesis: Herb can statically describe the useful update addresses in Rails
+templates.
+
+Implementation:
+
+- Parse each supported template with Herb.
+- Emit a reusable template manifest with parse status, root shape, render sites,
+  helper-lowered elements, and frontend tag targets.
+- Keep the existing Herb surface probe as a report over the manifest, not a
+  second visitor implementation.
+
+Gate:
+
+- Strict parse failures stay at zero for maintained benchmark templates.
+- Herb discovers every render site that the current proof depends on.
+- Single-root partial detection stays deterministic.
+- The probe report remains compatible enough to compare against existing
+  `results/herb_surface.json`.
+
+Worth proven when the manifest gives a reliable coverage map and clear reasons
+for narrow-update eligibility or fallback.
+
+## Phase 2: Compile-Time Instrumentation
+
+Hypothesis: Herb can insert stable frame and render-site markers without
+changing Rails rendering semantics.
+
+Implementation:
+
+- Use the manifest to add `data-upkeep-frame` to fragment roots.
+- Wrap render sites with `upkeep-render-site` markers.
+- Run only in a test or shadow path until output equivalence is proven.
+
+Gate:
+
+- Rendered HTML differs only by expected Upkeep markers.
+- Frame IDs are stable across repeated renders.
+- There are no missing or duplicate frame IDs.
+- ActionView helpers, locals, layouts, and partial rendering continue to behave
+  normally.
+
+Worth proven when DOM tagging can be deterministic and source-derived instead
+of inferred after rendering.
+
+## Phase 3: Runtime Attachment
+
+Hypothesis: runtime observation can attach reads to manifest-defined frame slots
+more cheaply and precisely than rediscovering structure at render time.
+
+Implementation:
+
+- Attach manifest path and fingerprint provenance to runtime frames.
+- Run the current recorder and manifest-backed recorder side by side.
+- Deliver using the current recorder until divergence is understood.
+- Deopt to page frame when runtime shape does not match the manifest.
+
+Gate:
+
+- Selected targets match current behavior, except where manifest mode is
+  strictly narrower and the DOM patch proof passes.
+- Identity signatures match current behavior.
+- Mismatches have concrete deopt reasons.
+- Runtime capture cost and graph size do not regress.
+
+Worth proven when correctness holds and runtime structure discovery shrinks.
+
+## Phase 4: Diff-Driven Manifest Updates
+
+Hypothesis: `Herb.diff` can classify template edits well enough to avoid broad
+manifest invalidation and explain topology changes.
+
+Implementation:
+
+- Diff old and new template source.
+- Keep frame topology when edits only touch text or attributes.
+- Rebuild affected manifest entries when ERB, roots, render sites, or
+  containment change.
+- Fall back to a full template rebuild when the diff is ambiguous.
+
+Gate:
+
+- No stale manifest after template edits.
+- No missed boundary-changing edits.
+- Diff classifications explain the rebuild scope.
+
+Worth proven when template edits become cheap and understandable in development
+and CI.
+
+## Phase 5: Manifest Replay
+
+Hypothesis: source-derived render-site IDs and manifest recipes make target
+replay cheaper and less fragile.
+
+Implementation:
+
+- Reference manifest entries from replay recipes.
+- Replay fragments, render sites, and pages against stable source-derived
+  addresses.
+- Keep full-render extraction as the correctness oracle while this is in
+  shadow mode.
+
+Gate:
+
+- Replayed target HTML equals the matching target from a fresh full render.
+- Replay time, SQL count, and payload size are lower or unchanged.
+- Subscriber identity partitioning still prevents payload leakage.
+
+Worth proven when narrow replay becomes stable enough to reduce full-page
+replay dependence.
+
+## Phase 6: Developer Report
+
+Hypothesis: Herb can make Upkeep adoption easier by explaining view coverage and
+fallbacks.
+
+Implementation:
+
+- Emit a report per app showing narrow-update eligibility, blocked templates,
+  render-site coverage, helper-hidden regions, and suggested fixes.
+- Track before/after coverage as templates are adjusted.
+
+Gate:
+
+- The report identifies actionable template changes.
+- Applying those changes increases fragment or render-site coverage.
+- Fewer proof cases fall back to page frames.
+
+Worth proven when Rails developers can improve Upkeep behavior without knowing
+the internal graph model.
+
+## Production Phase A: ActionView Integration
+
+Hypothesis: real Rails templates can use the same Herb source instrumentation
+as the proof renderer without changing ActionView semantics.
+
+Implementation:
+
+- Build manifests from Rails-resolved template source.
+- Apply `SourceInstrumenter` before ActionView compiles supported ERB
+  templates.
+- Attach manifest path and fingerprint to page, fragment, and render-site
+  frames in the Rails capture path.
+- Remove any ActionView-only duplicate marker path once the Herb path passes.
+
+Gate:
+
+- Existing ActionView capture tests still pass.
+- Rendered Rails HTML has exactly the expected Upkeep markers.
+- Target selection and replay correctness match the current Rails behavior.
+- Helpers, locals, controller requests, collection rendering, and layouts keep
+  working.
+
+Worth proven when the real gem runtime uses Herb for template structure, not
+just the proof renderer.
+
+## Production Phase B: Manifest Cache Updates
+
+Hypothesis: `ManifestDiff` can drive an actual manifest cache update strategy
+without stale topology.
+
+Implementation:
+
+- Add a manifest cache keyed by virtual template path and source digest.
+- Use `ManifestDiff` when a cached source changes.
+- Reuse stable topology for content-only changes.
+- Rebuild manifest entries for root, render-site, ERB, or parse-affecting
+  changes.
+- Emit the diff classification in developer/debug reports.
+
+Gate:
+
+- Template edit tests prove no stale manifest after content-only and
+  topology-changing edits.
+- Boundary-changing edits are never classified as safe topology reuse.
+- Cache invalidation explains every rebuild or fallback.
+
+Worth proven when development reloads and repeated captures avoid broad
+manifest rebuilding while staying correct.
+
+## Production Phase C: Manifest Replay Optimization
+
+Hypothesis: replay can use manifest addresses to avoid avoidable full-document
+work while keeping full-render extraction as the oracle during the transition.
+
+Implementation:
+
+- Reference manifest entries from Rails replay recipes.
+- Prefer manifest-known fragment and render-site addresses when extracting
+  replay targets.
+- Keep full-render extraction as a comparison oracle until every replay target
+  matches.
+- Remove any duplicate replay path after the manifest path matches.
+
+Gate:
+
+- Fragment, render-site, and page replay output still equals fresh full render
+  target HTML.
+- Replay SQL count, replay time, allocations, or extraction cost improves or
+  remains neutral.
+- Identity partitioning and authorization surfaces remain unchanged.
+
+Worth proven when manifest replay is cheaper or more explainable without
+weakening correctness.
+
+## Production Phase D: Performance Gates
+
+Hypothesis: Herb's source-derived structure reduces runtime work enough to
+justify making it the default.
+
+Implementation:
+
+- Add benchmark measurements for capture time, allocations, replay time, SQL
+  count, graph size, and payload bytes.
+- Compare the pre-Herb runtime path against the Herb path while both still
+  exist.
+- Report neutral, improved, and regressed metrics explicitly.
+
+Gate:
+
+- Correctness proofs pass.
+- Capture, replay, graph, SQL, and payload metrics do not regress beyond an
+  intentional and documented threshold.
+- Any regression has an explanation and a follow-up before Herb becomes the
+  only path.
+
+Worth proven when Herb is not just cleaner, but also measurably safe to enable
+by default.
+
+## Default Herb Cutover
+
+Hypothesis: after production phases A-D pass, Upkeep should have one template
+structure path: Herb.
+
+Implementation:
+
+- Make Herb instrumentation the default behavior.
+- Remove stale non-Herb marker, render-site, replay, and report paths rather
+  than keeping double paths.
+- Keep explicit deopts for unsupported templates instead of silently falling
+  back to an untracked structural model.
+
+Gate:
+
+- Full test suite, ActionView capture tests, end-to-end proof, identity proof,
+  auth proof, manifest diff tests, developer report tests, and performance
+  gates pass.
+- No duplicate source-instrumentation or post-render marker path remains.
+- The gem packaging decision is explicit: Herb support either ships as the
+  runtime path or is cleanly excluded with the feature disabled.
+
+Worth proven when Herb becomes Upkeep's single structural source of truth.
+
+## Cleanup Rule
+
+Each phase must retire or fold the previous experimental path into the new
+abstraction before it is committed. Temporary shadow paths are acceptable only
+inside the phase branch and must either become the main path or be removed
+before the phase commit.