ruby_reactor 0.8.7 → 0.8.8

data/specs/010-rollback-follow-ups/plan.md

@@ -0,0 +1,273 @@
+# Implementation Plan: Rollback and Resume Follow-ups
+
+**Branch**: `rollback_follow_ups` | **Date**: 2026-10-08 | **Spec**: [spec.md](spec.md)
+
+**Input**: Feature specification from `specs/010-rollback-follow-ups/spec.md`
+
+## Summary
+
+Seven follow-ups from 008, resolved with three shared mechanisms.
+
+1. **Every live run holds its liveness lock, and readers lock, then load** (US1, US2):
+   - A synchronous `Executor#execute` takes the run's `async:<id>` lock, auto-extended, and saves
+     before releasing it (R-01). The sweeper already skips locked runs, so a live caller-process run
+     is never re-enqueued. A killed one lapses within `context_lock_ttl`.
+   - `Worker#perform` takes the same lock **before** reading the run, waiting up to 2s, and hands
+     the owner to the executor, which re-enters the lock (R-02, R-03). A Worker resumed by a fan-out
+     map's completion can no longer race the caller's final save, in either order.
+2. **A resume is claimed per interrupt, and applied only by the run's lock owner** (US3, US4, US5):
+   - `continue` validates the payload in the calling process, then claims the interrupt with
+     `SET NX` (R-04). The claim holds the payload.
+   - It then tries to own the run: lock, reload, resume (R-05). Any path that cannot run now (the
+     run is busy, the reactor's lock or semaphore is held, a background resume) becomes one hand-off.
+     The Worker applies the claims under the lock, and `continue` returns a `DispatchResult`.
+   - The claim makes "exactly one resume per interrupt" hold in every mode, including inline
+     testing. Claiming per interrupt lets different interrupts be accepted while another resume
+     executes.
+   - Supporting changes:
+     - after admission, the snooze limit no longer escalates a run to `failed` without rollback
+       (R-07);
+     - interrupt attempts move to an atomic counter (R-08);
+     - `Reactor#undo` reloads under its lock (R-09).
+3. **Rollback state records what was cut off** (US6):
+   - An `aborted` run records its failing step's unfinished `compensate` (step, arguments, error) in
+     009's `context.rollback` (R-10).
+   - `Executor#undo_all` runs it before the undo stack. Manual undo, composed children and inline
+     elements all go through `undo_all`, so every depth is covered.
+   - The dashboard shows the outstanding `compensate` (R-11).
+
+Plus one new DSL method, **`map ... undo_all { |completed_results| }`** (US7). When it is declared,
+the map's rollback calls the block once with a lazy, index-ordered enumerable of completed
+results, instead of replaying each element (R-12, R-13):
+
+- a fan-out map reads its forward result slots and dispatches no rollback jobs;
+- an inline map reads element contexts one at a time, and still replays an `aborted` element
+  itself.
+
+## Technical Context
+
+**Language/Version**: Ruby >= 3.0
+
+**Primary Dependencies**:
+
+- Sidekiq and ActiveJob adapters, Redis, dry-validation.
+- Optional OpenTelemetry middleware.
+- GUI: React/Vite (`gui/`, built into `lib/ruby_reactor/web/public`).
+
+No new dependencies.
+
+**Storage**: Redis through `Storage::RedisAdapter` (see [data-model.md](data-model.md)).
+
+- New keys:
+  - `reactor:<Class>:context:<id>:resume:<step>`: the claim (`SET NX`, TTL `context_ttl`);
+  - `reactor:<Class>:context:<id>:resume_attempts:<step>`: the attempt counter (`INCR`).
+- `lock:async:<id>` gains three holders: `execute`, `Worker`, `continue`.
+- The `context.rollback` hash gains `arguments` and `error` for aborted runs.
+- The map `StepConfig` gains `undo_all_block`.
+- No migration. The one compatibility note: attempt counts restart for runs paused across the
+  upgrade (R-08).
+
+**Testing**: RSpec against real Redis (`redis://localhost:6780`), with Sidekiq fake mode plus
+`drain_async_jobs`.
+
+- **Concurrency specs** (US1, US4, US5): threads, a barrier or latch, real Redis.
+- **Interleaving spec** (US2): a public middleware hook drains jobs between the hand-off and the
+  caller's final save.
+- **Crash spec** (US1, FR-003): `fork` plus `SIGKILL`, tagged `:fork`.
+- **Inline mode**: one justified `Sidekiq::Testing.inline!` spec for US4 (FR-016).
+- **Scale**: `:slow` tag, 10,000 elements for `undo_all` (SC-007).
+- **Demo specs**: the shipped matchers only, plus the new `be_resume_deferred` and
+  `have_run_undo_all`.
+
+**Target Platform**: Ruby gem (MRI), Sidekiq/ActiveJob workers, Rails demo app in Docker.
+
+**Project Type**: Library (gem) with a bundled dashboard.
+
+**Performance Goals**:
+
+- **Synchronous run**: one lock acquire and release, and one extender thread, per run.
+- **Worker**: one extra lock round-trip, and usually no wait. It waits at most 2s, while a caller
+  finishes its hand-off.
+- **`continue`**: one `SET NX`, plus one `MGET` per resume for reactors that declare interrupts.
+- **`undo_all` over 10,000 elements**: one call. Memory is bounded by the enumerator's chunk
+  (1,000 result slots, or one element context).
+
+**Constraints**:
+
+- **Single writer**: `continue` writes the run's context only while holding its lock (J-5).
+- **Save before release** for every holder (J-3).
+- A claimed payload is never validated again (J-7).
+- An admitted run is never escalated to `failed` by snoozes (J-8).
+- Inline testing mode takes no context lock, as today (R-14).
+- Existing outcomes are unchanged for:
+  - uncontended resumes;
+  - background resumes;
+  - maps without `undo_all`;
+  - aborted runs whose `compensate` had finished.
+
+**Scale/Scope**:
+
+- About 14 library files touched. 2 new: `interrupt_claims.rb` and `error/recorded_failure.rb`.
+- About 10 new spec files and about 8 updated.
+- Demo: 4 reactor files, 3 rake tasks plus the `demo:rollback_follow_ups` group, 3 specs.
+- 6 documentation files, the README, CHANGELOG and `future_improvements.md`.
+- GUI: `ReactorDetail.tsx` (its API response is untyped, so `lib/reactors.ts` needs no change).
+
+## Constitution Check
+
+*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
+
+| Principle | Status | Note |
+| --- | --- | --- |
+| I. Gem-First Design | PASS | Everything is in `lib/` behind the existing DSL and `Reactor` API. One DSL method (`undo_all`). The Worker change is in the backend-agnostic `Worker` module, so Sidekiq and ActiveJob both get it. No host coupling. |
+| II. Saga Pattern Integrity | PASS (strengthened) | Closes four ways a run lost work or effects: a duplicate forward run (US1), overwritten progress (US2), a lost or double resume (US3–US5), and a `compensate` never finished after an interruption (US6). R-07 removes the snooze escalation that marked runs `failed` without rollback. `undo_all` keeps step-level order (P-6), and an `aborted` inline element still replays its own undos. |
+| III. Test-First with Real Infrastructure | PASS with one justified deviation | Each story's specs are written first and fail first, against real Redis. Concurrency is shown with threads, not mocks. The US2 interleaving uses a public middleware hook, not stubs. One new `inline!` spec, for FR-016, justified in Complexity Tracking. |
+| IV. Observability by Default | PASS | New key=value log lines: `resume.deferred`, `resume.waiting`, `map.rollback.undo_all.started` / `.completed`. A `:undo_all` trace entry. Dashboard: `pending_compensation` on aborted runs, and a "Resume accepted" API message. A re-run `compensate` fires the standard compensation middleware events. A hand-off never fires `:failed_reactor`. |
+| V. Simplicity & SemVer | PASS with notes | Three mechanisms, each serving several stories (Complexity Tracking). Code deleted: `reopen_paused` and the `past_gates?` contention branch in `continue`, and the blob write of attempts. MINOR (R-16), with CHANGELOG migration notes: `continue` returns a hand-off where it raised; "already resumed"; attempt counts restart; a synchronous run holds the liveness lock. No new config knobs. |
+| VI. Demo-App Proof of Feature | PASS (planned) | One reactor per file, all class-based steps (R-18): `BulkRefundDemoReactor` with `BulkRefundChargeReactor` (`undo_all`, fan-out, a later failure), `ContendedApprovalDemoReactor` (resume while locked), and `DualApprovalDemoReactor` (two interrupts at once). Rake: `demo:map_undo_all`, `demo:contended_resume`, `demo:concurrent_interrupts`, grouped as `demo:rollback_follow_ups` in `demo:all`. Specs use the shipped matchers. `TestSubject#resume(process_jobs:)`, `be_resume_deferred` and `have_run_undo_all` are added to `lib/ruby_reactor/rspec/`. No new Docker service or environment variable. |
+
+- [x] Documentation impact identified (research R-17):
+  - **README.md**: "Durability & Recovery", "Interrupts (Pause & Resume)", "Locks, Semaphores &
+    Ordered Locks" and "Map & Parallel Execution".
+  - **documentation/**: `interrupts.md`, `locks_and_semaphores.md`, `background_and_async.md`,
+    `data_pipelines.md`, `core_concepts.md`, `testing.md`.
+  - **demo_app/documentation/data_pipelines.md**.
+  - **CHANGELOG.md**.
+  - **specs/future_improvements.md**: remove the seven items, and update "Fenced context writes".
+  - Carried into tasks.md as a required task per user story.
+
+**Post-design re-check (after Phase 1)**: PASS.
+
+- New public surface:
+  - one DSL method;
+  - one changed return contract for `continue`, and one new rejection;
+  - two log events, plus the `undo_all` pair;
+  - one API field;
+  - three RSpec additions.
+- New internals:
+  - one small module (`InterruptClaims`) used by three callers;
+  - one error class;
+  - three storage methods.
+- The spec was revised for what the design fixed (see spec Status): FR-010, FR-017, FR-024, US6-AS3.
+
+## Project Structure
+
+### Documentation (this feature)
+
+```text
+specs/010-rollback-follow-ups/
+├── plan.md                        # This file
+├── research.md                    # Phase 0: R-01..R-19
+├── data-model.md                  # Phase 1: lock holders, claim, counter, rollback record, undo_all
+├── quickstart.md                  # Phase 1: validation guide
+├── contracts/
+│   ├── api-surface.md             # continue outcomes, undo on aborted, undo_all, logs, RSpec additions
+│   └── resume-protocol.md         # Invariants J-1..J-9, sequences P-1..P-6
+├── checklists/requirements.md
+└── tasks.md                       # Phase 2 (/speckit-tasks)
+```
+
+### Source Code (repository root)
+
+```text
+lib/ruby_reactor/
+├── interrupt_claims.rb                 # new: claim!(ctx, step, payload) (SET NX), unapplied(ctx) (MGET of
+│                                       #   interrupts without a result), apply!(ctx) → set_result (R-04, R-06)
+├── error/recorded_failure.rb           # new: StandardError carrying the original message + original_class (R-10)
+├── executor.rb                         # execute: acquire_context_lock (root, caller process); save before
+│                                       #   releasing it (R-01). context_lock_owner= for re-entry (R-03).
+│                                       #   resume_execution: InterruptClaims.apply! after the lock (R-04).
+│                                       #   mark_aborted: record the unfinished compensate in `rollback` (R-10).
+│                                       #   undo_all: compensate_pending! before the stack (R-10)
+├── executor/compensation_manager.rb    # @pending keeps rollback_arguments (R-10)
+├── reactor.rb                          # continue: validate → claim → lock → reload → resume, or hand off;
+│                                       #   delete reopen_paused (R-05). validate_continue_payload: attempt
+│                                       #   counter (R-08). undo: reload under lock, merge the aborted record,
+│                                       #   failure: kwarg (R-09, R-10)
+├── worker.rb                           # perform: lock, then load (R-02); status table (R-06); release in
+│                                       #   ensure; handle_snooze: uncapped once admitted, one
+│                                       #   resume.waiting warning (R-07)
+├── storage/redis_adapter.rb            # claim_interrupt_resume, retrieve_interrupt_resumes,
+│                                       #   increment_interrupt_attempts
+├── dsl/map_builder.rb                  # undo_all(&block): once, block required → undo_all_block (R-12)
+├── step/map_step.rb                    # compensate/undo: bulk_rollback when undo_all is declared (R-13)
+├── map/step_rollback.rb                # bulk_rollback: fan-out result slots / inline two passes; trace,
+│                                       #   logs, failure entry (R-13)
+├── web/api.rb                          # pending_compensation on aborted; continue → "Resume accepted" (R-11)
+└── rspec/{test_subject,matchers}.rb    # resume(process_jobs:), running runs; be_resume_deferred;
+                                        #   have_run_undo_all (R-18)
+
+gui/src/components/ReactorDetail.tsx    # outstanding-compensate line on aborted runs
+gui/src/lib/reactors.ts                 # pending_compensation type
+lib/ruby_reactor/web/public/            # rebuilt bundle
+
+spec/
+├── ruby_reactor/caller_process_liveness_spec.rb        # new: US1 (latch, long step, fork + SIGKILL)
+├── ruby_reactor/executor/caller_save_race_spec.rb      # new: US2 interleaving (middleware drain)
+├── ruby_reactor/worker_lock_then_load_spec.rb          # new: R-02, R-06 status table
+├── ruby_reactor/worker_snooze_admitted_spec.rb         # new: R-07
+├── ruby_reactor/interrupts/contended_resume_spec.rb    # new: US3
+├── ruby_reactor/interrupts/resume_claim_spec.rb        # new: US4, including the one inline! example
+├── ruby_reactor/interrupts/concurrent_interrupts_spec.rb  # new: US5
+├── ruby_reactor/dsl/map_undo_all_dsl_spec.rb           # new: R-12
+├── map/map_undo_all_spec.rb                            # new: US7 (fan-out, inline, atomic, undo, :slow)
+├── ruby_reactor/rollback/aborted_execution_spec.rb     # updated: US6 at top level, compose, inline element
+├── ruby_reactor/rollback/resume_guard_spec.rb          # updated: contention defers instead of raising
+├── ruby_reactor/sweeper_spec.rb                        # updated: caller-process runs skipped
+├── ruby_reactor/context_lock_spec.rb                   # updated: execute holds the lock; re-entry by owner
+├── ruby_reactor/interrupt_background_resume_spec.rb    # updated: claim-based background resume
+├── integration/interrupt_max_attempts_spec.rb          # updated: counter record; failure under lock
+├── ruby_reactor/web/*_spec.rb                          # updated: pending_compensation, continue message
+└── ruby_reactor/rspec/*_spec.rb                        # new or updated: resume(process_jobs:), new matchers
+
+demo_app/
+├── app/reactors/bulk_refund_demo_reactor.rb            # fan-out map with undo_all; a later step fails
+├── app/reactors/bulk_refund_charge_reactor.rb          # element reactor (class-based charge step)
+├── app/reactors/contended_approval_demo_reactor.rb     # with_lock + interrupt
+├── app/reactors/dual_approval_demo_reactor.rb          # finance (resume: :background) + legal interrupts
+├── lib/tasks/demo_reactors.rake  # demo:map_undo_all, :contended_resume, :concurrent_interrupts,
+│                                 #   :rollback_follow_ups (in demo:all)
+└── spec/reactors/{bulk_refund,contended_approval,dual_approval}_demo_reactor_spec.rb
+
+README.md, documentation/*.md, demo_app/documentation/data_pipelines.md, CHANGELOG.md,
+specs/future_improvements.md   # per R-17
+```
+
+**Structure Decision**: single gem project, existing `lib/ruby_reactor` layout. New interrupt specs
+go in a new `spec/ruby_reactor/interrupts/` folder. Rollback specs stay in
+`spec/ruby_reactor/rollback/`, and map specs in `spec/map/`.
+
+## Delivery Order
+
+The stories are independently testable. tasks.md is the authoritative order.
+
+1. **Foundation** (blocks US2, US3 and US5):
+   - `Executor#context_lock_owner=` (R-03);
+   - `Worker#perform` lock, then load, plus the status table (R-02, R-06);
+   - `Reactor#undo` reload under its lock (R-09).
+2. **US1 (P1)**: `execute` takes and releases the liveness lock, saving first (R-01). Specs: latch,
+   long step, fork kill.
+3. **US2 (P1)**: the interleaving spec, which passes with foundation plus US1. The inline-`continue`
+   variant lands with US3.
+4. **Claims foundation** (blocks US3–US5): storage methods, `InterruptClaims`, and `apply!` in
+   `resume_execution` (R-04).
+5. **US3 (P1)**: the `continue` flow with contention hand-off (R-05), the uncapped snooze after
+   admission (R-07), and the `resume_guard_spec` update.
+6. **US4 (P2)**: claim race specs, including the one `inline!` example.
+7. **US5 (P2)**: accept on `running` (R-05 step 1), the attempt counter (R-08), and the concurrency
+   specs.
+8. **US6 (P2)**: `@pending` arguments, the `mark_aborted` record, `undo_all`'s
+   `compensate_pending!`, the `Reactor#undo` merge (R-10), and the API and GUI (R-11).
+9. **US7 (P3)**: the DSL (R-12), `bulk_rollback` (R-13), and the matchers.
+10. **Demo, docs, CHANGELOG** per story. Close with the Docker acceptance run of
+    `demo:rollback_follow_ups`.
+
+## Complexity Tracking
+
+| Addition | Why needed | Simpler alternative rejected because |
+| --- | --- | --- |
+| Resume claim record per interrupt (`SET NX`) | FR-015 and FR-016: exactly one resume per interrupt **in every mode**. FR-017: different interrupts accepted concurrently. Also stores the payload until the lock owner applies it, so `continue` never writes the blob unlocked (J-5). | **The context lock** is skipped in inline mode. **A status compare-and-set in the blob** serializes all resumes of the run, which blocks US5, and still writes from outside the owning execution. |
+| The Worker takes the liveness lock before loading (wait ≤ 2s) | FR-005: closes the caller-save race in both orders. It also fixes the Workers enqueued by the rollback hand-off handshake and by `Reactor#undo` while holding the lock. | **Peek-then-snooze** leaves a check-then-read gap and adds a full Sidekiq poll (about 5s) to every hand-off from a caller process. **Full fenced writes** are out of scope (spec Assumptions). |
+| Uncapped snooze once admitted (R-07) | FR-010. Escalation marks a run with completed work `failed` **without rollback**, a Constitution II violation that also hits `resume: :background` today. | **Escalation with rollback** fails a run the user accepted, because of an unrelated holder. **Keeping the cap** strands an accepted resume. A one-time warning keeps leaked semaphores visible. |
+| One new `Sidekiq::Testing.inline!` spec, a deviation from Constitution III | FR-016 requires US4 to hold in inline job-testing mode, the mode that skips the run lock. That is the original 008 gap. | **Fake mode plus drain** cannot show it: the context lock is taken there, so the race under test never opens. Scoped to one example. |
+| Attempt counter moved to its own key (R-08) | Once `continue` accepts resumes of a `running` run (US5), the old blob save of the attempt count, from an unlocked snapshot, would overwrite the executing resume's progress. | **Counting under the run's lock** would make an invalid-payload response wait behind, or contend with, a live resume. **Not counting while running** makes `max_attempts` depend on timing. |

data/specs/010-rollback-follow-ups/pr-description.md

@@ -0,0 +1,110 @@
+# 010 Rollback and Resume Follow-ups
+
+Closes the seven "Rollback follow-ups (008)" items from `specs/future_improvements.md`.
+Spec: `specs/010-rollback-follow-ups/spec.md` · Plan: `plan.md` · Decisions: `research.md` (R-01..R-19).
+
+## What changes
+
+| Story | Fix |
+| --- | --- |
+| US1 (P1): the sweeper re-ran live caller-process runs | `Executor#execute` holds the run's `async:<id>` liveness lock, auto-extended, and saves before releasing it (R-01). The sweeper is unchanged: it already skips locked runs. A killed process is recovered once the lock lapses. |
+| US2 (P1): a caller's final save overwrote a worker's progress | With US1's lock, plus `Worker#perform` takes the run's lock **before** it reads the run (waiting up to 2s), and hands its owner to the executor, which re-enters the lock (R-02, R-03). |
+| US3 (P1): a resume contended on the reactor's lock/semaphore raised and was lost | `continue` validates the payload, claims the interrupt, then either owns the run (lock, then reload, then resume inline) or hands the resume to a worker and returns a `DispatchResult` (R-05). After admission, `lock_snooze_max_attempts` no longer escalates a run to `failed` without rollback; it warns once with `ruby_reactor.resume.waiting` (R-07). |
+| US4 (P2): two resumes of one interrupt could both run | Per-interrupt claim, `SET NX` on `reactor:<Class>:context:<id>:resume:<step>`, in every mode, including `Sidekiq::Testing.inline!` (R-04). The loser raises "already resumed" and writes nothing. |
+| US5 (P2): a resume for another interrupt was rejected while the run executed | A `running` run accepts a claim for a ready, unexecuted interrupt and hands it to a Worker. The Worker applies claims under the run's lock at resume start, and resumes a `paused` run only when it has an unapplied claim (R-06). Attempt counts move to their own `INCR` record, so `continue` never writes the run unlocked (R-08). `Reactor#undo` reloads under its lock (R-09). |
+| US6 (P2): a cut-off `compensate` of the failing step was never re-run | `mark_aborted` records the failing step, its arguments and its reason in `context.rollback`. `Executor#undo_all` re-runs it before the undo stack, so this works at any depth (composed child, inline map element). The dashboard API adds `pending_compensation`, and `ReactorDetail` shows it (R-10, R-11). |
+| US7 (P3): map-level `undo_all` | `map ... undo_all { |completed_results| }` is called once per rollback with a lazy, index-ordered Enumerable. A fan-out map reads its result slots 1,000 at a time and dispatches no rollback jobs. An inline map reads element contexts one at a time, and `aborted` elements still replay their own steps (R-12, R-13). |
+
+New RSpec surface:
+
+- `TestSubject#resume(process_jobs:)`, which also accepts `running` subjects;
+- `be_resume_deferred`;
+- `have_run_undo_all(:map).with_elements(n)`.
+
+Demo (Constitution VI):
+
+- `BulkRefundDemoReactor` with `BulkRefundChargeReactor`, `ContendedApprovalDemoReactor` and
+  `DualApprovalDemoReactor`;
+- rake `demo:map_undo_all`, `demo:contended_resume` and `demo:concurrent_interrupts`, grouped as
+  `demo:rollback_follow_ups` in `demo:all`;
+- matcher-only specs.
+
+Docs:
+
+- README: Durability, Interrupts, Locks, Map;
+- `documentation/`: `interrupts.md`, `locks_and_semaphores.md`, `background_and_async.md`,
+  `data_pipelines.md`, `core_concepts.md`, `testing.md`;
+- `demo_app/documentation/data_pipelines.md`;
+- CHANGELOG;
+- `specs/future_improvements.md`.
+
+## Migration notes (also in CHANGELOG)
+
+1. `continue` returns a `DispatchResult` where it raised `Lock::AcquisitionError` /
+   `Semaphore::AcquisitionError`; a `rescue` of those simply stops firing.
+2. A second resume of the same interrupt raises `ValidationError` "already resumed".
+3. A resume for another ready interrupt of a `running` run is accepted (`DispatchResult`).
+4. Interrupt attempt counts restart for runs paused across the upgrade.
+5. A synchronous `Reactor.run` holds the run's liveness lock; `Reactor.undo` of such a live run
+   waits, then raises `Lock::AcquisitionError`.
+
+## Verification
+
+- **Baseline (T002)**: `bundle exec rspec` on `5ca7d1d1`: 1479 examples, 0 failures, 2 pending.
+- **After (T072)**: 1570 examples, 0 failures, 2 pending (the same two pending: the 009
+  interrupt-inside-a-composed-child examples). Run on a dedicated test Redis DB
+  (`RUBY_REACTOR_TEST_REDIS_URL=redis://localhost:6780/11`).
+- **Tagged (T073)**: `--tag slow` (10,000-element `undo_all`: one call, memory flat across the
+  enumeration, about 0.16s for the call) and `--tag fork` (a `SIGKILL`ed caller process is
+  recovered): both pass.
+- **`bundle exec rubocop`**: no offenses. **GUI**: `vitest` 69 tests pass; the bundle is rebuilt
+  into `lib/ruby_reactor/web/public`.
+- **Demo app (T075)**: 152 examples, 0 failures (local, test Redis DB 5).
+- **Docker acceptance (T076)**: in an isolated compose project, `bin/rails demo:rollback_follow_ups`:
+  - `demo:map_undo_all`: bulk refund called once with 12 charges, 0 per-element refunds, run
+    `failed`. ✅
+  - `demo:contended_resume`: resume accepted (`DispatchResult`) while the lock was held, run
+    `completed` after release. ✅
+  - `demo:concurrent_interrupts`: finance accepted (background), legal accepted while running,
+    `completed` with both approvals. ✅
+
+### `demo:all`: failures that are already on `main`
+
+`demo:all` runs every demo against one Redis that is flushed once, with one Sidekiq process. On
+`main` (run the same way, in a separate compose project) it already fails:
+
+- `distributed_map_rollback` (70 charges for 40 orders);
+- `composed_fan_out`, happy path and rollback (8 shipped instead of 5);
+- `default_batch_size`;
+- one `async_step` unit wait;
+- `AsyncReactorDemoReactor(user_7)` (async wait timeout).
+
+On this branch the same set fails, minus `default_batch_size` and the `async_step` unit. Each of
+those demos passes when run alone (checked `demo:map_execution_undo` and
+`demo:async_reactor_demo` on this branch). The three new tasks wait up to 120s, so they pass
+inside `demo:all` too. The `demo:all` interference is out of scope here and worth its own issue.
+
+## Regression proofs (red before green)
+
+- **US1 (T014)**: with `Executor#execute`'s liveness lock removed, 5 of the 7 examples in
+  `caller_process_liveness_spec.rb` fail.
+- **US2 (T018)**: with the same line removed, the owner Worker started by the map's completion
+  writes the run four times while the caller is still inside `execute`. With it, the Worker
+  snoozes without reading or writing, and the run finishes once.
+- **US4 (T032)**: with the claim disabled, the 200-iteration race accepts both resumes.
+- **US6 (T046)**: with `compensate_pending!` disabled, 5 of the 9 examples fail.
+- **The inline payload (found by the demo suite)**: with the caller's payload not applied as
+  given, a string-keyed payload reached the step symbolized (`WebhookInterruptReactor` was
+  rejected). Guarded by `interrupt_claims_spec.rb` ("as given, string keys included").
+
+## Deviations from tasks.md
+
+- **Spec placement**:
+  - T015's example lives in `caller_process_liveness_spec.rb`;
+  - T046 and T047 in a new `rollback/aborted_compensate_spec.rb`.
+- **No `lib/reactors.ts` type for `pending_compensation`** (T054): the detail response is
+  untyped.
+- **`continue` on a `running` run always hands off.** It never tries the lock or runs another
+  execution's run inline (research R-05, updated).
+- **The `ruby_reactor.resume.waiting` line carries `error`** (whose message names the key), not a
+  separate `key` field (contracts updated).

data/specs/010-rollback-follow-ups/quickstart.md

@@ -0,0 +1,157 @@
+# Quickstart: Validate the Rollback and Resume Follow-ups
+
+A guide to proving each user story end to end. The behavior is specified in
+[contracts/](contracts/), and the records in [data-model.md](data-model.md).
+
+## 0. Prerequisites
+
+- Ruby >= 3.0 and `bundle install`.
+- Test Redis reachable at `redis://localhost:6780` (`docker start ruby_reactor_redis_test`, or
+  `docker run -d --name rr-test-redis -p 6780:6379 redis:7-alpine`).
+- If other suites share the test Redis, run the new specs alone first (known flake under load).
+- The concurrency specs use threads and real Redis in Sidekiq fake mode. Only the US4 spec marked
+  `inline` uses `Sidekiq::Testing.inline!` (R-14).
+
+## 1. US1: the sweeper leaves live caller-process runs alone (P1)
+
+```bash
+bundle exec rspec spec/ruby_reactor/caller_process_liveness_spec.rb
+bundle exec rspec spec/ruby_reactor/sweeper_spec.rb
+```
+
+| Scenario | Observable outcome |
+| --- | --- |
+| Synchronous run blocked in a step (latch), `Sweeper.run_once` called meanwhile | Returns 0 for that run, `be_locked` on `async:<id>`, and the step's counter is 1 after the latch opens |
+| Same, `context_lock_ttl = 2` (the lock extends at most once a second), the step blocks 4s, sweeping every 0.5s | Never re-enqueued (the auto-extender renews the lock) |
+| A forked child runs the reactor, and is `SIGKILL`ed mid-step | Once its lock lapses (`context_lock_ttl = 2`), `Sweeper.run_once` re-enqueues it once (spec tagged `:fork`) |
+| Synchronous run hands off at a fan-out map and returns | Swept as before (lock released after the final save) |
+
+## 2. US2: no lost progress from the caller's last save (P1)
+
+```bash
+bundle exec rspec spec/ruby_reactor/executor/caller_save_race_spec.rb
+bundle exec rspec spec/ruby_reactor/worker_lock_then_load_spec.rb
+```
+
+| Scenario | Observable outcome |
+| --- | --- |
+| A middleware on the map step's `:complete_step` drains jobs while the caller still holds the lock | The owner Worker waits or snoozes and loads nothing. After a second drain: `be_success`, and each step after the map ran once (`have_run_step`) |
+| Same, with the fix reverted (regression proof, run once while writing the spec) | The final save overwrites the Worker's progress, and the run is stuck `running` |
+| Worker started while another owner holds `async:<id>` for longer than 2s | Snoozes as `ContextLockContention`, nothing written; runs after release on fresh state |
+| Inline `continue` reaching a fan-out map | Same as the first row |
+
+## 3. US3: contended resume is accepted (P1)
+
+```bash
+bundle exec rspec spec/ruby_reactor/interrupts/contended_resume_spec.rb
+bundle exec rspec spec/ruby_reactor/rollback/resume_guard_spec.rb   # updated: no longer raises
+bundle exec rspec spec/ruby_reactor/worker_snooze_admitted_spec.rb
+```
+
+| Scenario | Observable outcome |
+| --- | --- |
+| `hold_lock(<reactor lock key>)`, then `continue` with a valid payload | Returns `DispatchResult`, the run is `running`, a `ruby_reactor.resume.deferred reason=lock` line is logged, and one Worker job is enqueued |
+| Release the lock, drain | `be_success`; the interrupt's result equals the payload |
+| Lock held across 25 snoozes (`lock_snooze_max_attempts = 20`) | Never `failed`; one `ruby_reactor.resume.waiting` warning; resumes after release |
+| Invalid payload while the lock is held | Validation failure returned (the class method raises `InputValidationError`); `be_paused`; no job enqueued; no claim stored |
+| A second `continue` for the same interrupt while deferred | Raises `ValidationError` "already resumed" |
+| Semaphore with no free slot | Same as the first three rows |
+| `Reactor.cancel` before draining | The Worker loads `cancelled` and does nothing |
+
+## 4. US4: exactly one of two simultaneous resumes (P2)
+
+```bash
+bundle exec rspec spec/ruby_reactor/interrupts/resume_claim_spec.rb
+```
+
+| Scenario | Observable outcome |
+| --- | --- |
+| 200 iterations: two threads released by a barrier call `continue` for the same interrupt | Exactly one returns, one raises "already resumed"; the steps after the interrupt run once per run; the stored result is the winner's payload |
+| Same, inside `Sidekiq::Testing.inline!` (the one justified inline spec) | Same |
+| `resume: :background` interrupt | Exactly one Worker job is enqueued |
+
+## 5. US5: several interrupts resumed at once (P2)
+
+```bash
+bundle exec rspec spec/ruby_reactor/interrupts/concurrent_interrupts_spec.rb
+bundle exec rspec spec/ruby_reactor/multiple_interrupts_spec.rb   # existing, unchanged outcomes
+```
+
+| Scenario | Observable outcome |
+| --- | --- |
+| Paused at A and B. Thread 1 resumes A, and a step after A blocks on a latch. Thread 2 resumes B | Thread 2 gets `DispatchResult` (`reason=run_busy`). The latch opens, the run pauses at B, then a drain lets the Worker apply B: `be_success`, both results, each applied once |
+| 2 to 5 ready interrupts all resumed from threads | All accepted; final `be_success`; each interrupt result applied once |
+| Resume of B with an invalid payload while A runs | Validation failure; no claim; B stays ready |
+| A's resume fails and rolls back after B was accepted | The Worker loads `failed` and returns; B's result never applied |
+| The run is executing its first run in a worker, and B's dependencies are complete | Accepted; the run does not stay paused at B |
+
+## 6. US6: manual undo finishes a cut-off `compensate` (P2)
+
+```bash
+bundle exec rspec spec/ruby_reactor/rollback/aborted_compensate_spec.rb
+```
+
+| Scenario | Observable outcome |
+| --- | --- |
+| Step X fails; X's `compensate` raises `Interrupt` part-way (caller process) | The run is `aborted`, and `rollback` carries `step: X`, `compensated: false` and the arguments; the API shows `pending_compensation` |
+| `Reactor.undo(id)` | X's `compensate` runs again with the same arguments and a `RecordedFailure` reason (or the original string), before every undo; `cancelled` |
+| Interruption after X's `compensate` returned (during the undo stack) | Manual undo does not re-run `compensate`, and replays only the remaining entries |
+| Re-run `compensate` returns `Failure` | A `:failed_compensation` event and trace entry; the undo stack is still replayed |
+| X inside a composed child; X inside an inline map element | Re-run before that child's or element's undos |
+
+## 7. US7: `undo_all` (P3)
+
+```bash
+bundle exec rspec spec/map/map_undo_all_spec.rb
+bundle exec rspec spec/ruby_reactor/dsl/map_undo_all_dsl_spec.rb
+bundle exec rspec spec/map/map_undo_all_spec.rb --tag slow   # 10,000 elements (SC-007)
+```
+
+| Scenario | Observable outcome |
+| --- | --- |
+| Fan-out map of 20 elements with `undo_all`, a later step fails | One call with 20 results in index order; no element undo ran; no rollback jobs enqueued; the steps before the map undone after the call; `have_run_undo_all(:map).with_elements(20)` |
+| Atomic fan-out map, element 7 fails | Called with the completed results only; element 7 rolled itself back |
+| Inline map, same reactor | Same outcome |
+| The block raises | `have_rollback_failure(:map)` with `kind: :undo_all`; the steps before the map are still undone |
+| No element completed | Not called |
+| `Reactor.undo(id)` of a completed run | Called once |
+| Declared twice or without a block | `ValidationError` at class load |
+| 10,000 completed elements (slow) | One call; zero per-element undos; the process's memory is flat across the enumeration (measured as in 009 SC-001) |
+
+## 8. Full suite and style
+
+```bash
+bundle exec rspec
+bundle exec rubocop
+```
+
+## 9. Demo app (Constitution VI)
+
+Locally, without Docker, against test Redis DB 5:
+
+```bash
+cd demo_app
+REDIS_URL=redis://localhost:6780/5 RAILS_ENV=test bundle exec rspec \
+  spec/reactors/bulk_refund_demo_reactor_spec.rb \
+  spec/reactors/contended_approval_demo_reactor_spec.rb \
+  spec/reactors/dual_approval_demo_reactor_spec.rb
+REDIS_URL=redis://localhost:6780/5 bin/rails demo:rollback_follow_ups
+```
+
+**Docker acceptance**: use an isolated compose project, because container names are fixed and
+collide across worktrees.
+
+```bash
+docker compose -p rr_rollback_follow_ups -f docker-compose.yml -f <override.yml> up -d --build demo-redis demo-sidekiq
+docker compose -p rr_rollback_follow_ups -f docker-compose.yml -f <override.yml> run --rm --no-deps demo-app \
+  bash -c "bin/rails db:prepare && bin/rails demo:rollback_follow_ups"
+```
+
+Expected rake output, per task:
+
+- **`demo:map_undo_all`**: "bulk refund called once with N charges", then the run `failed` with
+  the steps before the map undone.
+- **`demo:contended_resume`**: "resume accepted (DispatchResult) while the lock is held", then
+  `completed` after release.
+- **`demo:concurrent_interrupts`**: "finance accepted (background); legal accepted while running",
+  then `completed`, with both approvals in the result.