ruby_reactor 0.8.6 → 0.8.8

data/specs/010-rollback-follow-ups/data-model.md

@@ -0,0 +1,139 @@
+# Data Model: Rollback and Resume Follow-ups
+
+**Feature**: [spec.md](spec.md) | **Research**: [research.md](research.md)
+
+All records live in Redis through `Storage::RedisAdapter`. `<Class>` is the reactor's storage name
+(`RubyReactor.reactor_storage_name`), and `<id>` is the run's context id.
+
+## 1. Run liveness lock (changed holders)
+
+`lock:async:<id>` is the existing re-entrant lock: Lua owner/count hash, TTL `context_lock_ttl`,
+auto-extended by its holder.
+
+| Holder | Since | Wait | Taken | Released |
+| --- | --- | --- | --- | --- |
+| `Executor#resume_execution`, `#resume_rollback` | 004/009 | 0 | after the reactor-level gates' rate-limit check | after the final save (009 R-13) |
+| `Reactor#undo` | 009 R-12 | 5s | before undoing; **now also before reloading** (R-09) | after the save |
+| **`Executor#execute`** (root, caller process) | **010 R-01** | 0 | after input validation, before the reactor-level lock and semaphore | **after the final save** |
+| **`Worker#perform`** | **010 R-02** | 2s, polling | **before the context is read** | in `ensure`, after the executor released its re-entry |
+| **`Reactor#continue`** | **010 R-05** | 0 | after the claim, **before the reload** | after the resume returns, or before the hand-off enqueue |
+
+- **Re-entry**: the Worker and `continue` pass their owner to the executor
+  (`Executor#context_lock_owner=`). The executor's acquire then raises the count instead of
+  contending.
+- **Inline job-testing mode**: no holder takes it (unchanged).
+- **Reader**: `Sweeper#run_once`, which is unchanged. It now sees caller-process runs as live.
+
+## 2. Interrupt resume claim (new)
+
+| Field | Value |
+| --- | --- |
+| Key | `reactor:<Class>:context:<id>:resume:<step>` |
+| Value | `ContextSerializer.serialize_value(payload)`, as JSON |
+| Created | `SET NX EX context_ttl`, by `Reactor#continue` after the payload has validated |
+| Read | `MGET` over the reactor's interrupt steps that have no result yet: `Executor#resume_execution` right after the context lock, and `Worker#perform` for a `paused` run |
+| Deleted | never; it expires with TTL |
+
+**Lifecycle** of one interrupt:
+
+```text
+absent ──continue (valid payload, SET NX wins)──▶ claimed ──lock owner applies──▶ applied
+  │                                                  │                              │
+  │                     second continue: SET NX loses ─▶ "already resumed"          │
+  └──────────────── continue after apply: the result exists ─▶ "already resumed" ◀──┘
+```
+
+- **Applied** means `context.set_result(step, payload)` by the execution that owns the lock,
+  persisted by that execution's next save.
+- An applied claim stays as the dedupe marker until it expires. After that, the step's result in
+  `intermediate_results` rejects duplicates.
+
+**Validation rules**:
+
+- A claim is written only for an interrupt step that is ready, has no result and has passed payload
+  validation (FR-008, FR-017, FR-020).
+- The run must be `paused` or `running`, and not `cancelled`.
+
+## 3. Interrupt attempt counter (moved)
+
+| Field | Value |
+| --- | --- |
+| Key | `reactor:<Class>:context:<id>:resume_attempts:<step>` |
+| Value | an integer, raised by `INCR` once per invalid payload, with `EXPIRE context_ttl` |
+| Replaces | `context.private_data[:interrupt_attempts][step]`, no longer written |
+
+At `step_config.max_attempts`, the run fails through `Reactor#undo(failure:)` (R-08, R-09). The
+`failure_reason` is unchanged: `message`, `step_name`, `errors`, `payload`, `step_arguments`,
+`attempts`, `validation_errors`.
+
+## 4. `context.rollback` on an aborted run (extended)
+
+There is no new context attribute. 009 added `rollback`, and 010 adds keys for the aborted case
+(R-10).
+
+| Key | Type | Meaning |
+| --- | --- | --- |
+| `trigger` | `"failure"` | a step failure was rolling back when the process was interrupted |
+| `step` | String | the failing step's name, at this context's level |
+| `compensated` | Boolean | whether that step's `compensate` returned. `false` makes manual undo re-run it |
+| `arguments` | serialized Hash | the step's `rollback_arguments`. **Its presence marks the aborted record**; 009 hand-off records do not carry it |
+| `error` | `{ "class", "message" }`, or a String | the failure that `compensate` receives again |
+| `failures` | serialized Array | rollback failures collected before the interruption (009) |
+
+**State transitions** (top level; a composed child or inline element behaves the same at its own
+level):
+
+```text
+running ──step fails──▶ (compensate running) ──interrupted──▶ aborted + rollback{compensated:false}
+                                                                 │
+                       Reactor.undo: lock, reload, merge record, undo_all
+                                                                 ▼
+           compensate re-run (once) ─▶ compensated:true ─▶ undo stack replayed ─▶ cancelled
+running ──step fails──▶ compensate returns ──interrupted during undo stack──▶ aborted (no record;
+                                                         manual undo replays the remaining stack only)
+```
+
+`Web::API` derives `pending_compensation: { step: }` from this record when the status is `aborted`
+and `compensated` is false (R-11).
+
+## 5. Map step declaration: `undo_all` (new)
+
+| Field | Where | Value |
+| --- | --- | --- |
+| `undo_all_block` | map `StepConfig#arguments`, source `Template::Value` | the block given to `undo_all`; absent when not declared |
+
+**Validation rules** (`MapBuilder`, at class definition): a block is required, and the method may be
+declared once per map. Either violation raises `Error::ValidationError` naming the map (FR-027).
+
+### Bulk undo inputs and outcome
+
+| Item | Fan-out map | Inline map |
+| --- | --- | --- |
+| Completed results | forward result slots, in index order, `Success` values only (`Map::ResultEnumerator`) | element contexts with status `completed`, in index order, one deserialized at a time |
+| Replayed per element instead | none (no element is `aborted`) | elements with status `aborted` (pass 1, `Map::ElementRollback`) |
+| Reported unavailable | indexes with no result slot | element contexts that expired, as in 009 |
+| Rollback records or jobs | none | none |
+
+Outcome entries:
+
+- **Execution trace**: `{ type: :undo_all, step: <map step>, count: <completed>, result: <value or error>, timestamp: }`.
+- **Rollback failure**, on a raise or a returned `Failure`:
+  `{ step: <map step>, kind: :undo_all, reason: :raised | :returned_failure, message: }`. It is
+  aggregated into the run's final failure like any rollback failure.
+
+## 6. Worker decision by status (R-06)
+
+| Stored status | Action |
+| --- | --- |
+| `completed`, `failed`, `cancelled`, `skipped`, `aborted`, `halted` | return (release the lock) |
+| `paused` | resume if any interrupt without a result has a claim; otherwise return |
+| `rolling_back` | `resume_rollback` |
+| `running` | `resume_execution` (applies claims first) |
+
+## 7. Snooze limit (R-07)
+
+| Condition | Limited by `lock_snooze_max_attempts`? |
+| --- | --- |
+| Reactor-level lock, semaphore or rate limit before admission (`!context.admitted?`) | yes (unchanged): escalate to `failed` |
+| Reactor-level lock or semaphore once admitted | **no**: one `ruby_reactor.resume.waiting` warning at the limit, then keep snoozing |
+| `ContextLockContention`, `OrderedLock::WaitError`, `ExecutionParked` | no (unchanged) |

data/specs/010-rollback-follow-ups/plan.md

@@ -0,0 +1,273 @@
+# Implementation Plan: Rollback and Resume Follow-ups
+
+**Branch**: `rollback_follow_ups` | **Date**: 2026-10-08 | **Spec**: [spec.md](spec.md)
+
+**Input**: Feature specification from `specs/010-rollback-follow-ups/spec.md`
+
+## Summary
+
+Seven follow-ups from 008, resolved with three shared mechanisms.
+
+1. **Every live run holds its liveness lock, and readers lock, then load** (US1, US2):
+   - A synchronous `Executor#execute` takes the run's `async:<id>` lock, auto-extended, and saves
+     before releasing it (R-01). The sweeper already skips locked runs, so a live caller-process run
+     is never re-enqueued. A killed one lapses within `context_lock_ttl`.
+   - `Worker#perform` takes the same lock **before** reading the run, waiting up to 2s, and hands
+     the owner to the executor, which re-enters the lock (R-02, R-03). A Worker resumed by a fan-out
+     map's completion can no longer race the caller's final save, in either order.
+2. **A resume is claimed per interrupt, and applied only by the run's lock owner** (US3, US4, US5):
+   - `continue` validates the payload in the calling process, then claims the interrupt with
+     `SET NX` (R-04). The claim holds the payload.
+   - It then tries to own the run: lock, reload, resume (R-05). Any path that cannot run now (the
+     run is busy, the reactor's lock or semaphore is held, a background resume) becomes one hand-off.
+     The Worker applies the claims under the lock, and `continue` returns a `DispatchResult`.
+   - The claim makes "exactly one resume per interrupt" hold in every mode, including inline
+     testing. Claiming per interrupt lets different interrupts be accepted while another resume
+     executes.
+   - Supporting changes:
+     - after admission, the snooze limit no longer escalates a run to `failed` without rollback
+       (R-07);
+     - interrupt attempts move to an atomic counter (R-08);
+     - `Reactor#undo` reloads under its lock (R-09).
+3. **Rollback state records what was cut off** (US6):
+   - An `aborted` run records its failing step's unfinished `compensate` (step, arguments, error) in
+     009's `context.rollback` (R-10).
+   - `Executor#undo_all` runs it before the undo stack. Manual undo, composed children and inline
+     elements all go through `undo_all`, so every depth is covered.
+   - The dashboard shows the outstanding `compensate` (R-11).
+
+Plus one new DSL method, **`map ... undo_all { |completed_results| }`** (US7). When it is declared,
+the map's rollback calls the block once with a lazy, index-ordered enumerable of completed
+results, instead of replaying each element (R-12, R-13):
+
+- a fan-out map reads its forward result slots and dispatches no rollback jobs;
+- an inline map reads element contexts one at a time, and still replays an `aborted` element
+  itself.
+
+## Technical Context
+
+**Language/Version**: Ruby >= 3.0
+
+**Primary Dependencies**:
+
+- Sidekiq and ActiveJob adapters, Redis, dry-validation.
+- Optional OpenTelemetry middleware.
+- GUI: React/Vite (`gui/`, built into `lib/ruby_reactor/web/public`).
+
+No new dependencies.
+
+**Storage**: Redis through `Storage::RedisAdapter` (see [data-model.md](data-model.md)).
+
+- New keys:
+  - `reactor:<Class>:context:<id>:resume:<step>`: the claim (`SET NX`, TTL `context_ttl`);
+  - `reactor:<Class>:context:<id>:resume_attempts:<step>`: the attempt counter (`INCR`).
+- `lock:async:<id>` gains three holders: `execute`, `Worker`, `continue`.
+- The `context.rollback` hash gains `arguments` and `error` for aborted runs.
+- The map `StepConfig` gains `undo_all_block`.
+- No migration. The one compatibility note: attempt counts restart for runs paused across the
+  upgrade (R-08).
+
+**Testing**: RSpec against real Redis (`redis://localhost:6780`), with Sidekiq fake mode plus
+`drain_async_jobs`.
+
+- **Concurrency specs** (US1, US4, US5): threads, a barrier or latch, real Redis.
+- **Interleaving spec** (US2): a public middleware hook drains jobs between the hand-off and the
+  caller's final save.
+- **Crash spec** (US1, FR-003): `fork` plus `SIGKILL`, tagged `:fork`.
+- **Inline mode**: one justified `Sidekiq::Testing.inline!` spec for US4 (FR-016).
+- **Scale**: `:slow` tag, 10,000 elements for `undo_all` (SC-007).
+- **Demo specs**: the shipped matchers only, plus the new `be_resume_deferred` and
+  `have_run_undo_all`.
+
+**Target Platform**: Ruby gem (MRI), Sidekiq/ActiveJob workers, Rails demo app in Docker.
+
+**Project Type**: Library (gem) with a bundled dashboard.
+
+**Performance Goals**:
+
+- **Synchronous run**: one lock acquire and release, and one extender thread, per run.
+- **Worker**: one extra lock round-trip, and usually no wait. It waits at most 2s, while a caller
+  finishes its hand-off.
+- **`continue`**: one `SET NX`, plus one `MGET` per resume for reactors that declare interrupts.
+- **`undo_all` over 10,000 elements**: one call. Memory is bounded by the enumerator's chunk
+  (1,000 result slots, or one element context).
+
+**Constraints**:
+
+- **Single writer**: `continue` writes the run's context only while holding its lock (J-5).
+- **Save before release** for every holder (J-3).
+- A claimed payload is never validated again (J-7).
+- An admitted run is never escalated to `failed` by snoozes (J-8).
+- Inline testing mode takes no context lock, as today (R-14).
+- Existing outcomes are unchanged for:
+  - uncontended resumes;
+  - background resumes;
+  - maps without `undo_all`;
+  - aborted runs whose `compensate` had finished.
+
+**Scale/Scope**:
+
+- About 14 library files touched. 2 new: `interrupt_claims.rb` and `error/recorded_failure.rb`.
+- About 10 new spec files and about 8 updated.
+- Demo: 4 reactor files, 3 rake tasks plus the `demo:rollback_follow_ups` group, 3 specs.
+- 6 documentation files, the README, CHANGELOG and `future_improvements.md`.
+- GUI: `ReactorDetail.tsx` (its API response is untyped, so `lib/reactors.ts` needs no change).
+
+## Constitution Check
+
+*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
+
+| Principle | Status | Note |
+| --- | --- | --- |
+| I. Gem-First Design | PASS | Everything is in `lib/` behind the existing DSL and `Reactor` API. One DSL method (`undo_all`). The Worker change is in the backend-agnostic `Worker` module, so Sidekiq and ActiveJob both get it. No host coupling. |
+| II. Saga Pattern Integrity | PASS (strengthened) | Closes four ways a run lost work or effects: a duplicate forward run (US1), overwritten progress (US2), a lost or double resume (US3–US5), and a `compensate` never finished after an interruption (US6). R-07 removes the snooze escalation that marked runs `failed` without rollback. `undo_all` keeps step-level order (P-6), and an `aborted` inline element still replays its own undos. |
+| III. Test-First with Real Infrastructure | PASS with one justified deviation | Each story's specs are written first and fail first, against real Redis. Concurrency is shown with threads, not mocks. The US2 interleaving uses a public middleware hook, not stubs. One new `inline!` spec, for FR-016, justified in Complexity Tracking. |
+| IV. Observability by Default | PASS | New key=value log lines: `resume.deferred`, `resume.waiting`, `map.rollback.undo_all.started` / `.completed`. A `:undo_all` trace entry. Dashboard: `pending_compensation` on aborted runs, and a "Resume accepted" API message. A re-run `compensate` fires the standard compensation middleware events. A hand-off never fires `:failed_reactor`. |
+| V. Simplicity & SemVer | PASS with notes | Three mechanisms, each serving several stories (Complexity Tracking). Code deleted: `reopen_paused` and the `past_gates?` contention branch in `continue`, and the blob write of attempts. MINOR (R-16), with CHANGELOG migration notes: `continue` returns a hand-off where it raised; "already resumed"; attempt counts restart; a synchronous run holds the liveness lock. No new config knobs. |
+| VI. Demo-App Proof of Feature | PASS (planned) | One reactor per file, all class-based steps (R-18): `BulkRefundDemoReactor` with `BulkRefundChargeReactor` (`undo_all`, fan-out, a later failure), `ContendedApprovalDemoReactor` (resume while locked), and `DualApprovalDemoReactor` (two interrupts at once). Rake: `demo:map_undo_all`, `demo:contended_resume`, `demo:concurrent_interrupts`, grouped as `demo:rollback_follow_ups` in `demo:all`. Specs use the shipped matchers. `TestSubject#resume(process_jobs:)`, `be_resume_deferred` and `have_run_undo_all` are added to `lib/ruby_reactor/rspec/`. No new Docker service or environment variable. |
+
+- [x] Documentation impact identified (research R-17):
+  - **README.md**: "Durability & Recovery", "Interrupts (Pause & Resume)", "Locks, Semaphores &
+    Ordered Locks" and "Map & Parallel Execution".
+  - **documentation/**: `interrupts.md`, `locks_and_semaphores.md`, `background_and_async.md`,
+    `data_pipelines.md`, `core_concepts.md`, `testing.md`.
+  - **demo_app/documentation/data_pipelines.md**.
+  - **CHANGELOG.md**.
+  - **specs/future_improvements.md**: remove the seven items, and update "Fenced context writes".
+  - Carried into tasks.md as a required task per user story.
+
+**Post-design re-check (after Phase 1)**: PASS.
+
+- New public surface:
+  - one DSL method;
+  - one changed return contract for `continue`, and one new rejection;
+  - two log events, plus the `undo_all` pair;
+  - one API field;
+  - three RSpec additions.
+- New internals:
+  - one small module (`InterruptClaims`) used by three callers;
+  - one error class;
+  - three storage methods.
+- The spec was revised for what the design fixed (see spec Status): FR-010, FR-017, FR-024, US6-AS3.
+
+## Project Structure
+
+### Documentation (this feature)
+
+```text
+specs/010-rollback-follow-ups/
+├── plan.md                        # This file
+├── research.md                    # Phase 0: R-01..R-19
+├── data-model.md                  # Phase 1: lock holders, claim, counter, rollback record, undo_all
+├── quickstart.md                  # Phase 1: validation guide
+├── contracts/
+│   ├── api-surface.md             # continue outcomes, undo on aborted, undo_all, logs, RSpec additions
+│   └── resume-protocol.md         # Invariants J-1..J-9, sequences P-1..P-6
+├── checklists/requirements.md
+└── tasks.md                       # Phase 2 (/speckit-tasks)
+```
+
+### Source Code (repository root)
+
+```text
+lib/ruby_reactor/
+├── interrupt_claims.rb                 # new: claim!(ctx, step, payload) (SET NX), unapplied(ctx) (MGET of
+│                                       #   interrupts without a result), apply!(ctx) → set_result (R-04, R-06)
+├── error/recorded_failure.rb           # new: StandardError carrying the original message + original_class (R-10)
+├── executor.rb                         # execute: acquire_context_lock (root, caller process); save before
+│                                       #   releasing it (R-01). context_lock_owner= for re-entry (R-03).
+│                                       #   resume_execution: InterruptClaims.apply! after the lock (R-04).
+│                                       #   mark_aborted: record the unfinished compensate in `rollback` (R-10).
+│                                       #   undo_all: compensate_pending! before the stack (R-10)
+├── executor/compensation_manager.rb    # @pending keeps rollback_arguments (R-10)
+├── reactor.rb                          # continue: validate → claim → lock → reload → resume, or hand off;
+│                                       #   delete reopen_paused (R-05). validate_continue_payload: attempt
+│                                       #   counter (R-08). undo: reload under lock, merge the aborted record,
+│                                       #   failure: kwarg (R-09, R-10)
+├── worker.rb                           # perform: lock, then load (R-02); status table (R-06); release in
+│                                       #   ensure; handle_snooze: uncapped once admitted, one
+│                                       #   resume.waiting warning (R-07)
+├── storage/redis_adapter.rb            # claim_interrupt_resume, retrieve_interrupt_resumes,
+│                                       #   increment_interrupt_attempts
+├── dsl/map_builder.rb                  # undo_all(&block): once, block required → undo_all_block (R-12)
+├── step/map_step.rb                    # compensate/undo: bulk_rollback when undo_all is declared (R-13)
+├── map/step_rollback.rb                # bulk_rollback: fan-out result slots / inline two passes; trace,
+│                                       #   logs, failure entry (R-13)
+├── web/api.rb                          # pending_compensation on aborted; continue → "Resume accepted" (R-11)
+└── rspec/{test_subject,matchers}.rb    # resume(process_jobs:), running runs; be_resume_deferred;
+                                        #   have_run_undo_all (R-18)
+
+gui/src/components/ReactorDetail.tsx    # outstanding-compensate line on aborted runs
+gui/src/lib/reactors.ts                 # pending_compensation type
+lib/ruby_reactor/web/public/            # rebuilt bundle
+
+spec/
+├── ruby_reactor/caller_process_liveness_spec.rb        # new: US1 (latch, long step, fork + SIGKILL)
+├── ruby_reactor/executor/caller_save_race_spec.rb      # new: US2 interleaving (middleware drain)
+├── ruby_reactor/worker_lock_then_load_spec.rb          # new: R-02, R-06 status table
+├── ruby_reactor/worker_snooze_admitted_spec.rb         # new: R-07
+├── ruby_reactor/interrupts/contended_resume_spec.rb    # new: US3
+├── ruby_reactor/interrupts/resume_claim_spec.rb        # new: US4, including the one inline! example
+├── ruby_reactor/interrupts/concurrent_interrupts_spec.rb  # new: US5
+├── ruby_reactor/dsl/map_undo_all_dsl_spec.rb           # new: R-12
+├── map/map_undo_all_spec.rb                            # new: US7 (fan-out, inline, atomic, undo, :slow)
+├── ruby_reactor/rollback/aborted_execution_spec.rb     # updated: US6 at top level, compose, inline element
+├── ruby_reactor/rollback/resume_guard_spec.rb          # updated: contention defers instead of raising
+├── ruby_reactor/sweeper_spec.rb                        # updated: caller-process runs skipped
+├── ruby_reactor/context_lock_spec.rb                   # updated: execute holds the lock; re-entry by owner
+├── ruby_reactor/interrupt_background_resume_spec.rb    # updated: claim-based background resume
+├── integration/interrupt_max_attempts_spec.rb          # updated: counter record; failure under lock
+├── ruby_reactor/web/*_spec.rb                          # updated: pending_compensation, continue message
+└── ruby_reactor/rspec/*_spec.rb                        # new or updated: resume(process_jobs:), new matchers
+
+demo_app/
+├── app/reactors/bulk_refund_demo_reactor.rb            # fan-out map with undo_all; a later step fails
+├── app/reactors/bulk_refund_charge_reactor.rb          # element reactor (class-based charge step)
+├── app/reactors/contended_approval_demo_reactor.rb     # with_lock + interrupt
+├── app/reactors/dual_approval_demo_reactor.rb          # finance (resume: :background) + legal interrupts
+├── lib/tasks/demo_reactors.rake  # demo:map_undo_all, :contended_resume, :concurrent_interrupts,
+│                                 #   :rollback_follow_ups (in demo:all)
+└── spec/reactors/{bulk_refund,contended_approval,dual_approval}_demo_reactor_spec.rb
+
+README.md, documentation/*.md, demo_app/documentation/data_pipelines.md, CHANGELOG.md,
+specs/future_improvements.md   # per R-17
+```
+
+**Structure Decision**: single gem project, existing `lib/ruby_reactor` layout. New interrupt specs
+go in a new `spec/ruby_reactor/interrupts/` folder. Rollback specs stay in
+`spec/ruby_reactor/rollback/`, and map specs in `spec/map/`.
+
+## Delivery Order
+
+The stories are independently testable. tasks.md is the authoritative order.
+
+1. **Foundation** (blocks US2, US3 and US5):
+   - `Executor#context_lock_owner=` (R-03);
+   - `Worker#perform` lock, then load, plus the status table (R-02, R-06);
+   - `Reactor#undo` reload under its lock (R-09).
+2. **US1 (P1)**: `execute` takes and releases the liveness lock, saving first (R-01). Specs: latch,
+   long step, fork kill.
+3. **US2 (P1)**: the interleaving spec, which passes with foundation plus US1. The inline-`continue`
+   variant lands with US3.
+4. **Claims foundation** (blocks US3–US5): storage methods, `InterruptClaims`, and `apply!` in
+   `resume_execution` (R-04).
+5. **US3 (P1)**: the `continue` flow with contention hand-off (R-05), the uncapped snooze after
+   admission (R-07), and the `resume_guard_spec` update.
+6. **US4 (P2)**: claim race specs, including the one `inline!` example.
+7. **US5 (P2)**: accept on `running` (R-05 step 1), the attempt counter (R-08), and the concurrency
+   specs.
+8. **US6 (P2)**: `@pending` arguments, the `mark_aborted` record, `undo_all`'s
+   `compensate_pending!`, the `Reactor#undo` merge (R-10), and the API and GUI (R-11).
+9. **US7 (P3)**: the DSL (R-12), `bulk_rollback` (R-13), and the matchers.
+10. **Demo, docs, CHANGELOG** per story. Close with the Docker acceptance run of
+    `demo:rollback_follow_ups`.
+
+## Complexity Tracking
+
+| Addition | Why needed | Simpler alternative rejected because |
+| --- | --- | --- |
+| Resume claim record per interrupt (`SET NX`) | FR-015 and FR-016: exactly one resume per interrupt **in every mode**. FR-017: different interrupts accepted concurrently. Also stores the payload until the lock owner applies it, so `continue` never writes the blob unlocked (J-5). | **The context lock** is skipped in inline mode. **A status compare-and-set in the blob** serializes all resumes of the run, which blocks US5, and still writes from outside the owning execution. |
+| The Worker takes the liveness lock before loading (wait ≤ 2s) | FR-005: closes the caller-save race in both orders. It also fixes the Workers enqueued by the rollback hand-off handshake and by `Reactor#undo` while holding the lock. | **Peek-then-snooze** leaves a check-then-read gap and adds a full Sidekiq poll (about 5s) to every hand-off from a caller process. **Full fenced writes** are out of scope (spec Assumptions). |
+| Uncapped snooze once admitted (R-07) | FR-010. Escalation marks a run with completed work `failed` **without rollback**, a Constitution II violation that also hits `resume: :background` today. | **Escalation with rollback** fails a run the user accepted, because of an unrelated holder. **Keeping the cap** strands an accepted resume. A one-time warning keeps leaked semaphores visible. |
+| One new `Sidekiq::Testing.inline!` spec, a deviation from Constitution III | FR-016 requires US4 to hold in inline job-testing mode, the mode that skips the run lock. That is the original 008 gap. | **Fake mode plus drain** cannot show it: the context lock is taken there, so the race under test never opens. Scoped to one example. |
+| Attempt counter moved to its own key (R-08) | Once `continue` accepts resumes of a `running` run (US5), the old blob save of the attempt count, from an unlocked snapshot, would overwrite the executing resume's progress. | **Counting under the run's lock** would make an invalid-payload response wait behind, or contend with, a live resume. **Not counting while running** makes `max_attempts` depend on timing. |

data/specs/010-rollback-follow-ups/pr-description.md

@@ -0,0 +1,110 @@
+# 010 Rollback and Resume Follow-ups
+
+Closes the seven "Rollback follow-ups (008)" items from `specs/future_improvements.md`.
+Spec: `specs/010-rollback-follow-ups/spec.md` · Plan: `plan.md` · Decisions: `research.md` (R-01..R-19).
+
+## What changes
+
+| Story | Fix |
+| --- | --- |
+| US1 (P1): the sweeper re-ran live caller-process runs | `Executor#execute` holds the run's `async:<id>` liveness lock, auto-extended, and saves before releasing it (R-01). The sweeper is unchanged: it already skips locked runs. A killed process is recovered once the lock lapses. |
+| US2 (P1): a caller's final save overwrote a worker's progress | With US1's lock, plus `Worker#perform` takes the run's lock **before** it reads the run (waiting up to 2s), and hands its owner to the executor, which re-enters the lock (R-02, R-03). |
+| US3 (P1): a resume contended on the reactor's lock/semaphore raised and was lost | `continue` validates the payload, claims the interrupt, then either owns the run (lock, then reload, then resume inline) or hands the resume to a worker and returns a `DispatchResult` (R-05). After admission, `lock_snooze_max_attempts` no longer escalates a run to `failed` without rollback; it warns once with `ruby_reactor.resume.waiting` (R-07). |
+| US4 (P2): two resumes of one interrupt could both run | Per-interrupt claim, `SET NX` on `reactor:<Class>:context:<id>:resume:<step>`, in every mode, including `Sidekiq::Testing.inline!` (R-04). The loser raises "already resumed" and writes nothing. |
+| US5 (P2): a resume for another interrupt was rejected while the run executed | A `running` run accepts a claim for a ready, unexecuted interrupt and hands it to a Worker. The Worker applies claims under the run's lock at resume start, and resumes a `paused` run only when it has an unapplied claim (R-06). Attempt counts move to their own `INCR` record, so `continue` never writes the run unlocked (R-08). `Reactor#undo` reloads under its lock (R-09). |
+| US6 (P2): a cut-off `compensate` of the failing step was never re-run | `mark_aborted` records the failing step, its arguments and its reason in `context.rollback`. `Executor#undo_all` re-runs it before the undo stack, so this works at any depth (composed child, inline map element). The dashboard API adds `pending_compensation`, and `ReactorDetail` shows it (R-10, R-11). |
+| US7 (P3): map-level `undo_all` | `map ... undo_all { |completed_results| }` is called once per rollback with a lazy, index-ordered Enumerable. A fan-out map reads its result slots 1,000 at a time and dispatches no rollback jobs. An inline map reads element contexts one at a time, and `aborted` elements still replay their own steps (R-12, R-13). |
+
+New RSpec surface:
+
+- `TestSubject#resume(process_jobs:)`, which also accepts `running` subjects;
+- `be_resume_deferred`;
+- `have_run_undo_all(:map).with_elements(n)`.
+
+Demo (Constitution VI):
+
+- `BulkRefundDemoReactor` with `BulkRefundChargeReactor`, `ContendedApprovalDemoReactor` and
+  `DualApprovalDemoReactor`;
+- rake `demo:map_undo_all`, `demo:contended_resume` and `demo:concurrent_interrupts`, grouped as
+  `demo:rollback_follow_ups` in `demo:all`;
+- matcher-only specs.
+
+Docs:
+
+- README: Durability, Interrupts, Locks, Map;
+- `documentation/`: `interrupts.md`, `locks_and_semaphores.md`, `background_and_async.md`,
+  `data_pipelines.md`, `core_concepts.md`, `testing.md`;
+- `demo_app/documentation/data_pipelines.md`;
+- CHANGELOG;
+- `specs/future_improvements.md`.
+
+## Migration notes (also in CHANGELOG)
+
+1. `continue` returns a `DispatchResult` where it raised `Lock::AcquisitionError` /
+   `Semaphore::AcquisitionError`; a `rescue` of those simply stops firing.
+2. A second resume of the same interrupt raises `ValidationError` "already resumed".
+3. A resume for another ready interrupt of a `running` run is accepted (`DispatchResult`).
+4. Interrupt attempt counts restart for runs paused across the upgrade.
+5. A synchronous `Reactor.run` holds the run's liveness lock; `Reactor.undo` of such a live run
+   waits, then raises `Lock::AcquisitionError`.
+
+## Verification
+
+- **Baseline (T002)**: `bundle exec rspec` on `5ca7d1d1`: 1479 examples, 0 failures, 2 pending.
+- **After (T072)**: 1570 examples, 0 failures, 2 pending (the same two pending: the 009
+  interrupt-inside-a-composed-child examples). Run on a dedicated test Redis DB
+  (`RUBY_REACTOR_TEST_REDIS_URL=redis://localhost:6780/11`).
+- **Tagged (T073)**: `--tag slow` (10,000-element `undo_all`: one call, memory flat across the
+  enumeration, about 0.16s for the call) and `--tag fork` (a `SIGKILL`ed caller process is
+  recovered): both pass.
+- **`bundle exec rubocop`**: no offenses. **GUI**: `vitest` 69 tests pass; the bundle is rebuilt
+  into `lib/ruby_reactor/web/public`.
+- **Demo app (T075)**: 152 examples, 0 failures (local, test Redis DB 5).
+- **Docker acceptance (T076)**: in an isolated compose project, `bin/rails demo:rollback_follow_ups`:
+  - `demo:map_undo_all`: bulk refund called once with 12 charges, 0 per-element refunds, run
+    `failed`. ✅
+  - `demo:contended_resume`: resume accepted (`DispatchResult`) while the lock was held, run
+    `completed` after release. ✅
+  - `demo:concurrent_interrupts`: finance accepted (background), legal accepted while running,
+    `completed` with both approvals. ✅
+
+### `demo:all`: failures that are already on `main`
+
+`demo:all` runs every demo against one Redis that is flushed once, with one Sidekiq process. On
+`main` (run the same way, in a separate compose project) it already fails:
+
+- `distributed_map_rollback` (70 charges for 40 orders);
+- `composed_fan_out`, happy path and rollback (8 shipped instead of 5);
+- `default_batch_size`;
+- one `async_step` unit wait;
+- `AsyncReactorDemoReactor(user_7)` (async wait timeout).
+
+On this branch the same set fails, minus `default_batch_size` and the `async_step` unit. Each of
+those demos passes when run alone (checked `demo:map_execution_undo` and
+`demo:async_reactor_demo` on this branch). The three new tasks wait up to 120s, so they pass
+inside `demo:all` too. The `demo:all` interference is out of scope here and worth its own issue.
+
+## Regression proofs (red before green)
+
+- **US1 (T014)**: with `Executor#execute`'s liveness lock removed, 5 of the 7 examples in
+  `caller_process_liveness_spec.rb` fail.
+- **US2 (T018)**: with the same line removed, the owner Worker started by the map's completion
+  writes the run four times while the caller is still inside `execute`. With it, the Worker
+  snoozes without reading or writing, and the run finishes once.
+- **US4 (T032)**: with the claim disabled, the 200-iteration race accepts both resumes.
+- **US6 (T046)**: with `compensate_pending!` disabled, 5 of the 9 examples fail.
+- **The inline payload (found by the demo suite)**: with the caller's payload not applied as
+  given, a string-keyed payload reached the step symbolized (`WebhookInterruptReactor` was
+  rejected). Guarded by `interrupt_claims_spec.rb` ("as given, string keys included").
+
+## Deviations from tasks.md
+
+- **Spec placement**:
+  - T015's example lives in `caller_process_liveness_spec.rb`;
+  - T046 and T047 in a new `rollback/aborted_compensate_spec.rb`.
+- **No `lib/reactors.ts` type for `pending_compensation`** (T054): the detail response is
+  untyped.
+- **`continue` on a `running` run always hands off.** It never tries the lock or runs another
+  execution's run inline (research R-05, updated).
+- **The `ruby_reactor.resume.waiting` line carries `error`** (whose message names the key), not a
+  separate `key` field (contracts updated).